github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-12-02 11:06:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,459 Commits 60 Branches 51 Tags
e99edfcb51df48dec17498c60bbbd06baa293c22
Commit Graph

608 Commits

Author SHA1 Message Date
Boyuan Yao
1df98d5b66 [autoparallel] add rotor C version (#1658) 
* [autoparallel] add rotor c version

* [fx] remove metainfoprop in rotor solver

* [autoparallel] modify C
 code format

* [autoparallel] remove build.py

* [autoparallel] fix C extension build

* [autoparallel] add C solver consistency test

* [autoparallel] remove some unused imports

* [autoparallel] refactor rotor solver code

* [autoparallel] replace print with colossalai logger

* [autoparallel] ranks fixed
 2022-10-03 17:13:30 +08:00
YuliangLiu0306
11ec070e53 [hotfix]unit test (#1670) 2022-09-29 12:49:28 +08:00
Frank Lee
a60024e77a [autoparallel] added utils for broadcast operation (#1665) 
* [autoparallel] added utils for broadcast operation

* polish code
 2022-09-29 11:22:29 +08:00
YuliangLiu0306
3f068d1409 [autoparallel] update CommSpec (#1667) 2022-09-29 11:20:59 +08:00
YuliangLiu0306
746f8f979d [autoparallel] add batch norm handler v2 (#1666) 2022-09-29 11:02:49 +08:00
Kirigaya Kazuto
9708638ded [pipeline/pytree] add pytree to process args and kwargs | provide data_process_func to process args and kwargs after forward (#1642) 
* [pipeline/tuning] improve dispatch performance both time and space cost

* [pipeline/converge] add interface for testing convergence

* [NFC] polish colossalai/utils/multi_tensor_apply/multi_tensor_apply.py code style

* Update PipelineBase.py

* [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera

* [pipeline/chimera] test chimera | fix bug of initializing

* [pipeline/pytree] add pytree to process args and kwargs | provide  to process args and kwargs after forward
 2022-09-29 10:58:58 +08:00
Frank Lee
3a4d6f63a8 [autoparallel] added node handler for bmm (#1655) 2022-09-28 11:32:16 +08:00
YuliangLiu0306
095854477f [autoparallel] add conv handler v2 (#1663) 2022-09-28 11:24:59 +08:00
YuliangLiu0306
1e7816a460 [autoparallel] adapt solver with gpt (#1653) 2022-09-28 11:17:26 +08:00
Frank Lee
30e50c8b4a [autoparallel] implemented all matmul strategy generator (#1650) 2022-09-27 12:06:25 +08:00
YuliangLiu0306
03978aad45 [autoparallel] change the following nodes strategies generation logic (#1636) 
* [autoparallel] change the following nodes strategies generation logic

* fix unit test
 2022-09-27 11:20:52 +08:00
YuliangLiu0306
59f100510a [autoparallel] where handler (#1651) 
* [autoparallel] where handler

* fix unit test
 2022-09-27 11:20:43 +08:00
Boyuan Yao
5d0fdb9cb4 [fx] fix offload codegen test (#1648) 
* [fx] fix offload codegen test

* [fx] modify typing
 2022-09-27 10:25:27 +08:00
Frank Lee
45b39a692a [autoparallel] implemented linear projection strategy generator (#1639) 2022-09-26 16:58:14 +08:00
Frank Lee
154d3ef432 [fix] fixed the collective pattern name for consistency (#1649) 
* [fix] fixed the collective pattern name for consistency

* polish code
 2022-09-26 16:39:37 +08:00
YuliangLiu0306
b2b2a4af98 [autoparallel] adapt solver with mlp (#1638) 2022-09-26 15:26:14 +08:00
Jiarui Fang
c5d39215f6 Revert "[feature] new zero implementation (#1623)" (#1643) 
This reverts commit 5be118f405.
 2022-09-26 10:06:03 +08:00
HELSON
5be118f405 [feature] new zero implementation (#1623) 2022-09-24 19:58:18 +08:00
HELSON
95c35f73bd [moe] initialize MoE groups by ProcessGroup (#1640) 2022-09-23 17:20:41 +08:00
HELSON
a088022efc [moe] fix moe bugs (#1633) 2022-09-23 15:33:57 +08:00
YuliangLiu0306
702dbc5288 [tensor] use communication autograd func (#1617) 
* [tensor] use communication autograd func

* change all to all comm spec info

* rename pattern and distinguish fwd/bwd

* polish code
 2022-09-23 13:31:15 +08:00
YuliangLiu0306
0c703189b9 [autoparallel] add layernorm handler (#1629) 2022-09-23 12:00:25 +08:00
YuliangLiu0306
bf77d3ab65 [autoparallel] recover the merged node strategy index (#1613) 2022-09-23 11:52:42 +08:00
Boyuan Yao
d6b01feb66 [fx] Modify offload codegen (#1618) 
* [fx] modify offload codegen

* [fx] remove repeated hook definitions

* [fx] modify offload test
 2022-09-23 11:04:52 +08:00
YuliangLiu0306
9eae855408 [hotfix] add recompile after graph manipulatation (#1621) 2022-09-23 11:00:33 +08:00
Super Daniel
d967779a32 [fx/profiler] tuned the calculation of memory estimation (#1619) 
* [fx] tuned the meta info and rotor solver.

* [fx] remove import.

* [fx] remove import.

* [fx] remove import.

* [fx] tune the meta calculations.

* [fx] polish comments.

* [fx] remove assertions.

* [fx] modify test cases.

* [fx] modify test cases.

* [fx] optimize import.

* [fx
 2022-09-23 10:59:47 +08:00
HELSON
f7f2248771 [moe] fix MoE bugs (#1628) 
* remove forced FP32 modules

* correct no_shard-contexts' positions
 2022-09-22 13:56:30 +08:00
Jiarui Fang
38c68b5b9a [embedding] rollback for better FAW performance (#1625) 2022-09-22 11:16:25 +08:00
Frank Lee
d925122020 [autoparallel] added new linear module handler (#1616) 2022-09-21 12:23:21 +08:00
Kirigaya Kazuto
170fa81095 [pipeline/chimera] test chimera | fix bug of initializing (#1615) 
* [pipeline/tuning] improve dispatch performance both time and space cost

* [pipeline/converge] add interface for testing convergence

* [NFC] polish colossalai/utils/multi_tensor_apply/multi_tensor_apply.py code style

* Update PipelineBase.py

* [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera

* [pipeline/chimera] test chimera | fix bug of initializing
 2022-09-20 18:00:39 +08:00
Jiarui Fang
504ff1d101 [embeddings] use cache_ratio instead of cuda_row_num (#1611) 2022-09-20 14:33:04 +08:00
YuliangLiu0306
7d1bb71d5d [fx] PoC of runtime shape consistency application (#1607) 
* [fx] PoC of runtime shape consistency application

* polish code
 2022-09-20 14:00:04 +08:00
YuliangLiu0306
47b11c432c [autoparallel]add bcast matmul strategies (#1605) 2022-09-20 11:26:21 +08:00
Boyuan Yao
933b6c6367 [fx] Add pofo solver (#1608) 
* [fx] add pofo algorithm

* [fx] Add pofo solver

* [fx] code refactor

* [fx] fix test_linearize import
 2022-09-20 11:20:48 +08:00
Kirigaya Kazuto
edc9e419ad [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera (#1595) 
* [pipeline/tuning] improve dispatch performance both time and space cost

* [pipeline/converge] add interface for testing convergence

* [NFC] polish colossalai/utils/multi_tensor_apply/multi_tensor_apply.py code style

* Update PipelineBase.py

* [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera
 2022-09-19 11:44:18 +08:00
YuliangLiu0306
eac1b79371 [autoparallel] add bcast op handler (#1600) 
* [autoparallel] add bcast op handler

* polish code

* add more BCAST FUNC OP

* polish code

* add exception handler

* polish
 2022-09-16 11:33:01 +08:00
Boyuan Yao
a7cda6f57d [fx] Add offload codegen (#1598) 
* [fx] add input activation offload to codegen

* [fx] modify unit test

* [fx] remove two skips in torch11

* [fx] use all_input_nodes instead of _input_nodes
 2022-09-14 15:49:06 +08:00
Super Daniel
c8e9b2ad78 [hotfix/rotor] fix variable names (#1597) 
* [fx] add some comment and docstrings.

* [fx] add dataflow analysis for an autograd graph.

* add intepretation for graph analysis.

* [fx] before doing save_tensor_hooks.

* [fx] provide an accurate estimation of memory except for GPT-2.

* [fx] provide an accurate estimation of memory except for GPT-2.

* [fx] provide an accurate estimation of memory except for GPT-2.

* [fx] a very accurate version on GPT-2.

* [fx] refactor code.

* [fx] remove redundant inplace=True.

* [fx] refactor code.

* [fx] refactor code.

* [fx] refactor code.

* [fx] dive into backward memory.

* [fx] fix variable names in ckpt_solvers and unskip tests.

* [fx] commit my changes.

* [fx] restore skips.

* [fx] restore skips.

* [fx] chaange stage into phase.

* [fx] chaange stage into phase.

* [fx] chaange stage into phase.
 2022-09-14 14:27:04 +08:00
YuliangLiu0306
faa23b9d9a [autoparallel] add reshape handler (#1594) 
* [autoparallel] add reshape handler

* polish code
 2022-09-14 10:25:45 +08:00
Frank Lee
27fe8af60c [autoparallel] refactored shape consistency to remove redundancy (#1591) 
* [autoparallel] refactored shape consistency to remove redundancy

* polish code

* polish code

* polish code
 2022-09-13 18:30:18 +08:00
YuliangLiu0306
d164449d00 [autoparallel] add resnet autoparallel unit test and add backward weight communication cost (#1589) 2022-09-13 18:05:05 +08:00
Frank Lee
219f66c571 [autoparallel] added solver option dataclass (#1588) 2022-09-13 14:47:09 +08:00
YuliangLiu0306
82d4376c23 [autoparallel] adapt solver with resnet (#1583) 
* [autoparallel]adapt solver with resnet

* polish code

* polish code
 2022-09-13 12:07:09 +08:00
CsRic
f3403ff98e [embeddings] add already_split_along_rank flag for tablewise mode (#1584) 2022-09-13 10:50:34 +08:00
Boyuan Yao
f3687e4ee2 [fx] Add nested checkpoint in activation checkpoint codegen (#1585) 
* [fx] add nested activation_checkpoint codegen

* undo algorithms commits

* solver

* undo some commits

* [fx] torch11 add nested activation checkpoint codegen

* remove some imports

* [fx] add some comments in activation codegen

* [fx] codegen instance error fix
 2022-09-12 20:00:48 +08:00
アマデウス
e615cfc3a8 [NFC] polish test component gpt code style (#1567) 2022-09-08 16:34:09 +08:00
Kirigaya Kazuto
6159d45417 [pipeline/tuning] improve dispatch performance both time and space cost (#1544) 2022-09-07 19:01:06 +08:00
Super Daniel
4f59693207 [fx] provide a stable but not accurate enough version of profiler. (#1547) 
* [fx] compute memory stat and flop count for MetaInfoProp.

* [fx] modify node attribute.

* [fx] modify ckpt_chen.

* [fx] fix compatibility.

* [fx] fix import error.

* [fx] skip test for MetaInfoProp.

* [fx] skip test for MetaInfoProp.

* [fx] skip test for MetaInfoProp.

* [fx] skip test for MetaInfoProp.

* [fx] skip if torch 1.11.0.

* [fx] recover MetaInfoProp support for PyTorch 1.11.

* [fx] provide a stable but not accurate enough version of profiler.

* [fx] provide a stable but not accurate enough version of profiler.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix compatibility in tests.

* [fx] fix import error.
 2022-09-07 11:21:04 +08:00
YuliangLiu0306
0908d0fc61 [autoparallel]add backward cost info into strategies (#1524) 2022-09-07 11:19:00 +08:00
YuliangLiu0306
44c866a3e3 [autoparallel] change the merge node logic (#1533) 2022-09-07 11:18:19 +08:00