github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-08-30 23:56:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,335 Commits 58 Branches 51 Tags
c7925c5d089d9927d93525a92180a3b66c649091
Commit Graph

1335 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kirigaya Kazuto
9708638ded [pipeline/pytree] add pytree to process args and kwargs | provide data_process_func to process args and kwargs after forward (#1642) 
* [pipeline/tuning] improve dispatch performance both time and space cost

* [pipeline/converge] add interface for testing convergence

* [NFC] polish colossalai/utils/multi_tensor_apply/multi_tensor_apply.py code style

* Update PipelineBase.py

* [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera

* [pipeline/chimera] test chimera | fix bug of initializing

* [pipeline/pytree] add pytree to process args and kwargs | provide  to process args and kwargs after forward
 2022-09-29 10:58:58 +08:00
YuliangLiu0306
c27e701cb2 [autoparallel] remove no strategy nodes (#1652) 
* [autoparallel] remove no strategy nodes

* fix none object iteration issue
 2022-09-29 10:43:25 +08:00
Frank Lee
50f16a2850 [autoparallel] added compute resharding costs for node handler (#1662) 2022-09-28 19:55:44 +08:00
Frank Lee
9ec401a722 [autoparallel] added new strategy constructor template (#1661) 
* [autoparallel] added new strategy constructor template

* polish code
 2022-09-28 14:01:36 +08:00
Frank Lee
3a4d6f63a8 [autoparallel] added node handler for bmm (#1655) 2022-09-28 11:32:16 +08:00
YuliangLiu0306
095854477f [autoparallel] add conv handler v2 (#1663) 2022-09-28 11:24:59 +08:00
YuliangLiu0306
1e7816a460 [autoparallel] adapt solver with gpt (#1653) 2022-09-28 11:17:26 +08:00
Jiarui Fang
c638bec028 [embedding] polish async copy (#1657) 2022-09-27 14:37:03 +08:00
Jiarui Fang
988570e4a6 [embedding] add more detail profiling (#1656) 2022-09-27 13:43:59 +08:00
Jiarui Fang
e1f97fd2b8 [embedding] print profiling results (#1654) 2022-09-27 12:50:33 +08:00
Frank Lee
30e50c8b4a [autoparallel] implemented all matmul strategy generator (#1650) 2022-09-27 12:06:25 +08:00
YuliangLiu0306
03978aad45 [autoparallel] change the following nodes strategies generation logic (#1636) 
* [autoparallel] change the following nodes strategies generation logic

* fix unit test
 2022-09-27 11:20:52 +08:00
YuliangLiu0306
59f100510a [autoparallel] where handler (#1651) 
* [autoparallel] where handler

* fix unit test
 2022-09-27 11:20:43 +08:00
Super Daniel
6135e178b3 [fx] refactor code for profiler / enable fake tensor movement. (#1646) 
* [fx/profiling] provide summary for MetaInfoProp.

* [fx/profiler] provide a table of summary.

* [fx/profiler] provide a table of summary.

* [fx/profiler] provide a table of summary.

* [fx/profiler] provide a table of summary.

* [fx] optimize table repr.

* [fx] optimize table repr.

* [fx] refactor code for profiler.

* [fx] add docstring.

* [fx] add docstring.

* [fx] skip test.

* [fx] redo.

* [fx] redo.

* [fx] fix import error for torch11.

* [fx] fix import error for torch11.
 2022-09-27 10:26:52 +08:00
Boyuan Yao
5d0fdb9cb4 [fx] fix offload codegen test (#1648) 
* [fx] fix offload codegen test

* [fx] modify typing
 2022-09-27 10:25:27 +08:00
Frank Lee
45b39a692a [autoparallel] implemented linear projection strategy generator (#1639) 2022-09-26 16:58:14 +08:00
Frank Lee
154d3ef432 [fix] fixed the collective pattern name for consistency (#1649) 
* [fix] fixed the collective pattern name for consistency

* polish code
 2022-09-26 16:39:37 +08:00
YuliangLiu0306
b2b2a4af98 [autoparallel] adapt solver with mlp (#1638) 2022-09-26 15:26:14 +08:00
Jiarui Fang
04443605a5 [embedding] non-blocking cpu-gpu copy (#1647) 2022-09-26 14:57:57 +08:00
CsRic
0767f67a0f [embedding] isolate cache_op from forward (#1645) 
Co-authored-by: ric <mkkt_bkkt@mail.ustc.edu.cn>
 2022-09-26 11:18:59 +08:00
Jiarui Fang
c5d39215f6 Revert "[feature] new zero implementation (#1623)" (#1643) 
This reverts commit 5be118f405.
 2022-09-26 10:06:03 +08:00
HELSON
5be118f405 [feature] new zero implementation (#1623) 2022-09-24 19:58:18 +08:00
Boyuan Yao
f921733621 [autoparallel] Add pofo sequence annotation (#1637) 
* [autoparallel] annotate pofo sequence

* [autoparallel] remove unused print

* [autoparallel] fix some code
 2022-09-24 01:52:57 +08:00
Super Daniel
04bbabeea8 [fx/profiler] provide a table of summary. (#1634) 
* [fx/profiling] provide summary for MetaInfoProp.

* [fx/profiler] provide a table of summary.

* [fx] optimize table repr.
 2022-09-23 18:12:43 +08:00
HELSON
95c35f73bd [moe] initialize MoE groups by ProcessGroup (#1640) 2022-09-23 17:20:41 +08:00
Jiarui Fang
e57df80325 [embeddings] cache option (#1635) 2022-09-23 16:40:18 +08:00
HELSON
a088022efc [moe] fix moe bugs (#1633) 2022-09-23 15:33:57 +08:00
YuliangLiu0306
702dbc5288 [tensor] use communication autograd func (#1617) 
* [tensor] use communication autograd func

* change all to all comm spec info

* rename pattern and distinguish fwd/bwd

* polish code
 2022-09-23 13:31:15 +08:00
YuliangLiu0306
c7ac0f4ab2 [autoparallel] add elementwise handler (#1622) 
* [autoparallel] add elementwise handler

* polish code

* polish code

* reduce skipped strategies range

* polish code
 2022-09-23 13:27:31 +08:00
YuliangLiu0306
3a46215135 [autoparallel] add embedding handler (#1620) 2022-09-23 12:34:30 +08:00
YuliangLiu0306
69448f64c4 [autoparallel] protect bcast handler from invalid strategies (#1631) 2022-09-23 12:12:49 +08:00
YuliangLiu0306
0c703189b9 [autoparallel] add layernorm handler (#1629) 2022-09-23 12:00:25 +08:00
YuliangLiu0306
bf77d3ab65 [autoparallel] recover the merged node strategy index (#1613) 2022-09-23 11:52:42 +08:00
Boyuan Yao
d6b01feb66 [fx] Modify offload codegen (#1618) 
* [fx] modify offload codegen

* [fx] remove repeated hook definitions

* [fx] modify offload test
 2022-09-23 11:04:52 +08:00
YuliangLiu0306
9eae855408 [hotfix] add recompile after graph manipulatation (#1621) 2022-09-23 11:00:33 +08:00
Super Daniel
d967779a32 [fx/profiler] tuned the calculation of memory estimation (#1619) 
* [fx] tuned the meta info and rotor solver.

* [fx] remove import.

* [fx] remove import.

* [fx] remove import.

* [fx] tune the meta calculations.

* [fx] polish comments.

* [fx] remove assertions.

* [fx] modify test cases.

* [fx] modify test cases.

* [fx] optimize import.

* [fx
 2022-09-23 10:59:47 +08:00
HELSON
f7f2248771 [moe] fix MoE bugs (#1628) 
* remove forced FP32 modules

* correct no_shard-contexts' positions
 2022-09-22 13:56:30 +08:00
Jiarui Fang
38c68b5b9a [embedding] rollback for better FAW performance (#1625) 2022-09-22 11:16:25 +08:00
Frank Lee
d925122020 [autoparallel] added new linear module handler (#1616) 2022-09-21 12:23:21 +08:00
Kirigaya Kazuto
170fa81095 [pipeline/chimera] test chimera | fix bug of initializing (#1615) 
* [pipeline/tuning] improve dispatch performance both time and space cost

* [pipeline/converge] add interface for testing convergence

* [NFC] polish colossalai/utils/multi_tensor_apply/multi_tensor_apply.py code style

* Update PipelineBase.py

* [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera

* [pipeline/chimera] test chimera | fix bug of initializing
 2022-09-20 18:00:39 +08:00
Jiarui Fang
504ff1d101 [embeddings] use cache_ratio instead of cuda_row_num (#1611) 2022-09-20 14:33:04 +08:00
YuliangLiu0306
6a8f8cc05e [hotfix] got sliced types (#1614) 2022-09-20 14:32:42 +08:00
Frank Lee
d397842fa8 [autoparallel] added new node handler (#1612) 2022-09-20 14:17:21 +08:00
YuliangLiu0306
7d1bb71d5d [fx] PoC of runtime shape consistency application (#1607) 
* [fx] PoC of runtime shape consistency application

* polish code
 2022-09-20 14:00:04 +08:00
YuliangLiu0306
47b11c432c [autoparallel]add bcast matmul strategies (#1605) 2022-09-20 11:26:21 +08:00
Frank Lee
edb67cb378 [autoparallel] refactored the data structure for sharding strategy (#1610) 2022-09-20 11:20:54 +08:00
Boyuan Yao
933b6c6367 [fx] Add pofo solver (#1608) 
* [fx] add pofo algorithm

* [fx] Add pofo solver

* [fx] code refactor

* [fx] fix test_linearize import
 2022-09-20 11:20:48 +08:00
github-actions[bot]
d32cf84c46 Automated submodule synchronization (#1609) 
Co-authored-by: github-actions <github-actions@github.com>
 2022-09-20 10:47:08 +08:00
Frank Lee
725666d6a9 [workflow] deactivate conda environment before removing (#1606) 2022-09-19 12:05:33 +08:00
Kirigaya Kazuto
edc9e419ad [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera (#1595) 
* [pipeline/tuning] improve dispatch performance both time and space cost

* [pipeline/converge] add interface for testing convergence

* [NFC] polish colossalai/utils/multi_tensor_apply/multi_tensor_apply.py code style

* Update PipelineBase.py

* [pipeline/chimera] reconstruct PipelineBase and Worker to support more feasible custom schedule | finish Chimera
 2022-09-19 11:44:18 +08:00