github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-09 13:00:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,858 Commits 59 Branches 51 Tags
grpo-latest-dev-debug
Commit Graph

1213 Commits

Author SHA1 Message Date
Jiarui Fang
05bb28aacf [Gemini] mapping of preop timestep and param (#2124) 2022-12-13 12:50:24 +08:00
YuliangLiu0306
cd0af9f7f6 [autoparallel] gpt2lp runtimee test (#2113) 2022-12-12 18:06:40 +08:00
Jiarui Fang
9214d1fe28 [Gemini] chunk init using runtime visited param order (#2115) 2022-12-12 18:06:16 +08:00
HELSON
e7d3afc9cc [optimizer] add div_scale for optimizers (#2117) 
* [optimizer] add div_scale for optimizers

* [zero] use div_scale in zero optimizer

* fix testing error
 2022-12-12 17:58:57 +08:00
Jiarui Fang
e5aa8333e4 [NFC] update chunk manager API (#2119) 2022-12-12 16:57:22 +08:00
Jiarui Fang
e99edfcb51 [NFC] polish comments for Chunk class (#2116) 2022-12-12 15:39:31 +08:00
Ziyue Jiang
09d69e1c25 [PP Middleware] Add bwd and step for PP middleware (#2111) 
* add bwd and step for PP middleware

* pre-commit

Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>
 2022-12-12 12:40:03 +08:00
HELSON
63fbba3c19 [zero] add L2 gradient clipping for ZeRO (#2112) 
* [zero] add L2 gradient clipping

* [testing] add MlpModel

* [zero] add unit test for grad clipping

* fix atol
 2022-12-09 18:09:17 +08:00
Jiarui Fang
70a8556946 [gemini] get the param visited order during runtime (#2108) 2022-12-09 16:13:03 +08:00
YuliangLiu0306
d87baa85d9 [autoparallel] support linear function bias addition (#2104) 2022-12-09 10:31:36 +08:00
YuliangLiu0306
0fecbb9e20 [autoparallel] support addbmm computation (#2102) 2022-12-08 21:15:11 +08:00
YuliangLiu0306
d3d4630495 [autoparallel] add sum handler (#2101) 2022-12-08 17:02:54 +08:00
Ziyue Jiang
e4705ba4e2 [Pipeline Middleware] fix data race in Pipeline Scheduler for DAG (#2087) 
* add DAG test case

* fix datarace by adjusting theposition of lock

* polish code

* fix pytest for middleware

* remove test

Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>
 2022-12-08 13:32:27 +08:00
YuliangLiu0306
b175e6d58e [autoparallel] add bias addtion function class (#2098) 
* [autoparallel] add bias addtion function class

* polish code

* polish
 2022-12-08 11:31:51 +08:00
YuliangLiu0306
3af7e65dea [autoparallel] complete gpt related module search (#2097) 2022-12-08 10:04:09 +08:00
Jiarui Fang
85efb7ac2e [Gemini] gemini use the runtime memory tracer (RMT) (#2099) 2022-12-07 23:04:02 +08:00
Jiarui Fang
978242326a [Gemini] remove eval in gemini unittests! (#2092) 2022-12-07 11:58:37 +08:00
YuliangLiu0306
7f72eb0510 [autoparallel]add embedding handler (#2089) 
* [autoparallel] add embedding handler

* fix bugs
 2022-12-07 09:41:46 +08:00
Jiarui Fang
1fca5d79ea [Gemini] remove GLOBAL_MODEL_DATA_TRACER (#2091) 2022-12-06 22:30:16 +08:00
Jiarui Fang
25abae6d7f [Gemini] use MemStats in Runtime Memory tracer (#2088) 2022-12-06 19:48:20 +08:00
Jiarui Fang
33f4412102 [Gemini] use MemStats to store the tracing data. Seperate it from Collector. (#2084) 2022-12-06 16:43:06 +08:00
Jiarui Fang
1f99205827 [Gemini] remove static tracer (#2083) 2022-12-06 12:53:58 +08:00
YuliangLiu0306
0e9db368ef [autoparallel] add tensor constructor handler (#2082) 2022-12-06 10:20:10 +08:00
YuliangLiu0306
cdf537a648 [autoparallel] add non_split linear strategy (#2078) 
* [autoparallel] add non_split linear stategy

* polish
 2022-12-06 10:19:33 +08:00
Boyuan Yao
cf0268da93 [autoparallel] Add F.conv metainfo (#2069) 
* [fx] metainfo class for auto parallel

* [fx] add unit test for linear metainfo

* [fx] fix bwd param for linear

* [fx] modify unit test

* [fx] modify unit test

* [fx] modify import

* [fx] modify import

* [fx] modify import

* [fx] move meta profiler to auto parallel

* [fx] add conv metainfo class

* [fx] restore profiler

* [fx] restore meta profiler

* [autoparallel] modify unit test

* [fx] modify unit test

* [autoparallel] add batchnorm metainfo class

* [autoparallel] fix batchnorm unit test function declaration

* [fx] restore profiler

* [fx] add relu metainfo class

* [fx] restore profiler

* [autoparallel] modify metainfo input

* [autoparallel] add pooling metainfo

* [autoparallel] add F.linear metainfo generator

* [autoparallel] add binary elementwise metainfo

* [fx] recover profiler

* [autoparallel] fix forward memory calculation

* [autoparallel] modify constants.py

* [autoparallel] remove redundant print

* [autoparallel] add F.conv metainfo

* [autoparallel] linear fix
 2022-12-06 10:17:57 +08:00
YuliangLiu0306
f123476666 [autoparallel] complete gpt block searching (#2065) 
* [autoparallel] complete gpt block searching

* fix test
 2022-12-06 10:17:10 +08:00
Ziyue Jiang
597cdd3006 [Pipeline Middleware] Adapt scheduler for Topo (#2066) 
* adapt scheduler for Topo

* remoove comment

* fix set input

Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>
 2022-12-05 20:23:41 +08:00
Jiarui Fang
4f21c9e8d9 [Gemini] polish runtime tracer tests (#2077) 2022-12-05 16:22:49 +08:00
Jiarui Fang
a7adad9ccb [Gemini] rename hooks related to runtime mem tracer (#2076) 2022-12-05 15:00:03 +08:00
Jiarui Fang
40b7d55bf3 [Gemini] add albert in test models. (#2075) 2022-12-05 14:09:34 +08:00
Jiarui Fang
616ed91ecd [test] bert test in non-distributed way (#2074) 2022-12-05 13:32:16 +08:00
Jiarui Fang
223332ff7e [Gemini] rename ParamTracerWrapper -> RuntimeMemTracer (#2073) 2022-12-05 12:45:11 +08:00
Jiarui Fang
9f828ef36f [Gemini] remove not used MemtracerWrapper (#2072) 2022-12-05 11:57:59 +08:00
Boyuan Yao
616da17fab [autoparallel] add binary elementwise metainfo for auto parallel (#2058) 
* [fx] metainfo class for auto parallel

* [fx] add unit test for linear metainfo

* [fx] fix bwd param for linear

* [fx] modify unit test

* [fx] modify unit test

* [fx] modify import

* [fx] modify import

* [fx] modify import

* [fx] move meta profiler to auto parallel

* [fx] add conv metainfo class

* [fx] restore profiler

* [fx] restore meta profiler

* [autoparallel] modify unit test

* [fx] modify unit test

* [autoparallel] add batchnorm metainfo class

* [autoparallel] fix batchnorm unit test function declaration

* [fx] restore profiler

* [fx] add relu metainfo class

* [fx] restore profiler

* [autoparallel] modify metainfo input

* [autoparallel] add pooling metainfo

* [autoparallel] add F.linear metainfo generator

* [autoparallel] add binary elementwise metainfo

* [fx] recover profiler

* [autoparallel] fix forward memory calculation

* [autoparallel] modify constants.py

* [autoparallel] remove redundant print
 2022-12-04 15:18:51 +08:00
Ziyue Jiang
44ea461890 [Pipeline] Add Topo Class (#2059) 
* use Topo class to rewrite DAG

* polish code

* polish code

* polish code

* add comment

* add else to unended if

Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>
 2022-12-02 18:13:20 +08:00
YuliangLiu0306
e4293e5077 [hotfix] update test for latest version (#2060) 2022-12-02 18:12:30 +08:00
YuliangLiu0306
19438ea0ef [hotfix] skip gpt tracing test (#2064) 2022-12-02 16:48:28 +08:00
Zihao
38ea4ba1bd [Gemini] fix grad unreleased issue and param recovery issue (#2052) 2022-12-02 16:04:19 +08:00
YuliangLiu0306
1c1fe44305 [autoparallel] adapt solver with self attention (#2037) 
* [autoparallel] adapt solver with self attention

* polish code
 2022-12-01 17:53:15 +08:00
HELSON
f6178728a0 [gemini] fix init bugs for modules (#2047) 
* [gemini] fix init bugs for modules

* fix bugs
 2022-11-30 17:06:10 +08:00
Zihao
6a9158f1fa [Gemini] free and allocate cuda memory by tensor.storage, add grad hook (#2040) 2022-11-30 15:57:45 +08:00
Jiarui Fang
1e885329f4 [test] align model name with the file name. (#2045) 2022-11-30 15:45:26 +08:00
Jiarui Fang
31c644027b [hotfix] hotfix Gemini for no leaf modules bug (#2043) 2022-11-30 14:53:41 +08:00
HELSON
384cd26314 [zero] fix testing parameters (#2042) 2022-11-30 12:09:32 +08:00
HELSON
17a3c685b0 [zero] fix unit-tests (#2039) 2022-11-30 10:40:31 +08:00
Jiarui Fang
eb7742a4bb [Gemini] more tests for Gemini (#2038) 
* [Gemini] more tests for Gemini

* polish code
 2022-11-29 17:13:10 +08:00
HELSON
537e181705 [testing] fix testing models (#2036) 
* [testing] fix testing models

* roll back
 2022-11-29 13:42:06 +08:00
HELSON
a1ce02d740 [zero] test gradient accumulation (#1964) 
* [zero] fix memory leak for zero2

* [zero] test gradient accumulation

* [zero] remove grad clip test
 2022-11-29 13:00:30 +08:00
Ziyue Jiang
b0936e4a44 [rpc] split with dag (#2028) 
* add DAG to split_module

* add comment

* add test case for DAG

* remove print

* add DAG middleware in scheduler

* add test case for scheduler

* remove break

* recover old lifecycle

Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>
 2022-11-29 11:36:28 +08:00
Jiarui Fang
96134e7be3 [hotfix] add bert test for gemini fwd bwd (#2035) 2022-11-29 11:19:52 +08:00