github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-08-10 04:18:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
hotfix/example_test
ColossalAI/colossalai/zero/gemini/chunk
History
Baizhou Zhang 21ba89cab6
[gemini] support gradient accumulation (#4869) 
* add test

* fix no_sync bug in low level zero plugin

* fix test

* add argument for grad accum

* add grad accum in backward hook for gemini

* finish implementation, rewrite tests

* fix test

* skip stuck model in low level zero test

* update doc

* optimize communication & fix gradient checkpoint

* modify doc

* cleaning codes

* update cpu adam fp16 case
2023-10-17 14:07:21 +08:00
..
__init__.py [misc] update pre-commit and run all files (#4752) 2023-09-19 14:20:26 +08:00
chunk.py [gemini] support gradient accumulation (#4869) 2023-10-17 14:07:21 +08:00
manager.py [gemini] support gradient accumulation (#4869) 2023-10-17 14:07:21 +08:00
search_utils.py [misc] update pre-commit and run all files (#4752) 2023-09-19 14:20:26 +08:00
utils.py [misc] update pre-commit and run all files (#4752) 2023-09-19 14:20:26 +08:00