github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-04-12 15:14:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
119 Commits 62 Branches 51 Tags
b73a048ad8cf549c5ce3cecb72820923754fb520
Commit Graph

61 Commits

Author SHA1 Message Date
LuGY
b73a048ad8 [zero] cpu adam kernel (#288) 
* Added CPU Adam

* finished the cpu adam

* updated the license

* delete useless parameters, removed resnet

* modified the method off cpu adam unittest

* deleted some useless codes

* removed useless codes

Co-authored-by: ver217 <lhx0217@gmail.com>
Co-authored-by: Frank Lee <somerlee.9@gmail.com>
Co-authored-by: jiaruifang <fangjiarui123@gmail.com>
 2022-03-04 16:05:15 +08:00
Jiarui Fang
29521cba0a [zero] yet an improved sharded param (#311) 2022-03-04 15:49:23 +08:00
Jiarui Fang
2f6295bf78 [zero] polish shard strategy (#310) 
* init shard param from shape tuple

* add more unitest for shard param

* add set_payload method for ShardedParam

* [zero] add shareded tensor class

* polish code

* add shard stratgy

* move shard and gather logic to shard strategy from shard tensor.

* polish code
 2022-03-04 15:35:07 +08:00
ver217
b95f9b4670 polish code 2022-03-04 15:27:39 +08:00
ver217
2aa440358d fix sharded param hook and unit test 2022-03-04 15:27:39 +08:00
ver217
8c2327b93c impl shard optim v2 and add unit test 2022-03-04 15:27:39 +08:00
Jiarui Fang
88496b5b31 [zero] a shard strategy in granularity of tensor (#307) 2022-03-04 11:59:35 +08:00
Jiarui Fang
408cba655b [zero] sharded tensor (#305) 
* init shard param from shape tuple

* add more unitest for shard param

* add set_payload method for ShardedParam

* [zero] add shareded tensor class

* polish code
 2022-03-04 10:46:13 +08:00
Jie Zhu
ce5d94a604 [profiler] primary memory tracer 2022-03-04 09:35:23 +08:00
ver217
5cc84d94dc rename shared adam to sharded optim v2 2022-03-03 16:20:34 +08:00
ver217
df34bd0c7f fix master params dtype 2022-03-03 16:20:34 +08:00
ver217
6c290dbb08 add fp32 master params in sharded adam 2022-03-03 16:20:34 +08:00
ver217
6185b9772d add sharded adam 2022-03-03 16:20:34 +08:00
Jiarui Fang
6c78946fdd Polish sharded parameter (#297) 
* init shard param from shape tuple

* add more unitest for shard param

* add more unittests to shareded param
 2022-03-03 12:42:57 +08:00
ver217
9b07ac81d4 [zero] add sharded grad and refactor grad hooks for ShardedModel (#287) 2022-03-02 18:28:29 +08:00
Frank Lee
4fbb8db586 fixed typo in ShardParam (#294) 2022-03-02 17:26:23 +08:00
Frank Lee
193af3a8b7 added buffer sync to naive amp model wrapper (#291) 2022-03-02 16:47:17 +08:00
Jiarui Fang
6f22fb1906 add a common util for hooks registered on parameter. (#292) 2022-03-02 14:38:22 +08:00
Jie Zhu
3b64dcc439 bug fix: pass hook_list to engine (#273) 
* bug fix: pass hook_list to engine

* change parameter name
 2022-03-02 14:25:52 +08:00
Jiarui Fang
3280869358 Feature/zero (#279) 
* add zero1 (#209)

* add zero1

* add test zero1

* update zero stage 1 develop (#212)

* Implement naive zero3 (#240)

* naive zero3 works well

* add zero3 param manager

* add TODOs in comments

* add gather full param ctx

* fix sub module streams

* add offload

* fix bugs of hook and add unit tests

* fix bugs of hook and add unit tests (#252)

* add gather full param ctx

* fix sub module streams

* add offload

* fix bugs of hook and add unit tests

* polish code and add state dict hook

* fix bug

* update unit test

* refactor reconstructed zero code

* clip_grad support zero3 and add unit test

* add unit test for Zero3ParameterManager

* [WIP] initialize the shard param class

* [WIP] Yet another sharded model implementation (#274)

* [WIP] initialize the shard param class

* [WIP] Yes another implementation of shardModel. Using a better hook method.

* torch.concat -> torch.cat

* fix test_zero_level_1.py::test_zero_level_1 unitest

* remove deepspeed implementation and refactor for the reconstructed zero module

* polish zero dp unittests

Co-authored-by: ver217 <lhx0217@gmail.com>
Co-authored-by: Frank Lee <somerlee.9@gmail.com>
 2022-03-01 18:17:01 +08:00
1SAA
adf7e50325 Added TPExpert for special situation 2022-02-28 13:55:35 +08:00
HELSON
b5b612a26c Fixed parameter initialization in FFNExpert (#251) 2022-02-27 14:01:25 +08:00
アマデウス
726a4abb66 fixed CI dataset directory; fixed import error of 2.5d accuracy (#255) 2022-02-24 14:33:45 +08:00
1SAA
e7a1136436 Optimized MoE layer and fixed some bugs; 
Decreased moe tests;

Added FFNExperts and ViTMoE model
 2022-02-22 10:27:12 +08:00
zbian
218c061e2d fixed padding index issue for vocab parallel embedding layers; updated 3D linear to be compatible with examples in the tutorial 2022-02-22 02:05:43 +00:00
Frank Lee
f5ca88ec97 fixed apex import (#227) 2022-02-15 11:31:13 +08:00
Frank Lee
3a1a9820b0 fixed mkdir conflict and align yapf config with flake (#220) 2022-02-15 11:31:13 +08:00
アマデウス
9ee197d0e9 moved env variables to global variables; (#215) 
added branch context;
added vocab parallel layers;
moved split_batch from load_batch to tensor parallel embedding layers;
updated gpt model;
updated unit test cases;
fixed few collective communicator bugs
 2022-02-15 11:31:13 +08:00
Frank Lee
812357d63c fixed utils docstring and add example to readme (#200) 2022-02-03 11:37:17 +08:00
Frank Lee
765db512b5 fixed ddp bug on torch 1.8 (#194) 2022-01-28 15:14:04 +08:00
Jiarui Fang
569357fea0 add pytorch hooks (#179) 
* add pytorch hooks
fix #175

* remove licenses in src code

* add gpu memory tracer

* replacing print with logger in ophooks.
 2022-01-25 22:20:54 +08:00
ver217
708404d5f8 fix pipeline forward return tensors (#176) 2022-01-21 15:46:02 +08:00
HELSON
0f8c7f9804 Fixed docstring in colossalai (#171) 2022-01-21 10:44:30 +08:00
Frank Lee
e2089c5c15 adapted for sequence parallel (#163) 2022-01-20 13:44:51 +08:00
puck_WCR
9473a1b9c8 AMP docstring/markdown update (#160) 2022-01-18 18:33:36 +08:00
Frank Lee
f3802d6b06 fixed jit default setting (#154) 2022-01-18 13:37:20 +08:00
ver217
7bf1e98b97 pipeline last stage supports multi output (#151) 2022-01-17 15:57:47 +08:00
ver217
f68eddfb3d refactor kernel (#142) 2022-01-13 16:47:17 +08:00
BoxiangW
4a3d3446b0 Update layer integration documentations (#108) 
Update the documentations of layer integration

Update _log_hook.py

Update _operation.py
 2022-01-10 18:05:58 +08:00
ver217
9ef05ed1fc try import deepspeed when using zero (#130) 2022-01-07 17:24:57 +08:00
HELSON
dceae85195 Added MoE parallel (#127) 2022-01-07 15:08:36 +08:00
ver217
293fb40c42 add scatter/gather optim for pipeline (#123) 2022-01-07 13:22:22 +08:00
Jiarui Fang
2c0c85d3d3 fix a bug in timer (#114) 2022-01-05 16:07:06 +08:00
ver217
7904baf6e1 fix layers/schedule for hybrid parallelization (#111) (#112) 2022-01-04 20:52:31 +08:00
ver217
a951bc6089 update default logger (#100) (#101) 2022-01-04 20:03:26 +08:00
ver217
96780e6ee4 Optimize pipeline schedule (#94) 
* add pipeline shared module wrapper and update load batch

* added model parallel process group for amp and clip grad (#86)

* added model parallel process group for amp and clip grad

* update amp and clip with model parallel process group

* remove pipeline_prev/next group (#88)

* micro batch offload

* optimize pipeline gpu memory usage

* pipeline can receive tensor shape (#93)

* optimize pipeline gpu memory usage

* fix grad accumulation step counter

* rename classes and functions

Co-authored-by: Frank Lee <somerlee.9@gmail.com>
 2021-12-30 15:56:46 +08:00
アマデウス
01a80cd86d Hotfix/Colossalai layers (#92) 
* optimized 1d layer apis; reorganized nn.layer modules; fixed tests

* fixed 2.5d runtime issue

* reworked split batch, now called in trainer.schedule.load_batch

Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>
 2021-12-29 23:32:10 +08:00
アマデウス
0fedef4f3c Layer integration (#83) 
* integrated parallel layers for ease of building models

* integrated 2.5d layers

* cleaned codes and unit tests

* added log metric by step hook; updated imagenet benchmark; fixed some bugs

* reworked initialization; cleaned codes

Co-authored-by: BoxiangW <45734921+BoxiangW@users.noreply.github.com>
 2021-12-27 15:04:32 +08:00
shenggan
5c3843dc98 add colossalai kernel module (#55) 2021-12-21 12:19:52 +08:00
ver217
8f02a88db2 add interleaved pipeline, fix naive amp and update pipeline model initializer (#80) 2021-12-20 23:26:19 +08:00