9 Commits

Author	SHA1	Message	Date
Jie Zhu	345d32c182	[profiler] add adaptive sampling to memory profiler (#330) ... * fix merge conflict modify unit test remove unnessesary log info reformat file * remove unused module * remove unnecessary sync function * change doc string style from Google to Sphinx	2022-03-09 11:07:10 +08:00
ver217	ce5a7dcab0	[zero] Update sharded model v2 using sharded param v2 (#323)	2022-03-08 18:18:06 +08:00
Jiarui Fang	cec05b25c9	[zero] update zero context init with the updated test utils (#327)	2022-03-08 14:45:01 +08:00
ver217	2aa440358d	fix sharded param hook and unit test	2022-03-04 15:27:39 +08:00
ver217	8c2327b93c	impl shard optim v2 and add unit test	2022-03-04 15:27:39 +08:00
Jie Zhu	ce5d94a604	[profiler] primary memory tracer	2022-03-04 09:35:23 +08:00
ver217	9b07ac81d4	[zero] add sharded grad and refactor grad hooks for ShardedModel (#287)	2022-03-02 18:28:29 +08:00
Jiarui Fang	3280869358	Feature/zero (#279) ... * add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by: ver217 <lhx0217@gmail.com> Co-authored-by: Frank Lee <somerlee.9@gmail.com>	2022-03-01 18:17:01 +08:00
Jiarui Fang	569357fea0	add pytorch hooks (#179) ... * add pytorch hooks fix #175 * remove licenses in src code * add gpu memory tracer * replacing print with logger in ophooks.	2022-01-25 22:20:54 +08:00