github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-12-24 21:12:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,271 Commits 60 Branches 51 Tags
dac127d0ee9a6345295fc76aca0b43f42e22d3ff
Commit Graph

753 Commits

Author SHA1 Message Date
Hongxin Liu
f313babd11 [gemini] support save state dict in shards (#3581) 
* [gemini] support state dict shard

* [gemini] add test state dict shard

* [gemini] polish docstr

* [gemini] fix merge

* [gemini] polish code
 2023-04-17 17:11:09 +08:00
Hongxin Liu
152239bbfa [gemini] gemini supports lazy init (#3379) 
* [gemini] fix nvme optimizer init

* [gemini] gemini supports lazy init

* [gemini] add init example

* [gemini] add fool model

* [zero] update gemini ddp

* [zero] update init example

* add chunk method

* add chunk method

* [lazyinit] fix lazy tensor tolist

* [gemini] fix buffer materialization

* [misc] remove useless file

* [booster] update gemini plugin

* [test] update gemini plugin test

* [test] fix gemini plugin test

* [gemini] fix import

* [gemini] fix import

* [lazyinit] use new metatensor

* [lazyinit] use new metatensor

* [lazyinit] fix __set__ method
 2023-04-12 16:03:25 +08:00
jiangmingyan
52a933e175 [checkpoint] support huggingface style sharded checkpoint (#3461) 
* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

---------

Co-authored-by: luchen <luchen@luchendeMBP.lan>
 2023-04-06 16:23:39 +08:00
Frank Lee
80eba05b0a [test] refactor tests with spawn (#3452) 
* [test] added spawn decorator

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code
 2023-04-06 14:51:35 +08:00
ver217
933048ad3e [test] reorganize zero/gemini tests (#3445) 2023-04-06 09:38:25 +08:00
YuliangLiu0306
ffcdbf0f65 [autoparallel]integrate auto parallel feature with new tracer (#3408) 
* [autoparallel] integrate new analyzer in module level

* unify the profiling method

* polish

* fix no codegen bug

* fix pass bug

* fix liveness test

* polish
 2023-04-04 17:40:45 +08:00
Frank Lee
1beb85cc25 [checkpoint] refactored the API and added safetensors support (#3427) 
* [checkpoint] refactored the API and added safetensors support

* polish code
 2023-04-04 15:23:01 +08:00
ver217
26b7aac0be [zero] reorganize zero/gemini folder structure (#3424) 
* [zero] refactor low-level zero folder structure

* [zero] fix legacy zero import path

* [zero] fix legacy zero import path

* [zero] remove useless import

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor legacy zero import path

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor legacy zero import path

* [zero] fix test import path

* [zero] fix test

* [zero] fix circular import

* [zero] update import
 2023-04-04 13:48:16 +08:00
Frank Lee
638a07a7f9 [test] fixed gemini plugin test (#3411) 
* [test] fixed gemini plugin test

* polish code

* polish code
 2023-04-03 17:12:22 +08:00
ver217
5f2e34e6c9 [booster] implement Gemini plugin (#3352) 
* [booster] add gemini plugin

* [booster] update docstr

* [booster] gemini plugin add coloparam convertor

* [booster] fix coloparam convertor

* [booster] fix gemini plugin device

* [booster] add gemini plugin test

* [booster] gemini plugin ignore sync bn

* [booster] skip some model

* [booster] skip some model

* [booster] modify test world size

* [booster] modify test world size

* [booster] skip test
 2023-03-31 16:06:13 +08:00
HELSON
1a1d68b053 [moe] add checkpoint for moe models (#3354) 
* [moe] add checkpoint for moe models

* [hotfix] fix bugs in unit test
 2023-03-31 09:20:33 +08:00
YuliangLiu0306
fee2af8610 [autoparallel] adapt autoparallel with new analyzer (#3261) 
* [autoparallel] adapt autoparallel with new analyzer

* fix all node handler tests

* polish

* polish
 2023-03-30 17:47:24 +08:00
Frank Lee
73d3e4d309 [booster] implemented the torch ddd + resnet example (#3232) 
* [booster] implemented the torch ddd + resnet example

* polish code
 2023-03-27 10:24:14 +08:00
YuliangLiu0306
4d5d8f98a4 [API] implement device mesh manager (#3221) 
* [API] implement  device mesh manager

* polish
 2023-03-24 13:39:12 +08:00
YuliangLiu0306
045afa3ea2 [hotfix] skip torchaudio tracing test (#3211) 
* [hotfix] skip torchaudio tracing test

* fix lazy init test issue
 2023-03-24 12:15:33 +08:00
Frank Lee
cd142fbefa [api] implemented the checkpoint io module (#3205) 
* [api] implemented the checkpoint io module

* polish code

* polish code
 2023-03-23 10:53:17 +08:00
ver217
f8289d4221 [lazyinit] combine lazy tensor with dtensor (#3204) 
* [lazyinit] lazy tensor add distribute

* [lazyinit] refactor distribute

* [lazyinit] add test dist lazy init

* [lazyinit] add verbose info for dist lazy init

* [lazyinit] fix rnn flatten weight op

* [lazyinit] polish test

* [lazyinit] polish test

* [lazyinit] fix lazy tensor data setter

* [lazyinit] polish test

* [lazyinit] fix clean

* [lazyinit] make materialize inplace

* [lazyinit] refactor materialize

* [lazyinit] refactor test distribute

* [lazyinit] fix requires_grad

* [lazyinit] fix tolist after materialization

* [lazyinit] refactor distribute module

* [lazyinit] polish docstr

* [lazyinit] polish lazy init context

* [lazyinit] temporarily skip test

* [lazyinit] polish test

* [lazyinit] add docstr
 2023-03-23 10:53:06 +08:00
YuliangLiu0306
019a847432 [Analyzer] fix analyzer tests (#3197) 2023-03-22 13:38:11 +08:00
YuliangLiu0306
f57d34958b [FX] refactor experimental tracer and adapt it with hf models (#3157) 
* pass gpt trace and meta_prop

* pass t5 trace and meta_prop

* [FX] refactor experimental tracer and adapt it with hf models

* pass all mainstream model zoo

* fix CI

* fix CI

* fix CI

* fix CI

* fix CI

* fix CI

* fix CI

* fix CI

* skip tests

* fix CI

* using packaging version

* polish
 2023-03-22 10:40:33 +08:00
Frank Lee
e7f3bed2d3 [booster] added the plugin base and torch ddp plugin (#3180) 
* [booster] added the plugin base and torch ddp plugin

* polish code

* polish code

* polish code
 2023-03-21 17:39:30 +08:00
Zihao
18dbe76cae [auto-parallel] add auto-offload feature (#3154) 
* add auto-offload feature

* polish code

* fix syn offload runtime pass bug

* add offload example

* fix offload testing bug

* fix example testing bug
 2023-03-21 14:17:41 +08:00
zbian
7bc0afc901 updated flash attention usage 2023-03-20 17:57:04 +08:00
Frank Lee
085e7f4eff [test] fixed torchrec registration in model zoo (#3177) 
* [test] fixed torchrec registration in model zoo

* polish code

* polish code

* polish code
 2023-03-20 16:19:06 +08:00
Frank Lee
a9b8402d93 [booster] added the accelerator implementation (#3159) 2023-03-20 13:59:24 +08:00
Frank Lee
1ad3a636b1 [test] fixed torchrec model test (#3167) 
* [test] fixed torchrec model test

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code
 2023-03-20 11:40:25 +08:00
ver217
6ae8ed0407 [lazyinit] add correctness verification (#3147) 
* [lazyinit] fix shared module

* [tests] add lazy init test utils

* [tests] add torchvision for lazy init

* [lazyinit] fix pre op fn

* [lazyinit] handle legacy constructor

* [tests] refactor lazy init test models

* [tests] refactor lazy init test utils

* [lazyinit] fix ops don't support meta

* [tests] lazy init test timm models

* [lazyinit] fix set data

* [lazyinit] handle apex layers

* [tests] lazy init test transformers models

* [tests] lazy init test torchaudio models

* [lazyinit] fix import path

* [tests] lazy init test torchrec models

* [tests] update torch version in CI

* [tests] revert torch version in CI

* [tests] skip lazy init test
 2023-03-17 13:49:04 +08:00
Frank Lee
ed19290560 [booster] implemented mixed precision class (#3151) 
* [booster] implemented mixed precision class

* polish code
 2023-03-17 11:00:15 +08:00
YuliangLiu0306
ecd643f1e4 [test] add torchrec models to test model zoo (#3139) 2023-03-15 05:46:04 +00:00
ver217
14a115000b [tests] model zoo add torchaudio models (#3138) 
* [tests] model zoo add torchaudio models

* [tests] refactor torchaudio wavernn

* [tests] refactor fx torchaudio tests
 2023-03-15 11:51:16 +08:00
Frank Lee
6d48eb0560 [test] added transformers models to test model zoo (#3135) 2023-03-15 11:26:10 +08:00
Frank Lee
a674c63348 [test] added torchvision models to test model zoo (#3132) 
* [test] added torchvision models to test model zoo

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code
 2023-03-15 10:42:07 +08:00
HELSON
1216d1e7bd [tests] diffuser models in model zoo (#3136) 
* [tests] diffuser models in model zoo

* remove useless code

* [tests] add diffusers to requirement-test
 2023-03-14 17:20:28 +08:00
YuliangLiu0306
2eca4cd376 [DTensor] refactor dtensor with new components (#3089) 
* [DTensor] refactor dtensor with new components

* polish
 2023-03-14 16:25:47 +08:00
Frank Lee
86ac782d7c [test] added timm models to test model zoo (#3129) 
* [test] added timm models to test model zoo

* polish code

* polish code

* polish code

* polish code

* polish code
 2023-03-14 14:29:18 +08:00
Xuanlei Zhao
30dd13c450 [autochunk] support complete benchmark (#3121) 
* refact memory code

* dont log free var memory

* add memory align

* update chunk target

* update setting for new memory

* finish test

* update tracer

* update typo

* update test

* add unet test

* add bench

* update bench

* update bench

* init

* support vit

* move to cpu

* add cpu benchmark
 2023-03-13 17:42:37 +08:00
Super Daniel
fff98f06ed [analyzer] a minimal implementation of static graph analyzer (#2852) 
* [hotfix] meta tensor default device.

* [siu] add experimental submodules to main branch.

* [siu]

* [siu]

* [analyzer] init.

* [analyzer] readme.

* [analyzer] readme.

* [analyzer] readme.

* [analyzer] readme.

* [test] add test.

* Update symbolic_trace.py

* mark skip tests.

* try except.

* try except.

* try except.

* s

* init

* init

* fix

* skip

* skip

---------

Co-authored-by: Daniel Shao <superdainiu@MININT-PVARVID.fareast.corp.microsoft.com>
Co-authored-by: Daniel Shao <superdainiu@Daniels-Mac.local>
 2023-03-10 13:21:05 +08:00
Xuanlei Zhao
10c61de2f7 [autochunk] support vit (#3084) 
support vit for autochunk
* support some new ops for vit
* fix some bugs
* add test for vit
 2023-03-10 10:23:26 +08:00
YuliangLiu0306
8e4e8601b7 [DTensor] implement layout converter (#3055) 
* [DTensor] refactor LayoutConverter for DTensor

* polish code

* polish docstring
 2023-03-10 09:53:52 +08:00
Xuanlei Zhao
2ca9728cbb [autochunk] refactor chunk memory estimation (#2762) 
* refact memory code

* dont log free var memory

* add memory align

* update chunk target

* update setting for new memory

* finish test

* update tracer

* update typo

* update test
 2023-03-08 16:22:30 +08:00
YuliangLiu0306
29386a54e6 [DTensor] refactor CommSpec (#3034) 2023-03-08 10:45:31 +08:00
YuliangLiu0306
4269196c79 [hotfix] skip auto checkpointing tests (#3029) 
* [hotfix] skip auto checkpointing tests

* fix test name issue
 2023-03-07 15:50:00 +08:00
YuliangLiu0306
cd2b0eaa8d [DTensor] refactor sharding spec (#2987) 
* [autoparallel] refactor sharding spec

* rename function name
 2023-03-07 11:08:11 +08:00
YuliangLiu0306
e414e4092b [DTensor] implementation of dtensor (#2946) 
* [DTensor] implementation of dtensor

* test layout convert

* polish
 2023-03-01 16:34:58 +08:00
YuliangLiu0306
197d0bf4ed [autoparallel] apply repeat block to reduce solving time (#2912) 2023-02-28 11:03:30 +08:00
YuliangLiu0306
819e25d8b1 [hotfix] fix autoparallel compatibility test issues (#2754) 2023-02-23 17:28:36 +08:00
YuliangLiu0306
0f392d7403 [autoparallel] find repeat blocks (#2854) 
* [autoparallel] find repeat blocks

* polish

* polish

* polish
 2023-02-23 17:28:19 +08:00
Boyuan Yao
c7764d3f22 [autoparallel] Patch meta information of torch.where (#2822) 
* [autoparallel] patch meta information of torch.where

* [autoparallel] pre-commit modified
 2023-02-22 10:28:21 +08:00
Boyuan Yao
fcc4097efa [autoparallel] Patch meta information of torch.tanh() and torch.nn.Dropout (#2773) 
* [autoparallel] tanh meta information

* [autoparallel] remove redundant code

* [autoparallel] patch meta information of torch.nn.Dropout
 2023-02-22 10:27:59 +08:00
Boyuan Yao
7ea6bc7f69 [autoparallel] Patch tensor related operations meta information (#2789) 
* [autoparallel] tensor related meta information prototype

* [autoparallel] tensor related meta information

* [autoparallel] tensor related meta information

* [autoparallel] tensor related meta information

* [autoparallel] tensor related meta information
 2023-02-20 17:38:55 +08:00
HELSON
56ddc9ca7a [hotfix] add correct device for fake_param (#2796) 2023-02-17 15:29:07 +08:00