github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-07-17 02:00:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,669 Commits 62 Branches 51 Tags
aed20fb2dfb46041024dd56f79d3be1c751e03fe
Commit Graph

1266 Commits

Author SHA1 Message Date
duanjunwen
5e09c8b4e1 [feat] split communication and calculation; fix pop empty send_bwd_buffer error; 2024-08-27 06:29:13 +00:00
duanjunwen
1d75045c37 [feat] add test run_fwd_bwd automatic scheduling; 2024-08-26 11:21:56 +00:00
duanjunwen
fd5526b76e Merge branch 'main' into dev/zero_bubble 2024-08-26 04:03:20 +00:00
duanjunwen
107230d27a [update] update text; 2024-08-26 04:00:51 +00:00
duanjunwen
203033ea16 [fix] fix weight not close; 2024-08-23 08:57:27 +00:00
duanjunwen
c18ef060cf [feat] add dw test; 2024-08-23 06:04:12 +00:00
duanjunwen
ee9baedadf [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble; 2024-08-22 10:25:34 +00:00
Hongxin Liu
26493b97d3 [misc] update compatibility (#6008) 
* [misc] update compatibility

* [misc] update requirements

* [devops] disable requirements cache

* [test] fix torch ddp test

* [test] fix rerun on address in use

* [test] fix lazy init
 2024-08-16 18:49:14 +08:00
Edenzzzz
f5c84af0b0 [Feature] Zigzag Ring attention (#5905) 
* halfway

* fix cross-PP-stage position id length diff bug

* fix typo

* fix typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* unified cross entropy func for all shardformer models

* remove redundant lines

* add basic ring attn; debug cross entropy

* fwd bwd logic complete

* fwd bwd logic complete; add experimental triton rescale

* precision tests passed

* precision tests passed

* fix typos and remove misc files

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add sp_mode to benchmark; fix varlen interface

* update softmax_lse shape by new interface

* change tester name

* remove buffer clone; support packed seq layout

* add varlen tests

* fix typo

* all tests passed

* add dkv_group; fix mask

* remove debug statements

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-08-16 13:56:38 +08:00
Wang Binluo
75c963686f [lora] lora support hybrid parallel plugin (#5956) 
* lora support hybrid plugin

* fix

* fix

* fix

* fix
 2024-08-02 10:36:58 +08:00
botbw
62cdac6b7b [chore] remove redundant test case, print string & reduce test tokens 2024-08-01 10:06:59 +08:00
haze188
70793ce9ed [misc] fix ci failure: change default value to false in moe plugin 2024-08-01 10:06:59 +08:00
haze188
12d043ca00 [misc] remove incompatible test config 2024-08-01 10:06:59 +08:00
hxwang
cb01c0d5ce [moe] refactor mesh assignment 2024-08-01 10:06:59 +08:00
haze188
034020bd04 [misc] remove debug/print code 2024-08-01 10:06:59 +08:00
haze188
59bcf56c60 [misc] skip redunant test 2024-08-01 10:06:59 +08:00
hxwang
c3dc9b4dba [deepseek] replace attn (a workaround for bug in transformers) 2024-08-01 10:06:59 +08:00
hxwang
6c39f0b144 [test] add check 2024-08-01 10:06:59 +08:00
haze188
b2952a5982 [moe] deepseek moe sp support 2024-08-01 10:06:59 +08:00
hxwang
067e18f7e9 [test] fix test: test_zero1_2 2024-08-01 10:06:59 +08:00
hxwang
70c9924d0d [chore] solve moe ckpt test failure and some other arg pass failure 2024-08-01 10:06:59 +08:00
pre-commit-ci[bot]
52d346f2a5 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2024-08-01 10:06:59 +08:00
hxwang
46037c2ccd [chore] minor fix after rebase 2024-08-01 10:06:59 +08:00
hxwang
803878b2fd [moe] full test for deepseek and mixtral (pp + sp to fix) 2024-08-01 10:06:59 +08:00
hxwang
7077d38d5a [moe] finalize test (no pp) 2024-08-01 10:06:59 +08:00
haze188
2cddeac717 moe sp + ep bug fix 2024-08-01 10:06:59 +08:00
hxwang
877d94bb8c [moe] init moe plugin comm setting with sp 2024-08-01 10:06:59 +08:00
hxwang
09d6280d3e [chore] minor fix 2024-08-01 10:06:59 +08:00
Haze188
404b16faf3 [Feature] MoE Ulysses Support (#5918) 
* moe sp support

* moe sp bug solve

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-08-01 10:06:59 +08:00
hxwang
3e2b6132b7 [moe] clean legacy code 2024-08-01 10:06:59 +08:00
hxwang
74eccac0db [moe] test deepseek 2024-08-01 10:06:59 +08:00
botbw
dc583aa576 [moe] implement tp 2024-08-01 10:06:59 +08:00
botbw
0b5bbe9ce4 [test] add mixtral modelling test 2024-08-01 10:06:59 +08:00
hxwang
102b784a10 [chore] arg pass & remove drop token 2024-08-01 10:06:59 +08:00
botbw
9b9b76bdcd [moe] add mixtral dp grad scaling when not all experts are activated 2024-08-01 10:06:59 +08:00
botbw
e28e05345b [moe] implement submesh initialization 2024-08-01 10:06:59 +08:00
haze188
5ed5e8cfba solve hang when parallel mode = pp + dp 2024-08-01 10:06:59 +08:00
haze188
fe24789eb1 [misc] solve booster hang by rename the variable 2024-08-01 10:06:59 +08:00
botbw
13b48ac0aa [zero] solve hang 2024-08-01 10:06:59 +08:00
botbw
b5bfeb2efd [moe] implement transit between non moe tp and ep 2024-08-01 10:06:59 +08:00
botbw
37443cc7e4 [test] pass mixtral shardformer test 2024-08-01 10:06:59 +08:00
hxwang
46c069b0db [zero] solve hang 2024-08-01 10:06:59 +08:00
hxwang
a249e71946 [test] mixtra pp shard test 2024-08-01 10:06:59 +08:00
hxwang
0b76b57cd6 [test] add mixtral transformer test 2024-08-01 10:06:59 +08:00
Guangyao Zhang
1c961b20f3 [ShardFormer] fix qwen2 sp (#5903) 2024-07-15 13:58:06 +08:00
Hongxin Liu
c068ef0fa0 [zero] support all-gather overlap (#5898) 
* [zero] support all-gather overlap

* [zero] add overlap all-gather flag

* [misc] fix typo

* [zero] update api
 2024-07-11 18:59:59 +08:00
Guangyao Zhang
669849d74b [ShardFormer] Add Ulysses Sequence Parallelism support for Command-R, Qwen2 and ChatGLM (#5897) 2024-07-10 11:34:25 +08:00
Edenzzzz
fbf33ecd01 [Feature] Enable PP + SP for llama (#5868) 
* fix cross-PP-stage position id length diff bug

* fix typo

* fix typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use a one cross entropy func for all shardformer models

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-07-09 18:05:20 +08:00
Haze188
3420921101 [shardformer] DeepseekMoE support (#5871) 
* [Feature] deepseek moe expert parallel implement

* [misc] fix typo, remove redundant file (#5867)

* [misc] fix typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [Feature] deepseek support & unit test

* [misc] remove debug code & useless print

* [misc] fix typos (#5872)

* [Feature] remove modeling file, use auto config. (#5884)

* [misc] fix typos

* [Feature] deepseek support via auto model, remove modeling file

* [misc] delete useless file

* [misc] fix typos

* [Deepseek] remove redundant code (#5888)

* [misc] fix typos

* [Feature] deepseek support via auto model, remove modeling file

* [misc] delete useless file

* [misc] fix typos

* [misc] remove redundant code

* [Feature/deepseek] resolve comment. (#5889)

* [misc] fix typos

* [Feature] deepseek support via auto model, remove modeling file

* [misc] delete useless file

* [misc] fix typos

* [misc] remove redundant code

* [misc] mv module replacement into if branch

* [misc] add some warning message and modify some code in unit test

* [misc] fix typos

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-07-05 16:13:58 +08:00
pre-commit-ci[bot]
7c2f79fa98 [pre-commit.ci] pre-commit autoupdate (#5572) 
* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/PyCQA/autoflake: v2.2.1 → v2.3.1](https://github.com/PyCQA/autoflake/compare/v2.2.1...v2.3.1)
- [github.com/pycqa/isort: 5.12.0 → 5.13.2](https://github.com/pycqa/isort/compare/5.12.0...5.13.2)
- [github.com/psf/black-pre-commit-mirror: 23.9.1 → 24.4.2](https://github.com/psf/black-pre-commit-mirror/compare/23.9.1...24.4.2)
- [github.com/pre-commit/mirrors-clang-format: v13.0.1 → v18.1.7](https://github.com/pre-commit/mirrors-clang-format/compare/v13.0.1...v18.1.7)
- [github.com/pre-commit/pre-commit-hooks: v4.3.0 → v4.6.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.3.0...v4.6.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-07-01 17:16:41 +08:00