github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-01-24 22:35:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,630 Commits 60 Branches 51 Tags
6c2a120bed8658015f0f4e4ee95cbbe314b6ce5e
Commit Graph

2119 Commits

Author SHA1 Message Date
duanjunwen
ce58d8e8bf [fix] add output_obj_grad assert None at bwd b step; replace input_obj.require_grad_ with treemap; 2024-09-09 08:19:58 +00:00
duanjunwen
7568b34626 [fix] fix redundant detach & clone; add buffer assertation in the end; 2024-09-09 08:04:28 +00:00
duanjunwen
e6e1a97a6d [fix] fix requir grad position and detach position and input&output local buffer append position; 2024-09-04 03:31:08 +00:00
duanjunwen
20503cdfdf [fix] rm requir_grad for output; 2024-09-03 09:24:40 +00:00
duanjunwen
b4103f125c [fix] fix detach output & release output; 2024-09-03 09:09:41 +00:00
duanjunwen
4c1f81c683 [fix] fix bwd step if condition; remove useless comments and format info; 2024-09-03 08:56:08 +00:00
duanjunwen
ab643c9af7 [fix] rm output.data after send fwd; 2024-09-03 14:12:17 +08:00
duanjunwen
a48afc4a66 [fix] fix optim bwd; 2024-09-03 02:40:26 +00:00
duanjunwen
591a13bf7e [fix] fix optim bwd; 2024-09-02 11:19:42 +00:00
duanjunwen
77fe44286c [fix] rm zbv in hybridplugin 2024-09-02 10:00:43 +00:00
duanjunwen
6d18d38d5c [feat] update test; rm comments; 2024-09-02 09:50:47 +00:00
duanjunwen
a7b767b071 [fix] fix communication_map; 2024-08-30 05:56:02 +00:00
duanjunwen
8eb6eac225 [fix] fix optim bwd; add license for v_schedule; remove redundant attributes; fix schedule loop "while"--> "for"; add communication dict; 2024-08-30 05:42:43 +00:00
duanjunwen
6af81d8c0d [feat] add fwd_bwd_step, run_fwd_only; 2024-08-30 02:47:52 +00:00
duanjunwen
48ba22dbfd [feat] fix optimizer bwd b & w; support return accum loss & output 2024-08-29 08:54:45 +00:00
duanjunwen
4c4b01b859 [feat] add optim backward_b_by_grad 2024-08-29 03:16:59 +00:00
duanjunwen
fe209164f1 [feat] add apply v_schedule graph; p & p.grad assert err exist; 2024-08-27 10:29:39 +00:00
duanjunwen
8b37323f16 [feat] add run_fwd_bwd_with_microbatch (replace input) & test; add p&p.grad assert close test & all pass; 2024-08-27 09:31:38 +00:00
duanjunwen
9e0bd1af00 [fix] fix ci test; add pytest; 2024-08-27 08:00:23 +00:00
duanjunwen
283c9ff5d2 [fix] rm useless assign and comments; 2024-08-27 07:31:58 +00:00
duanjunwen
1b4bb2beeb [feat] add comments for ZBV func; 2024-08-27 07:11:50 +00:00
duanjunwen
5e09c8b4e1 [feat] split communication and calculation; fix pop empty send_bwd_buffer error; 2024-08-27 06:29:13 +00:00
duanjunwen
1d75045c37 [feat] add test run_fwd_bwd automatic scheduling; 2024-08-26 11:21:56 +00:00
duanjunwen
fd5526b76e Merge branch 'main' into dev/zero_bubble 2024-08-26 04:03:20 +00:00
Wenxuan Tan
7cf9df07bc [Hotfix] Fix llama fwd replacement bug (#6031) 
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
 2024-08-23 15:44:27 +08:00
duanjunwen
c18ef060cf [feat] add dw test; 2024-08-23 06:04:12 +00:00
duanjunwen
ee9baedadf [feat] add zerobubble pp (just a frame now); add POC test for dx_dw; add test for zerobubble; 2024-08-22 10:25:34 +00:00
Hongxin Liu
0d3b0bd864 [plugin] add cast inputs option for zero (#6003) (#6022) 2024-08-21 10:21:26 +08:00
Edenzzzz
dcc44aab8d [misc] Use dist logger in plugins (#6011) 
* use dist logger in plugins

* remove trash

* print on rank 0

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
 2024-08-20 10:32:41 +08:00
Edenzzzz
f1c3266a94 overlap kv comm with output rescale (#6017) 
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
 2024-08-19 14:08:17 +08:00
Hongxin Liu
26493b97d3 [misc] update compatibility (#6008) 
* [misc] update compatibility

* [misc] update requirements

* [devops] disable requirements cache

* [test] fix torch ddp test

* [test] fix rerun on address in use

* [test] fix lazy init
 2024-08-16 18:49:14 +08:00
Edenzzzz
f5c84af0b0 [Feature] Zigzag Ring attention (#5905) 
* halfway

* fix cross-PP-stage position id length diff bug

* fix typo

* fix typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* unified cross entropy func for all shardformer models

* remove redundant lines

* add basic ring attn; debug cross entropy

* fwd bwd logic complete

* fwd bwd logic complete; add experimental triton rescale

* precision tests passed

* precision tests passed

* fix typos and remove misc files

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add sp_mode to benchmark; fix varlen interface

* update softmax_lse shape by new interface

* change tester name

* remove buffer clone; support packed seq layout

* add varlen tests

* fix typo

* all tests passed

* add dkv_group; fix mask

* remove debug statements

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
 2024-08-16 13:56:38 +08:00
Haze188
887d2d579b [misc] Bypass the huggingface bug to solve the mask mismatch problem (#5991) 2024-08-15 14:40:26 +08:00
Hongxin Liu
406f984063 [plugin] add cast inputs option for zero (#6003) 2024-08-15 10:41:22 +08:00
Tong Li
ceb1e262e7 fix sync condition (#6000) 2024-08-14 11:22:39 +08:00
Edenzzzz
b4d2377d4c [Hotfix] Avoid fused RMSnorm import error without apex (#5985) 
Co-authored-by: Edenzzzz <wtan45@wisc.edu>
 2024-08-09 18:17:09 +08:00
Wang Binluo
75c963686f [lora] lora support hybrid parallel plugin (#5956) 
* lora support hybrid plugin

* fix

* fix

* fix

* fix
 2024-08-02 10:36:58 +08:00
botbw
62cdac6b7b [chore] remove redundant test case, print string & reduce test tokens 2024-08-01 10:06:59 +08:00
botbw
d1d1ab871e [moe] solve dp axis issue 2024-08-01 10:06:59 +08:00
botbw
65daa87627 [doc] add MoeHybridParallelPlugin docstring 2024-08-01 10:06:59 +08:00
hxwang
7bedd03739 [moe] remove force_overlap_comm flag and add warning instead 2024-08-01 10:06:59 +08:00
hxwang
f7c5485ed6 [chore] docstring 2024-08-01 10:06:59 +08:00
haze188
7e737df5ad [misc] remove useless condition 2024-08-01 10:06:59 +08:00
haze188
70793ce9ed [misc] fix ci failure: change default value to false in moe plugin 2024-08-01 10:06:59 +08:00
hxwang
606b0891ed [chore] change moe_pg_mesh to private 2024-08-01 10:06:59 +08:00
hxwang
5b4c12381b Revert "[moe] implement submesh initialization" 
This reverts commit 2f9bce6686.
 2024-08-01 10:06:59 +08:00
hxwang
cb01c0d5ce [moe] refactor mesh assignment 2024-08-01 10:06:59 +08:00
haze188
034020bd04 [misc] remove debug/print code 2024-08-01 10:06:59 +08:00
hxwang
c3dc9b4dba [deepseek] replace attn (a workaround for bug in transformers) 2024-08-01 10:06:59 +08:00
hxwang
6c39f0b144 [test] add check 2024-08-01 10:06:59 +08:00