github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-08-30 15:01:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
feature/zerobubble
ColossalAI/tests/test_shardformer/test_model
History
duanjunwen aed20fb2df
[feat] support zbv in mixtral benchmark; (#6083) 
* [feat] support zbv in mixtral benchmark;

* [fix] MixtralForCausalLMPolicy get_held_layer support zbv;

* [feat] update MixtralPipelineForwards --> mixtral_model_forward; support zbv;

* [feat] support MixtralPipelineForwards--> mixtral_for_causal_lm_forward for zbv

* [fix] fix llama, mixtral benchmark zbv loss none bug; update mixtral & llama policy and modeling;

* [feat] Linear1D_COL/ROW support zbv WeightGradStore;

* [feat] support use_zbv in llama, mixtral modeling; only replace Linear1D_Col/Row policy;

* [fix] fix test case; moe error in second iter

* [feat]EPMixtralSparseMoeBlock (op in MOE) support zbv;

* [fix] fix bwd b; now bwd w only for Layer replaced by Linear1D_Col/Row; other layer perform a fully bwd;

* [fix] debug zbv llama test;

* [fix] rm use_zbv flag in Shardconfig; rm debug info;

* [fix] add & fix  llama test

* [feat] support meta cache, meta_grad_send, meta_tensor_send; fix runtime too long in Recv Bwd; benchmark for llama + Hybrid(tp+pp);

* [fix\ fix fail case test_shard_llama

* [fix] fix test_shard_llama

* [fix] fix llama modeling policy;

* [fix] fix test_shard_llama ci;

* [fix] fix test zerobubble

* [fix] fix handle name; rm useless comments;

* [fix] fix send recv signature;

* [fix] fix comment in llama & benchmark

* [feat] support no tensor parallel Linear in shardformer; Add test for use weightGradStore and not use WeightGradStore

* [fix] fix linear (no tp) ops func name;
2024-10-31 18:17:29 +08:00
..
__init__.py [shardformer] adapted T5 and LLaMa test to use kit (#4049) 2023-07-04 16:05:01 +08:00
_utils.py [zerobubble] rebase main (#6075) 2024-10-08 15:58:00 +08:00
test_shard_bert.py [CI/tests] simplify some test case to reduce testing time (#5755) 2024-06-04 13:57:54 +08:00
test_shard_blip2.py [CI/tests] simplify some test case to reduce testing time (#5755) 2024-06-04 13:57:54 +08:00
test_shard_bloom.py [CI/tests] simplify some test case to reduce testing time (#5755) 2024-06-04 13:57:54 +08:00
test_shard_chatglm2.py [zerobubble] rebase main (#6075) 2024-10-08 15:58:00 +08:00
test_shard_command.py [zerobubble] rebase main (#6075) 2024-10-08 15:58:00 +08:00
test_shard_deepseek.py [zerobubble] rebase main (#6075) 2024-10-08 15:58:00 +08:00
test_shard_falcon.py [CI/tests] simplify some test case to reduce testing time (#5755) 2024-06-04 13:57:54 +08:00
test_shard_gpt2.py [zerobubble] rebase main (#6075) 2024-10-08 15:58:00 +08:00
test_shard_gptj.py [release] update version (#5752) 2024-05-31 19:40:26 +08:00
test_shard_llama.py [feat] support zbv in mixtral benchmark; (#6083) 2024-10-31 18:17:29 +08:00
test_shard_mistral.py [misc] refactor launch API and tensor constructor (#5666) 2024-04-29 10:40:11 +08:00
test_shard_mixtral.py [zerobubble] rebase main (#6075) 2024-10-08 15:58:00 +08:00
test_shard_opt.py [misc] refactor launch API and tensor constructor (#5666) 2024-04-29 10:40:11 +08:00
test_shard_qwen2.py [zerobubble] rebase main (#6075) 2024-10-08 15:58:00 +08:00
test_shard_sam.py [misc] refactor launch API and tensor constructor (#5666) 2024-04-29 10:40:11 +08:00
test_shard_t5.py [shardformer] Support the T5ForTokenClassification model (#5816) 2024-06-27 16:40:38 +08:00
test_shard_vit.py [CI/tests] simplify some test case to reduce testing time (#5755) 2024-06-04 13:57:54 +08:00
test_shard_whisper.py [misc] refactor launch API and tensor constructor (#5666) 2024-04-29 10:40:11 +08:00