github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-23 10:30:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

[example] add gpt2 HybridParallelPlugin example (#4653)

Browse Source 
* add gpt2 HybridParallelPlugin example

* update readme and testci

* update test ci

* fix test_ci bug

* update requirements

* add requirements

* update requirements

* add requirement

* rename file
This commit is contained in:
Bin Jia
2023-09-15 17:12:46 +08:00
committed by GitHub
parent 6a03c933a0
commit 608cffaed3
6 changed files with 449 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
examples/language/gpt/hybridparallelism/run.sh Normal file
View File

@@ -0,0 +1,5 @@
# load via internet
torchrun --standalone --nproc_per_node 4 --master_port 29800 finetune.py --target_f1 0.6 --plugin hybrid_parallel --model_type "gpt2"
# load from local
# torchrun --standalone --nproc_per_node 4 --master_port 29800 finetune.py --target_f1 0.6 --plugin hybrid_parallel --model_type "gpt2" --pretrained_path "your/path/to/pretrained_model"