github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-02 01:28:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add GPT PP Example (#2272)

Browse Source 
Co-authored-by: Ziyue Jiang <ziyue.jiang@gmail.com>
This commit is contained in:
Ziyue Jiang
2023-01-03 15:17:26 +08:00
committed by GitHub
parent 4b29112ab2
commit 9654df0e9a
3 changed files with 145 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
examples/language/gpt/README.md
View File

@@ -44,6 +44,11 @@ For simplicity, the input data is randonly generated here.
bash run.sh
```
Pipeline Parallel
```bash
bash run_pp.sh
```
### Training config
The `train_gpt_demo.py` provides three distributed plans, you can choose the plan you want in `run.sh`. The Colossal-AI leverages Tensor Parallel and Gemini + ZeRO DDP.