github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-01 17:17:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

[example] make gpt example directory more clear (#2353)

Browse Source
This commit is contained in:
Jiarui Fang
2023-01-06 11:11:26 +08:00
committed by GitHub
parent 7027540d3d
commit 509a87f3ff
13 changed files with 26 additions and 584 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
examples/language/gpt/README.md
View File

@@ -57,12 +57,6 @@ The `train_gpt_demo.py` provides three distributed plans, you can choose the pla
- Pytorch ZeRO
### Pipeline Parallel
```bash
bash run_pp.sh
```
## Performance
Testbed: a cluster of 8xA100 (80GB) and 1xAMD EPYC 7543 32-Core Processor (512 GB). GPUs are connected via PCI-e.
@@ -119,3 +113,9 @@ Touch the bar of model scale and batch size.
| model | #GPU | policy | TP | batch per DP | Tflops |
| ---------- | --------- |--------- |--------- |--------- |--------- |
| gpt2_20b | 8 | cpu | 2 | 8 | 46.895 |
### Experimental Features
#### [Pipeline Parallel](./experiments/pipeline_parallel/)
#### [Auto Parallel](./experiments/auto_parallel_with_gpt/)