mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2026-07-13 20:39:59 +00:00
60abd86d6a
* [example] enhence GPT demo * Update README.md Co-authored-by: binmakeswell <binmakeswell@gmail.com>
18 lines
516 B
Markdown
18 lines
516 B
Markdown
## Overview
|
|
This example shows how to use Colossal-AI to run huggingface GPT training in distributed manners.
|
|
|
|
## GPT
|
|
We use the GPT2 model from huggingface transformers. The input data is randonly generated.
|
|
|
|
## Our Modifications
|
|
The `train_gpt_demo.py` provides three distributed plans, i.e. Colossal-AI, PyTorch DDP and ZeRO.
|
|
The Colossal-AI leverages Tensor Parallel and Gemini.
|
|
|
|
## Quick Start
|
|
You can launch training by using the following bash script.
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
bash run.sh
|
|
```
|