github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-01 17:17:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

[example] enhance GPT demo (#1959)

Browse Source 
* [example] enhence GPT demo

* Update README.md

Co-authored-by: binmakeswell <binmakeswell@gmail.com>
This commit is contained in:
Jiarui Fang
2022-11-16 11:36:27 +08:00
committed by GitHub
parent acba142929
commit 60abd86d6a
3 changed files with 87 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
examples/language/gpt/README.md
View File

@@ -1,14 +1,15 @@
## Overview
This example shows how to use ColossalAI to run huggingface GPT training in distributed manners.
This example shows how to use Colossal-AI to run huggingface GPT training in distributed manners.
## GPT
We use the huggingface transformers GPT2 model. The input data is randonly generated.
We use the GPT2 model from huggingface transformers. The input data is randonly generated.
## Our Modifications
We adapt the OPT training code to ColossalAI by leveraging Gemini and ZeRO DDP.
The `train_gpt_demo.py` provides three distributed plans, i.e. Colossal-AI, PyTorch DDP and ZeRO.
The Colossal-AI leverages Tensor Parallel and Gemini.
## Quick Start
You can launch training by using the following bash script
You can launch training by using the following bash script.
```bash
pip install -r requirements.txt