mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2026-07-12 10:54:49 +00:00
60abd86d6a
* [example] enhence GPT demo * Update README.md Co-authored-by: binmakeswell <binmakeswell@gmail.com>
516 B
516 B
Overview
This example shows how to use Colossal-AI to run huggingface GPT training in distributed manners.
GPT
We use the GPT2 model from huggingface transformers. The input data is randonly generated.
Our Modifications
The train_gpt_demo.py provides three distributed plans, i.e. Colossal-AI, PyTorch DDP and ZeRO.
The Colossal-AI leverages Tensor Parallel and Gemini.
Quick Start
You can launch training by using the following bash script.
pip install -r requirements.txt
bash run.sh