mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2026-07-10 15:55:36 +00:00
17 lines
439 B
Markdown
17 lines
439 B
Markdown
## Overview
|
|
This example shows how to use ColossalAI to run huggingface GPT training with Gemini and ZeRO DDP.
|
|
|
|
## GPT
|
|
We use the huggingface transformers GPT2 model. The input data is randonly generated.
|
|
|
|
## Our Modifications
|
|
We adapt the OPT training code to ColossalAI by leveraging Gemini and ZeRO DDP.
|
|
|
|
## Quick Start
|
|
You can launch training by using the following bash script
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
bash run.sh
|
|
```
|