ColossalAI

mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-07-12 19:06:17 +00:00

Files

Jiarui Fang 60abd86d6a [example] enhance GPT demo (#1959 )

* [example] enhence GPT demo

* Update README.md

Co-authored-by: binmakeswell <binmakeswell@gmail.com>

2022-11-16 11:36:27 +08:00

README.md

[example] enhance GPT demo (#1959 )

2022-11-16 11:36:27 +08:00

requirements.txt

[example] simplify the GPT2 huggingface example (#1826 )

2022-11-08 16:14:07 +08:00

run.sh

[example] enhance GPT demo (#1959 )

2022-11-16 11:36:27 +08:00

train_gpt_demo.py

[example] enhance GPT demo (#1959 )

2022-11-16 11:36:27 +08:00

README.md

Overview

This example shows how to use Colossal-AI to run huggingface GPT training in distributed manners.

GPT

We use the GPT2 model from huggingface transformers. The input data is randonly generated.

Our Modifications

The train_gpt_demo.py provides three distributed plans, i.e. Colossal-AI, PyTorch DDP and ZeRO. The Colossal-AI leverages Tensor Parallel and Gemini.

Quick Start

You can launch training by using the following bash script.

pip install -r requirements.txt
bash run.sh