github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-01 17:17:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

[example] simplify the GPT2 huggingface example (#1826)

Browse Source
This commit is contained in:
Jiarui Fang
2022-11-08 16:14:07 +08:00
committed by GitHub
parent cd5a0d56fa
commit b1263d32ba
37 changed files with 177 additions and 27611 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
examples/language/opt/README.md
View File

@@ -22,6 +22,9 @@ The following example of [Colossal-AI](https://github.com/hpcaitech/ColossalAI)
We are using the pre-training weights of the OPT model provided by Hugging Face Hub on the raw WikiText-2 (no tokens were replaced before
the tokenization). This training script is adapted from the [HuggingFace Language Modelling examples](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling).
## Our Modifications
We adapt the OPT training code to ColossalAI by leveraging Gemini and ZeRO DDP.
## Quick Start
You can launch training by using the following bash script