github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-06 11:32:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

[tutorial] polish all README (#1946)

Browse Source
This commit is contained in:
binmakeswell
2022-11-14 19:49:32 +08:00
committed by GitHub
parent de56b563b9
commit 9183e0dec5
8 changed files with 264 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
examples/tutorial/sequence_parallel/README.md
View File

@@ -5,6 +5,15 @@ activation along the sequence dimension. This method can achieve better memory e
Paper: [Sequence Parallelism: Long Sequence Training from System Perspective](https://arxiv.org/abs/2105.13120)
## 🚀Quick Start
1. Run with the following command
```bash
export PYTHONPATH=$PWD
colossalai run --nproc_per_node 4 train.py -s
```
2. The default config is sequence parallel size = 2, pipeline size = 1, let’s change pipeline size to be 2 and try it again.
## How to Prepare WikiPedia Dataset
First, let's prepare the WikiPedia dataset from scratch. To generate a preprocessed dataset, we need four items: