mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2025-09-13 05:01:44 +00:00
add Chinese README
This commit is contained in:
64
README.md
64
README.md
@@ -13,9 +13,52 @@
|
||||
[](https://github.com/hpcaitech/ColossalAI/actions/workflows/PR_CI.yml)
|
||||
[](https://colossalai.readthedocs.io/en/latest/?badge=latest)
|
||||
[](https://codebeat.co/projects/github-com-hpcaitech-colossalai-main)
|
||||
|
||||
| [English](README.md) | [中文](README-zh-Hans.md) |
|
||||
</div>
|
||||
An integrated large-scale model training system with efficient parallelization techniques.
|
||||
|
||||
|
||||
## Features
|
||||
|
||||
Colossal-AI provides a collection of parallel training components for you. We aim to support you to write your
|
||||
distributed deep learning models just like how you write your single-GPU model. We provide friendly tools to kickstart
|
||||
distributed training in a few lines.
|
||||
|
||||
- Data Parallelism
|
||||
- Pipeline Parallelism
|
||||
- 1D, 2D, 2.5D, 3D tensor parallelism
|
||||
- Sequence parallelism
|
||||
- Friendly trainer and engine
|
||||
- Extensible for new parallelism
|
||||
- Mixed Precision Training
|
||||
- Zero Redundancy Optimizer (ZeRO)
|
||||
|
||||
## Examples
|
||||
### ViT
|
||||
|
||||
<img src="./docs/images/ViT_TP.png" width="400" />
|
||||
|
||||
- 14x larger batch size
|
||||
- 5x faster training
|
||||
|
||||
### GPT-3 & GPT-2
|
||||
|
||||

|
||||
|
||||
- Free 50% GPU resources, or 10.7% acceleration for GPT-3
|
||||
- 11x lower GPU RAM, or superlinear scaling for GPT-2
|
||||
|
||||
### BERT
|
||||
|
||||

|
||||
|
||||
- 2x faster training
|
||||
- 50% longer sequence length
|
||||
|
||||
Please visit our [documentation and tutorials](https://www.colossalai.org/) for more details.
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
### PyPI
|
||||
@@ -37,7 +80,7 @@ pip install colossalai[zero]
|
||||
|
||||
### Install From Source
|
||||
|
||||
> The documentation will be in line with the main branch of the repository. Feel free to raise an issue if you encounter any problem. :)
|
||||
> The version of Colossal-AI will be in line with the main branch of the repository. Feel free to raise an issue if you encounter any problem. :)
|
||||
|
||||
```shell
|
||||
git clone https://github.com/hpcaitech/ColossalAI.git
|
||||
@@ -107,13 +150,13 @@ train_dataloader = get_dataloader(dataset=dataset,
|
||||
)
|
||||
|
||||
|
||||
# build your
|
||||
# build your optimizer
|
||||
optimizer = ...
|
||||
|
||||
# build your loss function
|
||||
criterion = ...
|
||||
|
||||
# build your lr_scheduler
|
||||
# initialize colossalai
|
||||
engine, train_dataloader, _, _ = colossalai.initialize(
|
||||
model=model,
|
||||
optimizer=optimizer,
|
||||
@@ -157,21 +200,6 @@ class MLP_2D(nn.Module):
|
||||
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
Colossal-AI provides a collection of parallel training components for you. We aim to support you to write your
|
||||
distributed deep learning models just like how you write your single-GPU model. We provide friendly tools to kickstart
|
||||
distributed training in a few lines.
|
||||
|
||||
- Data Parallelism
|
||||
- Pipeline Parallelism
|
||||
- 1D, 2D, 2.5D, 3D and sequence parallelism
|
||||
- Friendly trainer and engine
|
||||
- Extensible for new parallelism
|
||||
- Mixed Precision Training
|
||||
- Zero Redundancy Optimizer (ZeRO)
|
||||
|
||||
Please visit our [documentation and tutorials](https://www.colossalai.org/) for more details.
|
||||
|
||||
## Cite Us
|
||||
|
||||
|
Reference in New Issue
Block a user