mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2025-09-09 13:00:52 +00:00
[doc] migrate the markdown files (#2652)
This commit is contained in:
37
docs/source/en/get_started/installation.md
Normal file
37
docs/source/en/get_started/installation.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# Setup
|
||||
|
||||
## Download From PyPI
|
||||
|
||||
You can install Colossal-AI with
|
||||
|
||||
```shell
|
||||
pip install colossalai
|
||||
```
|
||||
|
||||
If you want to build PyTorch extensions during installation, you can use the command below. Otherwise, the PyTorch extensions will be built during runtime.
|
||||
|
||||
```shell
|
||||
CUDA_EXT=1 pip install colossalai
|
||||
```
|
||||
|
||||
|
||||
## Download From Source
|
||||
|
||||
> The version of Colossal-AI will be in line with the main branch of the repository. Feel free to raise an issue if you encounter any problem. :)
|
||||
|
||||
```shell
|
||||
git clone https://github.com/hpcaitech/ColossalAI.git
|
||||
cd ColossalAI
|
||||
|
||||
# install dependency
|
||||
pip install -r requirements/requirements.txt
|
||||
|
||||
# install colossalai
|
||||
pip install .
|
||||
```
|
||||
|
||||
If you don't want to install and enable CUDA kernel fusion (compulsory installation when using fused optimizer):
|
||||
|
||||
```shell
|
||||
CUDA_EXT=1 pip install .
|
||||
```
|
19
docs/source/en/get_started/reading_roadmap.md
Normal file
19
docs/source/en/get_started/reading_roadmap.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# Reading Roadmap
|
||||
|
||||
Colossal-AI provides a collection of parallel training components for you. We aim to support you with your development
|
||||
of distributed deep learning models just like how you write single-GPU deep learning models. ColossalAI provides easy-to-use
|
||||
APIs to help you kickstart your training process. To better how ColossalAI works, we recommend you to read this documentation
|
||||
in the following order.
|
||||
|
||||
- If you are not familiar with distributed system or have never used Colossal-AI, you should first jump into the `Concepts`
|
||||
section to get a sense of what we are trying to achieve. This section can provide you with some background knowledge on
|
||||
distributed training as well.
|
||||
- Next, you can follow the `basics` tutorials. This section will cover the details about how to use Colossal-AI.
|
||||
- Afterwards, you can try out the features provided in Colossal-AI by reading `features` section. We will provide a codebase for each tutorial. These tutorials will cover the
|
||||
basic usage of Colossal-AI to realize simple functions such as data parallel and mixed precision training.
|
||||
- Lastly, if you wish to apply more complicated techniques such as how to run hybrid parallel on GPT-3, the
|
||||
`advanced tutorials` section is the place to go!
|
||||
|
||||
**We always welcome suggestions and discussions from the community, and we would be more than willing to help you if you
|
||||
encounter any issue. You can raise an [issue](https://github.com/hpcaitech/ColossalAI/issues) here or create a discussion
|
||||
topic in the [forum](https://github.com/hpcaitech/ColossalAI/discussions).**
|
43
docs/source/en/get_started/run_demo.md
Normal file
43
docs/source/en/get_started/run_demo.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# Quick Demo
|
||||
|
||||
Colossal-AI is an integrated large-scale deep learning system with efficient parallelization techniques. The system can
|
||||
accelerate model training on distributed systems with multiple GPUs by applying parallelization techniques. The system
|
||||
can also run on systems with only one GPU. Quick demos showing how to use Colossal-AI are given below.
|
||||
|
||||
## Single GPU
|
||||
|
||||
Colossal-AI can be used to train deep learning models on systems with only one GPU and achieve baseline
|
||||
performances. We provided an example to [train ResNet on CIFAR10 dataset](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/resnet)
|
||||
with only one GPU. You can find the example in [ColossalAI-Examples](https://github.com/hpcaitech/ColossalAI-Examples).
|
||||
Detailed instructions can be found in its `README.md`.
|
||||
|
||||
## Multiple GPUs
|
||||
|
||||
Colossal-AI can be used to train deep learning models on distributed systems with multiple GPUs and accelerate the
|
||||
training process drastically by applying efficient parallelization techniques. When we have several parallelism for you
|
||||
to try out.
|
||||
|
||||
#### 1. data parallel
|
||||
|
||||
You can use the same [ResNet example](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/resnet) as the
|
||||
single-GPU demo above. By setting `--nproc_per_node` to be the number of GPUs you have on your machine, the example
|
||||
is turned into a data parallel example.
|
||||
|
||||
#### 2. hybrid parallel
|
||||
|
||||
Hybrid parallel includes data, tensor, and pipeline parallelism. In Colossal-AI, we support different types of tensor
|
||||
parallelism (i.e. 1D, 2D, 2.5D and 3D). You can switch between different tensor parallelism by simply changing the configuration
|
||||
in the `config.py`. You can follow the [GPT example](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/language/gpt).
|
||||
Detailed instructions can be found in its `README.md`.
|
||||
|
||||
#### 3. MoE parallel
|
||||
|
||||
We provided [an example of WideNet](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/widenet) to demonstrate
|
||||
MoE parallelism. WideNet uses mixture of experts (MoE) to achieve better performance. More details can be found in
|
||||
[Tutorial: Integrate Mixture-of-Experts Into Your Model](../advanced_tutorials/integrate_mixture_of_experts_into_your_model.md)
|
||||
|
||||
#### 4. sequence parallel
|
||||
|
||||
Sequence parallel is designed to tackle memory efficiency and sequence length limit problems in NLP tasks. We provided
|
||||
[an example of BERT](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/language/bert/sequene_parallel) in
|
||||
[ColossalAI-Examples](https://github.com/hpcaitech/ColossalAI-Examples). You can follow the `README.md` to execute the code.
|
Reference in New Issue
Block a user