[doc] migrate the markdown files (#2652)

This commit is contained in:
Frank Lee
2023-02-09 14:21:38 +08:00
committed by GitHub
parent a020eecc70
commit 85b2303b55
84 changed files with 9729 additions and 0 deletions

View File

@@ -0,0 +1,37 @@
# Setup
## Download From PyPI
You can install Colossal-AI with
```shell
pip install colossalai
```
If you want to build PyTorch extensions during installation, you can use the command below. Otherwise, the PyTorch extensions will be built during runtime.
```shell
CUDA_EXT=1 pip install colossalai
```
## Download From Source
> The version of Colossal-AI will be in line with the main branch of the repository. Feel free to raise an issue if you encounter any problem. :)
```shell
git clone https://github.com/hpcaitech/ColossalAI.git
cd ColossalAI
# install dependency
pip install -r requirements/requirements.txt
# install colossalai
pip install .
```
If you don't want to install and enable CUDA kernel fusion (compulsory installation when using fused optimizer):
```shell
CUDA_EXT=1 pip install .
```

View File

@@ -0,0 +1,19 @@
# Reading Roadmap
Colossal-AI provides a collection of parallel training components for you. We aim to support you with your development
of distributed deep learning models just like how you write single-GPU deep learning models. ColossalAI provides easy-to-use
APIs to help you kickstart your training process. To better how ColossalAI works, we recommend you to read this documentation
in the following order.
- If you are not familiar with distributed system or have never used Colossal-AI, you should first jump into the `Concepts`
section to get a sense of what we are trying to achieve. This section can provide you with some background knowledge on
distributed training as well.
- Next, you can follow the `basics` tutorials. This section will cover the details about how to use Colossal-AI.
- Afterwards, you can try out the features provided in Colossal-AI by reading `features` section. We will provide a codebase for each tutorial. These tutorials will cover the
basic usage of Colossal-AI to realize simple functions such as data parallel and mixed precision training.
- Lastly, if you wish to apply more complicated techniques such as how to run hybrid parallel on GPT-3, the
`advanced tutorials` section is the place to go!
**We always welcome suggestions and discussions from the community, and we would be more than willing to help you if you
encounter any issue. You can raise an [issue](https://github.com/hpcaitech/ColossalAI/issues) here or create a discussion
topic in the [forum](https://github.com/hpcaitech/ColossalAI/discussions).**

View File

@@ -0,0 +1,43 @@
# Quick Demo
Colossal-AI is an integrated large-scale deep learning system with efficient parallelization techniques. The system can
accelerate model training on distributed systems with multiple GPUs by applying parallelization techniques. The system
can also run on systems with only one GPU. Quick demos showing how to use Colossal-AI are given below.
## Single GPU
Colossal-AI can be used to train deep learning models on systems with only one GPU and achieve baseline
performances. We provided an example to [train ResNet on CIFAR10 dataset](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/resnet)
with only one GPU. You can find the example in [ColossalAI-Examples](https://github.com/hpcaitech/ColossalAI-Examples).
Detailed instructions can be found in its `README.md`.
## Multiple GPUs
Colossal-AI can be used to train deep learning models on distributed systems with multiple GPUs and accelerate the
training process drastically by applying efficient parallelization techniques. When we have several parallelism for you
to try out.
#### 1. data parallel
You can use the same [ResNet example](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/resnet) as the
single-GPU demo above. By setting `--nproc_per_node` to be the number of GPUs you have on your machine, the example
is turned into a data parallel example.
#### 2. hybrid parallel
Hybrid parallel includes data, tensor, and pipeline parallelism. In Colossal-AI, we support different types of tensor
parallelism (i.e. 1D, 2D, 2.5D and 3D). You can switch between different tensor parallelism by simply changing the configuration
in the `config.py`. You can follow the [GPT example](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/language/gpt).
Detailed instructions can be found in its `README.md`.
#### 3. MoE parallel
We provided [an example of WideNet](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/image/widenet) to demonstrate
MoE parallelism. WideNet uses mixture of experts (MoE) to achieve better performance. More details can be found in
[Tutorial: Integrate Mixture-of-Experts Into Your Model](../advanced_tutorials/integrate_mixture_of_experts_into_your_model.md)
#### 4. sequence parallel
Sequence parallel is designed to tackle memory efficiency and sequence length limit problems in NLP tasks. We provided
[an example of BERT](https://github.com/hpcaitech/ColossalAI-Examples/tree/main/language/bert/sequene_parallel) in
[ColossalAI-Examples](https://github.com/hpcaitech/ColossalAI-Examples). You can follow the `README.md` to execute the code.