github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-07 03:52:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

[doc] update amp document

Browse Source
This commit is contained in:
Mingyan Jiang
2023-05-23 13:11:03 +08:00
committed by jiangmingyan
parent 9265f2d4d7
commit 8c62e50dbb
5 changed files with 24 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/source/en/basics/define_your_config.md
View File

@@ -2,9 +2,6 @@
Author: Guangyang Lu, Shenggui Li, Siqi Mai
> ⚠️ The information on this page is outdated and will be deprecated. Please check [Booster API](../basics/booster_api.md) for more information.
**Prerequisite:**
- [Distributed Training](../concepts/distributed_training.md)
- [Colossal-AI Overview](../concepts/colossalai_overview.md)
@@ -24,8 +21,7 @@ In this tutorial, we will cover how to define your configuration file.
## Configuration Definition
In a configuration file, there are two types of variables. One serves as feature specification and the other serves
as hyper-parameters. All feature-related variables are reserved keywords. For example, if you want to use mixed precision
training, you need to use the variable name `fp16` in the config file and follow a pre-defined format.
as hyper-parameters. All feature-related variables are reserved keywords. For example, if you want to use 1D tensor parallelism, you need to use the variable name `parallel` in the config file and follow a pre-defined format.
### Feature Specification
@@ -37,14 +33,13 @@ To illustrate the use of config file, we use mixed precision training as an exam
follow the steps below.
1. create a configuration file (e.g. `config.py`, the file name can be anything)
2. define the mixed precision configuration in the config file. For example, in order to use mixed precision training
natively provided by PyTorch, you can just write these lines of code below into your config file.
2. define the hybrid parallelism configuration in the config file. For example, in order to use 1D tensor parallel, you can just write these lines of code below into your config file.
```python
from colossalai.amp import AMP_TYPE
fp16 = dict(
mode=AMP_TYPE.TORCH
parallel = dict(
data=1,
pipeline=1,
tensor=dict(size=2, mode='1d'),
)
```
@@ -57,7 +52,7 @@ the current directory.
colossalai.launch(config='./config.py', ...)
```
In this way, Colossal-AI knows what features you want to use and will inject this feature during `colossalai.initialize`.
In this way, Colossal-AI knows what features you want to use and will inject this feature.
### Global Hyper-parameters
@@ -83,3 +78,4 @@ colossalai.launch(config='./config.py', ...)
print(gpc.config.BATCH_SIZE)
```
<!-- doc-test-command: echo "define_your_config.md does not need test" -->

3
docs/source/en/features/mixed_precision_training.md
View File

@@ -1,4 +1,4 @@
# Auto Mixed Precision Training
# Auto Mixed Precision Training (Outdated)
Author: Chuanrui Wang, Shenggui Li, Yongbin Li
@@ -365,3 +365,4 @@ Use the following command to start the training scripts. You can change `--nproc
```python
python -m torch.distributed.launch --nproc_per_node 4 --master_addr localhost --master_port 29500 train_with_engine.py --config config/config_AMP_torch.py
```
<!-- doc-test-command: torchrun --standalone --nproc_per_node=1 mixed_precision_training.py -->