mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2026-05-07 04:28:58 +00:00
fix zero3 fp16 and add zero3 model context (#62)
This commit is contained in:
@@ -23,7 +23,7 @@ ZeRO优化器可以切分三种模型状态(优化器状态、梯度、参数
|
||||
)
|
||||
|
||||
zero = dict(
|
||||
type='ZeroRedundancyOptimizer_Level_3',
|
||||
level=3,
|
||||
dynamic_loss_scale=True,
|
||||
clip_grad=1.0
|
||||
)
|
||||
@@ -78,4 +78,13 @@ ZeRO优化器可以切分三种模型状态(优化器状态、梯度、参数
|
||||
|
||||
### 使用ZeRO优化器进行训练
|
||||
|
||||
注意,当使用ZeRO-3时,如果您的模型过大以至于无法放入内存, 您应该使用`colossalai.zero.zero3_model_context`来构建您的模型:
|
||||
|
||||
```python
|
||||
from colossalai.zero import zero3_model_context
|
||||
|
||||
with zero3_model_context():
|
||||
model = Model()
|
||||
```
|
||||
|
||||
如果您完成了上述配置,可以运行`colossalai.initialize()`来开始您的训练。
|
||||
|
||||
Reference in New Issue
Block a user