github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-22 09:59:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

[hotfix]fix argument naming in docs and examples (#4083)

Browse Source
This commit is contained in:
Baizhou Zhang
2023-06-26 23:50:04 +08:00
committed by GitHub
parent e89b127d8e
commit 4da324cd60
8 changed files with 40 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/source/zh-Hans/advanced_tutorials/parallelize_your_training_like_Megatron.md
View File

@@ -165,7 +165,7 @@ def gemini_zero_dpp(model: torch.nn.Module, pg: ProcessGroup, placement_policy:
device=get_current_device(),
placement_policy=placement_policy,
pin_memory=True,
search_range_mb=32)
search_range_m=32)
return model
```
@@ -174,3 +174,6 @@ def gemini_zero_dpp(model: torch.nn.Module, pg: ProcessGroup, placement_policy:
我们做的上述优化让我们可以在单GPU上训练GPT-2模型，只需要将`run.sh`中设置参数`GPUNUM`=1，再运行文件时就可以在单个GPU上完成模型的训练。
GPT-2 示例在[Train GPT with Colossal-AI](https://github.com/hpcaitech/ColossalAI/tree/main/examples/language/gpt). 获得。
<!-- doc-test-command: torchrun --standalone --nproc_per_node=1 parallelize_your_training_like_Megatron.py -->