github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-21 17:40:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

[checkpointio] support unsharded checkpointIO for hybrid parallel (#4774)

Browse Source 
* support unsharded saving/loading for model

* support optimizer unsharded saving

* update doc

* support unsharded loading for optimizer

* small fix
This commit is contained in:
Baizhou Zhang
2023-09-26 10:58:03 +08:00
committed by GitHub
parent a2db75546d
commit 64a08b2dc3
4 changed files with 197 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/zh-Hans/basics/booster_plugins.md
View File

@@ -71,8 +71,6 @@ Zero-2 不支持局部梯度累积。如果您坚持使用，虽然可以积累
> ⚠ 在使用该插件的时候, 只有支持Shardformer的部分Huggingface transformers模型才能够使用张量并行、流水线并行以及优化工具。Llama 1、Llama 2、OPT、Bloom、Bert以及GPT2等主流transformers模型均已支持Shardformer。
> ⚠ 该插件当前只对模型和优化器支持分片的checkpoint方法。不分片的checkpoint方法会在未来的版本中被支持。
{{ autodoc:colossalai.booster.plugin.HybridParallelPlugin }}
### Torch DDP 插件