github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-08-01 07:46:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge e44e65d61c into 97f4bee9d8

Browse Source
This commit is contained in:
tongyu 2025-06-19 18:07:13 +08:00 committed by GitHub
commit 3c6e0adfab
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
colossalai/shardformer/README.md
View File

@ -83,6 +83,12 @@ Following are the description `ShardConfig`'s arguments:
- `extra_kwargs`: A dict to store extra kwargs for ShardFormer. - `extra_kwargs`: A dict to store extra kwargs for ShardFormer.
- - `tensor_parallel_mode`: A choice of parallel modes for tensor operations. Supported modes are:
- `'1d'`: 1D tensor parallelism
- `'2d'`: 2D tensor parallelism
- `'2.5d'`: 2.5D tensor parallelism
- `'3d'`: 3D tensor parallelism
### Write your own policy ### Write your own policy
If you have a custom model, you can also use Shardformer to parallelize it by writing your own sharding policy. More information about the sharding policy can be found in [API Design](#-api-design). If you have a custom model, you can also use Shardformer to parallelize it by writing your own sharding policy. More information about the sharding policy can be found in [API Design](#-api-design).