github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-13 13:11:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

[shardformer] add gpt2 test and layer class refactor (#4041)

Browse Source 
* add gpt2 test and layer class refactor

* add dropout in gpt2 policy
This commit is contained in:
FoolPlayer
2023-06-20 11:45:16 +08:00
committed by Frank Lee
parent d857f3dbba
commit 4021b9a8a2
14 changed files with 1400 additions and 840 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
colossalai/shardformer/layer/dropout.py
View File

@@ -4,7 +4,7 @@ import torch
import torch.nn as nn
from torch.distributed import ProcessGroup
from .layers import ParallelModule
from .parallelmodule import ParallelModule
from .utils import create_randomizer_with_offset