moved env variables to global variables; (#215)

added branch context;
added vocab parallel layers;
moved split_batch from load_batch to tensor parallel embedding layers;
updated gpt model;
updated unit test cases;
fixed few collective communicator bugs
This commit is contained in:
アマデウス
2022-02-14 11:15:02 +08:00
committed by Frank Lee
parent b82d60be02
commit 9ee197d0e9
63 changed files with 4304 additions and 1040 deletions

View File

@@ -4,8 +4,9 @@
import torch.nn as nn
try:
import apex.amp as apex_amp
except:
pass
except ImportError:
raise ImportError('Cannot import apex.amp correctly.')
from torch import Tensor
from colossalai.nn.optimizer import ColossalaiOptimizer