github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-05 02:51:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

[example] add zero1, zero2 example in GPT examples (#2146)

Browse Source 
* [example] add zero1 and zero2 for GPT

* update readme in gpt example

* polish code

* change init value

* update readme
This commit is contained in:
HELSON
2022-12-20 14:30:27 +08:00
committed by GitHub
parent 1cce6e36ca
commit a7d95b7024
5 changed files with 40 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
colossalai/zero/sharded_optim/low_level_optim.py
View File

@@ -35,13 +35,13 @@ class LowLevelZeroOptimizer(ColossalaiOptimizer):
optimizer: Optimizer,
# grad scaler config
initial_scale=2**32,
initial_scale=2**16,
min_scale=1,
growth_factor=2,
backoff_factor=0.5,
growth_interval=1000,
growth_interval=2000,
hysteresis=2,
max_scale: int = 2**32,
max_scale: int = 2**24,
# grad clipping
clip_grad_norm=0.0,