5 Commits

Author	SHA1	Message	Date
HELSON	1468e4bcfc	[zero] add constant placement policy (#1705) ... * fixes memory leak when paramter is in fp16 in ZeroDDP init. * bans chunk releasement in CUDA. Only when a chunk is about to offload, it is allowed to release. * adds a constant placement policy. With it, users can allocate a reserved caching memory space for parameters.	2022-10-14 17:53:16 +08:00
HELSON	95c35f73bd	[moe] initialize MoE groups by ProcessGroup (#1640)	2022-09-23 17:20:41 +08:00
Jiarui Fang	a445e118cf	[polish] polish singleton and global context (#500)	2022-03-23 18:03:39 +08:00
Jiarui Fang	65c0f380c2	[format] polish name format for MOE (#481)	2022-03-21 23:19:47 +08:00
HELSON	84fd7c1d4d	add moe context, moe utilities and refactor gradient handler (#455)	2022-03-18 16:38:32 +08:00