botbw 9b9b76bdcd [moe] add mixtral dp grad scaling when not all experts are activated

2024-08-01 10:06:59 +08:00

..

__init__.py

[MoE/ZeRO] Moe refactor with zero refactor (#5821)

2024-06-28 14:00:08 +08:00

_operation.py

[moe] add mixtral dp grad scaling when not all experts are activated

2024-08-01 10:06:59 +08:00

load_balance.py

[MoE/ZeRO] Moe refactor with zero refactor (#5821)

2024-06-28 14:00:08 +08:00

manager.py

fix some typo (#5307)

2024-01-25 13:56:27 +08:00

utils.py

[MoE/ZeRO] Moe refactor with zero refactor (#5821)

2024-06-28 14:00:08 +08:00