Baizhou Zhang c9625dbb63 [shardformer] support sharded optimizer checkpointIO of HybridParallelPlugin (#4540) ...

* implement sharded optimizer saving

* add more param info

* finish implementation of sharded optimizer saving

* fix bugs in optimizer sharded saving

* add pp+zero test

* param group loading

* greedy loading of optimizer

* fix bug when loading

* implement optimizer sharded saving

* add optimizer test & arrange checkpointIO utils

* fix gemini sharding state_dict

* add verbose option

* add loading of master params

* fix typehint

* fix master/working mapping in fp16 amp

2023-08-31 14:50:47 +08:00

..

components_to_test

[CI] fix typo with tests/ etc. (#3727)

2023-05-11 16:30:58 +08:00

kit

[shardformer] chatglm support sequence parallel (#4482)

2023-08-22 23:59:31 +08:00

test_amp

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_analyzer

[devops] update torch version of CI (#3725)

2023-05-15 17:20:56 +08:00

test_auto_parallel

[gemini] fix argument naming during chunk configuration searching

2023-06-25 13:34:15 +08:00

test_autochunk

[test] fixed tests failed due to dtensor change (#4082)

2023-07-04 16:05:01 +08:00

test_booster

[Shardformer] Merge flash attention branch to pipeline branch (#4362)

2023-08-15 23:25:14 +08:00

test_checkpoint_io

[shardformer] support sharded optimizer checkpointIO of HybridParallelPlugin (#4540)

2023-08-31 14:50:47 +08:00

test_cluster

[cluster] add process group mesh (#4039)

2023-08-15 23:25:14 +08:00

test_comm

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_config

[devops] add large-scale distributed test marker (#4452)

2023-08-16 18:56:52 +08:00

test_context

[devops] add large-scale distributed test marker (#4452)

2023-08-16 18:56:52 +08:00

test_data

[devops] add large-scale distributed test marker (#4452)

2023-08-16 18:56:52 +08:00

test_data_pipeline_tensor_parallel

[CI] fix typo with tests/ etc. (#3727)

2023-05-11 16:30:58 +08:00

test_ddp

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_device

[format] applied code formatting on changed files in pull request 4152 (#4157)

2023-07-04 16:07:47 +08:00

test_engine

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_fx

[misc] resolve code factor issues (#4433)

2023-08-15 23:25:14 +08:00

test_kernels

[Kernels] added triton-implemented of self attention for colossal-ai (#4241)

2023-07-18 23:53:38 +08:00

test_layers

[CI] fix typo with tests/ etc. (#3727)

2023-05-11 16:30:58 +08:00

test_lazy

[test] skip some not compatible models

2023-08-15 23:25:14 +08:00

test_moe

[CI] fix typo with tests/ etc. (#3727)

2023-05-11 16:30:58 +08:00

test_ops

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_optimizer

[bf16] add bf16 support (#3882)

2023-06-05 15:58:31 +08:00

test_pipeline

[shardformer] Pipeline/whisper (#4456)

2023-08-18 21:29:25 +08:00

test_shardformer

[shardformer] fix submodule replacement bug when enabling pp (#4544)

2023-08-31 09:57:18 +08:00

test_tensor

[test] fixed tests failed due to dtensor change (#4082)

2023-07-04 16:05:01 +08:00

test_trainer

[CI] fix typo with tests/ etc. (#3727)

2023-05-11 16:30:58 +08:00

test_utils

[devops] add large-scale distributed test marker (#4452)

2023-08-16 18:56:52 +08:00

test_zero

[hotfix] fix unsafe async comm in zero (#4404)

2023-08-11 15:09:24 +08:00

__init__.py

[zero] Update sharded model v2 using sharded param v2 (#323)

2022-03-11 15:50:28 +08:00