flybird1111 7a3dfd0c64 [shardformer] update shardformer to use flash attention 2 (#4392) ...

* cherry-pick flash attention 2

cherry-pick flash attention 2

* [shardformer] update shardformer to use flash attention 2

[shardformer] update shardformer to use flash attention 2, fix

[shardformer] update shardformer to use flash attention 2, fix

[shardformer] update shardformer to use flash attention 2, fix

2023-08-15 23:25:14 +08:00

..

test_checkpoint

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_activation_checkpointing.py

[CI] fix some spelling errors (#3707)

2023-05-10 17:12:03 +08:00

test_colo_checkpoint.py

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_commons.py

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_flash_attention.py

[shardformer] update shardformer to use flash attention 2 (#4392)

2023-08-15 23:25:14 +08:00

test_memory.py

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_norm_gradient_clipping.py

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00

test_zero_gradient_clippling.py

[test] refactor tests with spawn (#3452)

2023-04-06 14:51:35 +08:00