Edenzzzz fbf33ecd01 [Feature] Enable PP + SP for llama (#5868) ...

* fix cross-PP-stage position id length diff bug

* fix typo

* fix typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use a one cross entropy func for all shardformer models

---------

Co-authored-by: Edenzzzz <wtan45@wisc.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

2024-07-09 18:05:20 +08:00

..

moe

[MoE/ZeRO] Moe refactor with zero refactor (#5821)

2024-06-28 14:00:08 +08:00

__init__.py

[Feature] Enable PP + SP for llama (#5868)

2024-07-09 18:05:20 +08:00

_operation.py

[shardformer] Sequence Parallelism Optimization (#5533)

2024-04-03 17:15:47 +08:00

attn.py

[Inference] Fix flash-attn import and add model test (#5794)

2024-06-12 14:13:50 +08:00

dropout.py

[misc] update pre-commit and run all files (#4752)

2023-09-19 14:20:26 +08:00

embedding.py

[Inference] Fix bugs and docs for feat/online-server (#5598)

2024-05-08 15:20:53 +00:00

linear.py

[shardformer] refactor embedding resize (#5603)

2024-04-18 16:10:18 +08:00

loss.py

[Feature] Enable PP + SP for llama (#5868)

2024-07-09 18:05:20 +08:00

normalization.py

Remove CohereLayerNorm and use existing layernorm

2024-06-18 02:32:41 +00:00

parallel_module.py

[shardformer] refactor embedding resize (#5603)

2024-04-18 16:10:18 +08:00

qkv_fused_linear.py

[Inference]Lazy Init Support (#5785)

2024-06-27 18:02:15 +08:00

utils.py

[shardformer] Sequence Parallelism Optimization (#5533)

2024-04-03 17:15:47 +08:00