yuehuayingxueluo 87079cffe8 [Inference]Support FP16/BF16 Flash Attention 2 And Add high_precision Flag To Rotary Embedding (#5461) ...

* Support FP16/BF16 Flash Attention 2

* fix bugs in test_kv_cache_memcpy.py

* add context_kv_cache_memcpy_kernel.cu

* rm typename MT

* add tail process

* add high_precision

* add high_precision to config.py

* rm unused code

* change the comment for the high_precision parameter

* update test_rotary_embdding_unpad.py

* fix vector_copy_utils.h

* add comment for self.high_precision when using float32

2024-03-25 13:40:34 +08:00

..

benchmark_ops

[Inference/kernel]Add Fused Rotary Embedding and KVCache Memcopy CUDA Kernel (#5418)

2024-03-13 17:20:03 +08:00

benchmark_llama.py

[Inference]Support FP16/BF16 Flash Attention 2 And Add high_precision Flag To Rotary Embedding (#5461)

2024-03-25 13:40:34 +08:00

build_smoothquant_weight.py

[inference] refactor examples and fix schedule (#5077)

2023-11-21 10:46:03 +08:00

run_benchmark.sh

[Fix/Inference] Fix format of input prompts and input model in inference engine (#5395)

2024-02-23 10:51:35 +08:00

run_llama_inference.py

[npu] change device to accelerator api (#5239)

2024-01-09 10:20:05 +08:00