pre-commit-ci[bot]
|
d78817539e
|
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
|
2024-04-08 08:41:09 +00:00 |
|
yuehuayingxueluo
|
04aca9e55b
|
[Inference/Kernel]Add get_cos_and_sin Kernel (#5528)
* Add get_cos_and_sin kernel
* fix code comments
* fix code typos
* merge common codes of get_cos_and_sin kernel.
* Fixed a typo
* Changed 'asset allclose' to 'assert equal'.
|
2024-04-01 13:47:14 +08:00 |
|
yuehuayingxueluo
|
87079cffe8
|
[Inference]Support FP16/BF16 Flash Attention 2 And Add high_precision Flag To Rotary Embedding (#5461)
* Support FP16/BF16 Flash Attention 2
* fix bugs in test_kv_cache_memcpy.py
* add context_kv_cache_memcpy_kernel.cu
* rm typename MT
* add tail process
* add high_precision
* add high_precision to config.py
* rm unused code
* change the comment for the high_precision parameter
* update test_rotary_embdding_unpad.py
* fix vector_copy_utils.h
* add comment for self.high_precision when using float32
|
2024-03-25 13:40:34 +08:00 |
|
yuehuayingxueluo
|
f366a5ea1f
|
[Inference/kernel]Add Fused Rotary Embedding and KVCache Memcopy CUDA Kernel (#5418)
* add rotary embedding kernel
* add rotary_embedding_kernel
* add fused rotary_emb and kvcache memcopy
* add fused_rotary_emb_and_cache_kernel.cu
* add fused_rotary_emb_and_memcopy
* fix bugs in fused_rotary_emb_and_cache_kernel.cu
* fix ci bugs
* use vec memcopy and opt the gloabl memory access
* fix code style
* fix test_rotary_embdding_unpad.py
* codes revised based on the review comments
* fix bugs about include path
* rm inline
|
2024-03-13 17:20:03 +08:00 |
|
xs_courtesy
|
c1c45e9d8e
|
fix include path
|
2024-03-13 11:21:06 +08:00 |
|
xs_courtesy
|
095c070a6e
|
refactor code
|
2024-03-11 17:06:57 +08:00 |
|