Steve Luo 5cd75ce4c7 [Inference/Kernel] refactor kvcache manager and rotary_embedding and kvcache_memcpy oper… (#5663) ...

* refactor kvcache manager and rotary_embedding and kvcache_memcpy operator

* refactor decode_kv_cache_memcpy

* enable alibi in pagedattention

2024-04-30 15:52:23 +08:00

..

__init__.py

[Inference] Add CacheBlock and KV-Cache Manager (#5156)

2024-01-11 13:39:29 +00:00

block_cache.py

[doc] updated inference readme (#5343)

2024-02-02 14:31:10 +08:00

kvcache_manager.py

[Inference/Kernel] refactor kvcache manager and rotary_embedding and kvcache_memcpy oper… (#5663)

2024-04-30 15:52:23 +08:00