Yuanheng Zhao 5f98a9d68a [Infer] Optimize Blocked KVCache And Kernels Using It (#5325) ...

* revise shape of kvcache (context attn kernel)

* revise shape of kvcache (flash decoding kernel)

* revise shape of kvcache (kvcache copy) and attn func

* init of kvcache in kvcache manager

* revise llama modeling

* revise block size retrieval

* use torch for rms_norm benchmarking

* revise block size retrieval

2024-01-30 16:06:09 +08:00

..

__init__.py

[Inference] Add CacheBlock and KV-Cache Manager (#5156)

2024-01-11 13:39:29 +00:00

block_cache.py

[Inference] Add CacheBlock and KV-Cache Manager (#5156)

2024-01-11 13:39:29 +00:00

kvcache_manager.py

[Infer] Optimize Blocked KVCache And Kernels Using It (#5325)

2024-01-30 16:06:09 +08:00