mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2025-08-19 00:17:18 +00:00
3de2e62299
* [Inference] Add KVCache Manager * function refactored * add test for KVCache Manager * add attr beam width * Revise alloc func in CacheManager * Fix docs and pytests * add tp slicing for head number * optimize shapes of tensors used as physical cache * Apply using InferenceConfig on KVCacheManager * rm duplicate config file * Optimize cache allocation: use contiguous cache * Fix config in pytest (and config)
5 lines
124 B
Python
5 lines
124 B
Python
from .block_cache import CacheBlock
|
|
from .kvcache_manager import KVCacheManager
|
|
|
|
__all__ = ["CacheBlock", "KVCacheManager"]
|