Jianghai 0e616462a7 [Inference] add logit processor and request handler (#5166) ...

* add logit processor and request handler

* add

* add

* add

* fix

* add search tokens and update func

* finish request handler

* add running list test

* fix test

* fix some bug

* add

* add

* fix bugs

* fix some bugs

* fix bug

* fix

* fix

* add copy fun

* del useless attn

* fix request status

---------

Co-authored-by: CjhHa1 <cjh18671720497outlook.com>

2024-01-11 13:39:56 +00:00

..

__init__.py

[Inference] Add CacheBlock and KV-Cache Manager (#5156)

2024-01-11 13:39:29 +00:00

block_cache.py

[Inference] Add CacheBlock and KV-Cache Manager (#5156)

2024-01-11 13:39:29 +00:00

kvcache_manager.py

[Inference] add logit processor and request handler (#5166)

2024-01-11 13:39:56 +00:00