github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-12 20:54:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Pipeline inference] Combine kvcache with pipeline inference (#4938)

Browse Source 
* merge kvcache with pipeline inference and refactor the code structure

* support ppsize > 2

* refactor pipeline code

* do pre-commit

* modify benchmark

* fix bench mark

* polish code

* add docstring and update readme

* refactor the code

* fix some logic bug of ppinfer

* polish readme

* fix typo

* skip infer test
This commit is contained in:
Bin Jia
2023-10-27 16:19:54 +08:00
committed by GitHub
parent c6cd629e7a
commit 1db6727678
19 changed files with 922 additions and 745 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
colossalai/inference/__init__.py
View File

@@ -1,3 +1,4 @@
from .pipeline import PPInferEngine
__all__ = ["PPInferEngine"]
__all__ = ['PPInferEngine']