github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-08 04:24:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Inference] ADD async and sync Api server using FastAPI (#5396)

Browse Source 
* add api server

* fix

* add

* add completion service and fix bug

* add generation config

* revise shardformer

* fix bugs

* add docstrings and fix some bugs

* fix bugs and add choices for prompt template
This commit is contained in:
Jianghai
2024-03-01 14:47:36 +08:00
committed by CjhHa1
parent d482922035
commit 69cd7e069d
13 changed files with 789 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
colossalai/inference/struct.py
View File

@@ -164,6 +164,7 @@ class Sequence:
return (
f"(request_id={self.request_id}, "
f"prompt={self.prompt}, "
f"output_token_id={self.output_token_id},"
f"status={self.status.name}, "
f"sample_params={self.sample_params}, "
f"input_len={self.input_len},"