github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-07 03:52:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Online Server] Chat Api for streaming and not streaming response (#5470)

Browse Source 
* fix bugs

* fix bugs

* fix api server

* fix api server

* add chat api and test

* del request.n
This commit is contained in:
Jianghai
2024-04-07 14:45:43 +08:00
committed by CjhHa1
parent de378cd2ab
commit c064032865
7 changed files with 326 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
colossalai/inference/struct.py
View File

@@ -165,12 +165,13 @@ class Sequence:
def __repr__(self) -> str:
return (
f"(request_id={self.request_id}, "
f"prompt={self.prompt}, "
f"output_token_id={self.output_token_id},"
f"status={self.status.name}, "
f"sample_params={self.sample_params}, "
f"input_len={self.input_len},"
f"output_len={self.output_len})"
f"prompt={self.prompt},\n"
f"output_token_id={self.output_token_id},\n"
f"output={self.output},\n"
f"status={self.status.name},\n"
f"sample_params={self.sample_params},\n"
f"input_len={self.input_len},\n"
f"output_len={self.output_len})\n"
)