github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-09 04:50:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Inference] Fix API server, test and example (#5712)

Browse Source 
* fix api server

* fix generation config

* fix api server

* fix comments

* fix infer hanging bug

* resolve comments, change backend to free port
This commit is contained in:
Jianghai
2024-05-15 15:47:31 +08:00
committed by GitHub
parent 74c47921fa
commit f47f2fbb24
5 changed files with 73 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
colossalai/inference/server/completion_service.py
View File

@@ -23,7 +23,7 @@ class CompletionServing:
# it is not a intuitive way
self.engine.engine.generation_config = generation_config
result_generator = self.engine.generate(request_id, prompt=prompt)
result_generator = self.engine.generate(request_id, prompt=prompt, generation_config=generation_config)
if await request.is_disconnected():
# Abort the request if the client disconnects.