github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-10 05:20:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

[DOC] Update inference readme (#5280)

Browse Source 
* add readme

* add readme

* 1

* update engine

* finish readme

* add readme
This commit is contained in:
Jianghai
2024-01-29 16:21:06 +08:00
committed by GitHub
parent 1f8a75d470
commit c7c104cb7c
2 changed files with 79 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
colossalai/inference/core/engine.py
View File

@@ -242,6 +242,7 @@ class InferenceEngine:
finished_sequences = self.request_handler.update()
# Decode completed sentences.
# TODO : update decoding step
for seq in finished_sequences:
output_str = self.tokenizer.decode(seq.input_token_id + seq.output_token_id, skip_special_tokens=True)
output_list.append(output_str)