mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2025-09-21 09:29:47 +00:00
[Inference]Adapted to the triton attn kernels (#5264)
* adapted to the triton attn kernels * fix pad input * adapted to copy_kv_to_blocked_cache * fix ci test * update kv memcpy * remove print
This commit is contained in:
@@ -236,6 +236,7 @@ class InferenceEngine:
|
||||
output_list = []
|
||||
batch = self.request_handler.schedule()
|
||||
|
||||
# TODO: padding_id is used for generating attn_mask and will be removed if nopad version is supported.
|
||||
logits = self.model(
|
||||
batch,
|
||||
self.k_cahce,
|
||||
|
Reference in New Issue
Block a user