3 Commits

Author	SHA1	Message	Date
yuehuayingxueluo	9c2fe7935f	[Inference]Adapt temperature processing logic (#5689) ... * Adapt temperature processing logic * add ValueError for top_p and top_k * add GQA Test * fix except_msg	2024-05-08 17:58:29 +08:00
傅剑寒	e6496dd371	[Inference] Optimize request handler of llama (#5512) ... * optimize request_handler * fix ways of writing	2024-03-26 16:37:14 +08:00
Jianghai	0e616462a7	[Inference] add logit processor and request handler (#5166) ... * add logit processor and request handler * add * add * add * fix * add search tokens and update func * finish request handler * add running list test * fix test * fix some bug * add * add * fix bugs * fix some bugs * fix bug * fix * fix * add copy fun * del useless attn * fix request status --------- Co-authored-by: CjhHa1 <cjh18671720497outlook.com>	2024-01-11 13:39:56 +00:00