8 Commits

Author	SHA1	Message	Date
flybird11111	d0e13b85fd	[upgrade]Upgrade mixtral (#6317) ... * upgrade mixtral * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * upgrade infer * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * upgrade drafter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix * upgrade lazy * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * upgrade mixtral --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2025-05-21 16:14:05 +08:00
Yuanheng Zhao	7b249c76e5	[Fix] Fix spec-dec Glide LlamaModel for compatibility with transformers (#5837) ... * fix glide llama model * revise	2024-06-19 15:37:53 +08:00
Yuanheng Zhao	8bcfe360fd	[example] Update Inference Example (#5725) ... * [example] update inference example	2024-05-17 11:28:53 +08:00
Yuanheng Zhao	55cc7f3df7	[Fix] Fix Inference Example, Tests, and Requirements (#5688) ... * clean requirements * modify example inference struct * add test ci scripts * mark test_infer as submodule * rm deprecated cls & deps * import of HAS_FLASH_ATTN * prune inference tests to be run * prune triton kernel tests * increment pytest timeout mins * revert import path in openmoe	2024-05-08 11:30:15 +08:00
Yuanheng Zhao	e1acb58423	[doc] Add inference/speculative-decoding README (#5552) ... * add README for spec-dec * update roadmap	2024-04-10 11:07:52 +08:00
Yuanheng Zhao	d85d91435a	[Inference/SpecDec] Support GLIDE Drafter Model (#5455) ... * add glide-llama policy and modeling * update glide modeling, compitable with transformers 4.36.2 * revise glide llama modeling/usage * fix issues of glimpsing large kv * revise the way re-loading params for glide drafter * fix drafter and engine tests * enable convert to glide strict=False * revise glide llama modeling * revise vicuna prompt template * revise drafter and tests * apply usage of glide model in engine	2024-04-10 11:07:52 +08:00
Yuanheng Zhao	a37f82629d	[Inference/SpecDec] Add Speculative Decoding Implementation (#5423) ... * fix flash decoding mask during verification * add spec-dec * add test for spec-dec * revise drafter init * remove drafter sampling * retire past kv in drafter * (trivial) rename attrs * (trivial) rename arg * revise how we enable/disable spec-dec	2024-04-10 11:07:52 +08:00
Yuanheng Zhao	5a9b05f7b2	[Inference/SpecDec] Add Basic Drafter Model Container (#5405) ... * [Infer/Fix] Fix Dependency in test - RMSNorm kernel (#5399) fix dependency in pytest * add drafter model container (basic ver)	2024-04-10 11:07:51 +08:00