github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-01 17:17:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Fix] Fix spec-dec Glide LlamaModel for compatibility with transformers (#5837)

Browse Source 
* fix glide llama model

* revise
This commit is contained in:
Yuanheng Zhao
2024-06-19 15:37:53 +08:00
committed by GitHub
parent fd1dc417d8
commit 7b249c76e5
4 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
examples/inference/llama/README.md
View File

@@ -43,5 +43,8 @@ colossalai run --nproc_per_node 2 llama_generation.py -m PATH_MODEL --drafter_mo
If you want to try the GLIDE model (glide-vicuna7b) as the drafter model with vicuna-7B, you could provide the GLIDE model path or model card as drafter model and enable the feature by
```python
from colossalai.inference.modeling.models.glide_llama import GlideLlamaForCausalLM
drafter_model = GlideLlamaForCausalLM.from_pretrained(drafter_model_path_or_name)
...
engine.enable_spec_dec(drafter_model, use_glide_drafter=True)
```