github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-03 18:19:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix ColossalEval (#4992)

Browse Source 
Co-authored-by: Xu Yuanchen <yuanchen.xu00@gmail.com>
This commit is contained in:
Yuanchen
2023-10-31 10:30:03 +08:00
committed by GitHub
parent 459a88c806
commit abe071b663
3 changed files with 20 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
applications/ColossalEval/colossal_eval/models/huggingface.py
View File

@@ -96,7 +96,7 @@ class HuggingFaceModel(BaseModel):
self.logger.warning("pad_token_id is not set for the tokenizer. " "Using eos_token_id as pad_token_id.")
if self.tokenizer.eos_token:
self.tokenizer.pad_token = self.tokenizer.eos_token
elif self.tokenizer.eod_id:
elif hasattr(self.tokenizer, "eod_id"):
# Qwen has an eod token "<|endoftext|>".
self.tokenizer.pad_token_id = self.tokenizer.eod_id