Unverified Commit 3f0a3611 authored by Hailey Schoelkopf's avatar Hailey Schoelkopf Committed by GitHub
Browse files

generalize qwen pad token fix (#1146)

parent 42730d90
......@@ -238,7 +238,7 @@ class HFLM(LM):
elif self.tokenizer.eos_token:
self.tokenizer.pad_token_id = self.tokenizer.eos_token_id
else:
if "Qwen" in pretrained:
if self.config.model_type == "qwen":
# Qwen's trust_remote_code tokenizer does not allow for adding special tokens
self.tokenizer.pad_token = "<|endoftext|>"
else:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment