Skip to content

GitLab

Commit 3dc27bd2 authored by jon-tow's avatar jon-tow
Browse files

Remove stale tokenizer comment

parent 5b1beceb
Hide whitespace changes
Inline Side-by-side
Showing with 0 additions and 1 deletion
lm_eval/models/gpt2.py
View file @ 3dc27bd2
......@@ -42,7 +42,6 @@ class HFLM(BaseLM):
).to(self.device)
self.gpt2.eval()
# pretrained tokenizer for neo is broken for now so just hard-coding this to gpt2
self.tokenizer = transformers.AutoTokenizer.from_pretrained(
pretrained if tokenizer is None else tokenizer,
revision=revision,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment