Unverified Commit bef0c629 authored by Yiqing-Zhou's avatar Yiqing-Zhou Committed by GitHub
Browse files

fix

Remove '\n' before adding token into vocab
parent 897d0841
......@@ -69,6 +69,7 @@ def load_vocab(vocab_file):
with open(vocab_file, "r", encoding="utf-8") as reader:
tokens = reader.readlines()
for index, token in enumerate(tokens):
token = token[:-1]
vocab[token] = index
return vocab
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment