"examples/vscode:/vscode.git/clone" did not exist on "26858f27cb352b6bb1cda2b090413d1d2206a9ee"
  • Yiqing-Zhou's avatar
    read().splitlines() -> readlines() · 897d0841
    Yiqing-Zhou authored
    splitlines() does not work as what we expect here for bert-base-chinese because there is a '\u2028' (unicode line seperator) token in vocab file. Value of '\u2028'.splitlines() is ['', ''].
    Perhaps we should use readlines() instead.
    897d0841
tokenization_bert.py 18.9 KB