tests/test_tokenization_bert_japanese.py · 48c6c6139fbb2881ef16ac5d8afb6287467bf66e · chenpangpang / transformers

Masatoshi Suzuki authored Aug 17, 2020

* Update BERT Japanese tokenizers

* Update CircleCI config to download unidic

* Specify to use the latest dictionary packages

48c6c613

test_tokenization_bert_japanese.py 9.84 KB