tests/test_tokenization_xlm.py · 8b63a01d9511e68fca16b60d496a617a270d6c43 · chenpangpang / transformers · GitLab

Find file Blame History Permalink

XLM tokenizer should encode with bos token (#3791) · 8b63a01d
Lysandre Debut authored Apr 17, 2020
```
* XLM tokenizer should encode with bos token

* Update tests
```
8b63a01d

test_tokenization_xlm.py 3.25 KB