- 13 Feb, 2021 1 commit
-
-
Nicolas Patry authored
* Conversion from slow to fast for BPE spm vocabs contained an error. - There is only 1 test currently (tokenizers + slow) that used the modified path and it's reformer, which does not contain any ids modification so the bug was silent for now. - The real issue is that vocab variable was overloaded by SentencePieceExtractor, leading to Slow specific vocab oddities to be completely ignored - The bug was reported here https://github.com/huggingface/transformers/issues/9518 - Ran the complete tokenization test suite with slow without error (`RUN_SLOW=1 pytest -sv tests/test_tokenization_*`) * Remove rebase error. * Adding the fixture.
-
- 09 Oct, 2020 1 commit
-
-
Stas Bekman authored
-
- 06 Jan, 2020 2 commits
-
-
alberduris authored
-
alberduris authored
-
- 22 Dec, 2019 1 commit
-
-
Aymeric Augustin authored
-
- 26 Sep, 2019 1 commit
-
-
thomwolf authored
-
- 05 Jul, 2019 1 commit
-
-
thomwolf authored
-
- 02 Jul, 2019 1 commit
-
-
thomwolf authored
-
- 21 Jun, 2019 1 commit
-
-
thomwolf authored
-