1. 13 Feb, 2021 1 commit
    • Nicolas Patry's avatar
      Conversion from slow to fast for BPE spm vocabs contained an error. (#10120) · c9837a0d
      Nicolas Patry authored
      * Conversion from slow to fast for BPE spm vocabs contained an error.
      
      - There is only 1 test currently (tokenizers + slow) that used the modified path
      and it's reformer, which does not contain any ids modification so the
      bug was silent for now.
      - The real issue is that vocab variable was overloaded by
      SentencePieceExtractor, leading to Slow specific vocab oddities to be
      completely ignored
      - The bug was reported here https://github.com/huggingface/transformers/issues/9518
      - Ran the complete tokenization test suite with slow without error
      (`RUN_SLOW=1 pytest -sv tests/test_tokenization_*`)
      
      * Remove rebase error.
      
      * Adding the fixture.
      c9837a0d
  2. 09 Oct, 2020 1 commit
  3. 06 Jan, 2020 2 commits
  4. 22 Dec, 2019 1 commit
  5. 26 Sep, 2019 1 commit
  6. 05 Jul, 2019 1 commit
  7. 02 Jul, 2019 1 commit
  8. 21 Jun, 2019 1 commit