"official/modeling/privacy/configs_test.py" did not exist on "4bb36073fd6e52a54f86bcc0c9a0e4a2efb46cb1"
  • Nicolas Patry's avatar
    Conversion from slow to fast for BPE spm vocabs contained an error. (#10120) · c9837a0d
    Nicolas Patry authored
    * Conversion from slow to fast for BPE spm vocabs contained an error.
    
    - There is only 1 test currently (tokenizers + slow) that used the modified path
    and it's reformer, which does not contain any ids modification so the
    bug was silent for now.
    - The real issue is that vocab variable was overloaded by
    SentencePieceExtractor, leading to Slow specific vocab oddities to be
    completely ignored
    - The bug was reported here https://github.com/huggingface/transformers/issues/9518
    - Ran the complete tokenization test suite with slow without error
    (`RUN_SLOW=1 pytest -sv tests/test_tokenization_*`)
    
    * Remove rebase error.
    
    * Adding the fixture.
    c9837a0d
test_tokenization_camembert.py 3.33 KB