• Nicolas Patry's avatar
    Adding `skip_special_tokens=True` to FillMaskPipeline (#9783) · 781e4b13
    Nicolas Patry authored
    * We most likely don't want special tokens in this output.
    
    * Adding `skip_special_tokens=True` to FillMaskPipeline
    
    - It's backward incompatible.
    - It makes for sense for pipelines to remove references to
    special_tokens (all of the other pipelines do that).
    - Keeping special tokens makes it hard for users to actually remove them
      because all models have different tokens (<s>, <cls>, [CLS], ....)
    
    * Fixing `token_str` in the same vein, and actually fix the tests too !
    781e4b13
test_pipelines_fill_mask.py 9.77 KB