• SaulLu's avatar
    fix set truncation attribute in `__init__` of `PreTrainedTokenizerBase` (#15456) · 39b5d1a6
    SaulLu authored
    
    
    * change truncation_side in init of `PreTrainedTokenizerBase`
    Co-authored-by: default avatarLSinev <LSinev@users.noreply.github.com>
    
    * add test
    
    * Revert "replace assert with exception for `padding_side` arg in `PreTrainedTokenizerBase` `__init__`"
    
    This reverts commit 7a98b87962d2635c7e4d4f00db3948b694624843.
    
    * fix kwargs
    
    * Revert "fix kwargs"
    
    This reverts commit 67b0a5270e8cf1dbf70e6b0232e94c0452b6946f.
    
    * Update tests/test_tokenization_common.py
    Co-authored-by: default avatarNicolas Patry <patry.nicolas@protonmail.com>
    
    * delete truncation_side variable
    
    * reorganize test
    
    * format
    
    * complete doc
    
    * Revert "Revert "replace assert with exception for `padding_side` arg in `PreTrainedTokenizerBase` `__init__`""
    
    This reverts commit d5a10a7e2680539e5d9e98ae5d896c893d224b80.
    
    * fix typo
    
    * fix typos to render documentation
    
    * Revert "Revert "Revert "replace assert with exception for `padding_side` arg in `PreTrainedTokenizerBase` `__init__`"""
    
    This reverts commit 16cf58811943a08f43409a7c83eaa330686591d0.
    
    * format
    Co-authored-by: default avatarLSinev <LSinev@users.noreply.github.com>
    Co-authored-by: default avatarNicolas Patry <patry.nicolas@protonmail.com>
    39b5d1a6
test_tokenization_common.py 196 KB