• Philip May's avatar
    Add strip_accents to basic BertTokenizer. (#6280) · d5bc32ce
    Philip May authored
    * Add strip_accents to basic tokenizer
    
    * Add tests for strip_accents.
    
    * fix style with black
    
    * Fix strip_accents test
    
    * empty commit to trigger CI
    
    * Improved strip_accents check
    
    * Add code quality with is not False
    d5bc32ce
test_tokenization_bert.py 8.43 KB