• bastrob's avatar
    Add missing Flaubert tokenizer tests (#30492) · 464d986b
    bastrob authored
    * add flaubert tokenization test, enrich inheritance in FlaubertTokenizer.
    
    * fix quality code ci
    
    * ensure parameter consistency
    
    * fix ci
    
    * fix copyright year and flatten vocab list.
    
    * fix style
    464d986b
test_tokenization_flaubert.py 3.27 KB