• r-terada's avatar
    Add sudachi and jumanpp tokenizers for bert_japanese (#19043) · 2f53ab57
    r-terada authored
    * add sudachipy and jumanpp tokenizers for bert_japanese
    
    * use ImportError instead of ModuleNotFoundError in SudachiTokenizer and JumanppTokenizer
    
    * put test cases of test_tokenization_bert_japanese in one line
    
    * add require_sudachi and require_jumanpp decorator for testing
    
    * add sudachi and pyknp(jumanpp) to dependencies
    
    * remove sudachi_dict_small and sudachi_dict_full from dependencies
    
    * empty commit for ci
    2f53ab57
config.yml 35.4 KB