Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
libai.tokenizer.rst 279 Bytes
Newer Older
yuguo960516's avatar
bloom  
yuguo960516 committed
1
2
3
4
5
6
7
8
9
10
11
12 
libai.tokenizer
##############################

.. currentmodule:: libai.tokenizer
.. automodule:: libai.tokenizer
    :member-order: bysource
    :members:
        BertTokenizer,
        RobertaTokenizer,
        GPT2Tokenizer,
        T5Tokenizer,
        PreTrainedTokenizer,