tokenizer.rst 184 Bytes
Newer Older
thomwolf's avatar
thomwolf committed
1
2
3
4
5
6
7
8
Tokenizer
----------------------------------------------------

``PreTrainedTokenizer``
~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: pytorch_transformers.PreTrainedTokenizer
    :members: