Unverified Commit a882b9fa authored by Lewis Bails's avatar Lewis Bails Committed by GitHub
Browse files

Add tokenizer_file parameter to PreTrainedTokenizerFast docstring (#12624)


Co-authored-by: default avatarLewis Bails <Lewis.Bails@infomedia.dk>
parent f8f9a679
...@@ -59,6 +59,9 @@ INIT_TOKENIZER_DOCSTRING += """ ...@@ -59,6 +59,9 @@ INIT_TOKENIZER_DOCSTRING += """
tokenizer_object (:class:`tokenizers.Tokenizer`): tokenizer_object (:class:`tokenizers.Tokenizer`):
A :class:`tokenizers.Tokenizer` object from 🤗 tokenizers to instantiate from. See :doc:`Using tokenizers A :class:`tokenizers.Tokenizer` object from 🤗 tokenizers to instantiate from. See :doc:`Using tokenizers
from 🤗 tokenizers <../fast_tokenizers>` for more information. from 🤗 tokenizers <../fast_tokenizers>` for more information.
tokenizer_file (:class:`str`):
A path to a local JSON file representing a previously serialized :class:`tokenizers.Tokenizer` object from
🤗 tokenizers.
""" """
MODEL_TO_TRAINER_MAPPING = { MODEL_TO_TRAINER_MAPPING = {
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment