Unverified Commit 5eecf3ff authored by Steven Liu's avatar Steven Liu Committed by GitHub
Browse files

Clarify `use_fast` parameter in docstring (#20840)

* clarify use_fast parameter

* make style

* remove check frameworks, apply review
parent 2875fa97
......@@ -524,7 +524,9 @@ class AutoTokenizer:
In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for
facebook/rag-token-base), specify it here.
use_fast (`bool`, *optional*, defaults to `True`):
Whether or not to try to load the fast version of the tokenizer.
Use a [fast Rust-based tokenizer](https://huggingface.co/docs/tokenizers/index) if it is supported for
a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer
is returned instead.
tokenizer_type (`str`, *optional*):
Tokenizer type to be loaded.
trust_remote_code (`bool`, *optional*, defaults to `False`):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment