Skip to content

GitLab

Unverified Commit 147f7746 authored by Arthur's avatar Arthur Committed by GitHub
Browse files

[`PretrainedTokenizer`] add some of the most important functions to the doc (#27313)

parent 1ffc4dee
Hide whitespace changes
Inline Side-by-side
Showing with 4 additions and 0 deletions
docs/source/en/main_classes/tokenizer.md
View file @ 147f7746
...@@ -55,6 +55,8 @@ to a given token). ...@@ -55,6 +55,8 @@ to a given token).
[[autodoc]] PreTrainedTokenizer [[autodoc]] PreTrainedTokenizer
- __call__ - __call__
- add_tokens
- add_special_tokens
- apply_chat_template - apply_chat_template
- batch_decode - batch_decode
- decode - decode
...@@ -69,6 +71,8 @@ loaded very simply into 🤗 transformers. Take a look at the [Using tokenizers ...@@ -69,6 +71,8 @@ loaded very simply into 🤗 transformers. Take a look at the [Using tokenizers
[[autodoc]] PreTrainedTokenizerFast [[autodoc]] PreTrainedTokenizerFast
- __call__ - __call__
- add_tokens
- add_special_tokens
- apply_chat_template - apply_chat_template
- batch_decode - batch_decode
- decode - decode
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment