Skip to content

GitLab

"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "78f4a0e7e553c8472e2e13bdc1bea370a1ea4102"
Unverified Commit 823dbf8a authored by Gunjan Chhablani's avatar Gunjan Chhablani Committed by GitHub
Browse files

Remove MBart subclass of XLMRoberta in tokenzier docs (#16546) 

* Remove MBart subclass of XLMRoberta in tokenzier

* Fix style

* Copy docs from MBart50 tokenizer
parent 5fe06b9b
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 3 deletions
src/transformers/models/mbart/tokenization_mbart_fast.py
View file @ 823dbf8a
...@@ -62,9 +62,8 @@ class MBartTokenizerFast(PreTrainedTokenizerFast): ...@@ -62,9 +62,8 @@ class MBartTokenizerFast(PreTrainedTokenizerFast):
Construct a "fast" MBART tokenizer (backed by HuggingFace's *tokenizers* library). Based on Construct a "fast" MBART tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models). [BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models).
[`MBartTokenizerFast`] is a subclass of [`XLMRobertaTokenizerFast`]. Refer to superclass This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
[`XLMRobertaTokenizerFast`] for usage examples and documentation concerning the initialization parameters and other refer to this superclass for more information regarding those methods.
methods.
The tokenization method is `<tokens> <eos> <language code>` for source language documents, and ``<language code> The tokenization method is `<tokens> <eos> <language code>` for source language documents, and ``<language code>
<tokens> <eos>``` for target language documents. <tokens> <eos>``` for target language documents.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment