Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
ce111fee
Unverified
Commit
ce111fee
authored
Jul 08, 2021
by
Stas Bekman
Committed by
GitHub
Jul 08, 2021
Browse files
[doc] fix broken ref (#12597)
parent
f0dde601
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
3 additions
and
3 deletions
+3
-3
src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py
...mers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py
+1
-1
src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py
...ansformers/models/xlm_roberta/tokenization_xlm_roberta.py
+1
-1
src/transformers/models/xlm_roberta/tokenization_xlm_roberta_fast.py
...rmers/models/xlm_roberta/tokenization_xlm_roberta_fast.py
+1
-1
No files found.
src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py
View file @
ce111fee
...
@@ -56,7 +56,7 @@ def load_vocab(vocab_file):
...
@@ -56,7 +56,7 @@ def load_vocab(vocab_file):
class
XLMProphetNetTokenizer
(
PreTrainedTokenizer
):
class
XLMProphetNetTokenizer
(
PreTrainedTokenizer
):
"""
"""
Adapted from :class:`~transformers.RobertaTokenizer` and class:`~transformers.XLNetTokenizer`. Based on
Adapted from :class:`~transformers.RobertaTokenizer` and
:
class:`~transformers.XLNetTokenizer`. Based on
`SentencePiece <https://github.com/google/sentencepiece>`__.
`SentencePiece <https://github.com/google/sentencepiece>`__.
This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the main methods.
This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the main methods.
...
...
src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py
View file @
ce111fee
...
@@ -54,7 +54,7 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
...
@@ -54,7 +54,7 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
class
XLMRobertaTokenizer
(
PreTrainedTokenizer
):
class
XLMRobertaTokenizer
(
PreTrainedTokenizer
):
"""
"""
Adapted from :class:`~transformers.RobertaTokenizer` and class:`~transformers.XLNetTokenizer`. Based on
Adapted from :class:`~transformers.RobertaTokenizer` and
:
class:`~transformers.XLNetTokenizer`. Based on
`SentencePiece <https://github.com/google/sentencepiece>`__.
`SentencePiece <https://github.com/google/sentencepiece>`__.
This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the main methods.
This tokenizer inherits from :class:`~transformers.PreTrainedTokenizer` which contains most of the main methods.
...
...
src/transformers/models/xlm_roberta/tokenization_xlm_roberta_fast.py
View file @
ce111fee
...
@@ -67,7 +67,7 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
...
@@ -67,7 +67,7 @@ PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
class
XLMRobertaTokenizerFast
(
PreTrainedTokenizerFast
):
class
XLMRobertaTokenizerFast
(
PreTrainedTokenizerFast
):
"""
"""
Construct a "fast" XLM-RoBERTa tokenizer (backed by HuggingFace's `tokenizers` library). Adapted from
Construct a "fast" XLM-RoBERTa tokenizer (backed by HuggingFace's `tokenizers` library). Adapted from
:class:`~transformers.RobertaTokenizer` and class:`~transformers.XLNetTokenizer`. Based on `BPE
:class:`~transformers.RobertaTokenizer` and
:
class:`~transformers.XLNetTokenizer`. Based on `BPE
<https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models>`__.
<https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models>`__.
This tokenizer inherits from :class:`~transformers.PreTrainedTokenizerFast` which contains most of the main
This tokenizer inherits from :class:`~transformers.PreTrainedTokenizerFast` which contains most of the main
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment