Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
2886f7f0
Unverified
Commit
2886f7f0
authored
Sep 13, 2022
by
Yih-Dar
Committed by
GitHub
Sep 13, 2022
Browse files
Fix tokenizer for XLMRobertaXL (#19004)
Co-authored-by:
ydshieh
<
ydshieh@users.noreply.github.com
>
parent
2848c9ce
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
8 additions
and
2 deletions
+8
-2
src/transformers/models/auto/tokenization_auto.py
src/transformers/models/auto/tokenization_auto.py
+7
-1
src/transformers/models/xlm_roberta_xl/modeling_xlm_roberta_xl.py
...sformers/models/xlm_roberta_xl/modeling_xlm_roberta_xl.py
+1
-1
No files found.
src/transformers/models/auto/tokenization_auto.py
View file @
2886f7f0
...
...
@@ -271,7 +271,13 @@ else:
"XLMRobertaTokenizerFast"
if
is_tokenizers_available
()
else
None
,
),
),
(
"xlm-roberta-xl"
,
(
"RobertaTokenizer"
,
"RobertaTokenizerFast"
if
is_tokenizers_available
()
else
None
)),
(
"xlm-roberta-xl"
,
(
"XLMRobertaTokenizer"
if
is_sentencepiece_available
()
else
None
,
"XLMRobertaTokenizerFast"
if
is_tokenizers_available
()
else
None
,
),
),
(
"xlnet"
,
(
...
...
src/transformers/models/xlm_roberta_xl/modeling_xlm_roberta_xl.py
View file @
2886f7f0
...
...
@@ -54,7 +54,7 @@ logger = logging.get_logger(__name__)
_CHECKPOINT_FOR_DOC
=
"xlm-roberta-xlarge"
_CONFIG_FOR_DOC
=
"XLMRobertaXLConfig"
_TOKENIZER_FOR_DOC
=
"RobertaTokenizer"
_TOKENIZER_FOR_DOC
=
"
XLM
RobertaTokenizer"
XLM_ROBERTA_XL_PRETRAINED_MODEL_ARCHIVE_LIST
=
[
"facebook/xlm-roberta-xl"
,
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment