Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
330d83fd
"vscode:/vscode.git/clone" did not exist on "cde0c750af2fae6848ed3ee8be381b4f1230ecd0"
Unverified
Commit
330d83fd
authored
Sep 09, 2021
by
Koichi Yasuoka
Committed by
GitHub
Sep 08, 2021
Browse files
Typo in "end_of_word_suffix" (#13477)
But does it really work?
parent
2a15e8cc
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
1 addition
and
1 deletion
+1
-1
src/transformers/tokenization_utils_fast.py
src/transformers/tokenization_utils_fast.py
+1
-1
No files found.
src/transformers/tokenization_utils_fast.py
View file @
330d83fd
...
...
@@ -666,7 +666,7 @@ class PreTrainedTokenizerFast(PreTrainedTokenizerBase):
kwargs
[
"continuing_subword_prefix"
]
=
tokenizer_json
[
"model"
][
"continuing_subword_prefix"
]
if
(
tokenizer_json
[
"model"
][
"type"
]
==
"BPE"
and
"end_of_wor
k
_suffix"
not
in
kwargs
and
"end_of_wor
d
_suffix"
not
in
kwargs
and
tokenizer_json
[
"model"
][
"end_of_word_suffix"
]
is
not
None
):
kwargs
[
"end_of_word_suffix"
]
=
tokenizer_json
[
"model"
][
"end_of_word_suffix"
]
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment