Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
f778edb7
Unverified
Commit
f778edb7
authored
Jan 19, 2022
by
Li-Huai (Allan) Lin
Committed by
GitHub
Jan 19, 2022
Browse files
Fix typo in BERT tokenization file (#15228)
* Fix typo * Fix copies
parent
2a5a3849
Changes
5
Hide whitespace changes
Inline
Side-by-side
Showing
5 changed files
with
7 additions
and
7 deletions
+7
-7
src/transformers/models/bert/tokenization_bert.py
src/transformers/models/bert/tokenization_bert.py
+2
-2
src/transformers/models/bert/tokenization_bert_fast.py
src/transformers/models/bert/tokenization_bert_fast.py
+2
-2
src/transformers/models/layoutlmv2/tokenization_layoutlmv2.py
...transformers/models/layoutlmv2/tokenization_layoutlmv2.py
+1
-1
src/transformers/models/mpnet/tokenization_mpnet.py
src/transformers/models/mpnet/tokenization_mpnet.py
+1
-1
src/transformers/models/tapas/tokenization_tapas.py
src/transformers/models/tapas/tokenization_tapas.py
+1
-1
No files found.
src/transformers/models/bert/tokenization_bert.py
View file @
f778edb7
...
...
@@ -151,7 +151,7 @@ class BertTokenizer(PreTrainedTokenizer):
This should likely be deactivated for Japanese (see this
[issue](https://github.com/huggingface/transformers/issues/328)).
strip_accents
:
(`bool`, *optional*):
strip_accents (`bool`, *optional*):
Whether or not to strip all accents. If this option is not specified, then it will be determined by the
value for `lowercase` (as in the original BERT).
"""
...
...
@@ -382,7 +382,7 @@ class BasicTokenizer(object):
WordPieceTokenizer.
Args:
never_split (`L
I
st[str]`, *optional*)
never_split (`L
i
st[str]`, *optional*)
Kept for backward compatibility purposes. Now implemented directly at the base class level (see
[`PreTrainedTokenizer.tokenize`]) List of token not to split.
"""
...
...
src/transformers/models/bert/tokenization_bert_fast.py
View file @
f778edb7
...
...
@@ -147,10 +147,10 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
tokenize_chinese_chars (`bool`, *optional*, defaults to `True`):
Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see [this
issue](https://github.com/huggingface/transformers/issues/328)).
strip_accents
:
(`bool`, *optional*):
strip_accents (`bool`, *optional*):
Whether or not to strip all accents. If this option is not specified, then it will be determined by the
value for `lowercase` (as in the original BERT).
wordpieces_prefix
:
(`str`, *optional*, defaults to `"##"`):
wordpieces_prefix (`str`, *optional*, defaults to `"##"`):
The prefix for subwords.
"""
...
...
src/transformers/models/layoutlmv2/tokenization_layoutlmv2.py
View file @
f778edb7
...
...
@@ -1317,7 +1317,7 @@ class BasicTokenizer(object):
WordPieceTokenizer.
Args:
never_split (`L
I
st[str]`, *optional*)
never_split (`L
i
st[str]`, *optional*)
Kept for backward compatibility purposes. Now implemented directly at the base class level (see
[`PreTrainedTokenizer.tokenize`]) List of token not to split.
"""
...
...
src/transformers/models/mpnet/tokenization_mpnet.py
View file @
f778edb7
...
...
@@ -359,7 +359,7 @@ class BasicTokenizer(object):
WordPieceTokenizer.
Args:
never_split (`L
I
st[str]`, *optional*)
never_split (`L
i
st[str]`, *optional*)
Kept for backward compatibility purposes. Now implemented directly at the base class level (see
[`PreTrainedTokenizer.tokenize`]) List of token not to split.
"""
...
...
src/transformers/models/tapas/tokenization_tapas.py
View file @
f778edb7
...
...
@@ -2024,7 +2024,7 @@ class BasicTokenizer(object):
WordPieceTokenizer.
Args:
never_split (`L
I
st[str]`, *optional*)
never_split (`L
i
st[str]`, *optional*)
Kept for backward compatibility purposes. Now implemented directly at the base class level (see
[`PreTrainedTokenizer.tokenize`]) List of token not to split.
"""
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment