Skip to content

GitLab

"...git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "972535ea74c7b30987bc31c6621a2bbb58f82ca6"
Unverified Commit 90ec78b5 authored by Jangwon Park's avatar Jangwon Park Committed by GitHub
Browse files

Add missing arguments for BertWordPieceTokenizer (#5810)

parent 77cd0e13
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
src/transformers/tokenization_bert.py
View file @ 90ec78b5
...@@ -632,6 +632,8 @@ class BertTokenizerFast(PreTrainedTokenizerFast): ...@@ -632,6 +632,8 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
unk_token=unk_token, unk_token=unk_token,
sep_token=sep_token, sep_token=sep_token,
cls_token=cls_token, cls_token=cls_token,
pad_token=pad_token,
mask_token=mask_token,
clean_text=clean_text, clean_text=clean_text,
handle_chinese_chars=tokenize_chinese_chars, handle_chinese_chars=tokenize_chinese_chars,
strip_accents=strip_accents, strip_accents=strip_accents,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment