Add missing arguments for BertWordPieceTokenizer (#5810)

90ec78b5 · Jangwon Park · GitHub · 77cd0e13 · 90ec78b5
Unverified Commit 90ec78b5 authored Sep 07, 2020 by Jangwon Park Committed by GitHub Sep 07, 2020
Show whitespace changes
Inline Side-by-side

Showing with 2 additions and 0 deletions

src/transformers/tokenization_bert.py src/transformers/tokenization_bert.py +2 -0

No files found.
--- a/src/transformers/tokenization_bert.py
+++ b/src/transformers/tokenization_bert.py
@@ -632,6 +632,8 @@ class BertTokenizerFast(PreTrainedTokenizerFast):
                unk_token=unk_token,
                sep_token=sep_token,
                cls_token=cls_token,
+                pad_token=pad_token,
+                mask_token=mask_token,
                clean_text=clean_text,
                handle_chinese_chars=tokenize_chinese_chars,
                strip_accents=strip_accents,