"doc/git@developer.sourcefind.cn:wangsen/paddle_dbnet.git" did not exist on "44826b515a29d1d9db567d65ef1a5659b5f71d16"
Unverified Commit c18f5916 authored by Stas Bekman's avatar Stas Bekman Committed by GitHub
Browse files

typo (#7001)

apologies for the tiny PRs, just sending those as I find them.
parent 60fc0329
......@@ -1609,7 +1609,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin):
)
tokenizer.add_tokens(token, special_tokens=bool(token in special_tokens))
# Check all our special tokens are registrered as "no split" token (we don't cut them) and are in the vocab
# Check all our special tokens are registered as "no split" token (we don't cut them) and are in the vocab
added_tokens = tokenizer.sanitize_special_tokens()
if added_tokens:
logger.warning(
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment