"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "4c721c6b6aba636372b0c9e0ba449e182c6488d5"
Unverified Commit 1e51bb71 authored by Denis's avatar Denis Committed by GitHub
Browse files

Fix for #3865. PretrainedTokenizer mapped " do not" into " don't" when...

Fix for #3865. PretrainedTokenizer mapped " do not" into " don't" when .decode(...) is called. Removed the " do not" --> " don't" mapping from clean_up_tokenization(...). (#4024)
parent 24175910
......@@ -2195,7 +2195,6 @@ class PreTrainedTokenizer(SpecialTokensMixin):
.replace(" ' ", "'")
.replace(" n't", "n't")
.replace(" 'm", "'m")
.replace(" do not", " don't")
.replace(" 's", "'s")
.replace(" 've", "'ve")
.replace(" 're", "'re")
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment