"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "3b720ad9a5a3a854e2f11d43a61a056b5afd3b61"
Fix saving FlaubertTokenizer configs (#14991)
All specific tokenizer config properties must be passed to its base class (XLMTokenizer) in order to be saved. This was not the case for do_lowercase config. Thus it was not saved by save_pretrained() method and saving and reloading the tokenizer changed its behaviour. This commit fixes it.
Showing
Please register or sign in to comment