"...git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "23a71449c0e63595815b7e0a10adf5673c9db824"
Fix saving FlaubertTokenizer configs (#14991)
All specific tokenizer config properties must be passed to its base class (XLMTokenizer) in order to be saved. This was not the case for do_lowercase config. Thus it was not saved by save_pretrained() method and saving and reloading the tokenizer changed its behaviour. This commit fixes it.
Showing
Please register or sign in to comment