"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "7ea7eba39d2fc90ffde00bbb867521215da71f33"
Fix saving FlaubertTokenizer configs (#14991)
All specific tokenizer config properties must be passed to its base class (XLMTokenizer) in order to be saved. This was not the case for do_lowercase config. Thus it was not saved by save_pretrained() method and saving and reloading the tokenizer changed its behaviour. This commit fixes it.
Showing
Please register or sign in to comment