src/transformers/file_utils.py · 6bab83683bf46352b59ab01cf596804dfdf7d973 · chenpangpang / transformers

Partial local tokenizer load (#9807) · 6cb0a6f0

Lysandre Debut authored Jan 28, 2021



* Allow partial loading of a cached tokenizer

* Warning > Info

* Update src/transformers/tokenization_utils_base.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Raise error if not local_files_only
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

6cb0a6f0

file_utils.py 55 KB

Replace file_utils.py