tests/test_tokenization_fast.py · 08d609bfb8fbbaf508ae55c5cf414b262cc04061 · chenpangpang / transformers

"docs/source/vscode:/vscode.git/clone" did not exist on "c20b2c7e18424e35ce7217da1395928244ead78b"

Add tokenizers class mismatch detection between `cls` and checkpoint (#12619) · 08d609bf

Tomohiro Endo authored Jul 17, 2021



* Detect mismatch by analyzing config

* Fix comment

* Fix import

* Update src/transformers/tokenization_utils_base.py
Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>

* Revise based on reviews

* remove kwargs

* Fix exception

* Fix handling exception again

* Disable mismatch test in PreTrainedTokenizerFast
Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>

08d609bf

test_tokenization_fast.py 5.12 KB

Replace test_tokenization_fast.py