src/transformers/models/auto/tokenization_auto.py · e46ad22cd6cb28f78f4d9b6314e7581d8fd97dc5 · chenpangpang / transformers

"vscode:/vscode.git/clone" did not exist on "5d81a5683394dbcb151ba4842bd11f84c53edd6d"

Improve type checker performance (#13094) · e46ad22c

Bill Schnurr authored Aug 12, 2021



* conditional declare `TOKENIZER_MAPPING_NAMES` within a `if TYPE_CHECKING` block so that type checkers dont need to evaluate the RHS of the assignment.

this improves performance of the pylance/pyright type checkers

* Update src/transformers/models/auto/tokenization_auto.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* adding missing import

* format
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

e46ad22c

tokenization_auto.py 22.8 KB

Replace tokenization_auto.py