src/transformers/tokenization_utils_base.py · a0be960dcca468d9afc4f7f5213ad447e421a21c · chenpangpang / transformers

Fixing tokenizer when `transformers` is installed without `tokenizers` (#26236) · a0be960d

Uri Alon authored Sep 27, 2023

* Fixing tokenizer when tokenizers is not installed

* Adding __repr__ function and repr=True in dataclass

* Revert "Adding __repr__ function and repr=True in dataclass"

This reverts commit 18839505d1cada3170ed623744d3e75008a18bdc.

a0be960d

tokenization_utils_base.py 192 KB

Replace tokenization_utils_base.py