tests/test_tokenization_fast.py · e695470794f236392f249aeb815b62490126f595 · chenpangpang / transformers

Avoid using get_list_of_files (#15287) · e6954707

Sylvain Gugger authored Jan 25, 2022

* Avoid using get_list_of_files in config

* Wip, change tokenizer file getter

* Remove call in tokenizer files

* Remove last call to get_list_model_files

* Better tests

* Unit tests for new function

* Document bad API

e6954707

test_tokenization_fast.py 7.89 KB

Replace test_tokenization_fast.py