Skip to content

GitLab

"examples/research_projects/bertology/run_bertology.py" did not exist on "0f40e8d6a63656fbdbb3a31f91c0eb7c1168d2a8"
Switch branch/tag
Find file BlameHistoryPermalink
  • Nicolas Patry's avatar
    Upgrading to tokenizers 0.19.0 (#30289) · 8e5f76f5
    Nicolas Patry authored 
    * [DO NOT MERGE] Testing tokenizers 0.19.0rc0

* Accounting for the breaking change.

* Ruff.

* Upgrading to tokenizers `0.19` (new release with preprend_scheme fixed
and new surface for BPE tiktoken bug).
    8e5f76f5
t5_tokenizer_model.py 3.9 KB