examples/language-modeling/run_clm.py · acc851e1ff92835d2a3ee9774d9d0abfda6e3f36 · chenpangpang / transformers

[run_clm] clarify why we get the tokenizer warning on long input (#11145) · acc851e1

Stas Bekman authored Apr 08, 2021



* clarify why we get the warning here

* Update examples/language-modeling/run_clm.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* wording

* style
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

acc851e1

run_clm.py 18.8 KB

Replace run_clm.py