GitLab

cc @patil-suraj

* Allow validation split when loading dataset from local file

* Flax clm & t5, enable validation split for datasets loaded from local file

* fix_torch_device_generate_test

* remove @

* finish

* correct script

* correct script

* fix_torch_device_generate_test

* remove @

* start adding tests

* correct wav2vec2 pretraining

* up

* up
Co-authored-by: Patrick von Platen <patrick@huggingface.co>

* update scripts

* add copyright

* add logging

* cleanup

* add z loss

* add readme

* shard description

* update readme

* Update Transformers README, rename token_classification example to token-classification to be consistent with the others

* Update examples/tensorflow/README.md
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Add README for TF token classification

* Update examples/tensorflow/token-classification/README.md
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update examples/tensorflow/token-classification/README.md
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Adding TF translation example

* Fixes and style pass for TF translation example

* Remove unused postprocess_text copied from run_summarization

* Adding README

* Review fixes

* Move changes to model.config to after we've initialized the model

* Allow ByT5 type in Flax T5 script

* use T5TokenizerFast

* change up tokenizer config

* model_args

* reorder imports

* Update run_t5_mlm_flax.py

* Adding a TF summarization example

* Style pass

* Style fixes

* Updates for review comments

* Adding README

* Style pass

* Remove unused import

* fix_torch_device_generate_test

* remove @

* fix save problem

Successfully logging this requires Pytorch. For the purposes of this script we are not using Pytorch.

* fix_torch_device_generate_test

* remove @

* finish

* Validation split percentage to be used for custom data files also

Issue same as https://github.com/huggingface/transformers/issues/12406 fixed for pytorch branch run_mlm.py

* Validation split added in the right place

* Update run_clm.py

* validation split added for custom files

* Validation split added for custom files

* Update run_plm.py

* fixed validation split for custom files as input for pytorch examples in lm

* Update run_clm_no_trainer.py

* args modified

* add adafactor

* Update examples/flax/language-modeling/run_mlm_flax.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* fix_torch_device_generate_test

* remove @

* up

* up

* correct

* upload
Co-authored-by: Patrick von Platen <patrick@huggingface.co>

* fix saving and loading

* update readme

* fix_torch_device_generate_test

* remove @

* adapt flax pretrain script

* fix_torch_device_generate_test

* remove @

* up

* fix_torch_device_generate_test

* remove @

* push

* fix_torch_device_generate_test

* remove @

* add logging steps

* correct training scripts

* correct readme

* correct