GitLab

* Update no_trainer script for summarization

* removed unnecessary import

* fixes notation mistake

* removed: unused variable

* fixed bug which caused overwrite_cache to always be True (#18967).

* reformatting changes

* fixed incorrect param to hasattr

* simplified condition checks

* code cleanup

* Added accelerate gradient accumulation wrapper to run_image_classification_no_trainer.py example script

* make fixup changes

* PR comments

* changed input to Acceletor based on PR comment, ran make fixup

* Added comment explaining the sync_gradients statement

* Fixed lr scheduler max steps

* Changed run_clm_no_trainer.py script to use accelerate gradient accum wrapper

* Fixed all scripts except wav2vec2 pretraining to use accelerate gradient accum wrapper

* Added accelerate gradient accum wrapper for wav2vec2_pretraining_no_trainer.py script

* make fixup and lr_scheduler step inserted back into run_qa_beam_search_no_trainer.py

* removed changes to run_wav2vec2_pretraining_no_trainer.py script and fixed using wrong constant in qa_beam_search_no_trainer.py script

* zero chance anyone's using that constant no?

* `transformers-cli login` => `huggingface-cli login`

* `transformers-cli repo create` => `huggingface-cli repo create`

* `make style`

* Fix ROUGE add example check and update README

* Stay consistent in values

* Migrate metric to Evaluate in pytorch examples

* Remove unused imports

* Preliminary work on tokenizers

* Quality + fix tests

* Treat processors

* Fix pad

* Remove all uses of  in tests, docs and examples

* Replace all as_target_tokenizer

* Fix tests

* Fix quality

* Update examples/flax/image-captioning/run_image_captioning_flax.py
Co-authored-by: amyeroberts <amy@huggingface.co>

* Style
Co-authored-by: amyeroberts <amy@huggingface.co>

* Add summarization name mapping for MultiNews

* Add summarization name mapping for MultiNews

* Add ALL_LAYERNORM_LAYERS for LayerNorm

* fix bug of appending layer norm

Properly calculate the total train iterations and recalculate num epochs in no_trainer scripts (#17856)

* Add examples telemetry

* Alternative approach

* Add to all other examples

* Add to templates as well

* Put framework separately

* Same for TensorFlow

* improve no-trainer examples

* Trigger CI

* adding comment to clarify tracker init on main process

* Trigger CI

* Trigger CI

* Trigger CI

Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts (#17331)

* Fix length in no_trainer examples

* Add setup and teardown

* Use new accelerator config generator to automatically make tests able to run based on environment

* Black preview

* Fixup too!

* Fix check copies

* Use the same version as the CI

* Bump black

* Update all examples to properly calculate progress bar

* Propagate and fix imports

* Fixup all examples

* Fix multiple deletions of the same files in save_pretrained

* Add is_main_process argument

* Change tracking to store_true

* Remove step param and use it in the log dictionary directly

* use vars(args) when passing args to init_trackers

* Include tracking tests since tensorboard is already a dep

Adds checkpoint prefixes to the gitignore if `push_to_hub` is used along with `checkpointint_steps`

* Fixed some bugs involving saving during epochs
* Added tests mimicking the existing examples tests
* Added in json exporting to all `no_trainer` examples for consistency 

Adds logging and save/loading to the Accelerate scripts
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* fix formatting and remove use_auth

* Add use_auth_token to Flax examples

* [examples] max samples can't be bigger than then len of dataset

* do tf and flax

* Split file_utils in several submodules

* Fixes

* Add back more objects

* More fixes

* Who exactly decided to import that from there?

* Second suggestion to code with code review

* Revert wront move

* Fix imports

* Adapt all imports

* Adapt all imports everywhere

* Revert this import, will fix in a separate commit