GitLab

* Make training args fully immutable

* Working tests, PyTorch

* In test_trainer

* during testing

* Use proper dataclass way

* Fix test

* Another one

* Fix tf

* Lingering slow

* Exception

* Clean

* Update run_translation.py

Fixed link

* Update run_translation.py

* add missing tokens

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Load else

* New approach

* Propagate

chorse: adding new doc on train and val

* pytorch examples

* pytorch mim no trainer

* cookiecutter

* flax examples

* missed line in pytorch run_glue

* tensorflow examples

* tensorflow run_clip

* tensorflow run_mlm

* tensorflow run_ner

* tensorflow run_clm

* pytorch example from_configs

* pytorch no trainer examples

* Revert "tensorflow run_clip"

This reverts commit 261f86ac1f1c9e05dd3fd0291e1a1f8e573781d5.

* fix: duplicated argument

* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* pytorch examples

* tensorflow examples

* flax examples

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* make run_generation more generic for other devices

* use Accelerate to support any device type it supports.

* make style

* fix error usage of accelerator.prepare_model

* use `PartialState` to make sure everything is running on the right device

---------
Co-authored-by: statelesshz <jihuazhong1@huawei.com>

* Fix .push_to_hub and cleanup get_full_repo_name usage

* Do not rely on Python bool conversion magic

* request changes

replace `per_gpu_eval_batch_size` with `per_device_eval_batch_size` in readme of multiple-choice task (#25078)

replace `per_gpu_eval_batch_size` with `per_device_eval_batch_size`
in readme of multiple-choice

Change logic

* replace no_cuda with use_cpu in test_pytorch_examples

* remove codes that never be used

* fix style

* Add text classification example

* set the problem type and finetuning task

* ruff reformated

* fix bug for unseting label_to_id for regression

* update README.md

* fixed finetuning task

* update comment

* check if label exists in feature before removing

* add useful logging

* check if eval dataset is dict

* formatting

* Fix TypeError: Object of type int64 is not JSON serializable

* Convert numpy.float64 and numpy.int64 to float and int for json serialization

* Black reformatted examples/pytorch/token-classification/run_ner_no_trainer.py

* * make style

* Add mms ctc fine tuning

* make style

* More fixes that are needed

* make fix-copies

* make draft for README

* add new file

* move to new file

* make style

* make style

* add quick test

* make style

* make style

Fix step bugs in no trainer + load checkpoint + grad acc

Oops

Act on deprecation

* Re-enable squad test

* [all-test]

* [all-test] Fix all test command

* Fix the all-test

* Revamp test selection for the example tests

* Rename old XLA test and fake modif in run_glue

* Fixes

* Fake Trainer modif

* Remove fake modifs

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

* fix bug in group_texts function, that was inserting short batches

* fully exclude short batches and return empty dict instead

* fix style

* Fix translation

skip
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* convert numpy array to list before writing to json

per_category_iou and per_category_accuracy  are ndarray in the eval_metrics

* code reformatted with make style

trainer parameters changed to save tokenizer in addition to feature_extractor

Call str on id to make sure any ints are converted into the expected format for squad datasets

* Add run_mim_no_trainer.py draft from #20412

Add parse_args method and copy over other dependencies

Add Method call for sending telemetry

Initialize Accelerator

Make one log on every process

Set seed and Handle repository creation

Initialize dataset and Set validation split

Create Config

Adapt Config

Update Config

Create Feature Extractor

Create model

Set column names

Create transforms

Create mask generator

Create method to preprocess images

Shuffle datasets if needed and set transforms

Create Dataloaders

Add optimizer

Add learning rate scheduler

Prepare everything with our accelerator

Tie weights for TPU training

Recalculate training steps and training epochs

Set accelerator checkpointing steps

Initialize trackers and store configuration

Set total batch size

Fix typo: mlm -> mim

Log info at the start of training

Load in the weights and states from previous save

update the progress_bar if load from checkpoint

Define train loop

Add evaluation loop to training

Add to parse_args method

Push repo to hub

Save accelerator state

End training and save model and feature extractor

Remove unused imports

Fix trailing whitespace

* Update code based on comments, Rename feature_extractor to image_processor

* Fix linting

* Add argument for learning rate

* Add argument for setting number of training epochs

* Remove incorrect logger argument

* Convert max_train_steps to int for tqdm

---------
Co-authored-by: Saad Mahmud <shuvro.mahmud79@gmail.com>

Migration to Evaluate for metric is not quite complete

Save the tokenizer and image preprocessor after training a model with the contrastive image-text example (#23035)

Save tokenizer and image preprocessor

* Replace -100s in predictions by the pad token

* Style

* Try to catch them all