GitLab

* first draft

* benchmarks

* feedback

~transformer. -> ~transformers.

* fixes

* more fixes

* style fix

* more fix

* comments

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

if use_cpu: dataloader_pin_memory = False

* Add madlad-400 models

* Add madlad-400 to the doc table

* Update docs/source/en/model_doc/madlad-400.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Fill missing details in documentation

* Update docs/source/en/model_doc/madlad-400.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Do not doctest madlad-400

Tests are timing out.

---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* log

* log

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

update
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* First draft

* Add backwards compatibility

* More improvements

* More improvements

* Improve error message

* Address comment

* Add conversion script

* Fix style

* Update code snippet

* Adddress comment

* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* fix

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Fix passing scheduler-specific kwargs through TrainingArguments `lr_scheduler_kwargs`

* Added test for lr_scheduler_kwargs

* Add `model_docs`

* Add

* Update Model adoc

* Update docs/source/ja/model_doc/bark.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/model_doc/beit.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/model_doc/bit.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/model_doc/blenderbot.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/model_doc/blenderbot-small.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* update reiew-1

* Update toctree.yml

* translating docs and fixes of PR #27401

* Update docs/source/ja/model_doc/bert.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/model_doc/bert-generation.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update the model docs

---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* translate work

* update

* update

* update [[autodoc]]

* Update callback.md

---------
Co-authored-by: jiaqiw <wangjiaqi50@huawei.com>

* Update default ChatML template

* Update docs/warnings

* Update docs/source/en/chat_templating.md
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Slight rework

---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* docs: replace torch.distributed.run by torchrun

 `transformers` now officially support pytorch >= 1.10.
 The entrypoint `torchrun`` is present from 1.10 onwards.
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

* Update src/transformers/trainer.py

with @ArthurZucker's suggestion
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Fix code snippet

* Improve code snippet

* fix group_sub_entities bug

* add space

* Update forward signature

* Empty-Commit

* fix assisted decoding attention_cat

* fix attention_mask for assisted decoding

* fix attention_mask len

* fix attn len

* Use a more clean way to prepare assistant models inputs

* fix param meaning

* fix param name

* fix assistant model inputs

* update token type ids

* fix assistant kwargs copy

* add encoder-decoder tests of assisted decoding

* check if assistant kwargs contains updated keys

* revert test

* fix whisper tests

* fix assistant kwargs

* revert whisper test

* delete _extend funcs

* Fix mistral generate for long prompt / response

* Add unit test

* fix linter

* fix linter

* fix test

* add assisted generation test for mistral and load the model in 4 bit + fa2

Reorder

fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Fix sliding_window hasattr in Mistral

* hasattr -> getattr for sliding_window in Mistral

---------
Co-authored-by: Ilya Gusev <ilya.gusev@booking.com>

* fix

* fix

* fix

* fix

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Successfully resolved the ZeroDivisionError exception in the utils.notebook.y file.

* Now I update little code mentioned by Peter

* Using Black package to reformat my file

* Now I using ruff libary to reformated my file

* reflect RoCm support in the documentation

* Update docs/source/en/main_classes/trainer.md
Co-authored-by: Lysandre Debut <hi@lysand.re>

* fix review comments

* use ROCm instead of RoCm

---------
Co-authored-by: Lysandre Debut <hi@lysand.re>

[`DocString`] Support a revision in the docstring `add_code_sample_docstrings` to facilitate integrations (#27645)

* initial commit

* dummy changes

* style

* Update src/transformers/utils/doc.py
Co-authored-by: Alex McKinney <44398246+vvvm23@users.noreply.github.com>

* nits

* nit use ` if re.match(r'^refs/pr/\d*', revision):`

* restrict

* nit

* test the doc vuilder

* wow

* oke the order was wrong

---------
Co-authored-by: Alex McKinney <44398246+vvvm23@users.noreply.github.com>

Change "convert predictions to logits" to "convert logits to
predictions" to fix semantic error in the evaluation section. Logits
need to be converted to predictions to evaluate the accuracy, not the
other way round

* Update musicgen.md

please make it less hidden

* Add cleaner formatting

* Fix typo in warning message

The path of `default_cache_path` is hf_cache_home/hub. There is no
directory named transformers under hf_cache_home

* Fix a typo in comment

* Update the version number

v4.22.0 is the earlist version that contains those changes in PR #18492

* fix

* fix

* trigger

* Apply suggestions from code review
Co-authored-by: Lysandre Debut <hi@lysand.re>

* tic

* revert

* revert

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>

fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* add code changes

1. Refactor FSDP
2. Add `--save_only_model` option: When checkpointing, whether to only save the model, or also the optimizer, scheduler & rng state.
3. Bump up the minimum `accelerate` version to `0.21.0`

* quality

* fix quality?

* Revert "fix quality?"

This reverts commit 149330a6abc078827be274db84c8a2d26a76eba1.

* fix fsdp doc strings

* fix quality

* Update src/transformers/training_args.py
Co-authored-by: Zach Mueller <muellerzr@gmail.com>

* please fix the quality issue 😅



* Apply suggestions from code review
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

* address comment

* simplify conditional check as per the comment

* update documentation

---------
Co-authored-by: Zach Mueller <muellerzr@gmail.com>
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

* update

* fix

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Add resources

* Remove script

* Update docs/source/en/model_doc/dinov2.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

Update arxiv link

* Extended semantic segmentation

* Update image_segmentation.md

* Changed title

* Update docs/source/en/tasks/semantic_segmentation.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update docs/source/en/tasks/semantic_segmentation.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update docs/source/en/tasks/semantic_segmentation.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update docs/source/en/tasks/semantic_segmentation.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update docs/source/en/tasks/semantic_segmentation.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update semantic_segmentation.md

* Update docs/source/en/tasks/semantic_segmentation.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update docs/source/en/tasks/semantic_segmentation.md
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Addressed Niels' and Maria's comments

* Added detail on panoptic segmentation

* Added redirection and renamed the file

* Update _toctree.yml

* Update _redirects.yml

* Rename image_segmentation.md to semantic_segmentation.md

---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* added flash attention for opt

* added to list

* fix use cache (#3)

* style fix

* fix text

* test fix2

* reverted until 689f599

* torch fx tests are working now!

* small fix

* added TODO docstring

* changes

* comments and .md file modification

---------
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>