GitLab

* Remove old TF port guide

* repo-consistency

* Remove some translations as well for consistency

* Remove some translations as well for consistency

* Alias

* Note alias

* Tests and src

* Rest

* Clean

* Change typing?

* Fix tests

* Deprecation versions

* Configuring Translation Pipelines documents update #27753

Configuring Translation Pipelines documents update

* Language Format Addition

* adding supported list of languages list

* add `push_to_hub` to pipeline

* fix docs

* format with ruff

* update save_pretrained

* update save_pretrained

* remove unnecessary comment

* switch to push_to_hub method in DynamicPipelineTester

* remove unused imports

* update docs for add_new_pipeline

* fix docs for add_new_pipeline

* add comment

* fix italien docs

* changes to token retrieval for pipelines

* Update src/transformers/pipelines/base.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* fix: update `ko/_toctree.yml`

* fix: update ko/_toctree.yml

* Update docs/source/ko/_toctree.yml
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fix: delete `perf_infer_gpu_many`

* fix: Replace untranslated docs with `in_translation`
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fix: Replace untraslated docs with `in_translation`

---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Undo

* Use tokenizer

* Undo data collator

* ImportError: Trainer with PyTorch requires accelerate>=0.20.1 Fix

Adding the evaluate and accelerate installs at the beginning of the cell to fix the issue

* ImportError Fix: Trainer with PyTorch requires accelerate>=0.20.1

* Import Error Fix

* Update installation.md

* Update quicktour.md

* rollback other lang changes

* Update _config.py

* updates for other languages

* fixing error

* Tutorial Update

* Update tokenization_utils_base.py

* Just use an optimizer string to pass the doctest?

---------
Co-authored-by: Matt <rocketknight1@gmail.com>

* Add image processor to trainer

* Replace tokenizer=image_processor everywhere

* add USE_TORCH_XLA env

* rename torch_tpu to torch_xla

* better is_torch_xla_available; fix some fsdp and performance issues

* fix format

* fix bug when pjrt_device is cpu

* fix bug

* fix the deprecation handling

---------
Co-authored-by: anw90 <ang868@gmail.com>
Co-authored-by: wangang.wa <wangang.wa@alibaba-inc.com>

* Update ko _toctree.yml

* Create ko: generation_strategies.md

* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Apply suggestions from code review
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* Apply suggestions from code review
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* Script & Manual edition

* Update

* Translate contributing.md to German

* Fix formatting issues in contributing.md

* Address review comments

* Fix capitalization

Add language identifiers to code blocks

* Fix model documentation links in attention.md

* Fix external link syntax

* Fix target anchor names of section links

* Fix copyright statement comments

* Fix documentation headings

* Add missing entries to the language selector

* Add links to the Colab and AWS Studio notebooks for ONNX

* Use anchor links in CONTRIBUTING.md

* Fix broken hyperlinks due to spaces

* Fix links to OpenAI research articles

* Remove confusing footnote symbols from author names, as they are also considered invalid markup

Fix backticks in code blocks and documentation links

* Fix typos and grammar mistakes in docs and examples

* Fix typos in docstrings and comments

* Fix spelling of `tokenizer` in model tests

* Remove erroneous spaces in decorators

* Remove extra spaces in Markdown link texts

* [docs] Fix broken link
Signed-off-by: Hankyeol Kyung <kghnkl0103@gmail.com>

* [docs] Use shorter domain
Signed-off-by: Hankyeol Kyung <kghnkl0103@gmail.com>

---------
Signed-off-by: Hankyeol Kyung <kghnkl0103@gmail.com>

Switch to the conda-forge channel for transformer installation,
as the huggingface channel does not offer the latest version.

Fixes #28248

Update formats

* docs: replace torch.distributed.run by torchrun

 `transformers` now officially support pytorch >= 1.10.
 The entrypoint `torchrun`` is present from 1.10 onwards.
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

* Update src/transformers/trainer.py

with @ArthurZucker's suggestion
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

Update Korean tutorial for using LLMs, and refactor the nested conditional statements in hr_argparser.py (#27489)

docs: Update Korean LLM tutorial to use Mistral-7B, not Llama-v1

* Enable large-v3 downloading and update language list

* Fix type annotation

* make fixup

* Export Whisper feature extractor

* Fix error after extractor loading

* Do not use pre-computed mel filters

* Save the full preprocessor properly

* Update docs

* Remove comment
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Add alignment heads consistent with each Whisper version

* Remove alignment heads calculation

* Save fast tokenizer format as well

* Fix slow to fast conversion

* Fix bos/eos/pad token IDs in the model config

* Add decoder_start_token_id to config

---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

fixed the broken links belogs to dataset library of transformers

* try to stylify using ruff

* might need to remove these changes?

* use ruf format andruff check

* use isinstance instead of type comparision

* use # fmt: skip

* use # fmt: skip

* nits

* soem styling changes

* update ci job

* nits isinstance

* more files update

* nits

* more nits

* small nits

* check and format

* revert wrong changes

* actually use formatter instead of checker

* nits

* well docbuilder is overwriting this commit

* revert notebook changes

* try to nuke docbuilder

* style

* fix feature exrtaction test

* remve `indent-width = 4`

* fixup

* more nits

* update the ruff version that we use

* style

* nuke docbuilder styling

* leve the print for detected changes

* nits

* Remove file I/O
Co-authored-by: charliermarsh <charlie.r.marsh@gmail.com>

* style

* nits

* revert notebook changes

* Add # fmt skip when possible

* Add # fmt skip when possible

* Fix

* More `  # fmt: skip` usage

* More `  # fmt: skip` usage

* More `  # fmt: skip` usage

* NIts

* more fixes

* fix tapas

* Another way to skip

* Recommended way

* Fix two more fiels

* Remove asynch
Remove asynch

---------
Co-authored-by: charliermarsh <charlie.r.marsh@gmail.com>

* don't use `use_auth_token`internally

* let's use token everywhere

* fixup

* fixed links with 404

* make style

* docs: ko: big_models.md

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions
Co-Authored-By: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-Authored-By: heuristicwave <31366038+heuristicwave@users.noreply.github.com>
Co-Authored-By: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-Authored-By: heuristicwave <31366038+heuristicwave@users.noreply.github.com>
Co-Authored-By: bolizabeth <68984363+bolizabeth@users.noreply.github.com>

---------
Co-authored-by: bolizabeth <68984363+bolizabeth@users.noreply.github.com>
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-authored-by: heuristicwave <31366038+heuristicwave@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* docs: ko: sementic_segmentation.md

* feat: manual draft

* fix: manual edits

* fix: resolve suggestions
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>

* fix: resolve suggestions
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fix: edit the title

---------
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* docs: ko: toknenizer_summary.md
Co-Authored-By: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-Authored-By: Juntae <79131091+sronger@users.noreply.github.com>
Co-Authored-By: Injin Paek <71638597+eenzeenee@users.noreply.github.com>

* update review

* fix: resolve suggestions
Co-Authored-By: Nayeon Han <nayeon2.han@gmail.com>
Co-Authored-By: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fix: resolve suggestions
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>

---------
Co-authored-by: HanNayeoniee <nayeon2.han@gmail.com>
Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Juntae <79131091+sronger@users.noreply.github.com>
Co-authored-by: Injin Paek <71638597+eenzeenee@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>

* dos: ko: perf_train_gpu_many.mdx

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions

Change description
Follow the glossary
Fix discrepancies
Co-Authored-By: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-Authored-By: 이서정 <97655267+sjlee-wise@users.noreply.github.com>
Co-Authored-By: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------
Co-authored-by: Hyunho <105839613+hyunhp@users.noreply.github.com>
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* docs:ko:Debugging.md

* feat: chatgpt draft

* fix: resolve suggestions
Co-Authored-By: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-Authored-By: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------
Co-authored-by: Jang KyuJin <106062329+kj021@users.noreply.github.com>
Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* 🌐

 [i18n-KO] Translated  to Korean

* update translation

* fix some sentence editing and fixing punctuation

* Update docs/source/ko/_toctree.yml
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>

* Apply suggestions from code review
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>

---------
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>

* docs: ko-whisper.md

* fix: chatgpt draft

* feat: manual edits

* Feat: manual edits

* fix: resolve suggestions
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

---------
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* update

* update

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* docs: ko-contributing.md

* feat: chatGPT draft

* feat: manual edits

* feat: change linked document

* fix: resolve suggestion
Co-authored-by: Haewon Kim <ehdvkf02@naver.com>

* fix: resolve suggestion
Co-authored-by: Haewon Kim <ehdvkf02@naver.com>

* fix: resolve suggestion
Co-authored-by: Haewon Kim <ehdvkf02@naver.com>

* fix: resolve suggestion
Co-authored-by: Haewon Kim <ehdvkf02@naver.com>

* fix: resolve suggestion
Co-authored-by: Haewon Kim <ehdvkf02@naver.com>

* fix: resolve suggestion
Co-authored-by: Haewon Kim <ehdvkf02@naver.com>

* fix: resolve suggestion
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: resolve suggestion
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: resolve suggestion
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: resolve suggestion

* fix: resolve suggestion

* feat: delete file to resolve error

---------
Co-authored-by: Haewon Kim <ehdvkf02@naver.com>
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* docs: ko-llama2.md

* feat: chatGPT draft and manul edits

* feat: added inline TOC

* fix: inline TOC

* fix: resolve suggestions
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: resolve suggestion
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: resolve suggestion
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

---------
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* docs: ko-llama.md

* fix: chatgpt draft

* feat: manual edits

* fix: resolve suggestions

* docs: ko: llm_tutoroal.md

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions

* fix: resolve suggestions