GitLab

* initial commit, refactoring the text generation api reference

* removed repetitive code examples

* Refactoring the text generation docs to reduce repetition

* make style

* Part of the "text generation" rework: adding a high-level overview of the text generation strategies

* code samples update via make style

* fixed a few formatting issues

* Apply suggestions from review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* fixed spaces, and switched two links to markdown

* Apply Steven's suggestions from review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* new lines after headers to fix link rendering

* review feedback addressed. added links to image captioning and audio transcription examples

* minor capitalization fix

* addressed the review feedback

* Apply suggestions from review
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

* Applied review suggestions
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

* Added TF example for image classification

* Code style polishing

* code style polishing

* minor polishing

* fixed a link in a tip, and a typo in the inference TF content

* Apply Amy's suggestions from review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update docs/source/en/tasks/image_classification.mdx
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* review feedback addressed

* make style

* added PushToHubCallback with save_strategy="no"

* minor polishing

* added PushToHubCallback with save_strategy=no

* minor polishing

* Update docs/source/en/tasks/image_classification.mdx

* added data augmentation
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* make style
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Add resources

* Add more resources

* Remove pipeline tag

* Add more resources

* Add more resources
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>

fix: input -> input_string.

* Add resources

* Add more resources

* Add more resources

* Add TAPAS

* Fix pipeline tag

* Fix pipeline tags

* Remove pipeline tag

* Remove depth-estimation tag

* Update docs/source/en/model_doc/segformer.mdx
Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Apply suggestion

* Fix segformer
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* add draft logit processor

* add template functions

* update timesapmt processor parameters

* draft script

* simplify code

* cleanup

* fixup and clean

* update pipeline

* style

* clean up previous idea

* add tokenization utils

* update tokenizer and asr output

* fit whisper type

* style and update test

* clean test

* style test

* update tests

* update error test

* udpate code (not based on review yet)

* update tokenization

* update asr pipeline

* update code

* cleanup and update test

* fmt

* remove text verificatino

* cleanup

* cleanup

* add model test

* update tests

* update code add docstring

* update code and add docstring

* fix pipeline tests

* add draft logit processor

add template functions

update timesapmt processor parameters

draft script

simplify code

cleanup

fixup and clean

update pipeline

style

clean up previous idea

add tokenization utils

update tokenizer and asr output

fit whisper type

style and update test

clean test

style test

update tests

update error test

udpate code (not based on review yet)

update tokenization

update asr pipeline

update code

cleanup and update test

fmt

remove text verificatino

cleanup

cleanup

add model test

update tests

update code add docstring

update code and add docstring

fix pipeline tests

* Small update.

* Fixup.

* Tmp.

* More support.

* Making `forced_decoder_ids` non mandatory for users to set.

* update and fix first bug

* properly process sequence right after merge if last

* tofo

* allow list inputs + compute begin index better

* start adding tests

* add the 3 edge cases

* style

* format sequences

* fixup

* update

* update

* style

* test passes, edge cases should be good

* update last value

* remove Trie

* update tests and expec ted values

* handle bigger chunk_length

* clean tests a bit

* refactor chunk iter and clean pipeline

* update tests

* style

* refactor chunk iter and clean pipeline

* upade

* resolve comments

* Apply suggestions from code review
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* take stride right into account

* update test expected values

* Update code based on review
Co-authored-by: sgugger <sylvain.gugger@gmail.com>
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
Co-authored-by: sgugger <sylvain.gugger@gmail.com>

* Fixing offline mode for pipeline (when inferring task).

* Update src/transformers/pipelines/__init__.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Updating test to reflect change in exception.

* Fixing offline mode.

* Clean.
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Clarify and add missing typical_p docstring.

* Make the docstring easier to understand.

* Clarify typical_p docstring

Accept the suggestion by @stevhliu for paraphrasing the docstring.
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Use the same docstring as in GenerationConfig

Follow the suggestion suggested by @stevhliu in the pull request conversation.

* Fix docstring spacing.
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* feat: add standalone guide on XLA support.
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

* Empty commit to trigger CI

* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* address PR comments.
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

The max of top_k and min_tokens_to_keep performed on every call can just be done once up-front.

* Rename files

* Update file names in tests

* serializable hubert

* Add num_workers for prepare_tf_dataset

* Bugfix in the default collator and change default tensor type

* Remove the "num_workers" arg and move it to a new PR

* Adds Mask2Former to transformers
Co-authored-by: Shivalika Singh <shivalikasingh95@gmail.com>
Co-authored-by: Shivalika Singh <73357305+shivalikasingh95@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Fix training

* Add test

* Fix failing tests
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>

Update TFTapasEmbeddings
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Added clefourrier as ref point for graph models in bug reports

* Update PULL_REQUEST_TEMPLATE.md

fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Co-authored-by: susnato <susnato@tensorflow123456@gmail.com>

* Fixing #20783

* Update src/transformers/pipelines/base.py

* Fixing some tests.

* Fixup.

* Remove ffmpeg dep + a bit more relaxed for bigbird QA precision.

* Better dataset.

* Prevent failing on TF.

* Better condition. We can't use `can_use_iterator` since we cannot use it
directly.

Add `min_new_tokens` argument in generate() (implementation based on `MinNewTokensLengthLogitsProcessor`) (#21044)

add a new parameter min_new_tokens for generate()

- Remove duplicate encoder_attention_mask default value assignment

Fix docstring
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>

* First draft

* More improvements

* Add convnext backbone

* Add conversion script

* Add more improvements

* Comment out to_dict

* Add to_dict method

* Add default config

* Fix config

* Fix backbone

* Fix backbone some more

* Add docs, auto mapping, tests

* Fix some tests

* Fix more tests

* Fix more tests

* Add conversion script

* Improve conversion script

* Add support for getting reshaped undownsampled hidden states

* Fix forward pass

* Add print statements

* Comment out set_shift_and_window_size

* More improvements

* Correct downsampling layers conversion

* Fix style

* First draft

* Fix conversion script

* Remove config attribute

* Fix more tests

* Update READMEs

* Update ConvNextBackbone

* Fix ConvNext tests

* Align ConvNext with Swin

* Remove files

* Fix index

* Improve docs

* Add output_attentions to model forward

* Add backbone mixin, improve tests

* More improvements

* Update init_weights

* Fix interpolation of logits

* Add UperNetImageProcessor

* Improve image processor

* Fix image processor

* Remove print statements

* Remove script

* Update import

* Add image processor tests

* Remove print statements

* Fix test

* Add integration test

* Add convnext integration test

* Update docstring

* Fix README

* Simplify config

* Apply suggestions

* Improve docs

* Rename class

* Fix test_initialization

* Fix import

* Address review

* Fix confg

* Convert all checkpoints

* Fix default backbone

* Usage same processor as segformer

* Apply suggestions

* Fix init_weights, update conversion scripts

* Improve config

* Use Auto API instead of creating a new image processor

* Fix docs

* Add doctests

* Remove ResNetConfig dependency

* Add always_partition argument

* Fix rebaseé

* Improve docs

* Convert checkpoints
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MBP.localdomain>

Fixed typo

Suppress deprecation warning

* Remove random output

* remove values

* fix copy statements

* Rework automatic code samples in docstrings

* ImageProcessor->AutoImageProcessor

* Add models to fix copies

* Last typos

* A couple more models

* Fix copies

* Add community to toctree
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Copy English content
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Add some translations
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Add some translations
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Add some translations
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Fix position of community
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Fix translation
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Add translation
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Add translation
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Add translation
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* Add translation
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>
Signed-off-by: Shogo Hida <shogo.hida@gmail.com>

* first draft of new task summary

* make style

* review

* apply feedback

* apply feedbacks

* final touches

* typo

* change name in `__repr__`

* fix my mistake

fix torchscript tests for AltCLIP
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Fix for Past CI

* make style

* clean up

* unindent 2 blocks
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* [bnb optim] fixing test

* force 1 gpu

* fix

* fix

* fix

* finalize

* improve commentary

* fix

* cleanup

* more fixes

* Remove gradient_checkpointing from MarkupLMConfig

* Remove predict_special_tokens from OpenAIGPTConfig

* Remove enable_cls from RoCBertConfig

* Remove batch_size from TrajectoryTransformerConfig

* Remove searcher_seq_len from RealmConfig

* Remove feat_quantizer_dropout from WavLMConfig

* Remove position_biased_input from SEWDConfig

* Remove max_source_positions from Speech2Text2Config
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Fixed issue #21039 and added test for low_cpu_mem_usage

* Optimize inference only mode memory if ipex is used
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

* fix code style
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>