GitLab

fix typo

Error (also in original) model, scaling only q matrix not qk.T dot product (qk.T/sqrt(dim_per_head)) (#21627)

* Error in model, scaling only q matrix not qK.T dot product (qk.T/sqrt(dim_per_head))

As per Vaswani et al, 2017 p.4
Is torch.matmul(q, k.transpose(2, 3)) / math.sqrt(dim_per_head) not q / math.sqrt(dim_per_head)
https://arxiv.org/pdf/1912.05372.pdf

Error was in original FlauBERT repo and effectively scales queries but not values
cf. https://github.com/getalp/Flaubert/pull/45/commits/6d176880ca3a1a8dfa2b76c97030bb51c5e917b8

* Update modeling_flaubert.py

Update to https://github.com/huggingface/transformers/pull/21627
make fixup
make repo_consistency

* Update modeling_xlm.py

* Update modeling_flaubert.py

* Update modeling_xlm.py

* Remove redundant computations, comb variable names

* Fix scores to cur_scores

FInal cleanup of TOKENIZER_FOR_DOC

* v1

* make fixup

* add more methods

* Add in big model inference to issue template

* Trigger

* Untrigger

* empty test commit

* fix env issue

* fix env issue

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

clarify available pipelines

* [deepspeed] performance docs

* fix

* re-org

* update

* update

* a new NCCL Collectives section

* inference

* Update docs/source/en/main_classes/deepspeed.mdx
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* suggestion

* Update docs/source/en/main_classes/deepspeed.mdx

* suggestion

---------
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update setup.py

* suggestions

* Translate quicktour to French

* Traduction missing task

* use fp16

* use fp16

* use fp16

* use fp16

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

Bump ipython in /examples/research_projects/decision_transformer

Bumps [ipython](https://github.com/ipython/ipython) from 8.1.1 to 8.10.0.
- [Release notes](https://github.com/ipython/ipython/releases)
- [Commits](https://github.com/ipython/ipython/compare/8.1.1...8.10.0

)

---
updated-dependencies:
- dependency-name: ipython
  dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* annotated TFvisionEncoderDecoder input type hints
Co-authored-by: JuheonChu <chuj@dickinson.edu>
Co-authored-by: AdiaWu <wua@dickinson.edu>

* fixed failing tests

* make fix-copies

* failed test fix

* style fix

* revert

---------
Co-authored-by: JuheonChu <chuj@dickinson.edu>
Co-authored-by: AdiaWu <wua@dickinson.edu>
Co-authored-by: Matt <rocketknight1@gmail.com>

* fix bnb slow test

* make fixup

[MINOR] Fix link

I'm not sure this will also fix the currently broken link in the docs (Specifically here: https://huggingface.co/docs/transformers/model_doc/time_series_transformer) whereby clicking on `kashif` attempts to link to the following non-existent URL: https://huggingface.co/docs/transformers/model_doc/%3Chttps://huggingface.co/kashif

remove trailing word

skip test

* document question answering guide

* Added the list of supported models

* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* switched to AutoProcessor

* feedback addressed

* Apply suggestions from code review
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update docs/source/en/tasks/document_question_answering.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* more feedback addressed

* addressed comments about evaluation loss

* added appropriate image link

* make style

* typo fix

* resolving toc conflict

* fixed the image link

---------
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

add int8 support

* Remove unused decoder_layerdrop

* Update SPECIAL_CASES_TO_ALLOW for MT5Config

* Remove unused position_embedding_init_scale

* Remove unused decoder_max_relative_position

* Use unused decoder_max_relative_position

* Remove unused init_std

* Remove unused forgotten attributes

* Remove unused patch_norm

* Remove unused max_seq_len

* Update SPECIAL_CASES_TO_ALLOW for OneFormerConfig

---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>

* Added timesformer configuration
Co-authored-by: JuheonChu <chuj@dickinson.edu>

* Create documentation_tests.txt

* Update documentation_tests.txt
Co-authored-by: JuheonChu <chuj@dickinson.edu>

* Delete documentation_tests.txt

Updates, Deleting "src/transformers/utils/documentation_tests.txt" file.
Co-authored-by: JuheonChu <chuj@dickinson.edu>

* Create documentation_tests.txt
Co-authored-by: JuheonChu <chuj@dickinson.edu>

* Delete documentation_tests.txt
Co-authored-by: JuheonChu <chuj@dickinson.edu>

---------
Co-authored-by: JuheonChu <chuj@dickinson.edu>

* Replace input_values_prrocessing with unpack_inputs

* Skip test failing with OOM

* Update tests

* improving tests section

* documenting other  env variables

* [deepspeed] deal with models w/o config.hidden_size

* typo

* typo