Skip to content

GitLab

Switch branch/tag
  1. 01 Sep, 2021 1 commit
  3. 31 Aug, 2021 2 commits
    • Matt's avatar
      TF/Numpy variants for all DataCollator classes (#13105) · 854260ca
      Matt authored 
      

* Adding a TF variant of the DataCollatorForTokenClassification to get feedback

* Added a Numpy variant and a post_init check to fail early if a missing import is found

* Fixed call to Numpy variant

* Added a couple more of the collators

* Update src/transformers/data/data_collator.py
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Fixes, style pass, finished DataCollatorForSeqToSeq

* Added all the LanguageModeling DataCollators, except SOP and PermutationLanguageModeling

* Adding DataCollatorForPermutationLanguageModeling

* Style pass

* Add missing `__call__` for PLM

* Remove `post_init` checks for frameworks because the imports inside them were making us fail code quality checks

* Remove unused imports

* First attempt at some TF tests

* A second attempt to make any of those tests actually work

* TF tests, round three

* TF tests, round four

* TF tests, round five

* TF tests, all enabled!

* Style pass

* Merging tests into `test_data_collator.py`

* Merging tests into `test_data_collator.py`

* Fixing up test imports

* Fixing up test imports

* Trying shuffling the conditionals around

* Commenting out non-functional old tests

* Completed all tests for all three frameworks

* Style pass

* Fixed test typo

* Style pass

* Move standard `__call__` method to mixin

* Rearranged imports for `test_data_collator`

* Fix data collator typo "torch" -> "pt"

* Fixed the most embarrassingly obvious bug

* Update src/transformers/data/data_collator.py
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Renaming mixin

* Updating docs
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: default avatarDalton Walker <dalton_walker@icloud.com>
Co-authored-by: default avatarAndrew Romans <andrew.romans@hotmail.com>
      854260ca
    • Serhiy-Shekhovtsov's avatar
      Fixing a typo in the data_collator documentation (#13309) · 11fbc32e
      Serhiy-Shekhovtsov authored
      11fbc32e
  5. 30 Aug, 2021 1 commit
  7. 04 Aug, 2021 1 commit
    • Patrick von Platen's avatar
      [Flax] Correct flax docs (#12782) · fbf468b0
      Patrick von Platen authored 
      * fix_torch_device_generate_test

* remove @

* fix flax docs

* correct more docs in flax

* another correction

* fix flax docs

* Apply suggestions from code review
      fbf468b0
  9. 21 Jul, 2021 1 commit
  11. 17 Jul, 2021 1 commit
  13. 14 Jul, 2021 1 commit
  15. 13 Jul, 2021 1 commit
  17. 29 Jun, 2021 1 commit
  19. 25 Jun, 2021 1 commit
  21. 23 Jun, 2021 1 commit
  23. 22 Jun, 2021 2 commits
  25. 14 Jun, 2021 1 commit
  27. 08 Jun, 2021 2 commits
  29. 04 Jun, 2021 1 commit
  31. 02 Jun, 2021 1 commit
  33. 01 Jun, 2021 2 commits
  35. 26 May, 2021 1 commit
    • Patrick von Platen's avatar
      Flax Generate (#11777) · 996a315e
      Patrick von Platen authored 
      

* fix_torch_device_generate_test

* remove @

* add

* indexing

* correct a couple of tests

* fix tests

* add logits processor

* finish top_k, top_p, temp

* add docs

* correct flax prng key default

* improve generate

* add generation docs

* add docs

* make style

* revert model outputs change

* make style

* correct typo

* fix tests

* fix slow test

* add raise

* finish generation
Co-authored-by: default avatarPatrick von Platen <patrick@huggingface.co>
      996a315e
  37. 13 May, 2021 1 commit
  39. 07 May, 2021 1 commit
  41. 04 May, 2021 1 commit
  43. 30 Apr, 2021 2 commits
    • Stas Bekman's avatar
      [DeepSpeed] fp32 support (#11499) · 4e7bf94e
      Stas Bekman authored 
      * prep for deepspeed==0.3.16

* new version

* too soon

* support and test fp32 mode

* troubleshooting doc start

* workaround no longer needed

* add fp32 doc

* style

* cleanup, add tf32 note

* clarify

* release was made
      4e7bf94e
    • Nicolas Patry's avatar
      Adding `AutomaticSpeechRecognitionPipeline`. (#11337) · db9dd09c
      Nicolas Patry authored 
      

* Adding `AutomaticSpeechRecognitionPipeline`.

- Because we added everything to enable this pipeline, we probably
should add it to `transformers`.
- This PR tries to limit the scope and focuses only on the pipeline part
(what should go in, and out).
- The tests are very specific for S2T and Wav2vec2 to make sure both
architectures are supported by the pipeline. We don't use the mixin for
tests right now, because that requires more work in the `pipeline`
function (will be done in a follow up PR).
- Unsure about the "helper" function `ffmpeg_read`. It makes a lot of
  sense from a user perspective, it does not add any additional
dependencies (as in hard dependency, because users can always use their
own load mechanism). Meanwhile, it feels slightly clunky to have so much
optional preprocessing.
- The pipeline is not done to support streaming audio right now.

Future work:

- Add `automatic-speech-recognition` as a `task`. And add the
FeatureExtractor.from_pretrained within `pipeline` function.
- Add small models within tests
- Add the Mixin to tests.
- Make the logic between ForCTC vs ForConditionalGeneration better.

* Update tests/test_pipelines_automatic_speech_recognition.py
Co-authored-by: default avatarLysandre Debut <lysandre@huggingface.co>

* Adding docs + main import + type checking + LICENSE.

* Doc style !.

* Fixing TYPE_HINT.

* Specifying waveform shape in the docs.

* Adding asserts + specify in the documentation the shape of the input
np.ndarray.

* Update src/transformers/pipelines/automatic_speech_recognition.py
Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>

* Adding require to tests + move the `feature_extractor` doc.
Co-authored-by: default avatarLysandre Debut <lysandre@huggingface.co>
Co-authored-by: default avatarPatrick von Platen <patrick.v.platen@gmail.com>
      db9dd09c
  45. 27 Apr, 2021 1 commit
  47. 26 Apr, 2021 1 commit
  49. 23 Apr, 2021 1 commit
  51. 21 Apr, 2021 1 commit
  53. 13 Apr, 2021 2 commits
  55. 12 Apr, 2021 1 commit
  57. 09 Apr, 2021 1 commit
  59. 08 Apr, 2021 3 commits
  61. 05 Apr, 2021 2 commits