Skip to content

GitLab

Switch branch/tag
  1. 12 Oct, 2022 2 commits
    • Zhaoheng Ni's avatar
      Improve wav2vec2/hubert model for pre-training (#2716) · c5bd93b6
      Zhaoheng Ni authored 
      Summary:
This PR improves the Wav2Vec2/HuBERT model regarding model pre-training.

- The model initialization of positional embedding and transformer module is essential to model pre-training. The accuracy of unmasked frames should be higher than masked frames, as it is an easier task. but without the initialization, the accuracy of masked frames is higher than unmasked frames.
  Compared the performance after two epochs with 16 GPUs.
  - With model initialization, the accuracies of masked/unmasked frames are 0.08/0.11.
  - Without model initialization, the accuracies of masked/unmasked frames are 0.06/0.04.
- After adding the model initialization, the gradient is easy to overflow (aka `nan` gradient). In paper [Self-Supervised Learning for speech recognition with Intermediate layer supervision](https://arxiv.org/abs/2112.08778) the authors propose a simple but effective method to mitigate the overflow issue, by scaling down the multiplication of query and key and subtracting the maximum value from it (subtracting a constant value won't change the output of softmax). Then it guarantees the value won't be overflowed.
- In the original fairseq, the mask indices are generated by `numpy.random.choice`. Here replace `torch.multinomial` with `torch.randperm`. (cc carolineechen).

Other improvements within training scripts will be included in a separate PR.

Pull Request resolved: https://github.com/pytorch/audio/pull/2716

Reviewed By: xiaohui-zhang

Differential Revision: D39832189

Pulled By: nateanl

fbshipit-source-id: f4d2a473a79ad63add2dd16624bd155d5ce4de27
      c5bd93b6
    • Caroline Chen's avatar
      Skip hubert xlarge torchscript test (#2758) · c2ea6898
      Caroline Chen authored 
      Summary:
a couple of circleci unittests are failing during hubert xlarge torchscript test, which has been known to fail on Windows in the past (#65776). this PR disables this test on circleci

cc atalman

Pull Request resolved: https://github.com/pytorch/audio/pull/2758

Reviewed By: mthrok

Differential Revision: D40290535

Pulled By: carolineechen

fbshipit-source-id: 5c5fb43434a517b6c439a8cb8e853015d1550a57
      c2ea6898
  3. 11 Oct, 2022 4 commits
  5. 10 Oct, 2022 2 commits
    • Zhaoheng Ni's avatar
      Add unit test for LibriMix dataset (#2659) · c5b8e585
      Zhaoheng Ni authored 
      Summary:
Besides the unit test, the PR also addresses these issues:
- The original `LibriMix` dataset only supports "min" mode, which means the audio length is the minimum of all clean sources. It is default for source separation task. Users may also want to use "max" mode which allows for end-to-end separation and recognition. The PR adds ``mode`` argument to let users decide which dataset they want to use.
- If the task is ``"enh_both"``, the target is the audios in ``mix_clean`` instead of separate clean sources. The PR fixes it to use ``mix_clean`` as target.

Pull Request resolved: https://github.com/pytorch/audio/pull/2659

Reviewed By: carolineechen

Differential Revision: D40229227

Pulled By: nateanl

fbshipit-source-id: fc07e0d88a245e1367656d3767cf98168a799235
      c5b8e585
    • Zhaoheng Ni's avatar
      Fix HuBERT docstring (#2746) · be938e7e
      Zhaoheng Ni authored 
      Summary:
The docstring of `wav2vec2` argument is wrong. Fix it in this PR.

Pull Request resolved: https://github.com/pytorch/audio/pull/2746

Reviewed By: carolineechen

Differential Revision: D40225995

Pulled By: nateanl

fbshipit-source-id: 770e9c928ebebd7b6307e181601eb64625d668da
      be938e7e
  7. 09 Oct, 2022 1 commit
  9. 08 Oct, 2022 1 commit
  11. 07 Oct, 2022 3 commits
  13. 06 Oct, 2022 3 commits
  15. 05 Oct, 2022 1 commit
  17. 03 Oct, 2022 3 commits
  19. 01 Oct, 2022 1 commit
  21. 29 Sep, 2022 1 commit
  23. 28 Sep, 2022 3 commits
  25. 27 Sep, 2022 2 commits
  27. 26 Sep, 2022 2 commits
    • Andrey Talman's avatar
      Fix windows tests related to old conda on circleci (#2704) · 1c8accfc
      Andrey Talman authored 
      Summary:
Conda version on circleCI prints following message:
```
==> WARNING: A newer version of conda exists. <==
  current version: 4.6.14
  latest version: 4.14.0
```
and as a result this error:

```
+ /c/tools/miniconda3/Scripts/conda.exe install -v -y -c pytorch-nightly -c nvidia pytorch numpy ffmpeg pytorch-cuda=11.6
Collecting package metadata: ...working... done
Solving environment: ...working...

Too long with no output (exceeded 30m0s): context deadline exceeded
```

This should update the conda version running on the system and allow us to install pytorch and run some tests.

Pull Request resolved: https://github.com/pytorch/audio/pull/2704

Reviewed By: weiwangmeta

Differential Revision: D39820037

Pulled By: atalman

fbshipit-source-id: 4a82a7a6cbe3dc1a5807ac669e2fa79f454037fa
      1c8accfc
    • Andrey Talman's avatar
      Remove linux wheel from circleci (#2714) · 14714f29
      Andrey Talman authored 
      Summary:
Remove linux wheel from circleci

Pull Request resolved: https://github.com/pytorch/audio/pull/2714

Reviewed By: weiwangmeta

Differential Revision: D39816121

Pulled By: atalman

fbshipit-source-id: a3c99b530896888d7b4271d8b3f27f3c986b3480
      14714f29
  29. 24 Sep, 2022 2 commits
    • hwangjeff's avatar
      Fix CUDA check (#2710) · ce08f8d2
      hwangjeff authored 
      Summary:
`torch.version.cuda` can return a string of form X.X or X.X.X. This PR modifies the CUDA version check to account for this.

Pull Request resolved: https://github.com/pytorch/audio/pull/2710

Reviewed By: carolineechen, nateanl

Differential Revision: D39796810

Pulled By: hwangjeff

fbshipit-source-id: b483bd8200195844d65d0caddebaf1b10f939b64
      ce08f8d2
    • hwangjeff's avatar
      Add CUDA version check (#2707) · 0a5825ae
      hwangjeff authored 
      Summary:
Adds check to ensure that TorchAudio and PyTorch versions use the same CUDA version.

Pull Request resolved: https://github.com/pytorch/audio/pull/2707

Reviewed By: mthrok

Differential Revision: D39791154

Pulled By: hwangjeff

fbshipit-source-id: de00889c7bac897c6b8762502f9d37797016b71d
      0a5825ae
  31. 23 Sep, 2022 3 commits
  33. 22 Sep, 2022 2 commits
  35. 21 Sep, 2022 4 commits