GitLab

Summary:
Split from https://github.com/pytorch/audio/issues/2164
Add new test assets. Adding this commit separately so that
this commit message about the origin of the file is easier to find.

The original video is in public domain par
- https://svs.gsfc.nasa.gov/13013
- https://www.nasa.gov/multimedia/guidelines/index.html
(The YouTube page directly says so)
- https://www.youtube.com/watch?v=6zNsc0e3Zns

So, the video is modified to fit the needs for testing.
1. multiple audio/video streams
2. Non-audio/video (subtitle) streams
3. Different FPS and sampling rate
4. Ones without audio and video.

```
#!/usr/bin/env bash
original=https://svs.gsfc.nasa.gov/vis/a010000/a013000/a013013/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4
subtitle=https://svs.gsfc.nasa.gov/vis/a010000/a013000/a013013/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-SRT-CC.en_US.srt

# Fetch the original video, embed the subtitle
ffmpeg -i "${original}" -i "${subtitle}" -c:v copy -c:a copy -c:s mov_text -metadata:s:2 language=eng original.mp4 -y

# Extract, rescale video and resample audio
ffmpeg -i original.mp4 -ss 29 -to 42 -c:s copy -vf scale=480:270 -af aresample=16000 tmp1.mp4 -y
ffmpeg -i original.mp4 -ss 29 -to 42 -c:s copy -vf scale=320:180 -r 25 -af aresample=8000  tmp2.mp4 -y

# Merge them, retaining all the streams (6 in total)
ffmpeg -i tmp2.mp4 -i tmp1.mp4 -map 0 -map 1 -c:s copy nasa_13013.mp4 -y

# Make versions without audio / video
ffmpeg -i tmp2.mp4 -c copy -vn nasa_13013_no_video.mp4 -y
ffmpeg -i tmp2.mp4 -c copy -an nasa_13013_no_video.mp4 -y
```

Pull Request resolved: https://github.com/pytorch/audio/pull/2167

Reviewed By: carolineechen

Differential Revision: D33712954

Pulled By: mthrok

fbshipit-source-id: b7cfc1358043a4abd1c0b416e8a8fb0039867211

Summary:
s/gpu.small/gpu.nvidia.medium/

Pull Request resolved: https://github.com/pytorch/audio/pull/1791

Reviewed By: mthrok

Differential Revision: D33697984

Pulled By: malfet

fbshipit-source-id: 0aacad6d4badf023753fa874c8b80c7f65170d0d

Summary:
Closes https://github.com/pytorch/audio/issues/2162

Pull Request resolved: https://github.com/pytorch/audio/pull/2163

Reviewed By: nateanl

Differential Revision: D33666354

Pulled By: mthrok

fbshipit-source-id: 3e7a963b9ac85046317df8d5dab91af363e5668b

Summary:
Find out that tests are failing after change for tester GPU class, see https://github.com/pytorch/audio/pull/1791

Pull Request resolved: https://github.com/pytorch/audio/pull/2165

Reviewed By: mthrok

Differential Revision: D33674802

Pulled By: malfet

fbshipit-source-id: 2e39386c0f129cf44a30d5dfea67e9e2d0e875cf

Summary:
According to [the dataset discription](https://paperswithcode.com/dataset/ted-lium-3), the ``dev`` and ``test`` subsets of TEDLIUM v3 dataset are the same as v2. (under ``TEDLIUM_release-3/legacy`` directory). The ``train`` subset is under ``TEDLIUM_release-3/data`` directory. This PR adds subset support for it.

This also aligns with [TensorFlow's tedlium/release3](https://www.tensorflow.org/datasets/catalog/tedlium#tedliumrelease3) dataset.

Pull Request resolved: https://github.com/pytorch/audio/pull/2157

Reviewed By: mthrok

Differential Revision: D33585211

Pulled By: nateanl

fbshipit-source-id: 87cfe0d02b3a4c2cf7e2da0ccb7443fff5c43689

Summary:
update the labeling reminder to be triggered when PRs are closed rather than merged, because of the transition of merging through fbcode

Pull Request resolved: https://github.com/pytorch/audio/pull/2160

Reviewed By: nateanl

Differential Revision: D33642490

Pulled By: carolineechen

fbshipit-source-id: bb39c66653782694d967303065d40386689789a8

Summary:
additionally add decoding results for wav2vec2 large and also on the test-clean dataset

Pull Request resolved: https://github.com/pytorch/audio/pull/2161

Reviewed By: mthrok

Differential Revision: D33644670

Pulled By: carolineechen

fbshipit-source-id: a219a15af46f82a6bd90169bb3001dbad8f0a96e

Summary:
Currently, the doc build job uses `pytorch/manylinux-cuda100` as base environment image.
This PR changes that with `python:3.X`.

The problem with the previous one is
- The image is unnecessarily huge with tools not needed for build doc. (+3GB)
- No easy way to install ffmpeg>=4.1.

https://518849-90321822-gh.circle-artifacts.com/0/docs/index.html

Pull Request resolved: https://github.com/pytorch/audio/pull/2151

Reviewed By: carolineechen

Differential Revision: D33585043

Pulled By: mthrok

fbshipit-source-id: d6d2f6ab33511b8f5c7ca358bc6545e253c1b752

Summary:
- Change the version of nightly build to `Nightly Build (VERSION)`.
- Use `BUILD_VERSION` env var for release.
- Automatically change copyright year.
- Update the link to nightly in README so that the main branch directs to the corresponding document.

Because of the way CI job is setup, the resulting documentation says 0.8.0. This is fixed by https://github.com/pytorch/audio/issues/2151.

Pull Request resolved: https://github.com/pytorch/audio/pull/2152

Reviewed By: carolineechen, nateanl

Differential Revision: D33585053

Pulled By: mthrok

fbshipit-source-id: 3c2bf9fc3214c89f989f5ac65b74bc1e276a7161

[PyTorchLightning/pytorch-lightning] Add deprecation path for renamed training type plugins (#11227)

Summary:
### New commit log messages
  4eede7c30 Add deprecation path for renamed training type plugins (#11227)

Reviewed By: edward-io, daniellepintz

Differential Revision: D33409991

fbshipit-source-id: 373e48767e992d67db3c85e436648481ad16c9d0

Summary:
Add explanation and demonstration of different beam search decoder parameters.
Additionally use a better sample audio file and load in with token list instead of tokens file.

Pull Request resolved: https://github.com/pytorch/audio/pull/2141

Reviewed By: mthrok

Differential Revision: D33463230

Pulled By: carolineechen

fbshipit-source-id: d3dd6452b03d4fc2e095d778189c66f7161e4c68

Summary:
This commit enables ffmpeg-feature build in tests and
binary builds of all platforms.
(Linux/macOS/Windows x conda/wheel)

It also moves the definition of BUILD_FFMPEG env vars to the
top level `config.yml`.

 ---
Manual checking if all the build log contains `libtorchaudio_ffmpeg`.
### binary build
- [x] `binary_linux_conda_py3.7_cpu`
- [x] `binary_linux_conda_py3.7_cu102`
- [x] `binary_linux_wheel_py3.7_cpu`
- [x] `binary_linux_wheel_py3.7_cu102`
- [x] `binary_macos_conda_py3.7_cpu`
- [x] `binary_macos_wheel_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cu113`
- [x] `binary_windows_wheel_py3.7_cpu`
- [x] `binary_windows_wheel_py3.7_cu113`

### test
- [x] `unittest_linux_cpu_py3.7`
- [x] `unittest_linux_gpu_py3.7`
- [x] `unittest_macos_cpu_py3.7`
- [x] `unittest_windows_cpu_py3.7`
- [x] `unittest_windows_gpu_py3.7`
- [x] `integration test`

Pull Request resolved: https://github.com/pytorch/audio/pull/2140

Reviewed By: hwangjeff

Differential Revision: D33464430

Pulled By: mthrok

fbshipit-source-id: 2c5b72be75d49019bf1599036180d4e56074e46b

Summary:
- Unindent RNNTBundle components so that they show up on the right side bar
- Overwrite the sigunature of RNNTBundle methods so that back links are available

 ---

## Before

<img width="1440" alt="Screen Shot 2022-01-06 at 1 36 16 PM" src="https://user-images.githubusercontent.com/855818/148433552-9ba3051d-38b1-4825-9a8f-9173b23650ea.png">

## After

<img width="1436" alt="Screen Shot 2022-01-06 at 1 35 39 PM" src="https://user-images.githubusercontent.com/855818/148433525-733d138d-9a8b-43d6-bdf5-444b52d6a7a9.png">

Pull Request resolved: https://github.com/pytorch/audio/pull/2148

Reviewed By: hwangjeff

Differential Revision: D33458574

Pulled By: mthrok

fbshipit-source-id: ac34ffc4070261563a1f4ea9337997f0fe7b2212

Summary:
This PR:

- Replaces the `data_source` with `lengths`
- Adds a `shuffle` argument to decide whether to shuffle the samples in the buckets.
- Add `max_len` and `min_len` to filter out samples that are > max_len or < min_len.

cc nateanl

Pull Request resolved: https://github.com/pytorch/audio/pull/2147

Reviewed By: carolineechen

Differential Revision: D33454369

Pulled By: nateanl

fbshipit-source-id: 3835169ec7f808f8dd9650e7f183f79091efe886

Summary:
This commits supress the stderr output when shell commands are
executed `setup.py`.

`setup.py` performs multiple git commands to gather metadata.
When `git tag` command tries to find a tag, (which fails unless on release)
it produces `fatal: No names found, cannot describe anything.`

This is confusing especially when it is executed from higher level,
like `pip install git+https://...`.

Pull Request resolved: https://github.com/pytorch/audio/pull/2133

Reviewed By: nateanl

Differential Revision: D33455339

Pulled By: mthrok

fbshipit-source-id: 3e24451eb6fedcd0ad90f7e16e38fcdb70dc9704

Summary:
Drop support for python 3.6, and update dependencies documentation. More details [Issue 2051](https://github.com/pytorch/audio/issues/2051).

Pull Request resolved: https://github.com/pytorch/audio/pull/2139

Reviewed By: mthrok

Differential Revision: D33454583

Pulled By: wernerchao

fbshipit-source-id: 64eccb38e26853ba63f72fb92723e3f0155e806e

Summary:
### New commit log messages
  b64dea9dc Rename `DDPPlugin` to `DDPStrategy` (#11142)

Reviewed By: jjenniferdai

Differential Revision: D33259306

fbshipit-source-id: b4608c6b96b4a7977eaa4ed3f03c4b824882aef0

Summary:
Update the internal of `skipIfXXX` decorators so that tests in CI will not be automatically skipped.

Currently we automatically skip some tests based on the availability of related features/test tools.
This causes issues where we miss signals on certain important features. (CUDA on Windows) https://github.com/pytorch/audio/issues/1565

The new `skipIf` decorator will fail if in CI unless it is explicitly allowed to skip tests.
It does so by checking `CI` and `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` environment variables.

For non-CI environments, the behavior is same as before, but users can now set `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX=false` to disallow the automatic skip.

Results without `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` https://app.circleci.com/pipelines/github/pytorch/audio/9112/workflows/4e6db046-a1a2-4965-b0fe-d5baf4a1efac

Pull Request resolved: https://github.com/pytorch/audio/pull/2127

Reviewed By: hwangjeff

Differential Revision: D33430711

Pulled By: mthrok

fbshipit-source-id: d8954dd720469c5ab0f34ea062fd8cf04a8afa3e

Summary:
remove unnecessary RNNT Loss variables and comment as indicated in https://github.com/pytorch/audio/issues/1479 review comments
(will follow up on `workspace` comments separately depending on complexity)

Pull Request resolved: https://github.com/pytorch/audio/pull/2142

Reviewed By: mthrok

Differential Revision: D33433764

Pulled By: carolineechen

fbshipit-source-id: be0ecb77dabd63d733f0d33ff258eae32305eeaf

Summary:
This change adds a minimal ffmpeg installation step to the build wheel job so that later, we can use the resulting ffmpeg libraries for building torchaudio's ffmpeg-features.

The linux wheel build jobs run in CentOS 8 based environment, which does not provide an easy way to install ffmpeg without conda.

After https://github.com/pytorch/audio/pull/2124 is merged, then we can enable the ffmpeg-feature build in Linux wheel.

Pull Request resolved: https://github.com/pytorch/audio/pull/2137

Reviewed By: carolineechen

Differential Revision: D33430032

Pulled By: mthrok

fbshipit-source-id: bf946d394c0718ddbdc679d7970befc3221982b9

Summary:
Update ffmpeg discovery logic

Previously the build process used pkg-config to locate
an installation of ffmpeg, which does not work well Windows/CentOS.

This commit update the discovery process to use the custom
FindFFMPEG.cmake adopted from Kitware/VTK repository with addition of
conda environment.

 The custom discovery logic can support Windows and CentOS.

Pull Request resolved: https://github.com/pytorch/audio/pull/2124

Reviewed By: carolineechen

Differential Revision: D33429564

Pulled By: mthrok

fbshipit-source-id: 6cb50c1d8c58f51e0f3f3af5c5b541aa3a699bba

Summary:
after addition of tutorial and librispeech example with [WER results](https://github.com/pytorch/audio/pull/2130/files#diff-5f82be20f11a10a4cb411007df965b84eaab98fcc7ed49b42be1dd9916203193R20), we can remove README from decoder csrc directory and move the rest of the information to the corresponding documentation/main README

Pull Request resolved: https://github.com/pytorch/audio/pull/2136

Reviewed By: mthrok

Differential Revision: D33419449

Pulled By: carolineechen

fbshipit-source-id: 6cb29280f639af46834bec935b45f3c5a8ee350f

Summary:
add script for running CTC beam search decoder on librispeech dataset with torchaudio pretrained wav2vec2 models

Pull Request resolved: https://github.com/pytorch/audio/pull/2130

Reviewed By: mthrok

Differential Revision: D33419436

Pulled By: carolineechen

fbshipit-source-id: 0a0d00f4c17ecdbb497c9eda78673aa939d73c57

Summary:
MSVS is strict on header and would not allow `std::runtime_error`
without `<stdexcept>`.

Also this commit removes the stray `"libavutil/frame.h"`,
inserted by IDE, which I missed.

Ref https://github.com/pytorch/audio/issues/2124

Pull Request resolved: https://github.com/pytorch/audio/pull/2135

Reviewed By: carolineechen

Differential Revision: D33419120

Pulled By: mthrok

fbshipit-source-id: 328e723b70a3608133d9ddef9fc4a95e5d90e61d

Summary:
This commit enable ffmpeg-feature build on wheel-based binary distribution on macOS.

For macOS, since the underlying Python env is Conda, it installs `ffmpeg` from conda.

Depends on https://github.com/pytorch/audio/issues/1873

Pull Request resolved: https://github.com/pytorch/audio/pull/2122

Reviewed By: carolineechen, nateanl

Differential Revision: D33409113

Pulled By: mthrok

fbshipit-source-id: a73839087548010353422109b33e89e262c12a57

Summary:
* Before

https://pytorch.org/audio/main/models.html

<img width="852" alt="Screen Shot 2022-01-04 at 11 00 12 AM" src="https://user-images.githubusercontent.com/855818/148087255-3b94e63b-9870-4c7e-95c6-17acc1e65fef.png">

*After

https://503135-90321822-gh.circle-artifacts.com/0/docs/models.html

<img width="842" alt="Screen Shot 2022-01-04 at 10 59 40 AM" src="https://user-images.githubusercontent.com/855818/148087148-b951c7b0-d9cf-4014-8a50-b88c749f12ba.png">

Pull Request resolved: https://github.com/pytorch/audio/pull/2123

Reviewed By: carolineechen

Differential Revision: D33409661

Pulled By: mthrok

fbshipit-source-id: bb2dffea25ccc4356d257b2ab4a6e88f7f4e2bb3

Summary:
update the logic for initializing and updating submodule in setup, now that we have more than one submodule

Pull Request resolved: https://github.com/pytorch/audio/pull/2132

Reviewed By: mthrok

Differential Revision: D33410039

Pulled By: carolineechen

fbshipit-source-id: 6e18ec2904a01efb2f753f65345de063218c0b9b

Summary:
Preparation for updating the build process of ffmpeg-related feature
to support Windows.

Checking-in FindFFMPEG.cmake from Kitware/VTK repo without modification
so that later it's easy to follow our modification on it with git tool.

Pull Request resolved: https://github.com/pytorch/audio/pull/2125

Reviewed By: carolineechen

Differential Revision: D33405408

Pulled By: mthrok

fbshipit-source-id: 5faea8940d2dfcf0b2f647eda3754f713d21fcd1

Summary:
In [Fairseq](https://github.com/pytorch/fairseq/blob/main/examples/hubert/config/pretrain/hubert_base_librispeech.yaml#L48), the training applies additional penalty loss besides the cross-entropy losses. This PR adds the feature's mean square value to the model output to support such penalty loss.

Pull Request resolved: https://github.com/pytorch/audio/pull/2128

Reviewed By: mthrok

Differential Revision: D33403972

Pulled By: nateanl

fbshipit-source-id: f08fefa2c975a847c6075171b310f57c1980309d

Summary:
Currently, macOS CI jobs install `pkg-config` and `wget` with `brew`.
This is problematic as brew takes a long time with auto-update, and disabling the auto-update is not an ideal solution.
Conda also distributes these packages, so switching to conda.

Example issues with brew installation.
https://app.circleci.com/pipelines/github/pytorch/audio/7825/workflows/53965bcf-6ddf-4e42-ad52-83fd1bbab717

This commit removes the use of `brew` by
1. Replacing the use of `wget` with `curl` (pre-installed in most distro)
2. Install `pkg-condig` from conda.
    Note: All the macOS jobs, including binary build jobs, uses conda. Using `pkg-config` from Conda makes it easy to discover the packages installed from conda. (like `ffmpeg` in https://github.com/pytorch/audio/issues/2122)
3. Add `pkg-config` to conda build-time dependency
4. Make sure that the availability of `pkg-config` is explicitly checked when `sox` is being configured. (otherwise, it will fail at somewhere in the middle of build process with somewhat unintuitve error message)

Pull Request resolved: https://github.com/pytorch/audio/pull/1873

Reviewed By: carolineechen, nateanl

Differential Revision: D33404975

Pulled By: mthrok

fbshipit-source-id: ae512d3a3a422ebfe3b46c492bed44deecc36e72

Summary:
This commit enable ffmpeg-feature build on conda-based binary distribution on Linux and macOS.

It adds `ffmpeg` as build-time dependencies and enable the build with `BUILD_FFMPEG=1`.

Windows binaries, wheel-based binaries on Linux/macOS are not changed.

Pull Request resolved: https://github.com/pytorch/audio/pull/2120

Reviewed By: nateanl

Differential Revision: D33397473

Pulled By: mthrok

fbshipit-source-id: 67a23a40c0614c56fee60cc06a45f3265037f6df

Summary:
`libtorchaudio_decoder`, `torchaudio_decoder` and `libtorchaudio_ffmpeg` need not to be linked against `libtorchaudio`.

![dependency graph](https://user-images.githubusercontent.com/855818/147840753-5354bc43-19a5-4a24-a357-6dd3b6ba658e.png)

Pull Request resolved: https://github.com/pytorch/audio/pull/2121

Reviewed By: carolineechen

Differential Revision: D33389197

Pulled By: mthrok

fbshipit-source-id: ebb051e894e17519a87094b8056d26e7a1cd3281

Summary:
As per item 2 on [issue 2051](https://github.com/pytorch/audio/issues/2051), dropping support for python 3.6.
Removed 3.6 from test matrix and ran `.circleci/regenerate.py `.

Pull Request resolved: https://github.com/pytorch/audio/pull/2119

Reviewed By: mthrok

Differential Revision: D33379542

Pulled By: wernerchao

fbshipit-source-id: 6d0fb51b18c2fa7c8cf4eeee4a7f19c4a5210fac

Summary:
add documentaion for CTC decoder `Hypothesis` and include it in docs

Pull Request resolved: https://github.com/pytorch/audio/pull/2117

Reviewed By: mthrok

Differential Revision: D33370381

Pulled By: carolineechen

fbshipit-source-id: cf6501a499e5303cda0410f733f0fab4e1c39aff

Summary:
Preparation to land Python front-end of ffmpeg-related features.

- Set BUILD_FFMPEG=1 in Linux/macOS unit test jobs
- Install ffmpeg and pkg-config from conda-forge
- Add note about Windows build process
- Temporarily avoid `av_err2str`

Pull Request resolved: https://github.com/pytorch/audio/pull/2114

Reviewed By: hwangjeff

Differential Revision: D33371346

Pulled By: mthrok

fbshipit-source-id: b0e16a35959a49a2166109068f3e0cbbb836e888

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2115

Reviewed By: carolineechen

Differential Revision: D33370700

Pulled By: mthrok

fbshipit-source-id: 591b67870247f69cc542f649cd62444ee3c45934

Summary:
cc mthrok

Pull Request resolved: https://github.com/pytorch/audio/pull/2116

Reviewed By: mthrok

Differential Revision: D33368453

Pulled By: jdsgomes

fbshipit-source-id: 09cf3fe5ed6f771c2f16505633c0e59b0c27453c

Summary:
* Removes redundant declaration `right_context_blocks = []`, as flagged by kobenaxie.
* Adds random seed to tests, as flagged by carolineechen in other PRs.

Pull Request resolved: https://github.com/pytorch/audio/pull/2091

Reviewed By: mthrok

Differential Revision: D33340964

Pulled By: hwangjeff

fbshipit-source-id: a9de43e28d1bae7bd4806b280717b0d822bb42fc

Summary:
This PR adds `BUILD_FFMPEG` switch to torchaudio build process so that features related to ffmpeg are built.
The flag is false by default, so no CI jobs or development flow are affected.

This is because handling the dependencies around ffmpeg is a bit tricky.
Currently, the CMake file uses `pkg-config` to find an ffmpeg installation in the system.
This works fine for both conda-based installation and system-managed installation (like `apt`).

In subsequent PRs, I will find a solution that works for local development and binary distributions.

Pull Request resolved: https://github.com/pytorch/audio/pull/2048

Reviewed By: hwangjeff, nateanl

Differential Revision: D33367260

Pulled By: mthrok

fbshipit-source-id: 94517acecb62bd6d4e96d4b7cbc3ab3c2a25706c

Summary:
- Introduce AudioBuffer and VideoBuffer for different way of handling frames
- Update the way option dictionary is passed
- Remove unused AutoFrameUnref
- Add SrcStreamInfo/OutputStreamInfo classes

Pull Request resolved: https://github.com/pytorch/audio/pull/2113

Reviewed By: nateanl

Differential Revision: D33356144

Pulled By: mthrok

fbshipit-source-id: e837e84fae48baa7befd5c70599bcd2cbb61514d