GitLab

Summary:
This commit enables ffmpeg-feature build in tests and
binary builds of all platforms.
(Linux/macOS/Windows x conda/wheel)

It also moves the definition of BUILD_FFMPEG env vars to the
top level `config.yml`.

 ---
Manual checking if all the build log contains `libtorchaudio_ffmpeg`.
### binary build
- [x] `binary_linux_conda_py3.7_cpu`
- [x] `binary_linux_conda_py3.7_cu102`
- [x] `binary_linux_wheel_py3.7_cpu`
- [x] `binary_linux_wheel_py3.7_cu102`
- [x] `binary_macos_conda_py3.7_cpu`
- [x] `binary_macos_wheel_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cu113`
- [x] `binary_windows_wheel_py3.7_cpu`
- [x] `binary_windows_wheel_py3.7_cu113`

### test
- [x] `unittest_linux_cpu_py3.7`
- [x] `unittest_linux_gpu_py3.7`
- [x] `unittest_macos_cpu_py3.7`
- [x] `unittest_windows_cpu_py3.7`
- [x] `unittest_windows_gpu_py3.7`
- [x] `integration test`

Pull Request resolved: https://github.com/pytorch/audio/pull/2140

Reviewed By: hwangjeff

Differential Revision: D33464430

Pulled By: mthrok

fbshipit-source-id: 2c5b72be75d49019bf1599036180d4e56074e46b

Summary:
- Unindent RNNTBundle components so that they show up on the right side bar
- Overwrite the sigunature of RNNTBundle methods so that back links are available

 ---

## Before

<img width="1440" alt="Screen Shot 2022-01-06 at 1 36 16 PM" src="https://user-images.githubusercontent.com/855818/148433552-9ba3051d-38b1-4825-9a8f-9173b23650ea.png">

## After

<img width="1436" alt="Screen Shot 2022-01-06 at 1 35 39 PM" src="https://user-images.githubusercontent.com/855818/148433525-733d138d-9a8b-43d6-bdf5-444b52d6a7a9.png">

Pull Request resolved: https://github.com/pytorch/audio/pull/2148

Reviewed By: hwangjeff

Differential Revision: D33458574

Pulled By: mthrok

fbshipit-source-id: ac34ffc4070261563a1f4ea9337997f0fe7b2212

Summary:
This PR:

- Replaces the `data_source` with `lengths`
- Adds a `shuffle` argument to decide whether to shuffle the samples in the buckets.
- Add `max_len` and `min_len` to filter out samples that are > max_len or < min_len.

cc nateanl

Pull Request resolved: https://github.com/pytorch/audio/pull/2147

Reviewed By: carolineechen

Differential Revision: D33454369

Pulled By: nateanl

fbshipit-source-id: 3835169ec7f808f8dd9650e7f183f79091efe886

Summary:
This commits supress the stderr output when shell commands are
executed `setup.py`.

`setup.py` performs multiple git commands to gather metadata.
When `git tag` command tries to find a tag, (which fails unless on release)
it produces `fatal: No names found, cannot describe anything.`

This is confusing especially when it is executed from higher level,
like `pip install git+https://...`.

Pull Request resolved: https://github.com/pytorch/audio/pull/2133

Reviewed By: nateanl

Differential Revision: D33455339

Pulled By: mthrok

fbshipit-source-id: 3e24451eb6fedcd0ad90f7e16e38fcdb70dc9704

Summary:
Drop support for python 3.6, and update dependencies documentation. More details [Issue 2051](https://github.com/pytorch/audio/issues/2051).

Pull Request resolved: https://github.com/pytorch/audio/pull/2139

Reviewed By: mthrok

Differential Revision: D33454583

Pulled By: wernerchao

fbshipit-source-id: 64eccb38e26853ba63f72fb92723e3f0155e806e

Summary:
### New commit log messages
  b64dea9dc Rename `DDPPlugin` to `DDPStrategy` (#11142)

Reviewed By: jjenniferdai

Differential Revision: D33259306

fbshipit-source-id: b4608c6b96b4a7977eaa4ed3f03c4b824882aef0

Summary:
Update the internal of `skipIfXXX` decorators so that tests in CI will not be automatically skipped.

Currently we automatically skip some tests based on the availability of related features/test tools.
This causes issues where we miss signals on certain important features. (CUDA on Windows) https://github.com/pytorch/audio/issues/1565

The new `skipIf` decorator will fail if in CI unless it is explicitly allowed to skip tests.
It does so by checking `CI` and `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` environment variables.

For non-CI environments, the behavior is same as before, but users can now set `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX=false` to disallow the automatic skip.

Results without `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` https://app.circleci.com/pipelines/github/pytorch/audio/9112/workflows/4e6db046-a1a2-4965-b0fe-d5baf4a1efac

Pull Request resolved: https://github.com/pytorch/audio/pull/2127

Reviewed By: hwangjeff

Differential Revision: D33430711

Pulled By: mthrok

fbshipit-source-id: d8954dd720469c5ab0f34ea062fd8cf04a8afa3e

Summary:
remove unnecessary RNNT Loss variables and comment as indicated in https://github.com/pytorch/audio/issues/1479 review comments
(will follow up on `workspace` comments separately depending on complexity)

Pull Request resolved: https://github.com/pytorch/audio/pull/2142

Reviewed By: mthrok

Differential Revision: D33433764

Pulled By: carolineechen

fbshipit-source-id: be0ecb77dabd63d733f0d33ff258eae32305eeaf

Summary:
This change adds a minimal ffmpeg installation step to the build wheel job so that later, we can use the resulting ffmpeg libraries for building torchaudio's ffmpeg-features.

The linux wheel build jobs run in CentOS 8 based environment, which does not provide an easy way to install ffmpeg without conda.

After https://github.com/pytorch/audio/pull/2124 is merged, then we can enable the ffmpeg-feature build in Linux wheel.

Pull Request resolved: https://github.com/pytorch/audio/pull/2137

Reviewed By: carolineechen

Differential Revision: D33430032

Pulled By: mthrok

fbshipit-source-id: bf946d394c0718ddbdc679d7970befc3221982b9

Summary:
Update ffmpeg discovery logic

Previously the build process used pkg-config to locate
an installation of ffmpeg, which does not work well Windows/CentOS.

This commit update the discovery process to use the custom
FindFFMPEG.cmake adopted from Kitware/VTK repository with addition of
conda environment.

 The custom discovery logic can support Windows and CentOS.

Pull Request resolved: https://github.com/pytorch/audio/pull/2124

Reviewed By: carolineechen

Differential Revision: D33429564

Pulled By: mthrok

fbshipit-source-id: 6cb50c1d8c58f51e0f3f3af5c5b541aa3a699bba

Summary:
after addition of tutorial and librispeech example with [WER results](https://github.com/pytorch/audio/pull/2130/files#diff-5f82be20f11a10a4cb411007df965b84eaab98fcc7ed49b42be1dd9916203193R20), we can remove README from decoder csrc directory and move the rest of the information to the corresponding documentation/main README

Pull Request resolved: https://github.com/pytorch/audio/pull/2136

Reviewed By: mthrok

Differential Revision: D33419449

Pulled By: carolineechen

fbshipit-source-id: 6cb29280f639af46834bec935b45f3c5a8ee350f

Summary:
add script for running CTC beam search decoder on librispeech dataset with torchaudio pretrained wav2vec2 models

Pull Request resolved: https://github.com/pytorch/audio/pull/2130

Reviewed By: mthrok

Differential Revision: D33419436

Pulled By: carolineechen

fbshipit-source-id: 0a0d00f4c17ecdbb497c9eda78673aa939d73c57

Summary:
MSVS is strict on header and would not allow `std::runtime_error`
without `<stdexcept>`.

Also this commit removes the stray `"libavutil/frame.h"`,
inserted by IDE, which I missed.

Ref https://github.com/pytorch/audio/issues/2124

Pull Request resolved: https://github.com/pytorch/audio/pull/2135

Reviewed By: carolineechen

Differential Revision: D33419120

Pulled By: mthrok

fbshipit-source-id: 328e723b70a3608133d9ddef9fc4a95e5d90e61d

Summary:
This commit enable ffmpeg-feature build on wheel-based binary distribution on macOS.

For macOS, since the underlying Python env is Conda, it installs `ffmpeg` from conda.

Depends on https://github.com/pytorch/audio/issues/1873

Pull Request resolved: https://github.com/pytorch/audio/pull/2122

Reviewed By: carolineechen, nateanl

Differential Revision: D33409113

Pulled By: mthrok

fbshipit-source-id: a73839087548010353422109b33e89e262c12a57

Summary:
* Before

https://pytorch.org/audio/main/models.html

<img width="852" alt="Screen Shot 2022-01-04 at 11 00 12 AM" src="https://user-images.githubusercontent.com/855818/148087255-3b94e63b-9870-4c7e-95c6-17acc1e65fef.png">

*After

https://503135-90321822-gh.circle-artifacts.com/0/docs/models.html

<img width="842" alt="Screen Shot 2022-01-04 at 10 59 40 AM" src="https://user-images.githubusercontent.com/855818/148087148-b951c7b0-d9cf-4014-8a50-b88c749f12ba.png">

Pull Request resolved: https://github.com/pytorch/audio/pull/2123

Reviewed By: carolineechen

Differential Revision: D33409661

Pulled By: mthrok

fbshipit-source-id: bb2dffea25ccc4356d257b2ab4a6e88f7f4e2bb3

Summary:
update the logic for initializing and updating submodule in setup, now that we have more than one submodule

Pull Request resolved: https://github.com/pytorch/audio/pull/2132

Reviewed By: mthrok

Differential Revision: D33410039

Pulled By: carolineechen

fbshipit-source-id: 6e18ec2904a01efb2f753f65345de063218c0b9b

Summary:
Preparation for updating the build process of ffmpeg-related feature
to support Windows.

Checking-in FindFFMPEG.cmake from Kitware/VTK repo without modification
so that later it's easy to follow our modification on it with git tool.

Pull Request resolved: https://github.com/pytorch/audio/pull/2125

Reviewed By: carolineechen

Differential Revision: D33405408

Pulled By: mthrok

fbshipit-source-id: 5faea8940d2dfcf0b2f647eda3754f713d21fcd1

Summary:
In [Fairseq](https://github.com/pytorch/fairseq/blob/main/examples/hubert/config/pretrain/hubert_base_librispeech.yaml#L48), the training applies additional penalty loss besides the cross-entropy losses. This PR adds the feature's mean square value to the model output to support such penalty loss.

Pull Request resolved: https://github.com/pytorch/audio/pull/2128

Reviewed By: mthrok

Differential Revision: D33403972

Pulled By: nateanl

fbshipit-source-id: f08fefa2c975a847c6075171b310f57c1980309d

Summary:
Currently, macOS CI jobs install `pkg-config` and `wget` with `brew`.
This is problematic as brew takes a long time with auto-update, and disabling the auto-update is not an ideal solution.
Conda also distributes these packages, so switching to conda.

Example issues with brew installation.
https://app.circleci.com/pipelines/github/pytorch/audio/7825/workflows/53965bcf-6ddf-4e42-ad52-83fd1bbab717

This commit removes the use of `brew` by
1. Replacing the use of `wget` with `curl` (pre-installed in most distro)
2. Install `pkg-condig` from conda.
    Note: All the macOS jobs, including binary build jobs, uses conda. Using `pkg-config` from Conda makes it easy to discover the packages installed from conda. (like `ffmpeg` in https://github.com/pytorch/audio/issues/2122)
3. Add `pkg-config` to conda build-time dependency
4. Make sure that the availability of `pkg-config` is explicitly checked when `sox` is being configured. (otherwise, it will fail at somewhere in the middle of build process with somewhat unintuitve error message)

Pull Request resolved: https://github.com/pytorch/audio/pull/1873

Reviewed By: carolineechen, nateanl

Differential Revision: D33404975

Pulled By: mthrok

fbshipit-source-id: ae512d3a3a422ebfe3b46c492bed44deecc36e72

Summary:
This commit enable ffmpeg-feature build on conda-based binary distribution on Linux and macOS.

It adds `ffmpeg` as build-time dependencies and enable the build with `BUILD_FFMPEG=1`.

Windows binaries, wheel-based binaries on Linux/macOS are not changed.

Pull Request resolved: https://github.com/pytorch/audio/pull/2120

Reviewed By: nateanl

Differential Revision: D33397473

Pulled By: mthrok

fbshipit-source-id: 67a23a40c0614c56fee60cc06a45f3265037f6df

Summary:
`libtorchaudio_decoder`, `torchaudio_decoder` and `libtorchaudio_ffmpeg` need not to be linked against `libtorchaudio`.

![dependency graph](https://user-images.githubusercontent.com/855818/147840753-5354bc43-19a5-4a24-a357-6dd3b6ba658e.png)

Pull Request resolved: https://github.com/pytorch/audio/pull/2121

Reviewed By: carolineechen

Differential Revision: D33389197

Pulled By: mthrok

fbshipit-source-id: ebb051e894e17519a87094b8056d26e7a1cd3281

Summary:
As per item 2 on [issue 2051](https://github.com/pytorch/audio/issues/2051), dropping support for python 3.6.
Removed 3.6 from test matrix and ran `.circleci/regenerate.py `.

Pull Request resolved: https://github.com/pytorch/audio/pull/2119

Reviewed By: mthrok

Differential Revision: D33379542

Pulled By: wernerchao

fbshipit-source-id: 6d0fb51b18c2fa7c8cf4eeee4a7f19c4a5210fac

Summary:
add documentaion for CTC decoder `Hypothesis` and include it in docs

Pull Request resolved: https://github.com/pytorch/audio/pull/2117

Reviewed By: mthrok

Differential Revision: D33370381

Pulled By: carolineechen

fbshipit-source-id: cf6501a499e5303cda0410f733f0fab4e1c39aff

Summary:
Preparation to land Python front-end of ffmpeg-related features.

- Set BUILD_FFMPEG=1 in Linux/macOS unit test jobs
- Install ffmpeg and pkg-config from conda-forge
- Add note about Windows build process
- Temporarily avoid `av_err2str`

Pull Request resolved: https://github.com/pytorch/audio/pull/2114

Reviewed By: hwangjeff

Differential Revision: D33371346

Pulled By: mthrok

fbshipit-source-id: b0e16a35959a49a2166109068f3e0cbbb836e888

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2115

Reviewed By: carolineechen

Differential Revision: D33370700

Pulled By: mthrok

fbshipit-source-id: 591b67870247f69cc542f649cd62444ee3c45934

Summary:
cc mthrok

Pull Request resolved: https://github.com/pytorch/audio/pull/2116

Reviewed By: mthrok

Differential Revision: D33368453

Pulled By: jdsgomes

fbshipit-source-id: 09cf3fe5ed6f771c2f16505633c0e59b0c27453c

Summary:
* Removes redundant declaration `right_context_blocks = []`, as flagged by kobenaxie.
* Adds random seed to tests, as flagged by carolineechen in other PRs.

Pull Request resolved: https://github.com/pytorch/audio/pull/2091

Reviewed By: mthrok

Differential Revision: D33340964

Pulled By: hwangjeff

fbshipit-source-id: a9de43e28d1bae7bd4806b280717b0d822bb42fc

Summary:
This PR adds `BUILD_FFMPEG` switch to torchaudio build process so that features related to ffmpeg are built.
The flag is false by default, so no CI jobs or development flow are affected.

This is because handling the dependencies around ffmpeg is a bit tricky.
Currently, the CMake file uses `pkg-config` to find an ffmpeg installation in the system.
This works fine for both conda-based installation and system-managed installation (like `apt`).

In subsequent PRs, I will find a solution that works for local development and binary distributions.

Pull Request resolved: https://github.com/pytorch/audio/pull/2048

Reviewed By: hwangjeff, nateanl

Differential Revision: D33367260

Pulled By: mthrok

fbshipit-source-id: 94517acecb62bd6d4e96d4b7cbc3ab3c2a25706c

Summary:
- Introduce AudioBuffer and VideoBuffer for different way of handling frames
- Update the way option dictionary is passed
- Remove unused AutoFrameUnref
- Add SrcStreamInfo/OutputStreamInfo classes

Pull Request resolved: https://github.com/pytorch/audio/pull/2113

Reviewed By: nateanl

Differential Revision: D33356144

Pulled By: mthrok

fbshipit-source-id: e837e84fae48baa7befd5c70599bcd2cbb61514d

Reviewed By: zertosh

Differential Revision: D33361077

fbshipit-source-id: 007db010bd38c28f597ea66f68f97b13309e878c

Summary:
Part of https://github.com/pytorch/audio/issues/1986. Splitting the PR for easier review.

Add `Streamer` TorchBind.
For the overall architecture, see https://github.com/mthrok/audio/blob/ffmpeg/torchaudio/csrc/ffmpeg/README.md.

Note: Without a change to build process, the code added here won't be compiled. The build process will be updated later.
Needs to be imported after https://github.com/pytorch/audio/issues/2046.

Pull Request resolved: https://github.com/pytorch/audio/pull/2047

Reviewed By: hwangjeff

Differential Revision: D33355190

Pulled By: mthrok

fbshipit-source-id: a3ad4c2822ed3a7ddc19b1aaca9dddabd59ce2f8

Summary:
Adds parameter `p` to `TimeMasking` to allow for enforcing an upper bound on the proportion of time steps that it can mask. This behavior is consistent with the specifications provided in the SpecAugment paper (https://arxiv.org/abs/1904.08779).

Pull Request resolved: https://github.com/pytorch/audio/pull/2090

Reviewed By: carolineechen

Differential Revision: D33344772

Pulled By: hwangjeff

fbshipit-source-id: 6ff65f5304e489fa1c23e15c3d96b9946229fdcf

Summary:
Additionally accept list of tokens as CTC decoder input. This makes it possible to directly pass in something like `bundles.get_labels()` into the decoder factory function instead of requiring a separate tokens file.

Pull Request resolved: https://github.com/pytorch/audio/pull/2112

Reviewed By: hwangjeff, nateanl, mthrok

Differential Revision: D33352909

Pulled By: carolineechen

fbshipit-source-id: 6d22072e34f6cd7c6f931ce4eaf294ae4cf0c5cc

Summary:
Part of https://github.com/pytorch/audio/issues/1986. Splitting the PR for easier review.

Add `Streamer` class that bundles `StreamProcessor` and handle input.
For the overall architecture, see https://github.com/mthrok/audio/blob/ffmpeg/torchaudio/csrc/ffmpeg/README.md.

Note: Without a change to build process, the code added here won't be compiled. The build process will be updated later.
Needs to be imported after https://github.com/pytorch/audio/issues/2045.

Pull Request resolved: https://github.com/pytorch/audio/pull/2046

Reviewed By: carolineechen

Differential Revision: D33299863

Pulled By: mthrok

fbshipit-source-id: 6470cbe061057c8cb970ce7bb5692be04efb5fe9

Summary:
Regroup RNN-T components under `torchaudio.prototype.models` and `torchaudio.prototype.pipelines`.

Updated docs: https://492321-90321822-gh.circle-artifacts.com/0/docs/prototype.html

Pull Request resolved: https://github.com/pytorch/audio/pull/2110

Reviewed By: carolineechen, mthrok

Differential Revision: D33354116

Pulled By: hwangjeff

fbshipit-source-id: 9cf4afed548cb173d56211c16d31bcfa25a8e4cb

Summary:
Part of https://github.com/pytorch/audio/issues/1986. Splitting the PR for easier review.

Add StreamProcessor class that bundles `Buffer`, `FilterGraph` and `Decoder`.
Note: The API to retrieve the buffered Tensors is tentative.
For the overall architecture, see https://github.com/mthrok/audio/blob/ffmpeg/torchaudio/csrc/ffmpeg/README.md.

Note: Without a change to build process, the code added here won't be compiled. The build process will be updated later.
Needs to be imported after https://github.com/pytorch/audio/issues/2044.

Pull Request resolved: https://github.com/pytorch/audio/pull/2045

Reviewed By: carolineechen

Differential Revision: D33299858

Pulled By: mthrok

fbshipit-source-id: d85bececed475f45622743f137dd59cb1390ceed

Summary:
Add Sink class that bundles FilterGraph and Buffer. Part of https://github.com/pytorch/audio/issues/1986. Splitting the PR for easier review.

For the overall architecture, see https://github.com/mthrok/audio/blob/ffmpeg/torchaudio/csrc/ffmpeg/README.md.

Note: Without a change to build process, the code added here won't be compiled. The build process will be updated later.

Pull Request resolved: https://github.com/pytorch/audio/pull/2111

Reviewed By: carolineechen

Differential Revision: D33350388

Pulled By: mthrok

fbshipit-source-id: 8f42c5fe4be39ef2432c51fc0d0ac72ba3f06a26

Reviewed By: zertosh

Differential Revision: D33347867

fbshipit-source-id: 7672f65392e363c0359de2d86e745782a09cf9dc

Summary:
### Change list

* Split the documentation of prototypes
* Add a new API reference section dedicated for prototypes.
* Hide the signature of KenLMLexiconDecoder constructor. (cc carolineechen )
  * https://489516-90321822-gh.circle-artifacts.com/0/docs/prototype.ctc_decoder.html#torchaudio.prototype.ctc_decoder.KenLMLexiconDecoder
* Hide the signature of RNNT constructor. (cc hwangjeff )
  * https://489516-90321822-gh.circle-artifacts.com/0/docs/prototype.rnnt.html#torchaudio.prototype.RNNT
* Tweak CTC tutorial
  * Replace hyperlinks to API reference with backlinks
  * Add `progress=False` to download

### Follow-up

RNNT decoder and CTC decode returns their own `Hypothesis` classes. When I tried to add Hypothesis of CTC decode to the documentation, the build process complains that it's ambiguous.
I think the Hypothesis classes can be put inside of each decoder. (if TorchScript supports it) or make the name different, but in that case the interface of each Hypothesis has to be generic enough.

### Before

https://pytorch.org/audio/main/prototype.html

<img width="1390" alt="Screen Shot 2021-12-28 at 1 05 53 PM" src="https://user-images.githubusercontent.com/855818/147594425-6c7f8126-ab76-4edc-a616-a00901e7e9ef.png">

### After

https://489516-90321822-gh.circle-artifacts.com/0/docs/prototype.html

<img width="1202" alt="Screen Shot 2021-12-28 at 8 37 35 PM" src="https://user-images.githubusercontent.com/855818/147619281-8152b1ae-e127-40b2-a944-dc11b114b629.png">

https://489516-90321822-gh.circle-artifacts.com/0/docs/prototype.rnnt.html

<img width="1415" alt="Screen Shot 2021-12-28 at 8 38 27 PM" src="https://user-images.githubusercontent.com/855818/147619331-077b55b5-c5e9-47ab-bfe6-873e41c738c8.png">

https://489516-90321822-gh.circle-artifacts.com/0/docs/prototype.ctc_decoder.html

<img width="1417" alt="Screen Shot 2021-12-28 at 8 39 04 PM" src="https://user-images.githubusercontent.com/855818/147619364-63df3457-a4b2-4223-973f-f4301bd45280.png">

Pull Request resolved: https://github.com/pytorch/audio/pull/2108

Reviewed By: hwangjeff, carolineechen, nateanl

Differential Revision: D33340816

Pulled By: mthrok

fbshipit-source-id: 870edfadbe41d6f8abaf78fdb7017b3980dfe187

Summary:
Adds pretrained Emformer RNN-T inference pipeline that's capable of performing streaming and non-streaming ASR.

Includes demo script that uses pipeline to alternately perform streaming and non-streaming ASR on LibriSpeech test samples (video below).

https://user-images.githubusercontent.com/8345689/147590753-d5126557-d575-4551-8dfe-5977276cb4ad.mov

Pull Request resolved: https://github.com/pytorch/audio/pull/2093

Reviewed By: mthrok

Differential Revision: D33340776

Pulled By: hwangjeff

fbshipit-source-id: fbb3b1d471b4e9f1b93fa9dea9c464154537a8ac