GitLab

Summary:
This commit moves the Streaming API out of prototype module.

* The related classes are renamed as following

  - `Streamer` -> `StreamReader`.
  - `SourceStream` -> `StreamReaderSourceStream`
  - `SourceAudioStream` -> `StreamReaderSourceAudioStream`
  - `SourceVideoStream` -> `StreamReaderSourceVideoStream`
  - `OutputStream` -> `StreamReaderOutputStream`

This change is preemptive measurement for the possibility to add
`StreamWriter` API.

* Replace BUILD_FFMPEG build arg with USE_FFMPEG

We are not building FFmpeg, so USE_FFMPEG is more appropriate

 ---

After https://github.com/pytorch/audio/issues/2377

Remaining TODOs: (different PRs)
- [ ] Introduce `is_ffmpeg_binding_available` function.
- [ ] Refactor C++ code:
   - Rename `Streamer` to `StreamReader`.
   - Rename `streamer.[h|cpp]` to `stream_reader.[h|cpp]`.
   - Rename `prototype.cpp` to `stream_reader_binding.cpp`.
   - Introduce `stream_reader` directory.
- [x] Enable FFmpeg in smoke test (https://github.com/pytorch/audio/issues/2381)

Pull Request resolved: https://github.com/pytorch/audio/pull/2378

Reviewed By: carolineechen

Differential Revision: D36359299

Pulled By: mthrok

fbshipit-source-id: 6a57b702996af871e577fb7addbf3522081c1328

Summary:
This commit updates the lazy module initialization logic for
`torchaudio.prototype.io` and `torchaudio.prototype.ctc_decoder`.

- The modules are importable regarless of optional dependencies.
i.e. `import torchaudio.prototype.io` does not trigger the check for
optional dependencies.

- Optional dependencies are checked when the actual
API is imported for the first time.
i.e. `from torchaudio.prototype.io import Streamer` triggers the check
for optional dependencies.

The downside is that;

- `import torchaudio.prototype.io.Streamer` no longer works.

## Details:

Starting from Python 3.7, modules can bave `__getattr__` function,
which serves as a fallback if the import mechanism cannot find the
attribute.

This can be used to implement lazy import.

```python
def __getattr__(name):
    global pi
    if name == 'pi':
        import math
        pi = math.pi
        return pi
    raise AttributeError(...)
```

Ref: https://twitter.com/raymondh/status/1094686528440168453

The implementation performs lazy import for the APIs that work with
external/optional dependencies. In addition, it also check if the
binding is initialized only once.

## Why is this preferable approach?

Previously, the optional dependencies were checked at the tiem module
is imported;

https://github.com/pytorch/audio/blob/2f4eb4ac2f48a597825d3631a840afd855fe6b39/torchaudio/prototype/io/__init__.py#L1-L5

As long as this module is in `prototype`, which we ask users to import
explictly, users had control whether they want/do not want to install
the optional dependencies.

This approach only works for one optional dependencies per one module.
Say, we add different I/O library as an optional dependency, we need to
put all the APIs in dedicated submodule. This prevents us from having
flat namespace.
i.e. the I/O modules with multiple optional dependencies would look like

```python
# Client code
from torchaudio.io.foo import FooFeature
from torchaudio.io.bar import BarFeature
```

where the new approach would allow

```python
#client code
from torchaudio.io import FooFeature, BarFeature
```

Pull Request resolved: https://github.com/pytorch/audio/pull/2377

Reviewed By: nateanl

Differential Revision: D36305603

Pulled By: mthrok

fbshipit-source-id: c1eb6cac203f6dd0026d99f9a1de1af590a535ae

Summary:
- Use `apply_beamforming`, `rtf_evd`, `rtf_power`, `mvdr_weights_souden`, `mvdr_weights_rtf` methods under `torchaudio.functional` to replace the class methods.
- Refactor docstrings in `PSD` and `MVDR`.
- Put `_get_mvdr_vector` outside of `MVDR` class as it doesn't call self methods inside.
- Since MVDR uses einsum for matrix operations, packing and unpacking batches are not necessary. It can be tested by the [batch_consistency_test](https://github.com/pytorch/audio/blob/main/test/torchaudio_unittest/transforms/batch_consistency_test.py#L202). Removed it from the code.

Pull Request resolved: https://github.com/pytorch/audio/pull/2383

Reviewed By: carolineechen, mthrok

Differential Revision: D36338373

Pulled By: nateanl

fbshipit-source-id: a48a6ae2825657e5967a19656245596cdf037c5f

Summary:
- When cropping the waveform and corresponding label, we use the formula `torch.div(audio_start - kernel_size * sample_rate, stride * sample_rate, rounding_mode="floor")` to align the audio start and label start indices. However, sometimes the value can be negative, which result in an empty label. The training example will hurt the performance after zero-padding (i.e., the labels are all zero for the input waveform).
This PR fixes the bug by checking if `label_start` is negative, and change it to zero if so.
- If `pad` is True, the `length` should be the length of each waveform instead of the max length. Fix it to make the model ignore the padding component in pre-training.

Pull Request resolved: https://github.com/pytorch/audio/pull/2296

Reviewed By: mthrok

Differential Revision: D36323217

Pulled By: nateanl

fbshipit-source-id: 1ffa71e39bbc0e8dee55c3b829911bc2e785b423

Summary:
Applies the black-fbsource codemod with the new build of pyfmt.

paintitblack

Reviewed By: lisroach

Differential Revision: D36324783

fbshipit-source-id: 280c09e88257e5e569ab729691165d8dedd767bc

Summary:
Conda package build performs simple smoke test, which is different
from smoke_test jobs we define on our CI jobs.

Currently Conda packaging smoke test verifies the imporatability of
`torchaudio.prototype.io`, which requires FFmpeg 4.

1. We list FFmpeg 4 as runtime requirements, but this means that
conda's dependency resolver takes FFmpeg 4 into consideration.
FFmpeg 5 was release this year, and we can expect that user base
will move to FFmpeg gradually. If user environment has some constraint
on FFmpeg, torchaudio will have conflict and it will prevent users
from install torchaudio.

2. In #2377 the way optional dependency is checked/initialized is changed,
so this Conda smoke test will no longer check the integrity with FFmpeg libraries.

To solve the issues above, this commit moves the part that tests integrity with
FFmpeg libraries to the smoke test we define on CircleCI.

Pull Request resolved: https://github.com/pytorch/audio/pull/2381

Reviewed By: carolineechen

Differential Revision: D36323706

Pulled By: mthrok

fbshipit-source-id: 57ca816e0f3ad8e16d21e56062f6ed8a09ab93a3

Summary:
The modules include:
- PSD
- MVDR
- RTFMVDR
- SoudenMVDR

Pull Request resolved: https://github.com/pytorch/audio/pull/2382

Reviewed By: carolineechen

Differential Revision: D36314096

Pulled By: nateanl

fbshipit-source-id: 9d7d962b1c70cdc435a579191ad88838dd6fc0ba

Summary:
Since a while ago, CodeQL is always emitting red signal, but the team
does not know what this is / how to fix this. At this point, it is
purely noise while not providing a valuable signal.

Ref https://github.com/pytorch/audio/issues/2314

Pull Request resolved: https://github.com/pytorch/audio/pull/2380

Reviewed By: carolineechen

Differential Revision: D36305599

Pulled By: mthrok

fbshipit-source-id: 27ece58730066543600f3873397b9a239e54beb0

Summary:
On CircleCI, Windows unittests are failing for Python 3.7 with
`PermissionError` at the end of test when it cleans up temporary
directory.

According to the discussion https://github.com/python/cpython/issues/74168,
this is caused by a known issue with `shutil.rmtree`.

In the above thread it is advised to simply ignore the error as it
is not guaranteed that temp directories are cleaned up.

This commit follows the same path and simply ignore the error
so that our CI gets back to green.

Pull Request resolved: https://github.com/pytorch/audio/pull/2379

Reviewed By: carolineechen

Differential Revision: D36305595

Pulled By: mthrok

fbshipit-source-id: d9049c2ee3447712119786311f639a1f9f8911c5

Summary:
Modifies the example LibriSpeech Conformer RNN-T recipe as follows:
- Moves data loading and transforms logic from lightning module to data module (improves generalizability and reusability of lightning module and data module).
- Moves transforms logic from dataloader collator function to dataset (resolves dataloader multiprocessing issues on certain platforms).
- Replaces lambda functions with `partial` equivalents (resolves pickling issues in certain runtime environments).
- Modifies training script to allow for specifying path model checkpoint to restart training from.

Pull Request resolved: https://github.com/pytorch/audio/pull/2366

Reviewed By: mthrok

Differential Revision: D36305028

Pulled By: hwangjeff

fbshipit-source-id: 0b768da5d5909136c55418bf0a3c2ddd0c5683ba

Summary:
This commit refactor the constructor of wrapper classes so that
wrapper classes are only responsible for deallocation of underlying
FFmpeg custom structures.

The responsibility of custom initialization is moved to helper functions.

Context:

FFmpeg API uses bunch of raw pointers, which require dedicated allocater
and deallcoator. In torchaudio we wrap these pointers with
`std::unique_ptr<>` to adopt RAII semantics.

Currently all of the customization logics required for `Streamer` are
handled by the constructor of wrapper class. Like the following;

```
AVFormatContextPtr(
      const std::string& src,
      const std::string& device,
      const std::map<std::string, std::string>& option);
```

This constructor allocates the raw `AVFormatContext*` pointer,
while initializing it with the given option, then it parses the
input media.

As we consider the write/encode features, which require different way
of initializing the `AVFormatContext*`, making it the responsibility
of constructors of `AVFormatContextPtr` reduce the flexibility.

Thus this commit moves the customization to helper factory function.

- `AVFormatContextPtr(...)` -> `get_input_format_context(...)`
- `AVCodecContextPtr(...)` -> `get_decode_context(...)`

Pull Request resolved: https://github.com/pytorch/audio/pull/2373

Reviewed By: hwangjeff

Differential Revision: D36230148

Pulled By: mthrok

fbshipit-source-id: 202d57d549223904ee958193f3b386ef5a9cda3a

Summary:
Adds an implementation of the convolution-augmented streaming transformer (effectively Emformer with convolution block) described in https://arxiv.org/abs/2110.05241.

Continuation of https://github.com/pytorch/audio/issues/2324.

Pull Request resolved: https://github.com/pytorch/audio/pull/2358

Reviewed By: nateanl, xiaohui-zhang

Differential Revision: D36137992

Pulled By: hwangjeff

fbshipit-source-id: 9c7a7c233944fe9ef15b9ba397d7f0809da1f063

Summary:
Address https://github.com/pytorch/audio/issues/2375
The MVDR module internally transforms the dtype of complex tensors to `torch.complex128` for computation and transforms it back to the original dtype before returning the Tensor. However, it didn't convert back successfully due to `specgram_enhanced.to(dtype)`, which should be `specgram_enhanced = specgram_enhanced.to(dtype)`. Fix it to make the output dtype consistent with original input.

Pull Request resolved: https://github.com/pytorch/audio/pull/2376

Reviewed By: hwangjeff

Differential Revision: D36280851

Pulled By: nateanl

fbshipit-source-id: 553d1b98f899547209a4e3ebc59920c7ef1f3112

Summary:
previous update for rocm: https://github.com/pytorch/audio/pull/2186

Pull Request resolved: https://github.com/pytorch/audio/pull/2362

Reviewed By: seemethere

Differential Revision: D36283672

Pulled By: mthrok

fbshipit-source-id: bfd38940d027c8ccd72ab48991e5ab7f84b0e9c0

Summary:
Add a new design of MVDR module.
The RTFMVDR module supports the method based on the relative transfer function (RTF) and power spectral density (PSD) matrix of noise.
The input arguments are:
- multi-channel spectrum.
- RTF vector of the target speech
- PSD matrix of noise.
- reference channel in the microphone array.
- diagonal_loading option to enable or disable diagonal loading in matrix inverse computation.
- diag_eps for computing the inverse of the matrix.
- eps for computing the beamforming weight.
The output of the module is the single-channel complex-valued spectrum for the enhanced speech.

Pull Request resolved: https://github.com/pytorch/audio/pull/2368

Reviewed By: carolineechen

Differential Revision: D36214940

Pulled By: nateanl

fbshipit-source-id: 5f29f778663c96591e1b520b15f7876d07116937

Summary:
When computing the MVDR beamforming weights using the power iteration method, the PSD matrix of noise can be applied with diagonal loading to improve the robustness. This is also applicable to computing the RTF matrix (See https://github.com/espnet/espnet/blob/master/espnet2/enh/layers/beamformer.py#L614 as an example). This also aligns with current `torchaudio.transforms.MVDR` module to keep the consistency.

This PR adds the `diagonal_loading` argument with `True` as default value to `torchaudio.functional.rtf_power`.

Pull Request resolved: https://github.com/pytorch/audio/pull/2369

Reviewed By: carolineechen

Differential Revision: D36204130

Pulled By: nateanl

fbshipit-source-id: 93a58d5c2107841a16c4e32f0c16ab0d6b2d9420

Summary:
Add a new design of MVDR module.
The `SoudenMVDR` module supports the method proposed by [Souden et, al.](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.725.673&rep=rep1&type=pdf).
The input arguments are:
- multi-channel spectrum.
- PSD matrix of target speech.
- PSD matrix of noise.
- reference channel in the microphone array.
- diagonal_loading option to enable or disable diagonal loading in matrix inverse computation.
- diag_eps for computing the inverse of the matrix.
- eps for computing the beamforming weight.

The output of the module is the single-channel complex-valued spectrum for the enhanced speech.

Pull Request resolved: https://github.com/pytorch/audio/pull/2367

Reviewed By: hwangjeff

Differential Revision: D36198015

Pulled By: nateanl

fbshipit-source-id: 4027f4752a84aaef730ef3ea8c625e801cc35527

Summary:
This commits add `hw_accel` option to `Streamer::add_video_stream` method.
Specifying `hw_accel="cuda"` allows to create the chunk Tensor directly from CUDA,
when the following conditions are met.
1. the video format is H264,
2. underlying ffmpeg is compiled with NVENC, and
3. the client code specifies `decoder="h264_cuvid"`.

A simple benchmark yields x7 improvement in the decoding speed.

<details>

```python
import time

from torchaudio.prototype.io import Streamer

srcs = [
    "https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4",
    "./NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4",  # offline version
]

patterns = [
    ("h264_cuvid", None, "cuda:0"),  # NVDEC on CUDA:0 -> CUDA:0
    ("h264_cuvid", None, "cuda:1"),  # NVDEC on CUDA:1 -> CUDA:1
    ("h264_cuvid", None, None),  # NVDEC -> CPU
    (None, None, None),  # CPU
]

for src in srcs:
    print(src, flush=True)
    for (decoder, decoder_options, hw_accel) in patterns:
        s = Streamer(src)
        s.add_video_stream(5, decoder=decoder, decoder_options=decoder_options, hw_accel=hw_accel)

        t0 = time.monotonic()
        num_frames = 0
	for i, (chunk, ) in enumerate(s.stream()):
	    num_frames += chunk.shape[0]
        t1 = time.monotonic()
        print(chunk.dtype, chunk.shape, chunk.device)
        print(time.monotonic() - t0, num_frames, flush=True)
```
</details>

```
https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4
torch.uint8 torch.Size([5, 3, 1080, 1920]) cuda:0
10.781158386962488 6175
torch.uint8 torch.Size([5, 3, 1080, 1920]) cuda:1
10.771313901990652 6175
torch.uint8 torch.Size([5, 3, 1080, 1920]) cpu
27.88662809302332 6175
torch.uint8 torch.Size([5, 3, 1080, 1920]) cpu
83.22728440898936 6175
./NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4
torch.uint8 torch.Size([5, 3, 1080, 1920]) cuda:0
12.945253834011964 6175
torch.uint8 torch.Size([5, 3, 1080, 1920]) cuda:1
12.870224556012545 6175
torch.uint8 torch.Size([5, 3, 1080, 1920]) cpu
28.03406483103754 6175
torch.uint8 torch.Size([5, 3, 1080, 1920]) cpu
82.6120332319988 6175
```

With HW resizing

<details>

```python
import time

from torchaudio.prototype.io import Streamer

srcs = [
    "./NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4",
    "https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4",
]

patterns = [
    # Decode with NVDEC, CUDA HW scaling -> CUDA:0
    ("h264_cuvid", {"resize": "960x540"}, "", "cuda:0"),
    # Decoded with NVDEC, CUDA HW scaling -> CPU
    ("h264_cuvid", {"resize": "960x540"}, "", None),
    # CPU decoding, CPU scaling
    (None, None, "scale=width=960:height=540", None),
]

for src in srcs:
    print(src, flush=True)
    for (decoder, decoder_options, filter_desc, hw_accel) in patterns:
        s = Streamer(src)
        s.add_video_stream(
            5,
            decoder=decoder,
            decoder_options=decoder_options,
            filter_desc=filter_desc,
            hw_accel=hw_accel,
        )

        t0 = time.monotonic()
        num_frames = 0
        for i, (chunk, ) in enumerate(s.stream()):
            num_frames += chunk.shape[0]
        t1 = time.monotonic()
        print(chunk.dtype, chunk.shape, chunk.device)
        print(time.monotonic() - t0, num_frames, flush=True)
```

</details>

```
./NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4
torch.uint8 torch.Size([5, 3, 540, 960]) cuda:0
12.890056837990414 6175
torch.uint8 torch.Size([5, 3, 540, 960]) cpu
10.697489063022658 6175
torch.uint8 torch.Size([5, 3, 540, 960]) cpu
85.19899423001334 6175

https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4.mp4
torch.uint8 torch.Size([5, 3, 540, 960]) cuda:0
10.712715593050234 6175
torch.uint8 torch.Size([5, 3, 540, 960]) cpu
11.030170071986504 6175
torch.uint8 torch.Size([5, 3, 540, 960]) cpu
84.8515750519582 6175
```

Pull Request resolved: https://github.com/pytorch/audio/pull/2331

Reviewed By: hwangjeff

Differential Revision: D36217169

Pulled By: mthrok

fbshipit-source-id: 7979570b083cfc238ad4735b44305d8649f0607b

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2371

Reviewed By: xiaohui-zhang

Differential Revision: D36246167

Pulled By: carolineechen

fbshipit-source-id: 23042a1c393711864a18c9815d248c18d1d258b4

Summary:
Cleanup old version of cuda115 and other legacy versions

Pull Request resolved: https://github.com/pytorch/audio/pull/2374

Reviewed By: nateanl, mthrok

Differential Revision: D36250955

Pulled By: atalman

fbshipit-source-id: 6b7f0e2926eeb688991c939901c980428cf8e7ef

Summary:
This commit changes the way torchaudio binary distributions are built.

* For all the binary distributions (conda/pip on Linux/macOS/Windnows), build custom FFmpeg libraries.
* The custom FFmpeg libraries do not use `--use-gpl` nor `--use-nonfree`, so that they stay LGPL.
* The custom FFmpeg libraries employ rpath so that the torchaudio binary distributions look for the corresponding FFmpeg libraries installed in the runtime environment.
* The torchaudio binary build process will use them to bootstrap its build process.
* The custom FFmpeg libraries are NOT shipped.

This commit also add disclaimer about FFmpeg in README.

Pull Request resolved: https://github.com/pytorch/audio/pull/2355

Reviewed By: nateanl

Differential Revision: D36202087

Pulled By: mthrok

fbshipit-source-id: c30e5222ba190106c897e42f567cac9152dbd8ef

Summary:
The smoke test jobs simply perform `import torchaudio` to check
if the package artifacts are sane.

Originally, the CI was executing it in the root directory.
This was fine unless the source code is checked out.
When source code is checked out, performing `import torchaudio` in
root directory would import source torchaudio directory, instead of the
installed package.

This error is difficult to notice, so this commit introduces common script to
perform the smoke test, while moving out of root directory.

Pull Request resolved: https://github.com/pytorch/audio/pull/2365

Reviewed By: carolineechen

Differential Revision: D36202069

Pulled By: mthrok

fbshipit-source-id: 4396f85fec5c54869ada4c08f51304539f1b05cf

Summary:
Currently smoke tests are only executed on nightly jobs.
This is inconvenient as PRs that changes build process do not get
the signal naturally.

This commit changes it by always executing smoke tests.

Pull Request resolved: https://github.com/pytorch/audio/pull/2364

Reviewed By: atalman

Differential Revision: D36171267

Pulled By: mthrok

fbshipit-source-id: e549965ba139b5992177b7a094d87c9ef4432a7f

Summary:
This PR fixes Windows Smoke tests

Tested via  circleci :
https://app.circleci.com/pipelines/github/pytorch/audio/10572/workflows/970fd791-25cc-4af4-8183-a7835e1891bf/jobs/637607

Pull Request resolved: https://github.com/pytorch/audio/pull/2361

Reviewed By: nateanl, mthrok

Differential Revision: D36167317

Pulled By: atalman

fbshipit-source-id: 1418ebffd74614cc1110dc032d16ee9502a7d571

Summary:
libmad integration should be enabled only from source-build

Pull Request resolved: https://github.com/pytorch/audio/pull/2354

Reviewed By: nateanl

Differential Revision: D36012035

Pulled By: mthrok

fbshipit-source-id: adeda8cbfd418f96245909cae6862b648a6915a7

Summary:
Fix audio win smoke test to use GPU hosts for CUDA builds

Pull Request resolved: https://github.com/pytorch/audio/pull/2353

Reviewed By: mthrok

Differential Revision: D36006928

Pulled By: atalman

fbshipit-source-id: a27c4cc34093810c8cc08e01188e09b474478001

Summary:
This PR amends `RNNTBeamSearch`'s streaming decoding method to correctly unsqueeze `length` when its dimension is 0.

Original comment: Is "input.dim() == 0" unreachable as it could only be 2 or 3 in assertion of Line 329?

Pull Request resolved: https://github.com/pytorch/audio/pull/2344

Reviewed By: carolineechen, nateanl

Differential Revision: D35899740

Pulled By: hwangjeff

fbshipit-source-id: 84c1692b8cc9e5d35798d87f4a1bd052d94af9fb

Summary:
Add support for lexicon free decoding based on [fairseq's](https://github.com/pytorch/fairseq/blob/main/examples/speech_recognition/new/decoders/flashlight_decoder.py#L53) implementation. Reached numerical parity with fairseq's decoder in offline experimentation

Follow ups
- Add pretrained LM support for lex free decoding
- Add example in tutorial
- Replace flashlight C++ source code with flashlight text submodule
- [optional] fairseq compatibility test

Pull Request resolved: https://github.com/pytorch/audio/pull/2342

Reviewed By: nateanl

Differential Revision: D35856104

Pulled By: carolineechen

fbshipit-source-id: b64286550984df906ebb747e82f6fb1f21948ac7

Summary:
In different pre-training and fine-tuning settings, the `mask_prob`, `mask_channel_prob`, and `mask_channel_length` are different. For example, the settings in [pre-training](https://github.com/pytorch/fairseq/blob/main/examples/hubert/config/pretrain/hubert_base_librispeech.yaml#L70) and [fine-tuning](https://github.com/pytorch/fairseq/blob/main/examples/hubert/config/finetune/base_10h.yaml#L69-L73) are different. The motivation is to avoid overfitting when fine-tuning on a small dataset (example: [fine-tune on 10 minutes of audio](https://github.com/pytorch/fairseq/blob/main/examples/wav2vec/config/finetuning/vox_10m.yaml#L57-L59)).
This PR adds the required arguments in the factory functions to make them tunable for pre-training and fine-tuning. `mask_length` is set to `10` by default for all cases, hence it's not included in the factory function.

Pull Request resolved: https://github.com/pytorch/audio/pull/2345

Reviewed By: carolineechen, xiaohui-zhang

Differential Revision: D35845117

Pulled By: nateanl

fbshipit-source-id: 0cbb74d09535d189b8258aa8ee0f88779bdb77e7

Summary:
fix false shape

Pull Request resolved: https://github.com/pytorch/audio/pull/2347

Reviewed By: carolineechen

Differential Revision: D35921047

Pulled By: nateanl

fbshipit-source-id: 5b58820ee777920c68f13a15d80cd2bcc931af87

Summary:
The `LibriMix` dataset is missing on the [documentation webpage](https://pytorch.org/audio/stable/datasets.html).

Pull Request resolved: https://github.com/pytorch/audio/pull/2351

Reviewed By: carolineechen

Differential Revision: D35926695

Pulled By: nateanl

fbshipit-source-id: 168aed3bb15510d1b1ec57d77727932e481aca48

Summary:
Fix for torchaudio windows tests
Following is an example of such test failing:
https://app.circleci.com/pipelines/github/pytorch/audio/9408/workflows/e6e5a05c-7080-4fdc-b478-2182aed5f234/jobs/531612

The following code is failing:
`conda install -v -y $(ls ~/workspace/torchaudio*.tar.bz2)`

This is because the install package is generated in the following directory:
`/workspace/conda-bld/win-64/`

Pull Request resolved: https://github.com/pytorch/audio/pull/2350

Reviewed By: mthrok

Differential Revision: D35912424

Pulled By: atalman

fbshipit-source-id: fc4f66ffca24061cc768a5f1010b448f065b9410

Summary:
Fix python 3.10 smoke tests

Pull Request resolved: https://github.com/pytorch/audio/pull/2348

Reviewed By: mthrok

Differential Revision: D35906343

Pulled By: atalman

fbshipit-source-id: 6dbb39e69c9751da4b86d5da38a6d11816d527c5

Summary:
Cuda 11.5 remove since we introduced cuda 11.6

Pull Request resolved: https://github.com/pytorch/audio/pull/2346

Reviewed By: mthrok

Differential Revision: D35856758

Pulled By: atalman

fbshipit-source-id: d3c0cf7639fd20f9ccc52c0738f247b8598f1ed7

Summary:
Same change as done in this vision [PR](https://github.com/pytorch/vision/pull/5802)

As Ubuntu-1604 runners will no longer be available in early May
Update ubuntu-1604-cuda-10.1:201909-23 to ubuntu-2004-cuda-11.4:202110-01
Per [CircleCI Configuration reference](https://circleci.com/docs/2.0/configuration-reference/)

Resolves https://github.com/pytorch/audio/issues/2279

Pull Request resolved: https://github.com/pytorch/audio/pull/2343

Reviewed By: mthrok

Differential Revision: D35844880

Pulled By: atalman

fbshipit-source-id: 318a9fa42455e55664f3da6ab67625cb969f72e6

Summary:
When using customized `batch_sampler`, pytorch_lightning can't wrap the distributed sampler onto it. Hence we provide a `DistributedBatchSampler` that supports `BucketizeBatchSampler` in `ddp` mode.

The `DistributedBatchSampler` assumes `BucketizeBatchSampler.iter_list` is a list of lists, where each sub-list contains a batch of indices. Setting `shuffle` to `True` will shuffle the lists based on `seed` and current `epoch`.

The `shuffle` only happens in the initialization, and won't be changed if user don't reset it. The reason is shuffling `BucketizeBatchSampler` may have a different length than before, do shuffling in ``__iter__`` may result in mismatch between ``__len__`` and the real length value.
Hence users need to set `reload_dataloaders_every_n_epochs=1` in pytorch_lightning's Trainer. Then the value of ``__len__``  and the real length is the same.

Pull Request resolved: https://github.com/pytorch/audio/pull/2299

Reviewed By: hwangjeff

Differential Revision: D35781538

Pulled By: nateanl

fbshipit-source-id: 6e8396615497f1aeddab1ee5678830c0445c2b2a

Summary:
CUDA 11.6 for TorchAudio

Pull Request resolved: https://github.com/pytorch/audio/pull/2328

Reviewed By: mthrok

Differential Revision: D35826414

Pulled By: atalman

fbshipit-source-id: 0a471f0566286d69c0c73191aea7fd5ac0647e5f

Summary:
PyTorch Lite, which is becoming a standard for mobile PyTorch usage, does not support containers containing custom classes. Consequently, because TorchAudio's RNN-T decoder currently returns and accepts lists of `Hypothesis` namedtuples, it is not compatible with PyTorch Lite. This PR resolves said incompatibility by changing the underlying implementation of `Hypothesis` to tuple.

Pull Request resolved: https://github.com/pytorch/audio/pull/2339

Reviewed By: nateanl

Differential Revision: D35806529

Pulled By: hwangjeff

fbshipit-source-id: 9cbae5504722390511d35e7f9966af2519ccede5

Summary:
Introduces prototype of convolution-augmented Emformer layer. At a high level, it incorporates Conformer's macaron feedforward network structure and convolution module with Emformer.

Pull Request resolved: https://github.com/pytorch/audio/pull/2324

Reviewed By: mthrok

Differential Revision: D35734252

Pulled By: hwangjeff

fbshipit-source-id: c7ea0bdcfe53a948b00881a74f1f1e1928f5ac57

Summary:
implementation adapted from [s3prl](https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/quesst14_dtw/dataset.py)

modifying the s3prl downstream expert to [this](https://github.com/carolineechen/s3prl/commit/adc91a53d581a604f495f3795a865d84aa17f1a5) using this dataset implementation produces the same results as using the original s3prl pipeline

Pull Request resolved: https://github.com/pytorch/audio/pull/2290

Reviewed By: nateanl

Differential Revision: D35692551

Pulled By: carolineechen

fbshipit-source-id: 035ad161d4cbbd2072411cfdf89984b73a89868c