GitLab

Summary:
Adds unit tests for Emformer RNN-T LibriSpeech recipe. Also makes changes to recipe to resolve errors with pickling lambda functions in Windows.

Pull Request resolved: https://github.com/pytorch/audio/pull/2216

Reviewed By: nateanl

Differential Revision: D34171480

Pulled By: hwangjeff

fbshipit-source-id: 5fcebb457051f3041766324863728411180f5e1e

Summary:
This PR adds the prototype streaming API.
The implementation is based on ffmpeg libraries.

For the detailed usage, please refer to [the resulting tutorial](https://534376-90321822-gh.circle-artifacts.com/0/docs/tutorials/streaming_api_tutorial.html).

Pull Request resolved: https://github.com/pytorch/audio/pull/2164

Reviewed By: hwangjeff

Differential Revision: D33934457

Pulled By: mthrok

fbshipit-source-id: 92ade4aff2d25baf02c0054682d4fbdc9ba8f3fe

Summary:
This fixes:
```
Installed flake8: + flake8 --version
Traceback (most recent call last):
  File "/root/project/env/bin/flake8", line 6, in <module>
    from flake8.main.cli import main
  File "/root/project/env/lib/python3.7/site-packages/flake8/main/cli.py", line 6, in <module>
    from flake8.main import application
  File "/root/project/env/lib/python3.7/site-packages/flake8/main/application.py", line 24, in <module>
    from flake8.plugins import manager as plugin_manager
  File "/root/project/env/lib/python3.7/site-packages/flake8/plugins/manager.py", line 11, in <module>
    from flake8._compat import importlib_metadata
  File "/root/project/env/lib/python3.7/site-packages/flake8/_compat.py", line 7, in <module>
    import importlib_metadata
ModuleNotFoundError: No module named 'importlib_metata'
```

Pull Request resolved: https://github.com/pytorch/audio/pull/2191

Reviewed By: atalman

Differential Revision: D33930583

Pulled By: malfet

fbshipit-source-id: 68026743c29434113893cca38041596135d3bd53

Summary:
This commit enables ffmpeg-feature build in tests and
binary builds of all platforms.
(Linux/macOS/Windows x conda/wheel)

It also moves the definition of BUILD_FFMPEG env vars to the
top level `config.yml`.

 ---
Manual checking if all the build log contains `libtorchaudio_ffmpeg`.
### binary build
- [x] `binary_linux_conda_py3.7_cpu`
- [x] `binary_linux_conda_py3.7_cu102`
- [x] `binary_linux_wheel_py3.7_cpu`
- [x] `binary_linux_wheel_py3.7_cu102`
- [x] `binary_macos_conda_py3.7_cpu`
- [x] `binary_macos_wheel_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cu113`
- [x] `binary_windows_wheel_py3.7_cpu`
- [x] `binary_windows_wheel_py3.7_cu113`

### test
- [x] `unittest_linux_cpu_py3.7`
- [x] `unittest_linux_gpu_py3.7`
- [x] `unittest_macos_cpu_py3.7`
- [x] `unittest_windows_cpu_py3.7`
- [x] `unittest_windows_gpu_py3.7`
- [x] `integration test`

Pull Request resolved: https://github.com/pytorch/audio/pull/2140

Reviewed By: hwangjeff

Differential Revision: D33464430

Pulled By: mthrok

fbshipit-source-id: 2c5b72be75d49019bf1599036180d4e56074e46b

Summary:
Update the internal of `skipIfXXX` decorators so that tests in CI will not be automatically skipped.

Currently we automatically skip some tests based on the availability of related features/test tools.
This causes issues where we miss signals on certain important features. (CUDA on Windows) https://github.com/pytorch/audio/issues/1565

The new `skipIf` decorator will fail if in CI unless it is explicitly allowed to skip tests.
It does so by checking `CI` and `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` environment variables.

For non-CI environments, the behavior is same as before, but users can now set `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX=false` to disallow the automatic skip.

Results without `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` https://app.circleci.com/pipelines/github/pytorch/audio/9112/workflows/4e6db046-a1a2-4965-b0fe-d5baf4a1efac

Pull Request resolved: https://github.com/pytorch/audio/pull/2127

Reviewed By: hwangjeff

Differential Revision: D33430711

Pulled By: mthrok

fbshipit-source-id: d8954dd720469c5ab0f34ea062fd8cf04a8afa3e

Summary:
Currently, macOS CI jobs install `pkg-config` and `wget` with `brew`.
This is problematic as brew takes a long time with auto-update, and disabling the auto-update is not an ideal solution.
Conda also distributes these packages, so switching to conda.

Example issues with brew installation.
https://app.circleci.com/pipelines/github/pytorch/audio/7825/workflows/53965bcf-6ddf-4e42-ad52-83fd1bbab717

This commit removes the use of `brew` by
1. Replacing the use of `wget` with `curl` (pre-installed in most distro)
2. Install `pkg-condig` from conda.
    Note: All the macOS jobs, including binary build jobs, uses conda. Using `pkg-config` from Conda makes it easy to discover the packages installed from conda. (like `ffmpeg` in https://github.com/pytorch/audio/issues/2122)
3. Add `pkg-config` to conda build-time dependency
4. Make sure that the availability of `pkg-config` is explicitly checked when `sox` is being configured. (otherwise, it will fail at somewhere in the middle of build process with somewhat unintuitve error message)

Pull Request resolved: https://github.com/pytorch/audio/pull/1873

Reviewed By: carolineechen, nateanl

Differential Revision: D33404975

Pulled By: mthrok

fbshipit-source-id: ae512d3a3a422ebfe3b46c492bed44deecc36e72

Summary:
Preparation to land Python front-end of ffmpeg-related features.

- Set BUILD_FFMPEG=1 in Linux/macOS unit test jobs
- Install ffmpeg and pkg-config from conda-forge
- Add note about Windows build process
- Temporarily avoid `av_err2str`

Pull Request resolved: https://github.com/pytorch/audio/pull/2114

Reviewed By: hwangjeff

Differential Revision: D33371346

Pulled By: mthrok

fbshipit-source-id: b0e16a35959a49a2166109068f3e0cbbb836e888

Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/2096

run: `arc lint --apply-patches --paths-cmd 'hg files -I "./**/*.py"'`

Reviewed By: mthrok

Differential Revision: D33297351

fbshipit-source-id: 7bf5956edf0717c5ca90219f72414ff4eeaf5aa8

Summary:
The unit test failures seems to be caused by [conda 4.11](https://github.com/conda/conda/issues/11096)
Remove conda update line fixes the issue.

Pull Request resolved: https://github.com/pytorch/audio/pull/2069

Reviewed By: carolineechen

Differential Revision: D33023851

Pulled By: nateanl

fbshipit-source-id: 73246189d4ccc541e366a5367f532a5b456af8f8

Summary:
Similar to https://github.com/pytorch/vision/pull/4788
Make sure the workflow could download right PyTorch with cpu or cuda in case nightly build wasn't ready at that day.

https://app.circleci.com/pipelines/github/pytorch/audio/8427/workflows/11a80738-bcdd-45e3-b37f-328be36c60ee/jobs/438285?invite=true#step-107-542

![image](https://user-images.githubusercontent.com/16190118/142969390-142df7ec-6040-40c1-9a02-17d43f5de05e.png)

Pull Request resolved: https://github.com/pytorch/audio/pull/2026

Reviewed By: hwangjeff, nateanl

Differential Revision: D32634926

Pulled By: mthrok

fbshipit-source-id: 30d6349a0a2ce174b789a5888b1c8e0544a23a37

This commit changes the build process so that the third party code are fetched automatically when `setup.py` is invoked.

This is to allow
1. `build_sdist` to contain the third party library codes, so that source code distribution created at the time release tag is created is complete.
2. It makes it possible to do `pip install https://github.com/pytorch/audio.git`.
    Example:
    ```
    !pip install 'cmake>=3.18' ninja
    !pip install --verbose git+https://github.com/pytorch/audio.git@auto-source
    ```

* Default to BUILD_SOX=1 in non-Windows systems

Since the adaptation of CMake and restricting to the static linking of libsox,
the build process has become much robust with libsox integration enabled.

This commit makes it default behavior to build libsox integration in non-Windows systems.
The build process still checks BUILD_SOX env var so, setting `BUILD_SOX=0` disables it.

* Pin MKL to 2021.2.0

Fixes https://github.com/pytorch/pytorch/issues/62424

* Apply the same  constraint for conda builds

Use less-than-equals constraint to avoid dependency conflicts for
python-3.6 environment

There are some issues on CMake-based integration after 2021-06-19.

```
  Imported target "torch" includes non-existent path
    "/opt/conda/conda-bld/pytorch_1624259172741/work/torch/lib"
  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
  * The path was deleted, renamed, or moved to another location.
  * An install or uninstall procedure did not complete successfully.
  * The installation package was faulty and references files it does not provide.
```

This is being worked on PR like https://github.com/pytorch/pytorch/pull/60403.
The nightly builds are still causing the error, so as a workaround, 
we pin the nightly build version for Linux/macOS unittest.
The issue still happens on Windows unittest and binary builds.

This code was added as a part of investigation of inconsistent behavior
of file-like object support. Now the investigation is done and this code
does not provide much insight yet adding a bunch of lines to the log, so
this PR removes it.

Numpy is a dependency of pytorch so we should let it get resolved as it
install pytorch
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* Switch to cmake for build
* Hide symbols

So that when a test fails in flaky manner, we can look into the difference.

* Do not install sox system-wide, so that it will never get mixed up with static one torchaudio builds
* Do not install cmake system-wide
* Move libsndfile installation to Docker image

This reverts commit 84966ae1.