GitLab

Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/2256



Limits scope of unittesting to one python version for both macOS and
Windows. These types of workflows are particularly expensive and take a
long time so running them on every PR / every push is a bit wasteful
considering the value in signal between different python versions is
probably negligible.
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

Test Plan: Imported from OSS

Reviewed By: mthrok

Differential Revision: D34459626

Pulled By: seemethere

fbshipit-source-id: 47f5c317027f1b395edf9c1720b1b33ba689cad5

Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/2259



We're deprecating support for CUDA 11.1 binaries since CUDA 11.3 should
be forwards compatible with CUDA 11.1 drivers
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

Test Plan: Imported from OSS

Reviewed By: atalman

Differential Revision: D34458400

Pulled By: seemethere

fbshipit-source-id: 105d96a9a175a94d85ffe6e9abcce3c77163a72f

Summary:
Adding py3.10 to audio

Pull Request resolved: https://github.com/pytorch/audio/pull/2224

Reviewed By: malfet, atalman, mthrok

Differential Revision: D34442377

Pulled By: seemethere

fbshipit-source-id: 2656de73427063958d609a74c01b526a476cb06a

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2190

Reviewed By: mthrok

Differential Revision: D33930129

Pulled By: malfet

fbshipit-source-id: ddcbe79f6bdd3dc9b18c1dc337014142877b844b

Summary:
Also, retire cuda-10.2

Pull Request resolved: https://github.com/pytorch/audio/pull/2186

Reviewed By: mthrok

Differential Revision: D33917595

Pulled By: malfet

fbshipit-source-id: 060d3fa706279fe45ffd1f4f99e5727520612d56

Summary:
s/gpu.small/gpu.nvidia.medium/

Pull Request resolved: https://github.com/pytorch/audio/pull/1791

Reviewed By: mthrok

Differential Revision: D33697984

Pulled By: malfet

fbshipit-source-id: 0aacad6d4badf023753fa874c8b80c7f65170d0d

Summary:
Currently, the doc build job uses `pytorch/manylinux-cuda100` as base environment image.
This PR changes that with `python:3.X`.

The problem with the previous one is
- The image is unnecessarily huge with tools not needed for build doc. (+3GB)
- No easy way to install ffmpeg>=4.1.

https://518849-90321822-gh.circle-artifacts.com/0/docs/index.html

Pull Request resolved: https://github.com/pytorch/audio/pull/2151

Reviewed By: carolineechen

Differential Revision: D33585043

Pulled By: mthrok

fbshipit-source-id: d6d2f6ab33511b8f5c7ca358bc6545e253c1b752

Summary:
This commit enables ffmpeg-feature build in tests and
binary builds of all platforms.
(Linux/macOS/Windows x conda/wheel)

It also moves the definition of BUILD_FFMPEG env vars to the
top level `config.yml`.

 ---
Manual checking if all the build log contains `libtorchaudio_ffmpeg`.
### binary build
- [x] `binary_linux_conda_py3.7_cpu`
- [x] `binary_linux_conda_py3.7_cu102`
- [x] `binary_linux_wheel_py3.7_cpu`
- [x] `binary_linux_wheel_py3.7_cu102`
- [x] `binary_macos_conda_py3.7_cpu`
- [x] `binary_macos_wheel_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cpu`
- [x] `binary_windows_conda_py3.7_cu113`
- [x] `binary_windows_wheel_py3.7_cpu`
- [x] `binary_windows_wheel_py3.7_cu113`

### test
- [x] `unittest_linux_cpu_py3.7`
- [x] `unittest_linux_gpu_py3.7`
- [x] `unittest_macos_cpu_py3.7`
- [x] `unittest_windows_cpu_py3.7`
- [x] `unittest_windows_gpu_py3.7`
- [x] `integration test`

Pull Request resolved: https://github.com/pytorch/audio/pull/2140

Reviewed By: hwangjeff

Differential Revision: D33464430

Pulled By: mthrok

fbshipit-source-id: 2c5b72be75d49019bf1599036180d4e56074e46b

Summary:
Update the internal of `skipIfXXX` decorators so that tests in CI will not be automatically skipped.

Currently we automatically skip some tests based on the availability of related features/test tools.
This causes issues where we miss signals on certain important features. (CUDA on Windows) https://github.com/pytorch/audio/issues/1565

The new `skipIf` decorator will fail if in CI unless it is explicitly allowed to skip tests.
It does so by checking `CI` and `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` environment variables.

For non-CI environments, the behavior is same as before, but users can now set `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX=false` to disallow the automatic skip.

Results without `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` https://app.circleci.com/pipelines/github/pytorch/audio/9112/workflows/4e6db046-a1a2-4965-b0fe-d5baf4a1efac

Pull Request resolved: https://github.com/pytorch/audio/pull/2127

Reviewed By: hwangjeff

Differential Revision: D33430711

Pulled By: mthrok

fbshipit-source-id: d8954dd720469c5ab0f34ea062fd8cf04a8afa3e

Summary:
This change adds a minimal ffmpeg installation step to the build wheel job so that later, we can use the resulting ffmpeg libraries for building torchaudio's ffmpeg-features.

The linux wheel build jobs run in CentOS 8 based environment, which does not provide an easy way to install ffmpeg without conda.

After https://github.com/pytorch/audio/pull/2124 is merged, then we can enable the ffmpeg-feature build in Linux wheel.

Pull Request resolved: https://github.com/pytorch/audio/pull/2137

Reviewed By: carolineechen

Differential Revision: D33430032

Pulled By: mthrok

fbshipit-source-id: bf946d394c0718ddbdc679d7970befc3221982b9

Summary:
This commit enable ffmpeg-feature build on wheel-based binary distribution on macOS.

For macOS, since the underlying Python env is Conda, it installs `ffmpeg` from conda.

Depends on https://github.com/pytorch/audio/issues/1873

Pull Request resolved: https://github.com/pytorch/audio/pull/2122

Reviewed By: carolineechen, nateanl

Differential Revision: D33409113

Pulled By: mthrok

fbshipit-source-id: a73839087548010353422109b33e89e262c12a57

Summary:
Currently, macOS CI jobs install `pkg-config` and `wget` with `brew`.
This is problematic as brew takes a long time with auto-update, and disabling the auto-update is not an ideal solution.
Conda also distributes these packages, so switching to conda.

Example issues with brew installation.
https://app.circleci.com/pipelines/github/pytorch/audio/7825/workflows/53965bcf-6ddf-4e42-ad52-83fd1bbab717

This commit removes the use of `brew` by
1. Replacing the use of `wget` with `curl` (pre-installed in most distro)
2. Install `pkg-condig` from conda.
    Note: All the macOS jobs, including binary build jobs, uses conda. Using `pkg-config` from Conda makes it easy to discover the packages installed from conda. (like `ffmpeg` in https://github.com/pytorch/audio/issues/2122)
3. Add `pkg-config` to conda build-time dependency
4. Make sure that the availability of `pkg-config` is explicitly checked when `sox` is being configured. (otherwise, it will fail at somewhere in the middle of build process with somewhat unintuitve error message)

Pull Request resolved: https://github.com/pytorch/audio/pull/1873

Reviewed By: carolineechen, nateanl

Differential Revision: D33404975

Pulled By: mthrok

fbshipit-source-id: ae512d3a3a422ebfe3b46c492bed44deecc36e72

Summary:
As per item 2 on [issue 2051](https://github.com/pytorch/audio/issues/2051), dropping support for python 3.6.
Removed 3.6 from test matrix and ran `.circleci/regenerate.py `.

Pull Request resolved: https://github.com/pytorch/audio/pull/2119

Reviewed By: mthrok

Differential Revision: D33379542

Pulled By: wernerchao

fbshipit-source-id: 6d0fb51b18c2fa7c8cf4eeee4a7f19c4a5210fac

Summary:
cc mthrok

Pull Request resolved: https://github.com/pytorch/audio/pull/2116

Reviewed By: mthrok

Differential Revision: D33368453

Pulled By: jdsgomes

fbshipit-source-id: 09cf3fe5ed6f771c2f16505633c0e59b0c27453c

Summary:
*Before*
<img width="1094" alt="Screen Shot 2021-12-24 at 12 34 14 PM" src="https://user-images.githubusercontent.com/855818/147367213-b1e539c1-6e06-4e9b-aaf4-0458c502379b.png">

*After*

https://app.circleci.com/pipelines/github/pytorch/audio/8870/workflows/0445f1ac-ad48-412f-8045-2400d0cef4f4/jobs/482060

<img width="1096" alt="Screen Shot 2021-12-24 at 12 33 32 PM" src="https://user-images.githubusercontent.com/855818/147367210-a9b759bb-f992-4dc1-9359-0ec3912b3070.png">

Pull Request resolved: https://github.com/pytorch/audio/pull/2102

Reviewed By: carolineechen

Differential Revision: D33311253

Pulled By: mthrok

fbshipit-source-id: 6944921a8be58a2062b66a7dfd2c7ffe8c0866c3

Summary:
Follow-up of https://github.com/pytorch/audio/issues/2086

The CI job to download the third party code and cache daily has not been properly updated.

Pull Request resolved: https://github.com/pytorch/audio/pull/2095

Reviewed By: hwangjeff

Differential Revision: D33291738

Pulled By: mthrok

fbshipit-source-id: 6fc61f76b35c6f032085eda9d6053eefd2a1e0a9

Summary:
After discussing with Moto Hira, we decided to revert linting exemptions
introduced previously in order to keep the entire audio project as formatted
as possible, to reduce the time we spend on formatting discussion.

Pull Request resolved: https://github.com/pytorch/audio/pull/2087

Reviewed By: mthrok

Differential Revision: D33236949

Pulled By: jdsgomes

fbshipit-source-id: e13079f532c4534d8a168059b0ded6fa375fdecf

Summary:
Previously sox-related third-party source code was archived at
`third_party/sox/archives`.
Recently KenLM-related third-party source code was added and
they are archived at `third_party/archives`.

This PR changes the sox archive location to `third_party/archives`,
so that all the archvies are cached at the same location.

Pull Request resolved: https://github.com/pytorch/audio/pull/2086

Reviewed By: carolineechen

Differential Revision: D33236927

Pulled By: mthrok

fbshipit-source-id: 2f2aa5f4b386fefb46d7c98f7179c04995219f3c

Summary:
cc peterjc123 maxluk nbcsm guyang3532 gunandrose4u smartcat2010 mszhanyi

Pull Request resolved: https://github.com/pytorch/audio/pull/2067

Reviewed By: seemethere

Differential Revision: D33032607

Pulled By: atalman

fbshipit-source-id: a5767e9af27690d3a7ab762ddf30178b3069cd35

Summary:
Following up on [this comment ](https://github.com/pytorch/audio/pull/2056#issuecomment-988356439) I am separating the config changes from the formatting.

cc NicolasHug  mthrok

Pull Request resolved: https://github.com/pytorch/audio/pull/2066

Reviewed By: mthrok

Differential Revision: D32990377

Pulled By: jdsgomes

fbshipit-source-id: 67a6251a51901702ad10ae43c35609a09cbf5c5c

Summary:
The sox_effects test in `concurrent.future.ThreadPoolExecutor` started failing since couple of days. While investigate this, skipping the test.

Pull Request resolved: https://github.com/pytorch/audio/pull/2025

Reviewed By: nateanl

Differential Revision: D32615933

Pulled By: mthrok

fbshipit-source-id: 4f7301c0d3c0d11f687011e42e06d9c87ce4197f

* set cu113 for unittest_windows_gpu

* fix old logic

* Update .circleci/regenerate.py
Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com>

* Limit Windows GPU testing to CUDA-11.3 only

Which is the only CUDA version that planned to be supported on Windows
for the upcoming release

* Move unittests to 11.3 as well

* enable windows cudatests

* add this dir

* minor change

* vs integration

* Update cuda_install.bat

* add logs

* minor change

* minor change

* cp vision conda activate

* mv vc_env_helper.bat

* minor change

* exit if cuda not avaiable

* install numpy

* improt CMakeLists

* check cuda

* minor change

* change windows GPU image from previous to stable

* set libtorch audio suffix as pyd on Windows

* reduce changes

* check env settings

CCI says circleci/python:3.8 is deprecaetd. Migrating to cimg/python:3.8.

> CircleCI’s latest pre-built container images were designed from the ground up to help your team build more reliably.
> Our new images were built specifically for continuous integration projects and they are our most deterministic, performant, and efficient images yet.

> As of Dec 31, 2021, legacy images will no longer be supported on CircleCI.

Related links

- https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic
- https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034
- https://circleci.com/docs/2.0/next-gen-migration-guide

Cherry-pick of
https://github.com/pytorch/vision/commit/d2460a75de237cfef8e5c3415f7bb0ad8467c0e5
into this repo

Fixes https://github.com/pytorch/audio/issues/1750

Downloading CUDA-11.1 from conda-forge takes a while

* Remove some obsolete conditions about CUDA-10.0 from `pkg_helpers.bash`.
* Use `USE_CUDA` instead of `FORCE_CUDA` in `pkg_helpers.bash`
* Do not define `NO_CUDA_PACKAGE` in build_wheel.sh and build_cuda.sh
* Add conda-forge for Win cuda-11.1 builds
* Pass USE_CUDA / TORCH_CUDA_ARCH_LIST to conda build
* Add selected CUDA to path
* Don't define USE_CUDA for ROCM

TODO: Fix Windows CUDA builds

Copy `cuda_install.bat` from
https://github.com/pytorch/vision/blob/3d9d264198e2167030dc04bf453aeca47daa37db/packaging/windows/internal/cuda_install.bat

Add `windows_intall_cuda` step and include it in `binary_windows_conda`
and `binary_windows_wheel` workflows

No longer needed for Python 3.9
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* add rocm binary

use the term cu_version

update pkg_helpers

add docker wheel option

update config yml

add force_cuda option

* update config.yml

* remove conda builds