GitLab

Summary:
This commit enable ffmpeg-feature build on wheel-based binary distribution on macOS.

For macOS, since the underlying Python env is Conda, it installs `ffmpeg` from conda.

Depends on https://github.com/pytorch/audio/issues/1873

Pull Request resolved: https://github.com/pytorch/audio/pull/2122

Reviewed By: carolineechen, nateanl

Differential Revision: D33409113

Pulled By: mthrok

fbshipit-source-id: a73839087548010353422109b33e89e262c12a57

Summary:
Currently, macOS CI jobs install `pkg-config` and `wget` with `brew`.
This is problematic as brew takes a long time with auto-update, and disabling the auto-update is not an ideal solution.
Conda also distributes these packages, so switching to conda.

Example issues with brew installation.
https://app.circleci.com/pipelines/github/pytorch/audio/7825/workflows/53965bcf-6ddf-4e42-ad52-83fd1bbab717

This commit removes the use of `brew` by
1. Replacing the use of `wget` with `curl` (pre-installed in most distro)
2. Install `pkg-condig` from conda.
    Note: All the macOS jobs, including binary build jobs, uses conda. Using `pkg-config` from Conda makes it easy to discover the packages installed from conda. (like `ffmpeg` in https://github.com/pytorch/audio/issues/2122)
3. Add `pkg-config` to conda build-time dependency
4. Make sure that the availability of `pkg-config` is explicitly checked when `sox` is being configured. (otherwise, it will fail at somewhere in the middle of build process with somewhat unintuitve error message)

Pull Request resolved: https://github.com/pytorch/audio/pull/1873

Reviewed By: carolineechen, nateanl

Differential Revision: D33404975

Pulled By: mthrok

fbshipit-source-id: ae512d3a3a422ebfe3b46c492bed44deecc36e72

Summary:
As per item 2 on [issue 2051](https://github.com/pytorch/audio/issues/2051), dropping support for python 3.6.
Removed 3.6 from test matrix and ran `.circleci/regenerate.py `.

Pull Request resolved: https://github.com/pytorch/audio/pull/2119

Reviewed By: mthrok

Differential Revision: D33379542

Pulled By: wernerchao

fbshipit-source-id: 6d0fb51b18c2fa7c8cf4eeee4a7f19c4a5210fac

Summary:
cc mthrok

Pull Request resolved: https://github.com/pytorch/audio/pull/2116

Reviewed By: mthrok

Differential Revision: D33368453

Pulled By: jdsgomes

fbshipit-source-id: 09cf3fe5ed6f771c2f16505633c0e59b0c27453c

Summary:
*Before*
<img width="1094" alt="Screen Shot 2021-12-24 at 12 34 14 PM" src="https://user-images.githubusercontent.com/855818/147367213-b1e539c1-6e06-4e9b-aaf4-0458c502379b.png">

*After*

https://app.circleci.com/pipelines/github/pytorch/audio/8870/workflows/0445f1ac-ad48-412f-8045-2400d0cef4f4/jobs/482060

<img width="1096" alt="Screen Shot 2021-12-24 at 12 33 32 PM" src="https://user-images.githubusercontent.com/855818/147367210-a9b759bb-f992-4dc1-9359-0ec3912b3070.png">

Pull Request resolved: https://github.com/pytorch/audio/pull/2102

Reviewed By: carolineechen

Differential Revision: D33311253

Pulled By: mthrok

fbshipit-source-id: 6944921a8be58a2062b66a7dfd2c7ffe8c0866c3

Summary:
Follow-up of https://github.com/pytorch/audio/issues/2086

The CI job to download the third party code and cache daily has not been properly updated.

Pull Request resolved: https://github.com/pytorch/audio/pull/2095

Reviewed By: hwangjeff

Differential Revision: D33291738

Pulled By: mthrok

fbshipit-source-id: 6fc61f76b35c6f032085eda9d6053eefd2a1e0a9

Summary:
After discussing with Moto Hira, we decided to revert linting exemptions
introduced previously in order to keep the entire audio project as formatted
as possible, to reduce the time we spend on formatting discussion.

Pull Request resolved: https://github.com/pytorch/audio/pull/2087

Reviewed By: mthrok

Differential Revision: D33236949

Pulled By: jdsgomes

fbshipit-source-id: e13079f532c4534d8a168059b0ded6fa375fdecf

Summary:
Previously sox-related third-party source code was archived at
`third_party/sox/archives`.
Recently KenLM-related third-party source code was added and
they are archived at `third_party/archives`.

This PR changes the sox archive location to `third_party/archives`,
so that all the archvies are cached at the same location.

Pull Request resolved: https://github.com/pytorch/audio/pull/2086

Reviewed By: carolineechen

Differential Revision: D33236927

Pulled By: mthrok

fbshipit-source-id: 2f2aa5f4b386fefb46d7c98f7179c04995219f3c

Summary:
cc peterjc123 maxluk nbcsm guyang3532 gunandrose4u smartcat2010 mszhanyi

Pull Request resolved: https://github.com/pytorch/audio/pull/2067

Reviewed By: seemethere

Differential Revision: D33032607

Pulled By: atalman

fbshipit-source-id: a5767e9af27690d3a7ab762ddf30178b3069cd35

Summary:
Following up on [this comment ](https://github.com/pytorch/audio/pull/2056#issuecomment-988356439) I am separating the config changes from the formatting.

cc NicolasHug  mthrok

Pull Request resolved: https://github.com/pytorch/audio/pull/2066

Reviewed By: mthrok

Differential Revision: D32990377

Pulled By: jdsgomes

fbshipit-source-id: 67a6251a51901702ad10ae43c35609a09cbf5c5c

Summary:
The sox_effects test in `concurrent.future.ThreadPoolExecutor` started failing since couple of days. While investigate this, skipping the test.

Pull Request resolved: https://github.com/pytorch/audio/pull/2025

Reviewed By: nateanl

Differential Revision: D32615933

Pulled By: mthrok

fbshipit-source-id: 4f7301c0d3c0d11f687011e42e06d9c87ce4197f

* set cu113 for unittest_windows_gpu

* fix old logic

* Update .circleci/regenerate.py
Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com>

* Limit Windows GPU testing to CUDA-11.3 only

Which is the only CUDA version that planned to be supported on Windows
for the upcoming release

* Move unittests to 11.3 as well

* enable windows cudatests

* add this dir

* minor change

* vs integration

* Update cuda_install.bat

* add logs

* minor change

* minor change

* cp vision conda activate

* mv vc_env_helper.bat

* minor change

* exit if cuda not avaiable

* install numpy

* improt CMakeLists

* check cuda

* minor change

* change windows GPU image from previous to stable

* set libtorch audio suffix as pyd on Windows

* reduce changes

* check env settings

CCI says circleci/python:3.8 is deprecaetd. Migrating to cimg/python:3.8.

> CircleCI’s latest pre-built container images were designed from the ground up to help your team build more reliably.
> Our new images were built specifically for continuous integration projects and they are our most deterministic, performant, and efficient images yet.

> As of Dec 31, 2021, legacy images will no longer be supported on CircleCI.

Related links

- https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic
- https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034
- https://circleci.com/docs/2.0/next-gen-migration-guide

Cherry-pick of
https://github.com/pytorch/vision/commit/d2460a75de237cfef8e5c3415f7bb0ad8467c0e5
into this repo

Fixes https://github.com/pytorch/audio/issues/1750

Downloading CUDA-11.1 from conda-forge takes a while

* Remove some obsolete conditions about CUDA-10.0 from `pkg_helpers.bash`.
* Use `USE_CUDA` instead of `FORCE_CUDA` in `pkg_helpers.bash`
* Do not define `NO_CUDA_PACKAGE` in build_wheel.sh and build_cuda.sh
* Add conda-forge for Win cuda-11.1 builds
* Pass USE_CUDA / TORCH_CUDA_ARCH_LIST to conda build
* Add selected CUDA to path
* Don't define USE_CUDA for ROCM

TODO: Fix Windows CUDA builds

Copy `cuda_install.bat` from
https://github.com/pytorch/vision/blob/3d9d264198e2167030dc04bf453aeca47daa37db/packaging/windows/internal/cuda_install.bat

Add `windows_intall_cuda` step and include it in `binary_windows_conda`
and `binary_windows_wheel` workflows

No longer needed for Python 3.9
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* add rocm binary

use the term cu_version

update pkg_helpers

add docker wheel option

update config yml

add force_cuda option

* update config.yml

* remove conda builds

* Add CUDA binary builds

* Add "cuda_version": "cpu" to doc build jobs

* Add required cu_versions to pkg_helpers

* Regenerate config.yml

All macOS CI jobs were failing at `brew`.
The issue was the macOS version `10.13`, which is no longer supported by `brew`.
The macOS version is indirectly controlled by `xcode` version as listed [here](https://circleci.com/docs/2.0/testing-ios/#supported-xcode-versions).

Updating the `xcode` to `"12.0"` as in PyTorch core bump macOS version to `10.15` and it resolves the issue.
`Python 3.9` is still failing but it's packaging issue on upstream, which needs to be resolved separately.

Should be moved back to stable once
https://github.com/pytorch/pytorch/issues/56654 is closed

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* Switch to cmake for build
* Hide symbols

* Do not install sox system-wide, so that it will never get mixed up with static one torchaudio builds
* Do not install cmake system-wide
* Move libsndfile installation to Docker image