GitLab

Summary:
Currently, macOS CI jobs install `pkg-config` and `wget` with `brew`.
This is problematic as brew takes a long time with auto-update, and disabling the auto-update is not an ideal solution.
Conda also distributes these packages, so switching to conda.

Example issues with brew installation.
https://app.circleci.com/pipelines/github/pytorch/audio/7825/workflows/53965bcf-6ddf-4e42-ad52-83fd1bbab717

This commit removes the use of `brew` by
1. Replacing the use of `wget` with `curl` (pre-installed in most distro)
2. Install `pkg-condig` from conda.
    Note: All the macOS jobs, including binary build jobs, uses conda. Using `pkg-config` from Conda makes it easy to discover the packages installed from conda. (like `ffmpeg` in https://github.com/pytorch/audio/issues/2122)
3. Add `pkg-config` to conda build-time dependency
4. Make sure that the availability of `pkg-config` is explicitly checked when `sox` is being configured. (otherwise, it will fail at somewhere in the middle of build process with somewhat unintuitve error message)

Pull Request resolved: https://github.com/pytorch/audio/pull/1873

Reviewed By: carolineechen, nateanl

Differential Revision: D33404975

Pulled By: mthrok

fbshipit-source-id: ae512d3a3a422ebfe3b46c492bed44deecc36e72

Summary:
cc peterjc123 maxluk nbcsm guyang3532 gunandrose4u smartcat2010 mszhanyi

Pull Request resolved: https://github.com/pytorch/audio/pull/2067

Reviewed By: seemethere

Differential Revision: D33032607

Pulled By: atalman

fbshipit-source-id: a5767e9af27690d3a7ab762ddf30178b3069cd35

It seems that this is no longer necessary for recent macOS .

This is follow up after https://github.com/pytorch/builder/pull/823 that gets rids of `feature` and migrate it to `run_constrained`

* Remove some obsolete conditions about CUDA-10.0 from `pkg_helpers.bash`.
* Use `USE_CUDA` instead of `FORCE_CUDA` in `pkg_helpers.bash`
* Do not define `NO_CUDA_PACKAGE` in build_wheel.sh and build_cuda.sh
* Add conda-forge for Win cuda-11.1 builds
* Pass USE_CUDA / TORCH_CUDA_ARCH_LIST to conda build
* Add selected CUDA to path
* Don't define USE_CUDA for ROCM

TODO: Fix Windows CUDA builds

* Pin MKL to 2021.2.0

Fixes https://github.com/pytorch/pytorch/issues/62424

* Apply the same  constraint for conda builds

Use less-than-equals constraint to avoid dependency conflicts for
python-3.6 environment

No longer needed for Python 3.9
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* add rocm binary

use the term cu_version

update pkg_helpers

add docker wheel option

update config yml

add force_cuda option

* update config.yml

* remove conda builds

* Add CUDA binary builds

* Add "cuda_version": "cpu" to doc build jobs

* Add required cu_versions to pkg_helpers

* Regenerate config.yml

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

torchaudio can work with any version of pytorch so let's strip the
pytorch version suffix to denote that
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

Default to using the `cpu` directory since the base one is not reliable
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* packaging: Add test channels to pytorch dependency resolution
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* .circleci: Add test channel to smoke tests
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* .circleci: Put pytorch-test into a higher priority

pytorch-nightly was getting prioritized over pytorch-nightly which
shouldn't be the case
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* Add windows binary jobs

* Make wheel package platform-specific and python-version-specific

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

CUDA_SUFFIX was still being used here when it should've been swapped out
for PYTORCH_VERSION_SUFFIX, which is what's being used for conda below.
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
(cherry picked from commit 009b115d074ac5fcca2cc34662fe814df63324c1)
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* Refactor version suffix so conda packages don't get suffixes.
Signed-off-by: Edward Z. Yang <ezyang@fb.com>

* typofix
Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

Also bugfixes and updates to the packaging scripts.

* New entry points are packaging/build_wheel.sh and packaging/build_conda.sh. The only mandatory environment variable you have to set is PYTHON_VERSION
* CircleCI configuration uses 2.1-style parametrized builds to let you toggle python version, etc. as you do builds. We create a separate job per build configuration for maximum parallelism
* build_tools/packaging got moved to packaging, to be in-line with directory structure in torchvision
* The build_conda.sh and build_wheel.sh delegate most of the heavy lifting to pkg_helpers.bash, which defines a number of bash functions for performing common operations. The intent is that I'll copy-paste this file between other domain API projects.
* TORCHAUDIO_ prefix removed from envvars, so that I can more easily share packaging scripts between projects
* BUILD_VERSION is completely gone; just change the version number if you need to rebuild
* No more logic for cloning and checking out a fresh copy of torchaudio