GitLab

Extracted from #3604

Add Wall helper class and C++ unit test

Summary:
Reintroduce a conda environment within which we will do all deps installation, audio builds, and tests runs. This conda environment will use the python version set by the GHA job - previously this just defaulted to using the system 3.10 python which was default inside the container.

Pull Request resolved: https://github.com/pytorch/audio/pull/3477

Reviewed By: mthrok

Differential Revision: D47414572

Pulled By: osalpekar

fbshipit-source-id: 80760f82c7726205b29812d576e498db2a7a80a0

Summary:
- Moving the unit test scripts from .circleci to .github
- Remove docker file for unit test base
- Use the Conda from Docker image in Linux jobs.

Remaining follow-up items

- Reuse the unittest script in Linux GPU job like done in Linux CPU job.

The unit test script needs to be fixed to be used for Linux GPU job
in new GHA workflow. Keeping it as a separate follow-up work item.

Pull Request resolved: https://github.com/pytorch/audio/pull/3407

Differential Revision: D46498263

Pulled By: mthrok

fbshipit-source-id: d8256717a55bb4257151d819d3b2ebd453601eac

Summary:
Set the directory of JUnitText XML file to the one where test-infra picks up and put them in summary.

Example: https://github.com/pytorch/audio/actions/runs/5136305988

Pull Request resolved: https://github.com/pytorch/audio/pull/3394

Differential Revision: D46328832

Pulled By: mthrok

fbshipit-source-id: f0b5020a911ca4ec09345a965bdec769300859f0

Summary:
As discussed in the [Torchaudio Migration Proposal](https://docs.google.com/document/d/1PF8biwiGzsjzfEBM78mlLiRrkcsGsvuYkeqkI66Ym8A/edit), this PR moves the Linux CPU unittest job to Nova tooling. Note that this does not disable the existing CircleCI job at the moment.

Passing Job: https://github.com/pytorch/audio/actions/runs/4986115298/jobs/8926499354?pr=3323

Pull Request resolved: https://github.com/pytorch/audio/pull/3323

Reviewed By: atalman, mthrok

Differential Revision: D46113506

Pulled By: osalpekar

fbshipit-source-id: 1778c360e17b9d02c63bcc60100834c75798d380

Summary:
Update the internal of `skipIfXXX` decorators so that tests in CI will not be automatically skipped.

Currently we automatically skip some tests based on the availability of related features/test tools.
This causes issues where we miss signals on certain important features. (CUDA on Windows) https://github.com/pytorch/audio/issues/1565

The new `skipIf` decorator will fail if in CI unless it is explicitly allowed to skip tests.
It does so by checking `CI` and `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` environment variables.

For non-CI environments, the behavior is same as before, but users can now set `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX=false` to disallow the automatic skip.

Results without `TORCHAUDIO_TEST_ALLOW_SKIP_IF_XXX` https://app.circleci.com/pipelines/github/pytorch/audio/9112/workflows/4e6db046-a1a2-4965-b0fe-d5baf4a1efac

Pull Request resolved: https://github.com/pytorch/audio/pull/2127

Reviewed By: hwangjeff

Differential Revision: D33430711

Pulled By: mthrok

fbshipit-source-id: d8954dd720469c5ab0f34ea062fd8cf04a8afa3e

This code was added as a part of investigation of inconsistent behavior
of file-like object support. Now the investigation is done and this code
does not provide much insight yet adding a bunch of lines to the log, so
this PR removes it.

So that when a test fails in flaky manner, we can look into the difference.

This reverts commit 84966ae1.

This reverts commit 1ecbc249.

Reason: The macOS CI jobs are stuck with xdist recently.

We have been running unit test with editable installation. (i.e. `python setup.py develop`), with which we missed issues like #842. 

This CC makes installation in CI non-editable, and change test directory structure so that the source code will not shadow the installed version of `torchaudio`. With simple `pytest test`, `pytest` modifies `sys.path` and prepend checked out repository, which shadows the installed version.

To remedy this, the whole test suites has been moved from `./test` to `./test/torchaudio_unittest`. This adds nice module structure to our test code and we can do absolute import in each test module, which makes it possible again to run test with `python -m unittest torchaudio_unittest/XXX.py`

This change does not affect the regular development process (`python setup.py develop` && `pytest test`)

Currently our test suites automatically/silently skip tests on C++ extension
if it is not available. This is nice in local env, but in CI these tests should be
enforced and reported as failure if C++ extension is not available.

This PR adds switch for making tests fail if C++ extension is not available,
and make CI for *nix fail if that's the case.

Currently macOS CI job takes twice the time of linux unit tests.
This PR uses `pytest-xdist` to run unit tests in parallel on macOS CI job. 

Interestingly applying the same technique to linux unit tests make the test execution longer.
So the change is only applied to macOS.

macOS 27 mins -> 17 mins 😸
linux 14 mins -> 1 hours 😾

* Use cmake for third party

* Apply patch to libmad

* Update gitignore

* Update docker test image

* Separate style check from unit test

* TEST flake8 fail

* TEST clang-format fail

* Revert "TEST flake8 fail"

This reverts commit 88d90185cefe294b9a662129a05c2e26a9844fe4.

* Revert "TEST clang-format fail"

This reverts commit 93517d1ccba650a6ccd813268fd73a4fd63b252c.

* Clean up extension build mechanism and extension location

* Add back the switch to depend on external sox

* Remove print

* Fix

* Turn on tests when building through conda-build

* skip one failling test on windows.

* Switch to unittest

* Fix conda env activation

* add linux only condition

* Fix librosa installation

* conda-forge channel as a fallback

* Revert "conda-forge channel as a fallback"

This reverts commit 8c15f6638e34792640a3de3e057d9c2995425180.

* Try the pytorch way

* Use conda-forge

* Disable conda-forge

* Install future at setup env

* Move linux unittest CI job tools to dedicated directory

* Restore Linux scripts and move the updated souces to a new subdir

* Add back future

* Remove comments
Co-authored-by: Vincent Quenneville-Belair <vincentqb@gmail.com>
Co-authored-by: moto <855818+mthrok@users.noreply.github.com>

* Bust cache weekly

* Rename directory for utility

* Show 20 slowest tests

* fixup! Show 20 slowest tests

* Add test image with Kaldi executable and third_parties

* Use Ubuntu as bese image since pytorch/many-linux is huge

* wip

* Add test

* add caching

* fixup! add caching

* fixup! fixup! add caching