GitLab

Unpin mkl version and install pybind11 to get the windows CI working again

This fixes #3767

The test executable is not found. Need to investigate more later.

Extracted from #3604

Add Wall helper class and C++ unit test

Summary:
The new version of transformers changed the format of pre-trained weight. Fixing it is low-priority for the maintanance team so we disable the test.

See https://github.com/pytorch/audio/issues/3430

Pull Request resolved: https://github.com/pytorch/audio/pull/3431

Differential Revision: D46592883

Pulled By: mthrok

fbshipit-source-id: d8f54a281a92cac60c469c48f95345bcf0e959d6

Summary:
- Moving the unit test scripts from .circleci to .github
- Remove docker file for unit test base
- Use the Conda from Docker image in Linux jobs.

Remaining follow-up items

- Reuse the unittest script in Linux GPU job like done in Linux CPU job.

The unit test script needs to be fixed to be used for Linux GPU job
in new GHA workflow. Keeping it as a separate follow-up work item.

Pull Request resolved: https://github.com/pytorch/audio/pull/3407

Differential Revision: D46498263

Pulled By: mthrok

fbshipit-source-id: d8256717a55bb4257151d819d3b2ebd453601eac

Summary:
Continuing with the job migrations from CCI to Nova, this PR introduces the Windows CPU Unittest job as a Nova workflow.

The job is passing: https://github.com/pytorch/audio/actions/runs/5094569687/jobs/9159020192?pr=3329.

Pull Request resolved: https://github.com/pytorch/audio/pull/3329

Reviewed By: huydhn

Differential Revision: D46265649

Pulled By: atalman

fbshipit-source-id: 7659dfbcc8ad400f2e109ff64530e1f768e82ef9

Summary:
Remove temp channel for python 3.11, simplify logic around cuda

Pull Request resolved: https://github.com/pytorch/audio/pull/3250

Reviewed By: mthrok

Differential Revision: D44788219

Pulled By: atalman

fbshipit-source-id: 421ff9e0bf1818b41e395708cc4589d4a9c865bd

Summary:
As we migrate to use upstream flashlight-text and KenLM, this PR disable building CTC decoder by default.
This will stop shipping flashlight-text and KenLM bundle in torchaudio binary.

Ref: https://github.com/pytorch/audio/issues/3088

cc jacobkahn

Pull Request resolved: https://github.com/pytorch/audio/pull/3232

Reviewed By: hwangjeff

Differential Revision: D44650872

Pulled By: mthrok

fbshipit-source-id: 2415623abaf3cafa181135db5112d3c711137cd7

Summary:
Initial step to migrate to upstream CTC decoder.

https://circleci.com/api/v1.1/project/github/pytorch/audio/1174432/output/107/0?file=true&allocation-id=642636c6eaaa102ce75beb8e-0-build%2FB77C8AB

Pull Request resolved: https://github.com/pytorch/audio/pull/3225

Reviewed By: nateanl

Differential Revision: D44581338

Pulled By: mthrok

fbshipit-source-id: 1517fa0cd5e4ba001d136eb0dfc2a9349afcd2da

Summary:
In librosa 0.10 release, positional arguments are deprecated (see https://github.com/librosa/librosa/pull/1521 for details). The PR fixes the HiFiGAN unit test by using keyword arguments for `librosa.filters.mel` function.

Pull Request resolved: https://github.com/pytorch/audio/pull/3185

Reviewed By: mthrok

Differential Revision: D44218852

Pulled By: nateanl

fbshipit-source-id: 6171f7bec6a2144917697c1d640e701d95ec60d7

Summary:
replicate of https://github.com/pytorch/audio/issues/2644

Pull Request resolved: https://github.com/pytorch/audio/pull/2880

Reviewed By: mthrok

Differential Revision: D41633911

Pulled By: nateanl

fbshipit-source-id: 73cf145d75c389e996aafe96571ab86dc21f86e5

Summary:
CC atalman

Pull Request resolved: https://github.com/pytorch/audio/pull/2951

Reviewed By: mthrok

Differential Revision: D42459205

Pulled By: atalman

fbshipit-source-id: b2d7c5604ba1f3bb4d9a45a052ac41054acd52dd

Summary:
CC atalman

Pull Request resolved: https://github.com/pytorch/audio/pull/2623

Reviewed By: hwangjeff, nateanl

Differential Revision: D39036432

Pulled By: atalman

fbshipit-source-id: cd74a1bf8f74e31bd2c32c80d32c617f4b1766e8

Summary:
This commit adds FFmpeg-based encoder StreamWriter class.
StreamWriter is pretty much the opposite of StreamReader class, and
it supports;

* Encoding audio / still image / video
* Exporting to local file / streaming protocol / devices etc...
* File-like object support (in later commit)
* HW video encoding (in later commit)

See also: https://fburl.com/gslide/z85kn5a9 (Meta internal)

Pull Request resolved: https://github.com/pytorch/audio/pull/2628

Reviewed By: nateanl

Differential Revision: D38816650

Pulled By: mthrok

fbshipit-source-id: a9343b0d55755e186971dc96fb86eb52daa003c8

Summary:
Factory functions have been added to HDemucs class and test the implementation within the testing files.

Pull Request resolved: https://github.com/pytorch/audio/pull/2547

Reviewed By: carolineechen

Differential Revision: D37948600

Pulled By: skim0514

fbshipit-source-id: 7ac4e4a71519450cfbbc24ff7d7e70521f676040

Summary:
Fix for the cuda 11.6 and usage of cudatoolkit

Pull Request resolved: https://github.com/pytorch/audio/pull/2501

Reviewed By: mthrok

Differential Revision: D37388598

Pulled By: atalman

fbshipit-source-id: 41add7ad6fbb3d156cc1270625dc085c62f7a531

Summary:
The version of MKL that is installed alongside PyTorch has been bumped
to 2022.1 on Windows and it is causing installation issue in unit tests.

This commit pins the previous version

Backport of https://github.com/pytorch/audio/issues/2463

Pull Request resolved: https://github.com/pytorch/audio/pull/2486

Reviewed By: nateanl

Differential Revision: D37160156

Pulled By: mthrok

fbshipit-source-id: 7e3a30c25782b349a3cad2ee6d1141affc921881

Summary:
All the unittests jobs are failing due to import error due to protobuf and scipy.
This commit pins the versions of them to an older version.

## protobuf

https://app.circleci.com/pipelines/github/pytorch/audio/10979/workflows/42005226-ca7e-471c-80f4-db09f4bd2089/jobs/692078

```
E   TypeError: Descriptors cannot not be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```

https://github.com/protocolbuffers/protobuf/issues/10051
https://github.com/PyTorchLightning/pytorch-lightning/issues/13159

## scipy (pypocketfft)

1.8.1 is causing issue.

https://app.circleci.com/pipelines/github/pytorch/audio/10980/workflows/470a9361-4cc5-4d7c-9264-28fc8b86f1cb/jobs/692267

    ```
    ../env/lib/python3.9/site-packages/librosa/core/audio.py:11: in <module>
        import scipy.signal
    ../env/lib/python3.9/site-packages/scipy/signal/__init__.py:309: in <module>
        from . import _sigtools, windows
    ../env/lib/python3.9/site-packages/scipy/signal/windows/__init__.py:41: in <module>
        from ._windows import *
    ../env/lib/python3.9/site-packages/scipy/signal/windows/_windows.py:7: in <module>
        from scipy import linalg, special, fft as sp_fft
    ../env/lib/python3.9/site-packages/scipy/fft/__init__.py:91: in <module>
        from ._helper import next_fast_len
    ../env/lib/python3.9/site-packages/scipy/fft/_helper.py:3: in <module>
        from ._pocketfft import helper as _helper
    ../env/lib/python3.9/site-packages/scipy/fft/_pocketfft/__init__.py:3: in <module>
        from .basic import *
    ../env/lib/python3.9/site-packages/scipy/fft/_pocketfft/basic.py:6: in <module>
        from . import pypocketfft as pfft
    E   ImportError: /home/circleci/project/env/lib/python3.9/site-packages/torch/lib/../../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/circleci/project/env/lib/python3.9/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-39-x86_64-linux-gnu.so)

Pull Request resolved: https://github.com/pytorch/audio/pull/2422

Reviewed By: atalman

Differential Revision: D36764198

Pulled By: mthrok

fbshipit-source-id: 897a79fe9c3165206c2e747147fd0f257fc4f683

Summary:
Adding py3.10 to audio

Pull Request resolved: https://github.com/pytorch/audio/pull/2224

Reviewed By: malfet, atalman, mthrok

Differential Revision: D34442377

Pulled By: seemethere

fbshipit-source-id: 2656de73427063958d609a74c01b526a476cb06a

Summary:
Adds unit tests for Emformer RNN-T LibriSpeech recipe. Also makes changes to recipe to resolve errors with pickling lambda functions in Windows.

Pull Request resolved: https://github.com/pytorch/audio/pull/2216

Reviewed By: nateanl

Differential Revision: D34171480

Pulled By: hwangjeff

fbshipit-source-id: 5fcebb457051f3041766324863728411180f5e1e

Summary:
This PR adds the prototype streaming API.
The implementation is based on ffmpeg libraries.

For the detailed usage, please refer to [the resulting tutorial](https://534376-90321822-gh.circle-artifacts.com/0/docs/tutorials/streaming_api_tutorial.html).

Pull Request resolved: https://github.com/pytorch/audio/pull/2164

Reviewed By: hwangjeff

Differential Revision: D33934457

Pulled By: mthrok

fbshipit-source-id: 92ade4aff2d25baf02c0054682d4fbdc9ba8f3fe

Summary:
Similar to https://github.com/pytorch/vision/pull/4788
Make sure the workflow could download right PyTorch with cpu or cuda in case nightly build wasn't ready at that day.

https://app.circleci.com/pipelines/github/pytorch/audio/8427/workflows/11a80738-bcdd-45e3-b37f-328be36c60ee/jobs/438285?invite=true#step-107-542

![image](https://user-images.githubusercontent.com/16190118/142969390-142df7ec-6040-40c1-9a02-17d43f5de05e.png)

Pull Request resolved: https://github.com/pytorch/audio/pull/2026

Reviewed By: hwangjeff, nateanl

Differential Revision: D32634926

Pulled By: mthrok

fbshipit-source-id: 30d6349a0a2ce174b789a5888b1c8e0544a23a37

This commit changes the build process so that the third party code are fetched automatically when `setup.py` is invoked.

This is to allow
1. `build_sdist` to contain the third party library codes, so that source code distribution created at the time release tag is created is complete.
2. It makes it possible to do `pip install https://github.com/pytorch/audio.git`.
    Example:
    ```
    !pip install 'cmake>=3.18' ninja
    !pip install --verbose git+https://github.com/pytorch/audio.git@auto-source
    ```

* enable windows cudatests

* add this dir

* minor change

* vs integration

* Update cuda_install.bat

* add logs

* minor change

* minor change

* cp vision conda activate

* mv vc_env_helper.bat

* minor change

* exit if cuda not avaiable

* install numpy

* improt CMakeLists

* check cuda

* minor change

* change windows GPU image from previous to stable

* set libtorch audio suffix as pyd on Windows

* reduce changes

* check env settings

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

We have been running unit test with editable installation. (i.e. `python setup.py develop`), with which we missed issues like #842. 

This CC makes installation in CI non-editable, and change test directory structure so that the source code will not shadow the installed version of `torchaudio`. With simple `pytest test`, `pytest` modifies `sys.path` and prepend checked out repository, which shadows the installed version.

To remedy this, the whole test suites has been moved from `./test` to `./test/torchaudio_unittest`. This adds nice module structure to our test code and we can do absolute import in each test module, which makes it possible again to run test with `python -m unittest torchaudio_unittest/XXX.py`

This change does not affect the regular development process (`python setup.py develop` && `pytest test`)

* Use env var for switching SoX build and default to not

* Update README

* Fix packaging/CI script

* Enable CUDA tests for Windows

* Add back branch filters

* Turn on tests when building through conda-build

* skip one failling test on windows.

* Switch to unittest

* Fix conda env activation

* add linux only condition

* Fix librosa installation

* conda-forge channel as a fallback

* Revert "conda-forge channel as a fallback"

This reverts commit 8c15f6638e34792640a3de3e057d9c2995425180.

* Try the pytorch way

* Use conda-forge

* Disable conda-forge

* Install future at setup env

* Move linux unittest CI job tools to dedicated directory

* Restore Linux scripts and move the updated souces to a new subdir

* Add back future

* Remove comments
Co-authored-by: Vincent Quenneville-Belair <vincentqb@gmail.com>
Co-authored-by: moto <855818+mthrok@users.noreply.github.com>

* Add test image with Kaldi executable and third_parties

* Use Ubuntu as bese image since pytorch/many-linux is huge