GitLab

Summary:
This commit removes compute_kaldi_pitch function and the underlying Kaldi integration from torchaudio.

Kaldi pitch function was added in a short period of time by integrating the original Kaldi implementation, instead of reimplementing it in PyTorch.

The Kaldi integration employed a hack which replaces the base vector/matrix implementation of Kaldi with PyTorch Tensor so that there is only one blas library within torchaudio.

Recently, we are making torchaudio more lean, and we don't see a wide adoption of kaldi_pitch feature, so we decided to remove them.

See some of the discussion https://github.com/pytorch/audio/issues/1269

Pull Request resolved: https://github.com/pytorch/audio/pull/3368

Differential Revision: D46406176

Pulled By: mthrok

fbshipit-source-id: ee5e24d825188f379979ddccd680c7323b119b1e

Summary:
TorchAudio has migrated CTC decoder to flashlight-text, and code related CTC decoder was removed in https://github.com/pytorch/audio/issues/3236.

This commit cleans up the residual, removes the third party libraries used for CTC decoder, and mention to environment variable for CTC decoder.

Pull Request resolved: https://github.com/pytorch/audio/pull/3339

Reviewed By: nateanl

Differential Revision: D45920878

Pulled By: mthrok

fbshipit-source-id: 8d93e64138697781570e5b0b1c9f86e1a7923a89

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/3321

Reviewed By: atalman, mthrok

Differential Revision: D45673225

Pulled By: malfet

fbshipit-source-id: f2b915f3307ba95445702e3018254ad254fe2bb3

Summary:
https://github.com/pytorch/pytorch/pull/93155 Core has dropped python3.7

Pull Request resolved: https://github.com/pytorch/audio/pull/3020

Reviewed By: mthrok

Differential Revision: D42902346

Pulled By: weiwangmeta

fbshipit-source-id: 07ab1aff0e128c5960d87e5fa29e341310dea388

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2911

Reviewed By: carolineechen

Differential Revision: D41887854

Pulled By: mthrok

fbshipit-source-id: eb91773ec67b4cda2d70733df450956d83742509

Summary:
After reviewing the code for KenLM it turned out that we can build it without boost.

Pull Request resolved: https://github.com/pytorch/audio/pull/2552

Reviewed By: xiaohui-zhang

Differential Revision: D37949699

Pulled By: mthrok

fbshipit-source-id: 4a4ffae4220d0b764b53f52b93040670d91a84a3

Summary:
The source build is still saying its 0.12.

Pull Request resolved: https://github.com/pytorch/audio/pull/2509

Reviewed By: carolineechen

Differential Revision: D37427703

Pulled By: mthrok

fbshipit-source-id: a6e455ba7c583af7b1a2a355ca45a9e5ab5fe30d

Summary:
Applies new import merging and sorting from µsort v1.0.

When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.

Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.

For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting

Reviewed By: lisroach

Differential Revision: D36402214

fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2250

Reviewed By: mthrok

Differential Revision: D34302192

Pulled By: nateanl

fbshipit-source-id: 4ea7047503ef87e22b5ef6075ad010314d5e3885

Summary:
This commit fixes the feature to exclude `torchaudio.prototype` module.

In `setup.py` there is a special case that is triggered if the commit is on release branch or release tag, that  excludes `torchaudio.prototype`. This was introduced to make it easy for release-related work.
It turned out that the submodules under `torchaudio.prototype`, such as `torchaudio.prototype.pipelines`, are not properly excluded from packaging.
These sub modules did not exist in previous releases, so it was not an issue.

**Note** This feature is triggered only in release branch, so the fix is not visible in the CI of this PR.
https://app.circleci.com/pipelines/github/pytorch/audio/9674/workflows/d0c9a6f1-8ca9-441a-a5f5-08926075fa39/jobs/553985?invite=true#step-104-193

The following outputs were observed when running it on local env.

* Before the change

```
$ BUILD_FFMPEG=0 BUILD_SOX=0 BUILD_CTC_DECODER=0 BUILD_RNNT=0 BUILD_KALDI=0 python setup.py clean bdist_wheel
```
```
-- Git branch: prototype-exclusion
-- Git SHA: 0af1edaa420c46be10292cbea7150c34ef80a0e1
-- Git tag: None
-- PyTorch dependency: torch
-- Building version 0.11.0+0af1eda
 --- Initializing submodules
 --- Initialized submodule
Excluding torchaudio.prototype from the package.
...
creating build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype
creating build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/io
copying torchaudio/prototype/io/streamer.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/io
copying torchaudio/prototype/io/__init__.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/io
creating build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/pipelines
copying torchaudio/prototype/pipelines/__init__.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/pipelines
copying torchaudio/prototype/pipelines/rnnt_pipeline.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/pipelines
creating build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/ctc_decoder
copying torchaudio/prototype/ctc_decoder/ctc_decoder.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/ctc_decoder
copying torchaudio/prototype/ctc_decoder/__init__.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/prototype/ctc_decoder
warning: build_py: byte-compiling is disabled, skipping.
```

* After the change

```
$ BUILD_FFMPEG=0 BUILD_SOX=0 BUILD_CTC_DECODER=0 BUILD_RNNT=0 BUILD_KALDI=0 python setup.py clean bdist_wheel
```

```
-- Git branch: prototype-exclusion
-- Git SHA: 0af1edaa420c46be10292cbea7150c34ef80a0e1
-- Git tag: None
-- PyTorch dependency: torch
-- Building version 0.11.0+0af1eda
 --- Initializing submodules
 --- Initialized submodule
Excluding torchaudio.prototype from the package.
...
creating build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2
copying torchaudio/models/wav2vec2/__init__.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2
copying torchaudio/models/wav2vec2/model.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2
copying torchaudio/models/wav2vec2/components.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2
creating build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2/utils
copying torchaudio/models/wav2vec2/utils/__init__.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2/utils
copying torchaudio/models/wav2vec2/utils/import_huggingface.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2/utils
copying torchaudio/models/wav2vec2/utils/import_fairseq.py -> build/lib.macosx-11.0-arm64-3.9/torchaudio/models/wav2vec2/utils
warning: build_py: byte-compiling is disabled, skipping.
```

Pull Request resolved: https://github.com/pytorch/audio/pull/2225

Reviewed By: nateanl

Differential Revision: D34257128

Pulled By: mthrok

fbshipit-source-id: a3d6eca5803356e5aa3fe0eda82f6a9f5affb8e8

Summary:
This commits supress the stderr output when shell commands are
executed `setup.py`.

`setup.py` performs multiple git commands to gather metadata.
When `git tag` command tries to find a tag, (which fails unless on release)
it produces `fatal: No names found, cannot describe anything.`

This is confusing especially when it is executed from higher level,
like `pip install git+https://...`.

Pull Request resolved: https://github.com/pytorch/audio/pull/2133

Reviewed By: nateanl

Differential Revision: D33455339

Pulled By: mthrok

fbshipit-source-id: 3e24451eb6fedcd0ad90f7e16e38fcdb70dc9704

Summary:
Drop support for python 3.6, and update dependencies documentation. More details [Issue 2051](https://github.com/pytorch/audio/issues/2051).

Pull Request resolved: https://github.com/pytorch/audio/pull/2139

Reviewed By: mthrok

Differential Revision: D33454583

Pulled By: wernerchao

fbshipit-source-id: 64eccb38e26853ba63f72fb92723e3f0155e806e

Summary:
update the logic for initializing and updating submodule in setup, now that we have more than one submodule

Pull Request resolved: https://github.com/pytorch/audio/pull/2132

Reviewed By: mthrok

Differential Revision: D33410039

Pulled By: carolineechen

fbshipit-source-id: 6e18ec2904a01efb2f753f65345de063218c0b9b

Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/2096

run: `arc lint --apply-patches --paths-cmd 'hg files -I "./**/*.py"'`

Reviewed By: mthrok

Differential Revision: D33297351

fbshipit-source-id: 7bf5956edf0717c5ca90219f72414ff4eeaf5aa8

Summary:
Previously sox-related third-party source code was archived at
`third_party/sox/archives`.
Recently KenLM-related third-party source code was added and
they are archived at `third_party/archives`.

This PR changes the sox archive location to `third_party/archives`,
so that all the archvies are cached at the same location.

Pull Request resolved: https://github.com/pytorch/audio/pull/2086

Reviewed By: carolineechen

Differential Revision: D33236927

Pulled By: mthrok

fbshipit-source-id: 2f2aa5f4b386fefb46d7c98f7179c04995219f3c

Summary:
Add KenLM and its dependencies required for static build (`zlib`, `bzip2`, `lzma` and `boost-thread`).

The KenLM and its dependencies are build but since no corresponding code on torchaudio side is changed, the resulting torchaudio extension module is not changed. (therefore, as long as build process passes on CI this PR should be good to go.)

Pull Request resolved: https://github.com/pytorch/audio/pull/2076

Reviewed By: carolineechen

Differential Revision: D33189980

Pulled By: mthrok

fbshipit-source-id: 6096113128b939f3cf70990c99aacc4aaa954584

This commit changes the build process so that the third party code are fetched automatically when `setup.py` is invoked.

This is to allow
1. `build_sdist` to contain the third party library codes, so that source code distribution created at the time release tag is created is complete.
2. It makes it possible to do `pip install https://github.com/pytorch/audio.git`.
    Example:
    ```
    !pip install 'cmake>=3.18' ninja
    !pip install --verbose git+https://github.com/pytorch/audio.git@auto-source
    ```

Also update README.md to mention 1.9.1

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* Switch to cmake for build
* Hide symbols

Currently `setuptools` assumes that torhcaudio is zip safe and performs
egg (zip) installation when `python setup.py install` (or `pip install .`).

When `torchaudio` loads extension module written with Torchscript, 
the corresponding loading function requires the `so` file to be present as 
an actual file, and this does not work well with egg installation.

This CC fixes this by setting `zip_safe=False` in setup.py and disable egg
installation, so that `python setup.py install` installs uncompressed files
with regular directory structure.

setuptools warns that it should be imported before Distutils

```
site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
```

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* Clean up extension build mechanism and extension location

* Add back the switch to depend on external sox

* Remove print

* Fix

This PR
 - Changes the location of C++ codes from
     - from `torchaudio/torch_sox.cpp` to `torchaudio/csrc/sox.cpp`
     - from `torchaudio/torch_sox.h` to `torchaudio/csrc/sox.h`
 - Changes the location where the resulting library is placed
     - from `_torch_sox.so` (outside of `torchaudio` module)
        to `torchaudio/_torchaudio.so`.

Fix #628

* [WIP] Add Windows CI

* Remove cu_version

* checkout_merge -> checkout

* Add build script

* Switch backend to soundfile

* Remove soundfile as dependency

* Rename jobs

* Fix lint

* New entry points are packaging/build_wheel.sh and packaging/build_conda.sh. The only mandatory environment variable you have to set is PYTHON_VERSION
* CircleCI configuration uses 2.1-style parametrized builds to let you toggle python version, etc. as you do builds. We create a separate job per build configuration for maximum parallelism
* build_tools/packaging got moved to packaging, to be in-line with directory structure in torchvision
* The build_conda.sh and build_wheel.sh delegate most of the heavy lifting to pkg_helpers.bash, which defines a number of bash functions for performing common operations. The intent is that I'll copy-paste this file between other domain API projects.
* TORCHAUDIO_ prefix removed from envvars, so that I can more easily share packaging scripts between projects
* BUILD_VERSION is completely gone; just change the version number if you need to rebuild
* No more logic for cloning and checking out a fresh copy of torchaudio

Billing of changes:

* Can now toggle what branch is built using BRANCH envvar
* PyTorch nightly we were built against is hardcoded into conda meta.yaml
* Disable Conda Python 3.5 builds as we don't have nightlies for torch at the moment
* No more torchaudio-cpu or torchaudio-nightly suffix
* Remove librosa from testing stanza, so that conda build works with stock anaconda channel
* Change convention on PREFIX to be what we previously called $PREFIX/audio
* Make build_from_source.sh cd-neutral

Signed-off-by: Edward Z. Yang ezyang@fb.com