GitLab

Summary:
This commit gets rid of our copy of CTC decoder code and
replace it with upstream Flashlight-Text repo.

Pull Request resolved: https://github.com/pytorch/audio/pull/2580

Reviewed By: carolineechen

Differential Revision: D38244906

Pulled By: mthrok

fbshipit-source-id: d274240fc67675552d19ff35e9a363b9b9048721

Summary:
After reviewing the code for KenLM it turned out that we can build it without boost.

Pull Request resolved: https://github.com/pytorch/audio/pull/2552

Reviewed By: xiaohui-zhang

Differential Revision: D37949699

Pulled By: mthrok

fbshipit-source-id: 4a4ffae4220d0b764b53f52b93040670d91a84a3

Summary:
closes https://github.com/pytorch/audio/issues/2420

Pull Request resolved: https://github.com/pytorch/audio/pull/2479

Reviewed By: carolineechen

Differential Revision: D37142717

Pulled By: mthrok

fbshipit-source-id: c3d4cc1435a74dfa6992112590c988c2903511a8

Summary:
Remove the code related to libmad, which had been disabled in https://github.com/pytorch/audio/issues/2354

In https://github.com/pytorch/audio/issues/2419, we mp3 decoding to ffmpeg. But CI tests were still using libmad.
This commit completely removes libmad from torchaudio.

This is BC-breaking change as `apply_sox_effects_file` function cannot handle MP3, and it cannot fallback to ffmpeg.
The workaround for this is to use `torchaudio.load` then `apply_sox_effects_tensor`.

Pull Request resolved: https://github.com/pytorch/audio/pull/2428

Reviewed By: carolineechen

Differential Revision: D36851805

Pulled By: mthrok

fbshipit-source-id: f98795c59a1ac61cef511f2bbeac37f7c3c69d55

Summary:
libmad integration should be enabled only from source-build

Pull Request resolved: https://github.com/pytorch/audio/pull/2354

Reviewed By: nateanl

Differential Revision: D36012035

Pulled By: mthrok

fbshipit-source-id: adeda8cbfd418f96245909cae6862b648a6915a7

Summary:
This commit
1. Updates the config.guess and config.sub files and
2. applies them to all the third party libraries that use them.

This resolves the following build failure on M1 mac with newer SDK.

On MacBookPro with M1 chip, with the recent OS update, something
about the development environment has been changed (probably newer
version of XCode) and the build stopeed working with the following
errors from third party dependencies.

```
checking build system type... Invalid configuration ‘arm64-apple-darwin20.0.0': machine ‘arm64-apple' not recognized
```

note: config files are taken from https://www.gnu.org/software/gettext/manual/html_node/config_002eguess.html

Pull Request resolved: https://github.com/pytorch/audio/pull/2307

Reviewed By: nateanl

Differential Revision: D35318273

Pulled By: mthrok

fbshipit-source-id: 746ac51dd1816767aa78b88445f76a29acfd29e8

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2300

Reviewed By: xiaohui-zhang

Differential Revision: D35258323

Pulled By: nateanl

fbshipit-source-id: 4b9f86600399ba0f5ec47f1c402968a812aa557d

Summary:
This PR addresses https://github.com/pytorch/audio/issues/2295 by updating `zlib`'s url to the one on sourceforge.net.
`zlib` 1.2.11 source code is removed from the official site. According to https://zlib.net, ```Due to the bug fixes, any installations of 1.2.11 should be replaced with 1.2.12.```
sourceforge preserves the older versions thus is more stable. The PR keep 1.2.11 as currently there is no 1.2.12 on sourceforge.

Pull Request resolved: https://github.com/pytorch/audio/pull/2297

Reviewed By: mthrok

Differential Revision: D35251361

Pulled By: nateanl

fbshipit-source-id: 174c2c2e1c34bef9799bbacfd1e12c8ff13ff15d

Summary:
Originally, the global property TORCHAUDIO_THIRD_PARTIES was introduced
to handle the optional third party dependencies that can change based on
the build config.

After revising the CMake, it turned out this is not really necessary,
as our torchaudio/csrc/CMakeLists.txt properly branches out for
conditional dependencies. Rather we should leave the global scope untouched.

Pull Request resolved: https://github.com/pytorch/audio/pull/2282

Reviewed By: hwangjeff

Differential Revision: D35059838

Pulled By: mthrok

fbshipit-source-id: ed3557eaa9a669e4466d64893beab5089eca78b8

Summary:
When building Kaldi submodule, it requires to run `get_version.sh`, so that version header is available.
It was pointed that the script should run with `bash`, instead of `sh`.

Fixes https://github.com/pytorch/audio/issues/2268

Pull Request resolved: https://github.com/pytorch/audio/pull/2269

Reviewed By: carolineechen

Differential Revision: D34667726

Pulled By: mthrok

fbshipit-source-id: 761b82c54b58af2bfb2836cbe18c9708f853f1e1

Summary:
This commit fixes the issue with ffmpeg discovery at build time.
The original implementation had issues like.

1. Wrong usage of FindFFMPEG, which caused mixture of ffmpeg libraries from system directory and user directory.
2. The optional `FFMPEG_ROOT` variable was not set within cmake.

The issue 1 is problematic when a user does not have a permission to
modify the environment. For example, an old version of ffmpeg, which is
installed in a directory managed by the system (such as `/usr/local/lib`),
then there is no way to specify a path in which user installs a supported version
of ffmpeg.

This commit changes the behavior by first searching the library
in `FFMPEG_ROOT` environment variables, then
resorting to the original behavior of searching the custom paths with
system default path.

Also this commirt removes support for `libavresample`, which is deprecated in
ffmpeg 4 and removed in ffmpeg 5.

Pull Request resolved: https://github.com/pytorch/audio/pull/2204

Reviewed By: carolineechen

Differential Revision: D34225769

Pulled By: mthrok

fbshipit-source-id: 95b0bfaaef31e2e69e6df29f789010f48a48210b

Summary:
Update ffmpeg discovery logic

Previously the build process used pkg-config to locate
an installation of ffmpeg, which does not work well Windows/CentOS.

This commit update the discovery process to use the custom
FindFFMPEG.cmake adopted from Kitware/VTK repository with addition of
conda environment.

 The custom discovery logic can support Windows and CentOS.

Pull Request resolved: https://github.com/pytorch/audio/pull/2124

Reviewed By: carolineechen

Differential Revision: D33429564

Pulled By: mthrok

fbshipit-source-id: 6cb50c1d8c58f51e0f3f3af5c5b541aa3a699bba

Summary:
Currently, macOS CI jobs install `pkg-config` and `wget` with `brew`.
This is problematic as brew takes a long time with auto-update, and disabling the auto-update is not an ideal solution.
Conda also distributes these packages, so switching to conda.

Example issues with brew installation.
https://app.circleci.com/pipelines/github/pytorch/audio/7825/workflows/53965bcf-6ddf-4e42-ad52-83fd1bbab717

This commit removes the use of `brew` by
1. Replacing the use of `wget` with `curl` (pre-installed in most distro)
2. Install `pkg-condig` from conda.
    Note: All the macOS jobs, including binary build jobs, uses conda. Using `pkg-config` from Conda makes it easy to discover the packages installed from conda. (like `ffmpeg` in https://github.com/pytorch/audio/issues/2122)
3. Add `pkg-config` to conda build-time dependency
4. Make sure that the availability of `pkg-config` is explicitly checked when `sox` is being configured. (otherwise, it will fail at somewhere in the middle of build process with somewhat unintuitve error message)

Pull Request resolved: https://github.com/pytorch/audio/pull/1873

Reviewed By: carolineechen, nateanl

Differential Revision: D33404975

Pulled By: mthrok

fbshipit-source-id: ae512d3a3a422ebfe3b46c492bed44deecc36e72

Summary:
Preparation to land Python front-end of ffmpeg-related features.

- Set BUILD_FFMPEG=1 in Linux/macOS unit test jobs
- Install ffmpeg and pkg-config from conda-forge
- Add note about Windows build process
- Temporarily avoid `av_err2str`

Pull Request resolved: https://github.com/pytorch/audio/pull/2114

Reviewed By: hwangjeff

Differential Revision: D33371346

Pulled By: mthrok

fbshipit-source-id: b0e16a35959a49a2166109068f3e0cbbb836e888

Summary:
cc mthrok

Pull Request resolved: https://github.com/pytorch/audio/pull/2116

Reviewed By: mthrok

Differential Revision: D33368453

Pulled By: jdsgomes

fbshipit-source-id: 09cf3fe5ed6f771c2f16505633c0e59b0c27453c

Summary:
This PR adds `BUILD_FFMPEG` switch to torchaudio build process so that features related to ffmpeg are built.
The flag is false by default, so no CI jobs or development flow are affected.

This is because handling the dependencies around ffmpeg is a bit tricky.
Currently, the CMake file uses `pkg-config` to find an ffmpeg installation in the system.
This works fine for both conda-based installation and system-managed installation (like `apt`).

In subsequent PRs, I will find a solution that works for local development and binary distributions.

Pull Request resolved: https://github.com/pytorch/audio/pull/2048

Reviewed By: hwangjeff, nateanl

Differential Revision: D33367260

Pulled By: mthrok

fbshipit-source-id: 94517acecb62bd6d4e96d4b7cbc3ab3c2a25706c

Summary:
Previously sox-related third-party source code was archived at
`third_party/sox/archives`.
Recently KenLM-related third-party source code was added and
they are archived at `third_party/archives`.

This PR changes the sox archive location to `third_party/archives`,
so that all the archvies are cached at the same location.

Pull Request resolved: https://github.com/pytorch/audio/pull/2086

Reviewed By: carolineechen

Differential Revision: D33236927

Pulled By: mthrok

fbshipit-source-id: 2f2aa5f4b386fefb46d7c98f7179c04995219f3c

Summary: Pull Request resolved: https://github.com/pytorch/audio/pull/2085

Reviewed By: carolineechen

Differential Revision: D33235225

Pulled By: mthrok

fbshipit-source-id: 47fe9ec4c93a26322b3a362202ddd3c4654c3f8c

Summary:
After all the C++ code from https://github.com/pytorch/audio/issues/2072 are added, this commit will enable decoder/KenLM integration in the build process.

Pull Request resolved: https://github.com/pytorch/audio/pull/2078

Reviewed By: carolineechen

Differential Revision: D33198183

Pulled By: mthrok

fbshipit-source-id: 9d7fa76151d06fbbac3785183c7c2ff9862d3128

Summary:
Add KenLM and its dependencies required for static build (`zlib`, `bzip2`, `lzma` and `boost-thread`).

The KenLM and its dependencies are build but since no corresponding code on torchaudio side is changed, the resulting torchaudio extension module is not changed. (therefore, as long as build process passes on CI this PR should be good to go.)

Pull Request resolved: https://github.com/pytorch/audio/pull/2076

Reviewed By: carolineechen

Differential Revision: D33189980

Pulled By: mthrok

fbshipit-source-id: 6096113128b939f3cf70990c99aacc4aaa954584

It's wrong to depend on `${TORCH_LIBRARIES}` as it pulls in explicit
`libcuda.so.1` dependency, which violates the assumption that GPU
accelerated libraries should be loadable with no NVIDIA drivers installed

Instead, make it depend on `torch` target, which includes all necessary
Torch C++ API dependences

When building torchaudio from source, `get_version.sh` from kaldi is executed everytime,
which results in kaldi-bindings to be always rebuilt.

This commit add "if" guard to the part so that they are not always executed.

Needed to allow building on M1

Downloaded from https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD and https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD

Replacing the config.guess with a newer version to support newer hardware, such as Nvidia Jetson.

Obtained from: https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD

See: https://www.gnu.org/software/gettext/manual/html_node/config_002eguess.html

Replace the prototype RNNT implementation (using warp-transducer) with one without external library dependencies

`wavpack` is a format not supported/tested in torchaudio.
Leaving the option blank can cause the issue like #1461 in untested environment.

* Fix fileobj I/O undeterministic behavior

Ever since the file-like object support was added in #1158, the test
was occasionally failing in CI. This PR fixes this.

* Switch to cmake for build
* Hide symbols

Currently `libsox` on Linux is compiled with GPU OpenMP and it interferes with the version PyTorch uses (Intel in case of binary distribution). This PR disables OpenMP support for `libsox`, while we investigate the way to use the same OpenMP as PyTorch's version.