GitLab

Summary:
This commit gets rid of our copy of CTC decoder code and
replace it with upstream Flashlight-Text repo.

Pull Request resolved: https://github.com/pytorch/audio/pull/2580

Reviewed By: carolineechen

Differential Revision: D38244906

Pulled By: mthrok

fbshipit-source-id: d274240fc67675552d19ff35e9a363b9b9048721

Summary:
Remove the code related to libmad, which had been disabled in https://github.com/pytorch/audio/issues/2354

In https://github.com/pytorch/audio/issues/2419, we mp3 decoding to ffmpeg. But CI tests were still using libmad.
This commit completely removes libmad from torchaudio.

This is BC-breaking change as `apply_sox_effects_file` function cannot handle MP3, and it cannot fallback to ffmpeg.
The workaround for this is to use `torchaudio.load` then `apply_sox_effects_tensor`.

Pull Request resolved: https://github.com/pytorch/audio/pull/2428

Reviewed By: carolineechen

Differential Revision: D36851805

Pulled By: mthrok

fbshipit-source-id: f98795c59a1ac61cef511f2bbeac37f7c3c69d55

Summary:
This commit adds file-like object support to Streaming API.

## Features
- File-like objects are expected to implement `read(self, n)`.
- Additionally `seek(self, offset, whence)` is used if available.
- Without `seek` method, some formats cannot be decoded properly.
  - To work around this, one can use the existing `decoder` option to tell what decoder it should use.
  - The set of `decoder` and `decoder_option` arguments were added to `add_basic_[audio|video]_stream` method, similar to `add_[audio|video]_stream`.
  - So as to have the arguments common to both audio and video in front of the rest of the arguments, the order of the arguments are changed.
  - Also `dtype` and `format` arguments were changed to make them consistent across audio/video methods.

## Code structure

The approach is very similar to how file-like object is supported in sox-based I/O.
In Streaming API if the input src is string, it is passed to the implementation bound with TorchBind,
if the src has `read` attribute, it is passed to the same implementation bound via PyBind 11.

![Untitled drawing](https://user-images.githubusercontent.com/855818/169098391-6116afee-7b29-460d-b50d-1037bb8a359d.png)

## Refactoring involved
- Extracted to https://github.com/pytorch/audio/issues/2402
  - Some implementation in the original TorchBind surface layer is converted to Wrapper class so that they can be re-used from PyBind11 bindings. The wrapper class serves to simplify the binding.
  - `add_basic_[audio|video]_stream` methods were removed from C++ layer as it was just constructing string and passing it to `add_[audio|video]_stream` method, which is simpler to do in Python.
  - The original core Streamer implementation kept the use of types in `c10` namespace minimum. All the `c10::optional` and `c10::Dict` were converted to the equivalents of `std` at binding layer. But since they work fine with PyBind11, Streamer core methods deal them directly.

## TODO:
- [x] Check if it is possible to stream MP4 (yuv420p) from S3 and directly decode (with/without HW decoding).

Pull Request resolved: https://github.com/pytorch/audio/pull/2400

Reviewed By: carolineechen

Differential Revision: D36520073

Pulled By: mthrok

fbshipit-source-id: a11d981bbe99b1ff0cc356e46264ac8e76614bc6

Summary:
This commit moves the Streaming API out of prototype module.

* The related classes are renamed as following

  - `Streamer` -> `StreamReader`.
  - `SourceStream` -> `StreamReaderSourceStream`
  - `SourceAudioStream` -> `StreamReaderSourceAudioStream`
  - `SourceVideoStream` -> `StreamReaderSourceVideoStream`
  - `OutputStream` -> `StreamReaderOutputStream`

This change is preemptive measurement for the possibility to add
`StreamWriter` API.

* Replace BUILD_FFMPEG build arg with USE_FFMPEG

We are not building FFmpeg, so USE_FFMPEG is more appropriate

 ---

After https://github.com/pytorch/audio/issues/2377

Remaining TODOs: (different PRs)
- [ ] Introduce `is_ffmpeg_binding_available` function.
- [ ] Refactor C++ code:
   - Rename `Streamer` to `StreamReader`.
   - Rename `streamer.[h|cpp]` to `stream_reader.[h|cpp]`.
   - Rename `prototype.cpp` to `stream_reader_binding.cpp`.
   - Introduce `stream_reader` directory.
- [x] Enable FFmpeg in smoke test (https://github.com/pytorch/audio/issues/2381)

Pull Request resolved: https://github.com/pytorch/audio/pull/2378

Reviewed By: carolineechen

Differential Revision: D36359299

Pulled By: mthrok

fbshipit-source-id: 6a57b702996af871e577fb7addbf3522081c1328

Summary:
libmad integration should be enabled only from source-build

Pull Request resolved: https://github.com/pytorch/audio/pull/2354

Reviewed By: nateanl

Differential Revision: D36012035

Pulled By: mthrok

fbshipit-source-id: adeda8cbfd418f96245909cae6862b648a6915a7

Summary:
This PR adds `BUILD_FFMPEG` switch to torchaudio build process so that features related to ffmpeg are built.
The flag is false by default, so no CI jobs or development flow are affected.

This is because handling the dependencies around ffmpeg is a bit tricky.
Currently, the CMake file uses `pkg-config` to find an ffmpeg installation in the system.
This works fine for both conda-based installation and system-managed installation (like `apt`).

In subsequent PRs, I will find a solution that works for local development and binary distributions.

Pull Request resolved: https://github.com/pytorch/audio/pull/2048

Reviewed By: hwangjeff, nateanl

Differential Revision: D33367260

Pulled By: mthrok

fbshipit-source-id: 94517acecb62bd6d4e96d4b7cbc3ab3c2a25706c

Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/2096

run: `arc lint --apply-patches --paths-cmd 'hg files -I "./**/*.py"'`

Reviewed By: mthrok

Differential Revision: D33297351

fbshipit-source-id: 7bf5956edf0717c5ca90219f72414ff4eeaf5aa8

Summary:
After all the C++ code from https://github.com/pytorch/audio/issues/2072 are added, this commit will enable decoder/KenLM integration in the build process.

Pull Request resolved: https://github.com/pytorch/audio/pull/2078

Reviewed By: carolineechen

Differential Revision: D33198183

Pulled By: mthrok

fbshipit-source-id: 9d7fa76151d06fbbac3785183c7c2ff9862d3128

Summary:
Add KenLM and its dependencies required for static build (`zlib`, `bzip2`, `lzma` and `boost-thread`).

The KenLM and its dependencies are build but since no corresponding code on torchaudio side is changed, the resulting torchaudio extension module is not changed. (therefore, as long as build process passes on CI this PR should be good to go.)

Pull Request resolved: https://github.com/pytorch/audio/pull/2076

Reviewed By: carolineechen

Differential Revision: D33189980

Pulled By: mthrok

fbshipit-source-id: 6096113128b939f3cf70990c99aacc4aaa954584

Summary:
Resolves https://github.com/pytorch/audio/issues/2049, https://github.com/pytorch/audio/issues/1940

Pull Request resolved: https://github.com/pytorch/audio/pull/2050

Reviewed By: nateanl

Differential Revision: D32712513

Pulled By: mthrok

fbshipit-source-id: e1db81786bcca67605ff765d27e0527e20967d1c

Make the structure of library files somewhat similar to PyTorch core, which has the following pattern

```
torch/_C.so
torch/lib/libc10.so
torch/lib/libtorch.so
...
```

```
torchaudio/_torchaudio.so
torchaudio/lib/libtorchaudio.so
```

* Split `libtorchaudio` and `_torchaudio`

This change extract the core implementation from `_torchaudio` to `libtorchaudio`,
so that `libtorchaudio` is reusable in TorchScript-based app.

`_torchaudio` is a wrapper around `libtorchaudio` and only provides PyBind11-based
features. (currently file-like object support in I/O)

* Removed `BUILD_LIBTORCHAUDIO` option

When invoking `cmake`, `libtorchaudio` is always built, so this option is removed.

The new assumptions around the library discoverability

- In regular OSS workflow (`pip`/`conda`-based binary installation), both `libtorchaudio` and `_torchaudio` are present.
    In this case,`libtorchaudio` has to be loaded manually with `torch.ops.load_library` and/or `torch.classes.load_library` otherwise importing `_torchaudio` would not be able to resolve the symbols defined in `libtorchaudio`.
- When `torchaudio` is deployed with PEX format (single zip file)
  - We expect that`libtorchaudio.so` exists as a file in some search path configured by client code.
  - `_torchaudio` is still importable and because we do not know where `libtorchaudio` will exist, we will let the dynamic loader resolve the dependency from `_torchaudio` to `libtorchaudio`, which should work as long as `libtorchaudio` is in a library search path (search path is not modifiable from already-running Python process).

* fix build error on ROCM

* Update CMakeLists.txt
Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com>

* address comments and fix cuda detction on rocm
Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com>

Needed to support CUDA builds on CPU machine

Parse `TORCH_CUDA_ARCH_LIST` as new-CUDA-language Cmake-3.18+ style [CMAKE_CUDA_ARCHITECTURES](https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html#prop_tgt:CUDA_ARCHITECTURES)

* Default to BUILD_SOX=1 in non-Windows systems

Since the adaptation of CMake and restricting to the static linking of libsox,
the build process has become much robust with libsox integration enabled.

This commit makes it default behavior to build libsox integration in non-Windows systems.
The build process still checks BUILD_SOX env var so, setting `BUILD_SOX=0` disables it.

* Switch to cmake for build
* Hide symbols

With this change, `BUILD_TRANSDUCER=1 python setup.py build_ext` now sees `-D_GLIBCXX_USE_CXX11_ABI=` in the compilation command. (Note: sox is C-only so it is not relevant to sox build process)

See also:
 - https://github.com/pytorch/text/pull/931
 - https://stackoverflow.com/a/55406930

* Use cmake for third party

* Apply patch to libmad

* Update gitignore

* Update docker test image

* Use env var for switching SoX build and default to not

* Update README

* Fix packaging/CI script

* Clean up extension build mechanism and extension location

* Add back the switch to depend on external sox

* Remove print

* Fix