GitLab

* [BC-Breaking] Split pretraining and finetuning factory functions

Previously, factory functions of wav2vec2 only generated the architecture
for the fine-tuning architecture used in wav2ve2 paper for ASR task.
That is, pre-training architecture + Linear module, and it did not
provide a straightforward way to generate architectures for pre-training.

The goal of the original implementation was to allow the inference of
wav2vec2 in non-Python environment via TorchScript. Now we would like to
expand it to pre-training/fine-tuning and HuBERT model as well.

Therefore, we need to have factory functions for both pre-training and
fine-tuning. This commit introduces new factory functions and separate
functions for pre-training and fine-tuning.

1. New functions for ASR fine-tuning.

We introdcue `wav2vec2_asr_XXX` functions which generates the architecture
used for the fine-tuning task in wav2vec2 paper. *1

2. Re-purpse the old functions

The existing functions, `wav2vec2_XXX`, now generates the architecture with
pre-trainig module only. (no Linear module)

Note
*1 This architecture is just one way to define architecture for fine-tuning
and it is not universal definition. The new `wav2vec2_asr_XXX` functions are
designed to provide these specific fine-tuning configuration and they are not
meant to support generic architecture for downstream task.

This commit fixes the local build on Windows with CUDA.

Previously, the Linear module (called `readout`, which is used only for an ASR fine-tuning
task) was placed in encoder module. Conceptually, the encoder has nothing to
do with a module specific to fine-tuning / downstream task.

The problems here are that;
1. encoder can be also used in pre-training phase, in which such a module should
not present
2. The choice of Linear module is arbitral, and it is inconvenient for users
to have hard-coded module structure in encoder.

Therefore, this commit moves the Linear module out the encoder, and places it
as `aux` attribute of `Wav2Vec2Model`. (as a result `Wav2Vec2Model` has
`feature_extractor`, `encoder` and `aux` attributes.)

An alternative approach is to define another module and place `Wav2Vec2Model`
and aux module along each other. But that will introduce a new class we need
to maintain.
The expected use of `aux` is only  for 1. loading the pre-trained parameters 
published by `fairseq` (and it's variations from HF) and 2. creating the same model 
architectures for comparison experiment.
The newly introduced class will not be general enough for downstream adaptations, 
where there will be a bunch of different more complicated models. (i.e. s3prl)

Therefore, based on the minimalistic approach, we put them inside of `Wav2Vec2Model`.

* Fix HF model integration

Previously, when testing wav2vec models from HF transformers, all the model were
instantiated as `Wav2Vec2ForCTC` class, while some of them were supposed to be
`Wav2Vec2Model`.

Fixing this revealed that model importer cannot correctly handle `Wav2Vec2Model` import.

This PR fixes these issues.

Summary: Update fairseq reference from master to main elsewhere

Reviewed By: alexeib

Differential Revision: D30938472

fbshipit-source-id: 243b98550207f241c9d3265bf3d4060350aaf0a8
Co-authored-by: Diana Liskovich <dianaml@fb.com>

Apply tweak around the test names so that it's easier to see which tests are failing.

Before: `test_import_finetuned_model_2`
After:  `test_import_finetuned_model_2_wav2vec2_large_lv60k`

* [BC-Breaking] Update `extract_features` of Wav2Vec2Model

Originally, `extract_features` method was returning the result from
the convolutional feature extractor module.

The features commonly used in downstream tasks are outputs from intermediate
layers of transformer block in encoder.

This commit update the behavior of `extract_features` to allow selectively
retrieve such features.

Make the structure of library files somewhat similar to PyTorch core, which has the following pattern

```
torch/_C.so
torch/lib/libc10.so
torch/lib/libtorch.so
...
```

```
torchaudio/_torchaudio.so
torchaudio/lib/libtorchaudio.so
```

* Split `libtorchaudio` and `_torchaudio`

This change extract the core implementation from `_torchaudio` to `libtorchaudio`,
so that `libtorchaudio` is reusable in TorchScript-based app.

`_torchaudio` is a wrapper around `libtorchaudio` and only provides PyBind11-based
features. (currently file-like object support in I/O)

* Removed `BUILD_LIBTORCHAUDIO` option

When invoking `cmake`, `libtorchaudio` is always built, so this option is removed.

The new assumptions around the library discoverability

- In regular OSS workflow (`pip`/`conda`-based binary installation), both `libtorchaudio` and `_torchaudio` are present.
    In this case,`libtorchaudio` has to be loaded manually with `torch.ops.load_library` and/or `torch.classes.load_library` otherwise importing `_torchaudio` would not be able to resolve the symbols defined in `libtorchaudio`.
- When `torchaudio` is deployed with PEX format (single zip file)
  - We expect that`libtorchaudio.so` exists as a file in some search path configured by client code.
  - `_torchaudio` is still importable and because we do not know where `libtorchaudio` will exist, we will let the dynamic loader resolve the dependency from `_torchaudio` to `libtorchaudio`, which should work as long as `libtorchaudio` is in a library search path (search path is not modifiable from already-running Python process).

CCI says circleci/python:3.8 is deprecaetd. Migrating to cimg/python:3.8.

> CircleCI’s latest pre-built container images were designed from the ground up to help your team build more reliably.
> Our new images were built specifically for continuous integration projects and they are our most deterministic, performant, and efficient images yet.

> As of Dec 31, 2021, legacy images will no longer be supported on CircleCI.

Related links

- https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic
- https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034
- https://circleci.com/docs/2.0/next-gen-migration-guide

* Fix comparison between signed and unsigned integer expressions
* Remove unused variable

Applies style fixes identified in fbcode.

* fix build error on ROCM

* Update CMakeLists.txt
Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com>

* address comments and fix cuda detction on rocm
Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com>

* Update the way to access libsox global config

Preparation for splitting `libtorchaudio` and `_torchaudio`.

When two libraries are compiled separately, and each code does
`#include <sox.h>` independently, two copies of libsox's static global
variables (`sox_globals_t`) are created.

Our code should be referring to the same instance. To achieve this,
`_torchaudio` should be accessing the global variable defined in
`libtorchaudio` via the custom utility functions, and it should not
directly use `sox_get_globals`.

This PR moves the code related to PyBind11 to the dedicated directory `torchaudio/csrc/pybind`.

Before, features related to PyBind11 (I/O for file-like object) was implemented in `torchaudio/csrc/sox` and the binding was defined in `torchaudio/csrc/pybind.cpp`. We used macro definition `TORCH_API_INCLUDE_EXTENSION_H` to turn on/off the feature, in addition to including/excluding `torchaudio/csrc/pybind.cpp` in the list of compiled source.

In the previous manner, in C++ example, one had to rebuild libtorchaudio separately, but by splitting them completely at compile time, it should conceptually possible to distribute libtorchaudio within torchaudio Python package and reuse it for C++ example.

* put output tensor on device in `get_whitenoise()`

* Update `get_spectrogram()` so that window uses same device as waveform

* put window on proper device in `test_griffinlim()`

Cherry-pick of
https://github.com/pytorch/vision/commit/d2460a75de237cfef8e5c3415f7bb0ad8467c0e5
into this repo

Fixes https://github.com/pytorch/audio/issues/1750

Downloading CUDA-11.1 from conda-forge takes a while

* Remove some obsolete conditions about CUDA-10.0 from `pkg_helpers.bash`.
* Use `USE_CUDA` instead of `FORCE_CUDA` in `pkg_helpers.bash`
* Do not define `NO_CUDA_PACKAGE` in build_wheel.sh and build_cuda.sh
* Add conda-forge for Win cuda-11.1 builds
* Pass USE_CUDA / TORCH_CUDA_ARCH_LIST to conda build
* Add selected CUDA to path
* Don't define USE_CUDA for ROCM

TODO: Fix Windows CUDA builds

It's wrong to depend on `${TORCH_LIBRARIES}` as it pulls in explicit
`libcuda.so.1` dependency, which violates the assumption that GPU
accelerated libraries should be loadable with no NVIDIA drivers installed

Instead, make it depend on `torch` target, which includes all necessary
Torch C++ API dependences

Calling `torch.[ops|classes].load_library(<PATH_TO_LIBRARY_FILE>)` is problematic in case `torchaudio`
is deployed with PEX format, because the library file does not exist as a file.

Our extension module, when it exists, is guaranteed to have PyBind11 binding even when no function is bound.
This allows to load the library using the regular `import` statement in Python, and it works even in PEX format.
When the library is loaded, the static initialization kicks in and the custom kernels bound via TorchScript also
become available. This removes the need to call `torch.[ops|classe].load_library`.

This works even when the implementation of custom kernel is stripped from `_torchaudio.so` so long as `_torchaudio.so` properly depend on the library that has the kernel implementations and static initialization.

Needed to support CUDA builds on CPU machine

Parse `TORCH_CUDA_ARCH_LIST` as new-CUDA-language Cmake-3.18+ style [CMAKE_CUDA_ARCHITECTURES](https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html#prop_tgt:CUDA_ARCHITECTURES)

Copy `cuda_install.bat` from
https://github.com/pytorch/vision/blob/3d9d264198e2167030dc04bf453aeca47daa37db/packaging/windows/internal/cuda_install.bat

Add `windows_intall_cuda` step and include it in `binary_windows_conda`
and `binary_windows_wheel` workflows