pipelines.rst

.. py:module:: torchaudio.pipelines

torchaudio.pipelines
====================

.. currentmodule:: torchaudio.pipelines
		   
The ``torchaudio.pipelines`` module packages pre-trained models with support functions and meta-data into simple APIs tailored to perform specific tasks.

When using pre-trained models to perform a task, in addition to instantiating the model with pre-trained weights, the client code also needs to build pipelines for feature extractions and post processing in the same way they were done during the training. This requires to carrying over information used during the training, such as the type of transforms and the their parameters (for example, sampling rate the number of FFT bins).

To make this information tied to a pre-trained model and easily accessible, ``torchaudio.pipelines`` module uses the concept of a `Bundle` class, which defines a set of APIs to instantiate pipelines, and the interface of the pipelines.

The following figure illustrates this.

.. image:: https://download.pytorch.org/torchaudio/doc-assets/pipelines-intro.png

A pre-trained model and associated pipelines are expressed as an instance of ``Bundle``. Different instances of same ``Bundle`` share the interface, but their implementations are not constrained to be of same types. For example, :class:`SourceSeparationBundle` defines the interface for performing source separation, but its instance :data:`CONVTASNET_BASE_LIBRI2MIX` instantiates a model of :class:`~torchaudio.models.ConvTasNet` while :data:`HDEMUCS_HIGH_MUSDB` instantiates a model of :class:`~torchaudio.models.HDemucs`. Still, because they share the same interface, the usage is the same.

.. note::

   Under the hood, the implementations of ``Bundle`` use components from other ``torchaudio`` modules, such as :mod:`torchaudio.models` and :mod:`torchaudio.transforms`, or even third party libraries like `SentencPiece <https://github.com/google/sentencepiece>`__ and `DeepPhonemizer <https://github.com/as-ideas/DeepPhonemizer>`__. But this implementation detail is abstracted away from library users.

.. _RNNT:

RNN-T Streaming/Non-Streaming ASR
---------------------------------

Interface
~~~~~~~~~

``RNNTBundle`` defines ASR pipelines and consists of three steps: feature extraction, inference, and de-tokenization.

.. image:: https://download.pytorch.org/torchaudio/doc-assets/pipelines-rnntbundle.png

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   RNNTBundle
   RNNTBundle.FeatureExtractor
   RNNTBundle.TokenProcessor

.. rubric:: Tutorials using ``RNNTBundle``

.. minigallery:: torchaudio.pipelines.RNNTBundle

Pretrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   EMFORMER_RNNT_BASE_LIBRISPEECH


wav2vec 2.0 / HuBERT / WavLM - SSL
----------------------------------

Interface
~~~~~~~~~

``Wav2Vec2Bundle`` instantiates models that generate acoustic features that can be used for downstream inference and fine-tuning.

.. image:: https://download.pytorch.org/torchaudio/doc-assets/pipelines-wav2vec2bundle.png

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   Wav2Vec2Bundle

Pretrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   WAV2VEC2_BASE
   WAV2VEC2_LARGE
   WAV2VEC2_LARGE_LV60K
   WAV2VEC2_XLSR53
   WAV2VEC2_XLSR_300M
   WAV2VEC2_XLSR_1B
   WAV2VEC2_XLSR_2B
   HUBERT_BASE
   HUBERT_LARGE
   HUBERT_XLARGE
   WAVLM_BASE
   WAVLM_BASE_PLUS
   WAVLM_LARGE

wav2vec 2.0 / HuBERT - Fine-tuned ASR
-------------------------------------

Interface
~~~~~~~~~

``Wav2Vec2ASRBundle`` instantiates models that generate probability distribution over pre-defined labels, that can be used for ASR.

.. image:: https://download.pytorch.org/torchaudio/doc-assets/pipelines-wav2vec2asrbundle.png

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   Wav2Vec2ASRBundle

.. rubric:: Tutorials using ``Wav2Vec2ASRBundle``

.. minigallery:: torchaudio.pipelines.Wav2Vec2ASRBundle

Pretrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   WAV2VEC2_ASR_BASE_10M
   WAV2VEC2_ASR_BASE_100H
   WAV2VEC2_ASR_BASE_960H
   WAV2VEC2_ASR_LARGE_10M
   WAV2VEC2_ASR_LARGE_100H
   WAV2VEC2_ASR_LARGE_960H
   WAV2VEC2_ASR_LARGE_LV60K_10M
   WAV2VEC2_ASR_LARGE_LV60K_100H
   WAV2VEC2_ASR_LARGE_LV60K_960H
   VOXPOPULI_ASR_BASE_10K_DE
   VOXPOPULI_ASR_BASE_10K_EN
   VOXPOPULI_ASR_BASE_10K_ES
   VOXPOPULI_ASR_BASE_10K_FR
   VOXPOPULI_ASR_BASE_10K_IT
   HUBERT_ASR_LARGE
   HUBERT_ASR_XLARGE

wav2vec 2.0 / HuBERT - Forced Alignment
---------------------------------------

Interface
~~~~~~~~~

``Wav2Vec2FABundle`` bundles pre-trained model and its associated dictionary. Additionally, it supports appending ``star`` token dimension.

.. image:: https://download.pytorch.org/torchaudio/doc-assets/pipelines-wav2vec2fabundle.png

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   Wav2Vec2FABundle
   Wav2Vec2FABundle.Tokenizer
   Wav2Vec2FABundle.Aligner

.. rubric:: Tutorials using ``Wav2Vec2FABundle``

.. minigallery:: torchaudio.pipelines.Wav2Vec2FABundle

Pertrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   MMS_FA

.. _Tacotron2:
   
Tacotron2 Text-To-Speech
------------------------

``Tacotron2TTSBundle`` defines text-to-speech pipelines and consists of three steps: tokenization, spectrogram generation and vocoder. The spectrogram generation is based on :class:`~torchaudio.models.Tacotron2` model.

.. image:: https://download.pytorch.org/torchaudio/doc-assets/pipelines-tacotron2bundle.png

``TextProcessor`` can be rule-based tokenization in the case of characters, or it can be a neural-netowrk-based G2P model that generates sequence of phonemes from input text.

Similarly ``Vocoder`` can be an algorithm without learning parameters, like `Griffin-Lim`, or a neural-network-based model like `Waveglow`.

Interface
~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   Tacotron2TTSBundle
   Tacotron2TTSBundle.TextProcessor
   Tacotron2TTSBundle.Vocoder

.. rubric:: Tutorials using ``Tacotron2TTSBundle``

.. minigallery:: torchaudio.pipelines.Tacotron2TTSBundle

Pretrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   TACOTRON2_WAVERNN_PHONE_LJSPEECH
   TACOTRON2_WAVERNN_CHAR_LJSPEECH
   TACOTRON2_GRIFFINLIM_PHONE_LJSPEECH
   TACOTRON2_GRIFFINLIM_CHAR_LJSPEECH

Source Separation
-----------------

Interface
~~~~~~~~~

``SourceSeparationBundle`` instantiates source separation models which take single channel audio and generates multi-channel audio.

.. image:: https://download.pytorch.org/torchaudio/doc-assets/pipelines-sourceseparationbundle.png

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   SourceSeparationBundle

.. rubric:: Tutorials using ``SourceSeparationBundle``

.. minigallery:: torchaudio.pipelines.SourceSeparationBundle

Pretrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   CONVTASNET_BASE_LIBRI2MIX
   HDEMUCS_HIGH_MUSDB_PLUS
   HDEMUCS_HIGH_MUSDB

Squim Objective
---------------

Interface
~~~~~~~~~

:py:class:`SquimObjectiveBundle` defines speech quality and intelligibility measurement (SQUIM) pipeline that can predict **objecive** metric scores given the input waveform.

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   SquimObjectiveBundle

Pretrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   SQUIM_OBJECTIVE

Squim Subjective
----------------

Interface
~~~~~~~~~

:py:class:`SquimSubjectiveBundle` defines speech quality and intelligibility measurement (SQUIM) pipeline that can predict **subjective** metric scores given the input waveform.

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_class.rst

   SquimSubjectiveBundle

Pretrained Models
~~~~~~~~~~~~~~~~~

.. autosummary::
   :toctree: generated
   :nosignatures:
   :template: autosummary/bundle_data.rst

   SQUIM_SUBJECTIVE