index.rst 14.5 KB
Newer Older
moto's avatar
moto committed
1
2
3
Torchaudio Documentation
========================

moto's avatar
moto committed
4
5
.. image:: _static/img/logo.png

moto's avatar
moto committed
6
7
8
Torchaudio is a library for audio and signal processing with PyTorch.
It provides I/O, signal and data processing functions, datasets,
model implementations and application components.
Brian Johnson's avatar
Brian Johnson committed
9

moto's avatar
moto committed
10
11
12
..
   Generate Table Of Contents (left navigation bar)
   NOTE: If you are adding tutorials, add entries to toctree and customcarditem below
13

moto's avatar
moto committed
14
15
16
.. toctree::
   :maxdepth: 1
   :caption: Torchaudio Documentation
moto's avatar
moto committed
17
   :hidden:
18

moto's avatar
moto committed
19
   Index <self>
moto's avatar
moto committed
20
   supported_features
moto's avatar
moto committed
21
   feature_classifications
moto's avatar
moto committed
22
   logo
23
   references
24

25
26
27
28
29
30
31
32
33
34
.. toctree::
   :maxdepth: 2
   :caption: Installation
   :hidden:

   installation
   build
   build.linux
   build.windows
   build.jetson
35
   build.ffmpeg
36

moto's avatar
moto committed
37
38
39
40
41
42
43
44
45
46
.. toctree::
   :maxdepth: 1
   :caption: API Tutorials
   :hidden:

   tutorials/audio_io_tutorial
   tutorials/streamreader_basic_tutorial
   tutorials/streamreader_advanced_tutorial
   tutorials/streamwriter_basic_tutorial
   tutorials/streamwriter_advanced
moto's avatar
moto committed
47
   hw_acceleration_tutorial
moto's avatar
moto committed
48

moto's avatar
moto committed
49
   tutorials/effector_tutorial
moto's avatar
moto committed
50
51
52
53
   tutorials/audio_resampling_tutorial
   tutorials/audio_data_augmentation_tutorial
   tutorials/audio_feature_extractions_tutorial
   tutorials/audio_feature_augmentation_tutorial
54
   tutorials/ctc_forced_alignment_api_tutorial
moto's avatar
moto committed
55

moto's avatar
moto committed
56
   tutorials/oscillator_tutorial
moto's avatar
moto committed
57
   tutorials/additive_synthesis_tutorial
moto's avatar
moto committed
58
   tutorials/filter_design_tutorial
59
   tutorials/subtractive_synthesis_tutorial
moto's avatar
moto committed
60
61
62
63
64
65
66
67
68
69
70
71

   tutorials/audio_datasets_tutorial

.. toctree::
   :maxdepth: 1
   :caption: Pipeline Tutorials
   :hidden:

   tutorials/speech_recognition_pipeline_tutorial
   tutorials/asr_inference_with_ctc_decoder_tutorial
   tutorials/online_asr_tutorial
   tutorials/device_asr
72
   tutorials/forced_alignment_for_multilingual_data_tutorial
moto's avatar
moto committed
73
74
75
76
   tutorials/forced_alignment_tutorial
   tutorials/tacotron2_pipeline_tutorial
   tutorials/mvdr_tutorial
   tutorials/hybrid_demucs_tutorial
77
   tutorials/squim_tutorial
78

moto's avatar
moto committed
79
80
81
82
83
84
85
86
87
88
.. toctree::
   :maxdepth: 1
   :caption: Training Recipes
   :hidden:

   Conformer RNN-T ASR <https://github.com/pytorch/audio/tree/main/examples/asr/librispeech_conformer_rnnt>
   Emformer RNN-T ASR <https://github.com/pytorch/audio/tree/main/examples/asr/emformer_rnnt>
   Conv-TasNet Source Separation <https://github.com/pytorch/audio/tree/main/examples/source_separation>
   HuBERT Pre-training and Fine-tuning (ASR) <https://github.com/pytorch/audio/tree/main/examples/hubert>

89
.. toctree::
90
   :maxdepth: 1
moto's avatar
moto committed
91
   :caption: Python API Reference
moto's avatar
moto committed
92
   :hidden:
93

94
   torchaudio
95
   io
96
97
98
   backend
   functional
   transforms
99
   datasets
100
   models
101
   models.decoder
102
   pipelines
103
   sox_effects
jamarshon's avatar
jamarshon committed
104
   compliance.kaldi
105
   kaldi_io
moto's avatar
moto committed
106
   utils
moto's avatar
moto committed
107
108
109

.. toctree::
   :maxdepth: 1
moto's avatar
moto committed
110
   :caption: Python Prototype API Reference
moto's avatar
moto committed
111
   :hidden:
moto's avatar
moto committed
112

hwangjeff's avatar
hwangjeff committed
113
   prototype
hwangjeff's avatar
hwangjeff committed
114
   prototype.datasets
hwangjeff's avatar
hwangjeff committed
115
   prototype.functional
116
   prototype.models
117
   prototype.pipelines
hwangjeff's avatar
hwangjeff committed
118
   prototype.transforms
119

moto's avatar
moto committed
120
121
122
123
124
125
126
127
128
.. toctree::
   :maxdepth: 1
   :caption: C++ Prototype API Reference
   :hidden:

   libtorchaudio
   libtorchaudio.stream_reader
   libtorchaudio.stream_writer

moto's avatar
moto committed
129
.. toctree::
130
   :maxdepth: 1
moto's avatar
moto committed
131
132
   :caption: PyTorch Libraries
   :hidden:
moto's avatar
moto committed
133

moto's avatar
moto committed
134
135
136
137
138
139
140
   PyTorch <https://pytorch.org/docs>
   torchaudio <https://pytorch.org/audio>
   torchtext <https://pytorch.org/text>
   torchvision <https://pytorch.org/vision>
   TorchElastic <https://pytorch.org/elastic/>
   TorchServe <https://pytorch.org/serve>
   PyTorch on XLA Devices <http://pytorch.org/xla/>
141

moto's avatar
moto committed
142
143
Tutorials
---------
144

moto's avatar
moto committed
145
146
147
148
149
150
151
152
153
.. customcardstart::

.. customcarditem::
   :header: Loading waveform Tensors from files and saving them
   :card_description: Learn how to query/load audio files and save waveform tensors to files, using <code>torchaudio.info</code>, <code>torchaudio.load</code> and <code>torchaudio.save</code> functions.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/audio_io_tutorial.png
   :link: tutorials/audio_io_tutorial.html
   :tags: I/O

154
155
156
157
158
159
160
.. customcarditem::
   :header: CTC Forced Alignment API
   :card_description: Learn how to use TorchAudio's CTC forced alignment API (<code>torchaudio.functional.forced_align</code>).
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/ctc_forced_alignment_api_tutorial.png
   :link: tutorials/ctc_forced_alignment_api_tutorial.html
   :tags: CTC,Forced-Alignment

161
162
163
164
165
166
167
.. customcarditem::
   :header: Forced alignment for multilingual data
   :card_description: Learn how to use align multiligual data using TorchAudio's CTC forced alignment API (<code>torchaudio.functional.forced_align</code>) and a multiligual Wav2Vec2 model.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/forced_alignment_for_multilingual_data_tutorial.png
   :link: tutorials/forced_alignment_for_multilingual_data_tutorial.html
   :tags: Forced-Alignment

moto's avatar
moto committed
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
.. customcarditem::
   :header: Streaming media decoding with StreamReader
   :card_description: Learn how to load audio/video to Tensors using <code>torchaudio.io.StreamReader</code> class.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/streamreader_basic_tutorial.png
   :link: tutorials/streamreader_basic_tutorial.html
   :tags: I/O,StreamReader

.. customcarditem::
   :header: Device input, synthetic audio/video, and filtering with StreamReader
   :card_description: Learn how to load media from hardware devices, generate synthetic audio/video, and apply filters to them with <code>torchaudio.io.StreamReader</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/streamreader_advanced_tutorial.gif
   :link: tutorials/streamreader_advanced_tutorial.html
   :tags: I/O,StreamReader

.. customcarditem::
   :header: Streaming media encoding with StreamWriter
   :card_description: Learn how to save audio/video with <code>torchaudio.io.StreamWriter</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/streamwriter_basic_tutorial.gif
   :link: tutorials/streamwriter_basic_tutorial.html
   :tags: I/O,StreamWriter
Yuekai Zhang's avatar
Yuekai Zhang committed
188

moto's avatar
moto committed
189
190
191
192
193
194
195
196
197
198
199
200
201
202
.. customcarditem::
   :header: Playing media with StreamWriter
   :card_description: Learn how to play audio/video with <code>torchaudio.io.StreamWriter</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/streamwriter_advanced.gif
   :link: tutorials/streamwriter_advanced.html
   :tags: I/O,StreamWriter

.. customcarditem::
   :header: Hardware accelerated video I/O with NVDEC/NVENC
   :card_description: Learn how to setup and use HW accelerated video I/O.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/hw_acceleration_tutorial.png
   :link: hw_acceleration_tutorial.html
   :tags: I/O,StreamReader,StreamWriter

moto's avatar
moto committed
203
204
205
206
207
208
209
.. customcarditem::
   :header: Apply effects and codecs to waveform
   :card_description: Learn how to apply effects and codecs to waveform using <code>torchaudio.io.AudioEffector</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/effector_tutorial.png
   :link: tutorials/effector_tutorial.html
   :tags: Preprocessing

moto's avatar
moto committed
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
.. customcarditem::
   :header: Audio resampling with bandlimited sinc interpolation
   :card_description: Learn how to resample audio tensor with <code>torchaudio.functional.resample</code> and <code>torchaudio.transforms.Resample</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/audio_resampling_tutorial.png
   :link: tutorials/audio_resampling_tutorial.html
   :tags: Preprocessing

.. customcarditem::
   :header: Audio data augmentation
   :card_description: Learn how to use <code>torchaudio.functional</code> and <code>torchaudio.transforms</code> modules to perform data augmentation.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/audio_data_augmentation_tutorial.png
   :link: tutorials/audio_data_augmentation_tutorial.html
   :tags: Preprocessing

.. customcarditem::
   :header: Audio feature extraction
   :card_description: Learn how to use <code>torchaudio.functional</code> and <code>torchaudio.transforms</code> modules to extract features from waveform.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/audio_feature_extractions_tutorial.png
   :link: tutorials/audio_feature_extractions_tutorial.html
   :tags: Preprocessing

.. customcarditem::
   :header: Audio feature augmentation
   :card_description: Learn how to use <code>torchaudio.functional</code> and <code>torchaudio.transforms</code> modules to perform feature augmentation.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/audio_feature_augmentation_tutorial.png
   :link: tutorials/audio_feature_augmentation_tutorial.html
   :tags: Preprocessing

moto's avatar
moto committed
238
239
240
241
242
243
244
.. customcarditem::
   :header: Generating waveforms with oscillator
   :card_description:
   :image: _images/sphx_glr_oscillator_tutorial_003.png
   :link: tutorials/oscillator_tutorial.html
   :tags: DSP

moto's avatar
moto committed
245
246
247
248
249
250
251
.. customcarditem::
   :header: Additive Synthesis
   :card_description:
   :image: _images/sphx_glr_additive_synthesis_tutorial_001.png
   :link: tutorials/additive_synthesis_tutorial.html
   :tags: DSP

moto's avatar
moto committed
252
253
254
255
256
257
258
.. customcarditem::
   :header: Designing digital filters
   :card_description:
   :image: _images/sphx_glr_filter_design_tutorial_001.png
   :link: tutorials/filter_design_tutorial.html
   :tags: DSP

259
260
261
262
263
264
265
.. customcarditem::
   :header: Subtractive Synthesis
   :card_description:
   :image: _images/sphx_glr_subtractive_synthesis_tutorial_002.png
   :link: tutorials/subtractive_synthesis_tutorial.html
   :tags: DSP

moto's avatar
moto committed
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
.. customcarditem::
   :header: Audio dataset
   :card_description: Learn how to use <code>torchaudio.datasets</code> module.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/audio_datasets_tutorial.png
   :link: tutorials/audio_datasets_tutorial.html
   :tags: Dataset

.. customcarditem::
   :header: AM inference with Wav2Vec2
   :card_description: Learn how to perform acoustic model inference with Wav2Vec2 (<code>torchaudio.pipelines.Wav2Vec2ASRBundle</code>).
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/speech_recognition_pipeline_tutorial.png
   :link: tutorials/speech_recognition_pipeline_tutorial.html
   :tags: ASR,wav2vec2

.. customcarditem::
   :header: LM inference with CTC Beam Seach Decoder
   :card_description: Learn how to perform ASR beam search decoding with lexicon and language model, using <code>torchaudio.models.decoder.ctc_decoder</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/asr_inference_with_ctc_decoder_tutorial.png
   :link: tutorials/asr_inference_with_ctc_decoder_tutorial.html
   :tags: Pipelines,ASR,wav2vec2,CTC-Decoder

.. customcarditem::
   :header: Online ASR with Emformer RNN-T
   :card_description: Learn how to perform online ASR with Emformer RNN-T (<code>torchaudio.pipelines.RNNTBundle</code>) and <code>torchaudio.io.StreamReader</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/online_asr_tutorial.gif
   :link: tutorials/online_asr_tutorial.html
   :tags: Pipelines,ASR,RNNT,StreamReader
Yuekai Zhang's avatar
Yuekai Zhang committed
293

moto's avatar
moto committed
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
.. customcarditem::
   :header: Real-time microphone ASR with Emformer RNN-T
   :card_description: Learn how to transcribe speech fomr microphone with Emformer RNN-T (<code>torchaudio.pipelines.RNNTBundle</code>) and <code>torchaudio.io.StreamReader</code>.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/device_asr.png
   :link: tutorials/device_asr.html
   :tags: Pipelines,ASR,RNNT,StreamReader

.. customcarditem::
   :header: Forced Alignment with Wav2Vec2
   :card_description: Learn how to align text to speech with Wav2Vec 2 (<code>torchaudio.pipelines.Wav2Vec2ASRBundle</code>).
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/forced_alignment_tutorial.png
   :link: tutorials/forced_alignment_tutorial.html
   :tags: Pipelines,Forced-Alignment,wav2vec2

.. customcarditem::
   :header: Text-to-Speech with Tacotron2
   :card_description: Learn how to generate speech from text with Tacotron2 (<code>torchaudio.pipelines.Tacotron2TTSBundle</code>).
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/tacotron2_pipeline_tutorial.png
   :link: tutorials/tacotron2_pipeline_tutorial.html
   :tags: Pipelines,TTS-(Text-to-Speech)
Yuekai Zhang's avatar
Yuekai Zhang committed
314

moto's avatar
moto committed
315
316
317
318
319
320
321
322
323
324
325
326
327
328
.. customcarditem::
   :header: Speech Enhancement with MVDR Beamforming
   :card_description: Learn how to improve speech quality with MVDR Beamforming.
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/mvdr_tutorial.png
   :link: tutorials/mvdr_tutorial.html
   :tags: Pipelines,Speech-Enhancement

.. customcarditem::
   :header: Music Source Separation with Hybrid Demucs
   :card_description: Learn how to perform music source separation with pre-trained Hybrid Demucs (<code>torchaudio.pipelines.SourceSeparationBundle</code>).
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/hybrid_demucs_tutorial.png
   :link: tutorials/hybrid_demucs_tutorial.html
   :tags: Pipelines,Source-Separation

329
330
331
332
333
334
.. customcarditem::
   :header: Torchaudio-Squim: Non-intrusive Speech Assessment in TorchAudio
   :card_description: Learn how to estimate subjective and objective metrics with pre-trained TorchAudio-SQUIM models (<code>torchaudio.pipelines.SQUIMObjective</code>).
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/squim_tutorial.png
   :link: tutorials/squim_tutorial.html
   :tags: Pipelines,Speech Assessment,Speech Enhancement
moto's avatar
moto committed
335
.. customcardend::
336
337
338


Citing torchaudio
339
-----------------
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355

If you find torchaudio useful, please cite the following paper:

- Yang, Y.-Y., Hira, M., Ni, Z., Chourdia, A., Astafurov, A., Chen, C., Yeh, C.-F., Puhrsch, C.,
  Pollack, D., Genzel, D., Greenberg, D., Yang, E. Z., Lian, J., Mahadeokar, J., Hwang, J.,
  Chen, J., Goldsborough, P., Roy, P., Narenthiran, S., Watanabe, S., Chintala, S.,
  Quenneville-Bélair, V, & Shi, Y. (2021).
  TorchAudio: Building Blocks for Audio and Speech Processing. arXiv preprint arXiv:2110.15018.


In BibTeX format:

.. code-block:: bibtex

    @article{yang2021torchaudio,
      title={TorchAudio: Building Blocks for Audio and Speech Processing},
356
357
358
359
360
361
362
363
      author={Yao-Yuan Yang and Moto Hira and Zhaoheng Ni and
              Anjali Chourdia and Artyom Astafurov and Caroline Chen and
              Ching-Feng Yeh and Christian Puhrsch and David Pollack and
              Dmitriy Genzel and Donny Greenberg and Edward Z. Yang and
              Jason Lian and Jay Mahadeokar and Jeff Hwang and Ji Chen and
              Peter Goldsborough and Prabhat Roy and Sean Narenthiran and
              Shinji Watanabe and Soumith Chintala and
              Vincent Quenneville-Bélair and Yangyang Shi},
364
365
366
      journal={arXiv preprint arXiv:2110.15018},
      year={2021}
    }