Skip to content

GitLab

Switch branch/tag
  1. 01 Apr, 2023 1 commit
    • moto's avatar
      Add AudioEffector (#3163) · a4036248
      moto authored 
      Summary:
This commit adds a new feature AudioEffector, which can be used to
apply various effects and codecs to waveforms in Tensor.

Under the hood it uses StreamWriter and StreamReader to apply
filters and encode/decode.

This is going to replace the deprecated `apply_codec` and
`apply_sox_effect_tensor` functions.

It can also perform online, chunk-by-chunk filtering.

Tutorial to follow.

closes https://github.com/pytorch/audio/issues/3161

Pull Request resolved: https://github.com/pytorch/audio/pull/3163

Reviewed By: hwangjeff

Differential Revision: D44576660

Pulled By: mthrok

fbshipit-source-id: 2c5cc87082ab431315d29d56d6ac9efaf4cf7aeb
      a4036248
  3. 31 Mar, 2023 3 commits
  5. 30 Mar, 2023 2 commits
    • moto's avatar
      Support encode spec change in StreamWriter (#3207) · 1b648626
      moto authored 
      Summary:
This commit adds support for changing the spec of media
(such as sample rate, #channels, image size and frame rate)
on-the-fly at encoding time.

The motivation behind this addition is that certain media
formats support only limited number of spec, and it is
cumbersome to require client code to change the spec
every time.

For example, OPUS supports only 48kHz sampling rate, and
vorbis only supports stereo.

To make it easy to work with media of different formats,
this commit makes it so that anything that's not compatible
with the format is automatically converted, and allows
users to specify the override.

Notable implementation detail is that, for sample format and
pixel format, the default value of encoder has higher precedent
to source value, while for other attributes like sample rate and
#channels, the source value has higher precedent as long as
they are supported.

Pull Request resolved: https://github.com/pytorch/audio/pull/3207

Reviewed By: nateanl

Differential Revision: D44439622

Pulled By: mthrok

fbshipit-source-id: 09524f201d485d201150481884a3e9e4d2aab081
      1b648626
    • moto's avatar
      Support changing the number of channels in StreamReader (#3216) · 4bc4ca75
      moto authored 
      Summary:
This commit adds `num_channels` argument,
which allows one to change the number of channels on-the-fly.

Pull Request resolved: https://github.com/pytorch/audio/pull/3216

Reviewed By: hwangjeff

Differential Revision: D44516925

Pulled By: mthrok

fbshipit-source-id: 3e5a11b3fdbb19071f712a8148e27aff60341df3
      4bc4ca75
  7. 29 Mar, 2023 3 commits
    • Moto Hira's avatar
      Reduce io tests (#3217) · 09ccf7cc
      Moto Hira authored 
      Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/3217

This commit removes some tests for file-like object from StreamWriter test.

The rational is that testing things after the output file is opened are
same for file-like object and regular files. Things like filter-graph and
encoder format change does not affect how the encoded bynary are written.

Reviewed By: hwangjeff

Differential Revision: D44518626

fbshipit-source-id: 821ec20deca92e5e5c85bf4d47997eed51735374
      09ccf7cc
    • moto's avatar
      Reuse HW device context in GPU encoder (#3215) · c76fd58b
      moto authored 
      Summary:
In https://github.com/pytorch/audio/issues/3178, a mechanism to cache HW device context was introduced.
This commit applies the reuse in StreamWriter, so that
when using GPU video decoding and encoding, they are shared.

This gives back about 250 - 300 MB of GPU memory.

 ---

Q: What is HW device context?
From https://ffmpeg.org/doxygen/4.1/structAVHWDeviceContext.html#details
> This struct aggregates all the (hardware/vendor-specific) "high-level" state, i.e.
>
> state that is not tied to a concrete processing configuration. E.g., in an API that supports hardware-accelerated encoding and decoding, this struct will (if possible) wrap the state that is common to both encoding and decoding and from which specific instances of encoders or decoders can be derived.

Pull Request resolved: https://github.com/pytorch/audio/pull/3215

Reviewed By: nateanl

Differential Revision: D44504051

Pulled By: mthrok

fbshipit-source-id: 77579cdc8bd9e9b8a218e3f29031d091cda83860
      c76fd58b
    • moto's avatar
      Remove the note about AAC (#3214) · c07a96ab
      moto authored 
      Summary:
There is a part of StreamWriter tutorial that warns about corrupted AAC audio output, but this is no longer relevant thus this commit deletes it.

Pull Request resolved: https://github.com/pytorch/audio/pull/3214

Reviewed By: nateanl

Differential Revision: D44504030

Pulled By: mthrok

fbshipit-source-id: 4d26d582e9fb87d4e6fa674c05fe3192bc223eef
      c07a96ab
  9. 28 Mar, 2023 4 commits
  11. 27 Mar, 2023 2 commits
    • hwangjeff's avatar
      Revise encoder config arg and docstrings (#3203) · b1de9f1a
      hwangjeff authored 
      Summary:
For `StreamWriter`,
* Renames arg `config` to codec_config`.
* Renames struct `EncodingConfig` and dataclass `EncodeConfig` to `CodecConfig`.
* Adds docstrings for arg codec_config`.
* Updates `chunk` to `frames` in `write_*_chunk` methods.

Pull Request resolved: https://github.com/pytorch/audio/pull/3203

Reviewed By: mthrok

Differential Revision: D44350153

Pulled By: hwangjeff

fbshipit-source-id: 1b940b1366a43ec0565c362bfcbf62744088b343
      b1de9f1a
    • Moto Hira's avatar
      Refactor the initialization of EncodeProcess (#3205) · 4eac61a3
      Moto Hira authored 
      Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/3205

This commit refactors the initialization of EncodeProcess.

Interface-wise, the signature of the constructor of EncodeProcess
has made simpler just to take rvalues of its components, and the
initialization of the components have been moved to helper functions.

Implementat-wise, the order that the components are initialized is
revised, and the source of initialization parameters is also revised.

For example, the original implementation first creates AVCodecContext,
and passes it around to create the other components. This relied on
an assumption that parameters AVCodecContext has (such as image size
and sample rate) are same as the source data. This is not always right,
and as we will introduce custom filter graph and allow on-the-fly
transform of rates and dimensions, it will become even less correct.

The new initialization constructs source AVFrame, TensorConverter and
FilterGraph from source attributes. This makes it easy to introduce
on-the-fly transform.

Reviewed By: nateanl

Differential Revision: D44360650

fbshipit-source-id: bf0e77dc1a5a40fc8e9870c50d07339d812762e8
      4eac61a3
  13. 25 Mar, 2023 1 commit
    • moto's avatar
      Properly set #samples passed to encoder (#3204) · d8a37a21
      moto authored 
      Summary:
Some audio encoders expect specific, exact number of samples described as in `AVCodecContext.frame_size`.

The `AVFrame.nb_samples` is set for the frames passed to `AVFilterGraph`,
but frames coming out of the graph do not necessarily have the same numbr of frames.

This causes issues with encoding OPUS (among others).

This commit fixes it by inserting `asetnsamples` to filter graph if a fixed number of samples is requested.

Note:
It turned out that FFmpeg 4.1 has issue with OPUS encoding. It does not properly discard some sample.
We should probably move the minimum required FFmpeg to 4.2, but I am not sure if we can enforce it via ABI.
Work around will be to issue an warning if encoding OPUS with 4.1. (follow-up)

Pull Request resolved: https://github.com/pytorch/audio/pull/3204

Reviewed By: nateanl

Differential Revision: D44374668

Pulled By: mthrok

fbshipit-source-id: 10ef5333dc0677dfb83c8e40b78edd8ded1b21dc
      d8a37a21
  15. 23 Mar, 2023 6 commits
  17. 22 Mar, 2023 2 commits
    • moto's avatar
      Fix oscillator bank test (#3196) · aa590a1b
      moto authored 
      Summary:
Follow up of https://github.com/pytorch/audio/pull/3083

Pull Request resolved: https://github.com/pytorch/audio/pull/3196

Reviewed By: nateanl

Differential Revision: D44308940

Pulled By: mthrok

fbshipit-source-id: e3ef27656e74c28ae78b767517d8e0ba3a9ac4a6
      aa590a1b
    • atalman's avatar
      [Nova] Windows Adopt ffmpeg build to be executed from github actions (#3193) · da86fbc7
      atalman authored 
      Summary:
Adopt ffmpeg build to be executed from github actions for windows

Tested by manually invoking this script:
```
c:\actions-runner\_work\test-infra\test-infra\pytorch\audio
Chocolatey v1.2.1
Installing the following packages:
msys2
By installing, you accept licenses for the packages.
msys2 v20230318.0.0 already installed.
 Use --force to reinstall, specify a version to install, or try upgrade.

Chocolatey installed 0/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Warnings:
 - msys2 - msys2 v20230318.0.0 already installed.
 Use --force to reinstall, specify a version to install, or try upgrade.

Did you know the proceeds of Pro (and some proceeds from other
 licensed editions) go into bettering the community infrastructure?
 Your support ensures an active community, keeps Chocolatey tip-top,
 plus it nets you some awesome features!
 https://chocolatey.org/compare
warning: base-devel-...
      da86fbc7
  19. 21 Mar, 2023 6 commits
  21. 20 Mar, 2023 3 commits
    • Moto Hira's avatar
      Refactor StreamReader internals (#3184) · c17226a0
      Moto Hira authored 
      Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/3184

Tweak internals of StreamReader
1. Pass time_base to Buffer class so that
    * no need to pass frame_duration separately
    * Conversion of PTS to double type can be delayed until when it's popped
2. Merge `get_output_timebase` method into `get_output_stream_info`.
3. If filter description is not provided, fill in null filter at top-level StreamReader
4. Expose filer and filter description from Sink class to get rid of wrapper get methods.

Reviewed By: nateanl

Differential Revision: D44207976

fbshipit-source-id: f25ac9be69c9897e9dcec0c6e978f29b83b166e8
      c17226a0
    • Moto Hira's avatar
      Fix GPU memory leak on StreamReader (#3186) · 9533d300
      Moto Hira authored 
      Summary:
Pull Request resolved: https://github.com/pytorch/audio/pull/3186

Fix the GPU memory leak introduced in https://github.com/pytorch/audio/pull/3183

The HW frames context is owned by AVCodecContext.
The removed `av_buffer_ref` call increased the ferenrence counting unnecessarily,
and prevented AVCodecContext from feeing the resource.

(Note: this ignores all push blocking failures!)

Reviewed By: nateanl

Differential Revision: D44231876

fbshipit-source-id: 9be2c33049dd02a3fa82a85271de7fb62e5b09ea
      9533d300
    • moto's avatar
      Support CUDA frame in FilterGraph (#3183) · c5b96558
      moto authored 
      Summary:
This commit adds CUDA frame support to FilterGraph

It initializes and attaches CUDA frames context to FilterGraph,
so that CUDA frames can be processed in FilterGraph.

As a result, it enables
1. CUDA filter support such as `scale_cuda`
2. Properly retrieve the pixel format coming out of FilterGraph when
   CUDA HW acceleration is enabled. (currently it is reported as "cuda")

Resolves https://github.com/pytorch/audio/issues/3159

Pull Request resolved: https://github.com/pytorch/audio/pull/3183

Reviewed By: hwangjeff

Differential Revision: D44183722

Pulled By: mthrok

fbshipit-source-id: 522d21039c361ddfaa87fa89cf49c19d210ac62f
      c5b96558
  23. 17 Mar, 2023 4 commits
  25. 16 Mar, 2023 2 commits
    • jiyuntu-eero's avatar
      Fix initialization of `get_trellis`. (#3172) · a6b34a5d
      jiyuntu-eero authored 
      Summary:
Fix https://github.com/pytorch/audio/issues/3166. In `get_trellis` method, the index of blank symbol is regarded as 0 by default. It should be changed to `blank_id`.

Pull Request resolved: https://github.com/pytorch/audio/pull/3172

Reviewed By: mthrok

Differential Revision: D44090889

Pulled By: nateanl

fbshipit-source-id: d119f4ded895d31aeefd59f8d975224870100264
      a6b34a5d
    • moto's avatar
      Refactor Tensor conversion in StreamReader (#3170) · 014d7140
      moto authored 
      Summary:
Currently, when the Buffer converts AVFrame* to torch::Tensor,
it checks the format at each time a frame is passed, and
perform the conversion.

This commit changes it so that the conversion operation is
pre-instantiated at the time outside stream is configured.

It introduces Converter implementations for various formats,
and use template to embed them in Buffer class.
This way, branching like if/switch are eliminated from
decoding path.

Pull Request resolved: https://github.com/pytorch/audio/pull/3170

Reviewed By: xiaohui-zhang

Differential Revision: D44048293

Pulled By: mthrok

fbshipit-source-id: 30d8b240a5695d7513f499ce17853f2f0ffcab9f
      014d7140
  27. 15 Mar, 2023 1 commit
    • Carl Parker's avatar
      Enhance UX on TorchAudio pages to improve awareness of doc versioning (#3167) · 92f2ea89
      Carl Parker authored 
      Summary:
- Boldface the version-selection UX and increase size by three percent.
- Add text to breadcrumbs to indicate version and stability.
- New `breadcrumbs.html` in `_templates` overrides Sphinx version.

I create a new variable in `conf.py`, **version_stable**, which has the version number for the most-recent stable release. I define this variable in the **html_context** dictionary so that it is visible to the templates.

I use this approach because I was not able to find any other way of discerning the current stable release during the build. Note that the `versions.html` file--which identifies the current stable release--appears to be available only in the **gh-pages** branch and so it is not available at build time.

However, this means that someone will need to update `conf.py` whenever the current stable release changes.

Pull Request resolved: https://github.com/pytorch/audio/pull/3167

Reviewed By: mthrok

Differential Revision: D44112224

Pulled By: carljparker

fbshipit-source-id: e76f5cb6734a784d161342964459577aa9b64cac
      92f2ea89