GitLab

[FBcode->GH] torchvision decoder. fix artifacts, blury images for frame dimensions not multiple of 8 (#7523) (#7552)
Co-authored-by: Vadim Zubov <vadikrobot@meta.com>

* fix typos throughout the code base

* fix grammar

* revert formatting changes to gallery

* revert 'an uXX'

* remove 'number of the best'

Set pytorch vision decoder probesize for getting stream info based on the value from decode setting (#6900) (#6950)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/6900



1. Pass along the value of probesize from decode setting.
2. allow pytorch vision decoder to set the probesize for getting stream info

Reviewed By: jdsgomes

Differential Revision: D40967782

fbshipit-source-id: 642ae3b67c50b1fdafc20f9814a52fdf5db527d7
Co-authored-by: Chen Liu <cliu7@meta.com>

* add tensor as optional param

* add init from memory

* fix bug

* fix bug

* first working version

* apply formatting and add tests

* simplify tests

* fix tests

* fix wrong variable name

* add path as optional parameter

* add src as optional

* address pr comments

* Fix warning messages

* address pr comments

* make tests stricter

* Revert "make tests stricter"

This reverts commit 6c92e94e8372f381c9496c9f885c2c71b6a4356b.

Summary:
This diff adds **`skipOperationNotPermittedPackets`** to `DecoderParameters`.

Once that is set to `True`, it allows decoder to skip through those packets, that are causing `EPERM` errors and result into stoppage of consuming a stream.

Reviewed By: jdsgomes

Differential Revision: D38732706

fbshipit-source-id: a5cb64e14edda376e6ba240089ebee2e3a9865d0
Co-authored-by: Oleksandr Voietsa <ovoietsa@fb.com>

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/6432



According to the documentation the packet has to be freed after `av_read_frame()` call.
```
 If pkt->buf is NULL, then the packet is valid until the next
 av_read_frame() or until avformat_close_input(). Otherwise the packet
 is valid indefinitely. In both cases the packet must be freed with
 av_packet_unref when it is no longer needed.
```

Differential Revision: D38747612

fbshipit-source-id: 2e4ccc8365d0d97e5da756ff9c1dcdf27ed323f0
Co-authored-by: Oleksandr Voietsa <ovoietsa@fb.com>

Summary: provider callback uses video sampler to copy/scale input frames using sws_scale function, added option to use av_image_copy to copy image planes in case if sws_getContext fails (required to support AV_PIX_FMT_PAL8 pixel format)

Reviewed By: ovoietsa

Differential Revision: D36466749

fbshipit-source-id: 1674f98a362db835d147800c24bdda6f82d2f8ff
Co-authored-by: Sergiy Bilobrov <sergiy@fb.com>

Summary:
Resolves https://www.internalfb.com/tasks/?t=128004042

Caused by TorchVision's PR at https://github.com/pytorch/vision/pull/6322 which was in response to a change on PyTorch Core at https://github.com/pytorch/pytorch/pull/82032

Reviewed By: fmassa

Differential Revision: D38383266

fbshipit-source-id: 3f0ebd04a610031c4720123d1869a851f76455cd

Differential Revision: D36671123

fbshipit-source-id: eb99f293978ed150bca60073115971586f0fa02a
Co-authored-by: Oleksandr Voietsa <ovoietsa@fb.com>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* fix for FFMPEG 5.0

* Attempt to fix memory leak

* early stop when mem alloc fails

* fix c lint error

* fix bug

* revert changes

* remove incorrect av_packet_free

* add `av_packet_free` to subtitle stream

* Addressing subtitle stream comments

* addressing decoder changes

* nit

* addressing datumbox concernsz

* averror push

* addressing comments

* Apply suggestions from code review

* add new error code to documentation

* re-introducing timeout

* fix c++ syntax
Co-authored-by: Bruno Korbar <bkorbar@quansight.com>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Joao Gomes <jdsgomes@fb.com>
Co-authored-by: Joao Gomes <joaopsgomes@gmail.com>

* Add seek in GPU decoder

* Merge the two tests

* Refine unit test

* Return RGB frames as output of GPU decoder

* Move clamp to the conversion function

* Cleaned up a bit

* Remove utility functions from test

* Use data member width directly

* Fix linter error

* [WIP] Add video GPU decoder

* Expose use_dev_frame to python class and handle it internally

* Fixed invalid argument CUDA error

* Fixed empty and missing frames

* Free remaining frames in the queue

* Added nv12 to yuv420 conversion support for host frames

* Added unit test and cleaned up code

* Use CUDA_HOME inside if

* Undo commented out code

* Add Readme

* Remove output_format and use_device_frame optional arguments from the VideoReader API

* Cleaned up init()

* Fix warnings

* Fix python linter errors

* Fix linter issues in setup.py

* clang-format

* Make reformat private

* Member function naming

* Add comments

* Variable renaming

* Code cleanup

* Make return type of decode() void

* Replace printing errors with throwing runtime_error

* Replaced runtime_error with TORCH_CHECK in demuxer.h

* Use CUDAGuard instead of cudaSetDevice

* Remove printf

* Use Tensor instead of uint8* and remove cuMemAlloc/cuMemFree

* Use TORCH_CHECK instead of runtime_error

* Use TORCHVISION_INCLUDE and TORCHVISION_LIBRARY to pass video codec location

* Include ffmpeg_include_dir

* Remove space

* Removed use of runtime_error

* Update Readme

* Check for bsf.h

* Change struct initialisation style

* Clean-up get_operating_point

* Make variable naming convention uniform

* Move checking for bsf.h around

* Fix linter error
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* modify processPacket to support fast seek

* add fastSeek to ProcessPacket decoder definition

* add fastseek flag to DecoderParametersStruct

* add fastseek flag to the process packet call

* no default params in C++ implementation

* enable flag in C++ implementation

* make order of parameters more normal

* register new seek with python api

* [somewhat broken] test suite for keyframes using pyav

* revert " changes

* add type annotations to init

* Adding tests

* linter

* Flake doesn't show up :|

* Change from unitest to pytest syntax

* add return type
Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

* initial commit

* initial commit

* flake8

* last nits
Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

* document video_sampler.

* minor docs for decoder.cpp

* descriptive comments for the stream.c

* descriptive comments for decoder.cpp

* per-stream descriptive comments

* Fixing CLANG hopefully

* addressing prabhat's comments

* typo I think
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Fixed some ffmpeg deprecation warnings in decoder

* Fixed clang error
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Fixed comparison warnings in audio_stream and video_stream

* Fixed clang error

This reverts commit f6b6b510.

* remove unused functions

* shut up annoying warnings via cast

* prabhat's comments

* Fixed clang format error
Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

Co-authored-by: Prabhat Roy <prabhatroy@fb.com>

This reverts commit 3bfdb427.

* multithreading allowed in stream codec context

* numThreads is passed as a decoder parameter. At this stage code should be unchanged

* enabling multithreading in videoReader API

* moving defaults to header files

* replace long with int64_t because torchscript

* docstring for Num threads

* Enable codec-related heuristics as defaults

* Update torchvision/csrc/io/decoder/stream.cpp
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Fixing build errors

* minor docs

* Linting

* updating defaults for the C++ function calls to be single threaded

* adding special case for single threaded stuff
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Summary:
Noticed this copy-paste error while reading D26026259 (https://github.com/pytorch/vision/commit/f16322b596c7dc9e9d67d3b40907694f29e16357) for inspiration to solve a similar problem

Created from Diffusion's 'Open in Editor' feature.

Reviewed By: datumbox

Differential Revision: D26373056

fbshipit-source-id: d67f46243a280e8a59d12bad9dd730c1a3d512c9

Summary:
Upgrading service to platform009 enforces tls1.3 authentication.
Specify tls cert file to decoder through config.
Context: https://fb.workplace.com/groups/2020511478062223

Differential Revision: D26026259

fbshipit-source-id: e89f88ff7e41ec6b6a62e85d9015d78f6377aa65

Summary:
* Reduce unnecessary header inclusions in models and io.

* Move autocast to separate folder and hide autograd implementation in an anonymous namespace.

* Moving files in subfolders.

Reviewed By: fmassa

Differential Revision: D25461523

fbshipit-source-id: 756eeb6848aacaa474de4825ed4c1045d17e2cea