GitLab

* fix typos throughout the code base

* fix grammar

* revert formatting changes to gallery

* revert 'an uXX'

* remove 'number of the best'

* [fbsync] Bkorbar/pyavapi (#6943)

Summary:
* Test: add backend parameter

* VideoReader object now works on backend

* Frame reading now passes

* Keyframe seek now passes

* Pyav backend now supports metadata

* changes in test to reflect GPU decoder change

* Linter?

* Test GPU output

* Addressing Joao's comments

* lint

* lint

* Revert "Test GPU output"

This reverts commit f62e955d7dc81bcb23b40d58ea75413b9b62e76d.

* lint?

* lint

* lint

* Address issues in build?

* hopefully doc fix

* Arrgh

* arrgh

* fix typos

* fix input options

* remove read from memory option in pyav

* skip read from mem test for gpu and pyab be

* fix test

* remove unused import

* Hack to get reading from memory work with pyav

* patch audio test

* gallery change in a hope that docs won't break

* check video decoder inside io

* adding missing lib loading code

* remove unused input

Reviewed By: YosuaMichael

Differential Revision: D41376278

fbshipit-source-id: 1575f63a524227e69187990d88973fc50f705217
Co-authored-by: Bruno Korbar <bkorbar@quansight.com>
Co-authored-by: Joao Gomes <jdsgomes@fb.com>

* cherrypick chages made during fbsync
Co-authored-by: Joao Gomes <jdsgomes@meta.com>
Co-authored-by: Bruno Korbar <bkorbar@quansight.com>

* Test: add backend parameter

* VideoReader object now works on backend

* Frame reading now passes

* Keyframe seek now passes

* Pyav backend now supports metadata

* changes in test to reflect GPU decoder change

* Linter?

* Test GPU output

* Addressing Joao's comments

* lint

* lint

* Revert "Test GPU output"

This reverts commit f62e955d7dc81bcb23b40d58ea75413b9b62e76d.

* lint?

* lint

* lint

* Address issues in build?

* hopefully doc fix

* Arrgh

* arrgh

* fix typos

* fix input options

* remove read from memory option in pyav

* skip read from mem test for gpu and pyab be

* fix test

* remove unused import

* Hack to get reading from memory work with pyav

* patch audio test

* gallery change in a hope that docs won't break

* check video decoder inside io

* adding missing lib loading code

* remove unused input
Co-authored-by: Bruno Korbar <bkorbar@quansight.com>
Co-authored-by: Joao Gomes <jdsgomes@fb.com>

* fix bug in output format for pyav

* add read from memory with constructor overload

* Revert "add read from memory with constructor overload"

This reverts commit 14cbbab239165be05096fd6cbb88cb0448502436.

* run ufmt

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Dataset creation now supports "new" version of Kinetics dataset

* remove unnecessary warning for now

* provide kinetics option

* new reading somehow doesn't need BHWC to BCHW transform

* Addressing minor comments

* Adding kinetics deprication warning for the old Kinetics400 class

* lint error

* Update torchvision/datasets/kinetics.py
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* Updating README

* Remove BHWC to BCHW

* Put warning back

* formatting
Co-authored-by: Bruno Korbar <bkorbar@quansight.com>
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

* Fix shape mismatch error

* Update offset
Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* add api usage log for io

* cover VideoReader

* cover c++ APIs

* add api usage log for io

* cover VideoReader

* cover c++ APIs

* add _cpp suffix to c++ APIs

* use new API and change cpp format

* remove _cpp suffix

* adopt new API

* lint

Co-authored-by: Nicolas Hug <nicolashug@fb.com>

* add ufmt as code formatter

* cleanup

* quote ufmt requirement

* split imports into more groups

* regenerate circleci config

* fix CI

* clarify local testing utils section

* use ufmt pre-commit hook

* split relative imports into local category

* Revert "split relative imports into local category"

This reverts commit f2e224cde2008c56c9347c1f69746d39065cdd51.

* pin black and usort dependencies

* fix local test utils detection

* fix ufmt rev

* add reference utils to local category

* fix usort config

* remove custom categories sorting

* Run pre-commit without fixing flake8

* got a double import in merge
Co-authored-by: Nicolas Hug <nicolashug@fb.com>

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Fixed audio-video synchronisation problem in read_video() when using  as unit

* Addressed review comments

* Added unit test

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

Co-authored-by: Vasilis Vryniotis <datumbox@users.noreply.github.com>

* Add option to write audio to video file

Summary:
I was trying to use torchvision's `write_video` function and realized there was no option to add in the audio.

Thus, this diff contains the changes necessary such that this is possible. This is my first time trying to contribute to this project, so be as harsh as you need!

Reviewed By: fmassa

Differential Revision: D21480083

fbshipit-source-id: 2e11f2c8728d42f86c94068f75b843793d5a94aa

* Fix typo

* Try fix Windows

* Disable test on Windows
Co-authored-by: Joanna Bitton <jbitton@fb.com>

* enable typing check for torchvision.io

* fix existing errors

* Update torchvision/io/_video_opt.py
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* add ignores for FileFinder

* use python 3 type hints

* lint

* video_opt

* video

* try quote av type hints

* revert from .dim() to .ndim

* revert changes to _video_opt.py and ignore errors

* fix type hints

* fix type hints for read_video_timestamps

* change offset int to float

* remove unused import
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Refactoring to use contexts managers, list comprehensions when more idiomatic, and minor renaming to help reader clarity (#2335)

* Refactoring to use contexts managers, list comprehensions when more idiomatic, and minor renaming to help reader clarity.

* Fix flake8 warning in video_utils.py

Fix: PyAV does not support floating point numbers with decimals as FPS when writing and will throw in case this constraint is not satisfied. (#2334)

* get pts directly instead of storing full frames to get pts later

* fix linting

* add initial pts value
sort pts

* catch decoding errors for read_video_timestamp

* get pts directly instead of storing full frames to get pts later

* fix linting

* add initial pts value
sort pts

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/2077

Pull Request resolved: https://github.com/facebookresearch/SlowFast/pull/164

This is a follow-up diff from D18720474

We will be releasing a new version of torchvision soon and the signature of those functions is not ready yet, following my comment in https://our.intern.facebook.com/intern/diff/D18720474/?transaction_id=561239541337402



Reviewed By: stephenyan1231

Differential Revision: D20914571

fbshipit-source-id: 1a7560b8f8e46ab42ef376c50b494a4f73923e94
Co-authored-by: Francisco Massa <fmassa@fb.com>

* Base decoder for video. (#1747)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1747

Pull Request resolved: https://github.com/pytorch/vision/pull/1746

Added the implementation of ffmpeg based decoder with functionality that can be used in VUE and TorchVision.

Reviewed By: fmassa

Differential Revision: D19358914

fbshipit-source-id: abb672f89bfaca6351dda2354f0d35cf8e47fa0f

* Integrated base decoder into VideoReader class and video_utils.py (#1766)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1766

Replaced FfmpegDecoder (incompativle with VUE) by base decoder (compatible with VUE).
Modified python utilities video_utils.py for internal simplification. Public interface got preserved.

Reviewed By: fmassa

Differential Revision: D19415903

fbshipit-source-id: 4d7a0158bd77bac0a18732fe4183fdd9a57f6402

* Optimizating base decoder performance. (#1852)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1852

Changed base decoder internals for a faster clip processing.

Reviewed By: stephenyan1231

Differential Revision: D19748379

fbshipit-source-id: 58a435f0a0b25545e7bd1a3edb0b1d558176a806

* Minor fix and decoder class members access.

Summary:
Found and fix a bug in cropping algorithm (simple mistyping).
Also derived classes need access to some decoder class members, like initialization parameters - make it protected.

Reviewed By: stephenyan1231, fmassa

Differential Revision: D19895076

fbshipit-source-id: 691336c8e18526b085ae5792ac3546bc387a6db9

* Added missing header for less dependencies. (#1898)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1898

Include streams/samplers shouldn't depend on decoder headers. Add dependencies directly to the place where they are required.

Reviewed By: stephenyan1231

Differential Revision: D19911404

fbshipit-source-id: ef322a053708405c02cee4562b456b1602fb12fc

* Implemented VUE Asynchronous Decoder

Summary: For Mothership we have found that asynchronous decoder provides a better performance.

Differential Revision: D20026194

fbshipit-source-id: 627b91844b4e3f917002031dd32cb19c239f4ba8

* fix a bug in API read_video_from_memory (#1942)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1942

In D18720474, it introduces a bug in `read_video_from_memory` API. Thank weiyaowang for reporting it.

Reviewed By: weiyaowang

Differential Revision: D20270179

fbshipit-source-id: 66348c99a5ad1f9129b90e934524ddfaad59de03

* extend decoder to support new video_max_dimension argument (#1924)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1924

Extend `video reader` decoder python API in Torchvision to support a new argument `video_max_dimension`. This enables the new video decoding use cases. When setting `video_width=0`, `video_height=0`, `video_min_dimension != 0`, and `video_max_dimension != 0`, we can rescale the video clips so that its spatial resolution (height, width) becomes
 - (video_min_dimension, video_max_dimension) if original height < original width
 - (video_max_dimension, video_min_dimension) if original height >= original width

This is useful at video model testing stage, where we perform fully convolution evaluation and take entire video frames without cropping as input. Previously, for instance we can only set `video_width=0`, `video_height=0`, `video_min_dimension = 128`, which will preserve aspect ratio. In production dataset, there are a small number of videos where aspect ratio is either extremely large or small, and when the shorter edge is rescaled to 128, the longer edge is still large. This will easily cause GPU memory OOM when we sample multiple video clips, and put them in a single minibatch.

Now, we can set (for instance) `video_width=0`, `video_height=0`, `video_min_dimension = 128` and `video_max_dimension = 171` so that the rescale resolution is either (128, 171) or (171, 128) depending on whether original height is larger than original width. Thus, we are less likely to have gpu OOM because the spatial size of video clips is determined.

Reviewed By: putivsky

Differential Revision: D20182529

fbshipit-source-id: f9c40afb7590e7c45e6908946597141efa35f57c

* Fixing samplers initialization (#1967)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1967



No-ops for torchvision diff, which fixes samplers.

Differential Revision: D20397218

fbshipit-source-id: 6dc4d04364f305fbda7ca4f67a25ceecd73d0f20

* Exclude C++ test files
Co-authored-by: Yuri Putivsky <yuri@fb.com>
Co-authored-by: Zhicheng Yan <zyan3@fb.com>

* torchscriptable functions for video io (#1653)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1653



created new torchscriptable video io functions as part of the api: read_video_meta_data_from_memory and read_video_from_memory.

Updated the implementation of some of the internal functions to be torchscriptable.

Reviewed By: stephenyan1231

Differential Revision: D18720474

fbshipit-source-id: 4ee646b66afecd2dc338a71fd8f249f25a3263bc

* BugFix
Co-authored-by: Jon Guerin <54725679+jguerin-fb@users.noreply.github.com>

* Unify video backend interfaces

* Remove reference cycle

* Make functions private and enable tests on OSX

* Disable test if video_reader backend not available

* Lint

* Fix import after refactoring

* Fix lint

* Handle corrupted video headers in io

* Catch exceptions while decoding partly-corrupted files

* Add more tests

modified code of io.read_video and io.read_video_timestamps to intepret pts values in seconds (#1331)

* modified code of io.read_video and io.read_video_timestamps to interpret pts values in seconds

* changed default value for pts_unit to pts, corrected formatting

* hanndliing both fractions and floats for start_pts and end_pts, added test cases for pts_unit sec

* moved unit conversion logic to _read_from_stream method

Following a similar line of inquiry to pytorch/audio#217

* Expose docs for io and ops package

Had do modify the docstrings to use Napoleon NumPy style, because Napoleon Google Style doesn't support multiple return arguments

* Add video section

* Copy classification scripts for video classification

* Initial version of video classification

* add version

* Training of r2plus1d_18 on kinetics work

Gives even slightly better results than expected, with 57.336 top1 clip accuracy. But we count some clips twice in this evaluation

* Cleanups on training script

* Lint

* Minor improvements

* Remove some hacks

* Lint

* Optimize read_video_timestamps for some formats

* Add some tests

* Miscellaneous fixes and improvements

* Guard against videos without video stream

* Fix lint

* Add test for packed b-frames videos

* Fix missing import

Also extend video saving to support different codecs and options. Notably, we can now save with lossless compression

* Properly order videos with B-frames

* seek doesn't seek to pts, but dts

Find a way of overcoming this problem.