GitLab

* add mypy config

* fix syntax error

* fix annotations in torchvision/utils.py

* add mypy type check to CircleCI

* add mypy cache to ignore files

* try fix CI

* ignore flake8 F821 since it interferes with mypy

* add mypy type check to config generator

* explicitly set config files

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/2077

Pull Request resolved: https://github.com/facebookresearch/SlowFast/pull/164

This is a follow-up diff from D18720474

We will be releasing a new version of torchvision soon and the signature of those functions is not ready yet, following my comment in https://our.intern.facebook.com/intern/diff/D18720474/?transaction_id=561239541337402



Reviewed By: stephenyan1231

Differential Revision: D20914571

fbshipit-source-id: 1a7560b8f8e46ab42ef376c50b494a4f73923e94
Co-authored-by: Francisco Massa <fmassa@fb.com>

Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* Base decoder for video. (#1747)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1747

Pull Request resolved: https://github.com/pytorch/vision/pull/1746

Added the implementation of ffmpeg based decoder with functionality that can be used in VUE and TorchVision.

Reviewed By: fmassa

Differential Revision: D19358914

fbshipit-source-id: abb672f89bfaca6351dda2354f0d35cf8e47fa0f

* Integrated base decoder into VideoReader class and video_utils.py (#1766)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1766

Replaced FfmpegDecoder (incompativle with VUE) by base decoder (compatible with VUE).
Modified python utilities video_utils.py for internal simplification. Public interface got preserved.

Reviewed By: fmassa

Differential Revision: D19415903

fbshipit-source-id: 4d7a0158bd77bac0a18732fe4183fdd9a57f6402

* Optimizating base decoder performance. (#1852)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1852

Changed base decoder internals for a faster clip processing.

Reviewed By: stephenyan1231

Differential Revision: D19748379

fbshipit-source-id: 58a435f0a0b25545e7bd1a3edb0b1d558176a806

* Minor fix and decoder class members access.

Summary:
Found and fix a bug in cropping algorithm (simple mistyping).
Also derived classes need access to some decoder class members, like initialization parameters - make it protected.

Reviewed By: stephenyan1231, fmassa

Differential Revision: D19895076

fbshipit-source-id: 691336c8e18526b085ae5792ac3546bc387a6db9

* Added missing header for less dependencies. (#1898)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1898

Include streams/samplers shouldn't depend on decoder headers. Add dependencies directly to the place where they are required.

Reviewed By: stephenyan1231

Differential Revision: D19911404

fbshipit-source-id: ef322a053708405c02cee4562b456b1602fb12fc

* Implemented VUE Asynchronous Decoder

Summary: For Mothership we have found that asynchronous decoder provides a better performance.

Differential Revision: D20026194

fbshipit-source-id: 627b91844b4e3f917002031dd32cb19c239f4ba8

* fix a bug in API read_video_from_memory (#1942)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1942

In D18720474, it introduces a bug in `read_video_from_memory` API. Thank weiyaowang for reporting it.

Reviewed By: weiyaowang

Differential Revision: D20270179

fbshipit-source-id: 66348c99a5ad1f9129b90e934524ddfaad59de03

* extend decoder to support new video_max_dimension argument (#1924)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1924

Extend `video reader` decoder python API in Torchvision to support a new argument `video_max_dimension`. This enables the new video decoding use cases. When setting `video_width=0`, `video_height=0`, `video_min_dimension != 0`, and `video_max_dimension != 0`, we can rescale the video clips so that its spatial resolution (height, width) becomes
 - (video_min_dimension, video_max_dimension) if original height < original width
 - (video_max_dimension, video_min_dimension) if original height >= original width

This is useful at video model testing stage, where we perform fully convolution evaluation and take entire video frames without cropping as input. Previously, for instance we can only set `video_width=0`, `video_height=0`, `video_min_dimension = 128`, which will preserve aspect ratio. In production dataset, there are a small number of videos where aspect ratio is either extremely large or small, and when the shorter edge is rescaled to 128, the longer edge is still large. This will easily cause GPU memory OOM when we sample multiple video clips, and put them in a single minibatch.

Now, we can set (for instance) `video_width=0`, `video_height=0`, `video_min_dimension = 128` and `video_max_dimension = 171` so that the rescale resolution is either (128, 171) or (171, 128) depending on whether original height is larger than original width. Thus, we are less likely to have gpu OOM because the spatial size of video clips is determined.

Reviewed By: putivsky

Differential Revision: D20182529

fbshipit-source-id: f9c40afb7590e7c45e6908946597141efa35f57c

* Fixing samplers initialization (#1967)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1967



No-ops for torchvision diff, which fixes samplers.

Differential Revision: D20397218

fbshipit-source-id: 6dc4d04364f305fbda7ca4f67a25ceecd73d0f20

* Exclude C++ test files
Co-authored-by: Yuri Putivsky <yuri@fb.com>
Co-authored-by: Zhicheng Yan <zyan3@fb.com>

* torchscriptable functions for video io (#1653)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1653



created new torchscriptable video io functions as part of the api: read_video_meta_data_from_memory and read_video_from_memory.

Updated the implementation of some of the internal functions to be torchscriptable.

Reviewed By: stephenyan1231

Differential Revision: D18720474

fbshipit-source-id: 4ee646b66afecd2dc338a71fd8f249f25a3263bc

* BugFix
Co-authored-by: Jon Guerin <54725679+jguerin-fb@users.noreply.github.com>

* Unify video metadata in VideoClips

* Bugfix

* Make tests a bit more robust

* Unify video backend interfaces

* Remove reference cycle

* Make functions private and enable tests on OSX

* Disable test if video_reader backend not available

* Lint

* Fix import after refactoring

* Fix lint

* Handle corrupted video headers in io

* Catch exceptions while decoding partly-corrupted files

* Add more tests

* extend video reader to support fast video probing

* fix c++ lint

* small fix

* allow to accept input video of type torch.Tensor

* add metadata to video dataset classes. bug fix. more robustness

* query video backend within VideoClips class

* Fix tests

* Fix lint

modified code of io.read_video and io.read_video_timestamps to intepret pts values in seconds (#1331)

* modified code of io.read_video and io.read_video_timestamps to interpret pts values in seconds

* changed default value for pts_unit to pts, corrected formatting

* hanndliing both fractions and floats for start_pts and end_pts, added test cases for pts_unit sec

* moved unit conversion logic to _read_from_stream method

* add _backend argument to __init__() of class VideoClips

* minor fix

* minor fix

* Make backend private in VideoClips

* Fix lint

* [video reader] inception commit

* add method save_metadata to class VideoClips in video_utils.py

* add load_metadata() method to VideoClips class

* add Exception to not catch unexpected events such as memory erros, interrupt

* fix bugs in video_plus.py

* [video reader]remove logging. update setup.py

* remove time measurement in test_video_reader.py

* Remove glog and try making ffmpeg finding more robust

* Add ffmpeg to conda build

* Add ffmpeg to conda build [again]

* Make library path finding more robust

* Missing import

* One more missing fix for import

* Py2 compatibility and change package to av to avoid version conflict with ffmpeg

* Fix for python2

* [video reader] support to decode one stream only (e.g. video/audio stream)

* remove argument _precomputed_metadata_filepath

* remove save_metadata method

* add get_metadata method

* expose _precomputed_metadata and frame_rate arguments in video dataset __init__ method

* remove ssize_t

* remove size_t to pass CI check on Windows

* add PyInit__video_reader function to pass CI check on Windows

* minor fix to define PyInit_video_reader symbol

* Make c++ video reader optional

* Temporarily revert changes to test_io

* Revert changes to python files

* Rename files to make it private

* Fix python lint

* Fix C++ lint

* add a functor object EnumClassHash to make Enum class instances usable as key type of std::unordered_map

* fix cpp format check

Following a similar line of inquiry to pytorch/audio#217

* Expose docs for io and ops package

Had do modify the docstrings to use Napoleon NumPy style, because Napoleon Google Style doesn't support multiple return arguments

* Add video section

* Copy classification scripts for video classification

* Initial version of video classification

* add version

* Training of r2plus1d_18 on kinetics work

Gives even slightly better results than expected, with 57.336 top1 clip accuracy. But we count some clips twice in this evaluation

* Cleanups on training script

* Lint

* Minor improvements

* Remove some hacks

* Lint

* Optimize read_video_timestamps for some formats

* Add some tests

* Miscellaneous fixes and improvements

* Guard against videos without video stream

* Fix lint

* Add test for packed b-frames videos

* Fix missing import

Also extend video saving to support different codecs and options. Notably, we can now save with lossless compression

* Properly order videos with B-frames

* seek doesn't seek to pts, but dts

Find a way of overcoming this problem.

Also make logging less verbose

* Add VideoClips and Kinetics dataset

* Lint + add back missing line

* Adds ClipSampler following Bruno comment

* Change name following Bruno's suggestion

* Enable specifying a target framerate

* Fix test_io for new interface

* Add comment mentioning drop_last behavior

* Make compute_clips more robust

* Flake8

* Fix for Python2

* WIP

* WIP

* Add some documentation

* Improve tests and add GC collection

* [WIP] add timestamp getter

* Bugfixes

* Improvements and travis

* Add audio fine-grained alignment

* More doc

* Remove unecessary file

* Remove comment

* Lazy import av

* Remove hard-coded constants for the test

* Return info stats from read

* Fix for Python-2