GitLab

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/2077

Pull Request resolved: https://github.com/facebookresearch/SlowFast/pull/164

This is a follow-up diff from D18720474

We will be releasing a new version of torchvision soon and the signature of those functions is not ready yet, following my comment in https://our.intern.facebook.com/intern/diff/D18720474/?transaction_id=561239541337402



Reviewed By: stephenyan1231

Differential Revision: D20914571

fbshipit-source-id: 1a7560b8f8e46ab42ef376c50b494a4f73923e94
Co-authored-by: Francisco Massa <fmassa@fb.com>

* Add tests for negative samples for Mask R-CNN and Keypoint R-CNN

* Fix lint

* improve stability of test_nms_cuda

This change addresses two issues:

_create_tensors_with_iou() creates test data for the NMS tests. It
takes care to ensure at least one pair of boxes (1st and last) have
IoU around the threshold for the test. However, the constructed
IoU for that pair is _so_ close to the threshold that rounding
differences (presumably) between CPU and CUDA implementations may
result in one suppressing a box in the pair and the other not.
Adjust the construction to ensure the IoU for the box pair is
near the threshold, but far-enough above that both implementations
should agree.

Where 2 boxes have nearly or exactly the same score, the CPU and
CUDA implementations may order them differently. Adjust
test_nms_cuda() to check only that the non-suppressed box lists
include the same members, without regard for ordering.

* adjust assertion in test_nms_cuda

The CPU and CUDA nms implementations each sort the box scores
as part of their work, but the sorts they use are not stable. So
boxes with the same score maybe be processed in opposite order
by the two implmentations.

Relax the assertion in test_nms_cuda (following the model in
pytorch's test_topk()) to allow the test to pass if the output
differences are caused by similarly-scored boxes.

* improve stability of test_nms_cuda

Adjust _create_tensors_with_iou() to ensure we create at least
one box just over threshold that should be suppressed.

* Add CircleCI job for python lint

* Break lint

* Fix

* Fix lint

* Re-enable all tests and remove travis python lint

* Add test for large batches in DeformConv2d

* Clean-up and (try) fix DeformConv2d

* Simplifications and bugfixes

* Try fix CUDA now

* fixes and tests for variable input size

* transform test fix

* Fix comment

* Dynamic shape for keypoint_rcnn

* Update test_onnx.py

* Update rpn.py

* Fix for split on RPN

* Fixes for feedbacks

* flake8

* topk fix

* Fix build

* branch on tracing

* fix for scalar tensor

* Fixes for script type annotations

* Update rpn.py

* clean up

* clean up

* Update rpn.py

* Updated for feedback

* Fix for comments

* revert to use tensor

* Added test for box clip

* Fixes for feedback

* Fix for feedback

* ORT version revert

* Update ort

* Update .travis.yml

* Update test_onnx.py

* Update test_onnx.py

* Tensor sizes

* Fix for dynamic split

* Try disable tests

* pytest verbose

* revert one test

* enable tests

* Update .travis.yml

* Update .travis.yml

* Update .travis.yml

* Update test_onnx.py

* Update .travis.yml

* Passing device

* Fixes for test

* Fix for boxes datatype

* clean up
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* remove sys.version_info == 2

* remove sys.version_info < 3

* remove from __future__ imports

* remove six from python code

* remove six from setup.py

* remove six from tests

* remove six from references

* remove six from packaging

* revert str to torch._six._string_classes

* revert str to torch._six._string_classes

* Add fill option to RandomPerspective #1972

* Minor fix to docstring syntax

* Add _parse_fill() to get fillcolor (#1972)

* Minor refactoring as per comments.

* Added test for RandomPerspective with fillcolor.

* Force perspective transform in test.

* F.normalize unsqueeze mean & std if necessary

* added tests to F.normalize for 3d mean & std tensors

* modified FasterRCNN to accept negative samples

* remove debug lines

* Change torch.zeros_like to torch.zerros

* Add unit tests

* take the `device` into account
Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

* Base decoder for video. (#1747)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1747

Pull Request resolved: https://github.com/pytorch/vision/pull/1746

Added the implementation of ffmpeg based decoder with functionality that can be used in VUE and TorchVision.

Reviewed By: fmassa

Differential Revision: D19358914

fbshipit-source-id: abb672f89bfaca6351dda2354f0d35cf8e47fa0f

* Integrated base decoder into VideoReader class and video_utils.py (#1766)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1766

Replaced FfmpegDecoder (incompativle with VUE) by base decoder (compatible with VUE).
Modified python utilities video_utils.py for internal simplification. Public interface got preserved.

Reviewed By: fmassa

Differential Revision: D19415903

fbshipit-source-id: 4d7a0158bd77bac0a18732fe4183fdd9a57f6402

* Optimizating base decoder performance. (#1852)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1852

Changed base decoder internals for a faster clip processing.

Reviewed By: stephenyan1231

Differential Revision: D19748379

fbshipit-source-id: 58a435f0a0b25545e7bd1a3edb0b1d558176a806

* Minor fix and decoder class members access.

Summary:
Found and fix a bug in cropping algorithm (simple mistyping).
Also derived classes need access to some decoder class members, like initialization parameters - make it protected.

Reviewed By: stephenyan1231, fmassa

Differential Revision: D19895076

fbshipit-source-id: 691336c8e18526b085ae5792ac3546bc387a6db9

* Added missing header for less dependencies. (#1898)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1898

Include streams/samplers shouldn't depend on decoder headers. Add dependencies directly to the place where they are required.

Reviewed By: stephenyan1231

Differential Revision: D19911404

fbshipit-source-id: ef322a053708405c02cee4562b456b1602fb12fc

* Implemented VUE Asynchronous Decoder

Summary: For Mothership we have found that asynchronous decoder provides a better performance.

Differential Revision: D20026194

fbshipit-source-id: 627b91844b4e3f917002031dd32cb19c239f4ba8

* fix a bug in API read_video_from_memory (#1942)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1942

In D18720474, it introduces a bug in `read_video_from_memory` API. Thank weiyaowang for reporting it.

Reviewed By: weiyaowang

Differential Revision: D20270179

fbshipit-source-id: 66348c99a5ad1f9129b90e934524ddfaad59de03

* extend decoder to support new video_max_dimension argument (#1924)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1924

Extend `video reader` decoder python API in Torchvision to support a new argument `video_max_dimension`. This enables the new video decoding use cases. When setting `video_width=0`, `video_height=0`, `video_min_dimension != 0`, and `video_max_dimension != 0`, we can rescale the video clips so that its spatial resolution (height, width) becomes
 - (video_min_dimension, video_max_dimension) if original height < original width
 - (video_max_dimension, video_min_dimension) if original height >= original width

This is useful at video model testing stage, where we perform fully convolution evaluation and take entire video frames without cropping as input. Previously, for instance we can only set `video_width=0`, `video_height=0`, `video_min_dimension = 128`, which will preserve aspect ratio. In production dataset, there are a small number of videos where aspect ratio is either extremely large or small, and when the shorter edge is rescaled to 128, the longer edge is still large. This will easily cause GPU memory OOM when we sample multiple video clips, and put them in a single minibatch.

Now, we can set (for instance) `video_width=0`, `video_height=0`, `video_min_dimension = 128` and `video_max_dimension = 171` so that the rescale resolution is either (128, 171) or (171, 128) depending on whether original height is larger than original width. Thus, we are less likely to have gpu OOM because the spatial size of video clips is determined.

Reviewed By: putivsky

Differential Revision: D20182529

fbshipit-source-id: f9c40afb7590e7c45e6908946597141efa35f57c

* Fixing samplers initialization (#1967)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1967



No-ops for torchvision diff, which fixes samplers.

Differential Revision: D20397218

fbshipit-source-id: 6dc4d04364f305fbda7ca4f67a25ceecd73d0f20

* Exclude C++ test files
Co-authored-by: Yuri Putivsky <yuri@fb.com>
Co-authored-by: Zhicheng Yan <zyan3@fb.com>

* Fix for roi_align export

* Disable interpolate script module tests

Disable test until export of interpolate script module to ONNX is fixed

* add checkout/assert in roi_pool

* add checkout/assert in roi_align

* move check_roi_boxes_shape func to ops/_utils.py

* add tests

* fix CI

* fix CI
Co-authored-by: Guanheng Zhang <zhangguanheng@devfair0197.h2.fair>

* fix googlenet no aux logits

* small fix
Co-authored-by: eellison <eellison@fb.com>

* Aligned flag in the interfaces

* Aligned flag in the impl, and remove unused comments

* Handling empty bin in forward

* Remove raise error in roi_width

* Aligned flag in the Testcodes

* ci: Add verbosity to pytest results, store in ci

Makes the pytest runs for building conda packages more verbose and
stores the results for viewing inside of CircleCI
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* test: Skip inception v3 in test_quantized_models

Was causing timeouts on circleci due to long run time, re-enable when
tests can be brought to a reasonable time again.
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

* feat: Added __repr__ attribute to GeneralizedRCNNTransform

Added more details to default __repr__ attribute for printing.

* fix: Put back relative imports

* style: Fixed pep8 compliance

Switched strings with  syntax to f-strings.

* test: Added test for GeneralizedRCNNTransform __repr__

Checked integrity of __repr__ attribute

* test: Fixed unittest for __repr__

Fixed the formatted strings in the __repr__ integrity check for GeneralizedRCNNTransform

* fix: Fixed f-strings for earlier python versions

Switched back f-strings to .format syntax for Python3.5 compatibility.

* fix: Fixed multi-line string

Fixed multiple-line string syntax for compatibility

* fix: Fixed GeneralizedRCNNTransform unittest

Fixed formatting of min_size argument of the resizing part

* Force object annotiation to be an array

* Remove unecessary parentheses

* Change object check

* Remove check for list

* Add test coverage to xml parsing

* Tidy up whitespace

* Fix indentation

* Force object annotiation to be an array

* Remove unecessary parentheses

* Change object check

* Remove check for list

* Add test coverage to xml parsing

* Tidy up whitespace

* torchscriptable functions for video io (#1653)

Summary:
Pull Request resolved: https://github.com/pytorch/vision/pull/1653



created new torchscriptable video io functions as part of the api: read_video_meta_data_from_memory and read_video_from_memory.

Updated the implementation of some of the internal functions to be torchscriptable.

Reviewed By: stephenyan1231

Differential Revision: D18720474

fbshipit-source-id: 4ee646b66afecd2dc338a71fd8f249f25a3263bc

* BugFix
Co-authored-by: Jon Guerin <54725679+jguerin-fb@users.noreply.github.com>

* adding new_empty_tensor symbolic

* flake8

* fix for feedback

* skipping the ORT test

* fix for ORT test

* initial fix

* outsourced num bands lookup

* fix doc

* added pillow version requirement

* simplify number of bands extraction

* remove unrelated change

* remove indirect dependency on pillow>=5.2.0

* extend docstring to transform

* bug fix

* added test

* Fix AnchorGenerator if moving from one device to another

* Fixes for the test

* Testing CI

* Disable tests for Pillow 7

* scriptability checks

* tests upds

* linter upds

* linter upds

* upds

* tuple list changes

* linter updates

* remove windows skips from video_utils tests, now that they pass

* replace lambda in videoclips in order to be pickled on windows and update tests

* out_place checks

* lint ups

* tgz updates

* tgz updates

* tgz updates

* add scriptable transform: center_crop

* add test: center_crop

* add scriptable transform: five_crop

* add scriptable transform: five_crop

* add scriptable transform: fix minor issues

* Add Deformable Convolution operation.

This adds the deformable convolution operation, as described in Deformable Convolutional Networks (https://arxiv.org/abs/1703.06211).

- The code is based on https://github.com/open-mmlab/mmdetection/blob/master/mmdet/ops/dcn/src/deform_conv_cuda.cpp ; the whole code was modified and refactored to remove redundancies and increase clarity, and to adapt it to torchvision.

- The CPU part is a direct copy of the CUDA code; it might make sense to do follow-up adjustments in the CPU code to simplify it / optimize it, or to reuse functionality between CPU and CUDA..

- We also add tests (with a non-trivial set of parameters); they can be made more robust by randomizing the parameters and executing multiple times.

* Update DeformConv to be more consistent w/ Conv2d

* rename some variables and arguments to match Conv2d;
* add optional bias;
* add weight, offset and bias as module parameters;
* remove the n_parallel_imgs parameter;
* Fix __repr__;
* etc..

Initialization of weight and bias is the same as in Conv2d, and
initialization of offsets to zero is the same as in the paper.

This also includes some other small unrelated fixes/improvements.

* Apply clang-format in DeformConv files.

* Import Optional type annotation

* Remove offset param from DeformConv2d module

- We pass the offset in the forward of DeformConv2d, instead of having
an internal parameter. This adds some complexity to creating the module
(e.g. now you have to worry about the output size, to create the
offset), but it gives more flexibility.
- We also use make_tuple for tuple creation, in an attempt to fix error
w/ older compilers.

* Replace abs by std::abs

Old gcc versions were giving wrong results here, because they would
resolve abs as int -> int, thus causing undesired truncation. Replacing
abs by std::abs should allow for correct overloading of abs as float -> float.

* Reorder declarations for clarity

* Reorder weight and offset args in deform_conv2d

We place offset arg before the weight arg, to be more
consistent with DeformConv2d.forward(input, offset)

* Replace abs by std::abs in DeformConv_cuda