GitLab

Differential Revision: D30615605

fbshipit-source-id: d4d4550b6d1da4c75945ba674fbdd49a842ec6a9

Summary:
Enable the inference for boltnn (via running torchscript).
- merge rcnn's boltnn test with other export types.
- misc fixes.

Differential Revision: D30610386

fbshipit-source-id: 7b78136f8ca640b5fc179cb47e3218e709418d71

Summary:
Refactors the `MODEL.REGRESSOR.PREPROCESSORS` usage to allow for multiple preprocessors, and adds a new `ADD_COORD_CHANNELS` preprocessor.

Note: `MODEL.FBNET_V2.STEM_IN_CHANNELS` should be modified in your config to reflect the preprocessors that are enabled. Specifically, `ADD_COORD_CHANNELS` increases the input channels by 2, while `SPLIT_AND_CONCAT` decreases by a factor of the chunk size (typically 2). See the included `quick_pupil_3d_*` configs as an example.

Differential Revision: D30459924

fbshipit-source-id: dd8e3293a416a1a556e091cecc058a1be5288cc0

Summary: Support customized subclass selection.  Only the selected gestures are used for model training.

Reviewed By: sanjeevk42

Differential Revision: D30205443

fbshipit-source-id: 36337893aa5d06bb8be5d5587da11398b246b02e

Summary: d2go/modeling/misc.py is open source, and references an internal code path in its docstring. This diff removes that reference.

Reviewed By: wat3rBro

Differential Revision: D30578876

fbshipit-source-id: b255af215e6c096f62f17f65c5f72a0ab95458a9

Summary: All metrics should have been reduced on rank 0 by dataset evaluator.

Reviewed By: wat3rBro

Differential Revision: D30389938

fbshipit-source-id: f8dfb6f1f17635c2fb98391780fdefe90c630054

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/106

# 2-stage DF-DETR

DF-DETR supports 2-stage detection. In the 1st stage, we detect class-agnostic boxes using the feature pyramid (a.k.a. `memory` in the code) computed by the encoder.

Current implementation has a few flaws
- In `setcriterion.py`, when computing loss for encoder 1st stage predictions, `num_boxes` should be reduced across gpus and also clamped to be positive integer to avoid divide-by-zero bug. Current implementation will lead to divide-by-zero NaN issue when `num_boxes` is zero (e.g. no box annotation in the cropped input image).
- In `gen_encoder_output_proposals()`, it manually fill in `float("inf")` at invalid spatial positions outside of actual image size. However, it is not guaranteed that those positions won't be selected as top-scored positions.  `float("inf")` can easily cause affected parameters to be updated to NaN value.
- `class_embed` for encoder should has 1 channel rather than num_class channels because we only need to predict the probability of being a foreground box.

This diff fixes the issues above.

# Gradient blocking in decoder

Currently, gradient of reference point is blocked at each decoding layer to improve numerical stability during training.
In this diff, add an option `MODEL.DETR.DECODER_BLOCK_GRAD`. When False, we do NOT block the gradient. Empirically, we find this leads to better box AP.

Reviewed By: zhanghang1989

Differential Revision: D30325396

fbshipit-source-id: 7d7add1e05888adda6e46cc6886117170daa22d4

Summary:
Update weights path in [previous commit](https://github.com/facebookresearch/d2go/commit/477ab964e2165cb586b5c00425f6e463d7edeadd)
fixes https://github.com/facebookresearch/d2go/issues/108

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/109

Reviewed By: wat3rBro

Differential Revision: D30505672

Pulled By: zhanghang1989

fbshipit-source-id: dc946348549a171a6ce058411be2bfd9fa2dad2c

Summary: `export_predictor` is now not customizable, all customization will be done via `prepare_for_export` and `ModelExportMethod`

Reviewed By: zhanghang1989

Differential Revision: D28083607

fbshipit-source-id: e584fff185912ca3e985194b741860276f0943df

Summary:
A torch version which is batched allows us to avoid CPU <--> GPU copy which
gets us ~200ms per iteration saving. This new version of generating boundary
weight mask produces identical masks.

Reviewed By: wat3rBro

Differential Revision: D30176412

fbshipit-source-id: 877f4c6337e7870d3bafd8eb9157ac166ddd588a

Summary:
Added multi-tensor optimizer implementation for SGD, from `torch.optim._multi_tensor`. It can potentially provide ~5% QPS improvement by using `foreach` API to speed up the optimizer step.

Using it is optional, from the configuration file, by specifying `SGD_MT` in the `SOLVER.OPTIMIZER` setting.

Reviewed By: zhanghang1989

Differential Revision: D30377761

fbshipit-source-id: 06107f1b91e9807c1db5d1b0ca6be09fcbb13e67

Summary: The uint8 cast means that the floating point non_bd_weight is never assigned

Reviewed By: wat3rBro

Differential Revision: D30176377

fbshipit-source-id: 013602bb4313393f220ee0f1510bf1ff83bd56fc

Summary: Add a mask and keypoint head to FBNetV3_G to support more use cases

Reviewed By: zhanghang1989

Differential Revision: D30205169

fbshipit-source-id: 2fb88df3770fa749331021756ef5a5b6c9493bf5

Summary: Add FBNAS toolkit for HPO in D2 (https://github.com/facebookresearch/d2go/commit/adf223bdac5b534514a8ba80f6bd61fc9dd8b464)Go

Reviewed By: newstzpz

Differential Revision: D28672821

fbshipit-source-id: 6a378af2bb43ef6cb556d4158fd1b0d3e363e956

Summary:
`torch.optim._multi_tensor` provides faster Optimizer implementations as it uses foreach APIs. We can enable it by modifying from `OPTIMIZER: "ADAMW"` to `OPTIMIZER: "ADAMW_MT"` in the config file.

In order to profit from the speedup, we need to reduce the number of parameter groups as suggested in this post: https://fb.workplace.com/groups/1405155842844877/permalink/4971600462867046/

The current implementation uses one parameter group per parameter which is not optimal. The proposed change groups parameters by learning rate and weight decay combinations.

Reviewed By: zhanghang1989

Differential Revision: D30272112

fbshipit-source-id: d8d24298a59b52c2fc2930f7d614a0c6380a432f

Summary:
Pull Request resolved: https://github.com/facebookresearch/detectron2/pull/3350

`get_local_rank` relies on a global variable set by Detectron2's `launch` utils.
Since other frameworks might use Detectron2's distribute utils but don't launch with Detectron2's `launch` utils. Use `torch.cuda.current_device` to get the current device instead.

Reviewed By: HarounH, ppwwyyxx

Differential Revision: D30233746

fbshipit-source-id: 0b140ed5c1e7cd87ccf05235127f338ffc40a53d

Summary:
Before this fix, the EMA GAN model will have inf min_val/max_val when converting QAT models to int8 torchscript model (as shown in f290518237).

https://pxl.cl/1MNx0

Reviewed By: yc-fb

Differential Revision: D23387923

fbshipit-source-id: 5c2119e2c5170e30c6059e7374c22e367fcd2b26

Reviewed By: stephenyan1231

Differential Revision: D30225977

fbshipit-source-id: 479b96acc7f90a8ee2373ab44112e21086e9d1d2

Summary:
Implementing `prepare_for_export` using the boltnn conversion from https://fburl.com/diffusion/ql1i3358.
Implementing `prepare_for_quant` using the quantization from https://fburl.com/diffusion/8nre9o03.

Differential Revision: D29817424

fbshipit-source-id: 800571ecf7f07d01c0a3a12100525354b48fe568

Summary: Add MaskFormer to d2go

Reviewed By: bichenwu09

Differential Revision: D30006691

fbshipit-source-id: 15c85f4ab8b3d515805d639ad8cf47532af81f5e

Summary: The `do_test` method might be used to perform testing outside the training process. One might think it will load the weights of the models before testing similar to `do_train` method. This diff adds a comment that clarifies this confusion.

Reviewed By: ppwwyyxx

Differential Revision: D29082338

fbshipit-source-id: 6ec7d7f7f243503414fa904f4eb8856e62e9ed6d

Summary:
Pull Request resolved: https://github.com/facebookresearch/detectron2/pull/3322

avoid warnings like the following:
```
[W ProcessGroupNCCL.cpp:1569] Rank 0 using best-guess GPU 0 to perform barrier as devices used by
this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is
incorrect. Specify device_ids in barrier() to force use of a particular device.
```

maybe can fix the hang in https://github.com/facebookresearch/detectron2/issues/3319

Reviewed By: vaibhava0

Differential Revision: D30077957

fbshipit-source-id: b8827e66c5eecc06b650acde2e7ff44106327f69

Summary:
Add default config for RandomSubsetTrainingSampler in D2 (https://github.com/facebookresearch/d2go/commit/3ee8885047e7ffb9eadcc6a1ecf8253c7ce9f79e)go.

User can use use the RandomSubsetTrainingSampler with the following yaml configs
```
DATALOADER:
  SAMPLER_TRAIN: RandomSubsetTrainingSampler
  RANDOM_SUBSET_RATIO: [Desired_ratio]  # for RandomSubsetTrainingSampler
```

Reviewed By: XiaoliangDai

Differential Revision: D29892366

fbshipit-source-id: cabb67fb46e51a93a8342a42f77a8a4d23a933e9

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/105

exploring deformable attention in transformer

Reviewed By: bichenwu09

Differential Revision: D29093714

fbshipit-source-id: dd691754d9e439661e2eddecb3a1e7cefc8fe568

Reviewed By: newstzpz

Differential Revision: D29897245

fbshipit-source-id: 5f96fc17361e7dcc65b2b15c995ec6104496c5c7

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/102

- fix model_zoo model urls (missed in D27992340 (https://github.com/facebookresearch/d2go/commit/477ab964e2165cb586b5c00425f6e463d7edeadd))
- update mask rcnn fbnet V3G config
- update v3g retrained weights

Reviewed By: ppwwyyxx, wat3rBro

Differential Revision: D29627615

fbshipit-source-id: 0694772e47b9c58965e47492177a5d6de53364cb

Summary:
Deformable DETR training can be unstable due to iterative box refinement in the transformer decoder. To stabilize the training, introduce two changes
- Remove the unnecessary use of inverse sigmoid.
It is possible to completely avoid using inverse sigmoid when box refinement is turned on.
- In `DeformableTransformer` class, detach `init_reference_out` before passing it into decoder to update memory and computer per-decoder-layer reference points/

Reviewed By: zhanghang1989

Differential Revision: D29903599

fbshipit-source-id: a374ba161be0d7bcdfb42553044c4c6700e92623

Summary:
Add new backbone
Experimental results are https://fburl.com/7fyecmrc

Reviewed By: bichenwu09

Differential Revision: D26877909

fbshipit-source-id: ba3f97a1e4d84bec22d6a345f1fca06c741010cc

Summary: In case the height/width is None, the original version will cause a crash. So adding additional check to bypass this issue.

Reviewed By: ppwwyyxx

Differential Revision: D29807853

fbshipit-source-id: b2b1a7edb52b7911da79a11329d4cf93f343c279

Summary:
The GAN training pipeline is broken since last week, with the TypeError: cannot pickle
'_io.TextIOWrapper' object (refer to f283777469 for more details)

The issue is caused by D29379832 (https://github.com/facebookresearch/d2go/commit/5509a1383c1162081e9784d79eaf0b12ebbca1fd), in which the deepcopy is failed. This diff fixes this issue by disabling the FLOPs calculation when the deepcopy is failed.

Reviewed By: ppwwyyxx

Differential Revision: D29552182

fbshipit-source-id: 80d078b3d8ca68535e0366a412668e098b04ed04

Summary:
Adding test for previous diff.
Boltnn backend is supported on device -- so this test only checks if the conversion takes place and the output file is present.

Differential Revision: D29589245

fbshipit-source-id: ba66a733295304531d177086ce6459a50cfbaa07

Summary:
Added predictor_type `boltnn_int8` to export to BoltNN via torch delegate.

- `int8` needs to be in the name, otherwise the post-train quantization won't happen;

```
cfg.QUANTIZATION.BACKEND = "qnnpack"
// cfg.QUANTIZATION.CUSTOM_QSCHEME = "per_tensor_affine"
```

Seems that ` QUANTIZATION.CUSTOM_QSCHEME per_tensor_affine` is not needed - likely covered by "qnnpack".

Reviewed By: wat3rBro

Differential Revision: D29106043

fbshipit-source-id: 865ac5af86919fe7b4530b48433a1bd11e295bf4

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/101

In D2 (https://github.com/facebookresearch/d2go/commit/4f3f3401173ee842995ec69a7ce2635e2deb178a)GoDatasetMapper, when crop transform is applied to the image. "Inputs" should be updated to use the cropped images before other transforms are applied later.

Reviewed By: zhanghang1989

Differential Revision: D29551488

fbshipit-source-id: 48917ffc91c8a80286d61ba3ae8391541ec2c930

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/96

In `DETRRunner`, the method `build_optimizer` customized the following logics, which are actually redundant to parent class implementation and can be removed.
- Discount LR for certain modules, such as those with name `reference_points`, `backbone`, and `sampling_offsets`.
  - Those can be achieved by `SOLVER.LR_MULTIPLIER_OVERWRITE` after we update `get_default_optimizer_params` in `mobile-vision/d2go/d2go/optimizer/build.py`.
- Full model gradient clipping
  - This is also implemented in `mobile-vision/d2go/d2go/optimizer/build.py`

It also has minor issues
- It ignores `SOLVER.WEIGHT_DECAY_NORM` which can set a different weight decay for affine parameters in the norm modules.

Reviewed By: zhanghang1989

Differential Revision: D29420642

fbshipit-source-id: deeb9348c9d282231c540dde6161acedd8e3a119

Summary: Fix missing comma for extended coco load, which would ignore bbox_mode and keypoints field.

Reviewed By: zhanghang1989

Differential Revision: D29608815

fbshipit-source-id: 8c737df1dfef7f88494f7de25e06b0c37742ac30

Summary: Set find_unused_parameters according to DDP_FIND_UNUSED_PARAMETERS with DDPPlugin

Reviewed By: kazhang

Differential Revision: D29567013

fbshipit-source-id: f3ffac566a2ff046f55e692b3b24f9531913d4d4

Summary:
Add the fields which will be used in point-based modeling.
- `point_coords` : indicates the point_coords in the image.
- `point_labels`: indicates the foreground or background points.

Differential Revision: D29532103

fbshipit-source-id: 9af6c9b049e1d05fd0d77909b09de1feec391ce9

Summary:
In D29048363 (https://github.com/facebookresearch/d2go/commit/c480d4e4e213a850cced7758f7b62c20caad8820) we make the detaching of `reference_points` earlier in the hope of allowing more gradient flow to update weights in `self.bbox_embed`.
In this diff, we revert the changes as i) it does not improve box AP ii) it may potential cause in-stable optimization when iterative box refinement is turned on.

Reviewed By: zhanghang1989

Differential Revision: D29530735

fbshipit-source-id: 3217c863343836e129d53e07c0eedb2db8164fe6

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/98

https://github.com/facebookresearch/d2go/issues/60#issuecomment-863149605
#stamp2ship

Reviewed By: zhanghang1989

Differential Revision: D29495802

fbshipit-source-id: 0dc63b1ee1f7c8c0a694c39ce41ab77c25109c60

Summary: Removed quant/dequant between backbone and proposal generator, and roi_box_conv and the following avg_pool

Reviewed By: wat3rBro

Differential Revision: D29383036

fbshipit-source-id: ef07b3d1997b1fc7f92bcd9201523e9071510a8b