GitLab

Summary:
https://fb.workplace.com/groups/pythonfoundation/posts/2990917737888352

Remove `mobile-vision` from opt-out list; leaving `mobile-vision/SNPE` opted out because of 3rd-party code.

arc lint --take BLACK --apply-patches --paths-cmd 'hg files mobile-vision'

allow-large-files

Reviewed By: sstsai-adl

Differential Revision: D30721093

fbshipit-source-id: 9e5c16d988b315b93a28038443ecfb92efd18ef8

Differential Revision: D30624781

fbshipit-source-id: 6538813c886ffb9eae2e1d88d500f34c61cae5c0

Summary:
## Context
- The current limit on the number of detections per image (`K`) in LVIS is 300.
- Implementing AP_pool/AP_fixed requires removing this default limit on `K`
- [Literature](https://arxiv.org/pdf/2102.01066.pdf) has shown that increasing `K` correlates with AP gains

## This Diff
- Changed limit on number of detections per image (`K`) to be customizable for LVIS and COCO through `TEST.DETECTIONS_PER_IMAGE` in the config
   - For COCO:
       - Maintain the default `max_dets_per_image` to be [1, 10, 100] as from [COCOEval](https://www.internalfb.com/code/fbsource/[88bb57c3054a]/fbcode/deeplearning/projects/cocoApi/PythonAPI/pycocotools/cocoeval.py?lines=28-29)
       - Allow users to input a custom integer for `TEST.DETECTIONS_PER_IMAGE` in the config, and use  [1, 10, `TEST.DETECTIONS_PER_IMAGE`] for COCOEval
   - For LVIS:
       - Maintain the default `max_dets_per_image` to be 300 as from [LVISEval](https://www.internalfb.com/code/fbsource/[f6b86d023721]/fbcode/deeplearning/projects/lvisApi/lvis/eval.py?lines=528-529)
       - Allow users to input a custom integer for `TEST.DETECTIONS_PER_IMAGE` in the config, and use this in LVISEval
- Added `COCOevalMaxDets` for evaluating AP with the custom limit on number of detections per image (since default `COCOeval` uses 100 as limit on detections per image for evaluating AP)

## Inference Runs using this Diff
- Performed inference using `K = {300, 1000, 10000, 100000}`
- Launched fblearner flows for object detector baseline models with N1055536 (LVIS) and N1055756 (COCO)
  - Recorded [results of running inference](https://docs.google.com/spreadsheets/d/1rgdjN2KvxcYfKCkGUC4tMw0XQJ5oZL0dwjOIh84YRg8/edit?usp=sharing)

Reviewed By: ppwwyyxx

Differential Revision: D30077359

fbshipit-source-id: 372eb5e0d7c228fb77fe23bf80d53597ec66287b

Differential Revision: D30615605

fbshipit-source-id: d4d4550b6d1da4c75945ba674fbdd49a842ec6a9

Summary:
Enable the inference for boltnn (via running torchscript).
- merge rcnn's boltnn test with other export types.
- misc fixes.

Differential Revision: D30610386

fbshipit-source-id: 7b78136f8ca640b5fc179cb47e3218e709418d71

Summary:
Refactors the `MODEL.REGRESSOR.PREPROCESSORS` usage to allow for multiple preprocessors, and adds a new `ADD_COORD_CHANNELS` preprocessor.

Note: `MODEL.FBNET_V2.STEM_IN_CHANNELS` should be modified in your config to reflect the preprocessors that are enabled. Specifically, `ADD_COORD_CHANNELS` increases the input channels by 2, while `SPLIT_AND_CONCAT` decreases by a factor of the chunk size (typically 2). See the included `quick_pupil_3d_*` configs as an example.

Differential Revision: D30459924

fbshipit-source-id: dd8e3293a416a1a556e091cecc058a1be5288cc0

Summary: Support customized subclass selection.  Only the selected gestures are used for model training.

Reviewed By: sanjeevk42

Differential Revision: D30205443

fbshipit-source-id: 36337893aa5d06bb8be5d5587da11398b246b02e

Summary: d2go/modeling/misc.py is open source, and references an internal code path in its docstring. This diff removes that reference.

Reviewed By: wat3rBro

Differential Revision: D30578876

fbshipit-source-id: b255af215e6c096f62f17f65c5f72a0ab95458a9

Summary: All metrics should have been reduced on rank 0 by dataset evaluator.

Reviewed By: wat3rBro

Differential Revision: D30389938

fbshipit-source-id: f8dfb6f1f17635c2fb98391780fdefe90c630054

Summary:
Update weights path in [previous commit](https://github.com/facebookresearch/d2go/commit/477ab964e2165cb586b5c00425f6e463d7edeadd)
fixes https://github.com/facebookresearch/d2go/issues/108

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/109

Reviewed By: wat3rBro

Differential Revision: D30505672

Pulled By: zhanghang1989

fbshipit-source-id: dc946348549a171a6ce058411be2bfd9fa2dad2c

Summary: `export_predictor` is now not customizable, all customization will be done via `prepare_for_export` and `ModelExportMethod`

Reviewed By: zhanghang1989

Differential Revision: D28083607

fbshipit-source-id: e584fff185912ca3e985194b741860276f0943df

Summary:
Added multi-tensor optimizer implementation for SGD, from `torch.optim._multi_tensor`. It can potentially provide ~5% QPS improvement by using `foreach` API to speed up the optimizer step.

Using it is optional, from the configuration file, by specifying `SGD_MT` in the `SOLVER.OPTIMIZER` setting.

Reviewed By: zhanghang1989

Differential Revision: D30377761

fbshipit-source-id: 06107f1b91e9807c1db5d1b0ca6be09fcbb13e67

Summary: Add a mask and keypoint head to FBNetV3_G to support more use cases

Reviewed By: zhanghang1989

Differential Revision: D30205169

fbshipit-source-id: 2fb88df3770fa749331021756ef5a5b6c9493bf5

Summary:
`torch.optim._multi_tensor` provides faster Optimizer implementations as it uses foreach APIs. We can enable it by modifying from `OPTIMIZER: "ADAMW"` to `OPTIMIZER: "ADAMW_MT"` in the config file.

In order to profit from the speedup, we need to reduce the number of parameter groups as suggested in this post: https://fb.workplace.com/groups/1405155842844877/permalink/4971600462867046/

The current implementation uses one parameter group per parameter which is not optimal. The proposed change groups parameters by learning rate and weight decay combinations.

Reviewed By: zhanghang1989

Differential Revision: D30272112

fbshipit-source-id: d8d24298a59b52c2fc2930f7d614a0c6380a432f

Summary:
Pull Request resolved: https://github.com/facebookresearch/detectron2/pull/3350

`get_local_rank` relies on a global variable set by Detectron2's `launch` utils.
Since other frameworks might use Detectron2's distribute utils but don't launch with Detectron2's `launch` utils. Use `torch.cuda.current_device` to get the current device instead.

Reviewed By: HarounH, ppwwyyxx

Differential Revision: D30233746

fbshipit-source-id: 0b140ed5c1e7cd87ccf05235127f338ffc40a53d

Summary:
Before this fix, the EMA GAN model will have inf min_val/max_val when converting QAT models to int8 torchscript model (as shown in f290518237).

https://pxl.cl/1MNx0

Reviewed By: yc-fb

Differential Revision: D23387923

fbshipit-source-id: 5c2119e2c5170e30c6059e7374c22e367fcd2b26

Summary:
Implementing `prepare_for_export` using the boltnn conversion from https://fburl.com/diffusion/ql1i3358.
Implementing `prepare_for_quant` using the quantization from https://fburl.com/diffusion/8nre9o03.

Differential Revision: D29817424

fbshipit-source-id: 800571ecf7f07d01c0a3a12100525354b48fe568

Summary: Add MaskFormer to d2go

Reviewed By: bichenwu09

Differential Revision: D30006691

fbshipit-source-id: 15c85f4ab8b3d515805d639ad8cf47532af81f5e

Summary: The `do_test` method might be used to perform testing outside the training process. One might think it will load the weights of the models before testing similar to `do_train` method. This diff adds a comment that clarifies this confusion.

Reviewed By: ppwwyyxx

Differential Revision: D29082338

fbshipit-source-id: 6ec7d7f7f243503414fa904f4eb8856e62e9ed6d

Summary:
Pull Request resolved: https://github.com/facebookresearch/detectron2/pull/3322

avoid warnings like the following:
```
[W ProcessGroupNCCL.cpp:1569] Rank 0 using best-guess GPU 0 to perform barrier as devices used by
this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is
incorrect. Specify device_ids in barrier() to force use of a particular device.
```

maybe can fix the hang in https://github.com/facebookresearch/detectron2/issues/3319

Reviewed By: vaibhava0

Differential Revision: D30077957

fbshipit-source-id: b8827e66c5eecc06b650acde2e7ff44106327f69

Summary:
Add default config for RandomSubsetTrainingSampler in D2 (https://github.com/facebookresearch/d2go/commit/3ee8885047e7ffb9eadcc6a1ecf8253c7ce9f79e)go.

User can use use the RandomSubsetTrainingSampler with the following yaml configs
```
DATALOADER:
  SAMPLER_TRAIN: RandomSubsetTrainingSampler
  RANDOM_SUBSET_RATIO: [Desired_ratio]  # for RandomSubsetTrainingSampler
```

Reviewed By: XiaoliangDai

Differential Revision: D29892366

fbshipit-source-id: cabb67fb46e51a93a8342a42f77a8a4d23a933e9

Reviewed By: newstzpz

Differential Revision: D29897245

fbshipit-source-id: 5f96fc17361e7dcc65b2b15c995ec6104496c5c7

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/102

- fix model_zoo model urls (missed in D27992340 (https://github.com/facebookresearch/d2go/commit/477ab964e2165cb586b5c00425f6e463d7edeadd))
- update mask rcnn fbnet V3G config
- update v3g retrained weights

Reviewed By: ppwwyyxx, wat3rBro

Differential Revision: D29627615

fbshipit-source-id: 0694772e47b9c58965e47492177a5d6de53364cb

Summary: In case the height/width is None, the original version will cause a crash. So adding additional check to bypass this issue.

Reviewed By: ppwwyyxx

Differential Revision: D29807853

fbshipit-source-id: b2b1a7edb52b7911da79a11329d4cf93f343c279

Summary:
The GAN training pipeline is broken since last week, with the TypeError: cannot pickle
'_io.TextIOWrapper' object (refer to f283777469 for more details)

The issue is caused by D29379832 (https://github.com/facebookresearch/d2go/commit/5509a1383c1162081e9784d79eaf0b12ebbca1fd), in which the deepcopy is failed. This diff fixes this issue by disabling the FLOPs calculation when the deepcopy is failed.

Reviewed By: ppwwyyxx

Differential Revision: D29552182

fbshipit-source-id: 80d078b3d8ca68535e0366a412668e098b04ed04

Summary:
Adding test for previous diff.
Boltnn backend is supported on device -- so this test only checks if the conversion takes place and the output file is present.

Differential Revision: D29589245

fbshipit-source-id: ba66a733295304531d177086ce6459a50cfbaa07

Summary:
Added predictor_type `boltnn_int8` to export to BoltNN via torch delegate.

- `int8` needs to be in the name, otherwise the post-train quantization won't happen;

```
cfg.QUANTIZATION.BACKEND = "qnnpack"
// cfg.QUANTIZATION.CUSTOM_QSCHEME = "per_tensor_affine"
```

Seems that ` QUANTIZATION.CUSTOM_QSCHEME per_tensor_affine` is not needed - likely covered by "qnnpack".

Reviewed By: wat3rBro

Differential Revision: D29106043

fbshipit-source-id: 865ac5af86919fe7b4530b48433a1bd11e295bf4

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/101

In D2 (https://github.com/facebookresearch/d2go/commit/4f3f3401173ee842995ec69a7ce2635e2deb178a)GoDatasetMapper, when crop transform is applied to the image. "Inputs" should be updated to use the cropped images before other transforms are applied later.

Reviewed By: zhanghang1989

Differential Revision: D29551488

fbshipit-source-id: 48917ffc91c8a80286d61ba3ae8391541ec2c930

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/96

In `DETRRunner`, the method `build_optimizer` customized the following logics, which are actually redundant to parent class implementation and can be removed.
- Discount LR for certain modules, such as those with name `reference_points`, `backbone`, and `sampling_offsets`.
  - Those can be achieved by `SOLVER.LR_MULTIPLIER_OVERWRITE` after we update `get_default_optimizer_params` in `mobile-vision/d2go/d2go/optimizer/build.py`.
- Full model gradient clipping
  - This is also implemented in `mobile-vision/d2go/d2go/optimizer/build.py`

It also has minor issues
- It ignores `SOLVER.WEIGHT_DECAY_NORM` which can set a different weight decay for affine parameters in the norm modules.

Reviewed By: zhanghang1989

Differential Revision: D29420642

fbshipit-source-id: deeb9348c9d282231c540dde6161acedd8e3a119

Summary: Fix missing comma for extended coco load, which would ignore bbox_mode and keypoints field.

Reviewed By: zhanghang1989

Differential Revision: D29608815

fbshipit-source-id: 8c737df1dfef7f88494f7de25e06b0c37742ac30

Summary:
Add the fields which will be used in point-based modeling.
- `point_coords` : indicates the point_coords in the image.
- `point_labels`: indicates the foreground or background points.

Differential Revision: D29532103

fbshipit-source-id: 9af6c9b049e1d05fd0d77909b09de1feec391ce9

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/98

https://github.com/facebookresearch/d2go/issues/60#issuecomment-863149605
#stamp2ship

Reviewed By: zhanghang1989

Differential Revision: D29495802

fbshipit-source-id: 0dc63b1ee1f7c8c0a694c39ce41ab77c25109c60

Summary: Removed quant/dequant between backbone and proposal generator, and roi_box_conv and the following avg_pool

Reviewed By: wat3rBro

Differential Revision: D29383036

fbshipit-source-id: ef07b3d1997b1fc7f92bcd9201523e9071510a8b

Summary: "fb" -> "fn"

Reviewed By: ananthsub

Differential Revision: D29480559

fbshipit-source-id: 78a0cd3ddd25df2c877514d4a5c0c29c248267a2

Summary: Updated load_from_checkpoint method call inside lighting_task.py to include extra 'strict' keyword parameter

Reviewed By: kazhang

Differential Revision: D29446372

fbshipit-source-id: b14bc13db551f0876ca78d3ea164cfb08e71a757

Summary: A Lightning task for training StyleGAN2.

Reviewed By: tax313

Differential Revision: D28922408

fbshipit-source-id: bdc9e7370de1b7b7ca9086bc6c0acbe66810d5f8

Summary:
This diff introduces the D2 (https://github.com/facebookresearch/d2go/commit/9d9f438b191634dc38d16f3973e490909b7f67dd)Go GANs Lightning task for migrating D2 (https://github.com/facebookresearch/d2go/commit/9d9f438b191634dc38d16f3973e490909b7f67dd)Go's GANsRunner to Lightning based workflow.
The Lightning task could directly work with D2 (https://github.com/facebookresearch/d2go/commit/9d9f438b191634dc38d16f3973e490909b7f67dd)Go e2e workflow.

Reviewed By: tax313

Differential Revision: D28165835

fbshipit-source-id: 4d9d679e188f9d5f9a46f01f7d34a8f30c3e170b

Summary:
Currently we move EMA weights to expected device right after loading from checkpoint.
However, by the time on_load_checkpoint hook is called, current GPU device has not been assigned. This could lead to EMA weights on cuda:0 while the model is on cuda:1.
This diff move EMA weights to device in `on_pretrain_routine_end` instead.

Reviewed By: zhanghang1989

Differential Revision: D28429843

fbshipit-source-id: d864fb3687eb6958872300c5ec0af7ce90591f83

Reviewed By: zhanghang1989

Differential Revision: D29379832

fbshipit-source-id: 9283a8796a1dbee81b51611407c22f7d5a2069dc

Summary:
# Context
In post training quantization callback, we make a deepcopy of the Lightning module before validation start and prepare the copy with FX quantization API. The callback keeps the prepared model inside it.

# The problem
By the second time we run the validation epoch, we try to make a copy of the Lightning module, which has a reference to trainer, which has a reference to quantization callback, which has a prepared model, which is not deepcopiable.

# Mitigation
Delete the trainer before making a deepcopy.
We're already doing that in stl/callbacks/quantization, but the changes were not ported into D2 (https://github.com/facebookresearch/d2go/commit/4169abc18ec539a24081b179fcbbc5a5754d102b)Go.

Reviewed By: zhanghang1989

Differential Revision: D29409085

fbshipit-source-id: 24550124181673b2e567b2a04563bcdfb440e145