GitLab

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/134

Updating `all_steps_qat` example config to use learnable QAT method.

And add logic in `GeneralizedRCNNPatch.prepare_for_quant` to call the new `d2go.utils.qat_utils.get_qat_qconfig` to properly support QAT in D2 (https://github.com/facebookresearch/d2go/commit/7992f91324aee6ae59795063a007c6837e60cdb8)Go training workflow

Differential Revision: D32147216

fbshipit-source-id: 32831c6156bc5c0775196ad8edc890a5292d204f

Summary: Allow reading `.npy` format binary masks shaped (H, W,) in addition to `.png` image masks shaped (H, W, C).

Reviewed By: wat3rBro

Differential Revision: D30136542

fbshipit-source-id: 56df5a766ab15b6808a1327815857e5d38eac910

Summary:
In quantization callback, we prepare the model with FX quantization API and only use the prepared model in training.
However, when training in DDP, the parameters in the origin model still require grad, causing unused parameters RuntimeError.
Previously, Lightning trainer train the model with find_unused_param flag, but if user manually disable it, they will get the runtime error.

In this diff, the parameters in the origin model are frozen. We could consider deleting the origin model after preparation to save memory, but we might have to make some assumption on Lightning module structure, for example, `.model` is the origin model, so that we could `delattr(pl_module, "model")`.

Reviewed By: wat3rBro

Differential Revision: D31902368

fbshipit-source-id: 56eabb6b2296278529dd2b94d6aa4c9ec9e9ca6b

Summary: as title

Reviewed By: Cysu

Differential Revision: D31901433

fbshipit-source-id: 1749527c04c392c830e1a49bca8313ddf903d7b1

Summary: FCOS is registered only because we make an import from `get_default_cfg`, if user don't call it (eg. using their own runner), they might find that the meta-arch is not registered.

Reviewed By: ppwwyyxx

Differential Revision: D31920026

fbshipit-source-id: 59eeeb3d1bf30d6b08463c2814930b1cadd7d549

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/130

We want to make sure that after importing `d2go.modeling` all the meta-arch is registered.

Reviewed By: Maninae

Differential Revision: D31904303

fbshipit-source-id: 3f32b65b764b2458e2fb9c4e0bbd99824b37ecfc

Summary:
### New commit log messages
  1f7bd6650 Mark accelerator connector as protected (#10032)

Reviewed By: yifuwang

Differential Revision: D31905981

fbshipit-source-id: a7f0f03033b02b603d28203ae2c8e8df4933fb23

Reviewed By: sstsai-adl

Differential Revision: D31806054

fbshipit-source-id: 4ea98405e1f94176cb77ca69077adf9f4d22e77e

Summary:
### New commit log messages
  0aa220b46 Remove deprecated `distributed_backend` from `Trainer` (#10017)

Reviewed By: kandluis

Differential Revision: D31788128

fbshipit-source-id: 4d9394e119b3122014fc9681a5c56aac8df49141

Summary: this utility function was added in D30272112 (https://github.com/facebookresearch/d2go/commit/737d099b0a8b0fb1f548435e73f95e1252442827) and is useful to all D2 (https://github.com/facebookresearch/d2go/commit/7992f91324aee6ae59795063a007c6837e60cdb8) users as well

Differential Revision: D31833523

fbshipit-source-id: 0adfc612adb8b448fa7f3dbec1b1278c309554c5

Summary: see bottom diff

Reviewed By: newstzpz

Differential Revision: D31781835

fbshipit-source-id: 501b51e7bf92cf3505060a62822fa36f1ed3a7d4

Summary: helps debugging

Reviewed By: zhanghang1989

Differential Revision: D31806396

fbshipit-source-id: 870308990c4c0c71453d107628b8adcb9edcf391

Summary:
Add toy example to illustrate the Turing workflow.
- modify the model building, add converting to helios step. Note that we need to hide this from OSS, so create FB version of the runner, in order to modify `build_model` and `get_default_cfg`.
- make the `D2 (https://github.com/facebookresearch/d2go/commit/7992f91324aee6ae59795063a007c6837e60cdb8)GoCompatibleMNISTRunner` up-to-date, and use the "tutorial" meta-arch for writing unit test since it's the simplest model. Note that even `TutorialNet` is very simple, there's still a constraint that the FC has to run on 4D tensor with 1x1 spatial dimension because it's been mapped to 1x1 Conv by Helios, modify the `TutorialNet` to make it compatible.

Reviewed By: newstzpz

Differential Revision: D31705305

fbshipit-source-id: 77949dfbf08252be5495e9273210274c8ad86abb

Summary: see bottom diff

Reviewed By: newstzpz

Differential Revision: D31780235

fbshipit-source-id: ec1285c4c5457a631e1eb88bebd47c9f41b47e12

Summary:
Supported learnable qat.
* Added a config key `QUANTIZATION.QAT.FAKE_QUANT_METHOD` to specify the qat metod (`default` or `learnable`).
* Added a config key `QUANTIZATION.QAT.ENABLE_LEARNABLE_OBSERVER_ITER` to specify the start iteration for learnable observers (before that it is using static observers).
* Custom quantization code needs to call ` d2go.utils.qat_utils.get_qat_qconfig()` to get proper qconfig for learnable qat. An exception will raise if qat method is learnable but no learnable observers are used in the model.
* Set the weight decay for scale/zero_point to 0 for the optimizer automatically.
* The way to use larnable qat: enable static observers -> enable fake quant -> enable learnable observers -> freeze bn.

Differential Revision: D31370822

fbshipit-source-id: a5a5044a539d0d7fe1cc6b36e6821fc411ce752a

Summary:
Refactored qat related code.
* Moved `_prepare_model_for_qat` related code to a function.
* Moved `_setup_non_qat_to_qat_state_dict_map` related code to a function.
* Moved QATHook related code to the quantization file and implemented as a class.

Differential Revision: D31370819

fbshipit-source-id: 836550b2c8d68cd93a84d5877ad9cef6f0f0eb39

Summary: D2 (https://github.com/facebookresearch/d2go/commit/7992f91324aee6ae59795063a007c6837e60cdb8) does not add new yacs config for new models, but this simple wrapper with configs->arguments mapping is enough to make the model work with yacs config.

Reviewed By: zhanghang1989

Differential Revision: D30980180

fbshipit-source-id: 75a0cc66051800a3e9d553bb650ca5c900c0ffa3

Summary:
Supported specifying customized parameter groups from model.
* Allow model to specify customized parameter groups by implementing a function `model.get_optimizer_param_groups(cfg)`
* Supported model with ddp.

Reviewed By: zhanghang1989

Differential Revision: D31289315

fbshipit-source-id: c91ba8014508e9fd5f172601b9c1c83c188338fd

Summary:
Refactor for get_optimizer_param_groups.
* Split `get_default_optimizer_params()` into multiple functions:
  * `get_optimizer_param_groups_default()`
  * `get_optimizer_param_groups_lr()`
  * `get_optimizer_param_groups_weight_decay()`
* Regroup the parameters to create the minimal amount of groups.
* Print all parameter groups when the optimizer is created.
    Param group 0: {amsgrad: False, betas: (0.9, 0.999), eps: 1e-08, lr: 10.0, params: 1, weight_decay: 1.0}
    Param group 1: {amsgrad: False, betas: (0.9, 0.999), eps: 1e-08, lr: 1.0, params: 1, weight_decay: 1.0}
    Param group 2: {amsgrad: False, betas: (0.9, 0.999), eps: 1e-08, lr: 1.0, params: 2, weight_decay: 0.0}
* Add some unit tests.

Reviewed By: zhanghang1989

Differential Revision: D31287783

fbshipit-source-id: e87df0ae0e67343bb2130db945d8faced44d7411

Summary: Also modify launch() because it should not assume it's always called with a CfgNode object.

Differential Revision: D31494215

fbshipit-source-id: 8f07e9cb64969f8a14641956f7ef7c7160748bd9

Summary: The assert just below fails because `backend = "NCCL"` and we don't have a GPU

Reviewed By: ppwwyyxx

Differential Revision: D31506095

fbshipit-source-id: c1199eeb732d098c02fe5cd40efb85284deaa3b9

Summary: No usage: https://www.internalfb.com/code/search?q=filepath%3Ad2go%2F%20repo%3Afbcode%20_mock_func

Differential Revision: D31591868

fbshipit-source-id: 3fc6103c40713fa7bf278fd57a3e8fb4436a0902

Summary: Fix a failure bug in real image driving generating

Reviewed By: yc-fb

Differential Revision: D31362721

fbshipit-source-id: b222745aada1bd6680ca931d49a70d8b428828a6

Summary:
EMA is only applicable when testing non-predictor based models, this diff simply add a check so it won't evaluate ema models.

Side note: `do_test` should probably just handle single model, in the case of EMA, we could let `do_train` to return two models with and without ema, and call `do_test` on each of them. Then the temporary fix in this diff is not needed at all.

Reviewed By: wrlife

Differential Revision: D31450572

fbshipit-source-id: 8696922a9fd194f91315d2f3480dc8bfd8f36a3d

Summary:
the LR scheduler is cosine, so this config has no effect.
Remove it to avoid confusion.

Reviewed By: sstsai-adl

Differential Revision: D31444047

fbshipit-source-id: b40e0d7d923c3b55dfe23353050ea0238b3afd16

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/124

Update callsites from torch.quantization to torch.ao.quantization

Reviewed By: z-a-f, jerryzh168

Differential Revision: D31286125

fbshipit-source-id: ef24ca87d8db398c65bb5b89f035afe0423a5685

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/116

Reviewed By: newstzpz

Differential Revision: D30860098

fbshipit-source-id: 5c9422dd91d305193f9b43869f12423660217010

Summary:
Add get_tbx_writer to runner class and call that in the do_train. Make tbx writer overridable.

(see D31289763 for a use case)

Reviewed By: zhanghang1989

Differential Revision: D31289763

fbshipit-source-id: 19ddbbe8df62f9da0640f595532cd8f1296e3be8

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/118

This diff adds the proper support for using scripting when exporting model.

Rename tracing-related code:
- Previously `trace_and_save_torchscript` is the primary function to export model, replace it with `export_optimize_and_save_torchscript`.
- Also rename `D2 (https://github.com/facebookresearch/d2go/commit/7992f91324aee6ae59795063a007c6837e60cdb8)TorchscriptTracingExport` to `TracingAdaptedTorchscriptExport` since it's not only for tracing now.

Introduce `jit_mode`:
- Add `jit_mode` option as the `export_kwargs` of ExportMethod.
- Add `scripting` and `tracing` trigger words to overwrite `jit_mode`. Please note that the `tracing` now applies to all models, which is different from the previous meaning (using `TracingAdapter` for RCNN).
- Therefore there're two ways of using scripting mode, 1) setting `jit_mode` in prepare_for_export; 2) using `script` trigger words. Add unit test as examples to illustrate two ways.
- Don't use `TracingAdapter` when scripting since it's not scriptable.

Consolidate triggering words logic.
- Group logic of handling trigger words (eg. `_mobile`, `_int8`, `scripting`, `tracing`) into a single decorator `update_export_kwargs_from_export_method` for better structuring and readability.

Reviewed By: zhanghang1989

Differential Revision: D31181624

fbshipit-source-id: 5fbb0d4fa4c29ffa4a761af8ea8f93b4bad4cef9

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/119

Reviewed By: zhanghang1989

Differential Revision: D31181216

fbshipit-source-id: 428116f4f4144e20410222825a9a00f75253ef4a

Summary: deprecate terminate_on_nan in pytorch lightning's default trainer config

Reviewed By: kazhang, wat3rBro

Differential Revision: D30910709

fbshipit-source-id: cb22c1f5f1cf3a3236333f21be87756d3f657f78

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/117

Fix github ci failure due to lack of coco datset. It was cased by D31134064 (https://github.com/facebookresearch/d2go/commit/f018d4a7ceef437d8fc3ca8b2bba4b7321917e06)

Reviewed By: mattcyu1, wat3rBro

Differential Revision: D31179666

fbshipit-source-id: fe25129d167afcdcb577e5c8d82f3432ba939ca9

Reviewed By: zhanghang1989

Differential Revision: D31134064

fbshipit-source-id: 825ca14477243a53f84b8521f4430a2b080324bd

Summary:
D31134064 changes the default ExportMethod from `DefaultTorchscriptExport` to `D2 (https://github.com/facebookresearch/d2go/commit/7992f91324aee6ae59795063a007c6837e60cdb8)TorchscriptTracingExport` for all models. Without change, all models will be wrapped using `TracingAdapter`, which might cause unexpected effects (eg. it's not scripting friendly).

This diff add check for input/output data structure and only wrap the model when necessary.

Reviewed By: zhanghang1989

Differential Revision: D31136261

fbshipit-source-id: 4a8ffc986a5c5d61c493dd4ba0eb185aa0d54f38

Summary: write to file instead.

Reviewed By: sstsai-adl

Differential Revision: D31151549

fbshipit-source-id: 728e68182cedd625cdbe057da4162a441b80c2a4

Summary: fix optimizer setting in pytorch lightning

Reviewed By: wat3rBro

Differential Revision: D30988441

fbshipit-source-id: fcd2f4c77a87a790d7e99b0e3c833c291fd66e77

Summary: This diff adds the sum of all the losses as `total_loss` and logs it.

Reviewed By: kazhang

Differential Revision: D31063260

fbshipit-source-id: 3012dd49dd8f5fc60a7c32f3ad7a3477d2b6f5a0

Summary: Might causing https://github.com/facebookresearch/d2go/issues/113.

Reviewed By: kazhang

Differential Revision: D31066641

fbshipit-source-id: 563c2cb255b1cca4a12c8adfafc7380f140efde5

Reviewed By: ppwwyyxx

Differential Revision: D31035247

fbshipit-source-id: 7340e6f6bb813e284416e37060d0d511c5c79e03

Summary:
As title, sometimes new_ds_name is not registered so it crashes the program when calling remove(). Adding a check.
A side effect to this is if it's not registered, get() method will register it first and then remove() will remove it from registery.

Reviewed By: ppwwyyxx

Differential Revision: D31049303

fbshipit-source-id: 149168fb89fd3b661b60717ff2aafa7a9bd52849