GitLab

Enable torch tracing by changing assertions in d2go forwards to allow for torch.fx.proxy.Proxy type.

Summary:
X-link: https://github.com/facebookresearch/detectron2/pull/4227

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/241

Torch FX tracing propagates a type of `torch.fx.proxy.Proxy` through the graph.

Existing type assertions in the d2go code base trigger during torch FX tracing, causing tracing to fail.

This adds a check for FX tracing in progress and  adds a helper function `assert_fx_safe()`, that can be used in place of a standard assertion. This function only applies the assertion if one is not tracing, allowing d2go assertion tests to be compatible with FX tracing.

Reviewed By: wat3rBro

Differential Revision: D35518556

fbshipit-source-id: a9b5d3d580518ca74948544973ae89f8b9de3282

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/359

Currently, D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go is missing the Adam optimizer. This Diff addresses the gap.

Reviewed By: tglik, asanakoy

Differential Revision: D38492151

fbshipit-source-id: 27791c23c73942b7a466f2ca91f6b3631733ba16

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/352

Reviewed By: newstzpz

Differential Revision: D37872639

fbshipit-source-id: 61acdaa669bc541dcb715af1172926efb53c0b2b

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/357

This change makes it possible to unpickle TrainNetOutput which is currently cannot be unpickled because it's a part of main module which can be different for the binary that's unpickling this dataclass.

Reviewed By: miqueljubert

Differential Revision: D38536040

fbshipit-source-id: 856594251b2eca7630d69c7917bc4746859dab9f

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/356

Attaching PDB on failure is not working when running in distributed environment. This change allows to disable this behavior by passing a command line argument.

Reviewed By: miqueljubert

Differential Revision: D38514736

fbshipit-source-id: 2e0008d6fbc6a4518a605debe67d76f8354364fc

Summary:
X-link: https://github.com/facebookresearch/detectron2/pull/4458

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/353

- The QAT was using old code prior to D36786902, update to use public API
- Make `trainer:reset_data_loader` to take lazy lambda expression in order to delay the creation of dataloader. It's possible that we don't have enough RAM to hold two data loader at the same time, so we need to delete the first one, then create the second one.

Differential Revision: D38330148

fbshipit-source-id: aae28a48eabf211fe00cafe5d9ea8aeaf56e4e0c

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/349

This is to allow None, meaning model_configs is not used.

Added tasks for the other TODO.

Reviewed By: wat3rBro

Differential Revision: D38199075

fbshipit-source-id: 774ca42a82a972b7e4c642cc4306aec39e2c2f7f

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/344

we need access to the modeling hooks in EMA, e.g. build trainer.

Reviewed By: wat3rBro

Differential Revision: D37997773

fbshipit-source-id: bf4372cd310605fa35aa70f0604b084b047001d8

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/278

Allow skipping do_test after do_train.

Reviewed By: wat3rBro

Differential Revision: D36786790

fbshipit-source-id: 785556b5743ee9af2abfe6c0e9e78c7055697048

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/348

Add testcase to ensure loading from config in eval_only is covered.

Reviewed By: wat3rBro

Differential Revision: D38001319

fbshipit-source-id: e6a2edb5001ae87606a3bf48e1355037aee0f9a0

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/342

Add a cfg option to control the frequency of the writers.

Currently, the default writers include:
```
writers = [
    CommonMetricPrinter(max_iter),
    JSONWriter(os.path.join(cfg.OUTPUT_DIR, "metrics.json")),
    tbx_writer,
]
```

Reviewed By: wat3rBro

Differential Revision: D38065583

fbshipit-source-id: ebdc20aab71e03b4e18772af78b410f17ba4216d

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/347

land the fbnet architectures for video logo detection project in RCNN, FCOS and YOLO

Reviewed By: wat3rBro

Differential Revision: D38139055

fbshipit-source-id: 6ba21f482ed067c52d438e0c217e523896c2131c

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/322

TorchVision has recently added the AugMix Augmentantion. This diff adds support of the specific transform to D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)go

Reviewed By: newstzpz

Differential Revision: D37578243

fbshipit-source-id: b793715ccb24a3bd999a40c51d8c9a75f22110a3

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/345

Reviewed By: xiecong

Differential Revision: D38086885

fbshipit-source-id: 808e104ee50c8870ae091533ac67b440e1bb8351

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/343

Reviewed By: miqueljubert

Differential Revision: D38077850

fbshipit-source-id: a79541d899ce2b49a30c7f2a81a616f76321026f

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/340

Reviewed By: miqueljubert

Differential Revision: D37968017

fbshipit-source-id: a3953fdbb2c48ceaffcf94df081c0b3253d247d5

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/339

add is_qat to lightning codepath

Reviewed By: jerryzh168

Differential Revision: D37937336

fbshipit-source-id: 68debe57c7f7dcf8647fad6ab9e34eff2aaa851c

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/338

Now we should've separated all the `prepare_for_quant` for eager and FX mode, we can remove this branch.

Reviewed By: jerryzh168

Differential Revision: D37865628

fbshipit-source-id: cd8f3aa7c90201f44bcfdbd65eb2edf5eded0e0c

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/336

Reviewed By: jerryzh168

Differential Revision: D37860495

fbshipit-source-id: 1ce0bc7bc8071d3bfbe53cd61ed180da62e29327

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/335

Manually remove `example_input` from eager-mode-only `prepare_for_quant`.

Reviewed By: jerryzh168

Differential Revision: D37838155

fbshipit-source-id: 2d98e0264fc0c40dcf1b6f28f7fc635c52acd75e

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/333

Follow D36916149.

Reviewed By: jerryzh168

Differential Revision: D37830568

fbshipit-source-id: dbeb204ccf96dd2e90a6509f24a2864503083f60

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/332

Solve: "we have decoupled qat with `.training` in quantization, maybe we should use some flags in `cfg` instead of checking this attribute here as well"

Reviewed By: jerryzh168

Differential Revision: D37801241

fbshipit-source-id: ed9884d7b462da195ed2e07c42634acfe5beefb2

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/330

- re-enable the `test_qat`
- remove `example_input` from `DetMetaArchForTest`, since its `custom_prepare_fx` just create a tensor for avgpool.

Reviewed By: jerryzh168

Differential Revision: D37793260

fbshipit-source-id: ec7a825c61292d9c6d792f910a957c1c27832336

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/331

- remove `example_input` from `default_prepare_for_quant` since now it's eager mode only.
- rename `apply_prepare_for_quant` to `convert_to_fake_quant_model` to reflect what it's doing.

Reviewed By: jerryzh168

Differential Revision: D37794085

fbshipit-source-id: e6f12098976272d979a3aac66287d9b95432dcc8

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/328

Reviewed By: jerryzh168

Differential Revision: D37695452

fbshipit-source-id: 744b1085365d1e155ea9e9fe51a6237994d90fa7

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/326

since prepare_for_quant_convert/custom_convert_fx is FX only, we can remove eager mode code from it. Also adjust the logic in `d2go/export/exporter.py` accordingly to reflect this.

Reviewed By: jerryzh168

Differential Revision: D37676977

fbshipit-source-id: ebd05082ee81bc1ac32fcc2a87bc0dfaacedd5bd

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/325

`prepare_for_quant_convert` is a confusing name because it only does `convert`, there's no "prepare" in it. It's actually for fx only, because eager mode always calls `torch.quantization.convert`, there's no way to customize it. So just call this `custom_convert_fx`. The new name is also unique in fbcode, so easy to do codemod later on.

This diff simply does the renaming by biggrep + replace.

Reviewed By: jerryzh168

Differential Revision: D37676717

fbshipit-source-id: e7d05eaafddc383dd432986267c945c8ebf94df4

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/327

Fix the bug for not leaving avgpool quantized when its input is quantized. This is not captured by CI because currently we set `small_pooler_resolution` to True to make test running faster, but this skips the `avgpool` everytime.

Reviewed By: jerryzh168

Differential Revision: D37706151

fbshipit-source-id: fde9a190324a23b97ba803e000f4caa54f6f3738

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/324

Fixes bug introduced in D37600026 (https://github.com/facebookresearch/d2go/commit/1a18ba3420e3c823accf731a11c0a91dc3babd85) -- forgot to fix imports after moving modelinghook to registry/builtin.py

Differential Revision: D37646330

fbshipit-source-id: cb763d65e7bbfd07eea6eff61727a42a6fcfbc88

Summary:
Moving MODELING_HOOK_REGISTRY to `d2go/registry/builtin.py`, so that  bootstrap works with modeling hook registry.

This fixes the detectron2 vs mobile_cv registry.

Reviewed By: wat3rBro

Differential Revision: D37600026

fbshipit-source-id: 7aa8cdec3b5eaea89d9db6fc35c7389cc9fb3193

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/321

Following up the bc-breaking change from fx graph mode quantization: https://github.com/pytorch/pytorch/pull/76496 that
added example_inputs to prepare_fx and prepare_qat_fx, we fixes the callsite related to mobile-vision and exposed
extra example_inputs in some apis

Reviewed By: wat3rBro

Differential Revision: D37163018

fbshipit-source-id: 9f0bb56659345d174a39b6d3bb4408caa553b88d

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/320

MCV/D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go's `launch` now supports `kwargs`, which matches elastic launch. Let's always use `args=(cfg, output_dir, runner_name)` for all the binaries, and use `kwargs` for remaining binary arguments (which matches the `extra_args` in FBL's OperatorArgument).

Reviewed By: sstsai-adl

Differential Revision: D37535145

fbshipit-source-id: 9767e8d71421d2262aee1fd4b9019758aa4a6bbd

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/319

follow up on D37500599 (https://github.com/facebookresearch/d2go/commit/668b7ac29b0afb55d5923e72fe4f6428e5c85cbd), move lightning_train_net part of D37367360 to this diff.

Reviewed By: sstsai-adl

Differential Revision: D37534370

fbshipit-source-id: 7f48942a14ce16a9a9540b189441b540ce4f4b25

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/318

Reviewed By: mcimpoi

Differential Revision: D37501246

fbshipit-source-id: 6dbe5dcbaf7454f451d4a3bb3fa2d856cc87d5cc

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/317

1. Add eval-only only option in similar fashion with train_net
2. Use output_dir from config is not specified via command line

Reviewed By: wenliangzhao2018

Differential Revision: D37500599

fbshipit-source-id: 00c5804d08a449def3cc15fff49e27066d01f229

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/316

As title.

Reviewed By: wenliangzhao2018

Differential Revision: D37259150

fbshipit-source-id: 6b69d4bcc8706bf50dbbe3655b306d1d9cbfbdca

Summary:
X-link: https://github.com/facebookresearch/mobile-vision/pull/92

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/314

We may import d2go first, which populates some builtin meta archs in registry, then do the bootstrap.

Reviewed By: mcimpoi

Differential Revision: D37427101

fbshipit-source-id: 00e53bf0d2e327a90d5dfbc23ef2f4daa6ddfc63

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/315

This method is no longer needed given the there's a better
alternative: save/load_binary_output.

Reviewed By: tglik, wat3rBro

Differential Revision: D37307668

fbshipit-source-id: 598a6d7aeb68f65fa6b42a230175c37d19fd7798

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/313

It's not natural to put runner's default config functions under `d2go/utils/`, move them to `d2go/runner/config_defaults.py` and clean things up. This also reduce the inter-sub-package dependencies.

Reviewed By: mattcyu1

Differential Revision: D37407078

fbshipit-source-id: 432644bee4f12306a14bac3dba76ced08b3683aa

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/309

Right now multiple machines can try to write to the same output file,
since they get the same argument. Additionally, on the same machine, several
outputs can be saved which requires unncessary unpacking. This change makes
train_net only write output of the rank 0 trainer.

Reviewed By: wat3rBro

Differential Revision: D37310084

fbshipit-source-id: 9d5352a274e8fb1d2043393b12896d402333c17b