GitLab

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/437

# Problem
- We use `TRAIN_CATEGORIES` to overrider the classes for convenient experimentation, to not have to re-map the JSON file
- But it's not possible to use the WeightedTrainingSampler with specified repeat factors (`DATASETS.TRAIN_REPEAT_FACTOR`) when also overriding the classes to use for training (ad-hoc datasets), because the underlying dataset name doesn't match the datasets specified in the `TRAIN_REPEAT_FACTOR` pairs (mapping between <dataset_name, repeat_factor>)

# Fix

- Update the dataset names for the REPEAT_FACTORS mapping as well, if we have enabled the `WeightedTrainingSampler` and use ad-hoc datasets.

Reviewed By: wat3rBro

Differential Revision: D41765638

fbshipit-source-id: 51dad484e4d715d2de900b5d0b7c7caa19903fb7

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/448

Tracing d2go runners with using adamw optimizer yielded small operators being executed in the optimizer code. They can be fused together by using the foreach version.

QPS gain is ~4.5%.

Reviewed By: miqueljubert

Differential Revision: D42004110

fbshipit-source-id: 807e0a297bb0b4272f67cc4348389294145a20eb

Summary:
X-link: https://github.com/fairinternal/detectron2/pull/589

X-link: https://github.com/facebookresearch/detectron2/pull/4702

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/443

Add MinIoURandomCrop augmentation, compare model performance.
Example of aug
{F822053068}{F822053066}{F822053051}
Data overhead is around 0.04 sec
without aug {F822053812}
with aug {F822053818}

Reviewed By: zechenghe

Differential Revision: D41804643

fbshipit-source-id: 8f13f98fa8132378a803534b59e892fbc1b3058c

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/442

Reviewed By: tglik

Differential Revision: D41714933

fbshipit-source-id: 5b2b3610af554f6082a4025af0673b4bc34b17ca

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/436

Renaming `model_ema.py` to `ema.py` (as `modeling` is already in the folder name. Fixing dependencies after rename

Reviewed By: wat3rBro

Differential Revision: D41685115

fbshipit-source-id: 006999a020a901ea8be4b71e072d688bd36cdce2

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/439

As title

Reviewed By: mattcyu1

Differential Revision: D41759804

fbshipit-source-id: 929efa960be570f0fe8543600e012d1bf037ab3b

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/432

We support caching of tuples since they behave similarly to lists

Reviewed By: XiaoliangDai

Differential Revision: D41483876

fbshipit-source-id: 9d741074f8e2335ddd737ae3f1bdb288910f5564

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/431

Add a generic domain adaptation algorithm. This algorithm:
* gets domain0 data out of the dataloader
* runs domain0 data into the model and saves target layer output
* gets domain1 data of the dataloader
* runs domain1 data into the model and saves target layer output
* runs domain adaptation loss on domain0, domain1 outputs
* combines losses using model training iteration

This diffs adds `get_preprocess_domain0_input` and `get_preprocess_domain1_input` to the distillation helper. These are functions that the user can use to convert the dataloader output to something that will be used by the model (e.g., pull the domain0 or domain1 key out of a dataloader that returns a dict).

Differential Revision: D40970724

fbshipit-source-id: fff050fbe864654fa6cb0df927f6843855ec1c14

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/430

We add losses in distillation by instantiating them in the distillation algorithm's init and then running them during the forward pass.

However this has some issues:
* the losses are not registered as a module in the model since they we organize them as a list of layerlossmetadata => this means that things like AMP do not behave as expected
* the losses are not on the same device as the rest of the model since they are created potentially after the model is moved to a new device

This diff solves both of these issues by including a helper function that registers and moves the losses to the same device as the model. `register_layer_losses_and_to_device` takes as input `List[LayerLossMetadata]`, moves the losses to the same device as the model and then registers these losses to the model.

Differential Revision: D41296932

fbshipit-source-id: ae7ae0847bce1b5cc481d838b9cae69cea424f25

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/429

Add a teacher type called `no_teacher` which can be specified by the user in the case they ignore the teacher (e.g., domain adaptation). Building the teacher just returns a noop (`nn.Identity`)

Differential Revision: D40971788

fbshipit-source-id: fc49ac44224c92806a7be253eefb8454305814eb

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/428

add an augmentation to pad image to square.
* For example, image with shape (10, 7, 3) will become (10, 10, 3) and pad with value specified by `pad_value`.

Reviewed By: tax313, wat3rBro

Differential Revision: D41545182

fbshipit-source-id: 6d5fd9d16984a9904d44f22386920cdf130edda7

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/433

Distillation uses a module called `CachedLayer` to record the outputs of a layer to an internal dict. This dict is typically initialized by the object itself and any value is overwritten every time the model runs.

However, sometimes we need more than one output run of the layer (e.g., domain adaptation => we run the model on real, then synthetic data and need to use both outputs).

This diff adds a helper to set externally set the cache dict of a model. In other words, we can run `set_cache_dict` on some model to change the dict used by all `CachedLayer` in the model. This allows us to run the model and record some outputs, then change the cache dict and rerun the model to save different outputs.

Differential Revision: D40970577

fbshipit-source-id: 49cb851af49ae193d0c8ac9218e02fdaf4e6587b

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/427

Re-try previous reverted diff D41350485 (https://github.com/facebookresearch/d2go/commit/0ea6bc1b61ab736ccf1840c58c2b19ed2e9a1282). The problem was essentially because `DefaultTask` is not a subclass of `Runner`, so when we call `Runner`'s class methods from `DefaultTask`, it won't work if the `Runner`'s method also calls other methods that are in `Runner` but not `DefaultTask`. The solution is simply split the data related APIs out into a separate class (mixin), and let `DefaultTask` and `Runner` both subclass from it.

Reviewed By: tglik

Differential Revision: D41507448

fbshipit-source-id: 8b26c129811436c0bd35e1c6b0705e7035d7e823

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/426

This diff is reverting D41350485 (https://github.com/facebookresearch/d2go/commit/0ea6bc1b61ab736ccf1840c58c2b19ed2e9a1282)
D41350485 (https://github.com/facebookresearch/d2go/commit/0ea6bc1b61ab736ccf1840c58c2b19ed2e9a1282) has been identified to be causing the following test or build failures:
Tests affected:
- https://www.internalfb.com/intern/test/281475052494930/
- https://www.internalfb.com/intern/test/844425005925528/
- https://www.internalfb.com/intern/test/562950029222822/

Here's the Multisect link:
https://www.internalfb.com/intern/testinfra/multisect/1429252
Here are the tasks that are relevant to this breakage:
T120995919: 4 tests started failing for oncall d2go in the last 2 weeks
We're generating a revert to back out the changes in this diff, please note the backout may land if someone accepts it.

Reviewed By: wat3rBro

Differential Revision: D41470376

fbshipit-source-id: 7d2074e150e6b36a3c260317f859c7f2131295db

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/422

Add a logger to distillation so we can check when distillation is applied

Reviewed By: XiaoliangDai

Differential Revision: D40375046

fbshipit-source-id: bb1d821fa26fb2da75e82122a30307fcccf7e558

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/421

Add some reasonable defaults when running knowledge distillation
* get_default_kd_image_classification_layer_losses => returns cross entropy loss on the output of the student classification layer and the teacher output (this is what the imagenet distillation uses)
* DefaultLossCombiner => simple function to multiply the losses by some weights

Unsure if these should go in `distillation.py` or a separate place (e.g., defaults or classification)

Reviewed By: chihyaoma

Differential Revision: D40330718

fbshipit-source-id: 5887566d88e3a96d01aca133c51041126b2692cc

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/423

The `DefaultTask` has forked the implementation of some runner methods from `Detectron2GoRunner`, which is not necessary since there should be no differences. This might cause issue that we update the one from `Detectron2GoRunner` but forgot about `DefaultTask`.

Reviewed By: chihyaoma

Differential Revision: D41350485

fbshipit-source-id: 38a1764a7cc77dc13939ac7d59f35584bf9dab9b

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/420

Adds knowledge distillation as a generic algorithm that can be used by various projects.

If eval, the algorithm just returns the result of the student model.

If training, the algorithm feeds the input into both the student and teacher model. The user provides a list of `LayerLossMetadata` that provides the layers and losses run on these layers. The algorithm uses dynamic mixin to record the outputs of the relevant layers and compute the losses after both models are run.

We provide student and teacher preprocessing as a placeholder before we support a more generic dataloader which can provide different inputs to the student and teacher (e.g., as of now, if you want to provide the teacher with a larger input then the dataloader should return a large input and the student preprocessing can downsample the input).

We add the following functions as part of the user customizable distillation helper:
* get_teacher => return a teacher that can be used directly by the KD algorithm
* get_layer_losses => return a list of `LayerLossMetadata` that provides the layers and losses
* get_preprocess_student_input => manipulate the output of the dataloader before passing to the student
* get_preprocess_teacher_input => manipulate the output of the dataloader before passing to the teacher
* get_combine_losses => since we may want to weight the student and distillation losses, return a function that can manipulate the loss_dict

Reviewed By: chihyaoma

Differential Revision: D40326412

fbshipit-source-id: 2fb0e818a7d5b120d62fb7aba314ff96cc7e10c5

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/424

The diffusion model need to customized eps for the AdamW optimizer. For example the prior of Dalle2 trained with adam_eps=1.0e-06 while its decoder trained with adam_eps=1.0e-08

Add the config for AdamW's eps to d2go pipeline with its default vaule same as the official doc: https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html

Reviewed By: newstzpz

Differential Revision: D41363142

fbshipit-source-id: 72f9a4084e229312807aad28b7aba8fec9116013

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/396

Integrate PyTorch FSDP, which supports two sharding modes: 1. gradient + optimizer sharding; 2. full model sharding (params + gradient + optimizer). This feature is enabled in the train_net.py code path.

Sources
* Integration follows this tutorial: https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html

API changes
* Add new config keys to support the new feature. Refer to mobile-vision/d2go/d2go/trainer/fsdp.py for the full list of config options
* Add `FSDPCheckpointer` as an inheritance of `QATCheckpointer` to support special loading/saving logic for FSDP models

Reviewed By: wat3rBro

Differential Revision: D39228316

fbshipit-source-id: 342ecb3bcbce748453c3fba2d6e1b7b7e478473c

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/419

This diff adds a metadata class `LayerLossMetadata` to help keep track of the losses we want to compute over layers. The class contains the type of loss, loss name, and layer names.

This diff adds a helper function to iterate over a list of `LayerLossMetadata` and return a dict containing the results.

Reviewed By: chihyaoma

Differential Revision: D40286564

fbshipit-source-id: b269dc63cc90a437ca279379d759c3106016327c

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/418

This diff adds a function that can be used to add `CachedLayers` to a model. Function iterates over named modules and dynamically mixes in `CachedLayer` to target modules.

This diff adds a function to remove the cached layers.

Reviewed By: Minione

Differential Revision: D40285806

fbshipit-source-id: 3137d19927d8fb9ec924a77c9085aea29fe94d5e

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/417

This diff adds a layer `CachedLayer` which is meant to be used with dynamic mixin. This layer runs the original module and clones the output into a dictionary provided by the user.

The main use case is in distillation where we dynamically mixin these layers to the layers that the user wants to compute various losses.

See subsequent diffs to get integration with distillation.

Reviewed By: Minione

Differential Revision: D40285573

fbshipit-source-id: 2058deff8b96f63aebd1e9b9933a5352b5197111

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/416

Distillation assumes teacher model has an attribute "device". Sometimes this attribute is actually a property (e.g., generalizedrcnn) but there is zero guarantee that it exists. We add a helper function to move the model to the device and add this attribute if needed.

Reviewed By: chihyaoma

Differential Revision: D40283954

fbshipit-source-id: 42921653eac8a79499e22edac29aa6aeac016e8a

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/415

The user can build a teacher by providing a trained config. However this model may have been trained using gpu whereas the user wants to load the model on cpu, this diff supports this use case by allowing the user to specify `cfg.DISTILLATION.TEACHER.DEVICE` as override.

Reviewed By: sstsai-adl

Differential Revision: D40125236

fbshipit-source-id: f1fd797a155e12b31bb7fcbc5e4997ee8eb23539

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/388

Reviewed By: wat3rBro

Differential Revision: D40377653

fbshipit-source-id: 3f99d30480a801c794665e67bb2b0d28c7c5b0e5

Summary:
X-link: https://github.com/facebookresearch/detectron2/pull/4654

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/412

Support custom precision dtype [float16, bfloat16] for AMP training on D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb) backend. There's an old config key `SOLVER.AMP.PRECISION` that only works on lightning backend. This diff enables this config key on D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb) backend (train_net binary) as well.

Reviewed By: tax313, wat3rBro

Differential Revision: D40811604

fbshipit-source-id: 58da17ae1519a54243b5295eb4253c297e4d9296

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/413

Switch to using nearest pixel interpolation when warping and added unit test.

Reviewed By: wat3rBro

Differential Revision: D41042506

fbshipit-source-id: 92b817f21e862422428a0d0df969ec9e037f99fb

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/410

1. Fix bounding box coordinate warping
2. Add apply_segmentation warning (will followup in another fix)

Reviewed By: wat3rBro

Differential Revision: D41013775

fbshipit-source-id: 3652b04c1622fe35fa9893dc22350f7d59b37c6e

Summary:
X-link: https://github.com/facebookresearch/detectron2/pull/4633

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/406

Profiling yielded the training could benefit from using the parallel version of the SGD optimizer.
Passing `foreach=True` to enable.

Reviewed By: tglik

Differential Revision: D40798214

fbshipit-source-id: aa098d1fbbece0862bc9343df761765b0c3b15da

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/405

- Use the non-hacky way (added in D40818736, https://github.com/facebookresearch/detectron2/pull/4626) to customize offloaded backend for DatasetFromList.
- In `D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go`, switch to use `SharedList` (added in D40789062, https://github.com/facebookresearch/mobile-vision/pull/120) by default to save RAM and optionally use `DiskCachedList` to further save RAM.

Local benchmarking results (using a ~2.4 GiB dataset) using dev mode:
| RAM usage (RES, SHR) | No-dataset | Naive | NumpySerializedList | SharedList | DiskCachedList |
| -- | -- | -- | -- | -- | -- |
| Master GPU worker.         | 8.0g, 2.8g | 21.4g, 2.8g | 11.6g, 2.8g | 11.5g, 5.2g | -- |
| Non-master GPU worker  | 7.5g, 2.8g | 21.0g, 2.8g | 11.5g, 2.8g | 8.0g, 2.8g | -- |
| Per data loader worker     | 2.0g, 1.0g | 14.0g, 1.0g | 4.4g, 1.0g | 2.1g, 1.0g | -- |

- The memory usage (RES, SHR) is found from `top` command. `RES` is total memory used per process; `SHR` shows how much RAM can be shared inside `RES`.
- experiments are done using 2 GPU and 2 data loader workers per GPU, so there're 6 processes in total, the **numbers are per-process**.
- `No-dataset`: running the same job with tiny dataset (only 4.47 MiB after serialization), since RAM usage should be negligible, it shows the floor RAM usage.
- other experiments are running using a dataset of the size of **2413.57 MiB** after serialization.
  - `Naive`: vanilla version if we don't offload the dataset to other storage.
  - `NumpySerializedList`: this optimization was added a long time ago in D19896490. I recalled that the RAM was indeed shared for data loader worker, but seems that there was a regression. Now basically all the processes have a copy of data.
  - `SharedList`: is enabled in this diff. It shows that only the master GPU needs extra RAM. It's interesting that it uses 3.5GB RAM more than other rank, while the data itself is 2.4GB. I'm not so sure if it's overhead of the storage itself or the overhead caused by sharing it with other processes, since non-master GPU using `NumpySerializedList` also uses 11.5g of RAM, we probably don't need to worry too much about it.
  - `DiskCachedList`: didn't benchmark, should have no extra RAM usage.

Using the above number for a typical 8GPU, 4worker training, assuming the OS and other programs take 20-30GB RAM, the current training will use `11.6g * 8 + 4.4g * 8*4 = 233.6g` RAM, on the edge of causing OOM for a 256gb machine. This aligns with our experience that it supports ~2GB dataset. After the change, the training will use only `(11.5g * 7 + 8.0g) + 2.1g * 8*4 = 155.7g` RAM, which gives a much larger head room, we can thus train with much larger dataset (eg. 20GB) or use more DL workers (eg. 8 workers).

Reviewed By: sstsai-adl

Differential Revision: D40819959

fbshipit-source-id: fbdc9d2d1d440e14ae8496be65979a09f3ed3638

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/409

`assert_close` is preferred over `assert_allclose`: https://github.com/pytorch/pytorch/issues/61844

The `assert_allclose` was removed yesterday in https://github.com/pytorch/pytorch/pull/87974, causing test to fail, eg. https://github.com/facebookresearch/d2go/actions/runs/3389194553/jobs/5632021291

Reviewed By: sstsai-adl

Differential Revision: D41000306

fbshipit-source-id: 7bd1cb9d5edf0a4609a909e2283df411bcabdf13

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/408

Update the DPE training script in D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go to support Turing DPE QAT.

Reviewed By: newstzpz

Differential Revision: D40612406

fbshipit-source-id: 9379e4be248045b995293c5a522bab05e0b13c6e

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/407

As is, observers can be periodically updated even if observer is disabled. Add a check on whether observer is disabled before updating observers periodically.

Reviewed By: tglik

Differential Revision: D40734564

fbshipit-source-id: fd3e50e95c95ad39bd993f71e02dc7acc79744cc

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/403

`cfg.SOLVER.AMP.ENABLED` enabled mixed precision, but this only works for V100 GPUs.
For A100s, the equivalent is to enable TF32.

Reviewed By: tglik

Differential Revision: D40675242

fbshipit-source-id: 5cc3d12cd3d7ec76665e0907ecc87fc5f64d73f0

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/404

`get_default_cfg` is now class method since stack of D37294926 (https://github.com/facebookresearch/d2go/commit/b077a2c13845d4ef8481979d64345368864fe5ff), this diff updates call sites using biggrep to replace "Runner().get_default_cfg" with "Runner.get_default_cfg"

Reviewed By: itomatik

Differential Revision: D40707898

fbshipit-source-id: 2b56545769d930d34dad8814d5bfeba4c54224fd

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/392

1. Moved scale adjustment to a separate function and expose the option to disable it
2. Add option to keep the original image instead of creating a square image

Reviewed By: wat3rBro

Differential Revision: D40403705

fbshipit-source-id: 6c35a9a1fe3ef868e5f0b2204874fd028776e26a

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/401

as followup on D40001329 (https://github.com/facebookresearch/d2go/commit/69bf820c64cd0ffb6a84f465199c9134814cf58e).
The export is running main func without launching distributed workers, so it need to set the shared context explicitly.

Reviewed By: wat3rBro

Differential Revision: D40708631

fbshipit-source-id: 7689a45dff383ba2cce01d33d3be95d612269fbe

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/402

print out the optimizer for easier debug

Reviewed By: newstzpz

Differential Revision: D40701959

fbshipit-source-id: 7b610e8f5771409632ae056cb9d34138b331adbc

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/399

Freezing the model before running quantization causes an issue with loading a saved checkpoint bc fusing does not support FrozenBatchNorm2d (which means that the checkpoint could have a fused weight conv.bn.weight whereas the model would have an unfused weight bn.weight). The longer term solution is to add FrozenBatchNorm2d to the fusing support but there are some subtle issues there that will take some time to fix:
* need to move FrozenBatchNorm2d out of D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb) and into mobile_cv lib
* current fuser has options to add new bn ops (e.g., FrozenBatchNorm2d) which we use with ops like SyncBN but this currently is only tested with inference so we need to write some additional checks on training

The swap will make freezing compatible with QAT and should still work with standard models. One subtle potential issue is that the current BN swap assumes that BN is a leaf node. If a user runs QAT without fusing BN, the BN will no longer be the leaf node as it will obtain an activation_post_process module in order to record the output. The result is that BN will not be frozen in this specific instance. This should not occur as BN is usually fused. A small adjustment to the BN swap would just be to swap the BN regardless of whether it is a leaf node (but we have to check whether activation_post_process module is retained). Another long term consideration is moving both freezing and quant to modeling hooks so the user can decide the order.

Reviewed By: wat3rBro

Differential Revision: D40496052

fbshipit-source-id: 0d7e467b833821f7952cd2fce459ae1f76e1fa3b