Skip to content

GitLab

Switch branch/tag
  1. 01 May, 2023 3 commits
  3. 21 Apr, 2023 1 commit
    • Tao Xu's avatar
      enable the diffusion visualization evaluators to run on multiple datasets · c7bd7dfe
      Tao Xu authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/527

- Add model.reset_generation_counter() to enable the diffusion visualization evaluators to run on multiple test datasets.
  - Before this fix, the visualization evaluators will only run on the 1st test dataset since self.generation_counter will set to <0 after running on the 1st test datasaet. Thus the visualization evaluators will skip for all the other test sets since self.generation_counter < 0.
- Use the ddim for upsampler by default for better results

Reviewed By: zechenghe

Differential Revision: D45058672

fbshipit-source-id: 2f7919bf6ecd2e5f6f242ce3e7891cb3dc8d6af4
      c7bd7dfe
  5. 20 Apr, 2023 2 commits
  7. 18 Apr, 2023 1 commit
  9. 11 Apr, 2023 2 commits
  11. 05 Apr, 2023 2 commits
    • Mik Vyatskov's avatar
      Setup root logger once & on import time · abdeafb0
      Mik Vyatskov authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/523

To avoid setting it up multiple times, add run_once() decorator.

Additionally make sure logging is configured for datalodaing workers, which have a different entry point, by moving setting up logging to the import time. Right now when a dataloader worker is created using spawn method from multiprocessing module, a new Python interpreter is created, with all the modules imported anew and with the entry point set to the method specified. This means that the entry point of the training framework is skipped, together with the logging setup.

With this change, the logging is configured on the import time, which means that when a dataloading process is created, even though the training main is not invoked, the logging is still configured because even though train_net is not invoked as an entry point, it's still imported in the child process.

Reviewed By: miqueljubert

Differential Revision: D44641142

fbshipit-source-id: 06ea85363d965b31d7f9ade3c2615ed9db67470b
      abdeafb0
    • Anthony Chen's avatar
      change default FSDP strategy to grad_optim (ZERO2) · 35affd74
      Anthony Chen authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/522

Change d2go's default FSDP sharding strategy to grad_optim, which corresponds to ShardingStrategy.SHARD_GRAD_OP in FSDP API, or ZERO2 in literature. grad_optim is shown to have the best tradeoff between memory utilization and training speed for mid-sized models.

`FSDP.ALGORITHM = ""` was from the previous design to indicate that no FSDP is used. It will not work now

Reviewed By: tglik

Differential Revision: D44657184

fbshipit-source-id: 3888eea5f2b5042269e69453f3cdd8db7cf1581c
      35affd74
  13. 03 Apr, 2023 1 commit
  15. 31 Mar, 2023 2 commits
  17. 30 Mar, 2023 4 commits
    • David Yan's avatar
      Update AIInfraCheckpointer to use the new gather/scatter functions for EMA and optimizer states · fe8680c1
      David Yan authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/520

- Move gather/scatter functions to their own util function
- Make changes to onboard AIInfraCheckpointer to the gather/scatter functions for optimizer and ema state
- Add a test for fsdp checkpointer and ai infra checkpointer

Reviewed By: YanjunChen329

Differential Revision: D44400633

fbshipit-source-id: bcfe3e0a4fbf53f91a83e88f74c4538699a50293
      fe8680c1
    • David Yan's avatar
      Save and load model EMA state for sharded state dicts in FSDPCheckpointer · e7652751
      David Yan authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/519

Prior to this, FSDP checkpointer did not save EMA state which matched the model state when the model used sharded state dict. This diff adds this functionality.

Reviewed By: YanjunChen329

Differential Revision: D44270790

fbshipit-source-id: f522765ad56e8279f355c43a19f26c3b6bcf01e3
      e7652751
    • Mircea Cimpoi's avatar
      Run zoomer profiling · 67267821
      Mircea Cimpoi authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/518

Enable profiling for eval step only, not on every eval (which can be called during training)

Reviewed By: frabu6

Differential Revision: D44535915

fbshipit-source-id: 4497a3f74f5d751277df9ed41bc9bf21056341c4
      67267821
    • Anton Rigner's avatar
      Read metadata for actual dataset in Visualizer, if available · c4b2d09c
      Anton Rigner authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/516

# Context

D2go allows for training with more than one datasets, and as long as the categories are consistent, the IDs do not necessarily have to correspond to each other between annotations of two different data sets.

It is still loaded correctly to the data loader, and the training works as expected.

# Problem

However, I observed weird mis-labelleing issues in the Visualizer for Tensorboard. Originally I thought this was a data/conversion issue, but upon inspecting the logs I see that the data is loaded correctly. See example below.

{F924075931}

"Plant" labelled as "Refrigerator", "Floor" labelled as "Lamp"

{F924078113}

... but the loaded annotations doesn't actually contain any samples of "Refrigerator".

The reason is that the Visualizer always loads the metadata (and thus the labels) from the first train data set, but the order of the categories between the data sets may not be consistent, but still be a valid training run.

# Fix

If there is a data set name associated with the data to visualize, use that to fetch the metadata, and the correct labels, otherwise default to the first data set (current situation).

Reviewed By: wat3rBro

Differential Revision:
D44495363

Privacy Context Container: L1127277

fbshipit-source-id: 37b940d393aa794cd2f39aabdc66c6d23abd8000
      c4b2d09c
  19. 26 Mar, 2023 1 commit
  21. 24 Mar, 2023 2 commits
    • David Yan's avatar
      Add tests for sharded_state_dict and fix compatibility problems · 46606a02
      David Yan authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/511

Add tests for sharded_state_dict integration in AIF Checkpointer

Fix compatibility problems including:
1. small API errors of flatten_sharded_optim_state_dict
2. deprecate model.use_local_state_dict and model.load_local_state_dict
3. fix auto conversion for local_state_dict
4. fix T148056077: add metadata to differentiate between local_state_dict and sharded_state_dict when loading a directory with FSDPCheckpointer

Reviewed By: YanjunChen329

Differential Revision: D44160045

fbshipit-source-id: f607b7076d0e49b9407f9adfbc8ecfe439c3b0c9
      46606a02
    • David Yan's avatar
      Add support for FSDP SHARDED_STATE_DICT in D2Go · fbc1c2e8
      David Yan authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/512

Currently, when saving and loading checkpoints for FSDP-wrapped modules, we are saving and loading using `StateDictType.LOCAL_STATE_DICT`, where the state_dict becomes essentially a single flat tensor under the `_flat_param` key (or some other layer-specific key for flat weights). This means that
1. It's impossible to load weights directly from checkpoints, for example in notebooks
2. Converting from a local to a global checkpoint requires running a special workflow (https://fburl.com/code/6yqa4ldb) that occupies the same number of GPUs as was used during training

This diff adds an option, `FSDP.STATE_DICT_TYPE`, which allows selection of the type of state dict to save (local, sharded, full). In sharded mode, with AIF checkpointing, we are able to have the benefit of allowing local loading of state dicts in minutes with any number of GPUs, in notebooks and elsewhere.

Note: for backwards compatibility, `CFG.FSDP.use_local_state_dict` and `CFG.FSDP.load_local_state_dict` still need to work when the new config parameter (`CFG.FSDP.state_dict_type`) is not set. Also, it's used to signify that local/sharded state dicts need to be converted to a full state dict when loading. This functionality can be deprecated when everyone migrates to AIF checkpointing with sharded dicts.

Reviewed By: YanjunChen329

Differential Revision: D43840887

fbshipit-source-id: d112f7b7ad97ba82fd5bf1da986b95ad7fc61c42
      fbc1c2e8
  23. 23 Mar, 2023 1 commit
    • Mik Vyatskov's avatar
      Redirect prints to logging module · d912e9f8
      Mik Vyatskov authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/509

print function is used all over the place and it's not realistic to enforce not using print for everyone. So this diff attempts to improve the debuggability of the code that was written using prints by redirecting prints to the logging module.

Additionally call logger setup from `setup_after_launch` to make sure logging settings are applied in every of the spawned processes.

Reviewed By: frabu6, wat3rBro

Differential Revision: D44280241

fbshipit-source-id: 713400ac2b2edacef3c7a99067cbb1e684c3c5ad
      d912e9f8
  25. 22 Mar, 2023 2 commits
  27. 21 Mar, 2023 2 commits
  29. 16 Mar, 2023 1 commit
  31. 11 Mar, 2023 1 commit
  33. 09 Mar, 2023 1 commit
  35. 08 Mar, 2023 1 commit
  37. 06 Mar, 2023 1 commit
    • Alan Lin's avatar
      Add export registry for FCOS · 8cb50233
      Alan Lin authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/491

As titled, although FCOS usually requires no customized export methods. We found that our internal MUI platform asks the exported model to follow certain protocols. To avoid mixing-up with external code, adding a export func registry to bypass it.

Reviewed By: wat3rBro

Differential Revision: D43800839

fbshipit-source-id: 41c8ebb10610ec92d17461211315c15908277b28
      8cb50233
  39. 05 Mar, 2023 3 commits
    • Fei Sun's avatar
      Move EMA to after backward. · a7dc757c
      Fei Sun authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/494

Currently EMA computation is in the after step hook. It is in the critical path where no other work is available. This increases the training iteration time. This diff moves the EMA computation to after the backward but before the optimizer step. This way, the majority of the EMA computation time on the CPU can be hidden since CPU at that time is waiting for the GPU to finish the backward anyway. This change may completely hide the EMA CPU time. It reduces the EMA time from 20ms to 4ms, where the 4ms is the GPU time.

However, with this change, the EMA gets its value from the previous iteration value (since it is before step). but since we do many epochs of training, one iteration difference may not be significant.

Reviewed By: tglik

Differential Revision: D43527552

fbshipit-source-id: 1faa9d910b20cae0fc77da541bc0ad176bce18a8
      a7dc757c
    • Fei Sun's avatar
      Prefetch forward · 5f1ef548
      Fei Sun authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/492

Enable prefetching the FSDP all gathers. Forward prefetch may or may not improve performance. Its effectiveness is determined by other FSDP options, such as zero2/zero3, HSDP/FSDP. Need to do a HPO sweep to figure out the best configuration.

Reviewed By: wat3rBro

Differential Revision: D43027253

fbshipit-source-id: cbf1b4bcf5b0b8301b5b9547e3c22b1f0ffc7590
      5f1ef548
    • Fei Sun's avatar
      Use LERP to implement EMA · 255313d8
      Fei Sun authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/493

Currently the EMA implementation first does the multiplication and then does the addition. It requires two round trips from HBM. With the lerp operator, one kernel can do both. This change uses LERP to compute EMA instead. It reduces the GPU EMA computation time by 40%.

Reviewed By: newstzpz

Differential Revision: D43525938

fbshipit-source-id: ca1e14453bdfda958d3c412a52ff48efa65b3dd4
      255313d8
  41. 02 Mar, 2023 1 commit
    • Anthony Chen's avatar
      add correctness test for ai infra checkpointer with FSDP local mode in d2go runner · fd0cbb8f
      Anthony Chen authored 
      Summary:
X-link: https://github.com/facebookresearch/mobile-vision/pull/141

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/484

This diff adds a correctness test for ai infra checkpointer + FSDP local mode within a d2go runner. It verifies that ai infra checkpointer saves the exact same model as the old checkpointer, and that we can convert between ai infra checkpoints (local) and fsdp checkpoints (local + global) seamlessly.

Note: adapted from mattcyu1's script D43492498.

## Testing

Testing is done by saving with both ai infra and fsdp checkponter and compare the state dict produced. Here are the steps:

1. Build the model. Save a local ckpt using the FSDP checkpointer and another local ckpt using the AIInfra checkpointer
2. Reset the model. Load local ckpt using the FSDP checkpointer and convert it to global ckpt
3. Reset the model. Load local ckpt using the AIInfra checkpointer and re-save it as global ckpt using the FSDP checkpointer
4. Compare the two global state dicts

## Others

1. Add a launch decorator for d2go.distributed worker using the one in `fbcode/mobile-vision/mobile_cv/mobile_cv/torch/utils_pytorch/distributed_helper.py`

2. Remove `ema_state.load_state_dict()` in loading. This is needed because ai infra checkpointer loads state dict in place before `ema_state.load_state_dict()` is called. Since it's inplace loading, both ema_state and state_dict['ema_state'] points to the same tensor. Calling` ema.load_state_dict()` clears ema_state, effectively freeing the tensor and cause it to return an empty dict.
Solution: Don't call `ema_state.load_state_dict()` because it's already loaded. More info: https://www.internalfb.com/intern/wiki/Components_in_AI/Checkpoint/Getting_Started/Input_Output_Contract/#load

Reviewed By: xunnanxu

Differential Revision: D43423572

fbshipit-source-id: 8c4a47917670ea1205f952540d1e4cb9fc9232c0
      fd0cbb8f
  43. 01 Mar, 2023 2 commits
  45. 28 Feb, 2023 1 commit
  47. 25 Feb, 2023 1 commit
  49. 24 Feb, 2023 1 commit
    • Matthew Yu's avatar
      turn off interleaving if only saving on rank0 · 3111ae59
      Matthew Yu authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/482

We should avoid using interleaving during save if we are calling save on one process:
```
if comm.is_main_process():
  save()
```
this is because interleave calls comm.synchronize() so will just wait indefinitely.

This diff updates the FSDP checkpointer to use save(interleave=False) when running on one process.

Reviewed By: wat3rBro, YanjunChen329

Differential Revision: D43526328

fbshipit-source-id: 672993a87af627aca090384b0c218798bd42fcde
      3111ae59