Skip to content

GitLab

Switch branch/tag
  1. 27 May, 2023 1 commit
  3. 25 May, 2023 1 commit
    • Jiaxu Zhu's avatar
      Generic Reproducibility · edcdb731
      Jiaxu Zhu authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/548

As title, by setting
```
SOLVER.DETERMINISTIC = True
SEED = 42 # or other values
```
Training results are reproducible

Reviewed By: wat3rBro, rkaarimi

Differential Revision: D46174626

fbshipit-source-id: d6665b777376a176bd46a1286c3199ed0da26ae6
      edcdb731
  5. 24 May, 2023 1 commit
  7. 16 May, 2023 1 commit
  9. 10 May, 2023 1 commit
  11. 05 Apr, 2023 1 commit
    • Mik Vyatskov's avatar
      Setup root logger once & on import time · abdeafb0
      Mik Vyatskov authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/523

To avoid setting it up multiple times, add run_once() decorator.

Additionally make sure logging is configured for datalodaing workers, which have a different entry point, by moving setting up logging to the import time. Right now when a dataloader worker is created using spawn method from multiprocessing module, a new Python interpreter is created, with all the modules imported anew and with the entry point set to the method specified. This means that the entry point of the training framework is skipped, together with the logging setup.

With this change, the logging is configured on the import time, which means that when a dataloading process is created, even though the training main is not invoked, the logging is still configured because even though train_net is not invoked as an entry point, it's still imported in the child process.

Reviewed By: miqueljubert

Differential Revision: D44641142

fbshipit-source-id: 06ea85363d965b31d7f9ade3c2615ed9db67470b
      abdeafb0
  13. 31 Mar, 2023 2 commits
  15. 23 Mar, 2023 1 commit
    • Mik Vyatskov's avatar
      Redirect prints to logging module · d912e9f8
      Mik Vyatskov authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/509

print function is used all over the place and it's not realistic to enforce not using print for everyone. So this diff attempts to improve the debuggability of the code that was written using prints by redirecting prints to the logging module.

Additionally call logger setup from `setup_after_launch` to make sure logging settings are applied in every of the spawned processes.

Reviewed By: frabu6, wat3rBro

Differential Revision: D44280241

fbshipit-source-id: 713400ac2b2edacef3c7a99067cbb1e684c3c5ad
      d912e9f8
  17. 01 Feb, 2023 1 commit
    • Yanghan Wang's avatar
      Allow specifying extra lightning trainer params via `_DEFAULTS_` in yaml · 6940fa9c
      Yanghan Wang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/461

There're needs for extending trainer parameters that are not in (or conflict with) the base d2go config, this diff adds a way to inject those configs without touching the base d2go config.
- In `get_trainer_params`, it simply checks the `LIGHTNING_TRAINER` and use whatever configs under it.
- Adds `GeneralizedRCNNTaskNoDefaultConfig`, which allows specify default config via yaml file for `GeneralizedRCNNTask`. (also make some changes for prerequisite)
- (next diff) User can add their own config updater by registering it in `CONFIG_UPDATER_REGISTRY`.

Differential Revision: D42928992

fbshipit-source-id: f2a1d8a3f2bec9908bb1af03928611d963b92c0e
      6940fa9c
  19. 14 Nov, 2022 1 commit
  21. 23 Oct, 2022 1 commit
  23. 09 Aug, 2022 1 commit
  25. 24 Jun, 2022 1 commit
    • Yanghan Wang's avatar
      use runner class instead of instance outside of main · 8051775c
      Yanghan Wang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/312

As discussed, we decided to not use runner instance outside of `main`, previous diffs already solved the prerequisites, this diff mainly does the renaming.
- Use runner name (str) in the fblearner, ML pipeline.
- Use runner name (str) in FBL operator, MAST and binary operator.
- Use runner class as the interface of main, it can be either the name of class (str) or actual class. The main usage should be using `str`, so that the importing of class happens inside `main`. But it's also a common use case to import runner class and call `main` for things like ad-hoc scripts or tests, supporting actual class makes it easier modify code for those cases (eg. some local test class doesn't have a name, so it's not feasible to use runner name).

Reviewed By: newstzpz

Differential Revision: D37060338

fbshipit-source-id: 879852d41902b87d6db6cb9d7b3e8dc55dc4b976
      8051775c
  27. 22 Jun, 2022 1 commit
  29. 18 Jun, 2022 1 commit
  31. 16 Jun, 2022 2 commits
    • Yanghan Wang's avatar
      support for using config file with _DEFAULTS_ via cli · 521b3cad
      Yanghan Wang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/271

- set `get_default_cfg = None` to opt-in this new feature.
- support config with `_DEFAULTS_` when loading config file. Note that we don't check explicitly `"_DEFAULTS_" in config file` but check `runner.get_default_cfg == None`, this is because runner having `get_default_cfg` and config having `_DEFAULTS_` should be mutually exclusive, and `load_full_config_from_file` can raise proper error if `_DEFAULTS_` is missing.
- we also need to save `_DEFAULTS_` in the diff config.

Reviewed By: tglik

Differential Revision: D36868581

fbshipit-source-id: e0e19309c3df5a85383ce1454b321a68d0868dc4
      521b3cad
    • Mik Vyatskov's avatar
      Implement a central helper for converting arguments to CLI args · 3b64e76a
      Mik Vyatskov authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/301

This is a follow-up of earlier work to extract part responsible for the
centrally defined parameters from the helper in train_net closer to where the
parameters are defined.

Reviewed By: tglik

Differential Revision: D37176212

fbshipit-source-id: 226415f36f4872ac3d9ba41541b4389a18cc11e6
      3b64e76a
  33. 15 Jun, 2022 1 commit
  35. 14 Jun, 2022 1 commit
  37. 09 Jun, 2022 1 commit
  39. 15 May, 2022 1 commit
    • John Reese's avatar
      apply import merging for fbcode (7 of 11) · b3a9204c
      John Reese authored 
      Summary:
Applies new import merging and sorting from µsort v1.0.

When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.

Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.

For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting

Reviewed By: lisroach

Differential Revision: D36402205

fbshipit-source-id: a4efc688d02da80c6e96685aa8eb00411615a366
      b3a9204c
  41. 14 May, 2022 1 commit
  43. 26 Apr, 2022 1 commit
  45. 16 Mar, 2022 1 commit
    • Chengjiang Long's avatar
      Implement the user calibration model option 3 under D2GO · b30cf9d2
      Chengjiang Long authored 
      Summary:
Dataloader:
Rewrote the data loader via build_stream_dataset_reader with the DATASET_DEFINITION of "peopleai_face_eng_inference_results".

User Calibration Model (initial version):

nn.Sequential(
    nn.Conv1d(72, 128, 1),
    nn.BatchNorm1d(128),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(128, 72),
)

Differential Revision: D34202009

fbshipit-source-id: 55a2c579e463ed19eac38b5dd12e11c09cbccc11
      b30cf9d2
  47. 18 Jan, 2022 1 commit
    • Miquel Jubert Hermoso's avatar
      Fix type signature of create_runner · c74e23b0
      Miquel Jubert Hermoso authored 
      Summary: The type signature of create_runner is not accurate. We expect lightning runners to follow DefaultTask. Also change setup.py to not import directly, which was causing circular dependencies together with the change.

Reviewed By: wat3rBro

Differential Revision: D32792069

fbshipit-source-id: 0fbb55eb269dd681dbc8df49d71c9635f56293b8
      c74e23b0
  49. 02 Dec, 2021 1 commit
  51. 20 Oct, 2021 1 commit
    • Yuxin Wu's avatar
      print flow id in logs · 74a031b8
      Yuxin Wu authored 
      Summary: helps debugging

Reviewed By: zhanghang1989

Differential Revision: D31806396

fbshipit-source-id: 870308990c4c0c71453d107628b8adcb9edcf391
      74a031b8
  53. 09 Sep, 2021 1 commit
  55. 13 May, 2021 1 commit
    • Kai Zhang's avatar
      Auto scale config for multi-node training · e87ed5f0
      Kai Zhang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/62

Lightning trainer set max step to cfg.SOLVER.MAX_ITER. However, this is the max iteration for all nodes, in multi-node training, we need to scale it down, as well as eval period and other configs.
This diff calls `auto_scale_world_size` before passing the config to trainer.

Reviewed By: wat3rBro

Differential Revision: D28140877

fbshipit-source-id: 2639ae58773a4ec2a0cc59dfefb2f5d9b1afe1a8
      e87ed5f0
  57. 05 Mar, 2021 1 commit
    • Peizhao Zhang's avatar
      migrated to iopath for d2go - data. · e96e287b
      Peizhao Zhang authored 
      Summary: migrated to iopath for d2go - data.

Reviewed By: sstsai-adl

Differential Revision: D26632143

fbshipit-source-id: d85e4a00817115a8762e4444406d867070d4fd0c
      e96e287b
  59. 03 Mar, 2021 1 commit