Skip to content

GitLab

Switch branch/tag
  1. 22 Jul, 2022 1 commit
  3. 30 Jun, 2022 2 commits
  5. 29 Jun, 2022 1 commit
  7. 24 Jun, 2022 2 commits
    • Mik Vyatskov's avatar
      Only save results to file from rank 0 · f0297b81
      Mik Vyatskov authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/309

Right now multiple machines can try to write to the same output file,
since they get the same argument. Additionally, on the same machine, several
outputs can be saved which requires unncessary unpacking. This change makes
train_net only write output of the rank 0 trainer.

Reviewed By: wat3rBro

Differential Revision: D37310084

fbshipit-source-id: 9d5352a274e8fb1d2043393b12896d402333c17b
      f0297b81
    • Yanghan Wang's avatar
      use runner class instead of instance outside of main · 8051775c
      Yanghan Wang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/312

As discussed, we decided to not use runner instance outside of `main`, previous diffs already solved the prerequisites, this diff mainly does the renaming.
- Use runner name (str) in the fblearner, ML pipeline.
- Use runner name (str) in FBL operator, MAST and binary operator.
- Use runner class as the interface of main, it can be either the name of class (str) or actual class. The main usage should be using `str`, so that the importing of class happens inside `main`. But it's also a common use case to import runner class and call `main` for things like ad-hoc scripts or tests, supporting actual class makes it easier modify code for those cases (eg. some local test class doesn't have a name, so it's not feasible to use runner name).

Reviewed By: newstzpz

Differential Revision: D37060338

fbshipit-source-id: 879852d41902b87d6db6cb9d7b3e8dc55dc4b976
      8051775c
  9. 18 Jun, 2022 2 commits
  11. 16 Jun, 2022 2 commits
  13. 15 Jun, 2022 1 commit
  15. 14 Jun, 2022 1 commit
  17. 09 Jun, 2022 1 commit
  19. 02 Jun, 2022 1 commit
    • Miquel Jubert Hermoso's avatar
      Separate into API and Exporter · 24da990f
      Miquel Jubert Hermoso authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/238

*This diff is part of a stack which has the goal of "buckifying" D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go core and enabling autodeps and other tooling. The last diff in the stack introduces the TARGETS. The diffs earlier in the stack are resolving circular dependencies and other issues which prevent the buckification from occurring.*

Following the comments in an abandoned diff, split the export code into two files, which will have their corresponding dependencies: exporter and api. api.py contains the components which have little dependencies, so it can be imported basically anywhere without circular dependencies.

exporter.py contains the utilities, which are use for export operations, for example in the exporter binary.

Reviewed By: mcimpoi

Differential Revision: D36166603

fbshipit-source-id: 25ded0b3925464c05be4048472a4c2ddcdb17ecf
      24da990f
  21. 15 May, 2022 1 commit
    • John Reese's avatar
      apply import merging for fbcode (7 of 11) · b3a9204c
      John Reese authored 
      Summary:
Applies new import merging and sorting from µsort v1.0.

When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.

Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.

For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting

Reviewed By: lisroach

Differential Revision: D36402205

fbshipit-source-id: a4efc688d02da80c6e96685aa8eb00411615a366
      b3a9204c
  23. 14 May, 2022 1 commit
  25. 24 Mar, 2022 1 commit
    • Tsahi Glik's avatar
      refactor exporter and eval command line tools · 744d72d7
      Tsahi Glik authored 
      Summary: Tweak exporter and evaluator cli entry point func to support calling it as a module with args from custom launching code.

Reviewed By: sstsai-adl

Differential Revision: D35035813

fbshipit-source-id: c8b24099e94ccc58c184f8aac95b2a24a137e86a
      744d72d7
  27. 10 Mar, 2022 1 commit
  29. 05 Mar, 2022 1 commit
  31. 03 Mar, 2022 1 commit
  33. 14 Feb, 2022 1 commit
    • Tugrul Savran's avatar
      D2Go Fail Fast: Move exception coming from not implemented "compare accuracy" feature to the top. · eee4dfc1
      Tugrul Savran authored 
      Summary:
Currently, the exporter method takes in a compare_accuracy parameter, which after all the compute (exporting etc.) raises an exception if it is set to True.

This looks like an antipattern, and causes a waste of compute.

Therefore, I am proposing to raise the exception at the very beginning of method call to let the client know in advance that this argument's functionality isn't implemented yet.

NOTE: We might also choose to get rid of the entire parameter. I am open for suggestions.

Differential Revision: D34186578

fbshipit-source-id: d7fbe7589dfe2d2f688b870885ca61e6829c9329
      eee4dfc1
  35. 08 Jan, 2022 1 commit
    • Binh Tang's avatar
      Add deprecation path for renamed training type plugins (#11227) · fcd51171
      Binh Tang authored 
      Summary:
### New commit log messages
  4eede7c30 Add deprecation path for renamed training type plugins (#11227)

Reviewed By: edward-io, daniellepintz

Differential Revision: D33409991

fbshipit-source-id: 373e48767e992d67db3c85e436648481ad16c9d0
      fcd51171
  37. 06 Jan, 2022 1 commit
    • Binh Tang's avatar
      Rename `DDPPlugin` to `DDPStrategy` (#11142) · aeb15613
      Binh Tang authored 
      Summary:
### New commit log messages
  b64dea9dc Rename `DDPPlugin` to `DDPStrategy` (#11142)

Reviewed By: jjenniferdai

Differential Revision: D33259306

fbshipit-source-id: b4608c6b96b4a7977eaa4ed3f03c4b824882aef0
      aeb15613
  39. 29 Dec, 2021 1 commit
  41. 25 Nov, 2021 1 commit
  43. 24 Sep, 2021 1 commit
  45. 18 Sep, 2021 1 commit
  47. 09 Sep, 2021 1 commit
  49. 09 Jul, 2021 1 commit
    • Mircea Cimpoi's avatar
      Add BoltNN conversion to d2go exporter · ecf832da
      Mircea Cimpoi authored 
      Summary:
Added predictor_type `boltnn_int8` to export to BoltNN via torch delegate.

- `int8` needs to be in the name, otherwise the post-train quantization won't happen;

```
cfg.QUANTIZATION.BACKEND = "qnnpack"
// cfg.QUANTIZATION.CUSTOM_QSCHEME = "per_tensor_affine"
```

Seems that ` QUANTIZATION.CUSTOM_QSCHEME per_tensor_affine` is not needed - likely covered by "qnnpack".

Reviewed By: wat3rBro

Differential Revision: D29106043

fbshipit-source-id: 865ac5af86919fe7b4530b48433a1bd11e295bf4
      ecf832da
  51. 07 Jul, 2021 1 commit
  53. 09 Jun, 2021 1 commit
    • Sam Tsai's avatar
      allow for multiple datasets for test data loader creation · fc690b45
      Sam Tsai authored 
      Summary: Use all training dataset for export instead of just first. This is to support use cases where there is only a small amount of images per jsons but a number of jsons. Since calibration uses the first dataset, it is limited by the number of images in a single dataset.

Reviewed By: ppwwyyxx

Differential Revision: D28902673

fbshipit-source-id: f80146b02d2d1bc04703fbb21ef410f5e26ba64c
      fc690b45
  55. 07 Jun, 2021 1 commit
  57. 25 May, 2021 2 commits
  59. 22 May, 2021 2 commits
  61. 21 May, 2021 1 commit
    • Sanjeev Kumar's avatar
      Enable inference config in export step · 90aff5da
      Sanjeev Kumar authored 
      Summary:
- Enable sdk inference config specification in export step. This enables adding the sdk configuration as part of model file in the export step. The sdk config can be specified as infernece_config.yaml and is zipped together with torchscript model. The main goal of sdk configuration is to control the model inference behavior with model.
- SDK inference config design doc: https://docs.google.com/document/d/1j5qx8IrnFg1DJFzTnu4W8WmXFYJ-AgCDfSQHb2ACJsk/edit
- One click fblearner pipeline is in next diff on the stack

Differential Revision: D27881742

fbshipit-source-id: 34a3ab7a88f456b74841cf671ea1b3f678cdb733
      90aff5da
  63. 13 May, 2021 1 commit
    • Kai Zhang's avatar
      Auto scale config for multi-node training · e87ed5f0
      Kai Zhang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/62

Lightning trainer set max step to cfg.SOLVER.MAX_ITER. However, this is the max iteration for all nodes, in multi-node training, we need to scale it down, as well as eval period and other configs.
This diff calls `auto_scale_world_size` before passing the config to trainer.

Reviewed By: wat3rBro

Differential Revision: D28140877

fbshipit-source-id: 2639ae58773a4ec2a0cc59dfefb2f5d9b1afe1a8
      e87ed5f0
  65. 17 Apr, 2021 2 commits
    • Kai Zhang's avatar
      Delegate to model's customization · aeb24a92
      Kai Zhang authored 
      Summary: Delegate FX quantization callback's customization to model.

Reviewed By: wat3rBro

Differential Revision: D27669212

fbshipit-source-id: 2715546cf03134896da6f95ecddaf8503ff95d0b
      aeb24a92
    • Kai Zhang's avatar
      E2E QAT Workflow on Lightning · 845d0b2c
      Kai Zhang authored 
      Summary:
As per title and sanity test E2E QAT workflow on Lightning Trainer.

- add `post_training_opts`. This is required to use `all_steps_qat.json` with Lightning. We don't actually support the post_training_opts in this diff though - we leave it part of T83437359.
- Update .yaml to specify the Quantize-able modules.
- Update `lightning_train_net.py` to use the QuantizationAwareTraining callback.

Reviewed By: kandluis

Differential Revision: D26304879

fbshipit-source-id: 948bef4817d385d8a0969e4990d7f17ecd6994b7
      845d0b2c