GitLab

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/644

This diff consolidates deterministic settings in D2Go. In the `default_runner.py` file, the `torch.set_float32_matmul_precision("highest")` function is added to set the precision for matrix multiplication to the highest possible value. In the `setup.py` file, the `torch.backends.cudnn.deterministic` setting is set to `True` and the `torch.backends.cudnn.allow_tf32` setting is set to `False` to avoid random pytorch and CUDA algorithms during the training. The `torch.backends.cuda.matmul.allow_tf32` setting is also set to `False` to avoid random matrix multiplication algorithms. Additionally, the `seed` function is used to set the seed for reproducibility.

Reviewed By: wat3rBro

Differential Revision: D51796739

fbshipit-source-id: 50e44ea50b0311b56a885db9f633491ac3002bd4

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/456

`3.20.1` version has security issue, as https://github.com/facebookresearch/d2go/security/dependabot/1.`3.20.x+` has previously known compatibility issue, thus pin the version to `3.20.2`.

Reviewed By: YanjunChen329

Differential Revision: D42353218

fbshipit-source-id: 172d3a2d53f0c46326e0ae730ad0a96b33b8bade

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/453

Previous diffs updated the LRScheduler to public version (eg. https://github.com/facebookresearch/detectron2/pull/4709), this also requires newer version of pytorch-lightning. This diff upgrades the lightning version to 1.8.6, also fixes some deprecated call sites of old lightning versions.
- `deepcopy` seems to be supported now, remove `_deepcopy` (there's now not allowed to access `trainer` attributed when it is `None`)
- `dataloader_idx` is removed from `on_train_batch_start`.
- stop using `_accelerator_connector` (the AcceleratorConnector doesn't have those attributes anymore).
- deprecated `on_pretrain_routine_end` -> `on_fit_start`

Reviewed By: YanjunChen329

Differential Revision: D42319019

fbshipit-source-id: ba46abbd98da96783e15d187a361fda47dc7d4d6

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/266

CI has issue (eg. https://github.com/facebookresearch/d2go/runs/6662620169?check_suite_focus=true):
```
  File "/usr/share/miniconda/envs/__setup_conda/lib/python3.8/site-packages/caffe2/proto/__init__.py", line 15, in <module>
    from caffe2.proto import caffe2_pb2, metanet_pb2, torch_pb2
  File "/usr/share/miniconda/envs/__setup_conda/lib/python3.8/site-packages/caffe2/proto/caffe2_pb2.py", line 33, in <module>
    _descriptor.EnumValueDescriptor(
  File "/usr/share/miniconda/envs/__setup_conda/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 755, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```

It's caused by BC-breaking change of protobuf, fix the version.

Reviewed By: tglik

Differential Revision: D36781162

fbshipit-source-id: 8f13461a44a5a8f01abdc09a1ff8f9759fe55619

Summary:
Applies new import merging and sorting from µsort v1.0.

When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.

Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.

For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting

Reviewed By: lisroach

Differential Revision: D36402205

fbshipit-source-id: a4efc688d02da80c6e96685aa8eb00411615a366

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/189

X-link: https://github.com/facebookresearch/recipes/pull/14

X-link: https://github.com/facebookresearch/ReAgent/pull/616

### New commit log messages
- [9b011606f Add callout items to the Docs landing page (#12196)](https://github.com/PyTorchLightning/pytorch-lightning/pull/12196)

Reviewed By: edward-io

Differential Revision: D34687261

fbshipit-source-id: 3ef6be5169a855582384f9097a962d2261625882

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/187

Reviewed By: ananthsub, zhanghang1989

Differential Revision: D34650467

fbshipit-source-id: b9518e5dd673b709320b87e57a43d053eca3aabe

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/185

The `DiskCachedDatasetFromList` was originally in the `d2go/data/utils.py`, so the class is declared by default. Therefore the clean up call (https://fburl.com/code/cu7hswhx) is always called even when the feature is not enabled. This diff move it to a new place and delay the import, so the clean up won't run.

Reviewed By: tglik

Differential Revision: D34601363

fbshipit-source-id: 734bb9b2c7957d7437ad40c4bfe60a441ec2f23a

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/183

Reviewed By: zhanghang1989

Differential Revision: D34492204

fbshipit-source-id: 7fd459172e83a5015ca9eee0e2018ce8b22c3096

Summary: Pull Request resolved: https://github.com/facebookresearch/d2go/pull/152

Reviewed By: zhanghang1989

Differential Revision: D31591900

fbshipit-source-id: 6ee8124419d535caf03532eda4f729e707b6dda7

Summary:
https://fb.workplace.com/groups/pythonfoundation/posts/2990917737888352

Remove `mobile-vision` from opt-out list; leaving `mobile-vision/SNPE` opted out because of 3rd-party code.

arc lint --take BLACK --apply-patches --paths-cmd 'hg files mobile-vision'

allow-large-files

Reviewed By: sstsai-adl

Differential Revision: D30721093

fbshipit-source-id: 9e5c16d988b315b93a28038443ecfb92efd18ef8

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/75

Refactor the base test case
- make test_dir valid throughout the test (rather than under local context), so individual test can load back the export model
- refactor the `custom_setup_test` for easier override.
- move parameterized into base class to avoid copying naming function

Reviewed By: zhanghang1989

Differential Revision: D28651067

fbshipit-source-id: c59a311564f6114039e20ed3a23e5dd9c84f4ae4

Summary:
we need to load the [config file](https://github.com/facebookresearch/d2go/blob/master/tests/misc/test_configs.py#L35) during the unittest.

The `get_resource_path` need to use `d2go.tests`
https://fburl.com/whn41ma0

The test fail at
https://github.com/facebookresearch/d2go/runs/2272258836

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/35

Reviewed By: wat3rBro

Differential Revision: D27578008

Pulled By: zhanghang1989

fbshipit-source-id: 5fa24c7f74bc3e59ffee98d57f02a2a558c2a4b0

Summary: all utils code are moved to d2go.utils.testing

Reviewed By: newstzpz

Differential Revision: D27209943

fbshipit-source-id: 6c5cb14858155a8ed13478d65ee8e02ef74616d7

Summary: Not d2go.tests is not a library for oss, move utils code to d2go.utils.testing

Reviewed By: zhanghang1989

Differential Revision: D26706933

fbshipit-source-id: 85767b66bbb6c67db05e11823beb4840220b2aa3

fbshipit-source-id: f4a8ba78691d8cf46e003ef0bd2e95f170932778