Skip to content

GitLab

Switch branch/tag
  1. 15 Jun, 2022 2 commits
  3. 14 Jun, 2022 4 commits
    • Yanghan Wang's avatar
      make get_default_cfg a classmethod · 65dad512
      Yanghan Wang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/293

In order to pass runner during the workflow using "runner name" instead of runner instance, we need to make sure the `get_default_cfg` is not instance method. It can be either staticmethod or classmethod, but I choose classmethod for better inheritance.

code mode using following script:
```
#!/usr/bin/env python3

import json
import os
import subprocess

result = subprocess.check_output("fbgs --json 'def get_default_cfg('", shell=True)
fbgs = json.loads(result)
fbsource_root = os.path.expanduser("~")

def _indent(s):
    return len(s) - len(s.lstrip())

def resolve_instance_method(content):
    lines = content.split("\n")
    for idx, line in enumerate(lines):
        if "def get_default_cfg(self" in line:
            indent = _indent(line)
            # find the class
            for j in range(idx, 0, -1):
                if lines[j].startswith(" " * (indent - 4) + "class "):
                    class_line = lines[j]
                    break
            else:
                raise RuntimeError("Can't find class")
            print("class_line: ", class_line)
            if "Runner" in class_line:
                # check self if not used
                for j in range(idx + 1, len(lines)):
                    if _indent(lines[j]) < indent:
                        break
                    assert "self" not in lines[j], (j, lines[j])
                # update the content
                assert "def get_default_cfg(self)" in line
                lines[idx] = lines[idx].replace(
                    "def get_default_cfg(self)", "def get_default_cfg(cls)"
                )
                lines.insert(idx, " " * indent + "classmethod")
                return "\n".join(lines)
    return content

def resolve_static_method(content):
    lines = content.split("\n")
    for idx, line in enumerate(lines):
        if "def get_default_cfg()" in line:
            indent = _indent(line)
            # find the class
            for j in range(idx, 0, -1):
                if "class " in lines[j]:
                    class_line = lines[j]
                    break
            else:
                print("[WARNING] Can't find class!!!")
                continue
            if "Runner" in class_line:
                # check staticmethod is used
                for j in range(idx, 0, -1):
                    if lines[j] == " " * indent + "staticmethod":
                        staticmethod_line_idx = j
                        break
                else:
                    raise RuntimeError("Can't find staticmethod")
                # update the content
                lines[idx] = lines[idx].replace(
                    "def get_default_cfg()", "def get_default_cfg(cls)"
                )
                lines[staticmethod_line_idx] = " " * indent + "classmethod"
                return "\n".join(lines)
    return content

for result in fbgs["results"]:
    filename = os.path.join(fbsource_root, result["file_name"])
    print(f"processing: {filename}")
    with open(filename) as f:
        content = f.read()
    orig_content = content
    while True:
        old_content = content
        content = resolve_instance_method(content)
        content = resolve_static_method(content)
        if content == old_content:
            break
    if content != orig_content:
        print("Updating ...")
        with open(filename, "w") as f:
            f.write(content)
```

Reviewed By: tglik

Differential Revision: D37059264

fbshipit-source-id: b09d5518f4232de95d8313621468905cf10a731c
      65dad512
    • Yanghan Wang's avatar
      support diff config for lightning_train_net · 8cf2b879
      Yanghan Wang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/285

`setup_after_launch` can now take `DefaultTask` as well (the `runner_or_task` can still be `None`, for runner-less train_net).

Reviewed By: tglik

Differential Revision: D37011560

fbshipit-source-id: ce8a88242df0a16de8da97d94e8eb7def524c69c
      8cf2b879
    • Sam Tsai's avatar
      separating out save function and checking if it has been run previously for a dataset · 84dac84f
      Sam Tsai authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/282

Writing the of the converted coco file needs to be synced since the file is used by COCOEval. The whole function could be called multiple times by main thread so need to add additional checks to see if dataset has been previous called.

Reviewed By: wat3rBro

Differential Revision: D36957778

fbshipit-source-id: 5f9c5c038c13cbe24132ef1b44f6e94d5a26b66a
      84dac84f
    • Yanghan Wang's avatar
      fix DDP argument misalignment · 38958fcc
      Yanghan Wang authored 
      Summary:
X-link: https://github.com/facebookresearch/mobile-vision/pull/83

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/292

Reviewed By: tglik, newstzpz

Differential Revision: D37116726

fbshipit-source-id: 9956fe19916521b4fb512207512861afe4f09148
      38958fcc
  5. 13 Jun, 2022 2 commits
  7. 10 Jun, 2022 3 commits
  9. 09 Jun, 2022 1 commit
  11. 08 Jun, 2022 1 commit
    • Yanghan Wang's avatar
      making bootstrap more robust · 1f45cf04
      Yanghan Wang authored 
      Summary:
X-link: https://github.com/facebookresearch/mobile-vision/pull/81

Pull Request resolved: https://github.com/facebookresearch/d2go/pull/283

- add `MoreMagicMock`, which handles inheritance and comparison.
- also support lazy registering mocked objects (has to be `MoreMagicMock`).
- don't need to skip `skip files that doesn't contain ".register()" call` since we can handle most files pretty well now.
- also mock the open
- delay the import for `from detectron2.utils.testing import assert_instances_allclose`; for some reason python is doing magic things if you import anything starting with `assert`, so the mocked import doesn't work.
- makes log function nicer.

Reviewed By: tglik

Differential Revision: D36798327

fbshipit-source-id: ccda7e7583b95a24f3dde1bbe0468593dacb8663
      1f45cf04
  13. 07 Jun, 2022 1 commit
  15. 06 Jun, 2022 3 commits
  17. 05 Jun, 2022 2 commits
  19. 03 Jun, 2022 3 commits
  21. 02 Jun, 2022 2 commits
  23. 01 Jun, 2022 1 commit
  25. 31 May, 2022 1 commit
    • Yanghan Wang's avatar
      fix protobuf version issue · 3a89f35a
      Yanghan Wang authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/266

CI has issue (eg. https://github.com/facebookresearch/d2go/runs/6662620169?check_suite_focus=true):
```
  File "/usr/share/miniconda/envs/__setup_conda/lib/python3.8/site-packages/caffe2/proto/__init__.py", line 15, in <module>
    from caffe2.proto import caffe2_pb2, metanet_pb2, torch_pb2
  File "/usr/share/miniconda/envs/__setup_conda/lib/python3.8/site-packages/caffe2/proto/caffe2_pb2.py", line 33, in <module>
    _descriptor.EnumValueDescriptor(
  File "/usr/share/miniconda/envs/__setup_conda/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 755, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
```

It's caused by BC-breaking change of protobuf, fix the version.

Reviewed By: tglik

Differential Revision: D36781162

fbshipit-source-id: 8f13461a44a5a8f01abdc09a1ff8f9759fe55619
      3a89f35a
  27. 28 May, 2022 1 commit
  29. 27 May, 2022 1 commit
  31. 26 May, 2022 1 commit
  33. 25 May, 2022 2 commits
  35. 23 May, 2022 2 commits
    • Miquel Jubert Hermoso's avatar
      Change reference from modeling to directly qconfig · 5f71fde2
      Miquel Jubert Hermoso authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/239

*This diff is part of a stack which has the goal of "buckifying" D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go core and enabling autodeps and other tooling. The last diff in the stack introduces the TARGETS. The diffs earlier in the stack are resolving circular dependencies and other issues which prevent the buckification from occurring.*

Break cyclic dependency by referring directly to the source file, instead to a different file that imports it.

Reviewed By: tglik

Differential Revision: D36166602

fbshipit-source-id: 7deafc02a52ab978a21593184d1b3d3810dc9346
      5f71fde2
    • Miquel Jubert Hermoso's avatar
      Reformat d2go_dataset_mapper to break circular dependency · ca094a0a
      Miquel Jubert Hermoso authored 
      Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/225

*This diff is part of a stack which has the goal of "buckifying" D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)Go core and enabling autodeps and other tooling. The last diff in the stack introduces the TARGETS. The diffs earlier in the stack are resolving circular dependencies and other issues which prevent the buckification from occurring.*

The overriding pattern applied in d2go_dataset_mapper, with the d2go_dataset_mapper_impl and d2go_dataset_mapper_impl_fb makes it possible that internal users get the _fb behaviour and external users the regular one, with the same import. But, this makes it necessary to put both files in the same dependency.

This causes a circular dependency. In general, one reasonable assumption is that fb-only dependencies can import oss dependencies but not vice versa. In the current setup, grouping both _impl files in a buck target makes that buck target contain both fb and oss code, and depend on both. This is causing circular buck dependency issues.

To fix this, while keeping the transparent import behavior, the following changes are done:

1. The implementation file is moved to a directory under .fb. It will have it's own target.
2. The non-fb version is renamed to DualInputDatasetMapper, as per Yanghan's suggestion. This simplifies the change, since it seems the fb version's behavior is not used atm.
3. d2go_dataset_mapper is moved to have the implementation itself

Reviewed By: tglik

Differential Revision: D35930993

fbshipit-source-id: ac57337d221df24f53e360d5dcb38ffa4164fef5
      ca094a0a
  37. 21 May, 2022 1 commit
  39. 20 May, 2022 3 commits
  41. 19 May, 2022 3 commits