Fix coco conversion race condition with file lock

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/260

Training + Evaluation for OCR Detection based on D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)GO

Purpose: To restart training of OCR detection after abandonment. Updates filepaths and some configs, as well as addresses a race condition situation when creating temp json files for annotations read/write within D2 (https://github.com/facebookresearch/d2go/commit/87374efb134e539090e0b5c476809dc35bf6aedb)GO.

The training can be restarted from scratch, or specified from a previous model by setting the weights_pth variable. Test plan is purely on 1 particular dataset.

Reviewed By: sstsai-adl

Differential Revision: D36632783

fbshipit-source-id: fe8677a57d660c495458d083c3fb70b95128b260

d2go/data/extended_coco.py

@@ -124,8 +124,9 @@ def convert_coco_text_to_coco_detection_json(
                 x["image_id"] = image_id_remap[x["image_id"]]
 
     os.makedirs(os.path.dirname(target_json), exist_ok=True)
-    with open(target_json, "w") as f:
-        json.dump(coco_text_json, f)
+    if comm.get_local_rank() == 0:
+        with open(target_json, "w") as f:
+            json.dump(coco_text_json, f)
 
     return coco_text_json
 

d2go/runner/default_runner.py

@@ -325,6 +325,9 @@ class Detectron2GoRunner(BaseRunner):
 
             # NOTE: creating evaluator after dataset is loaded as there might be dependency.  # noqa
             data_loader = self.build_detection_test_loader(cfg, dataset_name)
+            # Synchronize to ensure data processing during data loader build is complete
+            comm.synchronize()
+
             evaluator = self.get_evaluator(
                 cfg, dataset_name, output_folder=output_folder
             )