synchronize before dumping model configs

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/541

The issue post https://fb.workplace.com/groups/277527419809135/permalink/1303604910534709/

The fix was suggested by the MV folks.

Reviewed By: dilinwang820, wat3rBro

Differential Revision: D45881863

fbshipit-source-id: b33345c4230067b78f27e7deb038c095d55f1360

d2go/utils/misc.py

@@ -92,9 +92,11 @@ def dump_trained_model_configs(
         config_file = os.path.join(trained_model_config_dir, "{}.yaml".format(name))
         trained_model_configs[name] = config_file
         if comm.is_main_process():
-            logger.info("Dump trained config file: {}".format(config_file))
+            logger.info("Dumping trained config file: {}".format(config_file))
             with PathManager.open(config_file, "w") as f:
                 f.write(trained_cfg.dump())
+        comm.synchronize()
+        logger.info("Finished dumping trained config file")
     return trained_model_configs