Only save results to file from rank 0

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/309

Right now multiple machines can try to write to the same output file,
since they get the same argument. Additionally, on the same machine, several
outputs can be saved which requires unncessary unpacking. This change makes
train_net only write output of the rank 0 trainer.

Reviewed By: wat3rBro

Differential Revision: D37310084

fbshipit-source-id: 9d5352a274e8fb1d2043393b12896d402333c17b

tools/train_net.py

@@ -97,10 +97,9 @@ def run_with_cmdline_args(args):
         args=(cfg, output_dir, runner_name, args.eval_only, args.resume),
     )
 
-    if args.save_return_file is not None:
-        save_binary_outputs(args.save_return_file, outputs)
-
-    return outputs
+    # Only save results from global rank 0 for consistency.
+    if args.save_return_file is not None and args.machine_rank == 0:
+        save_binary_outputs(args.save_return_file, outputs[0])
 
 
 def cli(args=None):