Commit 8eb45690 authored by Yanghan Wang's avatar Yanghan Wang Committed by Facebook GitHub Bot
Browse files

fix ddp init twice in oss test

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/192

Nowadays lightning will initialize process group when using ddp strategy, since `TestLightningTrainNet` does a training with ddp strategy (https://fburl.com/code/a9yp0kzy), the process group ended up initialized after running the test. However there're other tests that will also set up ddp and thus expect non-initialized process group, this is not a problem on sandcastle since the tests run separately, however in OSS env, the tests are running together, so the error happens (eg. https://github.com/facebookresearch/d2go/runs/5668912203?check_suite_focus=true).

This diff adds a clean up step in `TestLightningTrainNet`.

Reviewed By: tglik

Differential Revision: D35099944

fbshipit-source-id: f5b42b2a87d4efd9aa0ed97e6bd2140d80ab9522
parent a0ee06f3
...@@ -29,4 +29,4 @@ jobs: ...@@ -29,4 +29,4 @@ jobs:
env: env:
OSSRUN: 1 OSSRUN: 1
run: | run: |
python -m unittest discover tests python -m unittest discover -v -s tests
...@@ -5,6 +5,7 @@ import os ...@@ -5,6 +5,7 @@ import os
import unittest import unittest
import numpy as np import numpy as np
import torch.distributed as dist
from d2go.config import CfgNode from d2go.config import CfgNode
from d2go.config.utils import flatten_config_dict from d2go.config.utils import flatten_config_dict
from d2go.runner.lightning_task import GeneralizedRCNNTask from d2go.runner.lightning_task import GeneralizedRCNNTask
...@@ -56,3 +57,7 @@ class TestLightningTrainNet(unittest.TestCase): ...@@ -56,3 +57,7 @@ class TestLightningTrainNet(unittest.TestCase):
accuracy2 = flatten_config_dict(out2.accuracy) accuracy2 = flatten_config_dict(out2.accuracy)
for k in accuracy: for k in accuracy:
np.testing.assert_equal(accuracy[k], accuracy2[k]) np.testing.assert_equal(accuracy[k], accuracy2[k])
def tearDown(self):
if dist.is_initialized():
dist.destroy_process_group()
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment