[1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=606889206 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m__________________ ERROR collecting lightning/test_simple.py ___________________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/lightning/test_simple.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) lightning/test_simple.py:7: in from pytorch_lightning import LightningModule, Trainer E ModuleNotFoundError: No module named 'pytorch_lightning'[0m [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m lightning/test_simple.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.23s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=633467180 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m____________ ERROR collecting model/BingBertSquad/test_e2e_squad.py ____________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/model/BingBertSquad/test_e2e_squad.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) model/BingBertSquad/test_e2e_squad.py:14: in import evaluate as eval E ModuleNotFoundError: No module named 'evaluate'[0m [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m model/BingBertSquad/test_e2e_squad.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.23s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1594543543 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items [33m============================ [33mno tests ran[0m[33m in 0.90s[0m[33m =============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1727951244 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m_________________ ERROR collecting onebit/test_nccl_backend.py _________________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/onebit/test_nccl_backend.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) onebit/test_nccl_backend.py:13: in from deepspeed.runtime.comm.nccl import NcclBackend /usr/local/lib/python3.7/site-packages/deepspeed/runtime/comm/nccl.py:8: in import cupy E ModuleNotFoundError: No module named 'cupy'[0m [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m onebit/test_nccl_backend.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.24s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2547395270 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m_________________ ERROR collecting onebit/test_mpi_backend.py __________________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/onebit/test_mpi_backend.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) onebit/test_mpi_backend.py:12: in from deepspeed.runtime.comm.mpi import MpiBackend /usr/local/lib/python3.7/site-packages/deepspeed/runtime/comm/mpi.py:7: in import cupy E ModuleNotFoundError: No module named 'cupy'[0m ------------------------------- Captured stderr -------------------------------- -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: 26388537c721 Local device: mlx5_0 Local port: 1 CPCs attempted: rdmacm, udcm -------------------------------------------------------------------------- [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m onebit/test_mpi_backend.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.44s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2302194531 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m___________________ ERROR collecting onebit/test_mpi_perf.py ___________________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/onebit/test_mpi_perf.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) onebit/test_mpi_perf.py:10: in from deepspeed.runtime.comm.mpi import MpiBackend /usr/local/lib/python3.7/site-packages/deepspeed/runtime/comm/mpi.py:7: in import cupy E ModuleNotFoundError: No module named 'cupy'[0m ------------------------------- Captured stderr -------------------------------- -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: 26388537c721 Local device: mlx5_0 Local port: 1 CPCs attempted: rdmacm, udcm -------------------------------------------------------------------------- [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m onebit/test_mpi_perf.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.44s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=238941418 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m__________________ ERROR collecting onebit/test_nccl_perf.py ___________________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/onebit/test_nccl_perf.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) onebit/test_nccl_perf.py:13: in from deepspeed.runtime.comm.nccl import NcclBackend /usr/local/lib/python3.7/site-packages/deepspeed/runtime/comm/nccl.py:8: in import cupy E ModuleNotFoundError: No module named 'cupy'[0m [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m onebit/test_nccl_perf.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.22s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=319756802 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m__________ ERROR collecting small_model_debugging/test_mics_config.py __________[0m [1m[31msmall_model_debugging/test_mics_config.py[0m:69: in rank = [96mint[39;49;00m(os.environ[[33m'[39;49;00m[33mRANK[39;49;00m[33m'[39;49;00m])[90m[39;49;00m [1m[31m/usr/local/lib/python3.7/os.py[0m:681: in __getitem__ [94mraise[39;49;00m [96mKeyError[39;49;00m(key) [94mfrom[39;49;00m [94mNone[39;49;00m[90m[39;49;00m [1m[31mE KeyError: 'RANK'[0m [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m small_model_debugging/test_mics_config.py - KeyError: 'RANK' !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.30s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2887553514 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m_____________ ERROR collecting small_model_debugging/test_model.py _____________[0m [1m[31msmall_model_debugging/test_model.py[0m:66: in rank = [96mint[39;49;00m(os.environ[[33m'[39;49;00m[33mRANK[39;49;00m[33m'[39;49;00m])[90m[39;49;00m [1m[31m/usr/local/lib/python3.7/os.py[0m:681: in __getitem__ [94mraise[39;49;00m [96mKeyError[39;49;00m(key) [94mfrom[39;49;00m [94mNone[39;49;00m[90m[39;49;00m [1m[31mE KeyError: 'RANK'[0m [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m small_model_debugging/test_model.py - KeyError: 'RANK' !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.31s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=4135786312 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 10 items unit/autotuning/test_autotuning.py::test_resource_manager_arg_mappings[arg_mappings2] [32mPASSED[0m[32m [ 10%][0m unit/autotuning/test_autotuning.py::test_resource_manager_arg_mappings[arg_mappings3] [32mPASSED[0m[33m [ 20%][0m unit/autotuning/test_autotuning.py::test_autotuner_resources[active_resources1] [32mPASSED[0m[33m [ 30%][0m unit/autotuning/test_autotuning.py::test_command_line [32mPASSED[0m[33m [ 40%][0m unit/autotuning/test_autotuning.py::test_autotuner_resources[active_resources3] [32mPASSED[0m[33m [ 50%][0m unit/autotuning/test_autotuning.py::test_resource_manager_arg_mappings[arg_mappings4] [32mPASSED[0m[33m [ 60%][0m unit/autotuning/test_autotuning.py::test_resource_manager_arg_mappings[arg_mappings1] [32mPASSED[0m[33m [ 70%][0m unit/autotuning/test_autotuning.py::test_autotuner_resources[active_resources2] [32mPASSED[0m[33m [ 80%][0m unit/autotuning/test_autotuning.py::test_autotuner_resources[active_resources0] [32mPASSED[0m[33m [ 90%][0m unit/autotuning/test_autotuning.py::test_resource_manager_arg_mappings[None] [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m unit/autotuning/test_autotuning.py::test_resource_manager_arg_mappings[arg_mappings2] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/autotuning/test_autotuning.py::test_resource_manager_arg_mappings[arg_mappings2] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== (30 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m10 passed[0m, [33m[1m2 warnings[0m[33m in 0.97s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=169367059 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 22 items unit/comm/test_dist.py::TestGroupedDistTest::test_two[1138] [32mPASSED[0m[33m [ 4%][0m unit/comm/test_dist.py::TestGroupedDistTest::test_one[1138] [32mPASSED[0m[33m [ 9%][0m unit/comm/test_dist.py::TestDistributedFixture::test[2-16] [32mPASSED[0m[33m [ 13%][0m unit/comm/test_dist.py::TestDistributedFixture::test[4-32] [32mPASSED[0m[33m [ 18%][0m unit/comm/test_dist.py::TestDistributedFixture::test[4-16] [32mPASSED[0m[33m [ 22%][0m unit/comm/test_dist.py::TestDistributedFixture::test[2-32] [32mPASSED[0m[33m [ 27%][0m unit/comm/test_dist.py::TestInit::test [32mPASSED[0m[33m [ 31%][0m unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[True] [32mPASSED[0m[33m [ 36%][0m unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[False] [32mPASSED[0m[33m [ 40%][0m unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[True] [32mPASSED[0m[33m [ 45%][0m unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[False] [32mPASSED[0m[33m [ 50%][0m unit/comm/test_dist.py::TestDistInit::test_already_init[True] [32mPASSED[0m[33m [ 54%][0m unit/comm/test_dist.py::TestDistInit::test_already_init[False] [32mPASSED[0m[33m [ 59%][0m unit/comm/test_dist.py::TestDistInit::test_already_init[None] [32mPASSED[0m[33m [ 63%][0m unit/comm/test_dist.py::TestDistInit::test_no_init[True] [32mPASSED[0m[33m [ 68%][0m unit/comm/test_dist.py::TestDistInit::test_no_init[False] [32mPASSED[0m[33m [ 72%][0m unit/comm/test_dist.py::TestDistInit::test_no_init[None] [32mPASSED[0m[33m [ 77%][0m unit/comm/test_dist.py::TestDistAllReduce::test [32mPASSED[0m[33m [ 81%][0m unit/comm/test_dist.py::TestDistInitNoEnv::test [32mPASSED[0m[33m [ 86%][0m unit/comm/test_dist.py::TestDistArgs::test[hello-icosahedron-1138-purple] [32mPASSED[0m[33m [ 90%][0m unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_2 [32mPASSED[0m[33m [ 95%][0m unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_1 [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 unit/comm/test_dist.py::TestDistributedFixture::test[2-16] unit/comm/test_dist.py::TestDistributedFixture::test[2-16] unit/comm/test_dist.py::TestDistributedFixture::test[4-32] unit/comm/test_dist.py::TestDistributedFixture::test[4-32] unit/comm/test_dist.py::TestDistributedFixture::test[4-16] unit/comm/test_dist.py::TestDistributedFixture::test[4-16] unit/comm/test_dist.py::TestDistributedFixture::test[2-32] unit/comm/test_dist.py::TestDistributedFixture::test[2-32] /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/comm/test_dist.py:70 /home/aishsh/ds-v0.9.2/tests/unit/comm/test_dist.py:70: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(1) unit/comm/test_dist.py::TestGroupedDistTest::test_two[1138] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/comm/test_dist.py::TestGroupedDistTest::test_two[1138] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 13.64s call unit/comm/test_dist.py::TestDistAllReduce::test 6.02s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[False] 5.22s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_2 5.22s call unit/comm/test_dist.py::TestDistInit::test_already_init[True] 5.21s call unit/comm/test_dist.py::TestGroupedDistTest::test_one[1138] 5.12s call unit/comm/test_dist.py::TestDistInit::test_already_init[None] 5.11s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[True] 5.11s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-32] 5.11s call unit/comm/test_dist.py::TestDistInitWithModel::test_already_init[True] 4.77s call unit/comm/test_dist.py::TestGroupedDistTest::test_two[1138] 4.41s call unit/comm/test_dist.py::TestInit::test 4.32s call unit/comm/test_dist.py::TestDistInit::test_already_init[False] 4.31s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-32] 4.31s call unit/comm/test_dist.py::TestDistInit::test_no_init[True] 4.22s call unit/comm/test_dist.py::TestDistArgs::test[hello-icosahedron-1138-purple] 4.22s call unit/comm/test_dist.py::TestDistInit::test_no_init[None] 4.21s setup unit/comm/test_dist.py::TestDistributedFixture::test[2-16] 4.21s setup unit/comm/test_dist.py::TestDistributedFixture::test[4-16] 4.01s call unit/comm/test_dist.py::TestDistInitNoEnv::test 4.01s call unit/comm/test_dist.py::TestDistInitWithModel::test_no_init[False] 4.01s call unit/comm/test_dist.py::TestDistributedFixture::test[2-32] 4.01s call unit/comm/test_dist.py::TestDistributedFixture::test[4-16] 4.01s call unit/comm/test_dist.py::TestDistributedFixture::test[2-16] 4.01s call unit/comm/test_dist.py::TestDistributedFixture::test[4-32] 3.91s call unit/comm/test_dist.py::TestDistInit::test_no_init[False] 3.91s call unit/comm/test_dist.py::TestWorldSizeOverrideDistTest::test_world_size_1 (40 durations < 1s hidden. Use -vv to show these durations.) [33m================= [32m22 passed[0m, [33m[1m12 warnings[0m[33m in 127.68s (0:02:07)[0m[33m ==================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=3236324661 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 3 items unit/compression/test_compression.py::TestCompression::test_linear_layer_compress [32mPASSED[0m[33m [ 33%][0m unit/compression/test_compression.py::TestCompression::test_conv1d_convertion [32mPASSED[0m[33m [ 66%][0m unit/compression/test_compression.py::TestCompression::test_mpu_compress [33mSKIPPED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/compression/test_compression.py::TestCompression::test_linear_layer_compress /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/compression/test_compression.py::TestCompression::test_linear_layer_compress /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 4.86s call unit/compression/test_compression.py::TestCompression::test_linear_layer_compress 4.31s call unit/compression/test_compression.py::TestCompression::test_conv1d_convertion (6 durations < 1s hidden. Use -vv to show these durations.) [33m================== [32m2 passed[0m, [33m[1m1 skipped[0m, [33m[1m3 warnings[0m[33m in 10.10s[0m[33m ===================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=4149175560 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 12 items unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-1-4] [32mPASSED[0m[33m [ 8%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-2-2] [32mPASSED[0m[33m [ 16%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-False-2-2] [32mPASSED[0m[33m [ 25%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-False-2-2] [32mPASSED[0m[33m [ 33%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-False-1-4] [32mPASSED[0m[33m [ 41%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-True-1-4] [32mPASSED[0m[33m [ 50%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-1-2] [32mPASSED[0m[33m [ 58%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-False-1-2] [32mPASSED[0m[33m [ 66%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-False-1-2] [32mPASSED[0m[33m [ 75%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-False-1-4] [32mPASSED[0m[33m [ 83%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-True-2-2] [32mPASSED[0m[33m [ 91%][0m unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-True-1-2] [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-1-4] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-1-4] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 5.82s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-False-2-2] 5.82s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-False-1-2] 5.72s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-True-1-4] 5.72s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-False-1-4] 5.72s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-True-1-2] 5.62s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-2-2] 5.52s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-False-1-4] 5.52s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-True-2-2] 5.38s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-1-4] 4.82s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[True-False-1-2] 4.72s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-False-2-2] 4.72s call unit/moe/test_moe_tp.py::TestMOETensorParallel::test[False-True-1-2] (24 durations < 1s hidden. Use -vv to show these durations.) [33m================== [32m12 passed[0m, [33m[1m3 warnings[0m[33m in 66.11s (0:01:06)[0m[33m ===================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=379413680 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 6 items unit/moe/test_moe.py::TestPRMoE::test[2-False] [32mPASSED[0m[33m [ 16%][0m unit/moe/test_moe.py::TestPRMoE::test[2-True] [32mPASSED[0m[33m [ 33%][0m unit/moe/test_moe.py::TestMoE::test[True-4] [32mPASSED[0m[33m [ 50%][0m unit/moe/test_moe.py::TestMoE::test[True-2] [32mPASSED[0m[33m [ 66%][0m unit/moe/test_moe.py::TestMoE::test[False-4] [32mPASSED[0m[33m [ 83%][0m unit/moe/test_moe.py::TestMoE::test[False-2] [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/moe/test_moe.py::TestPRMoE::test[2-False] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/moe/test_moe.py::TestPRMoE::test[2-False] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 10.53s call unit/moe/test_moe.py::TestPRMoE::test[2-True] 10.41s call unit/moe/test_moe.py::TestPRMoE::test[2-False] 10.33s call unit/moe/test_moe.py::TestMoE::test[True-4] 9.93s call unit/moe/test_moe.py::TestMoE::test[False-4] 9.63s call unit/moe/test_moe.py::TestMoE::test[False-2] 9.13s call unit/moe/test_moe.py::TestMoE::test[True-2] (12 durations < 1s hidden. Use -vv to show these durations.) [33m=================== [32m6 passed[0m, [33m[1m3 warnings[0m[33m in 60.91s (0:01:00)[0m[33m ===================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2867340693 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 6 items unit/monitor/test_monitor.py::TestWandB::test_empty_wandb [31mFAILED[0m[31m [ 16%][0m unit/monitor/test_monitor.py::TestWandB::test_wandb [31mFAILED[0m[31m [ 33%][0m unit/monitor/test_monitor.py::TestCSVMonitor::test_csv_monitor [32mPASSED[0m[31m [ 50%][0m unit/monitor/test_monitor.py::TestCSVMonitor::test_empty_csv_monitor [32mPASSED[0m[31m [ 66%][0m unit/monitor/test_monitor.py::TestTensorBoard::test_empty_tensorboard [32mPASSED[0m[31m [ 83%][0m unit/monitor/test_monitor.py::TestTensorBoard::test_tensorboard [32mPASSED[0m[31m [100%][0m =================================== FAILURES =================================== [31m[1m__________________________ TestWandB.test_empty_wandb __________________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:03:22,266] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl If you want to use wandb logging, please `pip install wandb` and follow the instructions at https://docs.wandb.ai/quickstart If you want to use wandb logging, please `pip install wandb` and follow the instructions at https://docs.wandb.ai/quickstart ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:03:23.230263 463215 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:03:23.230233 462960 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:03:23.237951 463217 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:03:23.237938 462959 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:03:23.238451 462959 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:03:23.242074 462960 ProcessGroupNCCL.cpp:1669] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-2: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/monitor/test_monitor.py", line 66, in test_empty_wandb wandb_monitor = WandbMonitor(ds_config.monitor_config.wandb) File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/wandb.py", line 16, in __init__ check_wandb_availability() File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/utils.py", line 19, in check_wandb_availability import wandb # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/__init__.py", line 26, in from wandb import sdk as wandb_sdk File "/usr/local/lib/python3.7/site-packages/wandb/sdk/__init__.py", line 3, in from . import wandb_helper as helper # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/sdk/wandb_helper.py", line 6, in from .lib import config_util File "/usr/local/lib/python3.7/site-packages/wandb/sdk/lib/config_util.py", line 10, in from wandb.util import load_yaml File "/usr/local/lib/python3.7/site-packages/wandb/util.py", line 47, in import requests File "/usr/local/lib/python3.7/site-packages/requests/__init__.py", line 43, in import urllib3 File "/usr/local/lib/python3.7/site-packages/urllib3/__init__.py", line 39, in "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168 Process Process-1: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/monitor/test_monitor.py", line 66, in test_empty_wandb wandb_monitor = WandbMonitor(ds_config.monitor_config.wandb) File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/wandb.py", line 16, in __init__ check_wandb_availability() File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/utils.py", line 19, in check_wandb_availability import wandb # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/__init__.py", line 26, in from wandb import sdk as wandb_sdk File "/usr/local/lib/python3.7/site-packages/wandb/sdk/__init__.py", line 3, in from . import wandb_helper as helper # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/sdk/wandb_helper.py", line 6, in from .lib import config_util File "/usr/local/lib/python3.7/site-packages/wandb/sdk/lib/config_util.py", line 10, in from wandb.util import load_yaml File "/usr/local/lib/python3.7/site-packages/wandb/util.py", line 47, in import requests File "/usr/local/lib/python3.7/site-packages/requests/__init__.py", line 43, in import urllib3 File "/usr/local/lib/python3.7/site-packages/urllib3/__init__.py", line 39, in "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168 [31m[1m_____________________________ TestWandB.test_wandb _____________________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:03:27,083] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl If you want to use wandb logging, please `pip install wandb` and follow the instructions at https://docs.wandb.ai/quickstart If you want to use wandb logging, please `pip install wandb` and follow the instructions at https://docs.wandb.ai/quickstart ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:03:28.083428 463242 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:03:28.083456 463497 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:03:28.086393 463499 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:03:28.086391 463241 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:03:28.086941 463241 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:03:28.094296 463242 ProcessGroupNCCL.cpp:1669] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-3: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/monitor/test_monitor.py", line 57, in test_wandb wandb_monitor = WandbMonitor(ds_config.monitor_config.wandb) File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/wandb.py", line 16, in __init__ check_wandb_availability() File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/utils.py", line 19, in check_wandb_availability import wandb # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/__init__.py", line 26, in from wandb import sdk as wandb_sdk File "/usr/local/lib/python3.7/site-packages/wandb/sdk/__init__.py", line 3, in from . import wandb_helper as helper # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/sdk/wandb_helper.py", line 6, in from .lib import config_util File "/usr/local/lib/python3.7/site-packages/wandb/sdk/lib/config_util.py", line 10, in from wandb.util import load_yaml File "/usr/local/lib/python3.7/site-packages/wandb/util.py", line 47, in import requests File "/usr/local/lib/python3.7/site-packages/requests/__init__.py", line 43, in import urllib3 File "/usr/local/lib/python3.7/site-packages/urllib3/__init__.py", line 39, in "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168 Process Process-4: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/monitor/test_monitor.py", line 57, in test_wandb wandb_monitor = WandbMonitor(ds_config.monitor_config.wandb) File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/wandb.py", line 16, in __init__ check_wandb_availability() File "/usr/local/lib/python3.7/site-packages/deepspeed/monitor/utils.py", line 19, in check_wandb_availability import wandb # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/__init__.py", line 26, in from wandb import sdk as wandb_sdk File "/usr/local/lib/python3.7/site-packages/wandb/sdk/__init__.py", line 3, in from . import wandb_helper as helper # noqa: F401 File "/usr/local/lib/python3.7/site-packages/wandb/sdk/wandb_helper.py", line 6, in from .lib import config_util File "/usr/local/lib/python3.7/site-packages/wandb/sdk/lib/config_util.py", line 10, in from wandb.util import load_yaml File "/usr/local/lib/python3.7/site-packages/wandb/util.py", line 47, in import requests File "/usr/local/lib/python3.7/site-packages/requests/__init__.py", line 43, in import urllib3 File "/usr/local/lib/python3.7/site-packages/urllib3/__init__.py", line 39, in "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168 [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/monitor/test_monitor.py::TestWandB::test_empty_wandb /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/monitor/test_monitor.py::TestWandB::test_empty_wandb /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 5.62s call unit/monitor/test_monitor.py::TestTensorBoard::test_tensorboard 5.29s call unit/monitor/test_monitor.py::TestWandB::test_empty_wandb 5.22s call unit/monitor/test_monitor.py::TestTensorBoard::test_empty_tensorboard 4.82s call unit/monitor/test_monitor.py::TestWandB::test_wandb 4.22s call unit/monitor/test_monitor.py::TestCSVMonitor::test_csv_monitor 4.21s call unit/monitor/test_monitor.py::TestCSVMonitor::test_empty_csv_monitor (12 durations < 1s hidden. Use -vv to show these durations.) [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m unit/monitor/test_monitor.py::[1mTestWandB::test_empty_wandb[0m [31mFAILED[0m unit/monitor/test_monitor.py::[1mTestWandB::test_wandb[0m [31m=================== [31m[1m2 failed[0m, [32m4 passed[0m, [33m3 warnings[0m[31m in 30.42s[0m[31m ===================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2456007268 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 2 items unit/pipe/test_pipe_module.py::TestPipeModuleSequential::test[True] [32mPASSED[0m[33m [ 50%][0m unit/pipe/test_pipe_module.py::TestPipeModuleSequential::test[False] [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/pipe/test_pipe_module.py::TestPipeModuleSequential::test[True] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/pipe/test_pipe_module.py::TestPipeModuleSequential::test[True] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 12.17s call unit/pipe/test_pipe_module.py::TestPipeModuleSequential::test[True] 11.63s call unit/pipe/test_pipe_module.py::TestPipeModuleSequential::test[False] (4 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m2 passed[0m, [33m[1m3 warnings[0m[33m in 24.73s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2679895035 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 2 items unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test [32mPASSED[0m[33m [ 50%][0m unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test_flops_profiler_in_inference [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 9.01s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test 4.32s call unit/profiling/flops_profiler/test_flops_profiler.py::TestFlopsProfiler::test_flops_profiler_in_inference (4 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m2 passed[0m, [33m[1m3 warnings[0m[33m in 14.36s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=345477723 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 2 items unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_missing_latest [32mPASSED[0m[33m [ 50%][0m unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_existing_latest [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_missing_latest /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_missing_latest /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 8.22s call unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_existing_latest 5.20s call unit/checkpoint/test_latest_checkpoint.py::TestLatestCheckpoint::test_missing_latest (4 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m2 passed[0m, [33m[1m3 warnings[0m[33m in 14.44s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=849818478 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 12 items unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[0-False] [32mPASSED[0m[33m [ 8%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[3-False] [32mPASSED[0m[33m [ 16%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[2-False] [32mPASSED[0m[33m [ 25%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[2-True] [31mFAILED[0m[31m [ 33%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[3-True] [31mFAILED[0m[31m [ 41%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[2-False] [32mPASSED[0m[31m [ 50%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[1-False] [32mPASSED[0m[31m [ 58%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[0-False] [32mPASSED[0m[31m [ 66%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[1-False] [32mPASSED[0m[31m [ 75%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[3-False] [32mPASSED[0m[31m [ 83%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[3-True] [31mFAILED[0m[31m [ 91%][0m unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[2-True] [31mFAILED[0m[31m [100%][0m =================================== FAILURES =================================== [31m[1m______ TestLRSchedulerCheckpoint.test_checkpoint_no_lr_scheduler[2-True] _______[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:05:26,540] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:05:27,817] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:05:27,818] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:05:27,820] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:05:27,870] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:05:27.547930 467700 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:05:27.547915 467445 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:05:27.555192 467702 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:05:27.555171 467444 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:05:27.557018 467444 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:05:27.559008 467445 ProcessGroupNCCL.cpp:1669] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:05:27.822145 467444 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:05:27.822170 467726 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:05:27.824822 467727 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:05:27.824822 467445 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET Process Process-7: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 113, in test_checkpoint_no_lr_scheduler load_lr_scheduler_states=False) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' Process Process-8: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 113, in test_checkpoint_no_lr_scheduler load_lr_scheduler_states=False) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m________ TestLRSchedulerCheckpoint.test_checkpoint_lr_scheduler[3-True] ________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:05:32,292] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:05:33,504] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:05:33,504] [INFO] [partition_parameters.py:454:__exit__] finished initializing model with 0.00B parameters [2023-05-27 03:05:33,505] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:05:33,506] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:05:33,511] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:05:33.257259 467760 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:05:33.257406 468015 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:05:33.265378 468017 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:05:33.265358 467759 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:05:33.266268 467759 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:05:33.268513 467760 ProcessGroupNCCL.cpp:1669] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:05:33.508345 468041 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:05:33.508340 467760 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:05:33.510841 467759 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:05:33.510848 468042 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! Process Process-10: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 69, in test_checkpoint_lr_scheduler load_lr_scheduler_states=True) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' Process Process-9: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 69, in test_checkpoint_lr_scheduler load_lr_scheduler_states=True) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m______ TestLRSchedulerCheckpoint.test_checkpoint_no_lr_scheduler[3-True] _______[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:06:27,390] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:06:27,696] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:06:27,699] [INFO] [partition_parameters.py:454:__exit__] finished initializing model with 0.00B parameters [2023-05-27 03:06:27,700] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:06:27,702] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:06:27,708] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:06:27.427412 469878 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:06:27.427402 469623 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:06:27.432334 469622 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:06:27.432348 469880 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:06:27.432655 469622 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:06:27.438056 469623 ProcessGroupNCCL.cpp:1669] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:06:27.699025 469623 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:06:27.699033 469904 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:06:27.707826 469622 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:06:27.707891 469905 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! Process Process-21: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 113, in test_checkpoint_no_lr_scheduler load_lr_scheduler_states=False) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' Process Process-22: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 113, in test_checkpoint_no_lr_scheduler load_lr_scheduler_states=False) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m________ TestLRSchedulerCheckpoint.test_checkpoint_lr_scheduler[2-True] ________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:06:32,230] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:06:33,319] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:06:33,320] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:06:33,321] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:06:33,361] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:06:33.054982 470186 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:06:33.054977 469931 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:06:33.061168 470188 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:06:33.061168 469930 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:06:33.061695 469930 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:06:33.065780 469931 ProcessGroupNCCL.cpp:1669] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:06:33.323458 470212 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:06:33.323458 469931 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:06:33.325616 470213 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:06:33.325613 469930 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET Process Process-24: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 69, in test_checkpoint_lr_scheduler load_lr_scheduler_states=True) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' Process Process-23: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/test_lr_scheduler.py", line 69, in test_checkpoint_lr_scheduler load_lr_scheduler_states=True) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 159, in checkpoint_correctness_verification ds_model = create_deepspeed_model(config_dict=config_dict, model=models[0], base_optimizer=base_optimizers[0]) File "/home/aishsh/ds-v0.9.2/tests/unit/checkpoint/common.py", line 141, in create_deepspeed_model optimizer=base_optimizer) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[0-False] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[0-False] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 12.13s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[3-False] 10.92s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[3-False] 10.22s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[1-False] 10.12s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[2-False] 9.32s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[2-False] 9.22s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[0-False] 9.08s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[0-False] 9.03s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[1-False] 5.81s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[2-True] 5.61s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[3-True] 5.52s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[2-True] 4.78s call unit/checkpoint/test_lr_scheduler.py::TestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[3-True] (24 durations < 1s hidden. Use -vv to show these durations.) [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m unit/checkpoint/test_lr_scheduler.py::[1mTestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[2-True][0m [31mFAILED[0m unit/checkpoint/test_lr_scheduler.py::[1mTestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[3-True][0m [31mFAILED[0m unit/checkpoint/test_lr_scheduler.py::[1mTestLRSchedulerCheckpoint::test_checkpoint_no_lr_scheduler[3-True][0m [31mFAILED[0m unit/checkpoint/test_lr_scheduler.py::[1mTestLRSchedulerCheckpoint::test_checkpoint_lr_scheduler[2-True][0m [31m============= [31m[1m4 failed[0m, [32m8 passed[0m, [33m3 warnings[0m[31m in 102.86s (0:01:42)[0m[31m ==============[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1888914074 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 5 items unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[4-False] [32mPASSED[0m[33m [ 20%][0m unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[2-True] [32mPASSED[0m[33m [ 40%][0m unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe[4] [32mPASSED[0m[33m [ 60%][0m unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[4-True] [32mPASSED[0m[33m [ 80%][0m unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[2-False] [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[4-False] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[4-False] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 11.09s call unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[4-False] 10.63s call unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[4-True] 10.44s call unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[2-False] 10.33s call unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe_and_zero[2-True] 10.13s call unit/checkpoint/test_moe_checkpoint.py::TestMoECheckpoint::test_checkpoint_moe[4] (10 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m5 passed[0m, [33m[1m3 warnings[0m[33m in 53.57s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=3994728316 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 3 items unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fused_optimizer [32mPASSED[0m[33m [ 33%][0m unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_unfused_optimizer [32mPASSED[0m[33m [ 66%][0m unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fp32_optimizer [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fused_optimizer /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fused_optimizer /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 10.52s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fused_optimizer 10.42s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_unfused_optimizer 9.32s call unit/checkpoint/test_other_optimizer.py::TestOtherOptimizerCheckpoint::test_checkpoint_fp32_optimizer (6 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m3 passed[0m, [33m[1m3 warnings[0m[33m in 31.22s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1138316618 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 4 items unit/checkpoint/test_reshape_checkpoint.py::test_reshape_222_to_211 [32mPASSED[0m[32m [ 25%][0m unit/checkpoint/test_reshape_checkpoint.py::test_reshape_222_to_121 [32mPASSED[0m[33m [ 50%][0m unit/checkpoint/test_reshape_checkpoint.py::test_reshape_222_to_111 [32mPASSED[0m[33m [ 75%][0m unit/checkpoint/test_reshape_checkpoint.py::test_reshape_222_to_122 [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m unit/checkpoint/test_reshape_checkpoint.py::test_reshape_222_to_211 /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_reshape_checkpoint.py::test_reshape_222_to_211 /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== (12 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m4 passed[0m, [33m[1m2 warnings[0m[33m in 0.97s[0m[33m =========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=3543619067 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 4 items unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[IGNORE] [32mPASSED[0m[33m [ 25%][0m unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[WARN] [32mPASSED[0m[33m [ 50%][0m unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[FAIL] [32mPASSED[0m[33m [ 75%][0m unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unknown_tag_validation [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[IGNORE] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[IGNORE] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 6.42s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[IGNORE] 6.02s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[WARN] 5.22s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unknown_tag_validation 5.12s call unit/checkpoint/test_tag_validation.py::TestCheckpointValidationTag::test_checkpoint_unique_tag[FAIL] (8 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m4 passed[0m, [33m[1m3 warnings[0m[33m in 23.72s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1608075071 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 3 items unit/checkpoint/test_pipeline.py::TestPipelineCheckpoint::test_checkpoint_pipe_engine[0] [32mPASSED[0m[33m [ 33%][0m unit/checkpoint/test_pipeline.py::TestPipelineCheckpoint::test_checkpoint_pipe_engine[1] [32mPASSED[0m[33m [ 66%][0m unit/checkpoint/test_pipeline.py::TestPipelineCheckpoint::test_checkpoint_pipe_module[base_topo0-test_topo0] [33mSKIPPED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_pipeline.py::TestPipelineCheckpoint::test_checkpoint_pipe_engine[0] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_pipeline.py::TestPipelineCheckpoint::test_checkpoint_pipe_engine[0] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 13.90s call unit/checkpoint/test_pipeline.py::TestPipelineCheckpoint::test_checkpoint_pipe_engine[0] 13.75s call unit/checkpoint/test_pipeline.py::TestPipelineCheckpoint::test_checkpoint_pipe_engine[1] (6 durations < 1s hidden. Use -vv to show these durations.) [33m================== [32m2 passed[0m, [33m[1m1 skipped[0m, [33m[1m3 warnings[0m[33m in 28.58s[0m[33m ===================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2965898371 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 9 items unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-True] [32mPASSED[0m[33m [ 11%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-False-False] [32mPASSED[0m[33m [ 22%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-True] [32mPASSED[0m[33m [ 33%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-False-False] [32mPASSED[0m[33m [ 44%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-False] [32mPASSED[0m[33m [ 55%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-True] [32mPASSED[0m[33m [ 66%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-False-False] [32mPASSED[0m[33m [ 77%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-False] [32mPASSED[0m[33m [ 88%][0m unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-False] [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-True] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-True] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 5.77s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-True] 5.41s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-False-False] 5.21s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-False] 4.41s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-True-True-True] 4.41s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-False-False] 4.41s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-False] 4.31s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[True-False-True-False] 4.31s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-False-False] 4.31s call unit/checkpoint/test_sparse.py::TestSparseCheckpoint::test_non_strict_load_sparse[False-False-True-True] (18 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m9 passed[0m, [33m[1m3 warnings[0m[33m in 43.53s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2213585216 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 53 items unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[3] [32mPASSED[0m[33m [ 1%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_not_load_optimizer_state[2] [32mPASSED[0m[33m [ 3%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[2] [32mPASSED[0m[33m [ 5%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_module_only[3] [32mPASSED[0m[33m [ 7%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_not_load_optimizer_state[1] [32mPASSED[0m[33m [ 9%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_module_only[1] [32mPASSED[0m[33m [ 11%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_module_only[2] [32mPASSED[0m[33m [ 13%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_not_load_optimizer_state[3] [32mPASSED[0m[33m [ 15%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[1] [32mPASSED[0m[33m [ 16%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[1] [32mPASSED[0m[33m [ 18%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[3] [32mPASSED[0m[33m [ 20%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[3] [32mPASSED[0m[33m [ 22%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[0] [32mPASSED[0m[33m [ 24%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[0] [32mPASSED[0m[33m [ 26%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[2] [32mPASSED[0m[33m [ 28%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[0] [32mPASSED[0m[33m [ 30%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[2] [32mPASSED[0m[33m [ 32%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[1] [32mPASSED[0m[33m [ 33%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[1] [32mPASSED[0m[33m [ 35%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[2] [32mPASSED[0m[33m [ 37%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[3] [32mPASSED[0m[33m [ 39%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-True-True] [32mPASSED[0m[33m [ 41%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[False-False-False] [32mPASSED[0m[33m [ 43%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[True-True-True] [32mPASSED[0m[33m [ 45%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-False] [32mPASSED[0m[33m [ 47%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-True-False] [32mPASSED[0m[33m [ 49%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[True-True-False] [32mPASSED[0m[33m [ 50%][0m unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[True-False-False] [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-True-True] unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-True-True] unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-False] unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-False] unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-True-False] unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-True-False] /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[3] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[3] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 12.38s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[3] 11.93s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_not_load_optimizer_state[3] 11.74s setup unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-False] 10.83s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[False-False-False] 10.72s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[True-True-True] 10.52s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_module_only[3] 10.52s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[1] 10.33s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_fixed_dp[True-True-False] 10.32s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[3] 10.14s setup unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-True-False] 10.13s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[1] 10.02s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[2] 9.83s setup unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-True-True] 9.83s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_module_only[2] 9.83s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[3] 9.82s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_not_load_optimizer_state[1] 9.73s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[0] 9.62s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_not_load_optimizer_state[2] 9.52s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_module_only[1] 9.43s call unit/checkpoint/test_zero_optimizer.py::TestZeROCheckpointFrozenWeights::test_load_optimizer_state[2] 9.42s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_load_immediate_save[0] 9.32s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[1] 8.72s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_save_before_accum_grad_is_done[2] 7.23s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-False-False] 7.12s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[False-True-True] 6.92s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[3] 6.52s call unit/checkpoint/test_zero_optimizer.py::TestZeROElasticCheckpoint::test_elastic_checkpoint_change_dp[True-True-False] 6.32s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[1] 6.22s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[0] 6.12s call unit/checkpoint/test_zero_optimizer.py::TestZeROSaveLoadEdgeCase::test_immediate_save_load[2] (52 durations < 1s hidden. Use -vv to show these durations.) [33m================== [32m27 passed[0m, [33m[1m9 warnings[0m[33m in 882.83s (0:14:42)[0m[33m ==================[0m !!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Test hanged, exiting !!!!!!!!!!!!!!!!!! [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=3942688220 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 22 items unit/elasticity/test_elastic.py::TestElasticConfigChanged::test [32mPASSED[0m[33m [ 4%][0m unit/elasticity/test_elastic.py::TestNonElasticBatchParamsWithOverride::test [32mPASSED[0m[33m [ 9%][0m unit/elasticity/test_elastic.py::TestNonElasticBatchParams::test [32mPASSED[0m[33m [ 13%][0m unit/elasticity/test_elastic.py::test_missing_micro_batch [32mPASSED[0m[33m [ 18%][0m unit/elasticity/test_elastic.py::test_future_elastic_version [32mPASSED[0m[33m [ 22%][0m unit/elasticity/test_elastic.py::test_model_parallel_v1_invalid [32mPASSED[0m[33m [ 27%][0m unit/elasticity/test_elastic.py::test_valid_world_size [32mPASSED[0m[33m [ 31%][0m unit/elasticity/test_elastic.py::test_model_parallel_v2_invalid [32mPASSED[0m[33m [ 36%][0m unit/elasticity/test_elastic.py::test_proper_mbsz [32mPASSED[0m[33m [ 40%][0m unit/elasticity/test_elastic.py::test_invalid_config_values[micro_batch_sizes-5] [32mPASSED[0m[33m [ 45%][0m unit/elasticity/test_elastic.py::test_basic_10k [32mPASSED[0m[33m [ 50%][0m unit/elasticity/test_elastic.py::test_missing_max_batch [32mPASSED[0m[33m [ 54%][0m unit/elasticity/test_elastic.py::test_model_parallel_v2_valid [32mPASSED[0m[33m [ 59%][0m unit/elasticity/test_elastic.py::test_invalid_config_values[min_gpus--1] [32mPASSED[0m[33m [ 63%][0m unit/elasticity/test_elastic.py::test_invalid_world_size [32mPASSED[0m[33m [ 68%][0m unit/elasticity/test_elastic.py::test_disabled [32mPASSED[0m[33m [ 72%][0m unit/elasticity/test_elastic.py::test_invalid_config_values[max_gpus--1] [32mPASSED[0m[33m [ 77%][0m unit/elasticity/test_elastic.py::test_empty_config [32mPASSED[0m[33m [ 81%][0m unit/elasticity/test_elastic.py::test_invalid_config_values[micro_batch_sizes-value5] [32mPASSED[0m[33m [ 86%][0m unit/elasticity/test_elastic.py::test_invalid_config_values[micro_batch_sizes-value4] [32mPASSED[0m[33m [ 90%][0m unit/elasticity/test_elastic.py::test_old_version [32mPASSED[0m[33m [ 95%][0m unit/elasticity/test_elastic.py::test_invalid_config_values[micro_batch_sizes-value0] [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/elasticity/test_elastic.py::TestElasticConfigChanged::test /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/elasticity/test_elastic.py::TestElasticConfigChanged::test /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 6.12s call unit/elasticity/test_elastic.py::TestNonElasticBatchParamsWithOverride::test 4.79s call unit/elasticity/test_elastic.py::TestElasticConfigChanged::test 4.31s call unit/elasticity/test_elastic.py::TestNonElasticBatchParams::test (63 durations < 1s hidden. Use -vv to show these durations.) [33m======================= [32m22 passed[0m, [33m[1m3 warnings[0m[33m in 16.26s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1902720899 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 4 items / 4 deselected / 0 selected [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [33m======================= [33m[1m4 deselected[0m, [33m[1m1 warning[0m[33m in 0.92s[0m[33m =======================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2540101885 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m_________ ERROR collecting unit/inference/test_checkpoint_sharding.py __________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/unit/inference/test_checkpoint_sharding.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) unit/inference/test_checkpoint_sharding.py:12: in from transformers import AutoConfig, AutoModelForCausalLM /usr/local/lib/python3.7/site-packages/transformers/__init__.py:26: in from . import dependency_versions_check /usr/local/lib/python3.7/site-packages/transformers/dependency_versions_check.py:17: in from .utils.versions import require_version, require_version_core /usr/local/lib/python3.7/site-packages/transformers/utils/__init__.py:30: in from .generic import ( /usr/local/lib/python3.7/site-packages/transformers/utils/generic.py:29: in from .import_utils import is_flax_available, is_tf_available, is_torch_available, is_torch_fx_proxy /usr/local/lib/python3.7/site-packages/transformers/utils/import_utils.py:32: in from . import logging /usr/local/lib/python3.7/site-packages/transformers/utils/logging.py:35: in import huggingface_hub.utils as hf_hub_utils /usr/local/lib/python3.7/site-packages/huggingface_hub/utils/__init__.py:32: in from ._errors import ( /usr/local/lib/python3.7/site-packages/huggingface_hub/utils/_errors.py:3: in from requests import HTTPError, Response /usr/local/lib/python3.7/site-packages/requests/__init__.py:43: in import urllib3 /usr/local/lib/python3.7/site-packages/urllib3/__init__.py:39: in "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " E ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168[0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m unit/inference/test_checkpoint_sharding.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m========================= [33m1 warning[0m, [31m[1m1 error[0m[31m in 1.44s[0m[31m ==========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1244343831 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m______________ ERROR collecting unit/inference/test_inference.py _______________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/unit/inference/test_inference.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) unit/inference/test_inference.py:16: in from transformers import pipeline /usr/local/lib/python3.7/site-packages/transformers/__init__.py:26: in from . import dependency_versions_check /usr/local/lib/python3.7/site-packages/transformers/dependency_versions_check.py:17: in from .utils.versions import require_version, require_version_core /usr/local/lib/python3.7/site-packages/transformers/utils/__init__.py:30: in from .generic import ( /usr/local/lib/python3.7/site-packages/transformers/utils/generic.py:29: in from .import_utils import is_flax_available, is_tf_available, is_torch_available, is_torch_fx_proxy /usr/local/lib/python3.7/site-packages/transformers/utils/import_utils.py:32: in from . import logging /usr/local/lib/python3.7/site-packages/transformers/utils/logging.py:35: in import huggingface_hub.utils as hf_hub_utils /usr/local/lib/python3.7/site-packages/huggingface_hub/utils/__init__.py:32: in from ._errors import ( /usr/local/lib/python3.7/site-packages/huggingface_hub/utils/_errors.py:3: in from requests import HTTPError, Response /usr/local/lib/python3.7/site-packages/requests/__init__.py:43: in import urllib3 /usr/local/lib/python3.7/site-packages/urllib3/__init__.py:39: in "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " E ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168[0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m unit/inference/test_inference.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m========================= [33m1 warning[0m, [31m[1m1 error[0m[31m in 1.45s[0m[31m ==========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=208031903 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 0 items / 1 error ==================================== ERRORS ==================================== [31m[1m___________ ERROR collecting unit/inference/test_model_profiling.py ____________[0m [31mImportError while importing test module '/home/aishsh/ds-v0.9.2/tests/unit/inference/test_model_profiling.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/lib/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) unit/inference/test_model_profiling.py:11: in from transformers import pipeline /usr/local/lib/python3.7/site-packages/transformers/__init__.py:26: in from . import dependency_versions_check /usr/local/lib/python3.7/site-packages/transformers/dependency_versions_check.py:17: in from .utils.versions import require_version, require_version_core /usr/local/lib/python3.7/site-packages/transformers/utils/__init__.py:30: in from .generic import ( /usr/local/lib/python3.7/site-packages/transformers/utils/generic.py:29: in from .import_utils import is_flax_available, is_tf_available, is_torch_available, is_torch_fx_proxy /usr/local/lib/python3.7/site-packages/transformers/utils/import_utils.py:32: in from . import logging /usr/local/lib/python3.7/site-packages/transformers/utils/logging.py:35: in import huggingface_hub.utils as hf_hub_utils /usr/local/lib/python3.7/site-packages/huggingface_hub/utils/__init__.py:32: in from ._errors import ( /usr/local/lib/python3.7/site-packages/huggingface_hub/utils/_errors.py:3: in from requests import HTTPError, Response /usr/local/lib/python3.7/site-packages/requests/__init__.py:43: in import urllib3 /usr/local/lib/python3.7/site-packages/urllib3/__init__.py:39: in "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently " E ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168[0m [36m[1m=========================== short test summary info ============================[0m [31mERROR[0m unit/inference/test_model_profiling.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! [31m=============================== [31m[1m1 error[0m[31m in 1.49s[0m[31m ===============================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=259071128 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 7 items unit/launcher/test_ds_arguments.py::test_core_binding_arguments [32mPASSED[0m[32m [ 14%][0m unit/launcher/test_ds_arguments.py::test_no_ds_parser [32mPASSED[0m[33m [ 28%][0m unit/launcher/test_ds_arguments.py::test_no_ds_arguments [32mPASSED[0m[33m [ 42%][0m unit/launcher/test_ds_arguments.py::test_no_ds_config_argument [32mPASSED[0m[33m [ 57%][0m unit/launcher/test_ds_arguments.py::test_core_deepscale_arguments [32mPASSED[0m[33m [ 71%][0m unit/launcher/test_ds_arguments.py::test_no_ds_arguments_no_ds_parser [32mPASSED[0m[33m [ 85%][0m unit/launcher/test_ds_arguments.py::test_no_ds_enable_argument [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m unit/launcher/test_ds_arguments.py::test_core_binding_arguments /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/launcher/test_ds_arguments.py::test_core_binding_arguments /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== (21 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m7 passed[0m, [33m[1m2 warnings[0m[33m in 0.93s[0m[33m =========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2195940584 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 5 items unit/launcher/test_multinode_runner.py::test_slurm_runner [32mPASSED[0m[32m [ 20%][0m unit/launcher/test_multinode_runner.py::test_mvapich_runner [32mPASSED[0m[33m [ 40%][0m unit/launcher/test_multinode_runner.py::test_mpich_runner [32mPASSED[0m[33m [ 60%][0m unit/launcher/test_multinode_runner.py::test_openmpi_runner [32mPASSED[0m[33m [ 80%][0m unit/launcher/test_multinode_runner.py::test_pdsh_runner [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m unit/launcher/test_multinode_runner.py::test_slurm_runner /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/launcher/test_multinode_runner.py::test_slurm_runner /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== (15 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m5 passed[0m, [33m[1m2 warnings[0m[33m in 0.92s[0m[33m =========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1623618086 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 7 items unit/launcher/test_run.py::test_parser_mutual_exclusive [32mPASSED[0m[32m [ 14%][0m unit/launcher/test_run.py::test_num_plus_parser [32mPASSED[0m[33m [ 28%][0m unit/launcher/test_run.py::test_hostfiles_bad [32mPASSED[0m[33m [ 42%][0m unit/launcher/test_run.py::test_hostfile_good [32mPASSED[0m[33m [ 57%][0m unit/launcher/test_run.py::test_parser_errors [32mPASSED[0m[33m [ 71%][0m unit/launcher/test_run.py::test_parser_multinode [32mPASSED[0m[33m [ 85%][0m unit/launcher/test_run.py::test_parser_local [32mPASSED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m unit/launcher/test_run.py::test_parser_mutual_exclusive /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/launcher/test_run.py::test_parser_mutual_exclusive /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== (21 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [32m7 passed[0m, [33m[1m2 warnings[0m[33m in 0.94s[0m[33m =========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=761628897 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 3 items unit/model_parallelism/test_configurable_parallel_mp.py::TestConfigurableMP::test_gpt2_basic [33mSKIPPED[0m[33m [ 33%][0m unit/model_parallelism/test_configurable_parallel_mp.py::TestConfigurableMP::test_gpt2_mp2_no_resize [33mSKIPPED[0m[33m [ 66%][0m unit/model_parallelism/test_configurable_parallel_mp.py::TestConfigurableResizeMP::test [33mSKIPPED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/model_parallelism/test_configurable_parallel_mp.py:62 /home/aishsh/ds-v0.9.2/tests/unit/model_parallelism/test_configurable_parallel_mp.py:62: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(1) unit/model_parallelism/test_configurable_parallel_mp.py:90 /home/aishsh/ds-v0.9.2/tests/unit/model_parallelism/test_configurable_parallel_mp.py:90: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(2) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== (6 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [33m[1m3 skipped[0m, [33m[1m3 warnings[0m[33m in 0.91s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=625012969 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 7 items unit/model_parallelism/test_configurable_parallel_pp.py::TestConfigurablePP::test_pp_basic [33mSKIPPED[0m[33m [ 14%][0m unit/model_parallelism/test_configurable_parallel_pp.py::TestConfigurableResizePP::test_world_size_2to4[1-2-1-4] [33mSKIPPED[0m[33m [ 28%][0m unit/model_parallelism/test_configurable_parallel_pp.py::TestConfigurableResizePP::test_world_size_4to2[2-2-2-1] [33mSKIPPED[0m[33m [ 42%][0m unit/model_parallelism/test_configurable_parallel_pp.py::TestConfigurableResizePP::test_world_size_2to1[1-2-1-1] [33mSKIPPED[0m[33m [ 57%][0m unit/model_parallelism/test_configurable_parallel_pp.py::TestConfigurableResizePP::test_world_size_4to1[2-2-1-1] [33mSKIPPED[0m[33m [ 71%][0m unit/model_parallelism/test_configurable_parallel_pp.py::TestConfigurableResizePP::test_world_size_1to4[1-1-2-2] [33mSKIPPED[0m[33m [ 85%][0m unit/model_parallelism/test_configurable_parallel_pp.py::TestConfigurableResizePP::test_world_size_2to4[2-1-2-2] [33mSKIPPED[0m[33m [100%][0m [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/model_parallelism/test_configurable_parallel_pp.py:236 /home/aishsh/ds-v0.9.2/tests/unit/model_parallelism/test_configurable_parallel_pp.py:236: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(1) unit/model_parallelism/test_configurable_parallel_pp.py:243 /home/aishsh/ds-v0.9.2/tests/unit/model_parallelism/test_configurable_parallel_pp.py:243: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(1) unit/model_parallelism/test_configurable_parallel_pp.py:250 /home/aishsh/ds-v0.9.2/tests/unit/model_parallelism/test_configurable_parallel_pp.py:250: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(2) unit/model_parallelism/test_configurable_parallel_pp.py:257 /home/aishsh/ds-v0.9.2/tests/unit/model_parallelism/test_configurable_parallel_pp.py:257: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(4) unit/model_parallelism/test_configurable_parallel_pp.py:264 /home/aishsh/ds-v0.9.2/tests/unit/model_parallelism/test_configurable_parallel_pp.py:264: PytestUnknownMarkWarning: Unknown pytest.mark.world_size - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html @pytest.mark.world_size(4) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== (14 durations < 1s hidden. Use -vv to show these durations.) [33m======================== [33m[1m7 skipped[0m, [33m[1m6 warnings[0m[33m in 0.93s[0m[33m ========================[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=532039706 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 16 items unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-False-resulting_optimizer0] [32mPASSED[0m[33m [ 6%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-True-resulting_optimizer14] [31mFAILED[0m[31m [ 12%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-True-True-resulting_optimizer5] [32mPASSED[0m[31m [ 18%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-True-False-resulting_optimizer9] [32mPASSED[0m[31m [ 25%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-True-False-resulting_optimizer3] [32mPASSED[0m[31m [ 31%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-False-False-resulting_optimizer8] [32mPASSED[0m[31m [ 37%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-False-resulting_optimizer2] [31mFAILED[0m[31m [ 43%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-True-False-resulting_optimizer1] [32mPASSED[0m[31m [ 50%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-True-True-resulting_optimizer13] [32mPASSED[0m[31m [ 56%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-False-True-resulting_optimizer12] [32mPASSED[0m[31m [ 62%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-True-resulting_optimizer4] [32mPASSED[0m[31m [ 68%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-True-True-resulting_optimizer15] [32mPASSED[0m[31m [ 75%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-True-False-resulting_optimizer11] [32mPASSED[0m[31m [ 81%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-True-resulting_optimizer6] [31mFAILED[0m[31m [ 87%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-False-resulting_optimizer10] [31mFAILED[0m[31m [ 93%][0m unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-True-True-resulting_optimizer7] [32mPASSED[0m[31m [100%][0m =================================== FAILURES =================================== [31m[1m_______ TestAdamConfigs.test[Adam-True-False-True-resulting_optimizer14] _______[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:25:39,433] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:25:39,620] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:25:39,621] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:25:39,657] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:25:39.435680 493269 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:25:39.435679 493140 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:25:39.436115 493140 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:25:39.625730 493282 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:25:39.625730 493140 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET Process Process-2: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_adamw.py", line 69, in test model_parameters=model.parameters()) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m______ TestAdamConfigs.test[AdamW-True-False-False-resulting_optimizer2] _______[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:26:02,259] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:26:02,444] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:26:02,445] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:26:02,482] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:26:02.261610 494017 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:26:02.261608 493888 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:26:02.261962 493888 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:26:02.449623 494030 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:26:02.449623 493888 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET Process Process-7: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_adamw.py", line 69, in test model_parameters=model.parameters()) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m_______ TestAdamConfigs.test[AdamW-True-False-True-resulting_optimizer6] _______[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:26:35,648] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:26:35,804] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:26:35,805] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:26:35,840] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:26:35.649827 494931 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:26:35.649842 495060 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:26:35.650161 494931 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:26:35.809242 494931 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:26:35.809260 495073 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! Process Process-14: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_adamw.py", line 69, in test model_parameters=model.parameters()) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m______ TestAdamConfigs.test[Adam-True-False-False-resulting_optimizer10] _______[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:26:40,169] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2023-05-27 03:26:40,345] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+e0e8085, git-hash=e0e8085, git-branch=HEAD [2023-05-27 03:26:40,346] [WARNING] [config_utils.py:70:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead [2023-05-27 03:26:40,386] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:26:40.170698 495219 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:26:40.170684 495090 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:26:40.171097 495090 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:26:40.349159 495090 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:26:40.349182 495232 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! Process Process-15: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_adamw.py", line 69, in test model_parameters=model.parameters()) File "/usr/local/lib/python3.7/site-packages/deepspeed/__init__.py", line 175, in initialize config_class=config_class) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 308, in __init__ self._configure_optimizer(optimizer, model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer basic_optimizer = self._configure_basic_optimizer(model_parameters) File "/usr/local/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1220, in _configure_basic_optimizer adamw_mode=effective_adam_w_mode) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-False-resulting_optimizer0] /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-False-resulting_optimizer0] /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 5.71s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-True-resulting_optimizer4] 5.67s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-False-False-resulting_optimizer0] 5.51s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-False-True-resulting_optimizer12] 5.01s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-False-False-resulting_optimizer8] 4.81s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-True-resulting_optimizer14] 4.61s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-True-True-resulting_optimizer15] 4.51s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-True-resulting_optimizer6] 4.51s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-False-False-resulting_optimizer2] 4.51s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-False-False-resulting_optimizer10] 4.41s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-True-True-False-resulting_optimizer11] 4.41s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-True-True-resulting_optimizer5] 4.31s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-True-False-resulting_optimizer3] 4.31s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-True-True-resulting_optimizer13] 4.31s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[Adam-False-True-False-resulting_optimizer9] 4.21s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-True-True-True-resulting_optimizer7] 4.21s call unit/ops/adam/test_adamw.py::TestAdamConfigs::test[AdamW-False-True-False-resulting_optimizer1] (32 durations < 1s hidden. Use -vv to show these durations.) [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m unit/ops/adam/test_adamw.py::[1mTestAdamConfigs::test[Adam-True-False-True-resulting_optimizer14][0m [31mFAILED[0m unit/ops/adam/test_adamw.py::[1mTestAdamConfigs::test[AdamW-True-False-False-resulting_optimizer2][0m [31mFAILED[0m unit/ops/adam/test_adamw.py::[1mTestAdamConfigs::test[AdamW-True-False-True-resulting_optimizer6][0m [31mFAILED[0m unit/ops/adam/test_adamw.py::[1mTestAdamConfigs::test[Adam-True-False-False-resulting_optimizer10][0m [31m============= [31m[1m4 failed[0m, [32m12 passed[0m, [33m3 warnings[0m[31m in 76.13s (0:01:16)[0m[31m ==============[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=1468307262 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 21 items unit/ops/adam/test_cpu_adam.py::TestCPUAdamGPUError::test_cpu_adam_gpu_error [31mFAILED[0m[31m [ 4%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1024-fp32] [32mPASSED[0m[31m [ 9%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[22-fp16] [32mPASSED[0m[31m [ 14%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1024-fp16] [31mFAILED[0m[31m [ 19%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1048576-fp32] [31mFAILED[0m[31m [ 23%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1048576-fp16] [31mFAILED[0m[31m [ 28%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1024-fp32] [31mFAILED[0m[31m [ 33%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[22-fp32] [31mFAILED[0m[31m [ 38%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[128-fp32] [32mPASSED[0m[31m [ 42%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[64-fp16] [32mPASSED[0m[31m [ 47%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[22-fp32] [32mPASSED[0m[31m [ 52%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp32] [31mFAILED[0m[31m [ 57%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1048576-fp32] [32mPASSED[0m[31m [ 61%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp16] [31mFAILED[0m[31m [ 66%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1024-fp16] [32mPASSED[0m[31m [ 71%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[128-fp16] [31mFAILED[0m[31m [ 76%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[64-fp32] [32mPASSED[0m[31m [ 80%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[128-fp32] [31mFAILED[0m[31m [ 85%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1048576-fp16] [32mPASSED[0m[31m [ 90%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[22-fp16] [31mFAILED[0m[31m [ 95%][0m unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[128-fp16] [32mPASSED[0m[31m [100%][0m =================================== FAILURES =================================== [31m[1m_________________ TestCPUAdamGPUError.test_cpu_adam_gpu_error __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:26:55,522] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:26:55.670020 495796 ProcessGroupNCCL.cpp:601] [Rank 1] NCCL watchdog thread started! I0527 03:26:55.670010 495519 ProcessGroupNCCL.cpp:500] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:26:55.678297 495518 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:26:55.678390 495798 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:26:55.678975 495518 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. I0527 03:26:55.681064 495519 ProcessGroupNCCL.cpp:1669] Rank 1 using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-1: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 122, in test_cpu_adam_gpu_error optimizer = DeepSpeedCPUAdam([param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' Process Process-2: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 122, in test_cpu_adam_gpu_error optimizer = DeepSpeedCPUAdam([param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m_________________ TestCPUAdam.test_fused_adam_equal[1024-fp16] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:27:11,125] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:27:11.127720 496292 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:27:11.127709 496152 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:27:11.128178 496152 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-5: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m_______________ TestCPUAdam.test_fused_adam_equal[1048576-fp32] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:27:16,050] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:27:16.052475 496457 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:27:16.052475 496317 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:27:16.053001 496317 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-6: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m_______________ TestCPUAdam.test_fused_adam_equal[1048576-fp16] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:27:20,738] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:27:20.740389 496622 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:27:20.740391 496482 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:27:20.740841 496482 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-7: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m_________________ TestCPUAdam.test_fused_adam_equal[1024-fp32] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:27:25,475] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:27:25.477136 496850 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:27:25.477134 496710 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:27:25.477574 496710 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-8: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m__________________ TestCPUAdam.test_fused_adam_equal[22-fp32] __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:27:30,202] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:27:30.204545 497015 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:27:30.204533 496875 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:27:30.204886 496875 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-9: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m__________________ TestCPUAdam.test_fused_adam_equal[64-fp32] __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:27:47,805] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:27:47.807761 497639 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:27:47.807761 497499 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:27:47.808174 497499 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-13: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m__________________ TestCPUAdam.test_fused_adam_equal[64-fp16] __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:27:56,792] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:27:56.794697 497957 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:27:56.794689 497817 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:27:56.795138 497817 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-15: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m_________________ TestCPUAdam.test_fused_adam_equal[128-fp16] __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:05,868] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:05.870954 498275 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:05.870954 498135 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:05.871385 498135 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-17: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m_________________ TestCPUAdam.test_fused_adam_equal[128-fp32] __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:14,788] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:14.791132 498593 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:14.791034 498453 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:14.791811 498453 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-19: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [31m[1m__________________ TestCPUAdam.test_fused_adam_equal[22-fp16] __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:23,792] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:23.795222 498911 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:23.795218 498771 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:23.795699 498771 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-21: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/adam/test_cpu_adam.py", line 80, in test_fused_adam_equal cpu_optimizer = DeepSpeedCPUAdam([cpu_param]) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__ self.ds_opt_adam = CPUAdamBuilder().load() File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__ self.ds_opt_adam.destroy_adam(self.opt_id) AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' [33m=============================== warnings summary ===============================[0m ../../../../usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8 /usr/local/lib/python3.7/site-packages/_pytest/fixtures.py:8: PytestDeprecationWarning: A private pytest class or function was used. from collections import deque unit/ops/adam/test_cpu_adam.py::TestCPUAdamGPUError::test_cpu_adam_gpu_error /home/aishsh/ds-v0.9.2/tests/conftest.py:48: UserWarning: Running test without verifying torch version, please provide an expected torch version with --torch_ver "Running test without verifying torch version, please provide an expected torch version with --torch_ver") unit/ops/adam/test_cpu_adam.py::TestCPUAdamGPUError::test_cpu_adam_gpu_error /home/aishsh/ds-v0.9.2/tests/conftest.py:55: UserWarning: Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver "Running test without verifying cuda version, please provide an expected cuda version with --cuda_ver") -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 6.68s call unit/ops/adam/test_cpu_adam.py::TestCPUAdamGPUError::test_cpu_adam_gpu_error 5.43s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1024-fp32] 4.92s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1024-fp16] 4.81s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[22-fp32] 4.81s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp16] 4.72s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[22-fp16] 4.71s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1024-fp32] 4.71s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[22-fp16] 4.71s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1048576-fp16] 4.71s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[1048576-fp32] 4.71s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[128-fp16] 4.61s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[128-fp32] 4.61s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_fused_adam_equal[64-fp32] 4.41s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[128-fp16] 4.31s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1024-fp16] 4.31s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1048576-fp32] 4.31s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[22-fp32] 4.31s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[64-fp16] 4.31s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[1048576-fp16] 4.21s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[128-fp32] 4.21s call unit/ops/adam/test_cpu_adam.py::TestCPUAdam::test_torch_adamw_equal[64-fp32] (42 durations < 1s hidden. Use -vv to show these durations.) [36m[1m=========================== short test summary info ============================[0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdamGPUError::test_cpu_adam_gpu_error[0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[1024-fp16][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[1048576-fp32][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[1048576-fp16][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[1024-fp32][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[22-fp32][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[64-fp32][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[64-fp16][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[128-fp16][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[128-fp32][0m [31mFAILED[0m unit/ops/adam/test_cpu_adam.py::[1mTestCPUAdam::test_fused_adam_equal[22-fp16][0m [31m============ [31m[1m11 failed[0m, [32m10 passed[0m, [33m3 warnings[0m[31m in 100.88s (0:01:40)[0m[31m =============[0m [1m============================= test session starts ==============================[0m platform linux -- Python 3.7.12, pytest-7.3.1, pluggy-1.0.0 -- /usr/local/bin/python3.7 cachedir: .pytest_cache Using --randomly-seed=2927879481 rootdir: /home/aishsh/ds-v0.9.2/tests configfile: pytest.ini plugins: xdist-3.2.1, randomly-3.12.0, forked-1.6.0 [1mcollecting ... [0mcollected 64 items / 16 deselected / 48 selected unit/ops/aio/test_aio.py::TestRead::test_parallel_read[True-False-True] [31mFAILED[0m[31m [ 2%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-True-True] [31mFAILED[0m[31m [ 4%][0m unit/ops/aio/test_aio.py::TestRead::test_parallel_read[True-False-False] [31mFAILED[0m[31m [ 6%][0m unit/ops/aio/test_aio.py::TestRead::test_parallel_read[False-True-True] [31mFAILED[0m[31m [ 8%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-True-False-False] [31mFAILED[0m[31m [ 10%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-True-True-True] [31mFAILED[0m[31m [ 12%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-False-False] [31mFAILED[0m[31m [ 14%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-False-False-True] [31mFAILED[0m[31m [ 16%][0m unit/ops/aio/test_aio.py::TestRead::test_parallel_read[False-True-False] [31mFAILED[0m[31m [ 18%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-False-True] [31mFAILED[0m[31m [ 20%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-True-False-True] [31mFAILED[0m[31m [ 22%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-True-True-True] [31mFAILED[0m[31m [ 25%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-False-False-False] [31mFAILED[0m[31m [ 27%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-True-True-False] [31mFAILED[0m[31m [ 29%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-False-True-False] [31mFAILED[0m[31m [ 31%][0m unit/ops/aio/test_aio.py::TestRead::test_parallel_read[False-False-True] [31mFAILED[0m[31m [ 33%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-False-True-True] [31mFAILED[0m[31m [ 35%][0m unit/ops/aio/test_aio.py::TestRead::test_parallel_read[False-False-False] [31mFAILED[0m[31m [ 37%][0m unit/ops/aio/test_aio.py::TestRead::test_parallel_read[True-True-True] [31mFAILED[0m[31m [ 39%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-True-False-True] [31mFAILED[0m[31m [ 41%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[False-True-False-False] [31mFAILED[0m[31m [ 43%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-True-True-False] [31mFAILED[0m[31m [ 45%][0m unit/ops/aio/test_aio.py::TestRead::test_async_read[True-False-True-False] [31mFAILED[0m[31m [ 47%][0m unit/ops/aio/test_aio.py::TestRead::test_parallel_read[True-True-False] [31mFAILED[0m[31m [ 50%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-False-False-True] [31mFAILED[0m[31m [ 52%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[False-True-False] [31mFAILED[0m[31m [ 54%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-False-False-False] [31mFAILED[0m[31m [ 56%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-False-True-False] [31mFAILED[0m[31m [ 58%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-True-True-True] [31mFAILED[0m[31m [ 60%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-True-True-False] [31mFAILED[0m[31m [ 62%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-True-False-True] [31mFAILED[0m[31m [ 64%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-False-True-True] [31mFAILED[0m[31m [ 66%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[False-True-True] [31mFAILED[0m[31m [ 68%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-False-False-True] [31mFAILED[0m[31m [ 70%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[False-False-False] [31mFAILED[0m[31m [ 72%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-True-True] [31mFAILED[0m[31m [ 75%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-False-False] [31mFAILED[0m[31m [ 77%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-True-True-True] [31mFAILED[0m[31m [ 79%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-True-True-False] [31mFAILED[0m[31m [ 81%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-False-True-False] [31mFAILED[0m[31m [ 83%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-False-True-True] [31mFAILED[0m[31m [ 85%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[False-False-True] [31mFAILED[0m[31m [ 87%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-True-False-False] [31mFAILED[0m[31m [ 89%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-False-True] [31mFAILED[0m[31m [ 91%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-True-False-False] [31mFAILED[0m[31m [ 93%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[False-True-False-True] [31mFAILED[0m[31m [ 95%][0m unit/ops/aio/test_aio.py::TestWrite::test_async_write[True-False-False-False] [31mFAILED[0m[31m [ 97%][0m unit/ops/aio/test_aio.py::TestWrite::test_parallel_write[True-True-False] [31mFAILED[0m[31m [100%][0m =================================== FAILURES =================================== [31m[1m_________________ TestRead.test_parallel_read[True-False-True] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:36,901] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:36.903471 499200 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:36.903486 499329 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:36.903791 499200 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-1: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 94, in test_parallel_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_async_read[True-False-True-True] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:40,395] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:40.397357 499471 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:40.397351 499342 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:40.397672 499342 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-2: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_parallel_read[True-False-False] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:43,927] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:43.929081 499613 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:43.929071 499484 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:43.929389 499484 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-3: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 94, in test_parallel_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_________________ TestRead.test_parallel_read[False-True-True] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:47,446] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:47.447726 499755 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:47.447724 499626 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:47.448045 499626 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-4: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 94, in test_parallel_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[True-True-False-False] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:51,018] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:51.020327 499897 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:51.020316 499768 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:51.020874 499768 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-5: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_async_read[True-True-True-True] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:54,474] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:54.476365 500039 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:54.476356 499910 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:54.476680 499910 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-6: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[True-False-False-False] _______________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:28:58,007] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:28:58.008978 500181 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:28:58.008970 500052 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:28:58.009289 500052 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-7: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[False-False-False-True] _______________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:01,519] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:01.520992 500323 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:01.520982 500194 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:01.521299 500194 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-8: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_parallel_read[False-True-False] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:05,112] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:05.114812 500465 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:05.114802 500336 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:05.115303 500336 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-9: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 94, in test_parallel_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[True-False-False-True] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:08,644] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:08.645887 500607 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:08.645877 500478 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:08.646190 500478 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-10: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[False-True-False-True] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:12,184] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:12.186090 500620 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:12.186107 500749 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:12.186399 500620 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-11: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_async_read[False-True-True-True] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:15,727] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:15.729334 500891 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:15.729334 500762 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:15.729833 500762 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-12: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m______________ TestRead.test_async_read[False-False-False-False] _______________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:19,208] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:19.211542 501033 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:19.211529 500904 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:19.212108 500904 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-13: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[False-True-True-False] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:22,765] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:22.767371 501175 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:22.767364 501046 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:22.767691 501046 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-14: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[False-False-True-False] _______________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:26,297] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:26.300590 501317 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:26.300571 501188 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:26.301123 501188 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-15: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_parallel_read[False-False-True] _________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:29,838] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:29.840524 501459 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:29.840524 501330 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:29.841032 501330 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-16: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 94, in test_parallel_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_______________ TestRead.test_async_read[False-False-True-True] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:33,358] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:33.359935 501601 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:33.359926 501472 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:33.360242 501472 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-17: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_parallel_read[False-False-False] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:36,982] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:36.984362 501743 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:36.984362 501614 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:36.984858 501614 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-18: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 94, in test_parallel_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m_________________ TestRead.test_parallel_read[True-True-True] __________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:40,541] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:40.548139 501885 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:40.548128 501756 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:40.548662 501756 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-19: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 94, in test_parallel_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 670, in _load_unlocked File "", line 583, in module_from_spec File "", line 1043, in create_module File "", line 219, in _call_with_frames_removed ImportError: /usr/local/lib/python3.7/site-packages/deepspeed/ops/aio/async_io_op.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __kmpc_for_static_fini [31m[1m________________ TestRead.test_async_read[True-True-False-True] ________________[0m Worker [94m0[39;49;00m exited [94mwith[39;49;00m code [94m1[39;49;00m[90m[39;49;00m ----------------------------- Captured stdout call ----------------------------- [2023-05-27 03:29:44,113] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl ----------------------------- Captured stderr call ----------------------------- WARNING: Logging before InitGoogleLogging() is written to STDERR I0527 03:29:44.119480 502027 ProcessGroupNCCL.cpp:601] [Rank 0] NCCL watchdog thread started! I0527 03:29:44.119482 501898 ProcessGroupNCCL.cpp:500] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 NCCL_DEBUG: UNSET I0527 03:29:44.120364 501898 ProcessGroupNCCL.cpp:1669] Rank 0 using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. Process Process-20: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 206, in _dist_init raise e File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 201, in _dist_init self.run(**self._fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/common.py", line 331, in run self._current_test(**fixture_kwargs) File "/home/aishsh/ds-v0.9.2/tests/unit/ops/aio/test_aio.py", line 119, in test_async_read h = AsyncIOBuilder().load().aio_handle(BLOCK_SIZE, QUEUE_DEPTH, single_submit, overlap_events, IO_PARALLEL) File "/usr/local/lib/python3.7/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load return importlib.import_module(self.absolute_name()) File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "