单节点单卡训练报错
使用提供的镜像,构建singularity沙盒,在计算节点上运行沙盒,执行单机单卡训练任务,报错如下
warnings.warn( WARNING:torch.distributed.run:***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
Traceback (most recent call last): File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 193, in main() File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/usr/local/lib/python3.9/site-packages/torch/distributed/run.py", line 689, in run elastic_launch( File "/usr/local/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/usr/local/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 252, in launch_agent result = agent.run() File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper result = f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 709, in run result = self._invoke_run(role) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 837, in _invoke_run self._initialize_workers(self._worker_group) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper result = f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 678, in _initialize_workers self._rendezvous(worker_group) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper result = f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 538, in _rendezvous store, group_rank, group_world_size = spec.rdzv_handler.next_rendezvous() File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line 55, in next_rendezvous self._store = TCPStore( # type: ignore[call-arg] ValueError: host not found: Name or service not known