Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • Y yolov5_pytorch
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • CI/CD
    • Repository
    • Value stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • ModelZoo
  • yolov5_pytorch
  • Issues
  • #1

Closed
Open
Created May 18, 2023 by huhe_cnic@huhe_cnic

单节点单卡训练报错

使用提供的镜像,构建singularity沙盒,在计算节点上运行沙盒,执行单机单卡训练任务,报错如下

warnings.warn( WARNING:torch.distributed.run:***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.


Traceback (most recent call last): File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 193, in main() File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/usr/local/lib/python3.9/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/usr/local/lib/python3.9/site-packages/torch/distributed/run.py", line 689, in run elastic_launch( File "/usr/local/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/usr/local/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 252, in launch_agent result = agent.run() File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper result = f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 709, in run result = self._invoke_run(role) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 837, in _invoke_run self._initialize_workers(self._worker_group) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper result = f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 678, in _initialize_workers self._rendezvous(worker_group) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper result = f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/agent/server/api.py", line 538, in _rendezvous store, group_rank, group_world_size = spec.rdzv_handler.next_rendezvous() File "/usr/local/lib/python3.9/site-packages/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line 55, in next_rendezvous self._store = TCPStore( # type: ignore[call-arg] ValueError: host not found: Name or service not known

Assignee
Assign to
Time tracking