Unverified Commit 9ba28043 authored by Rui Qiao's avatar Rui Qiao Committed by GitHub
Browse files

[misc] Show driver IP info when Ray fails to allocate driver worker (#13858)


Signed-off-by: default avatarRui Qiao <ruisearch42@gmail.com>
parent 24679788
......@@ -229,9 +229,10 @@ class RayDistributedExecutor(DistributedExecutorBase):
logger.debug("driver_dummy_worker: %s", self.driver_dummy_worker)
if not self.use_ray_spmd_worker and self.driver_dummy_worker is None:
raise ValueError(
"Ray does not allocate any GPUs on the driver node. Consider "
"adjusting the Ray placement group or running the driver on a "
"GPU node.")
"Ray does not allocate any GPUs on the driver node."
f"Driver IP: {driver_ip}, worker IPs: {worker_ips}."
"Consider adjusting the Ray placement group or running "
"the driver on a GPU node.")
ip_counts: Dict[str, int] = {}
for ip in worker_ips:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment