Merge branch 'v0.9.2-dev-cus' into 'v0.9.2-dev'

解决custom allreduce在dp情况下的其服务错误问题

See merge request dcutoolkit/deeplearing/vllm!348

vllm/distributed/device_communicators/custom_all_reduce.py

@@ -121,7 +121,11 @@ class CustomAllreduce:
         else:
             device_ids = list(range(cuda_device_count_stateless()))
 
-        physical_device_id = device_ids[device.index]
+        if (world_size == len(device_ids)):
+            physical_device_id = device_ids[device.index % world_size]
+        else:
+            physical_device_id = device_ids[device.index]
+            
         tensor = torch.tensor([physical_device_id],
                               dtype=torch.int,
                               device="cpu")