-------------------------------------------------------------------------- WARNING: No preset parameters were found for the device that Open MPI detected: Local host: master Device name: mlx5_0 Device vendor ID: 0x02c9 Device vendor part ID: 4125 Default device parameters will be used, which may result in lower performance. You can edit any of the files specified by the btl_openib_device_param_files MCA parameter to set values for your device. NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_no_device_params_found to 0. -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There are more than one active ports on host 'master', but the default subnet GID prefix was detected on more than one of these ports. If these ports are connected to different physical IB networks, this configuration will fail in Open MPI. This version of Open MPI requires that every physically separate IB subnet that is used between connected MPI processes must have different subnet ID values. Please see this FAQ entry for more details: http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid NOTE: You can turn off this warning by setting the MCA parameter btl_openib_warn_default_gid_prefix to 0. -------------------------------------------------------------------------- -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: master Local device: mlx5_6 Local port: 1 CPCs attempted: udcm -------------------------------------------------------------------------- # nThreads: 1 nGpus: 1 nRanks: 1 minBytes: 7618 maxBytes: 1073741824 step: 2(factor) warmupIters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # # Using devices # Rank 0 Pid 6997 on master device 0 [0000:9f:00.0] BW200 # Rank 1 Pid 6998 on master device 1 [0000:56:00.0] BW200 # Rank 2 Pid 6999 on master device 2 [0000:5d:00.0] BW200 # Rank 3 Pid 7000 on master device 3 [0000:05:00.0] BW200 # Rank 4 Pid 7001 on master device 4 [0000:e8:00.0] BW200 # Rank 5 Pid 7002 on master device 5 [0000:c1:00.0] BW200 # Rank 6 Pid 7003 on master device 6 [0000:ca:00.0] BW200 # Rank 7 Pid 7004 on master device 7 [0000:b1:00.0] BW200 # Rank 8 Pid 6588 on node1 device 0 [0000:9f:00.0] BW200 # Rank 9 Pid 6589 on node1 device 1 [0000:56:00.0] BW200 # Rank 10 Pid 6590 on node1 device 2 [0000:5d:00.0] BW200 # Rank 11 Pid 6591 on node1 device 3 [0000:05:00.0] BW200 # Rank 12 Pid 6592 on node1 device 4 [0000:e8:00.0] BW200 # Rank 13 Pid 6593 on node1 device 5 [0000:c1:00.0] BW200 # Rank 14 Pid 6594 on node1 device 6 [0000:ca:00.0] BW200 # Rank 15 Pid 6595 on node1 device 7 [0000:b1:00.0] BW200 master:6997:6997 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:6997:6997 [0] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:6997:6997 [0] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:6997:6997 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:6997:6997 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:6997:6997 [0] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:6997:6997 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:6997:6997 [0] NCCL INFO ROCr version 1.1 master:6997:6997 [0] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 RCCL version 2.18.3+hip6.1 HEAD:037e9b3 node1:6589:6589 [1] NCCL INFO ROCr version 1.1 node1:6589:6589 [1] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 node1:6589:6589 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7002:7002 [5] NCCL INFO ROCr version 1.1 master:7002:7002 [5] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:6998:6998 [1] NCCL INFO ROCr version 1.1 master:6998:6998 [1] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:7002:7002 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6593:6593 [5] NCCL INFO ROCr version 1.1 node1:6593:6593 [5] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:6998:6998 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6593:6593 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6589:6589 [1] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> node1:6593:6593 [5] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> node1:6589:6589 [1] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6589:6589 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6593:6593 [5] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6593:6593 [5] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6589:6589 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6589:6589 [1] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6593:6593 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6593:6593 [5] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6593:6593 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6589:6589 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:7002:7002 [5] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:6998:6998 [1] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:6998:6998 [1] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:6998:6998 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:7002:7002 [5] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:7002:7002 [5] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:6998:6998 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:6998:6998 [1] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:7002:7002 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:7002:7002 [5] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:6998:6998 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:7002:7002 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6592:6592 [4] NCCL INFO ROCr version 1.1 node1:6592:6592 [4] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 node1:6592:6592 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6592:6592 [4] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> master:7004:7004 [7] NCCL INFO ROCr version 1.1 master:7004:7004 [7] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:7004:7004 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6592:6592 [4] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6592:6592 [4] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6591:6591 [3] NCCL INFO ROCr version 1.1 node1:6591:6591 [3] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 node1:6592:6592 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6592:6592 [4] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6592:6592 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6591:6591 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6595:6595 [7] NCCL INFO ROCr version 1.1 node1:6595:6595 [7] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:7001:7001 [4] NCCL INFO ROCr version 1.1 master:7001:7001 [4] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 node1:6595:6595 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7001:7001 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6588:6588 [0] NCCL INFO ROCr version 1.1 node1:6588:6588 [0] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 node1:6588:6588 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7003:7003 [6] NCCL INFO ROCr version 1.1 master:7003:7003 [6] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:6999:6999 [2] NCCL INFO ROCr version 1.1 master:6999:6999 [2] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:7003:7003 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:6999:6999 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6590:6590 [2] NCCL INFO ROCr version 1.1 node1:6590:6590 [2] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 node1:6590:6590 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6594:6594 [6] NCCL INFO ROCr version 1.1 node1:6594:6594 [6] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 node1:6594:6594 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:6997:7038 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. master:6997:7038 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7000:7000 [3] NCCL INFO ROCr version 1.1 master:7000:7000 [3] NCCL INFO Dmabuf feature disabled without NCCL_ENABLE_DMABUF_SUPPORT=1 master:7000:7000 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6595:6595 [7] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> node1:6591:6591 [3] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> node1:6594:6594 [6] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> node1:6588:6588 [0] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> node1:6590:6590 [2] NCCL INFO Bootstrap : Using p14p2:10.17.26.196<0> node1:6591:6591 [3] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6591:6591 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6595:6595 [7] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6595:6595 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6595:6595 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6590:6590 [2] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6590:6590 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6594:6594 [6] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6594:6594 [6] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6594:6594 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6594:6594 [6] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6588:6588 [0] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory node1:6588:6588 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation node1:6591:6591 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6591:6591 [3] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6595:6595 [7] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6595:6595 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6591:6591 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6590:6590 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6590:6590 [2] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6590:6590 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6594:6594 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6588:6588 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" node1:6588:6588 [0] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 node1:6588:6588 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:7004:7004 [7] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:6999:6999 [2] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:7001:7001 [4] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:7004:7004 [7] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:7004:7004 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:7004:7004 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:7003:7003 [6] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:6997:7038 [0] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7000:7000 [3] NCCL INFO Bootstrap : Using p14p2:10.17.26.195<0> master:7004:7004 [7] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:7004:7004 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:6999:6999 [2] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:6999:6999 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:7000:7000 [3] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:7000:7000 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:7003:7003 [6] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:7003:7003 [6] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:7001:7001 [4] NCCL INFO NET/Plugin : Plugin load (librccl-net.so) returned 2 : librccl-net.so: cannot open shared object file: No such file or directory master:7001:7001 [4] NCCL INFO NET/Plugin : No plugin found, using internal implementation master:6999:6999 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:7001:7001 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:7001:7001 [4] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:7001:7001 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:7003:7003 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:7003:7003 [6] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:6999:6999 [2] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:7003:7003 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:7000:7000 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:121 NCCL WARN NUMA auto balancing enabled which can lead to variability in the RCCL performance! Disable by "sudo sysctl kernel.numa_balancing=0" master:7000:7000 [3] NCCL INFO Kernel version: 4.19.90-89.11.v2401.ky10.x86_64 master:6999:6999 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! master:7000:7000 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:142 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang! node1:6589:6629 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6589:6629 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:6998:7039 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. master:6998:7039 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6589:6629 [1] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:6998:7039 [1] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 node1:6593:6628 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 0. master:7002:7040 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6593:6628 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7002:7040 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6593:6628 [5] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7002:7040 [5] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7000:7044 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. master:7000:7044 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:6999:7045 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. master:6999:7045 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6588:6635 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6591:6631 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6588:6635 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6590:6634 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6591:6631 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6590:6634 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7000:7044 [3] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:6999:7045 [2] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 node1:6592:6630 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6592:6630 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7004:7041 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 0. master:7003:7043 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6588:6635 [0] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7004:7041 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6591:6631 [3] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 node1:6590:6634 [2] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 node1:6592:6630 [4] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7003:7043 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6594:6632 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 0. master:7001:7042 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6594:6632 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 master:7001:7042 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6594:6632 [6] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 node1:6595:6633 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 0. node1:6595:6633 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to p14p2 node1:6595:6633 [7] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7004:7041 [7] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7003:7043 [6] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:7001:7042 [4] NCCL INFO NCCL_IB_HCA set to mlx5_3:1,mlx5_4:1 master:6997:7038 [0] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:6997:7038 [0] NCCL INFO Using network IB master:6998:7039 [1] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:6998:7039 [1] NCCL INFO Using network IB node1:6589:6629 [1] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6589:6629 [1] NCCL INFO Using network IB master:6999:7045 [2] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:6999:7045 [2] NCCL INFO Using network IB master:7001:7042 [4] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:7001:7042 [4] NCCL INFO Using network IB master:7003:7043 [6] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:7003:7043 [6] NCCL INFO Using network IB master:7000:7044 [3] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:7000:7044 [3] NCCL INFO Using network IB master:7002:7040 [5] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:7002:7040 [5] NCCL INFO Using network IB node1:6592:6630 [4] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6592:6630 [4] NCCL INFO Using network IB node1:6588:6635 [0] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6588:6635 [0] NCCL INFO Using network IB master:7004:7041 [7] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.195<0> master:7004:7041 [7] NCCL INFO Using network IB node1:6590:6634 [2] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6590:6634 [2] NCCL INFO Using network IB node1:6594:6632 [6] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6594:6632 [6] NCCL INFO Using network IB node1:6591:6631 [3] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6591:6631 [3] NCCL INFO Using network IB node1:6593:6628 [5] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6593:6628 [5] NCCL INFO Using network IB node1:6595:6633 [7] NCCL INFO NET/IB : Using [0]mlx5_3:1/IB [1]mlx5_4:1/IB [RO]; OOB p14p2:10.17.26.196<0> node1:6595:6633 [7] NCCL INFO Using network IB master:6997:7038 [0] NCCL INFO comm 0x2c7f290 rank 0 nranks 16 cudaDev 0 busId 9f000 commId 0x957f9fef67ebf659 - Init START master:6999:7045 [2] NCCL INFO comm 0x3bda8f0 rank 2 nranks 16 cudaDev 2 busId 5d000 commId 0x957f9fef67ebf659 - Init START master:6998:7039 [1] NCCL INFO comm 0x2a6c650 rank 1 nranks 16 cudaDev 1 busId 56000 commId 0x957f9fef67ebf659 - Init START master:7004:7041 [7] NCCL INFO comm 0x3aab000 rank 7 nranks 16 cudaDev 7 busId b1000 commId 0x957f9fef67ebf659 - Init START master:7002:7040 [5] NCCL INFO comm 0x39bdf90 rank 5 nranks 16 cudaDev 5 busId c1000 commId 0x957f9fef67ebf659 - Init START master:7000:7044 [3] NCCL INFO comm 0x2a2d870 rank 3 nranks 16 cudaDev 3 busId 5000 commId 0x957f9fef67ebf659 - Init START master:7003:7043 [6] NCCL INFO comm 0x3ef2cf0 rank 6 nranks 16 cudaDev 6 busId ca000 commId 0x957f9fef67ebf659 - Init START master:7001:7042 [4] NCCL INFO comm 0x205cf60 rank 4 nranks 16 cudaDev 4 busId e8000 commId 0x957f9fef67ebf659 - Init START node1:6590:6634 [2] NCCL INFO comm 0x231fa00 rank 10 nranks 16 cudaDev 2 busId 5d000 commId 0x957f9fef67ebf659 - Init START node1:6592:6630 [4] NCCL INFO comm 0x20724e0 rank 12 nranks 16 cudaDev 4 busId e8000 commId 0x957f9fef67ebf659 - Init START node1:6594:6632 [6] NCCL INFO comm 0x2537600 rank 14 nranks 16 cudaDev 6 busId ca000 commId 0x957f9fef67ebf659 - Init START node1:6591:6631 [3] NCCL INFO comm 0x3e87c90 rank 11 nranks 16 cudaDev 3 busId 5000 commId 0x957f9fef67ebf659 - Init START master:6997:7038 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6593:6628 [5] NCCL INFO comm 0x20c72e0 rank 13 nranks 16 cudaDev 5 busId c1000 commId 0x957f9fef67ebf659 - Init START node1:6589:6629 [1] NCCL INFO comm 0x2f865c0 rank 9 nranks 16 cudaDev 1 busId 56000 commId 0x957f9fef67ebf659 - Init START node1:6595:6633 [7] NCCL INFO comm 0x2ffe760 rank 15 nranks 16 cudaDev 7 busId b1000 commId 0x957f9fef67ebf659 - Init START node1:6588:6635 [0] NCCL INFO comm 0x36c8820 rank 8 nranks 16 cudaDev 0 busId 9f000 commId 0x957f9fef67ebf659 - Init START node1:6589:6629 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6593:6628 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6590:6634 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6591:6631 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6588:6635 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6592:6630 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6595:6633 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 node1:6594:6632 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:7000:7044 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:7003:7043 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:7004:7041 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:6999:7045 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:7002:7040 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:7001:7042 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:6998:7039 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:323 -> 2 master:6999:7045 [2] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml master:6998:7039 [1] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml master:7000:7044 [3] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml master:7003:7043 [6] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml master:6997:7038 [0] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml master:7004:7041 [7] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml master:7002:7040 [5] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml master:7001:7042 [4] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6590:6634 [2] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6594:6632 [6] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6595:6633 [7] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6592:6630 [4] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6591:6631 [3] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6593:6628 [5] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6588:6635 [0] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6589:6629 [1] NCCL INFO NCCL_TOPO_FILE set by environment to ./topo-1.xml node1:6589:6629 [1] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:6999:7045 [2] NCCL INFO rocm_smi_lib: version 2.8.0.0 node1:6589:6629 [1] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6589:6629 [1] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6589:6629 [1] NCCL INFO Setting affinity for GPU 1 to ffff0000 master:6999:7045 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:6999:7045 [2] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. master:6997:7038 [0] NCCL INFO rocm_smi_lib: version 2.8.0.0 node1:6594:6632 [6] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:6998:7039 [1] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:7004:7041 [7] NCCL INFO rocm_smi_lib: version 2.8.0.0 node1:6590:6634 [2] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:7003:7043 [6] NCCL INFO rocm_smi_lib: version 2.8.0.0 node1:6592:6630 [4] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:7000:7044 [3] NCCL INFO rocm_smi_lib: version 2.8.0.0 node1:6595:6633 [7] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:6997:7038 [0] NCCL INFO NCCL_TOPO_DUMP_FILE set by environment to ./topo.xml master:7002:7040 [5] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:7001:7042 [4] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:6997:7038 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:6997:7038 [0] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6593:6628 [5] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:6998:7039 [1] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:6998:7039 [1] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6588:6635 [0] NCCL INFO rocm_smi_lib: version 2.8.0.0 master:6998:7039 [1] NCCL INFO Setting affinity for GPU 1 to ffff0000 master:7004:7041 [7] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:7004:7041 [7] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6591:6631 [3] NCCL INFO rocm_smi_lib: version 2.8.0.0 node1:6594:6632 [6] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6594:6632 [6] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. master:7003:7043 [6] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:7003:7043 [6] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. master:7000:7044 [3] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:7000:7044 [3] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6590:6634 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6590:6634 [2] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. master:7002:7040 [5] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:7002:7040 [5] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. master:7001:7042 [4] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS master:7001:7042 [4] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6592:6630 [4] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6592:6630 [4] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. master:7002:7040 [5] NCCL INFO Setting affinity for GPU 5 to ffff0000,00000000,00000000 node1:6595:6633 [7] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6595:6633 [7] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6593:6628 [5] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6593:6628 [5] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6588:6635 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6588:6635 [0] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6591:6631 [3] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to SYS node1:6591:6631 [3] NCCL INFO NCCL_NET_GDR_READ set by environment to 0. node1:6593:6628 [5] NCCL INFO Setting affinity for GPU 5 to ffff0000,00000000,00000000 master:7002:7040 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 5 env param disable sdma copy master:7002:7040 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 4/-1/-1->5->6 [2] 6/-1/-1->5->4 [3] 4/-1/-1->5->6 [4] 6/-1/-1->5->4 [5] 4/-1/-1->5->6 [6] 6/-1/-1->5->4 [7] 4/-1/-1->5->6 [8] 6/-1/-1->5->4 [9] 4/-1/-1->5->6 [10] 6/-1/-1->5->4 [11] 4/-1/-1->5->6 [12] 6/-1/-1->5->4 [13] 4/-1/-1->5->6 [14] 6/-1/-1->5->4 [15] 4/-1/-1->5->6 [16] 6/-1/-1->5->4 [17] 4/-1/-1->5->6 [18] 6/-1/-1->5->4 [19] 4/-1/-1->5->6 [20] 6/-1/-1->5->4 [21] 4/-1/-1->5->6 [22] 6/-1/-1->5->4 [23] 4/-1/-1->5->6 [24] 6/-1/-1->5->4 [25] 4/-1/-1->5->6 [26] 6/-1/-1->5->4 [27] 4/-1/-1->5->6 [28] 6/-1/-1->5->4 [29] 4/-1/-1->5->6 [30] 6/-1/-1->5->4 [31] 4/-1/-1->5->6 comm 0x39bdf90 nRanks 16 busId c1000 master:7002:7040 [5] NCCL INFO -hcugon- create sdma group queue rank:5 localRanks:8 nRanks:16 sdma copy is not enabled master:7002:7040 [5] NCCL INFO P2P Chunksize set to 131072 master:7004:7041 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 7 env param disable sdma copy master:7003:7043 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 6 env param disable sdma copy master:7003:7043 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 5/-1/-1->6->7 [2] 7/-1/-1->6->5 [3] 5/-1/-1->6->7 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 7/-1/-1->6->5 [7] 5/-1/-1->6->7 [8] 7/-1/-1->6->5 [9] 5/-1/-1->6->7 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7 [12] 7/-1/-1->6->5 [13] 5/-1/-1->6->7 [14] 7/-1/-1->6->5 [15] 5/-1/-1->6->7 [16] 7/-1/-1->6->5 [17] 5/-1/-1->6->7 [18] 7/-1/-1->6->5 [19] 5/-1/-1->6->7 [20] 7/-1/-1->6->5 [21] 5/-1/-1->6->7 [22] 7/-1/-1->6->5 [23] 5/-1/-1->6->7 [24] 7/-1/-1->6->5 [25] 5/-1/-1->6->7 [26] 7/-1/-1->6->5 [27] 5/-1/-1->6->7 [28] 7/-1/-1->6->5 [29] 5/-1/-1->6->7 [30] 7/-1/-1->6->5 [31] 5/-1/-1->6->7 comm 0x3ef2cf0 nRanks 16 busId ca000 master:7003:7043 [6] NCCL INFO -hcugon- create sdma group queue rank:6 localRanks:8 nRanks:16 sdma copy is not enabled master:7003:7043 [6] NCCL INFO P2P Chunksize set to 131072 master:7001:7042 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 4 env param disable sdma copy master:6999:7045 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 2 env param disable sdma copy master:7004:7041 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 6/-1/-1->7->0 [2] -1/-1/-1->7->6 [3] 6/-1/-1->7->0 [4] -1/-1/-1->7->6 [5] 6/-1/-1->7->0 [6] -1/-1/-1->7->6 [7] 6/-1/-1->7->0 [8] -1/-1/-1->7->6 [9] 6/-1/-1->7->0 [10] -1/-1/-1->7->6 [11] 6/-1/-1->7->0 [12] -1/-1/-1->7->6 [13] 6/-1/-1->7->0 [14] -1/-1/-1->7->6 [15] 6/-1/-1->7->0 [16] -1/-1/-1->7->6 [17] 6/-1/-1->7->0 [18] -1/-1/-1->7->6 [19] 6/-1/-1->7->0 [20] -1/-1/-1->7->6 [21] 6/-1/-1->7->0 [22] -1/-1/-1->7->6 [23] 6/-1/-1->7->0 [24] -1/-1/-1->7->6 [25] 6/-1/-1->7->0 [26] -1/-1/-1->7->6 [27] 6/-1/-1->7->0 [28] -1/-1/-1->7->6 [29] 6/-1/-1->7->0 [30] -1/-1/-1->7->6 [31] 6/-1/-1->7->0 comm 0x3aab000 nRanks 16 busId b1000 master:7004:7041 [7] NCCL INFO -hcugon- create sdma group queue rank:7 localRanks:8 nRanks:16 sdma copy is not enabled master:7004:7041 [7] NCCL INFO P2P Chunksize set to 131072 master:7000:7044 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 3 env param disable sdma copy master:7000:7044 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 2/11/-1->3->-1 [2] 4/-1/-1->3->2 [3] 2/-1/-1->3->11 [4] 4/-1/-1->3->2 [5] 2/11/-1->3->-1 [6] 4/-1/-1->3->2 [7] 2/-1/-1->3->11 [8] 4/-1/-1->3->2 [9] 2/11/-1->3->-1 [10] 4/-1/-1->3->2 [11] 2/-1/-1->3->11 [12] 4/-1/-1->3->2 [13] 2/11/-1->3->-1 [14] 4/-1/-1->3->2 [15] 2/-1/-1->3->11 [16] 4/-1/-1->3->2 [17] 2/11/-1->3->-1 [18] 4/-1/-1->3->2 [19] 2/-1/-1->3->11 [20] 4/-1/-1->3->2 [21] 2/11/-1->3->-1 [22] 4/-1/-1->3->2 [23] 2/-1/-1->3->11 [24] 4/-1/-1->3->2 [25] 2/11/-1->3->-1 [26] 4/-1/-1->3->2 [27] 2/-1/-1->3->11 [28] 4/-1/-1->3->2 [29] 2/11/-1->3->-1 [30] 4/-1/-1->3->2 [31] 2/-1/-1->3->11 comm 0x2a2d870 nRanks 16 busId 5000 master:7000:7044 [3] NCCL INFO -hcugon- create sdma group queue rank:3 localRanks:8 nRanks:16 sdma copy is not enabled master:7000:7044 [3] NCCL INFO P2P Chunksize set to 131072 master:7001:7042 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] -1/-1/-1->4->5 [2] 5/-1/-1->4->3 [3] -1/-1/-1->4->5 [4] 5/-1/-1->4->3 [5] -1/-1/-1->4->5 [6] 5/-1/-1->4->3 [7] -1/-1/-1->4->5 [8] 5/-1/-1->4->3 [9] -1/-1/-1->4->5 [10] 5/-1/-1->4->3 [11] -1/-1/-1->4->5 [12] 5/-1/-1->4->3 [13] -1/-1/-1->4->5 [14] 5/-1/-1->4->3 [15] -1/-1/-1->4->5 [16] 5/-1/-1->4->3 [17] -1/-1/-1->4->5 [18] 5/-1/-1->4->3 [19] -1/-1/-1->4->5 [20] 5/-1/-1->4->3 [21] -1/-1/-1->4->5 [22] 5/-1/-1->4->3 [23] -1/-1/-1->4->5 [24] 5/-1/-1->4->3 [25] -1/-1/-1->4->5 [26] 5/-1/-1->4->3 [27] -1/-1/-1->4->5 [28] 5/-1/-1->4->3 [29] -1/-1/-1->4->5 [30] 5/-1/-1->4->3 [31] -1/-1/-1->4->5 comm 0x205cf60 nRanks 16 busId e8000 master:7001:7042 [4] NCCL INFO -hcugon- create sdma group queue rank:4 localRanks:8 nRanks:16 sdma copy is not enabled master:7001:7042 [4] NCCL INFO P2P Chunksize set to 131072 master:6999:7045 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 1/-1/-1->2->3 [2] 3/-1/-1->2->1 [3] 1/-1/-1->2->3 [4] 3/-1/-1->2->1 [5] 1/-1/-1->2->3 [6] 3/-1/-1->2->1 [7] 1/-1/-1->2->3 [8] 3/-1/-1->2->1 [9] 1/-1/-1->2->3 [10] 3/-1/-1->2->1 [11] 1/-1/-1->2->3 [12] 3/-1/-1->2->1 [13] 1/-1/-1->2->3 [14] 3/-1/-1->2->1 [15] 1/-1/-1->2->3 [16] 3/-1/-1->2->1 [17] 1/-1/-1->2->3 [18] 3/-1/-1->2->1 [19] 1/-1/-1->2->3 [20] 3/-1/-1->2->1 [21] 1/-1/-1->2->3 [22] 3/-1/-1->2->1 [23] 1/-1/-1->2->3 [24] 3/-1/-1->2->1 [25] 1/-1/-1->2->3 [26] 3/-1/-1->2->1 [27] 1/-1/-1->2->3 [28] 3/-1/-1->2->1 [29] 1/-1/-1->2->3 [30] 3/-1/-1->2->1 [31] 1/-1/-1->2->3 comm 0x3bda8f0 nRanks 16 busId 5d000 master:6999:7045 [2] NCCL INFO -hcugon- create sdma group queue rank:2 localRanks:8 nRanks:16 sdma copy is not enabled master:6999:7045 [2] NCCL INFO P2P Chunksize set to 131072 master:6997:7038 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 0 env param disable sdma copy master:6997:7038 [0] NCCL INFO Channel 00/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 01/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 02/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6998:7039 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 1 env param disable sdma copy master:6998:7039 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 0/-1/-1->1->2 [2] 2/-1/-1->1->0 [3] 0/-1/-1->1->2 [4] 2/-1/-1->1->0 [5] 0/-1/-1->1->2 [6] 2/-1/-1->1->0 [7] 0/-1/-1->1->2 [8] 2/-1/-1->1->0 [9] 0/-1/-1->1->2 [10] 2/-1/-1->1->0 [11] 0/-1/-1->1->2 [12] 2/-1/-1->1->0 [13] 0/-1/-1->1->2 [14] 2/-1/-1->1->0 [15] 0/-1/-1->1->2 [16] 2/-1/-1->1->0 [17] 0/-1/-1->1->2 [18] 2/-1/-1->1->0 [19] 0/-1/-1->1->2 [20] 2/-1/-1->1->0 [21] 0/-1/-1->1->2 [22] 2/-1/-1->1->0 [23] 0/-1/-1->1->2 [24] 2/-1/-1->1->0 [25] 0/-1/-1->1->2 [26] 2/-1/-1->1->0 [27] 0/-1/-1->1->2 [28] 2/-1/-1->1->0 [29] 0/-1/-1->1->2 [30] 2/-1/-1->1->0 [31] 0/-1/-1->1->2 comm 0x2a6c650 nRanks 16 busId 56000 master:6998:7039 [1] NCCL INFO -hcugon- create sdma group queue rank:1 localRanks:8 nRanks:16 sdma copy is not enabled master:6998:7039 [1] NCCL INFO P2P Chunksize set to 131072 master:6997:7038 [0] NCCL INFO Channel 03/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 04/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 05/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 06/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 07/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 08/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 09/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 10/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 11/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 node1:6594:6632 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 14 env param disable sdma copy node1:6594:6632 [6] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 13/-1/-1->14->15 [2] 15/-1/-1->14->13 [3] 13/-1/-1->14->15 [4] 15/-1/-1->14->13 [5] 13/-1/-1->14->15 [6] 15/-1/-1->14->13 [7] 13/-1/-1->14->15 [8] 15/-1/-1->14->13 [9] 13/-1/-1->14->15 [10] 15/-1/-1->14->13 [11] 13/-1/-1->14->15 [12] 15/-1/-1->14->13 [13] 13/-1/-1->14->15 [14] 15/-1/-1->14->13 [15] 13/-1/-1->14->15 [16] 15/-1/-1->14->13 [17] 13/-1/-1->14->15 [18] 15/-1/-1->14->13 [19] 13/-1/-1->14->15 [20] 15/-1/-1->14->13 [21] 13/-1/-1->14->15 [22] 15/-1/-1->14->13 [23] 13/-1/-1->14->15 [24] 15/-1/-1->14->13 [25] 13/-1/-1->14->15 [26] 15/-1/-1->14->13 [27] 13/-1/-1->14->15 [28] 15/-1/-1->14->13 [29] 13/-1/-1->14->15 [30] 15/-1/-1->14->13 [31] 13/-1/-1->14->15 comm 0x2537600 nRanks 16 busId ca000 node1:6594:6632 [6] NCCL INFO -hcugon- create sdma group queue rank:14 localRanks:8 nRanks:16 sdma copy is not enabled node1:6594:6632 [6] NCCL INFO P2P Chunksize set to 131072 master:6997:7038 [0] NCCL INFO Channel 12/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 13/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 node1:6591:6631 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 11 env param disable sdma copy node1:6591:6631 [3] NCCL INFO Trees [0] 12/-1/-1->11->10 [1] 10/-1/-1->11->3 [2] 12/-1/-1->11->10 [3] 10/3/-1->11->-1 [4] 12/-1/-1->11->10 [5] 10/-1/-1->11->3 [6] 12/-1/-1->11->10 [7] 10/3/-1->11->-1 [8] 12/-1/-1->11->10 [9] 10/-1/-1->11->3 [10] 12/-1/-1->11->10 [11] 10/3/-1->11->-1 [12] 12/-1/-1->11->10 [13] 10/-1/-1->11->3 [14] 12/-1/-1->11->10 [15] 10/3/-1->11->-1 [16] 12/-1/-1->11->10 [17] 10/-1/-1->11->3 [18] 12/-1/-1->11->10 [19] 10/3/-1->11->-1 [20] 12/-1/-1->11->10 [21] 10/-1/-1->11->3 [22] 12/-1/-1->11->10 [23] 10/3/-1->11->-1 [24] 12/-1/-1->11->10 [25] 10/-1/-1->11->3 [26] 12/-1/-1->11->10 [27] 10/3/-1->11->-1 [28] 12/-1/-1->11->10 [29] 10/-1/-1->11->3 [30] 12/-1/-1->11->10 [31] 10/3/-1->11->-1 comm 0x3e87c90 nRanks 16 busId 5000 node1:6591:6631 [3] NCCL INFO -hcugon- create sdma group queue rank:11 localRanks:8 nRanks:16 sdma copy is not enabled node1:6591:6631 [3] NCCL INFO P2P Chunksize set to 131072 node1:6592:6630 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 12 env param disable sdma copy node1:6592:6630 [4] NCCL INFO Trees [0] 13/-1/-1->12->11 [1] -1/-1/-1->12->13 [2] 13/-1/-1->12->11 [3] -1/-1/-1->12->13 [4] 13/-1/-1->12->11 [5] -1/-1/-1->12->13 [6] 13/-1/-1->12->11 [7] -1/-1/-1->12->13 [8] 13/-1/-1->12->11 [9] -1/-1/-1->12->13 [10] 13/-1/-1->12->11 [11] -1/-1/-1->12->13 [12] 13/-1/-1->12->11 [13] -1/-1/-1->12->13 [14] 13/-1/-1->12->11 [15] -1/-1/-1->12->13 [16] 13/-1/-1->12->11 [17] -1/-1/-1->12->13 [18] 13/-1/-1->12->11 [19] -1/-1/-1->12->13 [20] 13/-1/-1->12->11 [21] -1/-1/-1->12->13 [22] 13/-1/-1->12->11 [23] -1/-1/-1->12->13 [24] 13/-1/-1->12->11 [25] -1/-1/-1->12->13 [26] 13/-1/-1->12->11 [27] -1/-1/-1->12->13 [28] 13/-1/-1->12->11 [29] -1/-1/-1->12->13 [30] 13/-1/-1->12->11 [31] -1/-1/-1->12->13 comm 0x20724e0 nRanks 16 busId e8000 node1:6592:6630 [4] NCCL INFO -hcugon- create sdma group queue rank:12 localRanks:8 nRanks:16 sdma copy is not enabled node1:6592:6630 [4] NCCL INFO P2P Chunksize set to 131072 node1:6589:6629 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 9 env param disable sdma copy node1:6589:6629 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 8/-1/-1->9->10 [2] 10/-1/-1->9->8 [3] 8/-1/-1->9->10 [4] 10/-1/-1->9->8 [5] 8/-1/-1->9->10 [6] 10/-1/-1->9->8 [7] 8/-1/-1->9->10 [8] 10/-1/-1->9->8 [9] 8/-1/-1->9->10 [10] 10/-1/-1->9->8 [11] 8/-1/-1->9->10 [12] 10/-1/-1->9->8 [13] 8/-1/-1->9->10 [14] 10/-1/-1->9->8 [15] 8/-1/-1->9->10 [16] 10/-1/-1->9->8 [17] 8/-1/-1->9->10 [18] 10/-1/-1->9->8 [19] 8/-1/-1->9->10 [20] 10/-1/-1->9->8 [21] 8/-1/-1->9->10 [22] 10/-1/-1->9->8 [23] 8/-1/-1->9->10 [24] 10/-1/-1->9->8 [25] 8/-1/-1->9->10 [26] 10/-1/-1->9->8 [27] 8/-1/-1->9->10 [28] 10/-1/-1->9->8 [29] 8/-1/-1->9->10 [30] 10/-1/-1->9->8 [31] 8/-1/-1->9->10 comm 0x2f865c0 nRanks 16 busId 56000 node1:6589:6629 [1] NCCL INFO -hcugon- create sdma group queue rank:9 localRanks:8 nRanks:16 sdma copy is not enabled node1:6589:6629 [1] NCCL INFO P2P Chunksize set to 131072 node1:6588:6635 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 8 env param disable sdma copy node1:6588:6635 [0] NCCL INFO Trees [0] 9/-1/-1->8->0 [1] 15/-1/-1->8->9 [2] 9/0/-1->8->-1 [3] 15/-1/-1->8->9 [4] 9/-1/-1->8->0 [5] 15/-1/-1->8->9 [6] 9/0/-1->8->-1 [7] 15/-1/-1->8->9 [8] 9/-1/-1->8->0 [9] 15/-1/-1->8->9 [10] 9/0/-1->8->-1 [11] 15/-1/-1->8->9 [12] 9/-1/-1->8->0 [13] 15/-1/-1->8->9 [14] 9/0/-1->8->-1 [15] 15/-1/-1->8->9 [16] 9/-1/-1->8->0 [17] 15/-1/-1->8->9 [18] 9/0/-1->8->-1 [19] 15/-1/-1->8->9 [20] 9/-1/-1->8->0 [21] 15/-1/-1->8->9 [22] 9/0/-1->8->-1 [23] 15/-1/-1->8->9 [24] 9/-1/-1->8->0 [25] 15/-1/-1->8->9 [26] 9/0/-1->8->-1 [27] 15/-1/-1->8->9 [28] 9/-1/-1->8->0 [29] 15/-1/-1->8->9 [30] 9/0/-1->8->-1 [31] 15/-1/-1->8->9 comm 0x36c8820 nRanks 16 busId 9f000 node1:6588:6635 [0] NCCL INFO -hcugon- create sdma group queue rank:8 localRanks:8 nRanks:16 sdma copy is not enabled node1:6588:6635 [0] NCCL INFO P2P Chunksize set to 131072 master:6997:7038 [0] NCCL INFO Channel 14/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 15/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 16/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 17/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 18/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 19/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 20/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 21/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 22/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 23/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 24/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 25/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 26/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 27/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 28/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 29/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Channel 30/32 : 0 1 2 11 12 13 14 15 8 9 10 3 4 5 6 7 master:6997:7038 [0] NCCL INFO Channel 31/32 : 0 5 7 1 11 10 12 14 8 13 15 9 3 2 4 6 master:6997:7038 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] 7/-1/-1->0->1 [2] 1/-1/-1->0->8 [3] 7/-1/-1->0->1 [4] 1/8/-1->0->-1 [5] 7/-1/-1->0->1 [6] 1/-1/-1->0->8 [7] 7/-1/-1->0->1 [8] 1/8/-1->0->-1 [9] 7/-1/-1->0->1 [10] 1/-1/-1->0->8 [11] 7/-1/-1->0->1 [12] 1/8/-1->0->-1 [13] 7/-1/-1->0->1 [14] 1/-1/-1->0->8 [15] 7/-1/-1->0->1 [16] 1/8/-1->0->-1 [17] 7/-1/-1->0->1 [18] 1/-1/-1->0->8 [19] 7/-1/-1->0->1 [20] 1/8/-1->0->-1 [21] 7/-1/-1->0->1 [22] 1/-1/-1->0->8 [23] 7/-1/-1->0->1 [24] 1/8/-1->0->-1 [25] 7/-1/-1->0->1 [26] 1/-1/-1->0->8 [27] 7/-1/-1->0->1 [28] 1/8/-1->0->-1 [29] 7/-1/-1->0->1 [30] 1/-1/-1->0->8 [31] 7/-1/-1->0->1 comm 0x2c7f290 nRanks 16 busId 9f000 master:6997:7038 [0] NCCL INFO -hcugon- create sdma group queue rank:0 localRanks:8 nRanks:16 sdma copy is not enabled master:6997:7038 [0] NCCL INFO P2P Chunksize set to 131072 node1:6590:6634 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 10 env param disable sdma copy node1:6590:6634 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 9/-1/-1->10->11 [2] 11/-1/-1->10->9 [3] 9/-1/-1->10->11 [4] 11/-1/-1->10->9 [5] 9/-1/-1->10->11 [6] 11/-1/-1->10->9 [7] 9/-1/-1->10->11 [8] 11/-1/-1->10->9 [9] 9/-1/-1->10->11 [10] 11/-1/-1->10->9 [11] 9/-1/-1->10->11 [12] 11/-1/-1->10->9 [13] 9/-1/-1->10->11 [14] 11/-1/-1->10->9 [15] 9/-1/-1->10->11 [16] 11/-1/-1->10->9 [17] 9/-1/-1->10->11 [18] 11/-1/-1->10->9 [19] 9/-1/-1->10->11 [20] 11/-1/-1->10->9 [21] 9/-1/-1->10->11 [22] 11/-1/-1->10->9 [23] 9/-1/-1->10->11 [24] 11/-1/-1->10->9 [25] 9/-1/-1->10->11 [26] 11/-1/-1->10->9 [27] 9/-1/-1->10->11 [28] 11/-1/-1->10->9 [29] 9/-1/-1->10->11 [30] 11/-1/-1->10->9 [31] 9/-1/-1->10->11 comm 0x231fa00 nRanks 16 busId 5d000 node1:6590:6634 [2] NCCL INFO -hcugon- create sdma group queue rank:10 localRanks:8 nRanks:16 sdma copy is not enabled node1:6590:6634 [2] NCCL INFO P2P Chunksize set to 131072 node1:6595:6633 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 15 env param disable sdma copy node1:6595:6633 [7] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 14/-1/-1->15->8 [2] -1/-1/-1->15->14 [3] 14/-1/-1->15->8 [4] -1/-1/-1->15->14 [5] 14/-1/-1->15->8 [6] -1/-1/-1->15->14 [7] 14/-1/-1->15->8 [8] -1/-1/-1->15->14 [9] 14/-1/-1->15->8 [10] -1/-1/-1->15->14 [11] 14/-1/-1->15->8 [12] -1/-1/-1->15->14 [13] 14/-1/-1->15->8 [14] -1/-1/-1->15->14 [15] 14/-1/-1->15->8 [16] -1/-1/-1->15->14 [17] 14/-1/-1->15->8 [18] -1/-1/-1->15->14 [19] 14/-1/-1->15->8 [20] -1/-1/-1->15->14 [21] 14/-1/-1->15->8 [22] -1/-1/-1->15->14 [23] 14/-1/-1->15->8 [24] -1/-1/-1->15->14 [25] 14/-1/-1->15->8 [26] -1/-1/-1->15->14 [27] 14/-1/-1->15->8 [28] -1/-1/-1->15->14 [29] 14/-1/-1->15->8 [30] -1/-1/-1->15->14 [31] 14/-1/-1->15->8 comm 0x2ffe760 nRanks 16 busId b1000 node1:6595:6633 [7] NCCL INFO -hcugon- create sdma group queue rank:15 localRanks:8 nRanks:16 sdma copy is not enabled node1:6595:6633 [7] NCCL INFO P2P Chunksize set to 131072 node1:6593:6628 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/init.cc:1312 NCCL WARN -hcugon- rank 13 env param disable sdma copy node1:6593:6628 [5] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 12/-1/-1->13->14 [2] 14/-1/-1->13->12 [3] 12/-1/-1->13->14 [4] 14/-1/-1->13->12 [5] 12/-1/-1->13->14 [6] 14/-1/-1->13->12 [7] 12/-1/-1->13->14 [8] 14/-1/-1->13->12 [9] 12/-1/-1->13->14 [10] 14/-1/-1->13->12 [11] 12/-1/-1->13->14 [12] 14/-1/-1->13->12 [13] 12/-1/-1->13->14 [14] 14/-1/-1->13->12 [15] 12/-1/-1->13->14 [16] 14/-1/-1->13->12 [17] 12/-1/-1->13->14 [18] 14/-1/-1->13->12 [19] 12/-1/-1->13->14 [20] 14/-1/-1->13->12 [21] 12/-1/-1->13->14 [22] 14/-1/-1->13->12 [23] 12/-1/-1->13->14 [24] 14/-1/-1->13->12 [25] 12/-1/-1->13->14 [26] 14/-1/-1->13->12 [27] 12/-1/-1->13->14 [28] 14/-1/-1->13->12 [29] 12/-1/-1->13->14 [30] 14/-1/-1->13->12 [31] 12/-1/-1->13->14 comm 0x20c72e0 nRanks 16 busId c1000 node1:6593:6628 [5] NCCL INFO -hcugon- create sdma group queue rank:13 localRanks:8 nRanks:16 sdma copy is not enabled node1:6593:6628 [5] NCCL INFO P2P Chunksize set to 131072 node1:6588:6635 [0] NCCL INFO Channel 00/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 00/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 00/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 00/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 02/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 02/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 04/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 02/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 02/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 04/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 06/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 04/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 04/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 06/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 08/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 06/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 08/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 06/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 10/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 10/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 08/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 12/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 08/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 12/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 14/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 10/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 10/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 16/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 14/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 12/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 12/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 18/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 16/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 00/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 14/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 14/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 00/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 00/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 00/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 20/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 18/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 02/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 16/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 16/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 02/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 22/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 02/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 02/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 20/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 04/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 18/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 24/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 18/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 04/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 04/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 04/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 06/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 22/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 26/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 06/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 20/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 20/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 06/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 06/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 08/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 24/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 28/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 08/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 22/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 08/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 22/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 08/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 10/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 26/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 30/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 10/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 10/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 24/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 10/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 24/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 12/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 28/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 12/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 12/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 12/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 26/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 26/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 14/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 00/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 00/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 30/0 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 14/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 14/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 14/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 28/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 16/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 28/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 02/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 02/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 16/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 16/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 16/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 18/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 30/0 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 30/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 04/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 00/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 04/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 18/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 18/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 18/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 00/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 06/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 20/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 06/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 02/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 20/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 20/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 20/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 08/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 22/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 02/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 08/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 04/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 22/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 22/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 22/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 10/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 24/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 04/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 10/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 06/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 24/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 24/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 12/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 24/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 26/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 12/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 06/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 08/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 26/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 26/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 14/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 26/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 28/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 14/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 08/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 28/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 10/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 16/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 28/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 28/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 16/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 30/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 10/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 18/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 30/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 12/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 30/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 30/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 18/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 12/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 01/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 20/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 01/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 14/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 01/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 20/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 01/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 22/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 14/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 03/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 16/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 03/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 22/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 24/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 03/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 03/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 05/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 16/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 05/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 18/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 24/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 26/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 05/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 07/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 05/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 18/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 07/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 20/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 26/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 28/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 09/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 07/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 07/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 09/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 22/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 28/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 20/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 30/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 11/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 09/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 09/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 11/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 30/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 24/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 13/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 22/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 11/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 11/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 13/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 26/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 00/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 02/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 04/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 15/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 24/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 06/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 08/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 10/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 15/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 13/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 12/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 13/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 14/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 16/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 00/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 18/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 20/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 02/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 28/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 04/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 22/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 17/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 24/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 06/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 26/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 08/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 26/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 28/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 10/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 30/0 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 12/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 17/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 14/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 15/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 16/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 15/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 18/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 20/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 22/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 24/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 19/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 26/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 30/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 28/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 30/0 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 28/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 19/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 17/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 17/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 21/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 01/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 21/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 30/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 19/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 19/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 23/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 03/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 23/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 01/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 21/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 21/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 25/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 05/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 25/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 03/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 27/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 23/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 23/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 27/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 07/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 29/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 05/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 25/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 25/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 29/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 09/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 31/0 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 07/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 27/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 27/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 31/0 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 11/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 09/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 29/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 29/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 13/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 11/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 31/0 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 31/0 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 15/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 13/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 01/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 03/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 05/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 07/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 09/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 17/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 11/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 01/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 13/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 03/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 15/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 05/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 17/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 07/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 15/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 19/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 09/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 21/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 11/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 23/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 25/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 13/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 15/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 27/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 17/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 29/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 19/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 31/0 : 9[56000] -> 3[5000] [send] via NET/IB/0 comm 0x2f865c0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 19/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 21/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 23/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 25/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 27/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 29/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 31/0 : 1[56000] -> 11[5000] [send] via NET/IB/0 comm 0x2a6c650 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 17/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 21/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 19/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 23/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 01/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 21/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 25/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 03/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 27/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 01/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 23/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 05/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 03/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 29/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 07/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 25/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 09/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 31/0 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 05/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 27/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 11/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 29/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 13/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 07/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 00/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 15/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 31/0 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 09/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 02/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 17/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 11/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 00/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 04/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 19/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 13/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 02/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 21/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 06/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 15/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 04/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 23/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 08/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 17/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 06/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 25/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 10/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 19/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 08/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 27/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 12/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 21/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 10/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 29/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 14/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 23/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 12/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 31/0 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 14/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 25/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 16/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 01/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 16/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 27/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 18/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 00/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 02/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 04/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 18/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 03/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 29/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 06/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 20/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 08/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 10/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 31/0 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 05/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 12/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 14/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 16/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 18/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 20/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 20/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 22/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 24/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 26/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 28/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 22/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 30/0 : 2[5d000] -> 11[5000] [send] via NET/IB/1 comm 0x3bda8f0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 07/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 01/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 01/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 03/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 05/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 22/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 24/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 07/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 09/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 11/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 13/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 15/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 00/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 17/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 02/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 19/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 04/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 21/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 06/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 23/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 08/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 03/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 10/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 25/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 26/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 12/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 24/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 27/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 14/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 09/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 29/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 16/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 31/0 : 1[56000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 18/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 20/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 01/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 22/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 24/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 26/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 28/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 30/0 : 10[5d000] -> 3[5000] [send] via NET/IB/1 comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 28/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 05/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 26/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 01/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 03/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 05/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 03/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 30/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 07/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 07/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 09/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 11/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 11/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 28/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 13/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 15/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 17/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 19/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 21/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 23/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 25/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 27/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 05/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 09/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 29/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 31/0 : 9[56000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 13/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 01/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 30/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 07/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 11/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 15/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 03/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 09/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 17/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 13/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 05/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 11/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 19/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 07/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 15/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 13/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 21/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 09/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 17/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 15/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 23/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 19/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 11/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 25/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 17/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 21/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 13/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 27/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 19/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 23/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 29/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 21/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7042 [4] NCCL INFO Connected all rings comm 0x205cf60 nRanks 16 busId e8000 master:7001:7042 [4] NCCL INFO Channel 01/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 25/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 31/0 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 23/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 27/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 25/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 29/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 15/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 27/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 31/0 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Connected all rings comm 0x20724e0 nRanks 16 busId e8000 node1:6592:6630 [4] NCCL INFO Channel 01/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 29/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 17/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 31/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 19/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 21/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 23/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 25/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7040 [5] NCCL INFO Connected all rings comm 0x39bdf90 nRanks 16 busId c1000 master:7000:7044 [3] NCCL INFO Channel 27/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 01/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 01/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 29/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 03/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6628 [5] NCCL INFO Connected all rings comm 0x20c72e0 nRanks 16 busId c1000 node1:6595:6633 [7] NCCL INFO Channel 03/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 31/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 05/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 05/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 07/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 07/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 09/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 09/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 11/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 11/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 03/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 13/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 05/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 15/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 07/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 17/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 09/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 19/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 11/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 21/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 13/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 23/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 15/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 25/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 17/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 27/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 19/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 13/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 29/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 03/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 21/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 15/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 31/0 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 17/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 23/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 05/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 19/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 25/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 07/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 21/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 27/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 09/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 23/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 29/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 11/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 25/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 31/0 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 13/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 27/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 15/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 29/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 17/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 31/0 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 19/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 21/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 23/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 25/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 27/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 29/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 31/0 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 01/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 01/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 03/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 03/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 05/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 05/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 07/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 09/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 11/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 07/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 13/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 09/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 15/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 11/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7041 [7] NCCL INFO Connected all rings comm 0x3aab000 nRanks 16 busId b1000 master:6997:7038 [0] NCCL INFO Connected all rings comm 0x2c7f290 nRanks 16 busId 9f000 master:6997:7038 [0] NCCL INFO Channel 01/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 17/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Connected all rings comm 0x3ef2cf0 nRanks 16 busId ca000 node1:6593:6628 [5] NCCL INFO Channel 13/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 15/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 19/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 21/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 17/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 23/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 19/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 25/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 21/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 27/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 29/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 31/0 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 23/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Connected all rings comm 0x2537600 nRanks 16 busId ca000 node1:6588:6635 [0] NCCL INFO Connected all rings comm 0x36c8820 nRanks 16 busId 9f000 node1:6588:6635 [0] NCCL INFO Channel 01/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 25/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6633 [7] NCCL INFO Connected all rings comm 0x2ffe760 nRanks 16 busId b1000 node1:6593:6628 [5] NCCL INFO Channel 27/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 29/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 31/0 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 03/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 05/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 07/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 09/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 11/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 13/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 03/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 15/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 05/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 17/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 07/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 19/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 09/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 21/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 11/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 23/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 13/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 25/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 15/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 27/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 17/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 29/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 19/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 31/0 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 01/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 21/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 03/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 23/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 05/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 25/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 07/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 27/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 09/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 29/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 11/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 31/0 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 13/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 15/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 17/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 19/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 01/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 21/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 03/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 23/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 05/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 00/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 25/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 00/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 02/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 07/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 27/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 02/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 04/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 09/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 29/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 04/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 06/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 11/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 31/0 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 06/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 08/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 13/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 08/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 10/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 15/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 10/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 12/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 17/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 12/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 14/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 19/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 14/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 16/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 21/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 18/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 16/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 23/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 20/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 18/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 25/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 22/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 20/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 27/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 24/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 22/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 29/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 26/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 24/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 31/0 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 28/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 26/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6630 [4] NCCL INFO Channel 30/0 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 28/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7001:7042 [4] NCCL INFO Channel 30/0 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 00/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 01/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 02/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 03/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 04/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 05/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 06/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 07/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 08/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 00/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 09/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 10/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 01/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 00/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 11/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 02/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 01/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 12/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 03/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 02/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 13/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 03/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 04/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 14/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 04/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 05/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 15/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 05/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 06/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 16/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 06/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 07/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 17/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 07/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 08/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 08/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 09/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 18/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 09/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 10/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 19/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 10/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 11/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 20/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 11/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 00/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 12/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 21/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 12/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 01/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 13/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 22/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 13/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 02/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 14/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 23/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 14/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 03/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 15/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 24/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 15/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 04/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 25/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 16/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 05/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 26/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 16/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 17/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 06/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 27/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 18/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 17/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 07/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 28/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 19/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 18/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 08/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 29/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 20/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 19/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 09/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 30/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 21/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 20/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 10/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7043 [6] NCCL INFO Channel 31/0 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 22/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 21/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 11/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 23/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 22/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 12/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 24/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 23/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 13/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 25/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 24/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 14/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 26/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 25/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 15/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 27/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 26/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 16/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 28/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 27/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 17/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 29/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 28/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 18/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 30/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 29/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 19/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Channel 31/0 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 30/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 20/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6632 [6] NCCL INFO Channel 31/0 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 21/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 22/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 23/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 24/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 25/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 26/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 27/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 28/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 29/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 30/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6593:6628 [5] NCCL INFO Channel 31/0 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7040 [5] NCCL INFO Connected all trees master:7002:7040 [5] NCCL INFO NCCL_ALGO set by environment to tree,ring master:7002:7040 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:7002:7040 [5] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:7002:7040 [5] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6593:6628 [5] NCCL INFO Connected all trees node1:6593:6628 [5] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6593:6628 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6593:6628 [5] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer node1:6593:6628 [5] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP master:7000:7044 [3] NCCL INFO Connected all rings comm 0x2a2d870 nRanks 16 busId 5000 master:6999:7045 [2] NCCL INFO Connected all rings comm 0x3bda8f0 nRanks 16 busId 5d000 node1:6590:6634 [2] NCCL INFO Connected all rings comm 0x231fa00 nRanks 16 busId 5d000 node1:6589:6629 [1] NCCL INFO Connected all rings comm 0x2f865c0 nRanks 16 busId 56000 node1:6591:6631 [3] NCCL INFO Connected all rings comm 0x3e87c90 nRanks 16 busId 5000 master:6999:7045 [2] NCCL INFO Channel 00/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 01/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 02/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 00/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 03/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 01/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 04/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 05/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 02/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 06/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 03/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 07/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 04/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 08/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 05/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 09/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 06/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 10/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 07/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 11/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 08/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 12/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 09/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 01/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 13/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 10/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 03/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 14/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 11/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 05/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 15/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 12/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 07/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 16/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 13/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 09/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 17/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 14/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 18/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 11/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 15/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 19/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 13/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 16/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 20/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 15/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 17/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 21/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 17/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 18/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 22/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 19/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 19/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 23/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 21/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 20/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 24/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 23/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 21/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 25/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 25/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 26/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 22/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 27/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 27/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 23/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 29/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 28/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 24/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 31/0 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 29/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 25/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 01/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 30/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 26/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Connected all rings comm 0x2a6c650 nRanks 16 busId 56000 master:6999:7045 [2] NCCL INFO Channel 31/0 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 27/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 03/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 28/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 01/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 03/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 05/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 29/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 07/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 09/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 11/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 13/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 15/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 17/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 19/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 21/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 30/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 31/0 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 05/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 01/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 03/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 07/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 05/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 07/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 09/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 11/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 13/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 15/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 17/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 19/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 21/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 23/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 25/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 09/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 27/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 29/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 31/0 : 3[5000] -> 11[5000] [receive] via NET/IB/0/GDRDMA comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 01/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 03/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 05/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 07/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 09/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 11/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 13/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 15/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 17/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 19/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 11/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 21/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 23/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 25/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 27/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 29/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 31/0 : 11[5000] -> 3[5000] [send] via NET/IB/0 comm 0x3e87c90 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 13/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 15/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 17/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 19/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 21/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 23/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 25/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 27/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 29/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 31/0 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 00/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 01/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 02/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 03/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 04/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 01/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 05/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 00/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 02/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 04/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 06/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 08/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 10/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 12/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 01/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 14/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 16/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 06/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 18/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 03/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 20/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 22/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 24/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 26/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 28/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 30/0 : 0[9f000] -> 8[9f000] [receive] via NET/IB/1/GDRDMA comm 0x36c8820 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 03/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 00/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 02/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 04/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 07/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 06/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 08/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 05/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 10/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 12/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 14/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 16/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 18/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 05/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 20/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 22/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 24/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 26/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 08/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 28/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 node1:6588:6635 [0] NCCL INFO Channel 30/0 : 8[9f000] -> 0[9f000] [send] via NET/IB/1 comm 0x36c8820 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 07/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 09/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 09/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 10/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 11/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 11/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 13/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 12/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 15/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 13/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 17/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 07/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 14/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 19/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 09/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 21/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 15/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 11/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 23/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 16/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 25/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 00/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 13/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 17/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 27/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 01/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 18/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 15/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 29/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 02/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 19/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 17/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 31/0 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 03/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 20/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 19/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 04/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 01/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 21/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 21/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 05/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 22/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 23/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 06/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 23/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 25/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 07/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 24/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 27/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 08/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 25/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 03/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 29/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 09/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 26/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 05/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 10/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 31/0 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 27/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 07/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 11/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 28/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 12/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 09/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 29/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 13/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 11/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 30/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 14/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 13/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6634 [2] NCCL INFO Channel 31/0 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 15/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 15/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 16/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 17/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 17/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 19/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 18/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 21/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 19/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 23/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 20/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 21/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 25/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 22/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 27/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 00/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 23/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 29/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 24/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 01/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 31/0 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 25/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 02/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 26/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 03/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 01/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 27/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 00/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 02/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 04/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 04/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 06/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 08/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 28/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 10/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 12/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 14/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 16/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 18/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 20/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 22/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 03/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 05/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 24/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 26/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 28/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 30/0 : 8[9f000] -> 0[9f000] [receive] via NET/IB/1/GDRDMA comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 00/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 02/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 04/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 06/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 08/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 10/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 12/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 14/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 05/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 16/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 06/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 18/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 20/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 22/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 24/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 26/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 29/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 28/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:6997:7038 [0] NCCL INFO Channel 30/0 : 0[9f000] -> 8[9f000] [send] via NET/IB/1 comm 0x2c7f290 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 07/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 07/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 30/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 09/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6589:6629 [1] NCCL INFO Channel 31/0 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 08/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 11/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 09/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 13/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 10/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 15/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 11/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 17/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 12/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 19/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 13/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 21/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 14/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 23/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 00/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 15/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 25/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 01/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 16/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 27/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 02/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 17/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 29/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 03/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 18/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 31/0 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 04/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 19/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 05/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 20/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 06/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 21/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 07/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 22/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 08/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 23/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 00/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 09/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 00/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 24/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 01/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 10/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 01/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 25/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 02/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 11/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 02/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 26/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 03/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 12/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 03/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 27/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 04/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 13/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 04/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 28/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 05/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 14/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 05/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 29/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 06/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 15/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 06/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 30/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 07/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 16/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 07/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 08/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7045 [2] NCCL INFO Channel 31/0 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 17/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 08/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 09/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 18/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 09/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 10/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 19/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 10/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 11/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 20/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 11/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 12/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 12/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 21/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 13/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 13/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 22/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 14/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 14/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 15/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 23/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 15/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 16/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 24/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 16/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 17/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 25/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 17/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 18/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 26/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 18/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 19/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 27/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 19/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 20/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 28/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 20/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 21/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 29/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 21/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 22/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 30/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 22/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 23/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7039 [1] NCCL INFO Channel 31/0 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 23/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 24/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 24/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 25/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 25/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 26/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 26/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 27/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 27/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 28/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 28/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 29/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 30/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6595:6633 [7] NCCL INFO Channel 31/0 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 29/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 30/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7004:7041 [7] NCCL INFO Channel 31/0 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6633 [7] NCCL INFO Connected all trees node1:6595:6633 [7] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6595:6633 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6595:6633 [7] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:6998:7039 [1] NCCL INFO Connected all trees master:6998:7039 [1] NCCL INFO NCCL_ALGO set by environment to tree,ring master:6998:7039 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:6998:7039 [1] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:6998:7039 [1] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6595:6633 [7] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6589:6629 [1] NCCL INFO Connected all trees node1:6589:6629 [1] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6589:6629 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6589:6629 [1] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:7004:7041 [7] NCCL INFO Connected all trees master:7004:7041 [7] NCCL INFO NCCL_ALGO set by environment to tree,ring master:7004:7041 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:7004:7041 [7] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer node1:6589:6629 [1] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP master:7004:7041 [7] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6594:6632 [6] NCCL INFO Connected all trees node1:6594:6632 [6] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6594:6632 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6594:6632 [6] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:7003:7043 [6] NCCL INFO Connected all trees master:7003:7043 [6] NCCL INFO NCCL_ALGO set by environment to tree,ring master:7003:7043 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:7003:7043 [6] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer node1:6594:6632 [6] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP master:7003:7043 [6] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP master:7000:7044 [3] NCCL INFO Channel 23/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 25/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 27/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 29/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 31/0 : 11[5000] -> 3[5000] [receive] via NET/IB/0/GDRDMA comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 01/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 03/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 05/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 07/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 09/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 11/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 13/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 15/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 17/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 19/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 21/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 23/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 25/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 27/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 29/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 31/0 : 3[5000] -> 11[5000] [send] via NET/IB/0 comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 00/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 02/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 04/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 06/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 08/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 10/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 12/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 14/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 16/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 18/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 00/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 20/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 02/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 22/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 04/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 24/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 06/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 26/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 08/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 28/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 10/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7044 [3] NCCL INFO Channel 30/0 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 12/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 14/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 16/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 18/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 20/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 22/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 24/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 26/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 28/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6631 [3] NCCL INFO Channel 30/0 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7042 [4] NCCL INFO Connected all trees master:7001:7042 [4] NCCL INFO NCCL_ALGO set by environment to tree,ring master:7001:7042 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:7001:7042 [4] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:7001:7042 [4] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6592:6630 [4] NCCL INFO Connected all trees node1:6592:6630 [4] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6592:6630 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6592:6630 [4] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:6999:7045 [2] NCCL INFO Connected all trees master:6999:7045 [2] NCCL INFO NCCL_ALGO set by environment to tree,ring master:6999:7045 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:6999:7045 [2] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer node1:6592:6630 [4] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP master:6999:7045 [2] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6590:6634 [2] NCCL INFO Connected all trees node1:6590:6634 [2] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6590:6634 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6590:6634 [2] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer node1:6590:6634 [2] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP master:6997:7038 [0] NCCL INFO Connected all trees master:6997:7038 [0] NCCL INFO NCCL_ALGO set by environment to tree,ring master:6997:7038 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:6997:7038 [0] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:6997:7038 [0] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6588:6635 [0] NCCL INFO Connected all trees node1:6588:6635 [0] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6588:6635 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6588:6635 [0] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer node1:6588:6635 [0] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6591:6631 [3] NCCL INFO Connected all trees node1:6591:6631 [3] NCCL INFO NCCL_ALGO set by environment to tree,ring node1:6591:6631 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 node1:6591:6631 [3] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer node1:6591:6631 [3] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP master:7000:7044 [3] NCCL INFO Connected all trees master:7000:7044 [3] NCCL INFO NCCL_ALGO set by environment to tree,ring master:7000:7044 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 256 | 256 master:7000:7044 [3] NCCL INFO 32 coll channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer master:7000:7044 [3] NCCL INFO NCCL_LAUNCH_MODE set by environment to GROUP node1:6588:6635 [0] NCCL INFO Init config for nccl_context_test: 0 node1:6592:6630 [4] NCCL INFO Init config for nccl_context_test: 0 node1:6590:6634 [2] NCCL INFO Init config for nccl_context_test: 0 node1:6594:6632 [6] NCCL INFO Init config for nccl_context_test: 0 node1:6591:6631 [3] NCCL INFO Init config for nccl_context_test: 0 node1:6589:6629 [1] NCCL INFO Init config for nccl_context_test: 0 node1:6593:6628 [5] NCCL INFO Init config for nccl_context_test: 0 node1:6595:6633 [7] NCCL INFO Init config for nccl_context_test: 0 node1:6588:6635 [0] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6588:6635 [0] NCCL INFO comm 0x36c8820 rank 8 nranks 16 cudaDev 0 busId 9f000 commId 0x957f9fef67ebf659 localSize 464 used 1186349392 bytes - Init COMPLETE node1:6592:6630 [4] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6592:6630 [4] NCCL INFO comm 0x20724e0 rank 12 nranks 16 cudaDev 4 busId e8000 commId 0x957f9fef67ebf659 localSize 464 used 872825168 bytes - Init COMPLETE node1:6593:6628 [5] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6593:6628 [5] NCCL INFO comm 0x20c72e0 rank 13 nranks 16 cudaDev 5 busId c1000 commId 0x957f9fef67ebf659 localSize 464 used 1074151760 bytes - Init COMPLETE node1:6594:6632 [6] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6591:6631 [3] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6590:6634 [2] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6589:6629 [1] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6595:6633 [7] NCCL INFO Maximum number of GPUs in any NUMA node: 2 node1:6595:6633 [7] NCCL INFO comm 0x2ffe760 rank 15 nranks 16 cudaDev 7 busId b1000 commId 0x957f9fef67ebf659 localSize 464 used 906379600 bytes - Init COMPLETE node1:6594:6632 [6] NCCL INFO comm 0x2537600 rank 14 nranks 16 cudaDev 6 busId ca000 commId 0x957f9fef67ebf659 localSize 464 used 1074151760 bytes - Init COMPLETE node1:6591:6631 [3] NCCL INFO comm 0x3e87c90 rank 11 nranks 16 cudaDev 3 busId 5000 commId 0x957f9fef67ebf659 localSize 464 used 1108754768 bytes - Init COMPLETE node1:6590:6634 [2] NCCL INFO comm 0x231fa00 rank 10 nranks 16 cudaDev 2 busId 5d000 commId 0x957f9fef67ebf659 localSize 464 used 906379600 bytes - Init COMPLETE node1:6589:6629 [1] NCCL INFO comm 0x2f865c0 rank 9 nranks 16 cudaDev 1 busId 56000 commId 0x957f9fef67ebf659 localSize 464 used 1040597328 bytes - Init COMPLETE master:7001:7042 [4] NCCL INFO Init config for nccl_context_test: 0 master:7000:7044 [3] NCCL INFO Init config for nccl_context_test: 0 master:7003:7043 [6] NCCL INFO Init config for nccl_context_test: 0 master:6997:7038 [0] NCCL INFO Init config for nccl_context_test: 0 master:6999:7045 [2] NCCL INFO Init config for nccl_context_test: 0 master:7004:7041 [7] NCCL INFO Init config for nccl_context_test: 0 master:6998:7039 [1] NCCL INFO Init config for nccl_context_test: 0 master:7002:7040 [5] NCCL INFO Init config for nccl_context_test: 0 master:7001:7042 [4] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:7001:7042 [4] NCCL INFO comm 0x205cf60 rank 4 nranks 16 cudaDev 4 busId e8000 commId 0x957f9fef67ebf659 localSize 464 used 872825168 bytes - Init COMPLETE master:7000:7044 [3] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:6997:7038 [0] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:7000:7044 [3] NCCL INFO comm 0x2a2d870 rank 3 nranks 16 cudaDev 3 busId 5000 commId 0x957f9fef67ebf659 localSize 464 used 1108754768 bytes - Init COMPLETE master:6998:7039 [1] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:6998:7039 [1] NCCL INFO comm 0x2a6c650 rank 1 nranks 16 cudaDev 1 busId 56000 commId 0x957f9fef67ebf659 localSize 464 used 1040597328 bytes - Init COMPLETE master:6997:7038 [0] NCCL INFO comm 0x2c7f290 rank 0 nranks 16 cudaDev 0 busId 9f000 commId 0x957f9fef67ebf659 localSize 464 used 1186349392 bytes - Init COMPLETE master:6999:7045 [2] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:7003:7043 [6] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:6999:7045 [2] NCCL INFO comm 0x3bda8f0 rank 2 nranks 16 cudaDev 2 busId 5d000 commId 0x957f9fef67ebf659 localSize 464 used 906379600 bytes - Init COMPLETE master:7003:7043 [6] NCCL INFO comm 0x3ef2cf0 rank 6 nranks 16 cudaDev 6 busId ca000 commId 0x957f9fef67ebf659 localSize 464 used 1074151760 bytes - Init COMPLETE master:7002:7040 [5] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:7004:7041 [7] NCCL INFO Maximum number of GPUs in any NUMA node: 2 master:7004:7041 [7] NCCL INFO comm 0x3aab000 rank 7 nranks 16 cudaDev 7 busId b1000 commId 0x957f9fef67ebf659 localSize 464 used 906379600 bytes - Init COMPLETE master:7002:7040 [5] NCCL INFO comm 0x39bdf90 rank 5 nranks 16 cudaDev 5 busId c1000 commId 0x957f9fef67ebf659 localSize 464 used 1074151760 bytes - Init COMPLETE # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) master:6997:7080 [0] NCCL INFO Channel 00/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 7[b1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 9[56000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 15[b1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 5[c1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 7[b1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 6[ca000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 13[c1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 3[5000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 11[5000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 10[5d000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 15[b1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 12[e8000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 4[e8000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 14[ca000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 6[ca000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 1[56000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 2[5d000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 14[ca000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 10[5d000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 7[b1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 2[5d000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 7[b1000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 6[ca000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 12[e8000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 14[ca000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 15[b1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 5[c1000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 15[b1000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 13[c1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 5[c1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 11[5000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 4[e8000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 6[ca000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 13[c1000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 3[5000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 11[5000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 14[ca000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 15[b1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 12[e8000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 7[b1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 3[5000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 15[b1000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 4[e8000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 5[c1000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 5[c1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 7[b1000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 13[c1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 14[ca000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 13[c1000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 14[ca000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 6[ca000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 7[b1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 4[e8000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 6[ca000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 13[c1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 12[e8000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 15[b1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 6[ca000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 14[ca000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 5[c1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 15[b1000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 6[ca000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 12[e8000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 12[e8000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 4[e8000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 7[b1000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 14[ca000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 4[e8000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 13[c1000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 5[c1000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 7[b1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 15[b1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 14[ca000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 6[ca000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 15[b1000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 7[b1000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 4[e8000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 12[e8000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 11[5000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 3[5000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 5[c1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 10[5d000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 5[c1000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 15[b1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 13[c1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 13[c1000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 7[b1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 2[5d000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 3[5000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 6[ca000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 6[ca000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 5[c1000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 9[56000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 15[b1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 11[5000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 15[b1000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 14[ca000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 14[ca000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 2[5d000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 1[56000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 13[c1000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 0[9f000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 7[b1000] via P2P/IPC comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 7[b1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 7[b1000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 10[5d000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 15[b1000] via P2P/IPC comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 8[9f000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 8[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 6[ca000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 0[9f000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 14[ca000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 3[5000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 4[e8000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 12[e8000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 9[56000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 0[9f000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 11[5000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 12[e8000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 9[56000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 8[9f000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 11[5000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 4[e8000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 1[56000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 3[5000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 1[56000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 11[5000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 9[56000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 10[5d000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 2[5d000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 3[5000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 1[56000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 8[9f000] -> 1[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 0[9f000] -> 9[56000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 10[5d000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 2[5d000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 1[56000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 9[56000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 0[9f000] -> 10[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 8[9f000] -> 2[5d000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 1[56000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 9[56000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 11[5000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 3[5000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 6[ca000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 13[c1000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 5[c1000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 3[5000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 11[5000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 8[9f000] -> 3[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 0[9f000] -> 11[5000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 10[5d000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 14[ca000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 8[9f000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 12[e8000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 5[c1000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 2[5d000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 0[9f000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 4[e8000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 13[c1000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 0[9f000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 9[56000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 2[5d000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 1[56000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 4[e8000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 1[56000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 8[9f000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 9[56000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 12[e8000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 10[5d000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 3[5000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 10[5d000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 11[5000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 4[e8000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 8[9f000] -> 4[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x205cf60 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 10[5d000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 2[5d000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 3[5000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 9[56000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 11[5000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 2[5d000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 12[e8000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 1[56000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 11[5000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 0[9f000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 0[9f000] -> 12[e8000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x20724e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 0[9f000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 9[56000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 3[5000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 3[5000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 8[9f000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 5[c1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 1[56000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 8[9f000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 13[c1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 1[56000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 12[e8000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 8[9f000] -> 5[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 11[5000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 9[56000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 0[9f000] -> 13[c1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 2[5d000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 4[e8000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 2[5d000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 10[5d000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 13[c1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 10[5d000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 14[ca000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 10[5d000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 6[ca000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 10[5d000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 1[56000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 2[5d000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 5[c1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 9[56000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 2[5d000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 8[9f000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 9[56000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 0[9f000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 4[e8000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 1[56000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 11[5000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 3[5000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 0[9f000] -> 14[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x2537600 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 12[e8000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 8[9f000] -> 6[ca000] [receive] via NET/IB/0/GDRDMA/Shared comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x3bda8f0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 1[56000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 9[56000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 6[ca000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 4[e8000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 12[e8000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 15[b1000] [send] via NET/IB/1/Shared comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 14[ca000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 7[b1000] [send] via NET/IB/1/Shared comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 13[c1000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 2[5d000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 5[c1000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 9[56000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 10[5d000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 1[56000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 0[9f000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 3[5000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 8[9f000] -> 7[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x3aab000 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 00/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 01/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 02/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 00/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 03/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 8[9f000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 01/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 04/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 02/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 05/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 03/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 04/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 06/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 05/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 07/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 08/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 06/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 09/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 07/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 10/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 08/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 11/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 09/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 12/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 10/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 13/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 11/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 14/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 12/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 13/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 15/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 14/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 16/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 15/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 16/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 17/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 17/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 18/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 18/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 11[5000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 19/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 19/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 20/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 20/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 21/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 21/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 00/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 22/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 22/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 01/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 23/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 23/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 02/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 24/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 00/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 24/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 03/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 25/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 00/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 01/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 04/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 25/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 26/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 0[9f000] -> 15[b1000] [receive] via NET/IB/1/GDRDMA/Shared comm 0x2ffe760 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 00/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 01/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 02/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 27/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 05/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 26/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 02/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 01/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 00/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 00/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 03/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 28/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 03/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 06/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 27/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 02/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 01/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 01/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 29/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 04/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 04/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 07/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 28/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 02/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 03/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 02/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 05/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 30/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 00/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 29/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 03/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 04/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 06/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7003:7079 [6] NCCL INFO Channel 31/1 : 6[ca000] -> 5[c1000] via P2P/IPC comm 0x3ef2cf0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 03/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 08/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 01/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 30/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 04/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 05/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 07/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 05/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 09/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 04/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 00/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 02/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 05/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7004:7084 [7] NCCL INFO Channel 31/1 : 7[b1000] -> 6[ca000] via P2P/IPC comm 0x3aab000 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 08/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 06/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 05/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 01/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 03/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 09/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 06/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 07/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 00/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 01/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 02/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 10/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 06/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 06/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 03/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 04/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 10/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 05/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 06/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 04/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 02/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 07/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 07/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 08/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 09/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 10/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 08/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 11/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 12/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 13/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 07/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 14/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 11/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 07/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 15/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 11/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 16/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 08/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 17/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 05/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 03/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 18/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 19/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 20/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 21/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 09/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 22/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 12/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 23/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 08/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 24/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 25/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 08/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 09/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 26/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 12/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 27/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 06/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 28/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 04/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 29/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 30/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 13/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6997:7080 [0] NCCL INFO Channel 31/1 : 0[9f000] -> 15[b1000] [send] via NET/IB/0/Shared comm 0x2c7f290 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 10/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 09/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 10/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 09/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 13/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 00/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 07/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 05/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 14/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 11/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 00/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 10/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 11/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 14/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 08/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 10/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 01/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 15/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 06/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 12/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 01/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 12/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 11/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 16/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 15/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 09/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 02/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 11/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 07/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 13/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 13/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 02/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 12/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 17/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 16/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 10/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 08/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 14/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 14/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 18/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 03/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 13/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 11/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 03/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 12/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 09/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 15/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 19/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 04/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 14/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 17/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 12/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 04/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 20/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 13/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 16/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 10/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 15/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 05/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 18/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 13/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 05/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 21/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 17/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 14/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 11/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 16/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 14/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 19/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 06/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 22/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 06/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 18/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 15/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 12/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 17/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 23/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 15/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 07/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 20/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 07/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 19/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 00/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 16/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 01/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 13/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 02/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 24/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 18/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 03/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 21/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 16/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 04/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 08/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 20/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 05/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 06/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 07/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 17/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 08/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 25/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 09/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 10/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 19/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 14/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 22/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 11/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 17/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 12/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 21/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 09/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 13/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 14/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 08/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 26/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 15/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 18/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 16/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 17/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 20/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 23/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 15/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 18/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 18/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 15/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 22/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 19/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 10/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 20/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 27/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 21/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 09/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 22/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 23/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 19/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 24/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 24/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 21/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 25/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 19/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 23/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 16/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 26/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 28/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 11/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 27/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 28/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 10/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 29/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 30/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 20/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6588:6671 [0] NCCL INFO Channel 31/1 : 8[9f000] -> 7[b1000] [send] via NET/IB/0/Shared comm 0x36c8820 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 25/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 24/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 29/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 16/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 12/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 20/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 22/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 17/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 11/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 21/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 30/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 25/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 26/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 21/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 17/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 13/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 18/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 23/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 12/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7001:7081 [4] NCCL INFO Channel 31/1 : 4[e8000] -> 3[5000] via P2P/IPC comm 0x205cf60 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 22/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 26/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 27/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 18/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 14/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 22/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 24/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 19/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 13/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 27/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 23/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 28/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 19/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 23/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 15/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 25/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 20/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 14/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 28/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 24/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 29/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 16/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 24/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 20/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 26/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 21/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 15/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 29/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 25/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 30/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 21/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 25/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 17/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 27/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 30/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 22/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 16/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 26/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6589:6672 [1] NCCL INFO Channel 31/1 : 9[56000] -> 8[9f000] via P2P/IPC comm 0x2f865c0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 26/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 18/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 22/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:7002:7085 [5] NCCL INFO Channel 31/1 : 5[c1000] -> 4[e8000] via P2P/IPC comm 0x39bdf90 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 28/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 00/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 23/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 17/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 27/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 19/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 27/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 23/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 29/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 01/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 24/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 18/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 28/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 28/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 20/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 24/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 30/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 19/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 25/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 02/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 29/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 21/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 29/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 25/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 master:6998:7083 [1] NCCL INFO Channel 31/1 : 1[56000] -> 0[9f000] via P2P/IPC comm 0x2a6c650 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 20/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 26/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 03/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 30/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 30/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 22/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 26/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 21/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 27/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 04/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 23/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6594:6670 [6] NCCL INFO Channel 31/1 : 14[ca000] -> 13[c1000] via P2P/IPC comm 0x2537600 nRanks 16 master:6999:7078 [2] NCCL INFO Channel 31/1 : 2[5d000] -> 1[56000] via P2P/IPC comm 0x3bda8f0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 27/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 00/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 22/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 05/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 28/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 24/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 28/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 23/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 01/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 29/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 06/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 25/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 29/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 24/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 02/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 30/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 26/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 07/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 30/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 25/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 03/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 master:7000:7082 [3] NCCL INFO Channel 31/1 : 3[5000] -> 2[5d000] via P2P/IPC comm 0x2a2d870 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 27/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 08/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6595:6675 [7] NCCL INFO Channel 31/1 : 15[b1000] -> 14[ca000] via P2P/IPC comm 0x2ffe760 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 26/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 04/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 28/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 09/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 27/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 05/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 29/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 10/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 28/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 30/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 06/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 11/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 29/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6592:6669 [4] NCCL INFO Channel 31/1 : 12[e8000] -> 11[5000] via P2P/IPC comm 0x20724e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 07/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 12/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 30/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 13/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 08/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6590:6668 [2] NCCL INFO Channel 31/1 : 10[5d000] -> 9[56000] via P2P/IPC comm 0x231fa00 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 14/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 09/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 15/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 10/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 16/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 11/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 17/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 12/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 18/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 13/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 19/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 14/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 20/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 15/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 21/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 16/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 22/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 17/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 23/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 18/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 24/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 19/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 25/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 20/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 26/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 21/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 27/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 22/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 28/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 23/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 29/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 24/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 30/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 25/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6593:6673 [5] NCCL INFO Channel 31/1 : 13[c1000] -> 12[e8000] via P2P/IPC comm 0x20c72e0 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 26/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 27/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 28/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 29/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 30/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 node1:6591:6674 [3] NCCL INFO Channel 31/1 : 11[5000] -> 10[5d000] via P2P/IPC comm 0x3e87c90 nRanks 16 [master:06991] 175 more processes have sent help message help-mpi-btl-openib.txt / no device params found [master:06991] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [master:06991] 111 more processes have sent help message help-mpi-btl-openib.txt / default subnet prefix [master:06991] 15 more processes have sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port node1:6591:6660 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<44584> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6591:6660 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6591:6660 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6591:6660 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] node1:6593:6663 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<34454> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6593:6663 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6593:6663 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6593:6663 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] node1:6589:6661 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<52902> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6589:6661 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6589:6661 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6589:6661 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] node1:6595:6665 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<49602> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6595:6665 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6595:6665 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6595:6665 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] node1:6588:6664 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<32848> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6588:6664 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6588:6664 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6588:6664 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] node1:6592:6667 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<56546> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6592:6667 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6592:6667 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6592:6667 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] node1:6590:6662 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<47996> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6590:6662 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6590:6662 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6590:6662 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] node1:6594:6666 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.195<47126> with error 12, opcode 0, len 0, vendor err 129 (Recv) node1:6594:6666 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 node1:6594:6666 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 node1:6594:6666 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:7000:7070 [3] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<46788> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:7000:7070 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:7000:7070 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:7000:7070 [3] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:7003:7076 [6] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<47746> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:7003:7076 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:7003:7076 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:7003:7076 [6] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:7001:7077 [4] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<34840> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:7001:7077 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:7001:7077 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:7001:7077 [4] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:6999:7072 [2] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<47992> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:6999:7072 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:6999:7072 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:6999:7072 [2] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:7004:7075 [7] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<45098> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:7004:7075 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:7004:7075 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:7004:7075 [7] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:7002:7073 [5] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<39448> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:7002:7073 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:7002:7073 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:7002:7073 [5] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:6998:7071 [1] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<40246> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:6998:7071 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:6998:7071 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:6998:7071 [1] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread] master:6997:7074 [0] /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net_ib.cc:1332 NCCL WARN NET/IB : Got completion from peer 10.17.26.196<43682> with error 12, opcode 0, len 0, vendor err 129 (Recv) master:6997:7074 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/transport/net.cc:1286 -> 6 master:6997:7074 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:689 -> 6 master:6997:7074 [0] NCCL INFO /data/jenkins_workspace/workspace/rccl_release/build/hipify/src/proxy.cc:868 -> 6 [Proxy Thread]