Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
nccl_reducescatter.log 4.62 KB
Newer Older
Yifan Xiong's avatar
Release - SuperBench v0.4.0 (#278)  
Yifan Xiong committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53 
# nThread 1 nGpus 8 minBytes 1 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 validation: 0
#
# Using devices
#   Rank  0 Pid 112580 on localhost device  0 [0x00] A100-SXM4-40GB
#   Rank  1 Pid 112580 on localhost device  1 [0x00] A100-SXM4-40GB
#   Rank  2 Pid 112580 on localhost device  2 [0x00] A100-SXM4-40GB
#   Rank  3 Pid 112580 on localhost device  3 [0x00] A100-SXM4-40GB
#   Rank  4 Pid 112580 on localhost device  4 [0x00] A100-SXM4-40GB
#   Rank  5 Pid 112580 on localhost device  5 [0x00] A100-SXM4-40GB
#   Rank  6 Pid 112580 on localhost device  6 [0x00] A100-SXM4-40GB
#   Rank  7 Pid 112580 on localhost device  7 [0x00] A100-SXM4-40GB
#
#                                                     out-of-place                       in-place
#       size         count    type   redop     time   algbw   busbw  error     time   algbw   busbw  error
#        (B)    (elements)                     (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
hostname:3442:3442 [0] NCCL INFO Launch mode Parallel
           0             0   float     sum    34.88    0.00    0.00    N/A    33.65    0.00    0.00    N/A
           0             0   float     sum    33.54    0.00    0.00    N/A    33.72    0.00    0.00    N/A
           0             0   float     sum    33.45    0.00    0.00    N/A    33.44    0.00    0.00    N/A
           0             0   float     sum    34.07    0.00    0.00    N/A    33.44    0.00    0.00    N/A
           0             0   float     sum    33.55    0.00    0.00    N/A    33.43    0.00    0.00    N/A
          32             1   float     sum    35.06    0.00    0.00    N/A    35.14    0.00    0.00    N/A
          64             2   float     sum    34.82    0.00    0.00    N/A    34.76    0.00    0.00    N/A
         128             4   float     sum    34.38    0.00    0.00    N/A    34.52    0.00    0.00    N/A
         256             8   float     sum    34.75    0.01    0.01    N/A    34.32    0.01    0.01    N/A
         512            16   float     sum    34.71    0.01    0.01    N/A    35.43    0.01    0.01    N/A
        1024            32   float     sum    35.16    0.03    0.03    N/A    34.75    0.03    0.03    N/A
        2048            64   float     sum    35.43    0.06    0.05    N/A    35.29    0.06    0.05    N/A
        4096           128   float     sum    35.49    0.12    0.10    N/A    35.17    0.12    0.10    N/A
        8192           256   float     sum    35.18    0.23    0.20    N/A    35.77    0.23    0.20    N/A
       16384           512   float     sum    35.27    0.46    0.41    N/A    35.49    0.46    0.40    N/A
       32768          1024   float     sum    35.00    0.94    0.82    N/A    35.09    0.93    0.82    N/A
       65536          2048   float     sum    36.78    1.78    1.56    N/A    36.92    1.77    1.55    N/A
      131072          4096   float     sum    40.71    3.22    2.82    N/A    39.78    3.29    2.88    N/A
      262144          8192   float     sum    48.12    5.45    4.77    N/A    46.65    5.62    4.92    N/A
      524288         16384   float     sum    59.81    8.77    7.67    N/A    58.88    8.90    7.79    N/A
     1048576         32768   float     sum    72.37   14.49   12.68    N/A    74.95   13.99   12.24    N/A
     2097152         65536   float     sum    80.64   26.01   22.76    N/A    79.62   26.34   23.05    N/A
     4194304        131072   float     sum    108.9   38.53   33.72    N/A    109.3   38.37   33.57    N/A
     8388608        262144   float     sum    147.3   56.96   49.84    N/A    166.8   50.28   44.00    N/A
    16777216        524288   float     sum    152.4  110.11   96.34    N/A    152.8  109.82   96.09    N/A
    33554432       1048576   float     sum    240.5  139.50  122.06    N/A    240.8  139.33  121.91    N/A
    67108864       2097152   float     sum    356.1  188.45  164.89    N/A    352.1  190.57  166.75    N/A
   134217728       4194304   float     sum    618.1  217.15  190.01    N/A    615.2  218.18  190.90    N/A
   268435456       8388608   float     sum   1108.7  242.11  211.84    N/A   1112.6  241.27  211.11    N/A
   536870912      16777216   float     sum   2169.0  247.52  216.58    N/A   2181.8  246.07  215.31    N/A
  1073741824      33554432   float     sum   4203.0  255.47  223.54    N/A   4206.3  255.27  223.36    N/A
  2147483648      67108864   float     sum   8356.9  256.97  224.85    N/A   8323.5  258.00  225.75    N/A
  4294967296     134217728   float     sum    16400  261.89  229.15    N/A    16402  261.86  229.13    N/A
  8589934592     268435456   float     sum    32464  264.60  231.52    N/A    32502  264.29  231.25    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 60.168
#