gpcnet_network_load.log 10.8 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
NetworkLoad Tests v1.3
  Test with 10 MPI ranks (10 nodes)
  2 nodes running Network Tests
  8 nodes running Congestion Tests (min 100 nodes per congestor)

  Legend
   RR = random ring communication pattern
   Lat = latency
   BW = bandwidth
   BW+Sync = bandwidth with barrier
+------------------------------------------------------------------------------------------------------------------------------------------+
|                                                          Isolated Network Tests                                                          |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|                            Name |          Min |          Max |          Avg |   Avg(Worst) |          99% |        99.9% |        Units |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|          RR Two-sided Lat (8 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |         usec |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| RR Two-sided BW+Sync (131072 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|        Multiple Allreduce (8 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |         usec |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+

+------------------------------------------------------------------------------------------------------------------------------------------+
|                                                        Isolated Congestion Tests                                                         |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|                            Name |          Min |          Max |          Avg |   Avg(Worst) |          99% |        99.9% |        Units |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|               Alltoall (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|       Two-sided Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|             Put Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|              Get Bcast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+

+------------------------------------------------------------------------------------------------------------------------------------------+
|                             Network Tests running with Congestion Tests (    RR Two-sided Lat Network Test)                              |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|                            Name |          Min |          Max |          Avg |   Avg(Worst) |          99% |        99.9% |        Units |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|          RR Two-sided Lat (8 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |         usec |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|               Alltoall (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|       Two-sided Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|             Put Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|              Get Bcast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+

+------------------------------------------------------------------------------------------------------------------------------------------+
|                             Network Tests running with Congestion Tests (RR Two-sided BW+Sync Network Test)                              |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|                            Name |          Min |          Max |          Avg |   Avg(Worst) |          99% |        99.9% |        Units |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| RR Two-sided BW+Sync (131072 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|               Alltoall (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|       Two-sided Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|             Put Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|              Get Bcast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+

+------------------------------------------------------------------------------------------------------------------------------------------+
|                             Network Tests running with Congestion Tests (  Multiple Allreduce Network Test)                              |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|                            Name |          Min |          Max |          Avg |   Avg(Worst) |          99% |        99.9% |        Units |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|        Multiple Allreduce (8 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |         usec |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|               Alltoall (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|       Two-sided Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|             Put Incast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
|              Get Bcast (4096 B) |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |      10000.0 |   MiB/s/rank |
+---------------------------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+

+------------------------------------------------------------------------------+
|          Network Tests running with Congestion Tests - Key Results           |
+---------------------------------+--------------------------------------------+
|                            Name |                   Congestion Impact Factor |
+---------------------------------+----------------------+---------------------+
|                                 |                  Avg |                 99% |
+---------------------------------+----------------------+---------------------+
|          RR Two-sided Lat (8 B) |                 0.0X |                0.0X |
+---------------------------------+----------------------+---------------------+
| RR Two-sided BW+Sync (131072 B) |                 0.0X |                0.0X |
+---------------------------------+----------------------+---------------------+
|        Multiple Allreduce (8 B) |                 0.0X |                0.0X |
+---------------------------------+----------------------+---------------------+