Support topo-aware IB performance validation (#373)

* Support topo-aware IB performance validation

Add a new pattern `topo-aware`, so the user can run IB performance
test based on VM's topology information. This way, the user can
validate the IB performance across VM pairs with different distance
as a quick test instead of pair-wise test.

To run with topo-aware pattern, user needs to specify three required
(and two optional) parameters in YAML config file:
--pattern	topo-aware
--ibstat	path to ibstat output
--ibnetdiscover	path to ibnetdiscover output
--min_dist	minimum distance of VM pairs (optional, default 2)
--max_dist	maximum distance of VM pairs (optional, default 6)

The newly added topo_aware module then parses the topology
information, builds a graph, and generates the VM pairs with
the specified distance (# hops).

The specified IB test will then be running across these
generated VM pairs.
Signed-off-by: Jie Zhang <jessezhang1010@gmail.com>

* Add description about topology aware ib traffic tests
Signed-off-by: Jie Zhang <jessezhang1010@gmail.com>

* Add unit test to verify generated topology aware config file

This commit adds unit test to verify the generated topology aware
config file is correct. To do so, four new data files are added in
order to invoke gen_topo_aware_config function to generate topology
aware config file, then compares it with the expected config file.
Signed-off-by: Jie Zhang <jessezhang1010@gmail.com>

* Fix lint issue on Azure pipeline
Signed-off-by: Jie Zhang <jessezhang1010@gmail.com>

docs/user-tutorial/benchmarks/micro-benchmarks.md

@@ -302,6 +302,17 @@ Measure the InfiniBand performance under multi nodes' traffic pattern.
 The traffic pattern is defined in a config file, which is pre-defined for one-to-many, many-to-one and all-to-all patterns.
 Each row in the config is one round, and all pairs of nodes in a row run ib command simultaneously.
 
+Besides the above three patterns, ib-traffic also supports topology-aware traffic pattern. To run ib-traffic with topology-aware
+pattern, the user needs to specify 3 required (and 2 optional) parameters in YAML config file:
+   - --pattern	 **topo-aware**
+   - --ibstat	 **path to ibstat output**
+   - --ibnetdiscover	 **path to ibnetdiscover output**
+   - --min_dist	 **minimum distance of VM pairs (optional, default 2)**
+   - --max_dist	 **maximum distance of VM pairs (optional, default 6)**
+
+Each row in the config file has all VM pairs with a fixed distance (#hops). That's by default, 1st, 2nd, 3rd row has all VM pairs
+with topology distance of 2, 4, 6, respectively.
+
 #### Metrics
 
 | Metrics                                                          | Unit             | Description                                                                                                                                                                                                                        |

setup.py

@@ -150,6 +150,7 @@ def run(self):
         'markdown>=3.3.0',
         'matplotlib>=3.0.0',
         'natsort>=7.1.1',
+        'networkx>=1.11',
         'numpy>=1.19.2',
         'openpyxl>=3.0.7',
         'omegaconf==2.0.6',

superbench/benchmarks/micro_benchmarks/ib_validation_performance.py

@@ -6,6 +6,7 @@
 import os
 
 from superbench.common.utils import logger
+from superbench.common.utils import gen_topo_aware_config
 from superbench.benchmarks import BenchmarkRegistry, ReturnCode
 from superbench.common.devices import GPU
 from superbench.benchmarks.micro_benchmarks import MicroBenchmarkWithInvoke
@@ -26,7 +27,7 @@ def __init__(self, name, parameters=''):
         self.__support_ib_commands = [
             'ib_write_bw', 'ib_read_bw', 'ib_send_bw', 'ib_write_lat', 'ib_read_lat', 'ib_send_lat'
         ]
-        self.__patterns = ['one-to-one', 'one-to-many', 'many-to-one']
+        self.__patterns = ['one-to-one', 'one-to-many', 'many-to-one', 'topo-aware']
         self.__config_path = os.path.join(os.getcwd(), 'config.txt')
         self.__config = []
 
@@ -108,6 +109,34 @@ def add_parser_arguments(self):
             required=False,
             help='The path of hostfile on the target machines.',
         )
+        self._parser.add_argument(
+            '--min_dist',
+            type=int,
+            default=2,
+            required=False,
+            help='The minimum distance of VM pair in topo-aware pattern',
+        )
+        self._parser.add_argument(
+            '--max_dist',
+            type=int,
+            default=6,
+            required=False,
+            help='The maximum distance of VM pair in topo-aware pattern',
+        )
+        self._parser.add_argument(
+            '--ibstat',
+            type=str,
+            default=None,
+            required=False,
+            help='The path of ibstat output',
+        )
+        self._parser.add_argument(
+            '--ibnetdiscover',
+            type=str,
+            default=None,
+            required=False,
+            help='The path of ibnetdiscover output',
+        )
 
     def __one_to_many(self, n):
         """Generate one-to-many pattern config.
@@ -190,21 +219,26 @@ def __fully_one_to_one(self, n):
             candidates = non_moving + robin
         return config
 
-    def gen_traffic_pattern(self, n, mode, config_file_path):
+    def gen_traffic_pattern(self, hosts, mode, config_file_path):
         """Generate traffic pattern into config file.
 
         Args:
-            n (int): the number of nodes.
-            mode (str): the traffic mode, including 'one-to-one', 'one-to-many', 'many-to-one'.
+            hosts (list): the list of VM hostnames read from hostfile.
+            mode (str): the traffic mode, including 'one-to-one', 'one-to-many', 'many-to-one', 'topo-aware'.
             config_file_path (str): the path of config file to generate.
         """
         config = []
+        n = len(hosts)
         if mode == 'one-to-many':
             config = self.__one_to_many(n)
         elif mode == 'many-to-one':
             config = self.__many_to_one(n)
         elif mode == 'one-to-one':
             config = self.__fully_one_to_one(n)
+        elif mode == 'topo-aware':
+            config = gen_topo_aware_config(
+                hosts, self._args.ibstat, self._args.ibnetdiscover, self._args.min_dist, self._args.max_dist
+            )
         with open(config_file_path, 'w') as f:
             for line in config:
                 f.write(line + '\n')
@@ -223,7 +257,7 @@ def __prepare_config(self):
                 hosts = f.readlines()
             # Generate the config file if not define
             if self._args.config is None:
-                self.gen_traffic_pattern(len(hosts), self._args.pattern, self.__config_path)
+                self.gen_traffic_pattern(hosts, self._args.pattern, self.__config_path)
             # Use the config file defined in args
             else:
                 self.__config_path = self._args.config

superbench/common/utils/__init__.py

@@ -8,6 +8,7 @@
 from superbench.common.utils.file_handler import rotate_dir, create_sb_output_dir, get_sb_config
 from superbench.common.utils.lazy_import import LazyImport
 from superbench.common.utils.process import run_command
+from superbench.common.utils.topo_aware import gen_topo_aware_config
 
 device_manager = LazyImport('superbench.common.utils.device_manager')
 
@@ -22,4 +23,5 @@
     'network',
     'rotate_dir',
     'run_command',
+    'gen_topo_aware_config',
 ]

superbench/common/utils/topo_aware.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+"""Topology Aware Utilities."""
+
+import re
+import networkx as nx
+from superbench.common.utils import logger
+
+
+class quick_regexp(object):
+    """Quick regular expression class."""
+    def __init__(self):
+        """Constructor."""
+        self.groups = None
+        self.matched = False
+
+    def search(self, pattern, string, flags=0):
+        """Search with group function."""
+        match = re.search(pattern, string, flags)
+        if match:
+            self.matched = True
+            if match.groups():
+                self.groups = re.search(pattern, string, flags).groups()
+            else:
+                self.groups = True
+        else:
+            self.matched = False
+            self.groups = None
+
+        return self.matched
+
+
+def gen_topo_aware_config(host_list, ibstat_file, ibnetdiscover_file, min_dist, max_dist):    # noqa: C901
+    """Generate topology aware config list in specified distance range.
+
+    Args:
+        host_list (list): list of VM read from hostfile.
+        ibstat_file (str): path of ibstat output.
+        ibnetdiscover_file (str): path of ibnetdiscover output.
+        min_dist (int): minimum distance of VM pair.
+        max_dist (int): maximum distance of VM pair.
+
+    Returns:
+        list: the generated config list, each item in the list is a str
+        like "0,1;2,3", which represents all VM pairs with a fixed
+        topology distance (#hops).
+    """
+    config = []
+    if not ibstat_file or not ibnetdiscover_file:
+        logger.error('Either ibstat or ibnetdiscover not specified.')
+        return config
+
+    if min_dist > max_dist:
+        logger.error('Specified minimum distane ({}) is larger than maximum distance ({}).'.format(min_dist, max_dist))
+        return config
+
+    # index each hostname in hostfile
+    host_idx = dict()
+    idx = 0
+    for h in host_list:
+        host_idx[h.strip()] = idx
+        idx += 1
+
+    sysimgguid_to_vmhost = dict()
+    phyhost_to_sysimgguid = dict()
+    topology = dict()
+    topologyX = dict()
+    hosts = list()
+    current_dev = ''
+
+    # Read ibstat output, store mapping from sysimgguid to vmhost for each HCA
+    try:
+        with open(ibstat_file, mode='r', buffering=1) as f:
+            for line in f:
+                line = line.strip()
+                isinstance(line, str)
+                if line:
+                    r = quick_regexp()
+                    if r.search(r'^(VM_hostname)\s+(.+)', line):
+                        vmhost = r.groups[1]
+                    elif r.search(r'^(0x)(.+)', line):
+                        sysimgguid = r.groups[1]
+                        sysimgguid_to_vmhost[sysimgguid] = vmhost
+    except BaseException as e:
+        logger.error('Failed to read ibstate file, message: {}.'.format(str(e)))
+        return config
+
+    # Read ibnetdiscover output to
+    # store the information of each device (Swith/HCA)
+    # collect all physical hosts that are associated with HCA
+    # store mapping from physical hostname to sysimgguid for each HCA
+    try:
+        with open(ibnetdiscover_file, mode='r', buffering=1) as f:
+            for line in f:
+                line = line.strip()
+                isinstance(line, str)
+                if line:
+                    r = quick_regexp()
+                    # Read the device (Switch/HCA)'s port, GUID, and metadata
+                    if r.search(r'^(\w+)\s+(\d+)\s+\"(.+?)\"\s+#\s+\"(.+?)\"', line):
+                        current_dev = r.groups[2]
+                        topology[current_dev] = dict()
+                        topology[current_dev]['number_of_ports'] = int(r.groups[1])
+                        topology[current_dev]['sysimgguid'] = r.groups[2].split('-')[1]
+                        topology[current_dev]['metadata'] = r.groups[3]
+                        metadata = topology[current_dev]['metadata']
+
+                        if r.groups[0] == 'Switch':
+                            topology[current_dev]['node_type'] = 'switch'
+                            topologyX[metadata] = dict()
+                            topologyX[metadata]['node_type'] = 'switch'
+                        else:
+                            topology[current_dev]['node_type'] = 'hca'
+                            if topology[current_dev]['metadata'] == 'Mellanox Technologies Aggregation Node':
+                                topology[current_dev]['hca_type'] = 'AN'
+                            elif (topology[current_dev]['metadata'].find('ufm') != -1):
+                                topology[current_dev]['hca_type'] = 'UFM'
+                            else:
+                                topology[current_dev]['hca_type'] = 'HCA'
+                                hostname = metadata.split(' ')[0]
+                                topology[current_dev]['hostname'] = hostname
+                                curr_sysimgguid = topology[current_dev]['sysimgguid']
+                                # curr_sysimgguid in sysimgguid_to_vmhost is to check
+                                # if the physical host associated with current device has VM running on it.
+                                # If not, no need to include this physical host for distance calculation later.
+                                if (
+                                    not (hostname in topologyX) and hostname != 'MT4123'
+                                    and curr_sysimgguid in sysimgguid_to_vmhost
+                                ):
+                                    topologyX[hostname] = dict()
+                                    topologyX[hostname]['node_type'] = 'host'
+                                    hosts.append(hostname)
+                                    phyhost_to_sysimgguid[hostname] = curr_sysimgguid
+
+                    # Read the port connection lines under each device (switch/hca)
+                    if r.search(r'^\[(\d+)\].*?\"(.+?)\"\[(\d+)\]', line):
+                        local_port = int(r.groups[0])
+                        connected_to_remote_host = r.groups[1]
+                        connected_to_remote_port = int(r.groups[2])
+                        topology[current_dev][local_port] = {connected_to_remote_host: connected_to_remote_port}
+    except BaseException as e:
+        logger.error('Failed to read ibnetdiscover file, message: {}.'.format(str(e)))
+        return config
+
+    # Build a graph across physical hosts and switch nodes
+    Gnx = nx.Graph()
+    Gnx.add_nodes_from(topologyX)
+    for dev in topology:
+        numPorts = topology[dev]['number_of_ports']
+        if numPorts > 0:
+            for port in range(1, numPorts + 1):
+                if port in topology[dev].keys():
+                    remote_host = list(topology[dev][port].keys())[0]
+                    if topology[dev]['node_type'] == 'hca':
+                        if topology[dev]['hca_type'] == 'AN' or topology[dev]['hca_type'] == 'UFM':
+                            continue
+                        src = topology[dev]['hostname']
+                    else:
+                        src = topology[dev]['metadata']
+
+                    if remote_host in topology:
+                        if topology[remote_host]['node_type'] == 'hca':
+                            if topology[remote_host]['hca_type'] == 'AN' or topology[remote_host]['hca_type'] == 'UFM':
+                                continue
+                            dest = topology[remote_host]['hostname']
+                        else:
+                            dest = topology[remote_host]['metadata']
+
+                        Gnx.add_edge(src, dest)
+
+    all_paths_len_obj = nx.all_pairs_shortest_path_length(Gnx)
+    all_paths_len = dict(all_paths_len_obj)
+
+    # Generate VM pairs with different topology distance
+    for nodeDistance in range(min_dist, max_dist + 1):
+        visited = dict()
+        hostpairs = {}
+        row = []
+        for hostx in hosts:
+            if hostx in visited:
+                continue
+            visited[hostx] = 1
+            for hosty in hosts:
+                if hosty in visited:
+                    continue
+                if all_paths_len[hostx][hosty] == nodeDistance:
+                    hostpairs[hostx] = hosty
+                    visited[hosty] = 1
+                    break
+        for host in hostpairs:
+            vma = sysimgguid_to_vmhost[phyhost_to_sysimgguid[host]]
+            vmb = sysimgguid_to_vmhost[phyhost_to_sysimgguid[hostpairs[host]]]
+            idx_pair = '{},{}'.format(host_idx[vma], host_idx[vmb])
+            row.append(idx_pair)
+        if row:
+            row = ';'.join(row)
+            config.append(row)
+
+    return config

tests/benchmarks/micro_benchmarks/test_ib_traffic_performance.py

@@ -6,14 +6,24 @@
 import os
 import numbers
 import unittest
+import uuid
 from pathlib import Path
 from unittest import mock
 from collections import defaultdict
+from superbench.common.utils import gen_topo_aware_config
 
+from tests.helper import decorator
 from tests.helper.testcase import BenchmarkTestCase
 from superbench.benchmarks import BenchmarkRegistry, Platform, BenchmarkType, ReturnCode
 
 
+def gen_hostlist(hostlist, num):
+    """Generate a fake list of specified number of hosts."""
+    hostlist.clear()
+    for i in range(0, num):
+        hostlist.append(str(uuid.uuid4()))
+
+
 class IBBenchmarkTest(BenchmarkTestCase, unittest.TestCase):
     """Tests for IBBenchmark benchmark."""
     @classmethod
@@ -31,9 +41,12 @@ def tearDownClass(cls):
                 p.unlink()
         super().tearDownClass()
 
-    def test_generate_config(self):    # noqa: C901
+    @decorator.load_data('tests/data/ib_traffic_topo_aware_hostfile')    # noqa: C901
+    @decorator.load_data('tests/data/ib_traffic_topo_aware_expected_config')
+    def test_generate_config(self, tp_hosts, tp_expected_config):    # noqa: C901
         """Test util functions ."""
         test_config_file = 'test_gen_config.txt'
+        hostlist = []
 
         def read_config(filename):
             config = []
@@ -59,16 +72,18 @@ def read_config(filename):
         benchmark = benchmark_class(benchmark_name)
         # Small scale test
         node_num = 4
+        gen_hostlist(hostlist, node_num)
         for m in ['one-to-one', 'one-to-many', 'many-to-one']:
-            benchmark.gen_traffic_pattern(node_num, m, test_config_file)
+            benchmark.gen_traffic_pattern(hostlist, m, test_config_file)
             config = read_config(test_config_file)
             assert (config == expected_config[m])
         # Large scale test
         node_num = 1000
+        gen_hostlist(hostlist, node_num)
         # check for 'one-to-many' and 'many-to-one'
         # In Nth step, the count of N is (N-1), others are all 1
         for m in ['one-to-many', 'many-to-one']:
-            benchmark.gen_traffic_pattern(node_num, m, test_config_file)
+            benchmark.gen_traffic_pattern(hostlist, m, test_config_file)
             config = read_config(test_config_file)
             assert (len(config) == node_num)
             assert (len(config[0]) == node_num - 1)
@@ -93,7 +108,7 @@ def read_config(filename):
         # check for 'one-to-one'
         # Each index appears 1 time in each step
         # Each index has been combined once with all the remaining indexes
-        benchmark.gen_traffic_pattern(node_num, 'one-to-one', test_config_file)
+        benchmark.gen_traffic_pattern(hostlist, 'one-to-one', test_config_file)
         config = read_config(test_config_file)
         if node_num % 2 == 1:
             assert (len(config) == node_num)
@@ -115,6 +130,15 @@ def read_config(filename):
         for node in range(node_num):
             assert (sorted(test_pairs[node]) == [(i) for i in range(node_num) if i != node])
 
+        # check for 'topo-aware'
+        # compare generated config file with pre-saved expected config file
+        tp_ibstat_path = 'tests/data/ib_traffic_topo_aware_ibstat.txt'
+        tp_ibnetdiscover_path = 'tests/data/ib_traffic_topo_aware_ibnetdiscover.txt'
+        hostlist = tp_hosts.split()
+        expected_config = tp_expected_config.split()
+        config = gen_topo_aware_config(hostlist, tp_ibstat_path, tp_ibnetdiscover_path, 2, 6)
+        assert (config == expected_config)
+
         Path(test_config_file).unlink()
 
     @mock.patch('superbench.common.devices.GPU.vendor', new_callable=mock.PropertyMock)

tests/data/ib_traffic_topo_aware_expected_config

+0,1;2,3;4,5;6,7;8,9
+0,2;1,3;6,8;7,9
+0,6;1,7;2,8;3,9

tests/data/ib_traffic_topo_aware_hostfile

+vma414bbc00005I
+vma414bbc00005J
+vma414bbc00005K
+vma414bbc00005L
+vma414bbc00005M
+vma414bbc00005N
+vma414bbc00005O
+vma414bbc00005P
+vma414bbc00005Q
+vma414bbc00005R

tests/data/ib_traffic_topo_aware_ibnetdiscover.txt

+# Faked topology file (ibnetdiscover output)
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xb8599f0300c2f06a
+switchguid=0xff08c43213b3f30(ff08c43213b3f30)
+Switch  41 "S-0ff08c43213b3f30"         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/S01/U1" enhanced port 0 lid 387 lmc 0
+[1]     "S-0ff08c432172aa1a"[21]                # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L01/U1" lid 226 4xHDR s=4 w=2 v=4 e=4
+[2]     "S-0ff08c432172a97a"[21]                # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L02/U1" lid 969 4xHDR s=4 w=2 v=4 e=4
+[41]    "H-0ff08c43213b3f38"[1](ff08c43213b3f38)                # "Mellanox Technologies Aggregation Node" lid 2152 4xHDR s=4 w=2 v=2 e=4
+
+
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xb8599f0300c2f06a
+switchguid=0xff08c432172aa1a(ff08c432172aa1a)
+Switch  41 "S-0ff08c432172aa1a"         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L01/U1" base port 0 lid 226 lmc 0
+[1]     "S-0ff08c43215bae48"[25]                # "MF0;rmd70-0101-0919-01ib0:MQM8700/U1" lid 556 4xHDR s=4 w=2 v=4 e=4
+[2]     "S-0ff08c4321407312"[25]                # "MF0;rmd70-0101-0919-02ib0:MQM8700/U1" lid 164 4xHDR s=4 w=2 v=4 e=4
+[3]     "S-0ff08c43214073f2"[25]                # "MF0;rmd70-0101-0919-03ib0:MQM8700/U1" lid 1913 4xHDR s=4 w=2 v=4 e=4
+[21]    "S-0ff08c43213b3f30"[1]         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/S01/U1" lid 387 4xHDR s=4 w=2 v=4 e=4
+
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xb8599f0300c2f06a
+switchguid=0xff08c432172a97a(ff08c432172a97a)
+Switch  41 "S-0ff08c432172a97a"         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L02/U1" base port 0 lid 969 lmc 0
+[11]    "S-0ff08c432154169c"[26]                # "MF0;rmd70-0101-0919-11ib0:MQM8700/U1" lid 1229 4xHDR s=4 w=2 v=4 e=4
+[12]    "S-0ff08c43215bae68"[26]                # "MF0;rmd70-0101-0919-12ib0:MQM8700/U1" lid 825 4xHDR s=4 w=2 v=4 e=4
+[21]    "S-0ff08c43213b3f30"[2]         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/S01/U1" lid 387 4xHDR s=4 w=2 v=4 e=4
+
+
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xff08c43215bae48
+switchguid=0xff08c43215bae48(ff08c43215bae48)
+Switch  41 "S-0ff08c43215bae48"         # "MF0;rmd70-0101-0919-01ib0:MQM8700/U1" enhanced port 0 lid 556 lmc 0
+[1]     "H-0ff08c4321664e96"[1](ff08c4321664e96)                # "RMD701091901003 ibp11s0f0" lid 1181 4xHDR s=4 w=2 v=4 e=4
+[9]     "H-0ff08c43217299f2"[1](ff08c43217299f2)                # "RMD701091901019 ibp11s0f0" lid 3191 4xHDR s=4 w=2 v=4 e=4
+[25]    "S-0ff08c432172aa1a"[1]         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L01/U1" lid 226 4xHDR s=4 w=2 v=4 e=4
+
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xff08c4321407312
+switchguid=0xff08c4321407312(ff08c4321407312)
+Switch  41 "S-0ff08c4321407312"         # "MF0;rmd70-0101-0919-02ib0:MQM8700/U1" enhanced port 0 lid 164 lmc 0
+[1]     "H-0ff08c4321729742"[1](ff08c4321729742)                # "RMD701091901053 ibp11s0f0" lid 516 4xHDR s=4 w=2 v=4 e=4
+[9]     "H-0ff08c4321729986"[1](ff08c4321729986)                # "RMD701091901037 ibp11s0f0" lid 1062 4xHDR s=4 w=2 v=4 e=4
+[25]    "S-0ff08c432172aa1a"[2]         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L01/U1" lid 226 4xHDR s=4 w=2 v=4 e=4
+
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xff08c43214073f2
+switchguid=0xff08c43214073f2(ff08c43214073f2)
+Switch  41 "S-0ff08c43214073f2"         # "MF0;rmd70-0101-0919-03ib0:MQM8700/U1" enhanced port 0 lid 1913 lmc 0
+[1]     "H-1c34da03005baca4"[1](1c34da03005baca4)               # "RMD701091902003 ibp11s0f0" lid 1294 4xHDR s=4 w=2 v=4 e=4
+[9]     "H-0ff08c432166275a"[1](ff08c432166275a)                # "RMD701091902019 ibp11s0f0" lid 698 4xHDR s=4 w=2 v=4 e=4
+[25]    "S-0ff08c432172aa1a"[3]         # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L01/U1" lid 226 4xHDR s=4 w=2 v=4 e=4
+
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xff08c432154169c
+switchguid=0xff08c432154169c(ff08c432154169c)
+Switch  41 "S-0ff08c432154169c"         # "MF0;rmd70-0101-0919-11ib0:MQM8700/U1" enhanced port 0 lid 1229 lmc 0
+[1]     "H-0ff08c4321664b66"[1](ff08c4321664b66)                # "RMD701091906003 ibp11s0f0" lid 806 4xHDR s=4 w=2 v=4 e=4
+[9]     "H-0ff08c432166274e"[1](ff08c432166274e)                # "RMD701091906019 ibp11s0f0" lid 600 4xHDR s=4 w=2 v=4 e=4
+[26]    "S-0ff08c432172a97a"[11]                # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L02/U1" lid 969 4xHDR s=4 w=2 v=4 e=4
+
+
+vendid=0x2c9
+devid=0xd2f0
+sysimgguid=0xff08c43215bae68
+switchguid=0xff08c43215bae68(ff08c43215bae68)
+Switch  41 "S-0ff08c43215bae68"         # "MF0;rmd70-0101-0919-12ib0:MQM8700/U1" enhanced port 0 lid 825 lmc 0
+[1]     "H-0ff08c4321664f2a"[1](ff08c4321664f2a)                # "RMD701091906053 ibp11s0f0" lid 329 4xHDR s=4 w=2 v=4 e=4
+[9]     "H-043f720300e61112"[1](43f720300e61112)                # "RMD701091906037 ibp11s0f0" lid 150 4xHDR s=4 w=2 v=4 e=4
+[26]    "S-0ff08c432172a97a"[12]                # "MF0;rmd70-0101-0908-01ib1-A:MCS8500/L02/U1" lid 969 4xHDR s=4 w=2 v=4 e=4
+
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c4321664e96
+caguid=0xff08c4321664e96
+Ca      1 "H-0ff08c4321664e96"          # "RMD701091901003 ibp11s0f0"
+[1](ff08c4321664e96)    "S-0ff08c43215bae48"[1]         # lid 1181 lmc 0 "MF0;rmd70-0101-0919-01ib0:MQM8700/U1" lid 556 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c43217299f2
+caguid=0xff08c43217299f2
+Ca      1 "H-0ff08c43217299f2"          # "RMD701091901019 ibp11s0f0"
+[1](ff08c43217299f2)    "S-0ff08c43215bae48"[9]         # lid 3191 lmc 0 "MF0;rmd70-0101-0919-01ib0:MQM8700/U1" lid 556 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c4321729742
+caguid=0xff08c4321729742
+Ca      1 "H-0ff08c4321729742"          # "RMD701091901053 ibp11s0f0"
+[1](ff08c4321729742)    "S-0ff08c4321407312"[1]         # lid 516 lmc 0 "MF0;rmd70-0101-0919-02ib0:MQM8700/U1" lid 164 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c4321729986
+caguid=0xff08c4321729986
+Ca      1 "H-0ff08c4321729986"          # "RMD701091901037 ibp11s0f0"
+[1](ff08c4321729986)    "S-0ff08c4321407312"[9]         # lid 1062 lmc 0 "MF0;rmd70-0101-0919-02ib0:MQM8700/U1" lid 164 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0x1c34da03005baca4
+caguid=0x1c34da03005baca4
+Ca      1 "H-1c34da03005baca4"          # "RMD701091902003 ibp11s0f0"
+[1](1c34da03005baca4)   "S-0ff08c43214073f2"[1]         # lid 1294 lmc 0 "MF0;rmd70-0101-0919-03ib0:MQM8700/U1" lid 1913 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c432166275a
+caguid=0xff08c432166275a
+Ca      1 "H-0ff08c432166275a"          # "RMD701091902019 ibp11s0f0"
+[1](ff08c432166275a)    "S-0ff08c43214073f2"[9]         # lid 698 lmc 0 "MF0;rmd70-0101-0919-03ib0:MQM8700/U1" lid 1913 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c4321664b66
+caguid=0xff08c4321664b66
+Ca      1 "H-0ff08c4321664b66"          # "RMD701091906003 ibp11s0f0"
+[1](ff08c4321664b66)    "S-0ff08c432154169c"[1]         # lid 806 lmc 0 "MF0;rmd70-0101-0919-11ib0:MQM8700/U1" lid 1229 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c432166274e
+caguid=0xff08c432166274e
+Ca      1 "H-0ff08c432166274e"          # "RMD701091906019 ibp11s0f0"
+[1](ff08c432166274e)    "S-0ff08c432154169c"[9]         # lid 600 lmc 0 "MF0;rmd70-0101-0919-11ib0:MQM8700/U1" lid 1229 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0xff08c4321664f2a
+caguid=0xff08c4321664f2a
+Ca      1 "H-0ff08c4321664f2a"          # "RMD701091906053 ibp11s0f0"
+[1](ff08c4321664f2a)    "S-0ff08c43215bae68"[1]         # lid 329 lmc 0 "MF0;rmd70-0101-0919-12ib0:MQM8700/U1" lid 825 4xHDR s=4 w=2 v=3 e=4
+
+
+vendid=0x2c9
+devid=0x101b
+sysimgguid=0x43f720300e61112
+caguid=0x43f720300e61112
+Ca      1 "H-043f720300e61112"          # "RMD701091906037 ibp11s0f0"
+[1](43f720300e61112)    "S-0ff08c43215bae68"[9]         # lid 150 lmc 0 "MF0;rmd70-0101-0919-12ib0:MQM8700/U1" lid 825 4xHDR s=4 w=2 v=3 e=4
+

tests/data/ib_traffic_topo_aware_ibstat.txt

+VM_hostname vma414bbc00005I
+0x0ff08c4321664e96
+VM_hostname vma414bbc00005J
+0x0ff08c43217299f2
+VM_hostname vma414bbc00005K
+0x0ff08c4321729742
+VM_hostname vma414bbc00005L
+0x0ff08c4321729986
+VM_hostname vma414bbc00005M
+0x1c34da03005baca4
+VM_hostname vma414bbc00005N
+0x0ff08c432166275a
+VM_hostname vma414bbc00005O
+0x0ff08c4321664b66
+VM_hostname vma414bbc00005P
+0x0ff08c432166274e
+VM_hostname vma414bbc00005Q
+0x0ff08c4321664f2a
+VM_hostname vma414bbc00005R
+0x043f720300e61112
+