README.md 10.5 KB
Newer Older
Yifan Xiong's avatar
Yifan Xiong committed
1
# SuperBenchmark
2

3
[![MIT licensed](https://img.shields.io/badge/license-MIT-brightgreen.svg)](LICENSE)
Yifan Xiong's avatar
Yifan Xiong committed
4
[![Lint](https://github.com/microsoft/superbenchmark/workflows/Lint/badge.svg)](https://github.com/microsoft/superbenchmark/actions?query=workflow%3ALint)
5
6
7
8
9
10
[![Codecov](https://codecov.io/gh/microsoft/superbenchmark/branch/main/graph/badge.svg?token=DDiDLW7pSd)](https://codecov.io/gh/microsoft/superbenchmark)

| Azure Pipelines | Build Status |
| :---: | :---: |
| cpu-unit-test | [![Build Status](https://dev.azure.com/msrasrg/SuperBenchmark/_apis/build/status/microsoft.superbenchmark?branchName=main)](https://dev.azure.com/msrasrg/SuperBenchmark/_build/latest?definitionId=77&branchName=main) |
| gpu-unit-test | [![Build Status](https://dev.azure.com/msrasrg/SuperBenchmark/_apis/build/status/cuda-unit-test?branchName=main)](https://dev.azure.com/msrasrg/SuperBenchmark/_build/latest?definitionId=80&branchName=main) |
11
12


13
14
15
**SuperBench** is a validation and profiling tool for AI infrastructure, which supports:

* AI infrastructure validation and diagnosis
Yifan Xiong's avatar
Yifan Xiong committed
16
17
    * Distributed validation tools to validate hundreds or thousands of servers automatically
    * Consider both raw hardware and E2E model performance with ML workload patterns
18
19
20
21
    * Build a contract to identify hardware issues
    * Provide infrastructural-oriented criteria as Performance/Quality Gates for hardware and system release
    * Provide detailed performance report and advanced analysis tool  
* AI workload benchmarking and profiling
Yifan Xiong's avatar
Yifan Xiong committed
22
    * Provide comprehensive performance comparison between different existing hardware
23
    * Provide insights for hardware and software co-design
Yifan Xiong's avatar
Yifan Xiong committed
24
25
26
27

It includes micro-benchmark for primitive computation and communication benchmarking,
and model-benchmark to measure domain-aware end-to-end deep learning workloads.

28
29
> 🔴 __Note__:
SuperBench is in the early pre-alpha stage for open source, and not ready for general public yet.
30
If you want to jump in early, you can try building latest code yourself.
31

32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
## SuperBench capabilities, workflow and benchmarking metrics

The following graphic shows the capabilities provide by SuperBench core framework and its extension.

<img src="imgs/superbench_structure.png">

Benchmarking metrics provided by SuperBench are listed as below.

<table>
  <tbody>
    <tr align="center" valign="bottom">
      <td>
      </td>
      <td>
        <b>Micro Benchmark</b>
        <img src="imgs/bar.png"/>
      </td>
      <td>
        <b>Model Benchmark</b>
        <img src="imgs/bar.png"/>
      </td>
    </tr>
    <tr valign="top">
      <td align="center" valign="middle">
        <b>Metrics</b>
      </td>
      <td>
        <ul><li><b>Computation Benchmark</b></li>
          <ul><li><b>Kernel Performance</b></li>
            <ul>
              <li>GFLOPS</li>
              <li>TensorCore</li>
              <li>cuBLAS</li>
              <li>cuDNN</li>
            </ul>
          </ul>
          <ul><li><b>Kernel Launch Time</b></li>
            <ul>
              <li>Kernel_Launch_Event_Time</li>
              <li>Kernel_Launch_Wall_Time</li>
            </ul>
          </ul>
          <ul><li><b>Operator Performance</b></li>
            <ul><li>MatMul</li><li>Sharding_MatMul</li></ul>
          </ul>
          <ul><li><b>Memory</b></li>
            <ul><li>H2D_Mem_BW_&lt;GPU ID&gt;</li>
              <li>H2D_Mem_BW_&lt;GPU ID&gt;</li></ul>
          </ul>
        </ul>
        <ul><li><b>Communication Benchmark</b></li>
          <ul><li><b>Device P2P Bandwidth</b></li>
            <ul><li>P2P_BW_Max</li><li>P2P_BW_Min</li><li>P2P_BW_Avg</li></ul>
          </ul>
          <ul><li><b>RDMA</b></li>
            <ul><li>RDMA_Peak</li><li>RDMA_Avg</li></ul>
          </ul>
          <ul><li><b>NCCL</b></li>
            <ul><li>NCCL_AllReduce</li></ul>
            <ul><li>NCCL_AllGather</li></ul>
            <ul><li>NCCL_broadcast</li></ul>
            <ul><li>NCCL_reduce</li></ul>
            <ul><li>NCCL_reduce_scatter</li></ul>
          </ul>
        </ul>
        <ul><li><b>Computation-Communication Benchmark</b></li>
          <ul><li><b>Mul_During_NCCL</b></li><li><b>MatMul_During_NCCL</b></li></ul>
        </ul>
        <ul><li><b>Storage Benchmark</b></li>
          <ul><li><b>Disk</b></li>
            <ul>
              <li>Read/Write</li><li>Rand_Read/Rand_Write</li>
              <li>R/W_Read</li><li>R/W_Write</li><li>Rand_R/W_Read</li><li>Rand_R/W_Write</li>
            </ul>
          </ul>
        </ul>   
      </td>
      <td>
        <ul><li><b>CNN models</b></li>
          <ul>
            <li><b>ResNet</b></li>
              <ul><li>ResNet-50</li><li>ResNet-101</li><li>ResNet-152</li></ul>
          </ul>
          <ul>
            <li><b>DenseNet</b></li>
              <ul><li>DenseNet-169</li><li>DenseNet-201</li></ul>
          </ul>
          <ul>
            <li><b>VGG</b></li>
              <ul><li>VGG-11</li><li>VGG-13</li><li>VGG-16</li><li>VGG-19</li></ul>
          </ul>
          <ul><li><b>Other CNN models</b></li><ul><li>...</li></ul></ul>
        </ul>  
        <ul><li><b>BERT models</b></li>
          <ul><li><b>BERT</b></li><li><b>BERT_LARGE</b></li></ul>
        </ul>
        <ul><li><b>LSTM</b></li></ul>
        <ul><li><b>GPT-2</b></li></ul>
      </td>
    </tr>
  </tbody>
</table>

Yifan Xiong's avatar
Yifan Xiong committed
135
136
137

## Installation

138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
### Using Docker (_Preferred_)

__System Requirements__

* Platform: Ubuntu 18.04 or later (64-bit)
* Docker: Docker CE 19.03 or later

__Install SuperBench__

* Using Pre-Build Images

    ```sh
    docker pull superbench/superbench:dev-cuda11.1.1
    docker run -it --rm \
        --privileged --net=host --ipc=host --gpus=all \
        superbench/superbench:dev-cuda11.1.1 bash
    ```

* Building the Image

    ```sh
    docker build -f dockerfile/cuda11.1.1.dockerfile -t superbench/superbench:dev .
    ```

Yifan Xiong's avatar
Yifan Xiong committed
162
163
### Using Python

164
165
166
__System Requirements__

* Platform: Ubuntu 18.04 or later (64-bit); Windows 10 (64-bit) with WSL2
Yifan Xiong's avatar
Yifan Xiong committed
167
168
* Python: Python 3.6 or later, pip 18.0 or later

169
170
171
172
173
174
175
176
177
178
179
    Check whether Python environment is already configured:
    ```sh
    # check Python version
    python3 --version
    # check pip version
    python3 -m pip --version
    ```
    If not, install the followings:
    * [Python](https://www.python.org/)
    * [pip](https://pip.pypa.io/en/stable/installing/)
    * [venv](https://docs.python.org/3/library/venv.html)
Yifan Xiong's avatar
Yifan Xiong committed
180

181
182
183
184
185
186
187
188
189
190
191
    It's recommended to use a virtual environment (optional):
    ```sh
    # create a new virtual environment
    python3 -m venv --system-site-packages ./venv
    # activate the virtual environment
    source ./venv/bin/activate

    # exit the virtual environment later
    # after you finish running superbench
    deactivate
    ```
Yifan Xiong's avatar
Yifan Xiong committed
192

193
__Install SuperBench__
Yifan Xiong's avatar
Yifan Xiong committed
194

195
* PyPI Binary
Yifan Xiong's avatar
Yifan Xiong committed
196

197
198
199
    ```sh
    # not available yet
    ```
Yifan Xiong's avatar
Yifan Xiong committed
200

201
* From Source
Yifan Xiong's avatar
Yifan Xiong committed
202
203
204
205
206
207
208
209

    ```sh
    # get source code
    git clone https://github.com/microsoft/superbenchmark
    cd superbenchmark

    # install superbench
    python3 -m pip install .
210
    make postinstall
Yifan Xiong's avatar
Yifan Xiong committed
211
212
213
    ```


214
## Usage
Yifan Xiong's avatar
Yifan Xiong committed
215

216
### Run SuperBench
Yifan Xiong's avatar
Yifan Xiong committed
217

218
219
220
221
222
223
224
225
226
```sh
# run benchmarks in default settings
sb exec

# use a custom config
sb exec --config-file ./superbench/config/default.yaml
```

### Benchmark Gallary
Yifan Xiong's avatar
Yifan Xiong committed
227

228
Please find more benchmark examples [here](examples/benchmarks/).
Yifan Xiong's avatar
Yifan Xiong committed
229
230
231
232


## Developer Guide

233
234
235
236
237
If you want to develop new feature, please follow below steps to set up development environment.

### Check Environment

Follow __[System Requirements](#using-python)__.
Yifan Xiong's avatar
Yifan Xiong committed
238
239
240
241

### Set Up

```sh
242
243
# get latest code
git clone https://github.com/microsoft/superbenchmark
Yifan Xiong's avatar
Yifan Xiong committed
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
cd superbenchmark

# install superbench
python3 -m pip install -e .[dev,test]
```

### Lint and Test

```sh
# format code using yapf
python3 setup.py format

# check code style with mypy and flake8
python3 setup.py lint

# run all unit tests
python3 setup.py test
```

### Submit a Pull Request

Please install `pre-commit` before `git commit` to run all pre-checks.

```sh
pre-commit install
```

271
Open a pull request to main branch on GitHub.
Yifan Xiong's avatar
Yifan Xiong committed
272

273
274
275

## Contributing

276
277
### Contributor License Agreement

278
279
280
281
282
283
284
285
286
287
288
289
This project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

290
291
292
293
294
295
296
297
### Contributing principles

SuperBenchmark is an open-source project. Your participation and contribution are highly appreciated. There are several important things you need know before contributing to this project:

#### What content can be added to SuperBenchmark

1. Bug fixes for existing features.
2. New features for benchmark module (micro-benchmark, model-benchmark, etc.)
298

299
   If you would like to contribute a new feature on SuperBenchmark, please submit your proposal first. In [GitHub Issues](https://github.com/microsoft/superbenchmark/issues) module, choose `Enhancement Request` to finish the submission. If the proposal is accepted, you can submit pull requests to origin main branch.
300
301
302
303
304
305

#### Contribution steps

If you would like to contribute to the project, please follow below steps of joint development on GitHub.

1. `Fork` the repo first to your personal GitHub account.
306
2. Checkout from main branch for feature development.
307
3. When you finish the feature, please fetch the latest code from origin repo, merge to your branch and resolve conflict.
308
4. Submit pull requests to origin main branch.
309
310
5. Please note that there might be comments or questions from reviewers. It will need your help to update the pull request.

311
312
## Trademarks

313
314
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
315
316
317
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.