cli.md 13.5 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
---
id: cli
---

# CLI

SuperBench provides a command line interface to help you use, deploy and run benchmarks.
```
$ sb

   _____                       ____                  _
  / ____|                     |  _ \                | |
 | (___  _   _ _ __   ___ _ __| |_) | ___ _ __   ___| |__
  \___ \| | | | '_ \ / _ \ '__|  _ < / _ \ '_ \ / __| '_ \
  ____) | |_| | |_) |  __/ |  | |_) |  __/ | | | (__| | | |
 |_____/ \__,_| .__/ \___|_|  |____/ \___|_| |_|\___|_| |_|
              | |
              |_|

Welcome to the SB CLI!
```

## SuperBench CLI commands

The following lists `sb` commands usages and examples:

27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
### `sb benchmark list`

List benchmarks which match the regular expression.
```bash title="SB CLI"
sb benchmark list [--name]
```

#### Optional arguments

| Name          | Default | Description                           |
|---------------|---------|---------------------------------------|
| `--name` `-n` | `None`  | Benchmark name or regular expression. |

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

List all benchmarks:
```bash title="SB CLI"
sb benchmark list
```

List all benchmarks ending with "-bw":
```bash title="SB CLI"
sb benchmark list --name [a-z]+-bw
```

### `sb benchmark list-parameters`

List parameters for benchmarks which match the regular expression.
```bash title="SB CLI"
sb benchmark list-parameters [--name]
```

#### Optional arguments

| Name          | Default | Description                           |
|---------------|---------|---------------------------------------|
| `--name` `-n` | `None`  | Benchmark name or regular expression. |

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

List parameters for all benchmarks:
```bash title="SB CLI"
sb benchmark list-parameters
```

List parameters for all benchmarks which starts with "pytorch-":
```bash title="SB CLI"
sb benchmark list-parameters --name pytorch-[a-z]+
```

89
90
91
92
93
94
95
96
97
98
99
### `sb deploy`

Deploy the SuperBench environments to all managed nodes.
```bash title="SB CLI"
sb deploy [--docker-image]
          [--docker-password]
          [--docker-username]
          [--host-file]
          [--host-list]
          [--host-password]
          [--host-username]
100
          [--no-image-pull]
101
          [--output-dir]
102
103
104
105
106
          [--private-key]
```

#### Optional arguments

107
108
| Name                  | Default                 | Description                                                                       |
|-----------------------|-------------------------|-----------------------------------------------------------------------------------|
109
| `--docker-image` `-i` | `superbench/superbench` | Docker image URI, [here](./user-tutorial/container-images.mdx) listed all images. |
110
111
112
113
114
115
| `--docker-password`   | `None`                  | Docker registry password if authentication is needed.                             |
| `--docker-username`   | `None`                  | Docker registry username if authentication is needed.                             |
| `--host-file` `-f`    | `None`                  | Path to Ansible inventory host file.                                              |
| `--host-list` `-l`    | `None`                  | Comma separated host list.                                                        |
| `--host-password`     | `None`                  | Host password or key passphase if needed.                                         |
| `--host-username`     | `None`                  | Host username if needed.                                                          |
116
| `--no-image-pull`     | `False`                 | Skip pull and use local Docker image.                                             |
117
118
| `--output-dir`        | `None`                  | Path to output directory, outputs/{datetime} will be used if not specified.       |
| `--private-key`       | `None`                  | Path to private key if needed.                                                    |
119
120
121

#### Global arguments

122
123
124
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
125
126
127

#### Examples

128
Deploy default image on local GPU node:
129
```bash title="SB CLI"
130
131
132
133
134
135
sb deploy --host-list localhost
```

Deploy image `superbench/cuda:11.1` to all nodes in `./host.ini`:
```bash title="SB CLI"
sb deploy --docker-image superbench/cuda:11.1 --host-file ./host.ini
136
137
138
139
140
141
142
143
```

### `sb exec`

Execute the SuperBench benchmarks locally.
```bash title="SB CLI"
sb exec [--config-file]
        [--config-override]
144
        [--output-dir]
145
146
147
148
```

#### Optional arguments

149
150
151
152
153
| Name                     | Default | Description                                                                 |
|--------------------------|---------|-----------------------------------------------------------------------------|
| `--config-file` `-c`     | `None`  | Path to SuperBench config file.                                             |
| `--config-override` `-C` | `None`  | Extra arguments to override config_file.                                    |
| `--output-dir`           | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |
154
155
156

#### Global arguments

157
158
159
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
160
161
162
163
164
165
166
167

#### Examples

Execute GPT2 model benchmark in default configuration:
```bash title="SB CLI"
sb exec --config-override superbench.enable="['gpt2_models']"
```

168
169
170
171
172
### `sb result diagnosis`

Filter the defective machines automatically from benchmarking results according to rules defined in rule file.

```bash title="SB CLI"
173
174
175
176
177
sb result diagnosis --baseline-file
                    --data-file
                    --rule-file
                    [--decimal-place-value]
                    [--rule-file]
178
                    [--output-all]
179
180
                    [--output-dir]
                    [--output-file-format {excel, json, md, html}]
181
182
183
184
```

#### Required arguments

185
186
187
188
| Name               | Description            |
|--------------------|------------------------|
| `--data-file` `-d` | Path to raw data file. |
| `--rule-file` `-r` | Path to rule file.     |
189
190
191

#### Optional arguments

192
193
| Name                    | Default | Description                                                                 |
|-------------------------|---------|-----------------------------------------------------------------------------|
194
| `--baseline-file` `-b` | Path to baseline file. |
195
| `--decimal-place-value` | 2       | Number of valid decimal places to show in output. Default: 2.               |
196
| `--output-all`          | N/A     | Output diagnosis results for all nodes.                                     |
197
| `--output-dir`          | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |
198
| `--output-file-format`  | `excel` | Format of output file, 'excel', 'json', 'jsonl', 'md' or 'html'. Default: excel.     |
199
200
201
202
203
204
205
206
207
208
209

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

Run data diagnosis and output the results in excel format:
```bash title="SB CLI"
210
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format excel
211
212
```

213
Run data diagnosis and output the results in json format:
214
```bash title="SB CLI"
215
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format json
216
217
```

218
219
220
221
222
Run data diagnosis and output the results in jsonl format:
```bash title="SB CLI"
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format jsonl
```

223
224
225
226
227
Run data diagnosis and output the results in markdown format with 2 valid decimal places:
```bash title="SB CLI"
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format md --decimal-place-value 2
```

228
229
230
231
232
run data diagnosis and output the results of all nodes in json format:
```bash title="SB CLI"
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format json --output-all
```

233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
### `sb result summary`

Generate the readable summary report automatically from benchmarking results according to rules defined in rule file.

```bash title="SB CLI"
sb result summary --data-file
                  --rule-file
                  [--decimal-place-value]
                  [--output-dir]
                  [--output-file-format {md, excel, html}]
```

#### Required arguments

| Name               | Description            |
|--------------------|------------------------|
| `--data-file` `-d` | Path to raw data file. |
| `--rule-file` `-r` | Path to rule file.     |

#### Optional arguments

| Name                    | Default | Description                                                                 |
|-------------------------|---------|-----------------------------------------------------------------------------|
| `--decimal-place-value` | 2       | Number of valid decimal places to show in output. Default: 2.               |
| `--output-dir`          | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |
| `--output-file-format`  | `md`    | Format of output file, 'excel', 'md' or 'html'. Default: md.                |

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

268
Run result summary and output the results in markdown format with 2 valid decimal places:
269
270
271
272
```bash title="SB CLI"
sb result summary --data-file outputs/results-summary.jsonl --rule-file rule.yaml --output-file-format md --decimal-place-value 2
```

273
Run result summary and output the results in html format:
274
275
276
277
```bash title="SB CLI"
sb result summary --data-file outputs/results-summary.jsonl --rule-file rule.yaml --output-file-format html
```

278
279
280
281
282
283
284
285
286
287
288
289
290
### `sb run`

Run the SuperBench benchmarks distributedly.
```bash title="SB CLI"
sb run [--config-file]
       [--config-override]
       [--docker-image]
       [--docker-password]
       [--docker-username]
       [--host-file]
       [--host-list]
       [--host-password]
       [--host-username]
291
       [--no-docker]
292
       [--output-dir]
293
294
295
296
297
       [--private-key]
```

#### Optional arguments

298
299
300
301
302
303
304
305
306
307
308
| Name                     | Default                 | Description                                                                 |
|--------------------------|-------------------------|-----------------------------------------------------------------------------|
| `--config-file` `-c`     | `None`                  | Path to SuperBench config file.                                             |
| `--config-override` `-C` | `None`                  | Extra arguments to override config_file.                                    |
| `--docker-image` `-i`    | `superbench/superbench` | Docker image URI.                                                           |
| `--docker-password`      | `None`                  | Docker registry password if authentication is needed.                       |
| `--docker-username`      | `None`                  | Docker registry username if authentication is needed.                       |
| `--host-file` `-f`       | `None`                  | Path to Ansible inventory host file.                                        |
| `--host-list` `-l`       | `None`                  | Comma separated host list.                                                  |
| `--host-password`        | `None`                  | Host password or key passphase if needed.                                   |
| `--host-username`        | `None`                  | Host username if needed.                                                    |
309
| `--no-docker`            | `False`                 | Run on host directly without Docker.                                        |
310
311
| `--output-dir`           | `None`                  | Path to output directory, outputs/{datetime} will be used if not specified. |
| `--private-key`          | `None`                  | Path to private key if needed.                                              |
312
313
314

#### Global arguments

315
316
317
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
318
319
320

#### Examples

321
322
323
324
325
326
Run all benchmarks on local GPU node:
```bash title="SB CLI"
sb run --host-list localhost
```

Run all benchmarks on all managed nodes in `./host.ini` using image `superbench/cuda:11.1`
327
328
and default benchmarking configuration:
```bash title="SB CLI"
329
sb run --docker-image superbench/cuda:11.1 --host-file ./host.ini
330
331
```

332
333
334
335
336
337
Run kernel launch benchmarks on host directly without using Docker:
```bash title="SB CLI"
sb run --no-docker --host-list localhost --config-override \
  superbench.enable=kernel-launch superbench.env.SB_MICRO_PATH=/path/to/superbenchmark
```

338
339
340
341
342
343
344
345
346
### `sb version`

Print the current SuperBench CLI version.
```bash title="SB CLI"
sb version
```

#### Global arguments

347
348
349
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
350
351
352
353
354
355
356

#### Examples

Print version:
```bash title="SB CLI"
sb version
```