cli.md 17.3 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
---
id: cli
---

# CLI

SuperBench provides a command line interface to help you use, deploy and run benchmarks.
```
$ sb

   _____                       ____                  _
  / ____|                     |  _ \                | |
 | (___  _   _ _ __   ___ _ __| |_) | ___ _ __   ___| |__
  \___ \| | | | '_ \ / _ \ '__|  _ < / _ \ '_ \ / __| '_ \
  ____) | |_| | |_) |  __/ |  | |_) |  __/ | | | (__| | | |
 |_____/ \__,_| .__/ \___|_|  |____/ \___|_| |_|\___|_| |_|
              | |
              |_|

Welcome to the SB CLI!
```

## SuperBench CLI commands

The following lists `sb` commands usages and examples:

27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
### `sb benchmark list`

List benchmarks which match the regular expression.
```bash title="SB CLI"
sb benchmark list [--name]
```

#### Optional arguments

| Name          | Default | Description                           |
|---------------|---------|---------------------------------------|
| `--name` `-n` | `None`  | Benchmark name or regular expression. |

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

List all benchmarks:
```bash title="SB CLI"
sb benchmark list
```

List all benchmarks ending with "-bw":
```bash title="SB CLI"
sb benchmark list --name [a-z]+-bw
```

### `sb benchmark list-parameters`

List parameters for benchmarks which match the regular expression.
```bash title="SB CLI"
sb benchmark list-parameters [--name]
```

#### Optional arguments

| Name          | Default | Description                           |
|---------------|---------|---------------------------------------|
| `--name` `-n` | `None`  | Benchmark name or regular expression. |

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

List parameters for all benchmarks:
```bash title="SB CLI"
sb benchmark list-parameters
```

List parameters for all benchmarks which starts with "pytorch-":
```bash title="SB CLI"
sb benchmark list-parameters --name pytorch-[a-z]+
```

89
90
91
92
### `sb deploy`

Deploy the SuperBench environments to all managed nodes.
```bash title="SB CLI"
93
94
sb deploy [--container-name]
          [--docker-image]
95
96
97
98
99
100
          [--docker-password]
          [--docker-username]
          [--host-file]
          [--host-list]
          [--host-password]
          [--host-username]
101
          [--no-image-pull]
102
          [--output-dir]
103
104
105
106
107
          [--private-key]
```

#### Optional arguments

108
109
| Name                  | Default                 | Description                                                                       |
|-----------------------|-------------------------|-----------------------------------------------------------------------------------|
110
| `--container-name`    | `sb-workspace`          | Docker container name.                                                            |
111
| `--docker-image` `-i` | `superbench/superbench` | Docker image URI, [here](./user-tutorial/container-images.mdx) listed all images. |
112
113
114
115
| `--docker-password`   | `None`                  | Docker registry password if authentication is needed.                             |
| `--docker-username`   | `None`                  | Docker registry username if authentication is needed.                             |
| `--host-file` `-f`    | `None`                  | Path to Ansible inventory host file.                                              |
| `--host-list` `-l`    | `None`                  | Comma separated host list.                                                        |
omahs's avatar
omahs committed
116
| `--host-password`     | `None`                  | Host password or key passphrase if needed.                                        |
117
| `--host-username`     | `None`                  | Host username if needed.                                                          |
118
| `--no-image-pull`     | `False`                 | Skip pull and use local Docker image.                                             |
119
120
| `--output-dir`        | `None`                  | Path to output directory, outputs/{datetime} will be used if not specified.       |
| `--private-key`       | `None`                  | Path to private key if needed.                                                    |
121
122
123

#### Global arguments

124
125
126
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
127
128
129

#### Examples

130
Deploy default image on local GPU node:
131
```bash title="SB CLI"
132
133
134
sb deploy --host-list localhost
```

135
136
137
138
139
Deploy using custom container name:
```bash title="SB CLI"
sb deploy --host-list localhost --container-name my-benchmark
```

140
141
142
Deploy image `superbench/cuda:11.1` to all nodes in `./host.ini`:
```bash title="SB CLI"
sb deploy --docker-image superbench/cuda:11.1 --host-file ./host.ini
143
144
145
146
147
148
149
150
```

### `sb exec`

Execute the SuperBench benchmarks locally.
```bash title="SB CLI"
sb exec [--config-file]
        [--config-override]
151
        [--output-dir]
152
153
154
155
```

#### Optional arguments

156
157
158
159
160
| Name                     | Default | Description                                                                 |
|--------------------------|---------|-----------------------------------------------------------------------------|
| `--config-file` `-c`     | `None`  | Path to SuperBench config file.                                             |
| `--config-override` `-C` | `None`  | Extra arguments to override config_file.                                    |
| `--output-dir`           | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |
161
162
163

#### Global arguments

164
165
166
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
167
168
169
170
171
172
173
174

#### Examples

Execute GPT2 model benchmark in default configuration:
```bash title="SB CLI"
sb exec --config-override superbench.enable="['gpt2_models']"
```

175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
### `sb node info`
Get system info on the local node.

```bash title="SB CLI"
sb node info [--output-dir]
```

#### Optional arguments

| Name           | Default | Description                                                                 |
|----------------|---------|-----------------------------------------------------------------------------|
| `--output-dir` | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |

#### Examples

Get system info on the local node and save it into the `outputs` dir:
```bash title="SB CLI"
sb node info --output-dir outputs
```

195
196
197
198
199
### `sb result diagnosis`

Filter the defective machines automatically from benchmarking results according to rules defined in rule file.

```bash title="SB CLI"
200
201
202
203
204
sb result diagnosis --baseline-file
                    --data-file
                    --rule-file
                    [--decimal-place-value]
                    [--rule-file]
205
                    [--output-all]
206
207
                    [--output-dir]
                    [--output-file-format {excel, json, md, html}]
208
209
210
211
```

#### Required arguments

212
213
214
215
| Name               | Description            |
|--------------------|------------------------|
| `--data-file` `-d` | Path to raw data file. |
| `--rule-file` `-r` | Path to rule file.     |
216
217
218

#### Optional arguments

219
220
| Name                    | Default | Description                                                                 |
|-------------------------|---------|-----------------------------------------------------------------------------|
221
| `--baseline-file` `-b` | Path to baseline file. |
222
| `--decimal-place-value` | 2       | Number of valid decimal places to show in output. Default: 2.               |
223
| `--output-all`          | N/A     | Output diagnosis results for all nodes.                                     |
224
| `--output-dir`          | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |
225
| `--output-file-format`  | `excel` | Format of output file, 'excel', 'json', 'jsonl', 'md' or 'html'. Default: excel.     |
226
227
228
229
230
231
232
233
234
235
236

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

Run data diagnosis and output the results in excel format:
```bash title="SB CLI"
237
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format excel
238
239
```

240
Run data diagnosis and output the results in json format:
241
```bash title="SB CLI"
242
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format json
243
244
```

245
246
247
248
249
Run data diagnosis and output the results in jsonl format:
```bash title="SB CLI"
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format jsonl
```

250
251
252
253
254
Run data diagnosis and output the results in markdown format with 2 valid decimal places:
```bash title="SB CLI"
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format md --decimal-place-value 2
```

255
256
257
258
259
run data diagnosis and output the results of all nodes in json format:
```bash title="SB CLI"
sb result diagnosis --data-file outputs/results-summary.jsonl --rule-file rule.yaml --baseline-file baseline.json --output-file-format json --output-all
```

260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
### `sb result summary`

Generate the readable summary report automatically from benchmarking results according to rules defined in rule file.

```bash title="SB CLI"
sb result summary --data-file
                  --rule-file
                  [--decimal-place-value]
                  [--output-dir]
                  [--output-file-format {md, excel, html}]
```

#### Required arguments

| Name               | Description            |
|--------------------|------------------------|
| `--data-file` `-d` | Path to raw data file. |
| `--rule-file` `-r` | Path to rule file.     |

#### Optional arguments

| Name                    | Default | Description                                                                 |
|-------------------------|---------|-----------------------------------------------------------------------------|
| `--decimal-place-value` | 2       | Number of valid decimal places to show in output. Default: 2.               |
| `--output-dir`          | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |
| `--output-file-format`  | `md`    | Format of output file, 'excel', 'md' or 'html'. Default: md.                |

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

295
Run result summary and output the results in markdown format with 2 valid decimal places:
296
297
298
299
```bash title="SB CLI"
sb result summary --data-file outputs/results-summary.jsonl --rule-file rule.yaml --output-file-format md --decimal-place-value 2
```

300
Run result summary and output the results in html format:
301
302
303
304
```bash title="SB CLI"
sb result summary --data-file outputs/results-summary.jsonl --rule-file rule.yaml --output-file-format html
```

305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
### `sb result generate-baseline`

Generate the baseline file automatically from multiple machines results according to rules defined in rule file.

```bash title="SB CLI"
sb result generate-baseline --data-file
                            --summary-rule-file
                            [--diagnosis-rule-file]
                            [--baseline-file]
                            [--decimal-place-value]
                            [--output-dir]
```

#### Required arguments

| Name                        | Description                             |
|-----------------------------|-----------------------------------------|
| `--data-file` `-d`          | Path to raw data file.                  |
| `--summary-rule-file` `-sr` | Path to summary rule file.              |

#### Optional arguments

| Name                          | Default | Description                                                                 |
|-------------------------------|---------|-----------------------------------------------------------------------------|
| `--diagnosis-rule-file` `-dr` | `None`  | Path to diagnosis rule file. Default: None.                                 |
| `--baseline-file` `-b`        | `None`  | Path to previous baseline file. Default: None.                              |
| `--decimal-place-value`       | 2       | Number of valid decimal places to show in output. Default: 2.               |
| `--output-dir`                | `None`  | Path to output directory, outputs/{datetime} will be used if not specified. |

#### Global arguments

| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |

#### Examples

Run result generate-baseline to generate baseline.json file:
```bash title="SB CLI"
sb result generate-baseline --data-file outputs/results-summary.jsonl --summary-rule-file summary-rule.yaml --diagnosis-rule-file diagnosis-rule.yaml
```

Run result generate-baseline and merge with previous baseline:
```bash title="SB CLI"
sb result generate-baseline --data-file outputs/results-summary.jsonl --summary-rule-file summary-rule.yaml --diagnosis-rule-file diagnosis-rule.yaml --baseline-file previous-baseline.json
```

352
353
354
355
356
357
### `sb run`

Run the SuperBench benchmarks distributedly.
```bash title="SB CLI"
sb run [--config-file]
       [--config-override]
358
       [--container-name]
359
360
361
       [--docker-image]
       [--docker-password]
       [--docker-username]
362
       [--get-info]
363
364
365
366
       [--host-file]
       [--host-list]
       [--host-password]
       [--host-username]
367
       [--no-docker]
368
       [--output-dir]
369
370
371
372
373
       [--private-key]
```

#### Optional arguments

374
375
376
377
| Name                     | Default                 | Description                                                                 |
|--------------------------|-------------------------|-----------------------------------------------------------------------------|
| `--config-file` `-c`     | `None`                  | Path to SuperBench config file.                                             |
| `--config-override` `-C` | `None`                  | Extra arguments to override config_file.                                    |
378
| `--container-name`       | `sb-workspace`          | Docker container name.                                                      |
379
380
381
| `--docker-image` `-i`    | `superbench/superbench` | Docker image URI.                                                           |
| `--docker-password`      | `None`                  | Docker registry password if authentication is needed.                       |
| `--docker-username`      | `None`                  | Docker registry username if authentication is needed.                       |
382
| `--get-info`             | `False`                 | Collect system info.                                                        |
383
384
| `--host-file` `-f`       | `None`                  | Path to Ansible inventory host file.                                        |
| `--host-list` `-l`       | `None`                  | Comma separated host list.                                                  |
omahs's avatar
omahs committed
385
| `--host-password`        | `None`                  | Host password or key passphrase if needed.                                  |
386
| `--host-username`        | `None`                  | Host username if needed.                                                    |
387
| `--no-docker`            | `False`                 | Run on host directly without Docker.                                        |
388
389
| `--output-dir`           | `None`                  | Path to output directory, outputs/{datetime} will be used if not specified. |
| `--private-key`          | `None`                  | Path to private key if needed.                                              |
390
391
392

#### Global arguments

393
394
395
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
396
397
398

#### Examples

399
400
401
402
403
Run all benchmarks on local GPU node:
```bash title="SB CLI"
sb run --host-list localhost
```

404
405
406
407
408
Run all benchmarks using custom container name:
```bash title="SB CLI"
sb run --host-list localhost --container-name my-benchmark
```

409
Run all benchmarks on all managed nodes in `./host.ini` using image `superbench/cuda:11.1`
410
411
and default benchmarking configuration:
```bash title="SB CLI"
412
sb run --docker-image superbench/cuda:11.1 --host-file ./host.ini
413
414
```

415
416
417
418
419
420
Run kernel launch benchmarks on host directly without using Docker:
```bash title="SB CLI"
sb run --no-docker --host-list localhost --config-override \
  superbench.enable=kernel-launch superbench.env.SB_MICRO_PATH=/path/to/superbenchmark
```

421
422
423
424
425
426
427
428
429
430
Collect system info on all nodes in ./host.ini" distributed without running benchmarks:
```bash title="SB CLI"
sb run --get-info --host-file ./host.ini -C superbench.enable=none
```

Collect system info on all nodes in ./host.ini" distributed while running benchmarks:
```bash title="SB CLI"
sb run --get-info --host-file ./host.ini
```

431
432
433
434
435
436
437
438
439
### `sb version`

Print the current SuperBench CLI version.
```bash title="SB CLI"
sb version
```

#### Global arguments

440
441
442
| Name          | Default | Description        |
|---------------|---------|--------------------|
| `--help` `-h` | N/A     | Show help message. |
443
444
445
446
447
448
449

#### Examples

Print version:
```bash title="SB CLI"
sb version
```