Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
model-benchmarks.md 6.61 KB
Newer Older
Yifan Xiong's avatar
Docs - Update SuperBench documents (#101)  
Yifan Xiong committed
1
2
3
4
5
6 
---
id: model-benchmarks
---

# Model Benchmarks
Yifan Xiong's avatar
Docs - Add introduction and metrics in benchmarks docs (#233)  
Yifan Xiong committed
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95 
## PyTorch Model Benchmarks

### `gpt_models`

#### Introduction

Run training or inference tasks with single or half precision for GPT models,
including gpt2-small, gpt2-medium, gpt2-large and gpt2-xl.

#### Metrics

| Name                                                          | Unit                   | Description                                 |
|---------------------------------------------------------------|------------------------|---------------------------------------------|
| gpt_models/pytorch-${model_name}/steptime_train_float32       | time (ms)              | Train step time with single precision.      |
| gpt_models/pytorch-${model_name}/throughput_train_float32     | throughput (samples/s) | Train throughput with single precision.     |
| gpt_models/pytorch-${model_name}/steptime_inference_float32   | time (ms)              | Inference step time with single precision.  |
| gpt_models/pytorch-${model_name}/throughput_inference_float32 | throughput (samples/s) | Inference throughput with single precision. |
| gpt_models/pytorch-${model_name}/steptime_train_float16       | time (ms)              | Train step time with half precision.        |
| gpt_models/pytorch-${model_name}/throughput_train_float16     | throughput (samples/s) | Train throughput with half precision.       |
| gpt_models/pytorch-${model_name}/steptime_inference_float16   | time (ms)              | Inference step time with half precision.    |
| gpt_models/pytorch-${model_name}/throughput_inference_float16 | throughput (samples/s) | Inference throughput with half precision.   |

### `bert_models`

#### Introduction

Run training or inference tasks with single or half precision for BERT models, including bert-base and bert-large.

#### Metrics

| Name                                                           | Unit                   | Description                                 |
|----------------------------------------------------------------|------------------------|---------------------------------------------|
| bert_models/pytorch-${model_name}/steptime_train_float32       | time (ms)              | Train step time with single precision.      |
| bert_models/pytorch-${model_name}/throughput_train_float32     | throughput (samples/s) | Train throughput with single precision.     |
| bert_models/pytorch-${model_name}/steptime_inference_float32   | time (ms)              | Inference step time with single precision.  |
| bert_models/pytorch-${model_name}/throughput_inference_float32 | throughput (samples/s) | Inference throughput with single precision. |
| bert_models/pytorch-${model_name}/steptime_train_float16       | time (ms)              | Train step time with half precision.        |
| bert_models/pytorch-${model_name}/throughput_train_float16     | throughput (samples/s) | Train throughput with half precision.       |
| bert_models/pytorch-${model_name}/steptime_inference_float16   | time (ms)              | Inference step time with half precision.    |
| bert_models/pytorch-${model_name}/throughput_inference_float16 | throughput (samples/s) | Inference throughput with half precision.   |

### `lstm_models`

#### Introduction

Run training or inference tasks with single or half precision for one bidirectional LSTM model.

#### Metrics

| Name                                                  | Unit                   | Description                                 |
|-------------------------------------------------------|------------------------|---------------------------------------------|
| lstm_models/pytorch-lstm/steptime_train_float32       | time (ms)              | Train step time with single precision.      |
| lstm_models/pytorch-lstm/throughput_train_float32     | throughput (samples/s) | Train throughput with single precision.     |
| lstm_models/pytorch-lstm/steptime_inference_float32   | time (ms)              | Inference step time with single precision.  |
| lstm_models/pytorch-lstm/throughput_inference_float32 | throughput (samples/s) | Inference throughput with single precision. |
| lstm_models/pytorch-lstm/steptime_train_float16       | time (ms)              | Train step time with half precision.        |
| lstm_models/pytorch-lstm/throughput_train_float16     | throughput (samples/s) | Train throughput with half precision.       |
| lstm_models/pytorch-lstm/steptime_inference_float16   | time (ms)              | Inference step time with half precision.    |
| lstm_models/pytorch-lstm/throughput_inference_float16 | throughput (samples/s) | Inference throughput with half precision.   |

### `cnn_models`

#### Introduction

Run training or inference tasks with single or half precision for CNN models listed in
[`torchvision.models`](https://pytorch.org/vision/0.8/models.html), including:
* resnet: resnet18, resnet34, resnet50, resnet101, resnet152
* resnext: resnext50_32x4d, resnext101_32x8d
* wide_resnet: wide_resnet50_2, wide_resnet101_2
* densenet: densenet121, densenet169, densenet201, densenet161
* vgg: vgg11, vgg11_bn, vgg13, vgg13_bn, vgg16, vgg16_bn, vgg19_bn, vgg19
* mnasnet: mnasnet0_5, mnasnet0_75, mnasnet1_0, mnasnet1_3
* mobilenet: mobilenet_v2
* shufflenet: shufflenet_v2_x0_5, shufflenet_v2_x1_0, shufflenet_v2_x1_5, shufflenet_v2_x2_0
* squeezenet: squeezenet1_0, squeezenet1_1
* others: alexnet, googlenet, inception_v3

#### Metrics

| Name                                                          | Unit                   | Description                                 |
|---------------------------------------------------------------|------------------------|---------------------------------------------|
| cnn_models/pytorch-${model_name}/steptime_train_float32       | time (ms)              | Train step time with single precision.      |
| cnn_models/pytorch-${model_name}/throughput_train_float32     | throughput (samples/s) | Train throughput with single precision.     |
| cnn_models/pytorch-${model_name}/steptime_inference_float32   | time (ms)              | Inference step time with single precision.  |
| cnn_models/pytorch-${model_name}/throughput_inference_float32 | throughput (samples/s) | Inference throughput with single precision. |
| cnn_models/pytorch-${model_name}/steptime_train_float16       | time (ms)              | Train step time with half precision.        |
| cnn_models/pytorch-${model_name}/throughput_train_float16     | throughput (samples/s) | Train throughput with half precision.       |
| cnn_models/pytorch-${model_name}/steptime_inference_float16   | time (ms)              | Inference step time with half precision.    |
| cnn_models/pytorch-${model_name}/throughput_inference_float16 | throughput (samples/s) | Inference throughput with half precision.   |