Unverified Commit 6ecded20 authored by Yuting Jiang's avatar Yuting Jiang Committed by GitHub
Browse files

Docs - Unify metric and add doc for cublas and cudnn functions (#271)

**Description**
Unify metric and add doc for cuBLAS and cuDNN functions.
parent 74625527
...@@ -60,11 +60,40 @@ Large scale matmul operation using `torch.matmul` with one GPU. ...@@ -60,11 +60,40 @@ Large scale matmul operation using `torch.matmul` with one GPU.
### `cublas-function` ### `cublas-function`
TODO #### Introduction
Measure the performance of most common Nvidia cuBLAS functions with parameters in models training including ResNet, VGG, DenseNet, LSTM, BERT, and GPT-2.
The supported functions for cuBLAS are as follows:
- cublasSgemm
- cublasSgemmStridedBatched
- cublasGemmStridedBatchedEx
- cublasGemmEx
- cublasCgemm3mStridedBatched
- cublasCgemm
#### Metrics
| Name | Unit | Description |
|----------------------------------------------------------|-----------|-------------------------------------------------------------------|
| cublas-function/name_${function_name}_${parameters}_time | time (us) | The mean time to execute the cublas function with the parameters. |
### `cudnn-function` ### `cudnn-function`
TODO #### Introduction
Measure the performance of most common Nvidia cuDNN functions with parameters in models training including ResNet, VGG, DenseNet, LSTM, BERT, and GPT-2.
The supported functions for cuDNN are as follows:
- cudnnConvolutionBackwardFilter
- cudnnConvolutionBackwardData
- cudnnConvolutionForward
#### Metrics
| Name | Unit | Description |
|---------------------------------------------------------|-----------|------------------------------------------------------------------|
| cudnn-function/name_${function_name}_${parameters}_time | time (us) | The mean time to execute the cudnn function with the parameters. |
### `tensorrt-inference` ### `tensorrt-inference`
......
...@@ -291,8 +291,8 @@ def _process_raw_result(self, cmd_idx, raw_output): ...@@ -291,8 +291,8 @@ def _process_raw_result(self, cmd_idx, raw_output):
raw_data = raw_data.split(',') raw_data = raw_data.split(',')
raw_data.pop() raw_data.pop()
raw_data = [float(item) for item in raw_data] raw_data = [float(item) for item in raw_data]
self._result.add_result(metric, statistics.mean(raw_data)) self._result.add_result(metric.lower() + '_time', statistics.mean(raw_data))
self._result.add_raw_data(metric, raw_data) self._result.add_raw_data(metric.lower() + '_time', raw_data)
if 'Error' in line: if 'Error' in line:
error = True error = True
except BaseException as e: except BaseException as e:
......
...@@ -6,6 +6,7 @@ ...@@ -6,6 +6,7 @@
import os import os
import json import json
import yaml import yaml
import statistics
from superbench.common.utils import logger from superbench.common.utils import logger
from superbench.benchmarks import Platform, BenchmarkRegistry, ReturnCode from superbench.benchmarks import Platform, BenchmarkRegistry, ReturnCode
...@@ -424,8 +425,8 @@ def _process_raw_result(self, cmd_idx, raw_output): ...@@ -424,8 +425,8 @@ def _process_raw_result(self, cmd_idx, raw_output):
raw_data = raw_data.split(',') raw_data = raw_data.split(',')
raw_data.pop() raw_data.pop()
raw_data = [float(item) for item in raw_data] raw_data = [float(item) for item in raw_data]
self._result.add_result(metric, sum(raw_data) / len(raw_data)) self._result.add_result(metric.lower() + '_time', statistics.mean(raw_data) * 1000)
self._result.add_raw_data(metric, raw_data) self._result.add_raw_data(metric.lower() + '_time', raw_data)
if 'Error' in line: if 'Error' in line:
error = True error = True
except BaseException as e: except BaseException as e:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment