GitLab

__Description__

Refine test cases for microbenchmark:
* Refine test fixture, add BenchmarkTestCase class.
* Refine test data.
* Resolve no numa issue for test_ib_loopback_util case.

__Description__

Fix issues for Ansible and benchmarks:
* Cleanup Ansible runner private data dir to avoid out of disk space issue when node number is large.
* Support both absolute and relative paths when fecth results.
* Use a deterministic image in Ansible test to avoid image update.
* Update logging format.
* Delete torch models and inputs after export.

**Description**
Add usage for data diagnosis.

**Description**
Update docs for monitor.

Add transformers for TensorRT inference.

**Description**
Add benchmark metrics for cpu-memory-bw-latency.

**Description**
Benchmarks: Fix Comment - Correct benchmark name in test_gpu_copy_bw_performance.py.

**Description**
Add mlc memory bandwidth and latency micro benchmark to Superbench.

**Major Revision**
- Add mlc benchmark with test and example files

**Description**
Minor doc change to highlight sb CLI version is independent of the sb container version.

**Description**
Add ONNXRuntime inference benchmark based on ORT python API.

**Major Revision**
- Add `ORTInferenceBenchmark` class to export pytorch model to onnx model and do inference
- Add tests and example for `ort-inference` benchmark
- Update the introduction docs.

**Description**
Add basic analysis features.

**Major Revision**
- Add statistics, correlations of the raw data
- Add numeric outlier detection(inter_quartile_range)
- Add boxplot for selected metric

**Description**
Integrate monitor into Superbench.

**Major Revision**
- Initialize, start and stop monitor in SB executor.
- Parse the monitor data in SB runner and merge into benchmark results.
- Specify ReduceType for monitor metrics, such as MAX, MIN and LAST.
- Add monitor configs into config file.

**Description**
Set the `reduce_op` type for metirc `return_code` as `None`.

**Description**
Add cli to integrate data diagnosis module.

**Description**
Unify metric names of benchmarks.

**Description**
Add data diagnosis module.

**Major Revision**
- Add DataDiagnosis class to support rule-based data diagnosis for result summary jsonl file of multi nodes
- Add RuleOp class to define rule operators

Fix issues for distributed runs:
* fix config for memory bandwidth benchmarks
* add throttling for high concurrency docker pull
* update rsync path and exclude directories
* handle exceptions when creating summary
* tune for logging

**Description**
Add return_code metric into result and revise unit tests.

**Description**
Add doc for data diagnosis, including input, output and baseline file schema.

Add config file for Azure NDm A100 v4 SKU.

**Description**
Add gpt-small into config files.

**Description**
If `ignore_invalid` is True, and 'required' arguments are not set when register the benchmark, the arguments should be provided by user in config and skip the arguments checking.

**Description**
Replace `-c` argument with `-N` for `numactl` since the old `-c`/`--cpubind` argument is deprecated.

**Description**
Upgrade FIO benchmark tool from 3.27 to 3.28.

**Description**
Update ib validtion mirobenchmark metrics.

Bumps [algoliasearch-helper](https://github.com/algolia/algoliasearch-helper-js) from 3.5.5 to 3.6.2.
- [Release notes](https://github.com/algolia/algoliasearch-helper-js/releases)
- [Changelog](https://github.com/algolia/algoliasearch-helper-js/blob/develop/CHANGELOG)
- [Commits](https://github.com/algolia/algoliasearch-helper-js/compare/3.5.5...3.6.2

)

---
updated-dependencies:
- dependency-name: algoliasearch-helper
  dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

**Description**

Update gpu-copy benchmark metrics.

**Description**
Fix typo in description of kernel_launch_overhead.py

**Description**
Add the initial version of Monitor.

**Major Revision**
- Add `Monitor` class to launch background process for monitoring.
- Add `MonitorRecord` class to save the data one time capturing.

**Description**
Rename `nvidia_helper` utility as `device_manager` module and support more functions:
```
device_manager.get_device_count()
device_manager.get_device_utilization(idx)
device_manager.get_device_temperature(idx)
device_manager.get_device_power_limit(idx)
device_manager.get_device_memory(idx)
device_manager.get_device_row_remapped_info(idx)
device_manager.get_device_ecc_error(idx)
```

__Description__

Add TensorRT inference benchmark for torchvision models.

__Major Revision__
- Measure TensorRT inference performance.

**Description**

Update docs to add network benchmarks for tcp and gpcnet.

**Description**
Add ib traffic validation distributed benchmark.

**Major Revision**
- Add ib traffic validation distributed benchmark, example and test

Update docs to add ORT AMD benchmarks based on docker.

**Description**
This commit does the following:
1) Adds CPU-initiated copy benchmark;
2) Adds dtod benchmark;
3) Support scanning NUMA nodes and GPUs inside the benchmark program;
4) Change the name of gpu-sm-copy to gpu-copy.

**Description**
This commit fixes the URL of ROCm GPG file.

Add introduction and metrics for micro-benchmarks and model-benchmarks document.

Add RocmOnnxModelBenchmark class to run benchmarks packaged in superbench/benchmark:rocm4.3.1-onnxruntime1.9.0

**Description**
Add gpcnet microbenchmark

**Major Revision**
- add 2 microbenmark for gpcnet, gpc-network-test, gpc-network-load-test
- add related test and example file

Description
Add CudaDockerBenchmark and RocmDockerBenchmark to support amd and cuda platform for DockerBenchmark.