GitLab

Add ReduceType description into benchmarks doc.

**Description**
Set reduce type for current benchmarks' metrics, including model benchmarks and ShardingMatmul.

**Description**
Replace `sum(results) / len(results)` with `statistics.mean(results)`

**Description**
Add reduce function support for output summary.

**Major Revision**
- Add reducer class to maintain all reduce functions.
- Save reduce type of each metric into `BenchmarkResult`
- Fix UT.

**Description**
Add rocBLAS building logic in third_party.

**Major Revision**
- Add rocm_rocblas target in third_party/Makefile.
- Add rocblas building logic

**Description**
Add Executor and Benchmarks design doc

**Major Revision**
- Add Executor design doc
- Add Benchmarks design doc

**Description**
Add rccl bandwidth microbenchmark for rocm.

**Major Revision**
- Register rccl-bw benchmark.

**Description**
Support rocm in third_party/makefile and add rccl-tests as a submodule with building logic.

**Major Revision**
- Support rocm in third_party/makefile
- Add rccl-tests as a submodule 
- Add build logic in third_party/Makefile for rccl-tests 

__Description__
Cherry-pick bug fixes from v0.2.1 to main.

__Major Revisions__
* Fix bug of VGG models failed on A100 GPU with batch_size=128.
* Fix Ansible connection issue when running in localhost.
* Update version in packages and docs.

**Description**
Support rocm in third_party/makefile.

**Major Revision**
- Split rocm and cuda target in makefile
- Add target in dockerfile

**Description**
Add the source code of rocm kernel launch overhead benchmark. 

**Major Revision**
- Revise cmake build logic to support both cuda and rocm

**Description**
Support rocm cmake build. 

**Major Revision**
- Add  some envs in rocm_common.cmake to support rocm cmake build.

**Description**
Add NCCL performance microbenchmark.

**Major Revision**
- Add microbenchmark, example, test, config for NCCL

**Description**
Add RDMA Loopback performance microbenchmark.

**Major Revision**
- Add microbenchmark, example, test, config for RDMA Loopback

**Description**
Add disk performance microbenchmark.

**Major Revision**
- Add microbenchmark, example, test, config for disk performance.

**Minor Revision**
- Fix bugs in executor unit test related to default enabled tests.

**Description**
Add result contributing rules

**Description**
Add release process document.

**Description**
Fix bug of tests/executor/test_executor.py.

**Major Revision**
- Test default enabled benchmarks only instead of all benchmarks.

**Description**
Add FIO benchmark tool into third-party dependency.

**Major Revision**
- Add FIO submodule into third-party directory and modify Makefile to enable it.

Add perftest as a submodule and add build logic

Benchmarks: Build Pipeline - Add nccl-tests as a submodule and add build logic.

Benchmarks: Fix bug - fix bug of third_party/cuda-samples git checkout issue when building docker (#126)

* fix bug in docker build of third_party/cuda-samples

Add microbenchmark, example, test, config for cuda memory performance and Add cuda-samples(tag with cuda version) as git submodule and update related makefile

Update network common utils. Add get_ib_devices in network common utils and move get_free_port from test utils to network common utils

* Bug Fix - Fix race condition issue for multi ranks (#117)

Fix race condition issue when multi ranks rotating the same directory.

* Update pipeline for release branch (#122)

* Bug Fix - Fix bug when convert bool config to store_true argument. (#120)
Co-authored-by: Yifan Xiong <yifan.xiong@microsoft.com>

* add validation config file for AMD MI100

Support both NVIDIA and AMD GPU and check GPU vendor during deployment and execution.

* Add GPU environment check in sb deploy.
* Check GPU vendor in executor.

Update README and version for v0.2 release.

Fetch benchmarks results on all nodes, will rsync after each benchmark.
The results directory structure on control node is as follows:

```
outputs/
└── datetime
    ├── nodes
    │   └── node-0
    │       ├── benchmarks
    │       │   ├── benchmark-0
    │       │   │   ├── rank-0
    │       │   │   │   └── results.json
    │       └── sb-exec.log
    ├── sb-run.log
    └── sb.config.yaml
```

Support `--host-list` for deploy and run commands.

Before this change, an inventory file is needed to use `sb deploy/run`.
Now, `--host-list localhost` or `-l localhost` is sufficient for quick try.

* Support custom output directory.
* Update document.

* Add introduction and release documents.
* Fix some typos in documents.

* fix bug for nvidia v100
* hard code the supported dict for different arch.

* add config file for ndv4.

* replace torch.optim.AdamW with transformers.AdamW.

* Initialize SuperBench website.
* Add GitHub Actions for automatically build and publish.

Update SuperBench documents.

* Add `sb deploy` command content.
* Fix inline if-expression syntax in playbook.
* Fix quote escape issue in bash command.
* Add custom env in config.
* Update default config for multi GPU benchmarks.
* Update MANIFEST.in to include jinja2 template.
* Require jinja2 minimum version.
* Fix occasional duplicate output in Ansible runner.
* Fix mixed color from Ansible and Python colorlog.
* Update according to comments.
* Change superbench.env from list to dict in config file.