GitLab

* Bug Fix - Fix race condition issue for multi ranks (#117)

Fix race condition issue when multi ranks rotating the same directory.

* Update pipeline for release branch (#122)

* Bug Fix - Fix bug when convert bool config to store_true argument. (#120)
Co-authored-by: Yifan Xiong <yifan.xiong@microsoft.com>

Support both NVIDIA and AMD GPU and check GPU vendor during deployment and execution.

* Add GPU environment check in sb deploy.
* Check GPU vendor in executor.

Fetch benchmarks results on all nodes, will rsync after each benchmark.
The results directory structure on control node is as follows:

```
outputs/
└── datetime
    ├── nodes
    │   └── node-0
    │       ├── benchmarks
    │       │   ├── benchmark-0
    │       │   │   ├── rank-0
    │       │   │   │   └── results.json
    │       └── sb-exec.log
    ├── sb-run.log
    └── sb.config.yaml
```

* Support custom output directory.
* Update document.

* fix bug for nvidia v100
* hard code the supported dict for different arch.

* replace torch.optim.AdamW with transformers.AdamW.

* Add `sb deploy` command content.
* Fix inline if-expression syntax in playbook.
* Fix quote escape issue in bash command.
* Add custom env in config.
* Update default config for multi GPU benchmarks.
* Update MANIFEST.in to include jinja2 template.
* Require jinja2 minimum version.
* Fix occasional duplicate output in Ansible runner.
* Fix mixed color from Ansible and Python colorlog.
* Update according to comments.
* Change superbench.env from list to dict in config file.

Fix bugs and refine log in single GPU benchmarks:

* Fix none framework issue
* Fix empty parameter bug
* Remove missed mobilenet_v3 models
* Change benchmark registration log to debug level
* Add pid in logging
* Add missing benchmarks in default config
* Fix deprecated logging warn

* Clean up the cache.

* fix return code reset issue

* Support local mode in runner.

* Change default shape of sharding-matmul.

* add cuda flops performance benchmark.

* add python related cudnn microbenchmark

* Save benchmark results to json file.

* add benchmark for cublas test

* format

* revise error handling and test

* add interface to read json file, revise json file path and include .json in packaging

* add random_seed in arguments

* revise preprocess of cublas benchmark

* fix lint error and note error in source code

* update according comments

* revise input arguments from json file to custom str and convert json file to built-in dict list

* restore package config

* fit lint issue

* update platform and comments

* rename files to match source code dir and fix comments error
Co-authored-by: root <root@sb-validation-000001.51z1chmys5fuzfqyo4niepozre.bx.internal.cloudapp.net>

* Support `torch.distributed` mode in runner.
* Support given `proc_num` and `node_num` in `torch.distributed` mode.

* Add integration tests for Ansible playbooks
* Add `gpu_vendor` var to bypass gpu mount

Implement ansible client and runner:
* add ansible client
* add deploy and check_env playbooks

* add kernel launch overhead benchmark.

* use absolute path of input file
* parse registry uri from image
* merge common parts for arguments processing

* Revise MicroBenchmark class to be more flexible.
* use command index not the command as the parameter.
* changes according to discussion.

* Benchmarks: Add Benchmark - Add LSTM model benchmarks.

* Benchmarks: Add Benchmark - Add CNN model benchmarks.

Benchmarks: Code Revision - Fix some issue for BERT benchmark. (#58)

* Benchmarks: Add Benchmark - Add GPT2 model benchmark.

* Benchmarks: Add Feature - Add interface to get all predefine parameters of all benchmarks.

* Benchmarks: Add Benchmark - add computation and communication overlap micro benchmark

* Benchmarks: Add benchmark - fix some format issues and typo

* Benchmarks: Add Benchmark - update according comments and add test

* revise tests

* skip multi gpu test due to no multi gpu
Co-authored-by: v-yujiang <v-yujiang@microsoft.com>

* Superbenchmark: Revise tests - revise benchmark test util to support multi gpu test

* modify test_sharding_matmul.py to match the tests util

* fix missing package in dockerfile
* update benchmark list and parameters
* catch runtime errors
* refine logging info

* add tests for matmul and sharding-matmul benchmarks.

* add decorator for sharding_matmul tests.

* add __init__.py for utils of benchmarks tests.

* disable GPU tests for CPU platform validation.

* fix typo
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>
Co-authored-by: Peng Cheng <chengpeng5555@outlook.com>
Co-authored-by: Yifan Xiong <yifan.xiong@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>
Co-authored-by: Yifan Xiong <yifan.xiong@microsoft.com>

* CLI integration with Executor and Runner

* init runner class with not implemented

* skip unnecessary tests according to env var
* remove useless tests

Add superbench executor class

* add executor class
* update default config to exec benchmarks
* add micro benchmarks and model benchmarks

Benchmarks: Code Revision - Revise BenchmarkRegistry interfaces for integration with executor. (#33)

* revise BenchmarkRegistry interfaces.
* address comments
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Benchmarks: Add Benchmark - Add Pytorch BERT benchmarks, including bert-base and bert-large.   (#20)

* add pytorch bert benchmarks.

* revise code

* fix issue

* revise code.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* move benchmarks registration from registry.py to __init__.py

* revise __init__.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>