GitLab

__Description__

Upgrade version and release note.

__Major Revision__

- Upgrade package versions
- Add release note for v0.7.0

Support `--no-image-pull` in `sb deploy` to skip image pull to use local
cached images.

**Description**
Fix bug for incorrect datatype judgement in cublas-function source code.

**Description**
Update benchmark doc of cublaslt, cublas-function and model benchmarks

Fix bug in TE BERT model, create different instances for each layer
during init.

**Description**

I couldn't build the fp8 benchmark.

`
[ 25%] Building CXX object
CMakeFiles/cublaslt_utils.dir/cublaslt_utils.cc.o
In file included from
<scrubbed>/superbench/benchmarks/micro_benchmarks/cublaslt_fp8_gemm/cublaslt_utils.cc:4:

<scrubbed>/superbench/benchmarks/micro_benchmarks/cublaslt_fp8_gemm/cublaslt_utils.h:
In function ‘void checkCublasStatus(cublasStatus_t)’:

<scrubbed>/superbench/benchmarks/micro_benchmarks/cublaslt_fp8_gemm/cublaslt_utils.h:15:20:
error: ‘logic_error’ is not a member of ‘std’
   15 |         throw std::logic_error("cuBLAS API failed");
      |                    ^~~~~~~~~~~

<scrubbed>/superbench/benchmarks/micro_benchmarks/cublaslt_fp8_gemm/cublaslt_utils.cc:
In member function ‘size_t cublasLtGemm::GetAlgorithm(int, size_t)’:

<scrubbed>/superbench/benchmarks/micro_benchmarks/cublaslt_fp8_gemm/cublaslt_utils.cc:103:20:
error: ‘runtime_error’ is not a member of ‘std’
103 | throw std::runtime_error("Unable to find any suitable
algorithms");
      |                    ^~~~~~~~~~~~~
make[2]: *** [CMakeFiles/cublaslt_utils.dir/build.make:76:
CMakeFiles/cublaslt_utils.dir/cublaslt_utils.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:85:
CMakeFiles/cublaslt_utils.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
`

Adding stdexcept fixed this.

Support traffic patterns under the different devices in NCCL/RCCL test
* change the metrics format if specified the pattern

**Major Revision**

- Add an option for pattern to generate mpi_pattern.txt file if
specified the path.
- In mpi pattern, serial_index and parallel_index will add in each
benchmark as environment variables.

**Minor Revision**
- Fix typo

Support FP8 in PyTorch BERT models:

* add fp8 hybrid/e4m3/e5m2 in precision arguments
* build BERT encoders with `te.TransformerLayer` to repalce
`transformers.BertModel`
* wrap forward steps with fp8 autocast

**Description**
Support the following patterns  in `mpi` mode:
* `k-batch`
* `topo-aware`

Support GEMM benchmark on Hopper GPUs.

Integrate cublaslt-gemm micro-benchmark #451.

**Description**
 Add correctness check in cublas-function benchmark.

**Major Revision**
- add python code of correctness check in cublas-function benchmark and test

Add micro-benchmark for cublaslt fp8 gemm.

**Description**
Add c source code of correctness check for cublas functions.

**Major Revision**
- add correctness check for all supported cublas functions
- add --correctness option into binary

**Minor Revision**
- fix bug and template fill_data and prepare_tensor to get right memory-alignment output matrix for different datatype

**Description**
Add stdout logging util module and enable real-time logging flushing in executor

**Major Revision**
- Add stdout logging util module to redirect stdout into file log
- enable stdout logging in executor to write benchmark output into both stdout and file `sb-bench.log`
- enable real-time log flushing in run_command of microbenchmarks through config `log_flushing`

**Minor Revision**
- add log_n_step args to enable regular step time log in model benchmarks 
- udpate related docs

**Description**
* Reuse `gen_pair_wise_config` in micro-benchmark

Add Docker image for arch90 NVIDIA GPUs:

* add CUDA11.8 Dockerfile
* update archs in Makefile and benchmarks accordingly
* update image build pipeline

* Extract pair-wise pattern from ib_validation

Bumps [qs](https://github.com/ljharb/qs) and [express](https://github.com/expressjs/express). These dependencies needed to be updated together.

Updates `qs` from 6.7.0 to 6.11.0
- [Release notes](https://github.com/ljharb/qs/releases)
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ljharb/qs/compare/v6.7.0...v6.11.0)

Updates `express` from 4.17.1 to 4.18.2
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/4.17.1...4.18.2

)

---
updated-dependencies:
- dependency-name: qs
  dependency-type: indirect
- dependency-name: express
  dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [decode-uri-component](https://github.com/SamVerschueren/decode-uri-component) from 0.2.0 to 0.2.2.
- [Release notes](https://github.com/SamVerschueren/decode-uri-component/releases)
- [Commits](https://github.com/SamVerschueren/decode-uri-component/compare/v0.2.0...v0.2.2

)

---
updated-dependencies:
- dependency-name: decode-uri-component
  dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

**Description**
Add wait time option to resolve mem-bw unstable issue.

**Description**
Downgrage transformers version to fix tersorrt test failure.

* add mpi-parallels mode

* update according to comments

* fix and update doc

* update

* merge into 'mpi' mode

* udpate according to comments

* fix testcases

* fix ansible

* regard pattern as field

* udpate

* fix flake8 version

* add flake8 range

* remove map-by from host config

* udpate comments

Bumps [loader-utils](https://github.com/webpack/loader-utils) from 1.4.0 to 1.4.2.
- [Release notes](https://github.com/webpack/loader-utils/releases)
- [Changelog](https://github.com/webpack/loader-utils/blob/v1.4.2/CHANGELOG.md)
- [Commits](https://github.com/webpack/loader-utils/compare/v1.4.0...v1.4.2

)

---
updated-dependencies:
- dependency-name: loader-utils
  dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Add non-zero return code for `sb deploy` and `sb run` command when
there're Ansible failures in control plane.
Return code is set to count of failure.

For failures caused by benchmarks, return code is still set per benchmark
in results json file.

Update version to include revision hash and date in "{last tag}+g{git
hash}.d{date}" format, here're the examples:
* exact tag: 0.6.0
* commit after tag: 0.6.0+gcbb1b34
* commit after tag with local changes: 0.6.0+gcbb1b34.d20221028

Fix codecov status issue due to service upgrade.

Removes [ansi-html](https://github.com/Tjatse/ansi-html). It's no longer used after updating ancestor dependency [webpack-dev-server](https://github.com/webpack/webpack-dev-server). These dependencies need to be updated together.


Removes `ansi-html`

Updates `webpack-dev-server` from 3.11.2 to 3.11.3
- [Release notes](https://github.com/webpack/webpack-dev-server/releases)
- [Changelog](https://github.com/webpack/webpack-dev-server/blob/v3.11.3/CHANGELOG.md)
- [Commits](https://github.com/webpack/webpack-dev-server/compare/v3.11.2...v3.11.3

)

---
updated-dependencies:
- dependency-name: ansi-html
  dependency-type: indirect
- dependency-name: webpack-dev-server
  dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Update codecov flags in configuration.

Benchmarks - Add support to allow list of custom config string in cudnn-functions and cublas-functions (#414)

**Description**
Add support to allow list of custom config string in cudnn-functions and cublas-functions.

**Description**

Cherry-pick bug fixes from v0.6.0 to main.

**Major Revisions**

* Enable latency test in ib traffic validation distributed benchmark (#396)
* Enhance parameter parsing to allow spaces in value (#397)
* Update apt packages in dockerfile (#398)
* Upgrade colorlog for NO_COLOR support (#404)
* Analyzer - Update error handling to support exit code of sb result diagnosis (#403)
* Analyzer - Make baseline file optional in data diagnosis and fix bugs (#399)
* Enhance timeout cleanup to avoid possible hanging (#405)
* Auto generate ibstat file by pssh (#402)
* Analyzer - Format int type and unify empty value to N/A in diagnosis output file (#406)
* Docs - Upgrade version and release note (#407)
* Docs - Fix issues in document (#408)
Co-authored-by: Yang Wang <yangwang1@microsoft.com>
Co-authored-by: Yuting Jiang <yutingjiang@microsoft.com>

**Description**
Add support to store values of metrics in data diagnosis.

Take the following rules as example: 
```
    nccl_store_rule:
      categories: NCCL_DIS
      store: True
      metrics:
        - nccl-bw:allreduce-run0/allreduce_1073741824_busbw
        - nccl-bw:allreduce-run1/allreduce_1073741824_busbw
        - nccl-bw:allreduce-run2/allreduce_1073741824_busbw
        - nccl-bw:allreduce-run3/allreduce_1073741824_busbw
        - nccl-bw:allreduce-run4/allreduce_1073741824_busbw
    nccl_rule:
      function: multi_rules
      criteria: 'lambda label:True if min(label["nccl_store_rule"].values())/max(label["nccl_store_rule"].values())<0.95 else False'
      categories: NCCL_DIS
```
**nccl_store_rule** will store the values of the metrics in dict and save them into `label["nccl_store_rule"]` , and then **rccl_rule** can use the values of metrics through `label["nccl_store_rule"].values()` in criteria

**Description**
Add support for both jsonl and json format in data diagnosis.

**Major Revision**
- Add support for both jsonl and json format in data diagnosis


**Minor Revision**
- change related doc
- add jsonl support in cli

__Description__

Update Python setup for require packages.

__Major Revisions__
* downgrade requests version to be compatible with python 3.6, add corresponding pipeline for 3.6
* add extra entry in extras_require for nested packages
* update `pip install` contents accordingly

**Description**
Degrade perftest submodule to v4.4-0.37 to fix stability issue.

Issue: rdma-loopback is not stable on public version(v0.5/v0.6-rc1)
Docker Version: v0.6-rc1-cuda11.1
Testbed: 8 A100 40GB GPUs (1 NDv4 node)
Result: 
New perftest version introduce the variance, max-min/mean = 2% for v4.4-0.37, 8% for v4.5-0.2

An enhancement for topo-aware IB performance validation #373.
This PR will auto-generate a required ibstate file `ib_traffic_topo_aware_ibstat.txt` which is used as input to build a graph.

**Description**
Rename field in data diagnosis to be more readable.

**Major Revision**
- rename fields according to diagnosis/metric format

**Minor Revision**
- change type of diagnosis/issue_num to be int

Fix minimum timeout: use 60s if config is shorter.