Release - SuperBench v0.7.0 (#468)

**Description**

Cherry-pick bug fixes from v0.7.0 to main.

**Major Revisions**

* Benchmarks - Fix missing include in FP8 benchmark (#460)
* Fix bug in TE BERT model (#461)
* Doc - Update benchmark doc (#465)
* Bug: Fix bug for incorrect datatype judgement in cublas-function
source code (#464)
* Support `sb deploy` without pulling image (#466)
* Docs - Upgrade version and release note (#467)
Co-authored-by: Russell J. Hewett <russell.j.hewett@gmail.com>
Co-authored-by: Yuting Jiang <yutingjiang@microsoft.com>

superbench/config/azure_ndmv4.yaml

@@ -3,7 +3,7 @@
 # Azure NDm A100 v4
 # reference: https://docs.microsoft.com/en-us/azure/virtual-machines/ndm-a100-v4-series
 
-version: v0.6
+version: v0.7
 superbench:
   enable: null
   monitor:

superbench/config/azure_ndv4.yaml

 # SuperBench Config
-version: v0.6
+version: v0.7
 superbench:
   enable: null
   monitor:

superbench/config/default.yaml

 # SuperBench Config
-version: v0.6
+version: v0.7
 superbench:
   enable: null
   monitor:

superbench/runner/playbooks/deploy.yaml

@@ -92,6 +92,7 @@
       shell: |
         docker pull {{ docker_image }}
       become: yes
+      when: docker_pull | default(true)
       throttle: 32
     - name: Starting Container
       shell: |

superbench/runner/runner.py

@@ -183,6 +183,7 @@ def deploy(self):    # pragma: no cover
             'ssh_port': random.randint(1 << 14, (1 << 15) - 1),
             'output_dir': str(self._output_path),
             'docker_image': self._docker_config.image,
+            'docker_pull': bool(self._docker_config.pull),
         }
         if bool(self._docker_config.username) and bool(self._docker_config.password):
             extravars.update(

tests/cli/test_sb.py

@@ -60,6 +60,12 @@ def test_sb_deploy(self, mocked_failure_count):
         mocked_failure_count.return_value = 0
         self.cmd('sb deploy --host-list localhost', checks=[NoneCheck()])
 
+    @mock.patch('superbench.runner.SuperBenchRunner.get_failure_count')
+    def test_sb_deploy_skippull(self, mocked_failure_count):
+        """Test sb deploy without docker pull."""
+        mocked_failure_count.return_value = 0
+        self.cmd('sb deploy --host-list localhost --no-image-pull', checks=[NoneCheck()])
+
     def test_sb_deploy_no_host(self):
         """Test sb deploy, no host_file or host_list provided, should fail."""
         self.cmd('sb deploy', expect_failure=True)

website/blog/2023-01-20-release-0-7.md

+---
+slug: release-sb-v0.7
+title: Releasing SuperBench v0.7
+author: Peng Cheng
+author_title: SuperBench Team
+author_url: https://github.com/cp5555
+author_image_url: https://github.com/cp5555.png
+tags: [superbench, announcement, release]
+---
+
+We are very happy to announce that **SuperBench 0.7.0 version** is officially released today!
+
+You can install and try superbench by following [Getting Started Tutorial](https://microsoft.github.io/superbenchmark/docs/getting-started/installation).
+
+## SuperBench 0.7.0 Release Notes
+
+### SuperBench Improvement
+
+- Support non-zero return code when "sb deploy" or "sb run" fails in Ansible.
+- Support log flushing to the result file during runtime.
+- Update version to include revision hash and date.
+- Support "pattern" in mpi mode to run tasks in parallel.
+- Support topo-aware, all-pair, and K-batch pattern in mpi mode.
+- Fix Transformers version to avoid Tensorrt failure.
+- Add CUDA11.8 Docker image for NVIDIA arch90 GPUs.
+- Support "sb deploy" without pulling image.
+
+### Micro-benchmark Improvements
+
+- Support list of custom config string in cudnn-functions and cublas-functions.
+- Support correctness check in cublas-functions.
+- Support GEMM-FLOPS for NVIDIA arch90 GPUs.
+- Support cuBLASLt FP16 and FP8 GEMM.
+- Add wait time option to resolve mem-bw unstable issue.
+- Fix bug for incorrect datatype judgement in cublas-function source code.
+
+### Model Benchmark Improvements
+
+- Support FP8 in BERT model training.
+
+### Distributed Benchmark Improvements
+
+- Support pair-wise pattern in IB validation benchmark.
+- Support topo-aware, pair-wise, and K-batch pattern in nccl-bw benchmark.

website/docusaurus.config.js

@@ -101,7 +101,7 @@ module.exports = {
     announcementBar: {
       id: 'supportus',
       content:
-        '📢 <a href="https://microsoft.github.io/superbenchmark/blog/release-sb-v0.6">v0.6.0</a> has been released! ' +
+        '📢 <a href="https://microsoft.github.io/superbenchmark/blog/release-sb-v0.7">v0.7.0</a> has been released! ' +
         '⭐️ If you like SuperBench, give it a star on <a target="_blank" rel="noopener noreferrer" href="https://github.com/microsoft/superbenchmark">GitHub</a>! ⭐️',
     },
     algolia: {

website/package-lock.json

 {
   "name": "superbench-website",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {

website/package.json

 {
   "name": "superbench-website",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "private": true,
   "scripts": {
     "docusaurus": "docusaurus",