GitLab

* Rename dev branch to main and set it as default.

* Superbenchmark: Revise tests - revise benchmark test util to support multi gpu test

* modify test_sharding_matmul.py to match the tests util

* update README for public view

* add flags for different pipelines to merge reports
* add token for cuda pipeline
* update codecov/patch target

* fix missing package in dockerfile
* update benchmark list and parameters
* catch runtime errors
* refine logging info

* add tests for matmul and sharding-matmul benchmarks.

* add decorator for sharding_matmul tests.

* add __init__.py for utils of benchmarks tests.

* disable GPU tests for CPU platform validation.

* fix typo
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>
Co-authored-by: Peng Cheng <chengpeng5555@outlook.com>
Co-authored-by: Yifan Xiong <yifan.xiong@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>
Co-authored-by: Yifan Xiong <yifan.xiong@microsoft.com>

* add azure pipelines for CUDA tests

* CLI integration with Executor and Runner

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add cuda11.1.1 dockerfile

* init runner class with not implemented

* skip unnecessary tests according to env var
* remove useless tests

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Add superbench executor class

* add executor class
* update default config to exec benchmarks
* add micro benchmarks and model benchmarks

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* fix raw data validation bug.
* address comments.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Benchmarks: Add Benchmark - Add op-sharding microbenchmark, including matmul and sharding_matmul. (#36)

* add microbenchmark - sharding matmul.
* address comments.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Benchmarks: Code Revision - Revise BenchmarkRegistry interfaces for integration with executor. (#33)

* revise BenchmarkRegistry interfaces.
* address comments
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* revise result process interface

* add more comments
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Update logger class.
* add file handler along with stream handler
* add colored formatter

* add bert-large as the model benchmark example

* add more arguments.

* address comments.

* delete duplicated file.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Use omegaconf to replace hydra for configuration system:
* remove hydra
* use omegaconf to merge configurations

Benchmarks: Add Benchmark - Add Pytorch BERT benchmarks, including bert-base and bert-large.   (#20)

* add pytorch bert benchmarks.

* revise code

* fix issue

* revise code.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* move benchmarks registration from registry.py to __init__.py

* revise __init__.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Benchmarks: Add Feature - Add benchmark finish check according to num_warmup/num_steps and duration in ModelBenchmark class. (#25)

* add is_finished function

* reuse current time.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Benchmarks: Add Feature - Register all benchmarks by import model_benchmarks/micro_benchmarks/docker_benchmarks. (#21)

* clean code.

* minor change.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add sample_count argument.

* handle more condidatins.

* address comments.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* support benchmark re-registration.

* address comments
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add pytorch base tests.

* add more test cases.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

- Add CLI commands
  * sb version
  * sb deploy
  * sb exec
  * sb run
- Add interface with executor and runner
- Add cli test cases

* add random dataset.

* install pytorch-cpu for test docker.

* fix typo

* add more test cases.

* address comments.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add flag to disable GPU.

* fix spelling

* fix unittest.

* address comments.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>