GitLab

**Description**
Fix bugs in sync results on root rank for e2e model benchmarks.

Bugs:
 - results were not changed to sync results (grammer)
 - sync results not applyed to all ranks but only root rank
 - output result on local_rank 0 not global root rank

**Description**
Use config `log_raw_data` to control whether log the raw data into file or not. The default value is `no`. We can set it as `yes` for some particular benchmarks to save the raw data into file, such as NCCL/RCCL test.

**Description**
Please write a brief description and link the related issue if have.

**Major Revision**
- Sync (do allreduce max) the E2E training results among all workers.
- Avoid using ':0' in metric name if there has only one rank having output.

**Description**
Add 50th, 90th, 95th, 99th, 99.9th latency metrics for ORT and pytorch inference benchmarks.

**Description**
Unify metric names of benchmarks.

**Description**
Add option `force_fp32` to use fp32 instead of tf32, only takes effect on Ampere or newer GPUs.

**Description**
Set reduce type for current benchmarks' metrics, including model benchmarks and ShardingMatmul.

**Description**
Replace `sum(results) / len(results)` with `statistics.mean(results)`

* add config file for ndv4.

Benchmarks: Add Feature - Add DistributedImpl and DistributedBackend arguments for micro benchmark. (#100)

* fix return code reset issue

* revise result process interface

* add more comments
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Benchmarks: Add Feature - Add benchmark finish check according to num_warmup/num_steps and duration in ModelBenchmark class. (#25)

* add is_finished function

* reuse current time.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add sample_count argument.

* handle more condidatins.

* address comments.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add flag to disable GPU.

* fix spelling

* fix unittest.

* address comments.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add optimizer definition and function to create torch optimizer.

* move optimizer enum into model_base module.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* benchmarks init.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>