GitLab

**Description**
Fix bug of duration feature for model benchmarks in distributed mode.

**Major Revision**
- Add all_reduce to sync the result of is_finished(the function to judge whether the model benchmark should be stopped) in each step 
  - to avoid inconsistency between different ranks to determine duration end (some rank may enter one more step and can never finish)
- Add torch.cuda.synchronize() before and after step time measuring in train_step() for all model benchmarks
  - some operations in train_step() maybe async resulting incorrect step time records (for example, lstm) 

**Description**
Fix bugs in sync results on root rank for e2e model benchmarks.

Bugs:
 - results were not changed to sync results (grammer)
 - sync results not applyed to all ranks but only root rank
 - output result on local_rank 0 not global root rank

**Description**
Unify metric names of benchmarks.

**Description**
Add option `force_fp32` to use fp32 instead of tf32, only takes effect on Ampere or newer GPUs.

* replace torch.optim.AdamW with transformers.AdamW.

* Clean up the cache.

Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>
Co-authored-by: Yifan Xiong <yifan.xiong@microsoft.com>

* skip unnecessary tests according to env var
* remove useless tests

Benchmarks: Code Revision - Revise BenchmarkRegistry interfaces for integration with executor. (#33)

* revise BenchmarkRegistry interfaces.
* address comments
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* move benchmarks registration from registry.py to __init__.py

* revise __init__.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>

* add pytorch base tests.

* add more test cases.
Co-authored-by: Guoshuai Zhao <guzhao@microsoft.com>