GitLab

* Use --cuda_ext flag to build all supported extensions

* Don't remove --cuda_ext since it'll be needed to build other extensions

* Need to clear all cmdline args so setup.py doesn't complain

Add more unit tests for both distributed and extensions

Change python3.6 to python

Enable multihead atten

Co-authored-by: Jeff Daily <jeff.daily@amd.com>

Fix rocblas_gemmex namespace

Fix namespace

Clean up comments

Enable HIP floa to hald conversion

Fix some spacing

This reverts commit 964e61f1.

1) multihead_attn
2) xentropy
3) fused_adam and distributed_fused_adam

change chunking scheme for full-allreduce case, add parameter order argument, both to enable contiguous chunking of allgather (#1190)

1. remove the weight broadcast in the constructor
2. disable unnecessary allreduces for clip-after-ar

* run backward

* remove custom_fwd/custom_bwd