GitLab

IFU-2021-10-15 (+ remove redundant defines + C10_CUDA_CHECK)

Changes include
- THC headers removal
- TH macros replacement
- fix some typo in comment
 Conflicts:
	apex/contrib/csrc/multihead_attn/additive_masked_softmax_dropout_cuda.cu
	apex/contrib/csrc/multihead_attn/encdec_multihead_attn_cuda.cu
	apex/contrib/csrc/multihead_attn/encdec_multihead_attn_norm_add_cuda.cu
	apex/contrib/csrc/multihead_attn/masked_softmax_dropout_cuda.cu
	apex/contrib/csrc/multihead_attn/self_multihead_attn_bias_additive_mask_cuda.cu
	apex/contrib/csrc/multihead_attn/self_multihead_attn_bias_cuda.cu
	apex/contrib/csrc/multihead_attn/self_multihead_attn_cuda.cu
	apex/contrib/csrc/multihead_attn/self_multihead_attn_norm_add_cuda.cu
	apex/contrib/csrc/multihead_attn/strided_batched_gemm.h

* Use --cuda_ext flag to build all supported extensions

* Don't remove --cuda_ext since it'll be needed to build other extensions

* Need to clear all cmdline args so setup.py doesn't complain

Add more unit tests for both distributed and extensions

Change python3.6 to python

Enable multihead atten

Co-authored-by: Jeff Daily <jeff.daily@amd.com>

Fix rocblas_gemmex namespace

Fix namespace

Clean up comments

Enable HIP floa to hald conversion

Fix some spacing

This reverts commit 964e61f1.

1) multihead_attn
2) xentropy
3) fused_adam and distributed_fused_adam

change chunking scheme for full-allreduce case, add parameter order argument, both to enable contiguous chunking of allgather (#1190)

1. remove the weight broadcast in the constructor
2. disable unnecessary allreduces for clip-after-ar