GitLab

* fix warp size in WARP_SHFL* in layernorm

* enable fused_layer_norm tests on ROCm

* enable deprecated fused adam optimizer

* enable deprecated fused lamb

* enable xentropy extension

* add warpsize 32 for nv and 64 for amd

* update compiler arguments

* update the syncwarp conditions

* update syncwarp condition

* enable deprecated fused adam optimizer

* enable deprecated fused lamb

* reset the compiler arguments

* syntax error

* aligning the compiler arguments

* enable mlp cuda

* add setup changes and tests

* skip the unit tests

* updated conditions for empty array

* removed hip platform conditions

IFU-master 07/27/2020.

Asp sparse param dict update

1. Support to include in sparse_parameter_list an user-supplied custom layer type and its parameter name. This is useful when users have their own implementation of nn.Linear or nn.Conv2D. For example, huggingface repo has a custom implementation of nn.Linear called LinearActivation.
2. Print info of layers in the model that are not pruned.

Merge pull request #917 from a-maci/master

Fixing the case when grads are None

Fixing mask multiplication with grad tensors

Grads can be None type. Adding this fix to skip multiplication with masks if this is the case.

Updating my repo

Fixed weight init for fused weight matrices in fused MHA by adding correct gain factor

Fixed variable name

* Enable sync batchnorm

* enable syncbn properly

* update the unit tests

* update tests

* update conditions for welford_merge_element

* updated conditions based on comments.

IFU-07072020

* [sync BN]

support non-uniform batch size across process group.

TODO: test should be added once cleaned up.

* updating unit tests

* new unit tests for different inputs

* cleaning

* Only attempt to patch Tensor methods if defined

* syntax
Co-authored-by: Michael Carilli <mcarilli@nvidia.com>

add unit tests for FusedLAMB optimizer

Fix launch bounds for cleanup(...) call

fix bf16 layernorm bug

2d masking and sparsity

Minor edit

Importance and usage is 2d masking