GitLab

fix bf16 layernorm bug

IFU_06_05_2020

* bfloat16 support for apex DDP

* enable mgpu tests for fp16 and bf16

* update Dockerfile

Co-authored-by: Michael Carilli <mcarilli@nvidia.com>

Remove distributed lamb from __init__.py

Distributed lamb optimizer

Fuses dropout and softmax in backward pass, add bias support to CPP MHA, add additive mask support, separate Q/K/V parameters (#854)
Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>

make FusedLAMB async

[Documentation] Update the readme for ROCm.

Enable bfloat16 for optimizers