GitLab

Update softmax.h

Fixing NULL stream launch in dispatch_additive_masked_softmax

add streaming support for softmax kernels

Sparsity

Co-authored-by: Michael Carilli <mcarilli@nvidia.com>

Remove distributed lamb from __init__.py

Distributed lamb optimizer

Fuses dropout and softmax in backward pass, add bias support to CPP MHA, add additive mask support, separate Q/K/V parameters (#854)
Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>