GitLab

To avoid import error if one doesn't have rotary_emb installed

This is faster since we only need to do atomic adds on dq, instead of atomic
adds on both dk and dv.

WIP because there seems to be some race conditions for head dimensions other
than 16, 32, 64, 128.

build wheel and upload to release

Previously we were using block_size=256.

They don't have to have the same block size, number of threads, etc.

fix typo in function mha_fwd

as title.