GitLab

Previously we were using block_size=256.

They don't have to have the same block size, number of threads, etc.

fix typo in function mha_fwd

as title.

build wheel workflow

remove numpy dependency

According to the `setup.py` file, only dependencies are torch and einops. But the `bert_padding.py` file requires `numpy` only to multiply the elements of a `torch.Size` object. This change aims at allowing the use of FlashAttention without numpy.

Make flash attention compile on Windows.

Update flash_attention.py

Recasting query and key after rotary_emb()

Phil Tillet suggests calling it "experimental".

Theoretically this might have lower numerical error since the scaling is in
fp32 instead of fp16 (not sure, I haven't thought too carefully about it).
However, in practice, the numerical errors seem about the same.