Skip to content

GitLab

"scripts/convert_vq_diffusion_to_diffusers.py" did not exist on "269109dbfbbdbe2800535239b881e96e1828a0ef"
Commit 4577151f authored by Tri Dao's avatar Tri Dao
Browse files

Link to Triton implementation

parent bc2c2102
Hide whitespace changes
Inline Side-by-side
Showing with 10 additions and 0 deletions
README.md
View file @ 4577151f
...@@ -8,6 +8,16 @@ Paper: https://arxiv.org/abs/2205.14135 ...@@ -8,6 +8,16 @@ Paper: https://arxiv.org/abs/2205.14135
IEEE Spectrum [article](https://spectrum.ieee.org/mlperf-rankings-2022) about our submission to the MLPerf 2.0 benchmark using FlashAttention. IEEE Spectrum [article](https://spectrum.ieee.org/mlperf-rankings-2022) about our submission to the MLPerf 2.0 benchmark using FlashAttention.
![FlashAttention](assets/flashattn_banner.jpg) ![FlashAttention](assets/flashattn_banner.jpg)
#### Triton implementation of FlashAttention
Phil Tillet (OpenAI) has an implementation of FlashAttention in Triton:
https://github.com/openai/triton/blob/master/python/tutorials/06-fused-attention.py
As Triton is a higher-level language than CUDA, it might be easier to understand
and experiment with. The notations in the Triton implementation are also closer
to what's used in our paper.
## Alpha release (0.1). ## Alpha release (0.1).
To compile (requiring CUDA 11, NVCC, and an Turing or Ampere GPU): To compile (requiring CUDA 11, NVCC, and an Turing or Ampere GPU):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment