Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
README.md 344 Bytes
Newer Older
Tri Dao's avatar
Add GPT and ViT models  
Tri Dao committed
1 
This CUDA extension implements fused dropout + residual + LayerNorm, based on
Tri Dao's avatar
Add fused_dense and dropout_add_layernorm CUDA extensions  
Tri Dao committed
2
3 
Apex's [FastLayerNorm](https://github.com/NVIDIA/apex/tree/master/apex/contrib/layer_norm).
We add dropout and residual, and make it work for both pre-norm and post-norm architecture.
Tri Dao's avatar
Mention that some CUDA extensions have only been tested on A100s  
Tri Dao committed
4
5
6 

It has only been tested on A100s.
Tri Dao's avatar
Add fused_dense and dropout_add_layernorm CUDA extensions  
Tri Dao committed
7
8
9 
```sh
cd csrc/layer_norm && pip install .
```