"vscode:/vscode.git/clone" did not exist on "1679ef13d36e889a2f5dadd34b2e75f794126eae"
Commit 3c7cbfc1 authored by Tri Dao's avatar Tri Dao
Browse files

[Docs] Mention that dropout_layer_norm supports all dims up to 6k

parent 85b8e3d3
...@@ -82,7 +82,7 @@ cd ../csrc/rotary && pip install . ...@@ -82,7 +82,7 @@ cd ../csrc/rotary && pip install .
``` ```
5. Fused dropout + residual + LayerNorm, adapted from Apex's 5. Fused dropout + residual + LayerNorm, adapted from Apex's
[FastLayerNorm](https://github.com/NVIDIA/apex/tree/master/apex/contrib/layer_norm). We add dropout and residual, and make it work for both pre-norm and post-norm architecture. [FastLayerNorm](https://github.com/NVIDIA/apex/tree/master/apex/contrib/layer_norm). We add dropout and residual, and make it work for both pre-norm and post-norm architecture.
This only supports a limited set of dimensions, see `csrc/layer_norm/ln_fwd_cuda_kernel.cu`. This supports dimensions divisible by 8, up to 6144.
```sh ```sh
cd ../csrc/layer_norm && pip install . cd ../csrc/layer_norm && pip install .
``` ```
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment