Skip to content

GitLab

Commit 43798966 authored by Tri Dao's avatar Tri Dao
Browse files

[Docs] Fix formatting

parent 3c7cbfc1
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
training/README.md
View file @ 43798966
...@@ -156,13 +156,13 @@ python run.py experiment=pile/gpt3-2.7B-flash-hdim128 trainer.devices=8 # 2.7B ...@@ -156,13 +156,13 @@ python run.py experiment=pile/gpt3-2.7B-flash-hdim128 trainer.devices=8 # 2.7B
``` ```
The default parameters are set for 8 x A100 80GB. We train with bf16 by default. The default parameters are set for 8 x A100 80GB. We train with bf16 by default.
To train with rotary embedding, run the experiments `pile/gpt3{s,m,l,xl**-flash-rotary**. To train with rotary embedding, run the experiments `pile/gpt3{s,m,l,xl}-flash-rotary`.
### Training options ### Training options
**Gradient accumulation**: to adjust device batch size to fit into GPU memory **Gradient accumulation**: to adjust device batch size to fit into GPU memory
(the global batch size stays the same, and gradient accumulation is calculated (the global batch size stays the same, and gradient accumulation is calculated
automatically), set `datamodule.batch_size=blah**. automatically), set `datamodule.batch_size=blah`.
**Multi-node**: to train on multiple nodes, add `trainer.num_nodes=blah`. **Multi-node**: to train on multiple nodes, add `trainer.num_nodes=blah`.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment