Skip to content

GitLab

Commit 3dadd16d authored by Deepak Narayanan's avatar Deepak Narayanan
Browse files

Update T5 scripts

parent 83c4d95a
Hide whitespace changes
Inline Side-by-side
Showing with 9 additions and 6 deletions
examples/pretrain_t5.sh
View file @ 3dadd16d
...@@ -15,7 +15,7 @@ python pretrain_t5.py \ ...@@ -15,7 +15,7 @@ python pretrain_t5.py \
--encoder-seq-length 512 \ --encoder-seq-length 512 \
--decoder-seq-length 128 \ --decoder-seq-length 128 \
--micro-batch-size 16 \ --micro-batch-size 16 \
--global-batch-size 2048 \ --global-batch-size 16 \
--max-position-embeddings 512 \ --max-position-embeddings 512 \
--train-iters 1000000 \ --train-iters 1000000 \
--lr-decay-iters 1000000 \ --lr-decay-iters 1000000 \
...@@ -35,4 +35,5 @@ python pretrain_t5.py \ ...@@ -35,4 +35,5 @@ python pretrain_t5.py \
--save-interval 10000 \ --save-interval 10000 \
--eval-interval 1000 \ --eval-interval 1000 \
--eval-iters 10 \ --eval-iters 10 \
--fp16 --fp16 \
--vocab-extra-ids 100
examples/pretrain_t5_distributed.sh
View file @ 3dadd16d
...@@ -24,7 +24,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \ ...@@ -24,7 +24,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \
--encoder-seq-length 512 \ --encoder-seq-length 512 \
--decoder-seq-length 128 \ --decoder-seq-length 128 \
--micro-batch-size 16 \ --micro-batch-size 16 \
--global-batch-size 2048 \ --global-batch-size 128 \
--max-position-embeddings 512 \ --max-position-embeddings 512 \
--train-iters 1000000 \ --train-iters 1000000 \
--lr-decay-iters 1000000 \ --lr-decay-iters 1000000 \
...@@ -44,4 +44,5 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \ ...@@ -44,4 +44,5 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \
--save-interval 10000 \ --save-interval 10000 \
--eval-interval 1000 \ --eval-interval 1000 \
--eval-iters 10 \ --eval-iters 10 \
--fp16 --fp16 \
--vocab-extra-ids 100
examples/pretrain_t5_distributed_with_mp.sh
View file @ 3dadd16d
...@@ -24,7 +24,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \ ...@@ -24,7 +24,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \
--encoder-seq-length 512 \ --encoder-seq-length 512 \
--decoder-seq-length 128 \ --decoder-seq-length 128 \
--micro-batch-size 16 \ --micro-batch-size 16 \
--global-batch-size 2048 \ --global-batch-size 128 \
--seq-length 512 \ --seq-length 512 \
--max-position-embeddings 512 \ --max-position-embeddings 512 \
--train-iters 1000000 \ --train-iters 1000000 \
...@@ -45,4 +45,5 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \ ...@@ -45,4 +45,5 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \
--save-interval 10000 \ --save-interval 10000 \
--eval-interval 1000 \ --eval-interval 1000 \
--eval-iters 10 \ --eval-iters 10 \
--fp16 --fp16 \
--vocab-extra-ids 100
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment