Skip to content

GitLab

Commit 1f45adcc authored by Mohammad's avatar Mohammad
Browse files

latest

parent ca19a8b5
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
examples/pretrain_bert_distributed.sh
View file @ 1f45adcc
...@@ -32,6 +32,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \ ...@@ -32,6 +32,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \
--distributed-backend nccl \ --distributed-backend nccl \
--lr 0.0001 \ --lr 0.0001 \
--lr-decay-style linear \ --lr-decay-style linear \
--min-lr 1.0e-5 \
--lr-decay-iters 990000 \ --lr-decay-iters 990000 \
--weight-decay 1e-2 \ --weight-decay 1e-2 \
--clip-grad 1.0 \ --clip-grad 1.0 \
......
examples/pretrain_gpt2_distributed.sh
View file @ 1f45adcc
...@@ -36,6 +36,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \ ...@@ -36,6 +36,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \
--distributed-backend nccl \ --distributed-backend nccl \
--lr 0.00015 \ --lr 0.00015 \
--lr-decay-style cosine \ --lr-decay-style cosine \
--min-lr 1.0e-5 \
--weight-decay 1e-2 \ --weight-decay 1e-2 \
--clip-grad 1.0 \ --clip-grad 1.0 \
--warmup .01 \ --warmup .01 \
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment