large update including model parallelism and gpt2

Co-authored-by: shoeybi <shoeybim@gmail.com> Co-authored-by: raulpuric <raulpuric@berkeley.edu> Co-authored-by: jaredcasper <jaredcasper@gmail.com> Co-authored-by: mpatwary <mostofa.patwary@gmail.com> Co-authored-by: plegresl <plegresl@gmail.com>

large update including model parallelism and gpt2
Co-authored-by: shoeybi <shoeybim@gmail.com> Co-authored-by: raulpuric <raulpuric@berkeley.edu> Co-authored-by: jaredcasper <jaredcasper@gmail.com> Co-authored-by: mpatwary <mostofa.patwary@gmail.com> Co-authored-by: plegresl <plegresl@gmail.com>
abe36e2e · Raul Puri · 0399d32c · abe36e2e · abe36e2e · abe36e2e
Commit abe36e2e authored Jul 29, 2019 by Raul Puri
10 changed files
--- a/scripts/pretrain_bert_distributed.sh
+++ b/scripts/pretrain_bert_distributed.sh
--- a/scripts/pretrain_bert_model_parallel.sh
+++ b/scripts/pretrain_bert_model_parallel.sh
--- a/scripts/pretrain_bert_sentencepiece.sh
+++ b/scripts/pretrain_bert_sentencepiece.sh
--- a/scripts/pretrain_bert_tfrecords_distributed.sh
+++ b/scripts/pretrain_bert_tfrecords_distributed.sh
--- a/scripts/pretrain_gpt2.sh
+++ b/scripts/pretrain_gpt2.sh
--- a/scripts/pretrain_gpt2_distributed.sh
+++ b/scripts/pretrain_gpt2_distributed.sh
--- a/scripts/pretrain_gpt2_model_parallel.sh
+++ b/scripts/pretrain_gpt2_model_parallel.sh
--- a/scripts/run_gpt2_eval.py
+++ b/scripts/run_gpt2_eval.py
--- a/scripts/split_json.py
+++ b/scripts/split_json.py
--- a/utils.py
+++ b/utils.py