update

e1354f9d · liangjing · e1354f9d · e1354f9d · e1354f9d · e1354f9d
Commit e1354f9d authored May 22, 2024 by liangjing
20 changed files
--- a/examples_deepspeed/bert_with_pile/README.md
+++ b/examples_deepspeed/bert_with_pile/README.md
--- a/examples_deepspeed/bert_with_pile/ds_config_bert_TEMPLATE.json
+++ b/examples_deepspeed/bert_with_pile/ds_config_bert_TEMPLATE.json
--- a/examples_deepspeed/bert_with_pile/ds_finetune_bert_mnli.sh
+++ b/examples_deepspeed/bert_with_pile/ds_finetune_bert_mnli.sh
--- a/examples_deepspeed/bert_with_pile/ds_finetune_bert_qqp.sh
+++ b/examples_deepspeed/bert_with_pile/ds_finetune_bert_qqp.sh
--- a/examples_deepspeed/bert_with_pile/ds_finetune_bert_race.sh
+++ b/examples_deepspeed/bert_with_pile/ds_finetune_bert_race.sh
--- a/examples_deepspeed/bert_with_pile/ds_pretrain_bert.sh
+++ b/examples_deepspeed/bert_with_pile/ds_pretrain_bert.sh
--- a/examples_deepspeed/bert_with_pile/prepare_pile_data.py
+++ b/examples_deepspeed/bert_with_pile/prepare_pile_data.py
--- a/examples_deepspeed/compression/125M-Int8-test-64gpu-distilled-group48.sh
+++ b/examples_deepspeed/compression/125M-Int8-test-64gpu-distilled-group48.sh
--- a/examples_deepspeed/compression/125M-L10-Int8-test-64gpu-distilled-group48.sh
+++ b/examples_deepspeed/compression/125M-L10-Int8-test-64gpu-distilled-group48.sh
--- a/examples_deepspeed/compression/125M-L12-Int8-test-64gpu-distilled-group48.sh
+++ b/examples_deepspeed/compression/125M-L12-Int8-test-64gpu-distilled-group48.sh
--- a/examples_deepspeed/compression/ds_config_gpt_TEMPLATE.json
+++ b/examples_deepspeed/compression/ds_config_gpt_TEMPLATE.json
--- a/examples_deepspeed/compression/ds_config_gpt_TEMPLATE_compression.json
+++ b/examples_deepspeed/compression/ds_config_gpt_TEMPLATE_compression.json
--- a/examples_deepspeed/compression/ds_evalharness.sh
+++ b/examples_deepspeed/compression/ds_evalharness.sh
--- a/examples_deepspeed/compression/ds_pretrain_gpt_1.3B_dense_cl_kd.sh
+++ b/examples_deepspeed/compression/ds_pretrain_gpt_1.3B_dense_cl_kd.sh
--- a/examples_deepspeed/compression/ds_pretrain_gpt_125M_dense_cl_kd.sh
+++ b/examples_deepspeed/compression/ds_pretrain_gpt_125M_dense_cl_kd.sh
--- a/examples_deepspeed/compression/ds_pretrain_gpt_125M_dense_kd.sh
+++ b/examples_deepspeed/compression/ds_pretrain_gpt_125M_dense_kd.sh
--- a/examples_deepspeed/compression/ds_pretrain_gpt_350M_dense_kd.sh
+++ b/examples_deepspeed/compression/ds_pretrain_gpt_350M_dense_kd.sh
--- a/examples_deepspeed/curriculum_learning/README.md
+++ b/examples_deepspeed/curriculum_learning/README.md
+This is an example of how to use DeepSpeed's curriculum learning (CL) feature which provides faster and more stable language model pre-training. Currently it is only integrated for GPT pre-training. Note that there are two curriculum learning examples in two different repos for Megatron-LM GPT-2 pre-training. Both of them have some unique features and limitations. See details in our [tutorial](https://www.deepspeed.ai/tutorials/curriculum-learning/). For technical details please refer to our [paper](https://arxiv.org/abs/2108.06084).
\ No newline at end of file
--- a/examples_deepspeed/curriculum_learning/ds_config_gpt_slw_TEMPLATE.json
+++ b/examples_deepspeed/curriculum_learning/ds_config_gpt_slw_TEMPLATE.json
--- a/examples_deepspeed/curriculum_learning/ds_pretrain_gpt2.sh
+++ b/examples_deepspeed/curriculum_learning/ds_pretrain_gpt2.sh