ZeRO-2 (#217)

Updates for ZeRO stage 2 + ZeRO stage 1 w. RS Co-authored-by: Tunji Ruwase <olruwase@microsoft.com> Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com> Co-authored-by: Elton Zheng <eltonz@microsoft.com> Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com> Co-authored-by: yuxionghe <yuxhe@microsoft.com> Co-authored-by: Arash Ashari <arashari@microsoft.com>

ZeRO-2 (#217)
Updates for ZeRO stage 2 + ZeRO stage 1 w. RS Co-authored-by: Tunji Ruwase <olruwase@microsoft.com> Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com> Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com> Co-authored-by: Elton Zheng <eltonz@microsoft.com> Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com> Co-authored-by: yuxionghe <yuxhe@microsoft.com> Co-authored-by: Arash Ashari <arashari@microsoft.com>
f2ac7eaf · Jeff Rasley · GitHub · c61e23b4 · f2ac7eaf · f2ac7eaf
Unverified Commit f2ac7eaf authored May 19, 2020 by Jeff Rasley Committed by GitHub May 19, 2020
20 changed files
--- a/docs/index.md
+++ b/docs/index.md
--- a/tests/model/BingBertSquad/BingBertSquad_run_func_test.py
+++ b/tests/model/BingBertSquad/BingBertSquad_run_func_test.py
--- a/tests/model/BingBertSquad/deepspeed_bsz24_fp16_config.json
+++ b/tests/model/BingBertSquad/deepspeed_bsz24_fp16_config.json
--- a/tests/model/BingBertSquad/deepspeed_bsz24_fp16_zero2_config.json
+++ b/tests/model/BingBertSquad/deepspeed_bsz24_fp16_zero2_config.json
--- a/tests/model/BingBertSquad/deepspeed_bsz24_fp32_config.json
+++ b/tests/model/BingBertSquad/deepspeed_bsz24_fp32_config.json
--- a/tests/model/BingBertSquad/run_BingBertSquad_sanity.sh
+++ b/tests/model/BingBertSquad/run_BingBertSquad_sanity.sh
--- a/tests/model/Megatron_GPT2/ds_config_func_bs4.json
+++ b/tests/model/Megatron_GPT2/ds_config_func_bs4.json
--- a/tests/model/Megatron_GPT2/ds_config_func_bs4_zero2.json
+++ b/tests/model/Megatron_GPT2/ds_config_func_bs4_zero2.json
--- a/tests/model/Megatron_GPT2/ds_config_func_bs8_no_zero.json
+++ b/tests/model/Megatron_GPT2/ds_config_func_bs8_no_zero.json
--- a/tests/model/Megatron_GPT2/ds_config_func_bs8.json
+++ b/tests/model/Megatron_GPT2/ds_config_func_bs8.json
--- a/tests/model/Megatron_GPT2/ds_config_func_bs8_zero2.json
+++ b/tests/model/Megatron_GPT2/ds_config_func_bs8_zero2.json
--- a/tests/model/Megatron_GPT2/ds_config_func_scheduler.json
+++ b/tests/model/Megatron_GPT2/ds_config_func_scheduler.json
--- a/tests/model/Megatron_GPT2/ds_config_perf_bs16.json
+++ b/tests/model/Megatron_GPT2/ds_config_perf_bs16.json
--- a/tests/model/Megatron_GPT2/ds_config_perf_bs32.json
+++ b/tests/model/Megatron_GPT2/ds_config_perf_bs32.json
--- a/tests/model/Megatron_GPT2/ds_config_perf_bs8.json
+++ b/tests/model/Megatron_GPT2/ds_config_perf_bs8.json
--- a/tests/model/Megatron_GPT2/ds_gpt2_test.sh
+++ b/tests/model/Megatron_GPT2/ds_gpt2_test.sh
--- a/tests/model/Megatron_GPT2/run_checkpoint_test.py
+++ b/tests/model/Megatron_GPT2/run_checkpoint_test.py
--- a/tests/model/Megatron_GPT2/run_func_test.py
+++ b/tests/model/Megatron_GPT2/run_func_test.py
--- a/tests/model/run_sanity_check.py
+++ b/tests/model/run_sanity_check.py
--- a/tests/small_model_debugging/test_model.py
+++ b/tests/small_model_debugging/test_model.py