Skip to content

GitLab

Unverified Commit 838f53b7 authored by Shaden Smith's avatar Shaden Smith Committed by GitHub
Browse files

Switches BBS example to use mbsize=3 and gas=2 to fit in 16GB of memory. (#341)

parent e8dd47df
Show whitespace changes
Inline Side-by-side
Showing with 5 additions and 5 deletions
tests/model/BingBertSquad/deepspeed_bsz24_fp16_config.json
View file @ 838f53b7
{ {
"train_batch_size": 24, "train_batch_size": 24,
"train_micro_batch_size_per_gpu": 6, "train_micro_batch_size_per_gpu": 3,
"steps_per_print": 1, "steps_per_print": 1,
"optimizer": { "optimizer": {
"type": "Adam", "type": "Adam",
......
tests/model/BingBertSquad/deepspeed_bsz24_fp16_zero2_config.json
View file @ 838f53b7
{ {
"train_batch_size": 24, "train_batch_size": 24,
"train_micro_batch_size_per_gpu": 6, "train_micro_batch_size_per_gpu": 3,
"steps_per_print": 1, "steps_per_print": 1,
"optimizer": { "optimizer": {
"type": "Adam", "type": "Adam",
......
tests/model/BingBertSquad/deepspeed_bsz24_fp32_config.json
View file @ 838f53b7
{ {
"train_batch_size": 24, "train_batch_size": 24,
"train_micro_batch_size_per_gpu": 6, "train_micro_batch_size_per_gpu": 3,
"steps_per_print": 1, "steps_per_print": 1,
"optimizer": { "optimizer": {
"type": "Adam", "type": "Adam",
......
tests/model/BingBertSquad/run_BingBertSquad.sh
View file @ 838f53b7
...@@ -121,7 +121,7 @@ echo "deepspeed: ${enable_deepspeed}" ...@@ -121,7 +121,7 @@ echo "deepspeed: ${enable_deepspeed}"
echo "other_args: ${other_args}" echo "other_args: ${other_args}"
EFFECTIVE_BATCH_SIZE=${batch_size} EFFECTIVE_BATCH_SIZE=${batch_size}
MAX_GPU_BATCH_SIZE=6 MAX_GPU_BATCH_SIZE=3
PER_GPU_BATCH_SIZE=$((EFFECTIVE_BATCH_SIZE/num_gpus)) PER_GPU_BATCH_SIZE=$((EFFECTIVE_BATCH_SIZE/num_gpus))
if [[ $PER_GPU_BATCH_SIZE -lt $MAX_GPU_BATCH_SIZE ]]; then if [[ $PER_GPU_BATCH_SIZE -lt $MAX_GPU_BATCH_SIZE ]]; then
GRAD_ACCUM_STEPS=1 GRAD_ACCUM_STEPS=1
......
tests/model/BingBertSquad/run_BingBertSquad_sanity.sh
View file @ 838f53b7
...@@ -122,7 +122,7 @@ echo "deepspeed: ${enable_deepspeed}" ...@@ -122,7 +122,7 @@ echo "deepspeed: ${enable_deepspeed}"
echo "other_args: ${other_args}" echo "other_args: ${other_args}"
EFFECTIVE_BATCH_SIZE=${batch_size} EFFECTIVE_BATCH_SIZE=${batch_size}
MAX_GPU_BATCH_SIZE=6 MAX_GPU_BATCH_SIZE=3
PER_GPU_BATCH_SIZE=$((EFFECTIVE_BATCH_SIZE/num_gpus)) PER_GPU_BATCH_SIZE=$((EFFECTIVE_BATCH_SIZE/num_gpus))
if [[ $PER_GPU_BATCH_SIZE -lt $MAX_GPU_BATCH_SIZE ]]; then if [[ $PER_GPU_BATCH_SIZE -lt $MAX_GPU_BATCH_SIZE ]]; then
GRAD_ACCUM_STEPS=1 GRAD_ACCUM_STEPS=1
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment