Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
ChatGLM-6B_pytorch
Commits
4fd83505
Commit
4fd83505
authored
Sep 22, 2023
by
zhaoying1
Browse files
Add new file
parent
cdab2fb2
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
32 additions
and
0 deletions
+32
-0
ptuning/deepspeed_zero2.json
ptuning/deepspeed_zero2.json
+32
-0
No files found.
ptuning/deepspeed_zero2.json
0 → 100644
View file @
4fd83505
{
"train_micro_batch_size_per_gpu"
:
"auto"
,
"gradient_accumulation_steps"
:
"auto"
,
"zero_allow_untested_optimizer"
:
true
,
"fp16"
:
{
"enabled"
:
"auto"
,
"loss_scale"
:
0
,
"initial_scale_power"
:
16
,
"loss_scale_window"
:
1000
,
"hysteresis"
:
2
,
"min_loss_scale"
:
1
},
"optimizer"
:
{
"type"
:
"AdamW"
,
"params"
:
{
"lr"
:
"auto"
,
"betas"
:
"auto"
,
"eps"
:
"auto"
,
"weight_decay"
:
"auto"
}
},
"zero_optimization"
:
{
"stage"
:
2
,
"stage3_gather_16bit_weights_on_model_save"
:
true
,
"allgather_partitions"
:
true
,
"allgather_bucket_size"
:
5e8
,
"overlap_comm"
:
false
,
"reduce_scatter"
:
true
,
"reduce_bucket_size"
:
5e8
,
"contiguous_gradients"
:
true
}
}
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment