Skip to content

GitLab

"src/vscode:/vscode.git/clone" did not exist on "cd21b965d1d10266d399b887f77b7dfc429f5f04"
Unverified Commit 01e848b3 authored by Samyam Rajbhandari's avatar Samyam Rajbhandari Committed by GitHub
Browse files

Default Contiguous Gradients False (#239) 

Contiguous Gradients should be set to false by default. Its not useful unless the model is very large
parent d24d3de9
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
deepspeed/pt/deepspeed_zero_config.py
View file @ 01e848b3
......@@ -51,7 +51,7 @@ ZERO_OPTIMIZATION_OVERLAP_COMM = 'overlap_comm'
ZERO_OPTIMIZATION_OVERLAP_COMM_DEFAULT = False
ZERO_OPTIMIZATION_CONTIGUOUS_GRADIENTS = 'contiguous_gradients'
ZERO_OPTIMIZATION_CONTIGUOUS_GRADIENTS_DEFAULT = True
ZERO_OPTIMIZATION_CONTIGUOUS_GRADIENTS_DEFAULT = False
ZERO_OPTIMIZATION_REDUCE_BUCKET_SIZE = 'reduce_bucket_size'
ZERO_OPTIMIZATION_REDUCE_BUCKET_SIZE_DEFAULT = 500000000
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment