Skip to content

GitLab

Commit 72e4230b authored by Deepak Narayanan's avatar Deepak Narayanan
Browse files

Turn scatter-gather optimization on by default

parent dcef9069
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 2 deletions
megatron/arguments.py
View file @ 72e4230b
...@@ -578,8 +578,9 @@ def _add_distributed_args(parser): ...@@ -578,8 +578,9 @@ def _add_distributed_args(parser):
choices=['local', 'torch'], choices=['local', 'torch'],
help='which DistributedDataParallel implementation ' help='which DistributedDataParallel implementation '
'to use.') 'to use.')
group.add_argument('--scatter-gather-tensors-in-pipeline', action='store_true', group.add_argument('--no-scatter-gather-tensors-in-pipeline', action='store_false',
help='Use scatter/gather to optimize communication of tensors in pipeline') help='Use scatter/gather to optimize communication of tensors in pipeline',
dest='scatter_gather_tensors_in_pipeline')
group.add_argument('--local_rank', type=int, default=None, group.add_argument('--local_rank', type=int, default=None,
help='local rank passed from distributed launcher.') help='local rank passed from distributed launcher.')
group.add_argument('--lazy-mpu-init', type=bool, required=False, group.add_argument('--lazy-mpu-init', type=bool, required=False,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment