Revert incorrect fix.

cd499559 · Jared Casper · 2a34e0ec · cd499559
Commit cd499559 authored Jan 28, 2022 by Jared Casper
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

megatron/utils.py megatron/utils.py +1 -1

No files found.
--- a/megatron/utils.py
+++ b/megatron/utils.py
@@ -84,7 +84,7 @@ def average_losses_across_data_parallel_group(losses):
        [loss.clone().detach().view(1) for loss in losses])
    torch.distributed.all_reduce(averaged_losses,
                                 group=mpu.get_data_parallel_group())
-    averaged_losses = averaged_losses /
+    averaged_losses = averaged_losses / \
        torch.distributed.get_world_size(group=mpu.get_data_parallel_group())

    return averaged_losses