Skip to content

GitLab

Commit 8676baca authored by Rewon Child's avatar Rewon Child
Browse files

Update comments

parent 14c85e64
Show whitespace changes
Inline Side-by-side
Showing with 0 additions and 2 deletions
megatron/optimizer/clip_grads.py
View file @ 8676baca
...@@ -138,10 +138,8 @@ def count_zeros_fp32(parameters): ...@@ -138,10 +138,8 @@ def count_zeros_fp32(parameters):
grad = param.grad.detach() grad = param.grad.detach()
grads_to_count.append(grad) grads_to_count.append(grad)
# Norm parameters.
total_num_zeros = 0.0 total_num_zeros = 0.0
# Calculate norm.
for grad in grads_to_count: for grad in grads_to_count:
num_zeros = grad.numel() - torch.count_nonzero(grad) num_zeros = grad.numel() - torch.count_nonzero(grad)
total_num_zeros = num_zeros + total_num_zeros total_num_zeros = num_zeros + total_num_zeros
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment