Skip to content

GitLab

Commit 7a6d630e authored by Raul Puri's avatar Raul Puri
Browse files

undid padding merge

parent b9b6fe0d
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 3 deletions
megatron/utils.py
View file @ 7a6d630e
......@@ -275,9 +275,8 @@ def vocab_size_with_padding(num_tokens, args):
after = num_tokens
multiple = args.make_vocab_size_divisible_by * \
mpu.get_model_parallel_world_size()
if multiple > 0:
while (after % multiple) != 0:
after += 1
while (after % multiple) != 0:
after += 1
print_rank_0('> padded vocab (size: {}) with {} dummy '
'tokens (new size: {})'.format(
num_tokens, after - num_tokens, after))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment