Merge branch 'preprocess_fix' into 'main'

Add arg needed for tokenizer. See merge request ADLR/megatron-lm!272

Merge branch 'preprocess_fix' into 'main'
Add arg needed for tokenizer. See merge request ADLR/megatron-lm!272
bba90f79 · Mohammad Shoeybi · 37475717 · 2c247afb · bba90f79
Commit bba90f79 authored May 14, 2021 by Mohammad Shoeybi
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 0 deletions

tools/preprocess_data.py tools/preprocess_data.py +1 -0

No files found.
--- a/tools/preprocess_data.py
+++ b/tools/preprocess_data.py
@@ -137,6 +137,7 @@ def get_args():
    args.rank = 0
    args.make_vocab_size_divisible_by = 128
    args.tensor_model_parallel_size = 1
+    args.vocab_extra_ids = 0

    return args