Skip to content

GitLab

Commit bba90f79 authored by Mohammad Shoeybi's avatar Mohammad Shoeybi
Browse files

Merge branch 'preprocess_fix' into 'main' 

Add arg needed for tokenizer.

See merge request ADLR/megatron-lm!272
parents 37475717 2c247afb
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
tools/preprocess_data.py
View file @ bba90f79
...@@ -137,6 +137,7 @@ def get_args(): ...@@ -137,6 +137,7 @@ def get_args():
args.rank = 0 args.rank = 0
args.make_vocab_size_divisible_by = 128 args.make_vocab_size_divisible_by = 128
args.tensor_model_parallel_size = 1 args.tensor_model_parallel_size = 1
args.vocab_extra_ids = 0
return args return args
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment