Skip to content

GitLab

Commit 2c247afb authored by Jared Casper's avatar Jared Casper
Browse files

Add arg needed for tokenizer.

parent 37475717
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
tools/preprocess_data.py
View file @ 2c247afb
...@@ -137,6 +137,7 @@ def get_args(): ...@@ -137,6 +137,7 @@ def get_args():
args.rank = 0 args.rank = 0
args.make_vocab_size_divisible_by = 128 args.make_vocab_size_divisible_by = 128
args.tensor_model_parallel_size = 1 args.tensor_model_parallel_size = 1
args.vocab_extra_ids = 0
return args return args
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment