Skip to content

GitLab

Commit 2946e10e authored by wxj's avatar wxj
Browse files

Update arguments.py

parent 0d0a2a22
Pipeline #2032 passed with stage
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
megatron/training/arguments.py
View file @ 2946e10e
...@@ -1703,6 +1703,7 @@ def _add_data_args(parser): ...@@ -1703,6 +1703,7 @@ def _add_data_args(parser):
'GPTSentencePieceTokenizer', 'GPTSentencePieceTokenizer',
'HuggingFaceTokenizer', 'HuggingFaceTokenizer',
'Llama2Tokenizer', 'Llama2Tokenizer',
'Llama3Tokenizer',
'QwenTokenizer', 'QwenTokenizer',
'TikTokenizer', 'TikTokenizer',
'NullTokenizer'], 'NullTokenizer'],
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment