Commit 2946e10e authored by wxj's avatar wxj
Browse files

Update arguments.py

parent 0d0a2a22
Pipeline #2032 passed with stage
......@@ -1703,6 +1703,7 @@ def _add_data_args(parser):
'GPTSentencePieceTokenizer',
'HuggingFaceTokenizer',
'Llama2Tokenizer',
'Llama3Tokenizer',
'QwenTokenizer',
'TikTokenizer',
'NullTokenizer'],
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment