Skip to content

GitLab

"...git@developer.sourcefind.cn:OpenDAS/colossalai.git" did not exist on "6c0fa7b9a858f1d630a6addd6ba988b63bc9f61f"
Unverified Commit ec912b1b authored by 李文军's avatar 李文军 Committed by GitHub
Browse files

[NFC] polish... 

[NFC] polish applications/Colossal-LLaMA-2/colossal_llama2/tokenizer/init_tokenizer.py code style (#5228)
parent ddf879e2
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
applications/Colossal-LLaMA-2/colossal_llama2/tokenizer/init_tokenizer.py
View file @ ec912b1b
...@@ -6,12 +6,12 @@ Initialize new tokenizer for continual pre-training ...@@ -6,12 +6,12 @@ Initialize new tokenizer for continual pre-training
""" """
import argparse import argparse
import os
import json import json
import os
from typing import List, Union from typing import List, Union
from transformers.models.llama.tokenization_llama import LlamaTokenizer
from sentencepiece import sentencepiece_model_pb2 as sp_pb2_model from sentencepiece import sentencepiece_model_pb2 as sp_pb2_model
from transformers.models.llama.tokenization_llama import LlamaTokenizer
from colossalai.logging import get_dist_logger from colossalai.logging import get_dist_logger
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment