Skip to content

GitLab

"git@developer.sourcefind.cn:orangecat/ollama.git" did not exist on "c75cafdb58632b21b8208715623d2e601c897cbc"
Unverified Commit f9b81151 authored by researcher2's avatar researcher2 Committed by GitHub
Browse files

Update sort_13_gram_buckets.py

parent 985f1f1a
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
scripts/clean_training_data/sort_13_gram_buckets.py
View file @ f9b81151
""" """
Iteratively runs gnu sort on each bucket, which does automatic the multiprocessing up to 8. Iteratively runs gnu sort on each bucket, uses up to 8 cores.
Arguments Arguments
--------- ---------
...@@ -55,4 +55,4 @@ if __name__ == '__main__': ...@@ -55,4 +55,4 @@ if __name__ == '__main__':
setup_logger_tqdm(logfile_path) setup_logger_tqdm(logfile_path)
args = parser.parse_args() args = parser.parse_args()
sort_13_gram_buckets(args.working_directory) sort_13_gram_buckets(args.working_directory)
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment