Skip to content

GitLab

Commit a29890c3 authored by Mostofa Patwary's avatar Mostofa Patwary
Browse files

added threads

parent 90c9e3f2
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
tools/openwebtext/filter_ngrams.py
View file @ a29890c3
...@@ -284,7 +284,7 @@ if __name__ == '__main__': ...@@ -284,7 +284,7 @@ if __name__ == '__main__':
dedup_key = args.dedup_dataset[1] dedup_key = args.dedup_dataset[1]
# Setup multi-processing. # Setup multi-processing.
num_workers = 1 #40 num_workers = 40
fin = open(dedup_file, 'r', encoding='utf-8') fin = open(dedup_file, 'r', encoding='utf-8')
pool = multiprocessing.Pool(num_workers) pool = multiprocessing.Pool(num_workers)
free_ngram_x=partial(free_ngram, args=args, key=dedup_key, ngrams=ngrams, \ free_ngram_x=partial(free_ngram, args=args, key=dedup_key, ngrams=ngrams, \
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment