Merge branch 'staging' into 'master'

April 2020 Release See merge request ADLR/megatron-lm!69

Merge branch 'staging' into 'master'
April 2020 Release See merge request ADLR/megatron-lm!69
22c0e300 · Jared Casper · 70174ae3 · acf8780b · 22c0e300 · 22c0e300
Commit 22c0e300 authored Apr 16, 2020 by Jared Casper
Expand all Hide whitespace changes
Inline Side-by-side

Showing with 201 additions and 1 deletion

tools/openwebtext/remove_group_duplicates.py tools/openwebtext/remove_group_duplicates.py +1 -1

tools/preprocess_data.py tools/preprocess_data.py +200 -0

No files found.
--- a/openwebtext/remove_group_duplicates.py
+++ b/openwebtext/remove_group_duplicates.py
--- a/tools/preprocess_data.py
+++ b/tools/preprocess_data.py