Merge branch 'move_openwebtext' into 'staging'

Move openwebtext scripts to tools directory. See merge request ADLR/megatron-lm!50

Merge branch 'move_openwebtext' into 'staging'
Move openwebtext scripts to tools directory. See merge request ADLR/megatron-lm!50
c99fa80c · Mohammad Shoeybi · 69d8ff1c · 0dc10d09 · c99fa80c · c99fa80c
Commit c99fa80c authored Apr 14, 2020 by Mohammad Shoeybi
7 changed files
--- a/openwebtext/README.md
+++ b/openwebtext/README.md
--- a/openwebtext/blacklist_urls.py
+++ b/openwebtext/blacklist_urls.py
--- a/openwebtext/cleanup_dataset.py
+++ b/openwebtext/cleanup_dataset.py
--- a/openwebtext/find_duplicates.py
+++ b/openwebtext/find_duplicates.py
--- a/openwebtext/group_duplicates_url.py
+++ b/openwebtext/group_duplicates_url.py
--- a/openwebtext/merge_jsons.py
+++ b/openwebtext/merge_jsons.py
--- a/openwebtext/remove_group_duplicates.py
+++ b/openwebtext/remove_group_duplicates.py