fix one more issue

0c01c2fe · Mostofa Patwary · 43d307d5 · 0c01c2fe
Commit 0c01c2fe authored Mar 29, 2021 by Mostofa Patwary
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

tools/openwebtext/cleanup_fix_dataset.py tools/openwebtext/cleanup_fix_dataset.py +1 -1

No files found.
--- a/tools/openwebtext/cleanup_fix_dataset.py
+++ b/tools/openwebtext/cleanup_fix_dataset.py
@@ -75,7 +75,7 @@ def process_doc(json_line, args):
    except Exception as e:
        print('Error: *************************\n{}\ntext: {}'.format(e, \
            text), flush=True)
-        return output, text, True
+        return output, text, document, True

    # don't remove
    return output, text, document, False