Merge pull request #1233 from searchivarius/master

Fix to prevent crashing on assert len(tokens_b)>=1

Merge pull request #1233 from searchivarius/master
Fix to prevent crashing on assert len(tokens_b)>=1
99a54ac5 · Thomas Wolf · GitHub · 439b37b4 · eab980fd · 99a54ac5
Unverified Commit 99a54ac5 authored Sep 10, 2019 by Thomas Wolf Committed by GitHub Sep 10, 2019
Hide whitespace changes
Inline Side-by-side

Showing with 2 additions and 1 deletion

examples/lm_finetuning/pregenerate_training_data.py examples/lm_finetuning/pregenerate_training_data.py +2 -1

No files found.
--- a/examples/lm_finetuning/pregenerate_training_data.py
+++ b/examples/lm_finetuning/pregenerate_training_data.py
@@ -329,7 +329,8 @@ def main():
                    doc = []
                else:
                    tokens = tokenizer.tokenize(line)
-                    doc.append(tokens)
+                    if tokens:
+                        doc.append(tokens)
            if doc:
                docs.add_document(doc)  # If the last doc didn't end on a newline, make sure it still gets added
        if len(docs) <= 1: