Skip to content

GitLab

Commit 5bf07724 authored by alexeib's avatar alexeib Committed by Myle Ott
Browse files

sort descending when evaluating lm because it is faster (17k wps vs 11k) and will fail early if oom

parent a091d239
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
fairseq/data/monolingual_dataset.py
View file @ 5bf07724
...@@ -72,7 +72,7 @@ class MonolingualDataset(FairseqDataset): ...@@ -72,7 +72,7 @@ class MonolingualDataset(FairseqDataset):
order = [np.random.permutation(len(self))] order = [np.random.permutation(len(self))]
else: else:
order = [np.arange(len(self))] order = [np.arange(len(self))]
order.append(self.sizes) order.append(np.flip(self.sizes, 0))
return np.lexsort(order) return np.lexsort(order)
def valid_size(self, index, max_positions): def valid_size(self, index, max_positions):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment