Add fairspeq task to train ASR model with auxiliary data. (#813)

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/813

Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/663

Pull Request resolved: https://github.com/fairinternal/fairspeq/pull/4

Introduce new training for speech models which accept additional training data.

Reviewed By: liezl200

Differential Revision: D15846661

fbshipit-source-id: 8b2cbfd56a86cf03c0b34c4a025bebdd5db7204e

fairseq/data/data_utils.py

@@ -102,6 +102,11 @@ def filter_by_size(indices, size_fn, max_positions, raise_exception=False):
                 for key in intersect_keys
             )
         else:
+            # Hacky as heck, for the specific case of multilingual training with RoundRobin.
+            if isinstance(size_fn(idx), dict) and isinstance(max_positions, tuple):
+                return all(a is None or b is None or a <= b
+                           for a, b in zip(size_fn(idx).values(), max_positions)
+                )
             # For MultiCorpusSampledDataset, will generalize it later
             if not isinstance(size_fn(idx), Iterable):
                 return all(size_fn(idx) <= b for b in max_positions)