Support for BPE vocabs + denoising autoencoder in PyTorch Translate (#362)

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/362

Pull Request resolved: https://github.com/pytorch/translate/pull/254

This actually uses the fairseq logic which supports BPE cont / end word marker suffixes.

Reviewed By: xianxl

Differential Revision: D12952766

fbshipit-source-id: 35a1bbc38240e4145bec0fc419f2d0a6a73ae2e5

fairseq/data/noising.py

@@ -83,7 +83,7 @@ class WordDropout(WordNoising):
         assert 0 < dropout_prob < 1
 
         # be sure to drop entire words
-        word_idx = self._get_bpe_word_idx(x)
+        word_idx = self.get_word_idx(x)
         sentences = []
         modified_lengths = []
         for i in range(lengths.size(0)):