Fix multilingual translation bug for to-many case

Summary:
The logic for adding decoder side language token was wrongly implemented.
The way we inject the language token is by replacing the eos symbol with language token symbol. However, the parameter for source / target eos symbol was not set correctly.

Reviewed By: tangyuq

Differential Revision: D17129108

fbshipit-source-id: 6fae385b787370656fd7ca7ab74e6bb91fe5463b

fairseq/tasks/multilingual_translation.py

@@ -214,8 +214,9 @@ class MultilingualTranslationTask(FairseqTask):
             )
             return self.alter_dataset_langtok(
                 langpair_dataset,
-                src_eos=self.dicts[tgt].eos(),
+                src_eos=self.dicts[src].eos(),
                 src_lang=src,
+                tgt_eos=self.dicts[tgt].eos(),
                 tgt_lang=tgt,
             )
 
@@ -239,6 +240,7 @@ class MultilingualTranslationTask(FairseqTask):
                     ),
                     src_eos=self.source_dictionary.eos(),
                     src_lang=self.args.source_lang,
+                    tgt_eos=self.target_dictionary.eos(),
                     tgt_lang=self.args.target_lang,
                 ),
             )]),