Fixing BMUF warmup and sync strategy

Summary:
Bmuf sync started happening even before warmup is done.
This diff fixes the behavior and do bmuf sync once warmup is done or if it's zero.

TODO: write a unit test case so that these problems can be figure out faster.

Reviewed By: jay-mahadeokar

Differential Revision: D17356277

fbshipit-source-id: 21500e6ed1225b97794e4ee203e5d7d04a2840f8

fairseq/optim/bmuf.py

@@ -127,7 +127,9 @@ class FairseqBMUF(FairseqOptimizer):
 
     def _is_bmuf_iter(self):
         # Check whether train iterations is equal to bmuf sync iter
-        if self.get_num_updates() % self.sync_iter == 0:
+        if (self.get_num_updates() > self.warmup_iteration) and (
+            self.get_num_updates() % self.sync_iter == 0
+        ):
             return True
         return False