only automatically rescale the lr for sgd optimizers.

Summary:
Pull Request resolved: https://github.com/facebookresearch/d2go/pull/377

Only automatically rescale the lr for sgd optimizers.
* Seems that only sgd needs scaling the lr, so we change to not to scale lr automatically by default. This will work better for newly added optimizers (like adam).

Reviewed By: itomatik, lg-zhang

Differential Revision: D39899434

fbshipit-source-id: d6eebc5b07d4489b401c1fc3cea00f5a060fe19d

d2go/runner/default_runner.py

@@ -95,12 +95,11 @@ def default_scale_d2_configs(cfg, new_world_size):
     # lr scale
     lr_scales = {
         "sgd": gpu_scale,
-        "adamw": 1,
         "sgd_mt": gpu_scale,
-        "adamw_mt": 1,
     }
     optim_name = cfg.SOLVER.OPTIMIZER.lower()
-    lr_scale = lr_scales[optim_name] if optim_name in lr_scales else gpu_scale
+    # only scale the lr for the optimizers specified in `lr_scales`
+    lr_scale = lr_scales.get(optim_name, 1.0)
 
     # default configs in D2
     cfg.SOLVER.BASE_LR = base_lr * lr_scale