Internal changes for object detection. (#3656)

* Force cast of num_classes to integer

PiperOrigin-RevId: 188335318

* Updating config util to allow overwriting of cosine decay learning rates.

PiperOrigin-RevId: 188338852

* Make box_list_ops.py and box_list_ops_test.py work with C API enabled.

The C API has improved shape inference over the original Python
code. This causes some previously-working conds to fail. Switching to smart_cond fixes this.

Another effect of the improved shape inference is that one of the
failures tested gets caught earlier, so I modified the test to reflect
this.

PiperOrigin-RevId: 188409792

* Fix parallel event file writing issue.

Without this change, the event files might get corrupted when multiple evaluations are run in parallel.

PiperOrigin-RevId: 188502560

* Deprecating the boolean flag of from_detection_checkpoint.

Replace with a string field fine_tune_checkpoint_type to train_config to provide extensibility. The fine_tune_checkpoint_type can currently take value of `detection`, `classification`, or others when the restore_map is overwritten.

PiperOrigin-RevId: 188518685

* Automated g4 rollback of changelist 188502560

PiperOrigin-RevId: 188519969

* Introducing eval metrics specs for Coco Mask metrics. This allows metrics to be computed in tensorflow using the tf.learn Estimator.

PiperOrigin-RevId: 188528485

* Minor fix to make object_detection/metrics/coco_evaluation.py python3 compatible.

PiperOrigin-RevId: 188550683

* Updating eval_util to handle eval_metric_ops from multiple `DetectionEvaluator`s.

PiperOrigin-RevId: 188560474

* Allow tensor input for new_height and new_width for resize_image.

PiperOrigin-RevId: 188561908

* Fix typo in fine_tune_checkpoint_type name in trainer.

PiperOrigin-RevId: 188799033

* Adding mobilenet feature extractor to object detection.

PiperOrigin-RevId: 188916897

* Allow label maps to optionally contain an explicit background class with id zero.

PiperOrigin-RevId: 188951089

* Fix boundary conditions in random_pad_to_aspect_ratio to ensure that min_scale is always less than max_scale.

PiperOrigin-RevId: 189026868

* Fallback on from_detection_checkpoint option if fine_tune_checkpoint_type isn't set.

PiperOrigin-RevId: 189052833

* Add proper names for learning rate schedules so we don't see cryptic names on tensorboard.

PiperOrigin-RevId: 189069837

* Enforcing that all datasets are batched (and then unbatched in the model) with batch_size >= 1.

PiperOrigin-RevId: 189117178

* Adding regularization to total loss returned from DetectionModel.loss().

PiperOrigin-RevId: 189189123

* Standardize the names of loss scalars (for SSD, Faster R-CNN and R-FCN) in both training and eval so they can be compared on tensorboard.

Log localization and classification losses in evaluation.

PiperOrigin-RevId: 189189940

* Remove negative test from box list ops test.

PiperOrigin-RevId: 189229327

* Add an option to warmup learning rate in manual stepping schedule.

PiperOrigin-RevId: 189361039

* Replace tf.contrib.slim.tfexample_decoder.LookupTensor with object_detection.data_decoders.tf_example_decoder.LookupTensor.

PiperOrigin-RevId: 189388556

* Force regularization summary variables under specific family names.

PiperOrigin-RevId: 189393190

* Automated g4 rollback of changelist 188619139

PiperOrigin-RevId: 189396001

* Remove step 0 schedule since we do a hard check for it after cl/189361039

PiperOrigin-RevId: 189396697

* PiperOrigin-RevId: 189040463

* PiperOrigin-RevId: 189059229

* PiperOrigin-RevId: 189214402

* Force regularization summary variables under specific family names.

PiperOrigin-RevId: 189393190

* Automated g4 rollback of changelist 188619139

PiperOrigin-RevId: 189396001

* Make slim python3 compatible.

* Monir fixes.

* Add TargetAssignment summaries in a separate family.

PiperOrigin-RevId: 189407487

* 1. Setting `family` keyword arg prepends the summary names twice with the same name. Directly adding family suffix to the name gets rid of this problem.
2. Make sure the eval losses have the same name.

PiperOrigin-RevId: 189434618

* Minor fixes to make object detection tf 1.4 compatible.

PiperOrigin-RevId: 189437519

* Call the base of mobilenet_v1 feature extractor under the right arg scope and set batchnorm is_training based on the value passed in the constructor.

PiperOrigin-RevId: 189460890

* Automated g4 rollback of changelist 188409792

PiperOrigin-RevId: 189463882

* Update object detection syncing.

PiperOrigin-RevId: 189601955

* Add an option to warmup learning rate, hold it constant for a certain number of steps and cosine decay it.

PiperOrigin-RevId: 189606169

* Let the proposal feature extractor function in faster_rcnn meta architectures return the activations (end_points).

PiperOrigin-RevId: 189619301

* Fixed bug which caused masks to be mostly zeros (caused by detection_boxes being in absolute coordinates if scale_to_absolute=True.

PiperOrigin-RevId: 189641294

* Open sourcing Mobilenetv2 + SSDLite.

PiperOrigin-RevId: 189654520

* Remove unused files.

research/object_detection/samples/configs/mask_rcnn_inception_resnet_v2_atrous_coco.config

@@ -110,10 +110,6 @@ train_config: {
       learning_rate: {
         manual_step_learning_rate {
           initial_learning_rate: 0.0003
-          schedule {
-            step: 0
-            learning_rate: .0003
-          }
           schedule {
             step: 900000
             learning_rate: .00003

research/object_detection/samples/configs/mask_rcnn_inception_v2_coco.config

@@ -109,10 +109,6 @@ train_config: {
       learning_rate: {
         manual_step_learning_rate {
           initial_learning_rate: 0.0002
-          schedule {
-            step: 0
-            learning_rate: .0002
-          }
           schedule {
             step: 900000
             learning_rate: .00002

research/object_detection/samples/configs/mask_rcnn_resnet101_atrous_coco.config

@@ -110,10 +110,6 @@ train_config: {
       learning_rate: {
         manual_step_learning_rate {
           initial_learning_rate: 0.0003
-          schedule {
-            step: 0
-            learning_rate: .0003
-          }
           schedule {
             step: 900000
             learning_rate: .00003

research/object_detection/samples/configs/mask_rcnn_resnet101_pets.config

@@ -103,10 +103,6 @@ train_config: {
       learning_rate: {
         manual_step_learning_rate {
           initial_learning_rate: 0.0007
-          schedule {
-            step: 0
-            learning_rate: 0.0007
-          }
           schedule {
             step: 15000
             learning_rate: 0.00007

research/object_detection/samples/configs/mask_rcnn_resnet50_atrous_coco.config

@@ -110,10 +110,6 @@ train_config: {
       learning_rate: {
         manual_step_learning_rate {
           initial_learning_rate: 0.0003
-          schedule {
-            step: 0
-            learning_rate: .0003
-          }
           schedule {
             step: 900000
             learning_rate: .00003

research/object_detection/samples/configs/rfcn_resnet101_coco.config

@@ -85,10 +85,6 @@ train_config: {
       learning_rate: {
         manual_step_learning_rate {
           initial_learning_rate: 0.0003
-          schedule {
-            step: 0
-            learning_rate: .0003
-          }
           schedule {
             step: 900000
             learning_rate: .00003

research/object_detection/samples/configs/rfcn_resnet101_pets.config

@@ -85,10 +85,6 @@ train_config: {
       learning_rate: {
         manual_step_learning_rate {
           initial_learning_rate: 0.0003
-          schedule {
-            step: 0
-            learning_rate: .0003
-          }
           schedule {
             step: 900000
             learning_rate: .00003

research/object_detection/samples/configs/ssd_inception_v2_pets.config

@@ -162,6 +162,7 @@ train_config: {
     ssd_random_crop {
     }
   }
+  max_number_of_boxes: 50
 }
 
 train_input_reader: {

research/object_detection/samples/configs/ssd_mobilenet_v2_coco.config

+# SSD with Mobilenet v2 configuration for MSCOCO Dataset.
+# Users should configure the fine_tune_checkpoint field in the train config as
+# well as the label_map_path and input_path fields in the train_input_reader and
+# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
+# should be configured.
+
+model {
+  ssd {
+    num_classes: 90
+    box_coder {
+      faster_rcnn_box_coder {
+        y_scale: 10.0
+        x_scale: 10.0
+        height_scale: 5.0
+        width_scale: 5.0
+      }
+    }
+    matcher {
+      argmax_matcher {
+        matched_threshold: 0.5
+        unmatched_threshold: 0.5
+        ignore_thresholds: false
+        negatives_lower_than_unmatched: true
+        force_match_for_each_row: true
+      }
+    }
+    similarity_calculator {
+      iou_similarity {
+      }
+    }
+    anchor_generator {
+      ssd_anchor_generator {
+        num_layers: 6
+        min_scale: 0.2
+        max_scale: 0.95
+        aspect_ratios: 1.0
+        aspect_ratios: 2.0
+        aspect_ratios: 0.5
+        aspect_ratios: 3.0
+        aspect_ratios: 0.3333
+      }
+    }
+    image_resizer {
+      fixed_shape_resizer {
+        height: 300
+        width: 300
+      }
+    }
+    box_predictor {
+      convolutional_box_predictor {
+        min_depth: 0
+        max_depth: 0
+        num_layers_before_predictor: 0
+        use_dropout: false
+        dropout_keep_probability: 0.8
+        kernel_size: 3
+        box_code_size: 4
+        apply_sigmoid_to_scores: false
+        conv_hyperparams {
+          activation: RELU_6,
+          regularizer {
+            l2_regularizer {
+              weight: 0.00004
+            }
+          }
+          initializer {
+            truncated_normal_initializer {
+              stddev: 0.03
+              mean: 0.0
+            }
+          }
+          batch_norm {
+            train: true,
+            scale: true,
+            center: true,
+            decay: 0.9997,
+            epsilon: 0.001,
+          }
+        }
+      }
+    }
+    feature_extractor {
+      type: 'ssd_mobilenet_v2'
+      min_depth: 16
+      depth_multiplier: 1.0
+      use_depthwise: true
+      conv_hyperparams {
+        activation: RELU_6,
+        regularizer {
+          l2_regularizer {
+            weight: 0.00004
+          }
+        }
+        initializer {
+          truncated_normal_initializer {
+            stddev: 0.03
+            mean: 0.0
+          }
+        }
+        batch_norm {
+          train: true,
+          scale: true,
+          center: true,
+          decay: 0.9997,
+          epsilon: 0.001,
+        }
+      }
+      batch_norm_trainable: true
+    }
+    loss {
+      classification_loss {
+        weighted_sigmoid {
+        }
+      }
+      localization_loss {
+        weighted_smooth_l1 {
+        }
+      }
+      hard_example_miner {
+        num_hard_examples: 3000
+        iou_threshold: 0.99
+        loss_type: CLASSIFICATION
+        max_negatives_per_positive: 3
+        min_negatives_per_image: 3
+      }
+      classification_weight: 1.0
+      localization_weight: 1.0
+    }
+    normalize_loss_by_num_matches: true
+    post_processing {
+      batch_non_max_suppression {
+        score_threshold: 1e-8
+        iou_threshold: 0.6
+        max_detections_per_class: 100
+        max_total_detections: 100
+      }
+      score_converter: SIGMOID
+    }
+  }
+}
+
+train_config: {
+  batch_size: 24
+  optimizer {
+    rms_prop_optimizer: {
+      learning_rate: {
+        exponential_decay_learning_rate {
+          initial_learning_rate: 0.004
+          decay_steps: 800720
+          decay_factor: 0.95
+        }
+      }
+      momentum_optimizer_value: 0.9
+      decay: 0.9
+      epsilon: 1.0
+    }
+  }
+  fine_tune_checkpoint: "PATH_TO_BE_CONFIGURED/model.ckpt"
+  fine_tune_checkpoint_type:  "detection"
+  # Note: The below line limits the training process to 200K steps, which we
+  # empirically found to be sufficient enough to train the pets dataset. This
+  # effectively bypasses the learning rate schedule (the learning rate will
+  # never decay). Remove the below line to train indefinitely.
+  num_steps: 200000
+  data_augmentation_options {
+    random_horizontal_flip {
+    }
+  }
+  data_augmentation_options {
+    ssd_random_crop {
+    }
+  }
+}
+
+train_input_reader: {
+  tf_record_input_reader {
+    input_path: "PATH_TO_BE_CONFIGURED/mscoco_train.record"
+  }
+  label_map_path: "PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt"
+}
+
+eval_config: {
+  num_examples: 8000
+  # Note: The below line limits the evaluation process to 10 evaluations.
+  # Remove the below line to evaluate indefinitely.
+  max_evals: 10
+}
+
+eval_input_reader: {
+  tf_record_input_reader {
+    input_path: "PATH_TO_BE_CONFIGURED/mscoco_val.record"
+  }
+  label_map_path: "PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt"
+  shuffle: false
+  num_readers: 1
+}
\ No newline at end of file

research/object_detection/trainer.py

@@ -254,7 +254,7 @@ def train(create_tensor_dict_fn, create_model_fn, train_config, master, task,
       training_optimizer, optimizer_summary_vars = optimizer_builder.build(
           train_config.optimizer)
       for var in optimizer_summary_vars:
-        tf.summary.scalar(var.op.name, var)
+        tf.summary.scalar(var.op.name, var, family='LearningRate')
 
     sync_optimizer = None
     if train_config.sync_replicas:
@@ -267,8 +267,16 @@ def train(create_tensor_dict_fn, create_model_fn, train_config, master, task,
     # Create ops required to initialize the model from a given checkpoint.
     init_fn = None
     if train_config.fine_tune_checkpoint:
+      if not train_config.fine_tune_checkpoint_type:
+        # train_config.from_detection_checkpoint field is deprecated. For
+        # backward compatibility, fine_tune_checkpoint_type is set based on
+        # from_detection_checkpoint.
+        if train_config.from_detection_checkpoint:
+          train_config.fine_tune_checkpoint_type = 'detection'
+        else:
+          train_config.fine_tune_checkpoint_type = 'classification'
       var_map = detection_model.restore_map(
-          from_detection_checkpoint=train_config.from_detection_checkpoint,
+          fine_tune_checkpoint_type=train_config.fine_tune_checkpoint_type,
           load_all_detection_checkpoint_vars=(
               train_config.load_all_detection_checkpoint_vars))
       available_var_map = (variables_helper.
@@ -320,11 +328,13 @@ def train(create_tensor_dict_fn, create_model_fn, train_config, master, task,
 
     # Add summaries.
     for model_var in slim.get_model_variables():
-      global_summaries.add(tf.summary.histogram(model_var.op.name, model_var))
+      global_summaries.add(tf.summary.histogram('ModelVars/' +
+                                                model_var.op.name, model_var))
     for loss_tensor in tf.losses.get_losses():
-      global_summaries.add(tf.summary.scalar(loss_tensor.op.name, loss_tensor))
+      global_summaries.add(tf.summary.scalar('Losses/' + loss_tensor.op.name,
+                                             loss_tensor))
     global_summaries.add(
-        tf.summary.scalar('TotalLoss', tf.losses.get_total_loss()))
+        tf.summary.scalar('Losses/TotalLoss', tf.losses.get_total_loss()))
 
     # Add the summaries from the first clone. These contain the summaries
     # created by model_fn and either optimize_clones() or _gather_clone_loss().

research/object_detection/trainer_test.py

@@ -157,13 +157,14 @@ class FakeDetectionModel(model.DetectionModel):
     }
     return loss_dict
 
-  def restore_map(self, from_detection_checkpoint=True):
+  def restore_map(self, fine_tune_checkpoint_type='detection'):
     """Returns a map of variables to load from a foreign checkpoint.
 
     Args:
-      from_detection_checkpoint: whether to restore from a full detection
+      fine_tune_checkpoint_type: whether to restore from a full detection
         checkpoint (with compatible variable names) or to restore from a
         classification checkpoint for initialization prior to training.
+        Valid values: `detection`, `classification`. Default 'detection'.
 
     Returns:
       A dict mapping variable names to variables.

research/object_detection/utils/config_util.py

@@ -285,6 +285,9 @@ def merge_external_params_with_configs(configs, hparams=None, **kwargs):
 def _update_initial_learning_rate(configs, learning_rate):
   """Updates `configs` to reflect the new initial learning rate.
 
+  This function updates the initial learning rate. For learning rate schedules,
+  all other defined learning rates in the pipeline config are scaled to maintain
+  their same ratio with the initial learning rate.
   The configs dictionary is updated in place, and hence not returned.
 
   Args:
@@ -322,6 +325,13 @@ def _update_initial_learning_rate(configs, learning_rate):
     manual_lr.initial_learning_rate = learning_rate
     for schedule in manual_lr.schedule:
       schedule.learning_rate *= learning_rate_scaling
+  elif learning_rate_type == "cosine_decay_learning_rate":
+    cosine_lr = optimizer_config.learning_rate.cosine_decay_learning_rate
+    learning_rate_base = cosine_lr.learning_rate_base
+    warmup_learning_rate = cosine_lr.warmup_learning_rate
+    warmup_scale_factor = warmup_learning_rate / learning_rate_base
+    cosine_lr.learning_rate_base = learning_rate
+    cosine_lr.warmup_learning_rate = warmup_scale_factor * learning_rate
   else:
     raise TypeError("Learning rate %s is not supported." % learning_rate_type)
 

research/object_detection/utils/config_util_test.py

@@ -59,6 +59,14 @@ def _update_optimizer_with_manual_step_learning_rate(
     schedule.learning_rate = initial_learning_rate * learning_rate_scaling**i
 
 
+def _update_optimizer_with_cosine_decay_learning_rate(
+    optimizer, learning_rate, warmup_learning_rate):
+  """Adds a new cosine decay learning rate."""
+  cosine_lr = optimizer.learning_rate.cosine_decay_learning_rate
+  cosine_lr.learning_rate_base = learning_rate
+  cosine_lr.warmup_learning_rate = warmup_learning_rate
+
+
 class ConfigUtilTest(tf.test.TestCase):
 
   def test_get_configs_from_pipeline_file(self):
@@ -154,6 +162,7 @@ class ConfigUtilTest(tf.test.TestCase):
     """Asserts successful updating of all learning rate schemes."""
     original_learning_rate = 0.7
     learning_rate_scaling = 0.1
+    warmup_learning_rate = 0.07
     hparams = tf.contrib.training.HParams(learning_rate=0.15)
     pipeline_config_path = os.path.join(self.get_temp_dir(), "pipeline.config")
 
@@ -201,6 +210,24 @@ class ConfigUtilTest(tf.test.TestCase):
       self.assertAlmostEqual(hparams.learning_rate * learning_rate_scaling**i,
                              schedule.learning_rate)
 
+    # Cosine decay learning rate.
+    pipeline_config = pipeline_pb2.TrainEvalPipelineConfig()
+    optimizer = getattr(pipeline_config.train_config.optimizer, optimizer_name)
+    _update_optimizer_with_cosine_decay_learning_rate(optimizer,
+                                                      original_learning_rate,
+                                                      warmup_learning_rate)
+    _write_config(pipeline_config, pipeline_config_path)
+
+    configs = config_util.get_configs_from_pipeline_file(pipeline_config_path)
+    configs = config_util.merge_external_params_with_configs(configs, hparams)
+    optimizer = getattr(configs["train_config"].optimizer, optimizer_name)
+    cosine_lr = optimizer.learning_rate.cosine_decay_learning_rate
+
+    self.assertAlmostEqual(hparams.learning_rate, cosine_lr.learning_rate_base)
+    warmup_scale_factor = warmup_learning_rate / original_learning_rate
+    self.assertAlmostEqual(hparams.learning_rate * warmup_scale_factor,
+                           cosine_lr.warmup_learning_rate)
+
   def testRMSPropWithNewLearingRate(self):
     """Tests new learning rates for RMSProp Optimizer."""
     self._assertOptimizerWithNewLearningRate("rms_prop_optimizer")

research/object_detection/utils/label_map_util.py

@@ -32,8 +32,10 @@ def _validate_label_map(label_map):
     ValueError: if label map is invalid.
   """
   for item in label_map.item:
-    if item.id < 1:
-      raise ValueError('Label map ids should be >= 1.')
+    if item.id < 0:
+      raise ValueError('Label map ids should be >= 0.')
+    if item.id == 0 and item.name != 'background':
+      raise ValueError('Label map id 0 is reserved for the background label')
 
 
 def create_category_index(categories):

research/object_detection/utils/label_map_util_test.py

@@ -95,6 +95,30 @@ class LabelMapUtilTest(tf.test.TestCase):
     with self.assertRaises(ValueError):
       label_map_util.load_labelmap(label_map_path)
 
+  def test_load_label_map_with_background(self):
+    label_map_string = """
+      item {
+        id:0
+        name:'background'
+      }
+      item {
+        id:2
+        name:'cat'
+      }
+      item {
+        id:1
+        name:'dog'
+      }
+    """
+    label_map_path = os.path.join(self.get_temp_dir(), 'label_map.pbtxt')
+    with tf.gfile.Open(label_map_path, 'wb') as f:
+      f.write(label_map_string)
+
+    label_map_dict = label_map_util.get_label_map_dict(label_map_path)
+    self.assertEqual(label_map_dict['background'], 0)
+    self.assertEqual(label_map_dict['dog'], 1)
+    self.assertEqual(label_map_dict['cat'], 2)
+
   def test_keep_categories_with_unique_id(self):
     label_map_proto = string_int_label_map_pb2.StringIntLabelMap()
     label_map_string = """

research/object_detection/utils/learning_schedules.py

@@ -56,14 +56,15 @@ def exponential_decay_with_burnin(global_step,
   return tf.where(
       tf.less(tf.cast(global_step, tf.int32), tf.constant(burnin_steps)),
       tf.constant(burnin_learning_rate),
-      post_burnin_learning_rate)
+      post_burnin_learning_rate, name='learning_rate')
 
 
 def cosine_decay_with_warmup(global_step,
                              learning_rate_base,
                              total_steps,
                              warmup_learning_rate=0.0,
-                             warmup_steps=0):
+                             warmup_steps=0,
+                             hold_base_rate_steps=0):
   """Cosine decay schedule with warm up period.
 
   Cosine annealing learning rate as described in:
@@ -79,6 +80,8 @@ def cosine_decay_with_warmup(global_step,
     total_steps: total number of training steps.
     warmup_learning_rate: initial learning rate for warm up.
     warmup_steps: number of warmup steps.
+    hold_base_rate_steps: Optional number of steps to hold base learning rate
+      before decaying.
 
   Returns:
     a (scalar) float tensor representing learning rate.
@@ -93,21 +96,24 @@ def cosine_decay_with_warmup(global_step,
   if total_steps < warmup_steps:
     raise ValueError('total_steps must be larger or equal to '
                      'warmup_steps.')
-  learning_rate = 0.5 * learning_rate_base * (
-      1 + tf.cos(np.pi * (tf.cast(global_step, tf.float32) - warmup_steps
-                         ) / float(total_steps - warmup_steps)))
+  learning_rate = 0.5 * learning_rate_base * (1 + tf.cos(
+      np.pi *
+      (tf.cast(global_step, tf.float32) - warmup_steps - hold_base_rate_steps
+      ) / float(total_steps - warmup_steps - hold_base_rate_steps)))
+  if hold_base_rate_steps > 0:
+    learning_rate = tf.where(global_step > warmup_steps + hold_base_rate_steps,
+                             learning_rate, learning_rate_base)
   if warmup_steps > 0:
     slope = (learning_rate_base - warmup_learning_rate) / warmup_steps
-    pre_cosine_learning_rate = slope * tf.cast(
-        global_step, tf.float32) + warmup_learning_rate
-    learning_rate = tf.where(
-        tf.less(tf.cast(global_step, tf.int32), warmup_steps),
-        pre_cosine_learning_rate,
-        learning_rate)
-  return learning_rate
+    warmup_rate = slope * tf.cast(global_step,
+                                  tf.float32) + warmup_learning_rate
+    learning_rate = tf.where(global_step < warmup_steps, warmup_rate,
+                             learning_rate)
+  return tf.where(global_step > total_steps, 0.0, learning_rate,
+                  name='learning_rate')
 
 
-def manual_stepping(global_step, boundaries, rates):
+def manual_stepping(global_step, boundaries, rates, warmup=False):
   """Manually stepped learning rate schedule.
 
   This function provides fine grained control over learning rates.  One must
@@ -124,6 +130,8 @@ def manual_stepping(global_step, boundaries, rates):
     rates: a list of (float) learning rates corresponding to intervals between
       the boundaries.  The length of this list must be exactly
       len(boundaries) + 1.
+    warmup: Whether to linearly interpolate learning rate for steps in
+      [0, boundaries[0]].
 
   Returns:
     a (scalar) float tensor representing learning rate
@@ -131,6 +139,7 @@ def manual_stepping(global_step, boundaries, rates):
     ValueError: if one of the following checks fails:
       1. boundaries is a strictly increasing list of positive integers
       2. len(rates) == len(boundaries) + 1
+      3. boundaries[0] != 0
   """
   if any([b < 0 for b in boundaries]) or any(
       [not isinstance(b, int) for b in boundaries]):
@@ -142,16 +151,21 @@ def manual_stepping(global_step, boundaries, rates):
   if len(rates) != len(boundaries) + 1:
     raise ValueError('Number of provided learning rates must exceed '
                      'number of boundary points by exactly 1.')
-  if not boundaries: return tf.constant(rates[0])
-  step_boundaries = tf.constant(boundaries, tf.int32)
+
+  if boundaries and boundaries[0] == 0:
+    raise ValueError('First step cannot be zero.')
+
+  if warmup and boundaries:
+    slope = (rates[1] - rates[0]) * 1.0 / boundaries[0]
+    warmup_steps = range(boundaries[0])
+    warmup_rates = [rates[0] + slope * step for step in warmup_steps]
+    boundaries = warmup_steps + boundaries
+    rates = warmup_rates + rates[1:]
+  else:
+    boundaries = [0] + boundaries
   num_boundaries = len(boundaries)
-  learning_rates = tf.constant(rates, tf.float32)
-  index = tf.reduce_min(
-      tf.where(
-          # Casting global step to tf.int32 is dangerous, but necessary to be
-          # compatible with TPU.
-          tf.greater(step_boundaries, tf.cast(global_step, tf.int32)),
-          tf.constant(range(num_boundaries), dtype=tf.int32),
-          tf.constant([num_boundaries] * num_boundaries, dtype=tf.int32)))
-  return tf.reduce_sum(learning_rates * tf.one_hot(index, len(rates),
-                                                   dtype=tf.float32))
+  rate_index = tf.reduce_max(tf.where(tf.greater_equal(global_step, boundaries),
+                                      range(num_boundaries),
+                                      [0] * num_boundaries))
+  return tf.reduce_sum(rates * tf.one_hot(rate_index, depth=num_boundaries),
+                       name='learning_rate')

research/object_detection/utils/learning_schedules_test.py

@@ -33,6 +33,7 @@ class LearningSchedulesTest(test_case.TestCase):
       learning_rate = learning_schedules.exponential_decay_with_burnin(
           global_step, learning_rate_base, learning_rate_decay_steps,
           learning_rate_decay_factor, burnin_learning_rate, burnin_steps)
+      assert learning_rate.op.name.endswith('learning_rate')
       return (learning_rate,)
 
     output_rates = [
@@ -51,6 +52,7 @@ class LearningSchedulesTest(test_case.TestCase):
       learning_rate = learning_schedules.cosine_decay_with_warmup(
           global_step, learning_rate_base, total_steps,
           warmup_learning_rate, warmup_steps)
+      assert learning_rate.op.name.endswith('learning_rate')
       return (learning_rate,)
     exp_rates = [0.1, 0.5, 0.9, 1.0, 0]
     input_global_steps = [0, 4, 8, 9, 100]
@@ -60,12 +62,53 @@ class LearningSchedulesTest(test_case.TestCase):
     ]
     self.assertAllClose(output_rates, exp_rates)
 
+  def testCosineDecayAfterTotalSteps(self):
+    def graph_fn(global_step):
+      learning_rate_base = 1.0
+      total_steps = 100
+      warmup_learning_rate = 0.1
+      warmup_steps = 9
+      learning_rate = learning_schedules.cosine_decay_with_warmup(
+          global_step, learning_rate_base, total_steps,
+          warmup_learning_rate, warmup_steps)
+      assert learning_rate.op.name.endswith('learning_rate')
+      return (learning_rate,)
+    exp_rates = [0]
+    input_global_steps = [101]
+    output_rates = [
+        self.execute(graph_fn, [np.array(step).astype(np.int64)])
+        for step in input_global_steps
+    ]
+    self.assertAllClose(output_rates, exp_rates)
+
+  def testCosineDecayWithHoldBaseLearningRateSteps(self):
+    def graph_fn(global_step):
+      learning_rate_base = 1.0
+      total_steps = 120
+      warmup_learning_rate = 0.1
+      warmup_steps = 9
+      hold_base_rate_steps = 20
+      learning_rate = learning_schedules.cosine_decay_with_warmup(
+          global_step, learning_rate_base, total_steps,
+          warmup_learning_rate, warmup_steps, hold_base_rate_steps)
+      assert learning_rate.op.name.endswith('learning_rate')
+      return (learning_rate,)
+    exp_rates = [0.1, 0.5, 0.9, 1.0, 1.0, 1.0, 0.999702, 0.874255, 0.577365,
+                 0.0]
+    input_global_steps = [0, 4, 8, 9, 10, 29, 30, 50, 70, 120]
+    output_rates = [
+        self.execute(graph_fn, [np.array(step).astype(np.int64)])
+        for step in input_global_steps
+    ]
+    self.assertAllClose(output_rates, exp_rates)
+
   def testManualStepping(self):
     def graph_fn(global_step):
       boundaries = [2, 3, 7]
       rates = [1.0, 2.0, 3.0, 4.0]
       learning_rate = learning_schedules.manual_stepping(
           global_step, boundaries, rates)
+      assert learning_rate.op.name.endswith('learning_rate')
       return (learning_rate,)
 
     output_rates = [
@@ -75,6 +118,22 @@ class LearningSchedulesTest(test_case.TestCase):
     exp_rates = [1.0, 1.0, 2.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0]
     self.assertAllClose(output_rates, exp_rates)
 
+  def testManualSteppingWithWarmup(self):
+    def graph_fn(global_step):
+      boundaries = [4, 6, 8]
+      rates = [0.02, 0.10, 0.01, 0.001]
+      learning_rate = learning_schedules.manual_stepping(
+          global_step, boundaries, rates, warmup=True)
+      assert learning_rate.op.name.endswith('learning_rate')
+      return (learning_rate,)
+
+    output_rates = [
+        self.execute(graph_fn, [np.array(i).astype(np.int64)])
+        for i in range(9)
+    ]
+    exp_rates = [0.02, 0.04, 0.06, 0.08, 0.10, 0.10, 0.01, 0.01, 0.001]
+    self.assertAllClose(output_rates, exp_rates)
+
   def testManualSteppingWithZeroBoundaries(self):
     def graph_fn(global_step):
       boundaries = []

research/object_detection/utils/ops.py

@@ -657,7 +657,7 @@ def position_sensitive_crop_regions(image,
     position_sensitive_features = tf.add_n(image_crops) / len(image_crops)
     # Then average over spatial positions within the bins.
     position_sensitive_features = tf.reduce_mean(
-        position_sensitive_features, [1, 2], keepdims=True)
+        position_sensitive_features, [1, 2], keep_dims=True)
   else:
     # Reorder height/width to depth channel.
     block_size = bin_crop_size[0]

research/object_detection/utils/ops_test.py

@@ -840,7 +840,7 @@ class OpsTestPositionSensitiveCropRegions(tf.test.TestCase):
     # All channels are equal so position-sensitive crop and resize should
     # work as the usual crop and resize for just one channel.
     crop = tf.image.crop_and_resize(image, boxes, box_ind, crop_size)
-    crop_and_pool = tf.reduce_mean(crop, [1, 2], keepdims=True)
+    crop_and_pool = tf.reduce_mean(crop, [1, 2], keep_dims=True)
 
     ps_crop_and_pool = ops.position_sensitive_crop_regions(
         tiled_image,
@@ -866,7 +866,7 @@ class OpsTestPositionSensitiveCropRegions(tf.test.TestCase):
     # When a single bin is used, position-sensitive crop and pool should be
     # the same as non-position sensitive crop and pool.
     crop = tf.image.crop_and_resize(image, boxes, box_ind, crop_size)
-    crop_and_pool = tf.reduce_mean(crop, [1, 2], keepdims=True)
+    crop_and_pool = tf.reduce_mean(crop, [1, 2], keep_dims=True)
 
     ps_crop_and_pool = ops.position_sensitive_crop_regions(
         image, boxes, box_ind, crop_size, num_spatial_bins, global_pool=True)
@@ -1054,7 +1054,7 @@ class OpsTestPositionSensitiveCropRegions(tf.test.TestCase):
       ps_crop = ops.position_sensitive_crop_regions(
           image, boxes, box_ind, crop_size, num_spatial_bins, global_pool=False)
       ps_crop_and_pool = tf.reduce_mean(
-          ps_crop, reduction_indices=(1, 2), keepdims=True)
+          ps_crop, reduction_indices=(1, 2), keep_dims=True)
 
       with self.test_session() as sess:
         output = sess.run(ps_crop_and_pool)

research/object_detection/utils/variables_helper.py

@@ -42,7 +42,7 @@ def filter_variables(variables, filter_regex_list, invert=False):
     a list of filtered variables.
   """
   kept_vars = []
-  variables_to_ignore_patterns = list(filter(None, filter_regex_list))
+  variables_to_ignore_patterns = filter(None, filter_regex_list)
   for var in variables:
     add = True
     for pattern in variables_to_ignore_patterns: