Latest updates from DELF code. (#6651)

* Internal change.

PiperOrigin-RevId: 185564155

* Fix small bug when reading DELF features when file is empty.
A new test is added that catches this bug.

PiperOrigin-RevId: 213839503

* Refactors DELF example code to expose a function to create a DELF feature extractor.

PiperOrigin-RevId: 241492615

* Merged commit includes the following changes:
244073180  by Andre Araujo:

    Internal change

--
243646498  by Andre Araujo:

    Detect and save bounding boxes for a list of images.
    A new proto is added, along with auxiliary read/write functions, and tests.

--

PiperOrigin-RevId: 244073180

* Merged commit includes the following changes:
244872693  by Andre Araujo:

    Small change.

--
244871213  by Andre Araujo:

    Small edit to DELF extractor function.

--
244790715  by Andre Araujo:

    internal automated change

--
244773146  by Andre Araujo:

    Add option to visualize extracted boxes, and change slightly model input.

--
244288992  by Andre Araujo:

    Script to cluster DELF features using K-means.

--
244275164  by Andre Araujo:

    Scripts to extract DELF features and boxes from Revisited Oxford/Paris datasets, used in Detect-to-Retrieve paper.

--

PiperOrigin-RevId: 244872693

* Merged commit includes the following changes:

PiperOrigin-RevId: 244876167

* Updates instructions to include new released models, instructions for DELF and detector extraction, etc.

* Small fix

* Merged commit includes the following changes:
244909336  by Andre Araujo:

    Update to DELF config example to use latest model.

--

PiperOrigin-RevId: 244909336

* Revert "Merged commit includes the following changes:"

This reverts commit f41be9f159330f62846d5d8b3d3d5d55cd1874f4.

* Update delf_config_example model

* Small edits to detection instructions

* Specifying GLB location in README

research/delf/DETECTION.md

+## Quick start: landmark detection
+
+### Download Oxford buildings dataset
+
+To illustrate detector usage, please download the Oxford buildings dataset, by
+following the instructions
+[here](EXTRACTION_MATCHING.md#download-oxford-buildings-dataset). Then, create
+the file `list_images_detector.txt` as follows:
+
+```bash
+# From tensorflow/models/research/delf/delf/python/examples/
+echo data/oxford5k_images/all_souls_000002.jpg >> list_images_detector.txt
+echo data/oxford5k_images/all_souls_000035.jpg >> list_images_detector.txt
+```
+
+### Download detector model
+
+Also, you will need to download the pre-trained detector model:
+
+```bash
+# From tensorflow/models/research/delf/delf/python/examples/
+mkdir parameters && cd parameters
+wget http://storage.googleapis.com/delf/d2r_frcnn_20190411.tar.gz
+tar -xvzf d2r_frcnn_20190411.tar.gz
+```
+
+**Note**: this is the Faster-RCNN based model. We also release a MobileNet-SSD
+model, see the [README](README.md#pre-trained-models) for download link. The
+instructions should work seamlessly for both models.
+
+### Detecting landmarks
+
+Now that you have everything in place, running this command should detect boxes
+for the images `all_souls_000002.jpg` and `all_souls_000035.jpg`, with a
+threshold of 0.8, and produce visualizations.
+
+```bash
+# From tensorflow/models/research/delf/delf/python/examples/
+python extract_boxes.py \
+  --detector_path parameters/d2r_frcnn_20190411 \
+  --detector_thresh 0.8 \
+  --list_images_path list_images_detector.txt \
+  --output_dir data/oxford5k_boxes \
+  --output_viz_dir data/oxford5k_boxes_viz
+```
+
+Two images are generated in the `data/oxford5k_boxes_viz` directory, they should
+look similar to these ones:
+
+![DetectionExample1](delf/python/examples/detection_example_1.jpg)
+![DetectionExample2](delf/python/examples/detection_example_2.jpg)
+
+### Troubleshooting
+
+#### `matplotlib`
+
+`matplotlib` may complain with a message such as `no display name and no
+$DISPLAY environment variable`. To fix this, one option is add the line
+`backend : Agg` to the file `.config/matplotlib/matplotlibrc`. On this problem,
+see the discussion
+[here](https://stackoverflow.com/questions/37604289/tkinter-tclerror-no-display-name-and-no-display-environment-variable).

research/delf/EXTRACTION_MATCHING.md

 ## Quick start: DELF extraction and matching
 
+### Download Oxford buildings dataset
+
 To illustrate DELF usage, please download the Oxford buildings dataset. To
 follow these instructions closely, please download the dataset to the
 `tensorflow/models/research/delf/delf/python/examples` directory, as in the
@@ -16,13 +18,15 @@ echo data/oxford5k_images/hertford_000056.jpg >> list_images.txt
 echo data/oxford5k_images/oxford_000317.jpg >> list_images.txt
 ```
 
+### Download pre-trained DELF model
+
 Also, you will need to download the trained DELF model:
 
 ```bash
 # From tensorflow/models/research/delf/delf/python/examples/
 mkdir parameters && cd parameters
-wget http://download.tensorflow.org/models/delf_v1_20171026.tar.gz
-tar -xvzf delf_v1_20171026.tar.gz
+wget http://storage.googleapis.com/delf/delf_gld_20190411.tar.gz
+tar -xvzf delf_gld_20190411.tar.gz
 ```
 
 ### DELF feature extraction
@@ -54,7 +58,7 @@ python match_images.py \
 
 The image `matched_images.png` is generated and should look similar to this one:
 
-![MatchedImagesExample](delf/python/examples/matched_images_example.png)
+![MatchedImagesExample](delf/python/examples/matched_images_example.jpg)
 
 ### Troubleshooting
 

research/delf/README.md

 # DELF: DEep Local Features
 
 This project presents code for extracting DELF features, which were introduced
-with the paper ["Large-Scale Image Retrieval with Attentive Deep Local
-Features"](https://arxiv.org/abs/1612.06321). A simple application is also
-illustrated, where two images containing the same landmark can be matched to
-each other, to obtain local image correspondences.
+with the paper
+["Large-Scale Image Retrieval with Attentive Deep Local Features"](https://arxiv.org/abs/1612.06321).
+It also contains code for the follow-up paper
+["Detect-to-Retrieve: Efficient Regional Aggregation for Image Search"](https://arxiv.org/abs/1812.01584).
+
+We also released pre-trained models based on the
+[Google Landmarks dataset](https://www.kaggle.com/google/google-landmarks-dataset).
 
 DELF is particularly useful for large-scale instance-level image recognition. It
 detects and describes semantic local features which can be geometrically
-verified between images showing the same object instance. The pre-trained model
-released here has been optimized for landmark recognition, so expect it to work
+verified between images showing the same object instance. The pre-trained models
+released here have been optimized for landmark recognition, so expect it to work
 well in this area. We also provide tensorflow code for building the DELF model,
 which could then be used to train models for other types of objects.
 
-If you make use of this code, please consider citing:
+If you make use of this code, please consider citing the following papers:
 
 ```
 "Large-Scale Image Retrieval with Attentive Deep Local Features",
-Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, Bohyung Han,
+H. Noh, A. Araujo, J. Sim, T. Weyand and B. Han,
 Proc. ICCV'17
 ```
 
+and/or
+
+```
+"Detect-to-Retrieve: Efficient Regional Aggregation for Image Search",
+M. Teichmann*, A. Araujo*, M. Zhu and J. Sim,
+Proc. CVPR'19
+```
+
 ## News
 
--   DELF achieved state-of-the-art results in a CVPR'18 image retrieval paper:
-    [Radenovic et al., "Revisiting Oxford and Paris: Large-Scale Image Retrieval
-    Benchmarking"](https://arxiv.org/abs/1803.11285).
--   DELF was featured in
+-   [Apr'19] Check out our CVPR'19 paper:
+    ["Detect-to-Retrieve: Efficient Regional Aggregation for Image Search"](https://arxiv.org/abs/1812.01584)
+-   [Jun'18] DELF achieved state-of-the-art results in a CVPR'18 image retrieval
+    paper: [Radenovic et al., "Revisiting Oxford and Paris: Large-Scale Image
+    Retrieval Benchmarking"](https://arxiv.org/abs/1803.11285).
+-   [Apr'18] DELF was featured in
     [ModelDepot](https://modeldepot.io/mikeshi/delf/overview)
--   DELF is now available in
+-   [Mar'18] DELF is now available in
     [TF-Hub](https://www.tensorflow.org/hub/modules/google/delf/1)
 
 ## Dataset
 
-The Google-Landmarks dataset has been released as part of two Kaggle challenges:
-[Landmark Recognition](https://www.kaggle.com/c/landmark-recognition-challenge)
-and [Landmark Retrieval](https://www.kaggle.com/c/landmark-retrieval-challenge).
-If you make use of the dataset in your research, please consider citing the
-paper mentioned above.
+We have two Google-Landmarks dataset versions:
+
+-   Initial version (v1) can be found
+    [here](https://www.kaggle.com/google/google-landmarks-dataset). In includes
+    the Google Landmark Boxes which were described in the Detect-to-Retrieve
+    paper.
+-   Second version (v2) has been released as part of two Kaggle challenges:
+    [Landmark Recognition](https://www.kaggle.com/c/landmark-recognition-2019)
+    and [Landmark Retrieval](https://www.kaggle.com/c/landmark-retrieval-2019).
+    It can be downloaded from CVDF
+    [here](https://github.com/cvdfoundation/google-landmark).
+
+If you make use of these datasets in your research, please consider citing the
+papers mentioned above.
 
 ## Installation
 
-To be able to use this code, please follow [these
-instructions](INSTALL_INSTRUCTIONS.md) to properly install the DELF library.
+To be able to use this code, please follow
+[these instructions](INSTALL_INSTRUCTIONS.md) to properly install the DELF
+library.
+
+## Quick start
 
-## Quick start: DELF extraction and matching
+### Pre-trained models
+
+We release several pre-trained models. See instructions in the following
+sections for examples on how to use the models.
+
+**DELF pre-trained on the Google-Landmarks dataset v1**
+([link](http://storage.googleapis.com/delf/delf_gld_20190411.tar.gz)). Presented
+in the [CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
+Boosts performance by ~4% mAP compared to ICCV'17 DELF model.
+
+**DELF pre-trained on Landmarks-Clean/Landmarks-Full dataset**
+([link](http://storage.googleapis.com/delf/delf_v1_20171026.tar.gz)). Presented
+in the [ICCV'17 DELF paper](https://arxiv.org/abs/1612.06321), model was trained
+on the dataset released by the [DIR paper](https://arxiv.org/abs/1604.01325).
+
+**Faster-RCNN detector pre-trained on Google Landmark Boxes**
+([link](http://storage.googleapis.com/delf/d2r_frcnn_20190411.tar.gz)).
+Presented in the
+[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
+
+**MobileNet-SSD detector pre-trained on Google Landmark Boxes**
+([link](http://storage.googleapis.com/delf/d2r_mnetssd_20190411.tar.gz)).
+Presented in the
+[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
+
+### DELF extraction and matching
 
 Please follow [these instructions](EXTRACTION_MATCHING.md). At the end, you
 should obtain a nice figure showing local feature matches, as:
 
-![MatchedImagesExample](delf/python/examples/matched_images_example.png)
+![MatchedImagesExample](delf/python/examples/matched_images_example.jpg)
+
+### Landmark detection
+
+Please follow [these instructions](DETECTION.md). At the end, you should obtain
+a nice figure showing a detection, as:
+
+![DetectionExample1](delf/python/examples/detection_example_1.jpg)
 
 ## Code overview
 
-DELF's code is located under the `delf` directory. There are two directories
+DELF/D2R code is located under the `delf` directory. There are two directories
 therein, `protos` and `python`.
 
 ### `delf/protos`
 
-This directory contains three protobufs:
+This directory contains protobufs:
 
+-   `box.proto`: protobuf for serializing detected boxes.
 -   `datum.proto`: general-purpose protobuf for serializing float tensors.
--   `feature.proto`: protobuf for serializing DELF features.
 -   `delf_config.proto`: protobuf for configuring DELF extraction.
+-   `feature.proto`: protobuf for serializing DELF features.
 
 ### `delf/python`
 
 This directory contains files for several different purposes:
 
--   `datum_io.py`, `feature_io.py` are helper files for reading and writing
-    tensors and features.
+-   `box_io.py`, `datum_io.py`, `feature_io.py` are helper files for reading and
+    writing tensors and features.
 -   `delf_v1.py` contains the code to create DELF models.
 -   `feature_extractor.py` contains the code to extract features using DELF.
     This is particularly useful for extracting features over multiple scales,
@@ -80,13 +138,17 @@ Besides these, other files in this directory contain tests for different
 modules.
 
 The subdirectory `delf/python/examples` contains sample scripts to run DELF
-feature extraction and matching:
+feature extraction/matching, and object detection:
 
+-   `delf_config_example.pbtxt` shows an example instantiation of the DelfConfig
+    proto, used for DELF feature extraction.
+-   `extract_boxes.py` enables object detection from a list of images.
 -   `extract_features.py` enables DELF extraction from a list of images.
 -   `match_images.py` supports image matching using DELF features extracted
     using `extract_features.py`.
--   `delf_config_example.pbtxt` shows an example instantiation of the DelfConfig
-    proto, used for DELF feature extraction.
+
+The subdirectory `delf/python/detect_to_retrieve` contains sample scripts
+related to the Detect-to-Retrieve paper (work in progress).
 
 ## Maintainers
 
@@ -94,10 +156,20 @@ André Araujo (@andrefaraujo)
 
 ## Release history
 
-### October 26, 2017
+### April, 2019
+
+Detect-to-Retrieve code released (work in progress).
+
+Includes pre-trained models to detect landmark boxes, and DELF model pre-trained
+on Google Landmarks v1 dataset.
+
+**Thanks to contributors**: André Araujo, Marvin Teichmann, Menglong Zhu,
+Jack Sim.
+
+### October, 2017
 
 Initial release containing DELF-v1 code, including feature extraction and
-matching examples.
+matching examples. Pre-trained DELF model from ICCV'17 paper is released.
 
 **Thanks to contributors**: André Araujo, Hyeonwoo Noh, Youlong Cheng,
 Jack Sim.

research/delf/delf/__init__.py

@@ -12,18 +12,22 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
-
 """Module to extract deep local features."""
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
 # pylint: disable=unused-import
+from delf.protos import box_pb2
 from delf.protos import datum_pb2
 from delf.protos import delf_config_pb2
 from delf.protos import feature_pb2
+from delf.python import box_io
 from delf.python import datum_io
 from delf.python import delf_v1
+from delf.python import detect_to_retrieve
 from delf.python import feature_extractor
 from delf.python import feature_io
+from delf.python.examples import extract_boxes
+from delf.python.examples import extract_features
 # pylint: enable=unused-import

research/delf/delf/python/box_io.py

@@ -24,7 +24,7 @@ from __future__ import print_function
 import numpy as np
 import tensorflow as tf
 
-from google3.third_party.tensorflow_models.delf.protos import box_pb2
+from delf import box_pb2
 
 
 def ArraysToBoxes(boxes, scores, class_indices):

research/delf/delf/python/detect_to_retrieve/__init__.py

+# Copyright 2019 The TensorFlow Authors All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Module for Detect-to-Retrieve technique."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+# pylint: disable=unused-import
+from delf.python.detect_to_retrieve import dataset
+# pylint: enable=unused-import

research/delf/delf/python/detect_to_retrieve/cluster_delf_features.py

+# Copyright 2019 The TensorFlow Authors All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Clusters DELF features using the K-means algorithm.
+
+All DELF local feature descriptors for a given dataset's index images are loaded
+as the input.
+
+Note that:
+-  we only use features extracted from whole images (no features from boxes are
+   used).
+-  the codebook should be trained on Paris images for Oxford retrieval
+   experiments, and vice-versa.
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import os
+import sys
+import time
+
+import numpy as np
+import tensorflow as tf
+
+from tensorflow.python.platform import app
+from delf import feature_io
+from delf.detect_to_retrieve import dataset
+
+cmd_args = None
+
+# Extensions.
+_DELF_EXTENSION = '.delf'
+
+# Default DELF dimensionality.
+_DELF_DIM = 128
+
+# Pace to report log when collecting features.
+_STATUS_CHECK_ITERATIONS = 100
+
+
+class _IteratorInitHook(tf.train.SessionRunHook):
+  """Hook to initialize data iterator after session is created."""
+
+  def __init__(self):
+    super(_IteratorInitHook, self).__init__()
+    self.iterator_initializer_fn = None
+
+  def after_create_session(self, session, coord):
+    """Initialize the iterator after the session has been created."""
+    del coord
+    self.iterator_initializer_fn(session)
+
+
+def main(argv):
+  if len(argv) > 1:
+    raise RuntimeError('Too many command-line arguments.')
+
+  # Process output directory.
+  if os.path.exists(cmd_args.output_cluster_dir):
+    raise RuntimeError(
+        'output_cluster_dir = %s already exists. This may indicate that a '
+        'previous run already wrote checkpoints in this directory, which would '
+        'lead to incorrect training. Please re-run this script by specifying an'
+        ' inexisting directory.' % cmd_args.output_cluster_dir)
+  else:
+    os.makedirs(cmd_args.output_cluster_dir)
+
+  # Read list of index images from dataset file.
+  print('Reading list of index images from dataset file...')
+  _, index_list, _ = dataset.ReadDatasetFile(cmd_args.dataset_file_path)
+  num_images = len(index_list)
+  print('done! Found %d images' % num_images)
+
+  # Loop over list of index images and collect DELF features.
+  features_for_clustering = []
+  start = time.clock()
+  print('Starting to collect features from index images...')
+  for i in range(num_images):
+    if i > 0 and i % _STATUS_CHECK_ITERATIONS == 0:
+      elapsed = (time.clock() - start)
+      print('Processing index image %d out of %d, last %d '
+            'images took %f seconds' %
+            (i, num_images, _STATUS_CHECK_ITERATIONS, elapsed))
+      start = time.clock()
+
+    features_filename = index_list[i] + _DELF_EXTENSION
+    features_fullpath = os.path.join(cmd_args.features_dir, features_filename)
+    _, _, features, _, _ = feature_io.ReadFromFile(features_fullpath)
+    if features.size != 0:
+      assert features.shape[1] == _DELF_DIM
+    for feature in features:
+      features_for_clustering.append(feature)
+
+  features_for_clustering = np.array(features_for_clustering, dtype=np.float32)
+  print('All features were loaded! There are %d features, each with %d '
+        'dimensions' %
+        (features_for_clustering.shape[0], features_for_clustering.shape[1]))
+
+  # Run K-means clustering.
+  def _get_input_fn():
+    """Helper function to create input function and hook for training.
+
+    Returns:
+      input_fn: Input function for k-means Estimator training.
+      init_hook: Hook used to load data during training.
+    """
+    init_hook = _IteratorInitHook()
+
+    def _input_fn():
+      """Produces tf.data.Dataset object for k-means training.
+
+      Returns:
+        Tensor with the data for training.
+      """
+      features_placeholder = tf.placeholder(tf.float32,
+                                            features_for_clustering.shape)
+      delf_dataset = tf.data.Dataset.from_tensor_slices((features_placeholder))
+      delf_dataset = delf_dataset.shuffle(1000).batch(
+          features_for_clustering.shape[0])
+      iterator = delf_dataset.make_initializable_iterator()
+
+      def _initializer_fn(sess):
+        """Initialize dataset iterator, feed in the data."""
+        sess.run(
+            iterator.initializer,
+            feed_dict={features_placeholder: features_for_clustering})
+
+      init_hook.iterator_initializer_fn = _initializer_fn
+      return iterator.get_next()
+
+    return _input_fn, init_hook
+
+  input_fn, init_hook = _get_input_fn()
+
+  kmeans = tf.estimator.experimental.KMeans(
+      num_clusters=cmd_args.num_clusters,
+      model_dir=cmd_args.output_cluster_dir,
+      use_mini_batch=False,
+  )
+
+  print('Starting K-means clustering...')
+  start = time.clock()
+  for i in range(cmd_args.num_iterations):
+    kmeans.train(input_fn, hooks=[init_hook])
+    average_sum_squared_error = kmeans.evaluate(
+        input_fn, hooks=[init_hook])['score'] / features_for_clustering.shape[0]
+    elapsed = (time.clock() - start)
+    print('K-means iteration %d (out of %d) took %f seconds, '
+          'average-sum-of-squares: %f' %
+          (i, cmd_args.num_iterations, elapsed, average_sum_squared_error))
+    start = time.clock()
+
+  print('K-means clustering finished!')
+
+
+if __name__ == '__main__':
+  parser = argparse.ArgumentParser()
+  parser.register('type', 'bool', lambda v: v.lower() == 'true')
+  parser.add_argument(
+      '--dataset_file_path',
+      type=str,
+      default='/tmp/gnd_roxford5k.mat',
+      help="""
+      Dataset file for Revisited Oxford or Paris dataset, in .mat format. The
+      list of index images loaded from this file is used to collect local
+      features, which are assumed to be in <image_name>.delf file format.
+      """)
+  parser.add_argument(
+      '--features_dir',
+      type=str,
+      default='/tmp/features',
+      help="""
+      Directory where DELF feature files are to be found.
+      """)
+  parser.add_argument(
+      '--num_clusters',
+      type=int,
+      default=1024,
+      help="""
+      Number of clusters to use.
+      """)
+  parser.add_argument(
+      '--num_iterations',
+      type=int,
+      default=50,
+      help="""
+      Number of iterations to use.
+      """)
+  parser.add_argument(
+      '--output_cluster_dir',
+      type=str,
+      default='/tmp/cluster',
+      help="""
+      Directory where clustering outputs are written to. This directory should
+      not exist before running this script; it will be created during
+      clustering.
+      """)
+  cmd_args, unparsed = parser.parse_known_args()
+  app.run(main=main, argv=[sys.argv[0]] + unparsed)

research/delf/delf/python/detect_to_retrieve/dataset.py

+# Copyright 2017 The TensorFlow Authors All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Python interface for Revisited Oxford/Paris dataset."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+from scipy.io import matlab
+import tensorflow as tf
+
+_GROUND_TRUTH_KEYS = ['easy', 'hard', 'junk', 'ok']
+
+
+def ReadDatasetFile(dataset_file_path):
+  """Reads dataset file in Revisited Oxford/Paris ".mat" format.
+
+  Args:
+    dataset_file_path: Path to dataset file, in .mat format.
+
+  Returns:
+    query_list: List of query image names.
+    index_list: List of index image names.
+    ground_truth: List containing ground-truth information for dataset. Each
+      entry is a dict corresponding to the ground-truth information for a query.
+      The dict may have keys 'easy', 'hard', 'junk' or 'ok', mapping to a list
+      of integers; additionally, it has a key 'bbx' mapping to a list of floats
+      with bounding box coordinates.
+  """
+  with tf.gfile.GFile(dataset_file_path, 'r') as f:
+    cfg = matlab.loadmat(f)
+
+  # Parse outputs according to the specificities of the dataset file.
+  query_list = [str(im_array[0]) for im_array in np.squeeze(cfg['qimlist'])]
+  index_list = [str(im_array[0]) for im_array in np.squeeze(cfg['imlist'])]
+  ground_truth_raw = np.squeeze(cfg['gnd'])
+  ground_truth = []
+  for query_ground_truth_raw in ground_truth_raw:
+    query_ground_truth = {}
+    for ground_truth_key in _GROUND_TRUTH_KEYS:
+      if ground_truth_key in query_ground_truth_raw.dtype.names:
+        adjusted_labels = query_ground_truth_raw[ground_truth_key] - 1
+        query_ground_truth[ground_truth_key] = adjusted_labels.flatten()
+
+    query_ground_truth['bbx'] = np.squeeze(query_ground_truth_raw['bbx'])
+    ground_truth.append(query_ground_truth)
+
+  return query_list, index_list, ground_truth

research/delf/delf/python/detect_to_retrieve/delf_gld_config.pbtxt

+model_path: "parameters/delf_gld_20190411/model"
+image_scales: .25
+image_scales: .3536
+image_scales: .5
+image_scales: .7071
+image_scales: 1.0
+image_scales: 1.4142
+image_scales: 2.0
+delf_local_config {
+  use_pca: true
+  # Note that for the exported model provided as an example, layer_name and
+  # iou_threshold are hard-coded in the checkpoint. So, the layer_name and
+  # iou_threshold variables here have no effect on the provided
+  # extract_features.py script.
+  layer_name: "resnet_v1_50/block3"
+  iou_threshold: 1.0
+  max_feature_num: 1000
+  score_threshold: 100.0
+  pca_parameters {
+    mean_path: "parameters/delf_gld_20190411/pca/mean.datum"
+    projection_matrix_path: "parameters/delf_gld_20190411/pca/pca_proj_mat.datum"
+    pca_dim: 128
+    use_whitening: false
+  }
+}

research/delf/delf/python/detect_to_retrieve/extract_index_boxes_and_features.py

+# Copyright 2017 The TensorFlow Authors All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Extracts DELF and boxes from the Revisited Oxford/Paris index datasets.
+
+Boxes are saved to <image_name>.boxes files. DELF features are extracted for the
+entire image and saved into <image_name>.delf files. In addition, DELF features
+are extracted for each high-confidence bounding box in the image, and saved into
+files named <image_name>_0.delf, <image_name>_1.delf, etc.
+
+The program checks if descriptors/boxes already exist, and skips computation for
+those.
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import csv
+import math
+import os
+import sys
+import time
+
+import numpy as np
+from PIL import Image
+from PIL import ImageFile
+import tensorflow as tf
+
+from google.protobuf import text_format
+from tensorflow.python.platform import app
+from delf import delf_config_pb2
+from delf import box_io
+from delf import feature_io
+from delf.detect_to_retrieve import dataset
+from delf import extract_boxes
+from delf import extract_features
+
+cmd_args = None
+
+# Extension of feature files.
+_BOX_EXTENSION = '.boxes'
+_DELF_EXTENSION = '.delf'
+_IMAGE_EXTENSION = '.jpg'
+
+# Pace to report extraction log.
+_STATUS_CHECK_ITERATIONS = 100
+
+# To avoid crashing for truncated (corrupted) images.
+ImageFile.LOAD_TRUNCATED_IMAGES = True
+
+
+def _PilLoader(path):
+  """Helper function to read image with PIL.
+
+  Args:
+    path: Path to image to be loaded.
+
+  Returns:
+    PIL image in RGB format.
+  """
+  with tf.gfile.GFile(path, 'rb') as f:
+    img = Image.open(f)
+    return img.convert('RGB')
+
+
+def _WriteMappingBasenameToIds(index_names_ids_and_boxes, output_path):
+  """Helper function to write CSV mapping from DELF file name to IDs.
+
+  Args:
+    index_names_ids_and_boxes: List containing 3-element lists with name, image
+      ID and box ID.
+    output_path: Output CSV path.
+  """
+  with tf.gfile.GFile(output_path, 'w') as f:
+    csv_writer = csv.DictWriter(
+        f, fieldnames=['name', 'index_image_id', 'box_id'])
+    csv_writer.writeheader()
+    for name_imid_boxid in index_names_ids_and_boxes:
+      csv_writer.writerow({
+          'name': name_imid_boxid[0],
+          'index_image_id': name_imid_boxid[1],
+          'box_id': name_imid_boxid[2],
+      })
+
+
+def main(argv):
+  if len(argv) > 1:
+    raise RuntimeError('Too many command-line arguments.')
+
+  tf.logging.set_verbosity(tf.logging.INFO)
+
+  # Read list of index images from dataset file.
+  tf.logging.info('Reading list of index images from dataset file...')
+  _, index_list, _ = dataset.ReadDatasetFile(cmd_args.dataset_file_path)
+  num_images = len(index_list)
+  tf.logging.info('done! Found %d images', num_images)
+
+  # Parse DelfConfig proto.
+  config = delf_config_pb2.DelfConfig()
+  with tf.gfile.GFile(cmd_args.delf_config_path, 'r') as f:
+    text_format.Merge(f.read(), config)
+
+  # Create output directories if necessary.
+  if not os.path.exists(cmd_args.output_features_dir):
+    os.makedirs(cmd_args.output_features_dir)
+  if not os.path.exists(cmd_args.output_boxes_dir):
+    os.makedirs(cmd_args.output_boxes_dir)
+
+  index_names_ids_and_boxes = []
+  with tf.Graph().as_default():
+    with tf.Session() as sess:
+      # Initialize variables, construct detector and DELF extractor.
+      init_op = tf.global_variables_initializer()
+      sess.run(init_op)
+      detector_fn = extract_boxes.MakeDetector(
+          sess, cmd_args.detector_model_dir, import_scope='detector')
+      delf_extractor_fn = extract_features.MakeExtractor(
+          sess, config, import_scope='extractor_delf')
+
+      start = time.clock()
+      for i in range(num_images):
+        if i == 0:
+          print('Starting to extract features/boxes from index images...')
+        elif i % _STATUS_CHECK_ITERATIONS == 0:
+          elapsed = (time.clock() - start)
+          print('Processing index image %d out of %d, last %d '
+                'images took %f seconds' %
+                (i, num_images, _STATUS_CHECK_ITERATIONS, elapsed))
+          start = time.clock()
+
+        index_image_name = index_list[i]
+        input_image_filename = os.path.join(cmd_args.images_dir,
+                                            index_image_name + _IMAGE_EXTENSION)
+        output_feature_filename_whole_image = os.path.join(
+            cmd_args.output_features_dir, index_image_name + _DELF_EXTENSION)
+        output_box_filename = os.path.join(cmd_args.output_boxes_dir,
+                                           index_image_name + _BOX_EXTENSION)
+
+        pil_im = _PilLoader(input_image_filename)
+        width, height = pil_im.size
+
+        # Extract and save boxes.
+        if tf.gfile.Exists(output_box_filename):
+          tf.logging.info('Skipping box computation for %s', index_image_name)
+          (boxes_out, scores_out,
+           class_indices_out) = box_io.ReadFromFile(output_box_filename)
+        else:
+          (boxes_out, scores_out,
+           class_indices_out) = detector_fn(np.expand_dims(pil_im, 0))
+          # Using only one image per batch.
+          boxes_out = boxes_out[0]
+          scores_out = scores_out[0]
+          class_indices_out = class_indices_out[0]
+          box_io.WriteToFile(output_box_filename, boxes_out, scores_out,
+                             class_indices_out)
+
+        # Select boxes with scores greater than threshold. Those will be the
+        # ones with extracted DELF features (besides the whole image, whose DELF
+        # features are extracted in all cases).
+        num_delf_files = 1
+        selected_boxes = []
+        for box_ind, box in enumerate(boxes_out):
+          if scores_out[box_ind] >= cmd_args.detector_thresh:
+            selected_boxes.append(box)
+        num_delf_files += len(selected_boxes)
+
+        # Extract and save DELF features.
+        for delf_file_ind in range(num_delf_files):
+          if delf_file_ind == 0:
+            index_box_name = index_image_name
+            output_feature_filename = output_feature_filename_whole_image
+          else:
+            index_box_name = index_image_name + '_' + str(delf_file_ind - 1)
+            output_feature_filename = os.path.join(
+                cmd_args.output_features_dir, index_box_name + _DELF_EXTENSION)
+
+          index_names_ids_and_boxes.append(
+              [index_box_name, i, delf_file_ind - 1])
+
+          if tf.gfile.Exists(output_feature_filename):
+            tf.logging.info('Skipping DELF computation for %s', index_box_name)
+            continue
+
+          if delf_file_ind >= 1:
+            bbox_for_cropping = selected_boxes[delf_file_ind - 1]
+            bbox_for_cropping_pil_convention = [
+                int(math.floor(bbox_for_cropping[1] * width)),
+                int(math.floor(bbox_for_cropping[0] * height)),
+                int(math.ceil(bbox_for_cropping[3] * width)),
+                int(math.ceil(bbox_for_cropping[2] * height))
+            ]
+            pil_cropped_im = pil_im.crop(bbox_for_cropping_pil_convention)
+            im = np.array(pil_cropped_im)
+          else:
+            im = np.array(pil_im)
+
+          (locations_out, descriptors_out, feature_scales_out,
+           attention_out) = delf_extractor_fn(im)
+
+          feature_io.WriteToFile(output_feature_filename, locations_out,
+                                 feature_scales_out, descriptors_out,
+                                 attention_out)
+
+  # Save mapping from output DELF name to index image id and box id.
+  _WriteMappingBasenameToIds(index_names_ids_and_boxes,
+                             cmd_args.output_index_mapping)
+
+
+if __name__ == '__main__':
+  parser = argparse.ArgumentParser()
+  parser.register('type', 'bool', lambda v: v.lower() == 'true')
+  parser.add_argument(
+      '--delf_config_path',
+      type=str,
+      default='/tmp/delf_config_example.pbtxt',
+      help="""
+      Path to DelfConfig proto text file with configuration to be used for DELF
+      extraction.
+      """)
+  parser.add_argument(
+      '--detector_model_dir',
+      type=str,
+      default='/tmp/detector_model',
+      help="""
+      Directory where detector SavedModel is located.
+      """)
+  parser.add_argument(
+      '--detector_thresh',
+      type=float,
+      default=0.1,
+      help="""
+      Threshold used to decide if an image's detected box undergoes feature
+      extraction. For all detected boxes with detection score larger than this,
+      a .delf file is saved containing the box features. Note that this
+      threshold is used only to select which boxes are used in feature
+      extraction; all detected boxes are actually saved in the .boxes file, even
+      those with score lower than detector_thresh.
+      """)
+  parser.add_argument(
+      '--dataset_file_path',
+      type=str,
+      default='/tmp/gnd_roxford5k.mat',
+      help="""
+      Dataset file for Revisited Oxford or Paris dataset, in .mat format.
+      """)
+  parser.add_argument(
+      '--images_dir',
+      type=str,
+      default='/tmp/images',
+      help="""
+      Directory where dataset images are located, all in .jpg format.
+      """)
+  parser.add_argument(
+      '--output_boxes_dir',
+      type=str,
+      default='/tmp/boxes',
+      help="""
+      Directory where detected boxes will be written to. Each image's boxes
+      will be written to a file with same name, and extension replaced by
+      .boxes.
+      """)
+  parser.add_argument(
+      '--output_features_dir',
+      type=str,
+      default='/tmp/features',
+      help="""
+      Directory where DELF features will be written to. Each image's features
+      will be written to a file with same name, and extension replaced by .delf,
+      eg: <image_name>.delf. In addition, DELF features are extracted for each
+      high-confidence bounding box in the image, and saved into files named
+      <image_name>_0.delf, <image_name>_1.delf, etc.
+      """)
+  parser.add_argument(
+      '--output_index_mapping',
+      type=str,
+      default='/tmp/index_mapping.csv',
+      help="""
+      CSV file which maps each .delf file name to the index image ID and
+      detected box ID. The format is 'name,index_image_id,box_id', including a
+      header. The 'name' refers to the .delf file name without extension.
+
+      For example, a few lines may be like:
+        'radcliffe_camera_000158,2,-1'
+        'radcliffe_camera_000158_0,2,0'
+        'radcliffe_camera_000158_1,2,1'
+        'radcliffe_camera_000158_2,2,2'
+      """)
+  cmd_args, unparsed = parser.parse_known_args()
+  app.run(main=main, argv=[sys.argv[0]] + unparsed)

research/delf/delf/python/detect_to_retrieve/extract_query_features.py

+# Copyright 2017 The TensorFlow Authors All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Extracts DELF features for query images from Revisited Oxford/Paris datasets.
+
+Note that query images are cropped before feature extraction, as required by the
+evaluation protocols of these datasets.
+
+The program checks if descriptors already exist, and skips computation for
+those.
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import os
+import sys
+import time
+
+import numpy as np
+from PIL import Image
+from PIL import ImageFile
+import tensorflow as tf
+
+from google.protobuf import text_format
+from tensorflow.python.platform import app
+from delf import delf_config_pb2
+from delf import feature_io
+from delf.detect_to_retrieve import dataset
+from delf import extract_features
+
+cmd_args = None
+
+# Extensions.
+_DELF_EXTENSION = '.delf'
+_IMAGE_EXTENSION = '.jpg'
+
+# To avoid PIL crashing for truncated (corrupted) images.
+ImageFile.LOAD_TRUNCATED_IMAGES = True
+
+
+def _PilLoader(path):
+  """Helper function to read image with PIL.
+
+  Args:
+    path: Path to image to be loaded.
+
+  Returns:
+    PIL image in RGB format.
+  """
+  with tf.gfile.GFile(path, 'rb') as f:
+    img = Image.open(f)
+    return img.convert('RGB')
+
+
+def main(argv):
+  if len(argv) > 1:
+    raise RuntimeError('Too many command-line arguments.')
+
+  tf.logging.set_verbosity(tf.logging.INFO)
+
+  # Read list of query images from dataset file.
+  tf.logging.info('Reading list of query images and boxes from dataset file...')
+  query_list, _, ground_truth = dataset.ReadDatasetFile(
+      cmd_args.dataset_file_path)
+  num_images = len(query_list)
+  tf.logging.info('done! Found %d images', num_images)
+
+  # Parse DelfConfig proto.
+  config = delf_config_pb2.DelfConfig()
+  with tf.gfile.GFile(cmd_args.delf_config_path, 'r') as f:
+    text_format.Merge(f.read(), config)
+
+  # Create output directory if necessary.
+  if not os.path.exists(cmd_args.output_features_dir):
+    os.makedirs(cmd_args.output_features_dir)
+
+  with tf.Graph().as_default():
+    with tf.Session() as sess:
+      # Initialize variables, construct DELF extractor.
+      init_op = tf.global_variables_initializer()
+      sess.run(init_op)
+      extractor_fn = extract_features.MakeExtractor(sess, config)
+
+      start = time.clock()
+      for i in range(num_images):
+        query_image_name = query_list[i]
+        input_image_filename = os.path.join(cmd_args.images_dir,
+                                            query_image_name + _IMAGE_EXTENSION)
+        output_feature_filename = os.path.join(
+            cmd_args.output_features_dir, query_image_name + _DELF_EXTENSION)
+        if tf.gfile.Exists(output_feature_filename):
+          tf.logging.info('Skipping %s', query_image_name)
+          continue
+
+        # Crop query image according to bounding box.
+        bbox = [int(round(b)) for b in ground_truth[i]['bbx']]
+        im = np.array(_PilLoader(input_image_filename).crop(bbox))
+
+        # Extract and save features.
+        (locations_out, descriptors_out, feature_scales_out,
+         attention_out) = extractor_fn(im)
+
+        feature_io.WriteToFile(output_feature_filename, locations_out,
+                               feature_scales_out, descriptors_out,
+                               attention_out)
+
+      elapsed = (time.clock() - start)
+      print('Processed %d query images in %f seconds' % (num_images, elapsed))
+
+
+if __name__ == '__main__':
+  parser = argparse.ArgumentParser()
+  parser.register('type', 'bool', lambda v: v.lower() == 'true')
+  parser.add_argument(
+      '--delf_config_path',
+      type=str,
+      default='/tmp/delf_config_example.pbtxt',
+      help="""
+      Path to DelfConfig proto text file with configuration to be used for DELF
+      extraction.
+      """)
+  parser.add_argument(
+      '--dataset_file_path',
+      type=str,
+      default='/tmp/gnd_roxford5k.mat',
+      help="""
+      Dataset file for Revisited Oxford or Paris dataset, in .mat format.
+      """)
+  parser.add_argument(
+      '--images_dir',
+      type=str,
+      default='/tmp/images',
+      help="""
+      Directory where dataset images are located, all in .jpg format.
+      """)
+  parser.add_argument(
+      '--output_features_dir',
+      type=str,
+      default='/tmp/features',
+      help="""
+      Directory where DELF features will be written to. Each image's features
+      will be written to a file with same name, and extension replaced by .delf.
+      """)
+  cmd_args, unparsed = parser.parse_known_args()
+  app.run(main=main, argv=[sys.argv[0]] + unparsed)

research/delf/delf/python/examples/delf_config_example.pbtxt

-model_path: "parameters/delf_v1_20171026/model/"
+model_path: "parameters/delf_gld_20190411/model/"
 image_scales: .25
 image_scales: .3536
 image_scales: .5
@@ -17,8 +17,8 @@ delf_local_config {
   max_feature_num: 1000
   score_threshold: 100.0
   pca_parameters {
-    mean_path: "parameters/delf_v1_20171026/pca/mean.datum"
-    projection_matrix_path: "parameters/delf_v1_20171026/pca/pca_proj_mat.datum"
+    mean_path: "parameters/delf_gld_20190411/pca/mean.datum"
+    projection_matrix_path: "parameters/delf_gld_20190411/pca/pca_proj_mat.datum"
     pca_dim: 40
     use_whitening: false
   }

research/delf/delf/python/examples/extract_boxes.py

@@ -27,6 +27,9 @@ import os
 import sys
 import time
 
+import matplotlib.patches as patches
+import matplotlib.pyplot as plt
+import numpy as np
 import tensorflow as tf
 
 from tensorflow.python.platform import app
@@ -34,8 +37,12 @@ from delf import box_io
 
 cmd_args = None
 
-# Extension of feature files.
+# Extension/suffix of produced files.
 _BOX_EXT = '.boxes'
+_VIZ_SUFFIX = '_viz.jpg'
+
+# Used for plotting boxes.
+_BOX_EDGE_COLORS = ['r', 'y', 'b', 'm', 'k', 'g', 'c', 'w']
 
 # Pace to report extraction log.
 _STATUS_CHECK_ITERATIONS = 100
@@ -56,45 +63,109 @@ def _ReadImageList(list_path):
   return image_paths
 
 
-def _MakeDetector(sess, model_dir):
+def MakeDetector(sess, model_dir, import_scope=None):
   """Creates a function to detect objects in an image.
 
   Args:
     sess: TensorFlow session to use.
     model_dir: Directory where SavedModel is located.
+    import_scope: Optional scope to use for model.
 
   Returns:
     Function that receives an image and returns detection results.
   """
-  tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING],
-                             model_dir)
-  input_images = sess.graph.get_tensor_by_name('input_images:0')
-  input_detection_thresh = sess.graph.get_tensor_by_name(
-      'input_detection_thresh:0')
-  boxes = sess.graph.get_tensor_by_name('detection_boxes:0')
-  scores = sess.graph.get_tensor_by_name('detection_scores:0')
-  class_indices = sess.graph.get_tensor_by_name('detection_classes:0')
-
-  def DetectorFn(images, threshold):
+  tf.saved_model.loader.load(
+      sess, [tf.saved_model.tag_constants.SERVING],
+      model_dir,
+      import_scope=import_scope)
+  import_scope_prefix = import_scope + '/' if import_scope is not None else ''
+  input_images = sess.graph.get_tensor_by_name('%sinput_images:0' %
+                                               import_scope_prefix)
+  boxes = sess.graph.get_tensor_by_name('%sdetection_boxes:0' %
+                                        import_scope_prefix)
+  scores = sess.graph.get_tensor_by_name('%sdetection_scores:0' %
+                                         import_scope_prefix)
+  class_indices = sess.graph.get_tensor_by_name('%sdetection_classes:0' %
+                                                import_scope_prefix)
+
+  def DetectorFn(images):
     """Receives an image and returns detected boxes.
 
     Args:
       images: Uint8 array with shape (batch, height, width 3) containing a batch
         of RGB images.
-      threshold: Detector threshold (float).
 
     Returns:
       Tuple (boxes, scores, class_indices).
     """
     return sess.run([boxes, scores, class_indices],
-                    feed_dict={
-                        input_images: images,
-                        input_detection_thresh: threshold,
-                    })
+                    feed_dict={input_images: images})
 
   return DetectorFn
 
 
+def _FilterBoxesByScore(boxes, scores, class_indices, score_threshold):
+  """Filter boxes based on detection scores.
+
+  Boxes with detection score >= score_threshold are returned.
+
+  Args:
+    boxes: [N, 4] float array denoting bounding box coordinates, in format [top,
+      left, bottom, right].
+    scores: [N] float array with detection scores.
+    class_indices: [N] int array with class indices.
+    score_threshold: Float detection score threshold to use.
+
+  Returns:
+    selected_boxes: selected `boxes`.
+    selected_scores: selected `scores`.
+    selected_class_indices: selected `class_indices`.
+  """
+  selected_boxes = []
+  selected_scores = []
+  selected_class_indices = []
+  for i, box in enumerate(boxes):
+    if scores[i] >= score_threshold:
+      selected_boxes.append(box)
+      selected_scores.append(scores[i])
+      selected_class_indices.append(class_indices[i])
+
+  return np.array(selected_boxes), np.array(selected_scores), np.array(
+      selected_class_indices)
+
+
+def _PlotBoxesAndSaveImage(image, boxes, output_path):
+  """Plot boxes on image and save to output path.
+
+  Args:
+    image: Numpy array containing image.
+    boxes: [N, 4] float array denoting bounding box coordinates, in format [top,
+      left, bottom, right].
+    output_path: String containing output path.
+  """
+  height = image.shape[0]
+  width = image.shape[1]
+
+  fig, ax = plt.subplots(1)
+  ax.imshow(image)
+  for i, box in enumerate(boxes):
+    scaled_box = [
+        box[0] * height, box[1] * width, box[2] * height, box[3] * width
+    ]
+    rect = patches.Rectangle([scaled_box[1], scaled_box[0]],
+                             scaled_box[3] - scaled_box[1],
+                             scaled_box[2] - scaled_box[0],
+                             linewidth=3,
+                             edgecolor=_BOX_EDGE_COLORS[i %
+                                                        len(_BOX_EDGE_COLORS)],
+                             facecolor='none')
+    ax.add_patch(rect)
+
+  ax.axis('off')
+  plt.savefig(output_path, bbox_inches='tight')
+  plt.close(fig)
+
+
 def main(argv):
   if len(argv) > 1:
     raise RuntimeError('Too many command-line arguments.')
@@ -107,9 +178,11 @@ def main(argv):
   num_images = len(image_paths)
   tf.logging.info('done! Found %d images', num_images)
 
-  # Create output directory if necessary.
+  # Create output directories if necessary.
   if not os.path.exists(cmd_args.output_dir):
     os.makedirs(cmd_args.output_dir)
+  if cmd_args.output_viz_dir and not os.path.exists(cmd_args.output_viz_dir):
+    os.makedirs(cmd_args.output_viz_dir)
 
   # Tell TensorFlow that the model will be built into the default Graph.
   with tf.Graph().as_default():
@@ -124,7 +197,7 @@ def main(argv):
       init_op = tf.global_variables_initializer()
       sess.run(init_op)
 
-      detector_fn = _MakeDetector(sess, cmd_args.detector_path)
+      detector_fn = MakeDetector(sess, cmd_args.detector_path)
 
       # Start input enqueue threads.
       coord = tf.train.Coordinator()
@@ -154,12 +227,21 @@ def main(argv):
           tf.logging.info('Skipping %s', image_path)
           continue
 
-        # Extract and save features.
-        (boxes_out, scores_out,
-         class_indices_out) = detector_fn(im, cmd_args.detector_thresh)
-
-        box_io.WriteToFile(out_boxes_fullpath, boxes_out[0], scores_out[0],
-                           class_indices_out[0])
+        # Extract and save boxes.
+        (boxes_out, scores_out, class_indices_out) = detector_fn(im)
+        (selected_boxes, selected_scores,
+         selected_class_indices) = _FilterBoxesByScore(boxes_out[0],
+                                                       scores_out[0],
+                                                       class_indices_out[0],
+                                                       cmd_args.detector_thresh)
+
+        box_io.WriteToFile(out_boxes_fullpath, selected_boxes, selected_scores,
+                           selected_class_indices)
+        if cmd_args.output_viz_dir:
+          out_viz_filename = base_boxes_filename + _VIZ_SUFFIX
+          out_viz_fullpath = os.path.join(cmd_args.output_viz_dir,
+                                          out_viz_filename)
+          _PlotBoxesAndSaveImage(im[0], selected_boxes, out_viz_fullpath)
 
       # Finalize enqueue threads.
       coord.request_stop()
@@ -200,5 +282,14 @@ if __name__ == '__main__':
       will be written to a file with same name, and extension replaced by
       .boxes.
       """)
+  parser.add_argument(
+      '--output_viz_dir',
+      type=str,
+      default='',
+      help="""
+      Optional. If set, a visualization of the detected boxes overlaid on the
+      image is produced, and saved to this directory. Each image is saved with
+      _viz.jpg suffix.
+      """)
   cmd_args, unparsed = parser.parse_known_args()
   app.run(main=main, argv=[sys.argv[0]] + unparsed)

research/delf/delf/python/examples/extract_features.py

@@ -59,27 +59,37 @@ def _ReadImageList(list_path):
   return image_paths
 
 
-def MakeExtractor(sess, config):
+def MakeExtractor(sess, config, import_scope=None):
   """Creates a function to extract features from an image.
 
   Args:
     sess: TensorFlow session to use.
     config: DelfConfig proto containing the model configuration.
+    import_scope: Optional scope to use for model.
 
   Returns:
     Function that receives an image and returns features.
   """
-  tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING],
-                             config.model_path)
-  input_image = sess.graph.get_tensor_by_name('input_image:0')
-  input_score_threshold = sess.graph.get_tensor_by_name('input_abs_thres:0')
-  input_image_scales = sess.graph.get_tensor_by_name('input_scales:0')
+  tf.saved_model.loader.load(
+      sess, [tf.saved_model.tag_constants.SERVING],
+      config.model_path,
+      import_scope=import_scope)
+  import_scope_prefix = import_scope + '/' if import_scope is not None else ''
+  input_image = sess.graph.get_tensor_by_name('%sinput_image:0' %
+                                              import_scope_prefix)
+  input_score_threshold = sess.graph.get_tensor_by_name('%sinput_abs_thres:0' %
+                                                        import_scope_prefix)
+  input_image_scales = sess.graph.get_tensor_by_name('%sinput_scales:0' %
+                                                     import_scope_prefix)
   input_max_feature_num = sess.graph.get_tensor_by_name(
-      'input_max_feature_num:0')
-  boxes = sess.graph.get_tensor_by_name('boxes:0')
-  raw_descriptors = sess.graph.get_tensor_by_name('features:0')
-  feature_scales = sess.graph.get_tensor_by_name('scales:0')
-  attention_with_extra_dim = sess.graph.get_tensor_by_name('scores:0')
+      '%sinput_max_feature_num:0' % import_scope_prefix)
+  boxes = sess.graph.get_tensor_by_name('%sboxes:0' % import_scope_prefix)
+  raw_descriptors = sess.graph.get_tensor_by_name('%sfeatures:0' %
+                                                  import_scope_prefix)
+  feature_scales = sess.graph.get_tensor_by_name('%sscales:0' %
+                                                 import_scope_prefix)
+  attention_with_extra_dim = sess.graph.get_tensor_by_name('%sscores:0' %
+                                                           import_scope_prefix)
   attention = tf.reshape(attention_with_extra_dim,
                          [tf.shape(attention_with_extra_dim)[0]])