Refactor DELG extraction + Release latest DELG paper models (#9145)

* Merged commit includes the following changes:
253126424  by Andre Araujo:

    Scripts to compute metrics for Google Landmarks dataset.

    Also, a small fix to metric in retrieval case: avoids duplicate predicted images.

--
253118971  by Andre Araujo:

    Metrics for Google Landmarks dataset.

--
253106953  by Andre Araujo:

    Library to read files from Google Landmarks challenges.

--
250700636  by Andre Araujo:

    Handle case of aggregation extraction with empty set of input features.

--
250516819  by Andre Araujo:

    Add minimum size for DELF extractor.

--
250435822  by Andre Araujo:

    Add max_image_size/min_image_size for open-source DELF proto / module.

--
250414606  by Andre Araujo:

    Refactor extract_aggregation to allow reuse with different datasets.

--
250356863  by Andre Araujo:

    Remove unnecessary cmd_args variable from boxes_and_features_extraction.

--
249783379  by Andre Araujo:

    Create directory for writing mapping file if it does not exist.

--
249581591  by Andre Araujo:

    Refactor scripts to extract boxes and features from images in Revisited datasets.
    Also, change tf.logging.info --> print for easier logging in open source code.

--
249511821  by Andre Araujo:

    Small change to function for file/directory handling.

--
249289499  by Andre Araujo:

    Internal change.

--

PiperOrigin-RevId: 253126424

* Updating DELF init to adjust to latest changes

* Editing init files for python packages

* Edit D2R dataset reader to work with py3.

PiperOrigin-RevId: 253135576

* DELF package: fix import ordering

* Adding new requirements to setup.py

* Adding init file for training dir

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* Adding init file for training subdirs

* Working version of DELF training

* Internal change.

PiperOrigin-RevId: 253248648

* Fix variance loading in open-source code.

PiperOrigin-RevId: 260619120

* Separate image re-ranking as a standalone library, and add metric writing to dataset library.

PiperOrigin-RevId: 260998608

* Tool to read written D2R Revisited datasets metrics file. Test is added.

Also adds a unit test for previously-existing SaveMetricsFile function.

PiperOrigin-RevId: 263361410

* Add optional resize factor for feature extraction.

PiperOrigin-RevId: 264437080

* Fix NumPy's new version spacing changes.

PiperOrigin-RevId: 265127245

* Maker image matching function visible, and add support for RANSAC seed.

PiperOrigin-RevId: 277177468

* Avoid matplotlib failure due to missing display backend.

PiperOrigin-RevId: 287316435

* Removes tf.contrib dependency.

PiperOrigin-RevId: 288842237

* Fix tf contrib removal for feature_aggregation_extractor.

PiperOrigin-RevId: 289487669

* Merged commit includes the following changes:
309118395  by Andre Araujo:

    Make DELF open-source code compatible with TF2.

--
309067582  by Andre Araujo:

    Handle image resizing rounding properly for python extraction.

    New behavior is tested with unit tests.

--
308690144  by Andre Araujo:

    Several changes to improve DELF model/training code and make it work in TF 2.1.0:
    - Rename some files for better clarity
    - Using compat.v1 versions of functions
    - Formatting changes
    - Using more appropriate TF function names

--
308689397  by Andre Araujo:

    Internal change.

--
308341315  by Andre Araujo:

    Remove old slim dependency in DELF open-source model.

    This avoids issues with requiring old TF-v1, making it compatible with latest TF.

--
306777559  by Andre Araujo:

    Internal change

--
304505811  by Andre Araujo:

    Raise error during geometric verification if local features have different dimensionalities.

--
301739992  by Andre Araujo:

    Transform some geometric verification constants into arguments, to allow custom matching.

--
301300324  by Andre Araujo:

    Apply name change(experimental_run_v2 -> run) for all callers in Tensorflow.

--
299919057  by Andre Araujo:

    Automated refactoring to make code Python 3 compatible.

--
297953698  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297521242  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297278247  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297270405  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297238741  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297108605  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
294676131  by Andre Araujo:

    Add option to resize images to square resolutions without aspect ratio preservation.

--
293849641  by Andre Araujo:

    Internal change.

--
293840896  by Andre Araujo:

    Changing Slim import to tf_slim codebase.

--
293661660  by Andre Araujo:

    Allow the delf training script to read from TFRecords dataset.

--
291755295  by Andre Araujo:

    Internal change.

--
291448508  by Andre Araujo:

    Internal change.

--
291414459  by Andre Araujo:

    Adding train script.

--
291384336  by Andre Araujo:

    Adding model export script and test.

--
291260565  by Andre Araujo:

    Adding placeholder for Google Landmarks dataset.

--
291205548  by Andre Araujo:

    Definition of DELF model using Keras ResNet50 as backbone.

--
289500793  by Andre Araujo:

    Add TFRecord building script for delf.

--

PiperOrigin-RevId: 309118395

* Updating README, dependency versions

* Updating training README

* Fixing init import of export_model

* Fixing init import of export_model_utils

* tkinter in INSTALL_INSTRUCTIONS

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* INSTALL_INSTRUCTIONS mentioning different cloning options

* Updating required TF version, since 2.1 is not available in pip

* Internal change.

PiperOrigin-RevId: 309136003

* Fix missing string_input_producer and start_queue_runners in TF2.

PiperOrigin-RevId: 309437512

* Handle RANSAC from skimage's latest versions.

PiperOrigin-RevId: 310170897

* DELF 2.1 version: badge and setup.py updated

* Add TF version badge in INSTALL_INSTRUCTIONS and paper badges in README

* Add paper badges in paper instructions

* Add paper badge to landmark detection instructions

* Small update to DELF training README

* Merged commit includes the following changes:
312614961  by Andre Araujo:

    Instructions/code to reproduce DELG paper results.

--
312523414  by Andre Araujo:

    Fix a minor bug when post-process extracted features, format config.delf_global_config.image_scales_ind to a list.

--
312340276  by Andre Araujo:

    Add support for global feature extraction in DELF open-source codebase.

--
311031367  by Andre Araujo:

    Add use_square_images as an option in DELF config. The default value is false. if it is set, then images are resized to square resolution before feature extraction (e.g. Starburst use case. ) Thought for a while, whether to have two constructor of DescriptorToImageTemplate, but in the end, decide to only keep one, may be less confusing.

--
310658638  by Andre Araujo:

    Option for producing local feature-based image match visualization.

--

PiperOrigin-RevId: 312614961

* DELF README update / DELG instructions

* DELF README update

* DELG instructions update

* Merged commit includes the following changes:

PiperOrigin-RevId: 312695597

* Merged commit includes the following changes:
312754894  by Andre Araujo:

    Code edits / instructions to reproduce GLDv2 results.

--

PiperOrigin-RevId: 312754894

* Markdown updates after adding GLDv2 stuff

* Small updates to DELF README

* Clarify that library must be installed before reproducing results

* Merged commit includes the following changes:
319114828  by Andre Araujo:

    Upgrade global feature model exporting to TF2.

--

PiperOrigin-RevId: 319114828

* Properly merging README

* small edits to README

* small edits to README

* small edits to README

* global feature exporting in training README

* Update to DELF README, install instructions

* Centralizing installation instructions

* Small readme update

* Fixing commas

* Mention DELG acceptance into ECCV'20

* Merged commit includes the following changes:
326723075  by Andre Araujo:

    Move image resize utility into utils.py.

--

PiperOrigin-RevId: 326723075

* Adding back matched_images_demo.png

* Merged commit includes the following changes:
327279047  by Andre Araujo:

    Adapt extractor to handle new form of joint local+global extraction.

--
326733524  by Andre Araujo:

    Internal change.

--

PiperOrigin-RevId: 327279047

* Updated DELG instructions after model extraction refactoring

* Updating GLDv2 paper model baseline

research/delf/README.md

@@ -112,11 +112,12 @@ in the [Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584). Boosts
 performance by ~4% mAP compared to ICCV'17 DELF model.
 
 **DELG pre-trained on the Google-Landmarks dataset v1**
-([link](http://storage.googleapis.com/delf/delg_gld_20200520.tar.gz)). Presented
-in the [DELG paper](https://arxiv.org/abs/2001.05027).
+([R101-DELG](http://storage.googleapis.com/delf/r101delg_gld_20200814.tar.gz),
+[R50-DELG](http://storage.googleapis.com/delf/r50delg_gld_20200814.tar.gz)).
+Presented in the [DELG paper](https://arxiv.org/abs/2001.05027).
 
 **RN101-ArcFace pre-trained on the Google-Landmarks dataset v2 (train-clean)**
-([link](https://storage.googleapis.com/delf/rn101_af_gldv2clean_20200521.tar.gz)).
+([link](https://storage.googleapis.com/delf/rn101_af_gldv2clean_20200814.tar.gz)).
 Presented in the [GLDv2 paper](https://arxiv.org/abs/2004.01804).
 
 **DELF pre-trained on Landmarks-Clean/Landmarks-Full dataset**

research/delf/delf/python/delg/DELG_INSTRUCTIONS.md

@@ -38,22 +38,29 @@ wget http://cmp.felk.cvut.cz/revisitop/data/datasets/rparis6k/gnd_rparis6k.mat
 
 ### Download model
 
-This is necessary to reproduce the main paper results:
+This is necessary to reproduce the main paper results. This example shows the
+R50-DELG model; the
+[R101-DELG model](http://storage.googleapis.com/delf/r101delg_gld_20200814.tar.gz)
+can be used as well, with similar steps.
 
 ```bash
 # From models/research/delf/delf/python/delg
 mkdir parameters && cd parameters
 
-# DELG-GLD model.
-wget http://storage.googleapis.com/delf/delg_gld_20200520.tar.gz
-tar -xvzf delg_gld_20200520.tar.gz
+# R50-DELG-GLD model.
+wget http://storage.googleapis.com/delf/r50delg_gld_20200814.tar.gz
+tar -xvzf r50delg_gld_20200814.tar.gz
 ```
 
 ### Feature extraction
 
-We present here commands for extraction on `roxford5k`. To extract on `rparis6k`
-instead, please edit the arguments accordingly (especially the
-`dataset_file_path` argument).
+We present here commands for R50-DELG extraction on `roxford5k`.
+
+-   To use the R101-DELG model, first download it as done above for the R50
+    variant; then, replace the below argument `delf_config_path` by
+    `r101delg_gld_config.pbtxt`
+-   To extract on `rparis6k` instead, please edit the arguments accordingly
+    (especially the `dataset_file_path` argument).
 
 #### Query feature extraction
 
@@ -67,7 +74,7 @@ Query feature extraction can be run as follows:
 ```bash
 # From models/research/delf/delf/python/delg
 python3 extract_features.py \
-  --delf_config_path delg_gld_config.pbtxt \
+  --delf_config_path r50delg_gld_config.pbtxt \
   --dataset_file_path ~/delg/data/gnd_roxford5k.mat \
   --images_dir ~/delg/data/oxford5k_images \
   --image_set query \
@@ -81,7 +88,7 @@ Run index feature extraction as follows:
 ```bash
 # From models/research/delf/delf/python/delg
 python3 extract_features.py \
-  --delf_config_path delg_gld_config.pbtxt \
+  --delf_config_path r50delg_gld_config.pbtxt \
   --dataset_file_path ~/delg/data/gnd_roxford5k.mat \
   --images_dir ~/delg/data/oxford5k_images \
   --image_set index \

research/delf/delf/python/delg/r101delg_gld_config.pbtxt

+use_local_features: true
+use_global_features: true
+model_path: "parameters/r101delg_gld_20200814"
+image_scales: 0.25
+image_scales: 0.35355338
+image_scales: 0.5
+image_scales: 0.70710677
+image_scales: 1.0
+image_scales: 1.4142135
+image_scales: 2.0
+delf_local_config {
+  use_pca: false
+  max_feature_num: 1000
+  score_threshold: 166.1
+}
+delf_global_config {
+  use_pca: false
+  image_scales_ind: 3
+  image_scales_ind: 4
+  image_scales_ind: 5
+}
+max_image_size: 1024

research/delf/delf/python/delg/delg_gld_config.pbtxt → research/delf/delf/python/delg/r50delg_gld_config.pbtxt

 use_local_features: true
 use_global_features: true
-model_path: "parameters/delg_gld_20200520"
+model_path: "parameters/r50delg_gld_20200814"
 image_scales: 0.25
 image_scales: 0.35355338
 image_scales: 0.5

research/delf/delf/python/examples/extractor.py

@@ -43,18 +43,24 @@ def MakeExtractor(config):
   Raises:
     ValueError: if config is invalid.
   """
-  # Assert the configuration
-  if config.use_global_features and hasattr(
+  # Assert the configuration.
+  # TODO(andrearaujo): Handle this case.
+  if config.use_global_features and config.use_local_features and hasattr(
       config, 'is_tf2_exported') and config.is_tf2_exported:
-    raise ValueError('use_global_features is incompatible with is_tf2_exported')
+    raise ValueError(
+        'Joint local+global extraction is currently incompatible with '
+        'is_tf2_exported')
 
   # Load model.
   model = tf.saved_model.load(config.model_path)
 
-  # Input/output end-points/tensors.
+  # Input image scales to use for extraction.
+  image_scales_tensor = tf.convert_to_tensor(list(config.image_scales))
+
+  # Input (feeds) and output (fetches) end-points. These are only needed when
+  # using a model that was exported using TF1.
   feeds = ['input_image:0', 'input_scales:0']
   fetches = []
-  image_scales_tensor = tf.convert_to_tensor(list(config.image_scales))
 
   # Custom configuration needed when local features are used.
   if config.use_local_features:
@@ -96,8 +102,14 @@ def MakeExtractor(config):
 
   # Custom configuration needed when global features are used.
   if config.use_global_features:
-    # Extra output end-point.
+    # Extra input/output end-points/tensors.
+    feeds.append('input_global_scales_ind:0')
     fetches.append('global_descriptors:0')
+    if config.delf_global_config.image_scales_ind:
+      global_scales_ind_tensor = tf.constant(
+          list(config.delf_global_config.image_scales_ind))
+    else:
+      global_scales_ind_tensor = tf.range(len(config.image_scales))
 
     # If using PCA, pre-load required parameters.
     global_pca_parameters = {}
@@ -181,23 +193,31 @@ def MakeExtractor(config):
             output_dict['scales'], output_dict['scores']
         ]
       else:
-        output = model(image_tensor, image_scales_tensor,
-                       score_threshold_tensor, max_feature_num_tensor)
+        if config.use_global_features:
+          output = model(image_tensor, image_scales_tensor,
+                         score_threshold_tensor, max_feature_num_tensor,
+                         global_scales_ind_tensor)
+        else:
+          output = model(image_tensor, image_scales_tensor,
+                         score_threshold_tensor, max_feature_num_tensor)
     else:
-      output = model(image_tensor, image_scales_tensor)
+      if hasattr(config, 'is_tf2_exported') and config.is_tf2_exported:
+        predict = model.signatures['serving_default']
+        output_dict = predict(
+            input_image=image_tensor,
+            input_scales=image_scales_tensor,
+            input_global_scales_ind=global_scales_ind_tensor)
+        output = [output_dict['global_descriptors']]
+      else:
+        output = model(image_tensor, image_scales_tensor,
+                       global_scales_ind_tensor)
 
     # Post-process extracted features: normalize, PCA (optional), pooling.
     if config.use_global_features:
       raw_global_descriptors = output[-1]
-      if config.delf_global_config.image_scales_ind:
-        raw_global_descriptors_selected_scales = tf.gather(
-            raw_global_descriptors,
-            list(config.delf_global_config.image_scales_ind))
-      else:
-        raw_global_descriptors_selected_scales = raw_global_descriptors
       global_descriptors_per_scale = feature_extractor.PostProcessDescriptors(
-          raw_global_descriptors_selected_scales,
-          config.delf_global_config.use_pca, global_pca_parameters)
+          raw_global_descriptors, config.delf_global_config.use_pca,
+          global_pca_parameters)
       unnormalized_global_descriptor = tf.reduce_sum(
           global_descriptors_per_scale, axis=0, name='sum_pooling')
       global_descriptor = tf.nn.l2_normalize(

research/delf/delf/python/google_landmarks_dataset/README.md

@@ -52,8 +52,8 @@ wget http://cmp.felk.cvut.cz/revisitop/data/datasets/rparis6k/gnd_rparis6k.mat
 mkdir parameters && cd parameters
 
 # RN101-ArcFace model trained on GLDv2-clean.
-wget https://storage.googleapis.com/delf/rn101_af_gldv2clean_20200521.tar.gz
-tar -xvzf rn101_af_gldv2clean_20200521.tar.gz
+wget https://storage.googleapis.com/delf/rn101_af_gldv2clean_20200814.tar.gz
+tar -xvzf rn101_af_gldv2clean_20200814.tar.gz
 ```
 
 ### Feature extraction

research/delf/delf/python/google_landmarks_dataset/rn101_af_gldv2clean_config.pbtxt

 use_local_features: false
 use_global_features: true
-model_path: "parameters/rn101_af_gldv2clean_20200521"
+model_path: "parameters/rn101_af_gldv2clean_20200814"
 image_scales: 0.70710677
 image_scales: 1.0
 image_scales: 1.4142135