Skip to content

GitLab

Unverified Commit 8a0f2272 authored by André Araujo's avatar André Araujo Committed by GitHub
Browse files

DELF codebase general cleanup (#9930) 

* Merged commit includes the following changes:
253126424  by Andre Araujo:

    Scripts to compute metrics for Google Landmarks dataset.

    Also, a small fix to metric in retrieval case: avoids duplicate predicted images.

--
253118971  by Andre Araujo:

    Metrics for Google Landmarks dataset.

--
253106953  by Andre Araujo:

    Library to read files from Google Landmarks challenges.

--
250700636  by Andre Araujo:

    Handle case of aggregation extraction with empty set of input features.

--
250516819  by Andre Araujo:

    Add minimum size for DELF extractor.

--
250435822  by Andre Araujo:

    Add max_image_size/min_image_size for open-source DELF proto / module.

--
250414606  by Andre Araujo:

    Refactor extract_aggregation to allow reuse with different datasets.

--
250356863  by Andre Araujo:

    Remove unnecessary cmd_args variable from boxes_and_features_extraction.

--
249783379  by Andre Araujo:

    Create directory for writing mapping file if it does not exist.

--
249581591  by Andre Araujo:

    Refactor scripts to extract boxes and features from images in Revisited datasets.
    Also, change tf.logging.info --> print for easier logging in open source code.

--
249511821  by Andre Araujo:

    Small change to function for file/directory handling.

--
249289499  by Andre Araujo:

    Internal change.

--

PiperOrigin-RevId: 253126424

* Updating DELF init to adjust to latest changes

* Editing init files for python packages

* Edit D2R dataset reader to work with py3.

PiperOrigin-RevId: 253135576

* DELF package: fix import ordering

* Adding new requirements to setup.py

* Adding init file for training dir

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* Adding init file for training subdirs

* Working version of DELF training

* Internal change.

PiperOrigin-RevId: 253248648

* Fix variance loading in open-source code.

PiperOrigin-RevId: 260619120

* Separate image re-ranking as a standalone library, and add metric writing to dataset library.

PiperOrigin-RevId: 260998608

* Tool to read written D2R Revisited datasets metrics file. Test is added.

Also adds a unit test for previously-existing SaveMetricsFile function.

PiperOrigin-RevId: 263361410

* Add optional resize factor for feature extraction.

PiperOrigin-RevId: 264437080

* Fix NumPy's new version spacing changes.

PiperOrigin-RevId: 265127245

* Maker image matching function visible, and add support for RANSAC seed.

PiperOrigin-RevId: 277177468

* Avoid matplotlib failure due to missing display backend.

PiperOrigin-RevId: 287316435

* Removes tf.contrib dependency.

PiperOrigin-RevId: 288842237

* Fix tf contrib removal for feature_aggregation_extractor.

PiperOrigin-RevId: 289487669

* Merged commit includes the following changes:
309118395  by Andre Araujo:

    Make DELF open-source code compatible with TF2.

--
309067582  by Andre Araujo:

    Handle image resizing rounding properly for python extraction.

    New behavior is tested with unit tests.

--
308690144  by Andre Araujo:

    Several changes to improve DELF model/training code and make it work in TF 2.1.0:
    - Rename some files for better clarity
    - Using compat.v1 versions of functions
    - Formatting changes
    - Using more appropriate TF function names

--
308689397  by Andre Araujo:

    Internal change.

--
308341315  by Andre Araujo:

    Remove old slim dependency in DELF open-source model.

    This avoids issues with requiring old TF-v1, making it compatible with latest TF.

--
306777559  by Andre Araujo:

    Internal change

--
304505811  by Andre Araujo:

    Raise error during geometric verification if local features have different dimensionalities.

--
301739992  by Andre Araujo:

    Transform some geometric verification constants into arguments, to allow custom matching.

--
301300324  by Andre Araujo:

    Apply name change(experimental_run_v2 -> run) for all callers in Tensorflow.

--
299919057  by Andre Araujo:

    Automated refactoring to make code Python 3 compatible.

--
297953698  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297521242  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297278247  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297270405  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297238741  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297108605  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
294676131  by Andre Araujo:

    Add option to resize images to square resolutions without aspect ratio preservation.

--
293849641  by Andre Araujo:

    Internal change.

--
293840896  by Andre Araujo:

    Changing Slim import to tf_slim codebase.

--
293661660  by Andre Araujo:

    Allow the delf training script to read from TFRecords dataset.

--
291755295  by Andre Araujo:

    Internal change.

--
291448508  by Andre Araujo:

    Internal change.

--
291414459  by Andre Araujo:

    Adding train script.

--
291384336  by Andre Araujo:

    Adding model export script and test.

--
291260565  by Andre Araujo:

    Adding placeholder for Google Landmarks dataset.

--
291205548  by Andre Araujo:

    Definition of DELF model using Keras ResNet50 as backbone.

--
289500793  by Andre Araujo:

    Add TFRecord building script for delf.

--

PiperOrigin-RevId: 309118395

* Updating README, dependency versions

* Updating training README

* Fixing init import of export_model

* Fixing init import of export_model_utils

* tkinter in INSTALL_INSTRUCTIONS

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* INSTALL_INSTRUCTIONS mentioning different cloning options

* Updating required TF version, since 2.1 is not available in pip

* Internal change.

PiperOrigin-RevId: 309136003

* Fix missing string_input_producer and start_queue_runners in TF2.

PiperOrigin-RevId: 309437512

* Handle RANSAC from skimage's latest versions.

PiperOrigin-RevId: 310170897

* DELF 2.1 version: badge and setup.py updated

* Add TF version badge in INSTALL_INSTRUCTIONS and paper badges in README

* Add paper badges in paper instructions

* Add paper badge to landmark detection instructions

* Small update to DELF training README

* Merged commit includes the following changes:
312614961  by Andre Araujo:

    Instructions/code to reproduce DELG paper results.

--
312523414  by Andre Araujo:

    Fix a minor bug when post-process extracted features, format config.delf_global_config.image_scales_ind to a list.

--
312340276  by Andre Araujo:

    Add support for global feature extraction in DELF open-source codebase.

--
311031367  by Andre Araujo:

    Add use_square_images as an option in DELF config. The default value is false. if it is set, then images are resized to square resolution before feature extraction (e.g. Starburst use case. ) Thought for a while, whether to have two constructor of DescriptorToImageTemplate, but in the end, decide to only keep one, may be less confusing.

--
310658638  by Andre Araujo:

    Option for producing local feature-based image match visualization.

--

PiperOrigin-RevId: 312614961

* DELF README update / DELG instructions

* DELF README update

* DELG instructions update

* Merged commit includes the following changes:

PiperOrigin-RevId: 312695597

* Merged commit includes the following changes:
312754894  by Andre Araujo:

    Code edits / instructions to reproduce GLDv2 results.

--

PiperOrigin-RevId: 312754894

* Markdown updates after adding GLDv2 stuff

* Small updates to DELF README

* Clarify that library must be installed before reproducing results

* Merged commit includes the following changes:
319114828  by Andre Araujo:

    Upgrade global feature model exporting to TF2.

--

PiperOrigin-RevId: 319114828

* Properly merging README

* small edits to README

* small edits to README

* small edits to README

* global feature exporting in training README

* Update to DELF README, install instructions

* Centralizing installation instructions

* Small readme update

* Fixing commas

* Mention DELG acceptance into ECCV'20

* Merged commit includes the following changes:
326723075  by Andre Araujo:

    Move image resize utility into utils.py.

--

PiperOrigin-RevId: 326723075

* Adding back matched_images_demo.png

* Merged commit includes the following changes:
327279047  by Andre Araujo:

    Adapt extractor to handle new form of joint local+global extraction.

--
326733524  by Andre Araujo:

    Internal change.

--

PiperOrigin-RevId: 327279047

* Updated DELG instructions after model extraction refactoring

* Updating GLDv2 paper model baseline

* Merged commit includes the following changes:
328982978  by Andre Araujo:

    Updated DELG model training so that the size of the output tensor is unchanged by the GeM pooling layer. Export global model trained with DELG global features.

--
328218938  by Andre Araujo:

    Internal change.

--

PiperOrigin-RevId: 328982978

* Updated training README after recent changes

* Updated training README to fix small typo

* Merged commit includes the following changes:
330022709  by Andre Araujo:

    Export joint local+global TF2 DELG model, and enable such joint extraction.

    Also, rename export_model.py -> export_local_model.py for better clarity.

    To check that the new exporting code is doing the right thing, I compared features extracted from the new exported model against those extracted from models exported with a single modality, using the same checkpoint. They are identical.

    Some other small changes:
    - small automatic reformating
    - small documentation improvements

--

PiperOrigin-RevId: 330022709

* Updated DELG exporting instructions

* Updated DELG exporting instructions: fix small typo

* Adding DELG pre-trained models on GLDv2-clean

* Merged commit includes the following changes:
331625297  by Andre Araujo:

    Internal change.

--
330062115  by Andre Araujo:

    Fix small (non-critical) typo in the DELG extractor.

--

PiperOrigin-RevId: 331625297

* Merged commit includes the following changes:
347479009  by Andre Araujo:

    Fix image size setting for GLD training.

--

PiperOrigin-RevId: 347479009

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_0/google3/..

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_0/google3/..

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_1/google3/..

* Add whiten module import
parent d9ed5232
Hide whitespace changes
Inline Side-by-side
Showing with 41 additions and 44 deletions
research/delf/delf/python/training/train.py
View file @ 8a0f2272
......@@ -41,12 +41,10 @@ FLAGS = flags.FLAGS
flags.DEFINE_boolean('debug', False, 'Debug mode.')
flags.DEFINE_string('logdir', '/tmp/delf', 'WithTensorBoard logdir.')
flags.DEFINE_string(
'train_file_pattern', '/tmp/data/train*',
'File pattern of training dataset files.')
flags.DEFINE_string(
'validation_file_pattern', '/tmp/data/validation*',
'File pattern of validation dataset files.')
flags.DEFINE_string('train_file_pattern', '/tmp/data/train*',
'File pattern of training dataset files.')
flags.DEFINE_string('validation_file_pattern', '/tmp/data/validation*',
'File pattern of validation dataset files.')
flags.DEFINE_enum(
'dataset_version', 'gld_v1', ['gld_v1', 'gld_v2', 'gld_v2_clean'],
'Google Landmarks dataset version, used to determine the number of '
......@@ -56,8 +54,8 @@ flags.DEFINE_float('initial_lr', 0.01, 'Initial learning rate.')
flags.DEFINE_integer('batch_size', 32, 'Global batch size.')
flags.DEFINE_integer('max_iters', 500000, 'Maximum iterations.')
flags.DEFINE_boolean('block3_strides', True, 'Whether to use block3_strides.')
flags.DEFINE_boolean(
'use_augmentation', True, 'Whether to use ImageNet style augmentation.')
flags.DEFINE_boolean('use_augmentation', True,
'Whether to use ImageNet style augmentation.')
flags.DEFINE_string(
'imagenet_checkpoint', None,
'ImageNet checkpoint for ResNet backbone. If None, no checkpoint is used.')
......@@ -65,11 +63,10 @@ flags.DEFINE_float(
'attention_loss_weight', 1.0,
'Weight to apply to the attention loss when calculating the '
'total loss of the model.')
flags.DEFINE_boolean(
'delg_global_features', False, 'Whether to train a DELG model.')
flags.DEFINE_boolean('delg_global_features', False,
'Whether to train a DELG model.')
flags.DEFINE_float(
'delg_gem_power', 3.0,
'Power for Generalized Mean pooling. Used only if '
'delg_gem_power', 3.0, 'Power for Generalized Mean pooling. Used only if '
'delg_global_features=True.')
flags.DEFINE_integer(
'delg_embedding_layer_dim', 2048,
......@@ -79,12 +76,11 @@ flags.DEFINE_float(
'delg_scale_factor_init', 45.25,
'Initial value of the scaling factor of the cosine logits. The default '
'value is sqrt(2048). Used only if delg_global_features=True.')
flags.DEFINE_float(
'delg_arcface_margin', 0.1,
'ArcFace margin. Used only if delg_global_features=True.')
flags.DEFINE_float('delg_arcface_margin', 0.1,
'ArcFace margin. Used only if delg_global_features=True.')
flags.DEFINE_integer('image_size', 321, 'Size of each image side to use.')
flags.DEFINE_boolean(
'use_autoencoder', True, 'Whether to train an autoencoder.')
flags.DEFINE_boolean('use_autoencoder', True,
'Whether to train an autoencoder.')
flags.DEFINE_float(
'reconstruction_loss_weight', 10.0,
'Weight to apply to the reconstruction loss from the autoencoder when'
......@@ -329,9 +325,9 @@ def main(argv):
reconstruction_loss = 0
# Cumulate global loss, attention loss and reconstruction loss.
total_loss = (desc_loss
+ FLAGS.attention_loss_weight * attn_loss
+ FLAGS.reconstruction_loss_weight * reconstruction_loss)
total_loss = (
desc_loss + FLAGS.attention_loss_weight * attn_loss +
FLAGS.reconstruction_loss_weight * reconstruction_loss)
# Perform backpropagation through the descriptor and attention layers
# together. Note that this will increment the number of iterations of
......@@ -537,8 +533,8 @@ def main(argv):
# TODO(andrearaujo): save only in one of the two ways. They are
# identical, the only difference is that the manager adds some extra
# prefixes and variables (eg, optimizer variables).
if (global_step_value %
save_interval == 0) or (global_step_value >= max_iters):
if (global_step_value % save_interval
== 0) or (global_step_value >= max_iters):
save_path = manager.save(checkpoint_number=global_step_value)
logging.info('Saved (%d) at %s', global_step_value, save_path)
......
research/delf/delf/python/whiten.py
View file @ 8a0f2272
......@@ -19,7 +19,9 @@ import os
import numpy as np
def apply_whitening(descriptors, mean_descriptor_vector, projection,
def apply_whitening(descriptors,
mean_descriptor_vector,
projection,
output_dim=None):
"""Applies the whitening to the descriptors as a post-processing step.
......@@ -42,7 +44,7 @@ def apply_whitening(descriptors, mean_descriptor_vector, projection,
descriptors = np.dot(projection[:output_dim, :],
descriptors - mean_descriptor_vector)
descriptors_whitened = descriptors / (
np.linalg.norm(descriptors, ord=2, axis=0, keepdims=True) + eps)
np.linalg.norm(descriptors, ord=2, axis=0, keepdims=True) + eps)
return descriptors_whitened
......@@ -72,14 +74,17 @@ def learn_whitening(descriptors, qidxs, pidxs):
mean_descriptor_vector = descriptors[:, qidxs].mean(axis=1, keepdims=True)
# Interclass (matching pairs) difference.
interclass_difference = descriptors[:, qidxs] - descriptors[:, pidxs]
covariance_matrix = (np.dot(interclass_difference, interclass_difference.T) /
interclass_difference.shape[1])
covariance_matrix = (
np.dot(interclass_difference, interclass_difference.T) /
interclass_difference.shape[1])
# Whitening part.
projection = np.linalg.inv(cholesky(covariance_matrix))
projected_X = np.dot(projection, descriptors - mean_descriptor_vector)
non_matching_covariance_matrix = np.dot(projected_X, projected_X.T)
projected_descriptors = np.dot(projection,
descriptors - mean_descriptor_vector)
non_matching_covariance_matrix = np.dot(projected_descriptors,
projected_descriptors.T)
eigval, eigvec = np.linalg.eig(non_matching_covariance_matrix)
order = eigval.argsort()[::-1]
eigvec = eigvec[:, order]
......@@ -116,6 +121,5 @@ def cholesky(matrix):
alpha = 1e-10
else:
alpha *= 10
print(
">>>> {}::cholesky: Matrix is not positive definite, adding {:.0e} "
"on the diagonal".format(os.path.basename(__file__), alpha))
print(">>>> {}::cholesky: Matrix is not positive definite, adding {:.0e} "
"on the diagonal".format(os.path.basename(__file__), alpha))
research/delf/delf/python/whiten_test.py
View file @ 8a0f2272
......@@ -18,15 +18,14 @@
import numpy as np
import tensorflow as tf
from delf.python import whiten
from delf import whiten
class WhitenTest(tf.test.TestCase):
def testApplyWhitening(self):
# Testing the application of the learned whitening.
vectors = np.array([[0.14022471, 0.96360618],
[0.37601032, 0.25528411]])
vectors = np.array([[0.14022471, 0.96360618], [0.37601032, 0.25528411]])
# Learn whitening for the `vectors`. First element in the `vectors` is
# viewed is the example query and the second element is the corresponding
# positive.
......@@ -40,12 +39,10 @@ class WhitenTest(tf.test.TestCase):
def testLearnWhitening(self):
# Testing whitening learning function.
input = np.array([[0.14022471, 0.96360618],
[0.37601032, 0.25528411]])
descriptors = np.array([[0.14022471, 0.96360618], [0.37601032, 0.25528411]])
# Obtain the mean descriptor vector and the projection matrix.
mean_vector, projection = whiten.learn_whitening(input, [0], [1])
expected_mean_vector = np.array([[0.14022471],
[0.37601032]])
mean_vector, projection = whiten.learn_whitening(descriptors, [0], [1])
expected_mean_vector = np.array([[0.14022471], [0.37601032]])
expected_projection = np.array([[1.18894378e+00, -1.74326044e-01],
[1.45071361e+04, 9.89421193e+04]])
# Check that the both calculated values are close to the expected values.
......@@ -54,13 +51,13 @@ class WhitenTest(tf.test.TestCase):
def testCholeskyPositiveDefinite(self):
# Testing the Cholesky decomposition for the positive definite matrix.
input = np.array([[1, -2j], [2j, 5]])
output = whiten.cholesky(input)
descriptors = np.array([[1, -2j], [2j, 5]])
output = whiten.cholesky(descriptors)
expected_output = np.array([[1. + 0.j, 0. + 0.j], [0. + 2.j, 1. + 0.j]])
# Check that the expected output is obtained.
self.assertAllClose(output, expected_output)
# Check that the properties of the Cholesky decomposition are satisfied.
self.assertAllClose(np.matmul(output, output.T.conj()), input)
self.assertAllClose(np.matmul(output, output.T.conj()), descriptors)
def testCholeskyNonPositiveDefinite(self):
# Testing the Cholesky decomposition for a non-positive definite matrix.
......@@ -68,8 +65,8 @@ class WhitenTest(tf.test.TestCase):
decomposition = whiten.cholesky(input_matrix)
expected_output = np.array([[2., -2.], [-2., 2.]])
# Check that the properties of the Cholesky decomposition are satisfied.
self.assertAllClose(np.matmul(decomposition, decomposition.T),
expected_output)
self.assertAllClose(
np.matmul(decomposition, decomposition.T), expected_output)
if __name__ == '__main__':
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment