Skip to content

GitLab

Unverified Commit 979e296a authored by André Araujo's avatar André Araujo Committed by GitHub
Browse files

Add GLDv2 baseline in DELF codebase (#8557) 

* Merged commit includes the following changes:
253126424  by Andre Araujo:

    Scripts to compute metrics for Google Landmarks dataset.

    Also, a small fix to metric in retrieval case: avoids duplicate predicted images.

--
253118971  by Andre Araujo:

    Metrics for Google Landmarks dataset.

--
253106953  by Andre Araujo:

    Library to read files from Google Landmarks challenges.

--
250700636  by Andre Araujo:

    Handle case of aggregation extraction with empty set of input features.

--
250516819  by Andre Araujo:

    Add minimum size for DELF extractor.

--
250435822  by Andre Araujo:

    Add max_image_size/min_image_size for open-source DELF proto / module.

--
250414606  by Andre Araujo:

    Refactor extract_aggregation to allow reuse with different datasets.

--
250356863  by Andre Araujo:

    Remove unnecessary cmd_args variable from boxes_and_features_extraction.

--
249783379  by Andre Araujo:

    Create directory for writing mapping file if it does not exist.

--
249581591  by Andre Araujo:

    Refactor scripts to extract boxes and features from images in Revisited datasets.
    Also, change tf.logging.info --> print for easier logging in open source code.

--
249511821  by Andre Araujo:

    Small change to function for file/directory handling.

--
249289499  by Andre Araujo:

    Internal change.

--

PiperOrigin-RevId: 253126424

* Updating DELF init to adjust to latest changes

* Editing init files for python packages

* Edit D2R dataset reader to work with py3.

PiperOrigin-RevId: 253135576

* DELF package: fix import ordering

* Adding new requirements to setup.py

* Adding init file for training dir

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* Adding init file for training subdirs

* Working version of DELF training

* Internal change.

PiperOrigin-RevId: 253248648

* Fix variance loading in open-source code.

PiperOrigin-RevId: 260619120

* Separate image re-ranking as a standalone library, and add metric writing to dataset library.

PiperOrigin-RevId: 260998608

* Tool to read written D2R Revisited datasets metrics file. Test is added.

Also adds a unit test for previously-existing SaveMetricsFile function.

PiperOrigin-RevId: 263361410

* Add optional resize factor for feature extraction.

PiperOrigin-RevId: 264437080

* Fix NumPy's new version spacing changes.

PiperOrigin-RevId: 265127245

* Maker image matching function visible, and add support for RANSAC seed.

PiperOrigin-RevId: 277177468

* Avoid matplotlib failure due to missing display backend.

PiperOrigin-RevId: 287316435

* Removes tf.contrib dependency.

PiperOrigin-RevId: 288842237

* Fix tf contrib removal for feature_aggregation_extractor.

PiperOrigin-RevId: 289487669

* Merged commit includes the following changes:
309118395  by Andre Araujo:

    Make DELF open-source code compatible with TF2.

--
309067582  by Andre Araujo:

    Handle image resizing rounding properly for python extraction.

    New behavior is tested with unit tests.

--
308690144  by Andre Araujo:

    Several changes to improve DELF model/training code and make it work in TF 2.1.0:
    - Rename some files for better clarity
    - Using compat.v1 versions of functions
    - Formatting changes
    - Using more appropriate TF function names

--
308689397  by Andre Araujo:

    Internal change.

--
308341315  by Andre Araujo:

    Remove old slim dependency in DELF open-source model.

    This avoids issues with requiring old TF-v1, making it compatible with latest TF.

--
306777559  by Andre Araujo:

    Internal change

--
304505811  by Andre Araujo:

    Raise error during geometric verification if local features have different dimensionalities.

--
301739992  by Andre Araujo:

    Transform some geometric verification constants into arguments, to allow custom matching.

--
301300324  by Andre Araujo:

    Apply name change(experimental_run_v2 -> run) for all callers in Tensorflow.

--
299919057  by Andre Araujo:

    Automated refactoring to make code Python 3 compatible.

--
297953698  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297521242  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297278247  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297270405  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297238741  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297108605  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
294676131  by Andre Araujo:

    Add option to resize images to square resolutions without aspect ratio preservation.

--
293849641  by Andre Araujo:

    Internal change.

--
293840896  by Andre Araujo:

    Changing Slim import to tf_slim codebase.

--
293661660  by Andre Araujo:

    Allow the delf training script to read from TFRecords dataset.

--
291755295  by Andre Araujo:

    Internal change.

--
291448508  by Andre Araujo:

    Internal change.

--
291414459  by Andre Araujo:

    Adding train script.

--
291384336  by Andre Araujo:

    Adding model export script and test.

--
291260565  by Andre Araujo:

    Adding placeholder for Google Landmarks dataset.

--
291205548  by Andre Araujo:

    Definition of DELF model using Keras ResNet50 as backbone.

--
289500793  by Andre Araujo:

    Add TFRecord building script for delf.

--

PiperOrigin-RevId: 309118395

* Updating README, dependency versions

* Updating training README

* Fixing init import of export_model

* Fixing init import of export_model_utils

* tkinter in INSTALL_INSTRUCTIONS

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* INSTALL_INSTRUCTIONS mentioning different cloning options

* Updating required TF version, since 2.1 is not available in pip

* Internal change.

PiperOrigin-RevId: 309136003

* Fix missing string_input_producer and start_queue_runners in TF2.

PiperOrigin-RevId: 309437512

* Handle RANSAC from skimage's latest versions.

PiperOrigin-RevId: 310170897

* DELF 2.1 version: badge and setup.py updated

* Add TF version badge in INSTALL_INSTRUCTIONS and paper badges in README

* Add paper badges in paper instructions

* Add paper badge to landmark detection instructions

* Small update to DELF training README

* Merged commit includes the following changes:
312614961  by Andre Araujo:

    Instructions/code to reproduce DELG paper results.

--
312523414  by Andre Araujo:

    Fix a minor bug when post-process extracted features, format config.delf_global_config.image_scales_ind to a list.

--
312340276  by Andre Araujo:

    Add support for global feature extraction in DELF open-source codebase.

--
311031367  by Andre Araujo:

    Add use_square_images as an option in DELF config. The default value is false. if it is set, then images are resized to square resolution before feature extraction (e.g. Starburst use case. ) Thought for a while, whether to have two constructor of DescriptorToImageTemplate, but in the end, decide to only keep one, may be less confusing.

--
310658638  by Andre Araujo:

    Option for producing local feature-based image match visualization.

--

PiperOrigin-RevId: 312614961

* DELF README update / DELG instructions

* DELF README update

* DELG instructions update

* Merged commit includes the following changes:

PiperOrigin-RevId: 312695597

* Merged commit includes the following changes:
312754894  by Andre Araujo:

    Code edits / instructions to reproduce GLDv2 results.

--

PiperOrigin-RevId: 312754894

* Markdown updates after adding GLDv2 stuff

* Small updates to DELF README
parent e64d4f56
Hide whitespace changes
Inline Side-by-side
Showing with 220 additions and 43 deletions
research/delf/README.md
View file @ 979e296a
......@@ -3,20 +3,19 @@
[![TensorFlow 2.1](https://img.shields.io/badge/tensorflow-2.1-brightgreen)](https://github.com/tensorflow/tensorflow/releases/tag/v2.1.0)
[![Python 3.6](https://img.shields.io/badge/python-3.6-blue.svg)](https://www.python.org/downloads/release/python-360/)
This project presents code for extracting local and global image features, which
are particularly useful for large-scale instance-level image recognition. These
were introduced in the [DELF](https://arxiv.org/abs/1612.06321),
[Detect-to-Retrieve](https://arxiv.org/abs/1812.01584) and
[DELG](https://arxiv.org/abs/2001.05027) papers.
This project presents code for deep local and global image feature methods,
which are particularly useful for the computer vision tasks of instance-level
recognition and retrieval. These were introduced in the
[DELF](https://arxiv.org/abs/1612.06321),
[Detect-to-Retrieve](https://arxiv.org/abs/1812.01584),
[DELG](https://arxiv.org/abs/2001.05027) and
[Google Landmarks Dataset v2](https://arxiv.org/abs/2004.01804) papers.
We also released pre-trained models based on the
[Google Landmarks dataset](https://www.kaggle.com/google/google-landmarks-dataset).
We provide Tensorflow code for building and training models, and python code for
image retrieval and local feature matching. Pre-trained models for the landmark
recognition domain are also provided.
The pre-trained models released here have been optimized for landmark
recognition, so expect it to work well in this area. We also provide tensorflow
code for building and training models.
If you make use of this code, please consider citing the following papers:
If you make use of this codebase, please consider citing the following papers:
DELF:
[![Paper](http://img.shields.io/badge/paper-arXiv.1612.06321-B3181B.svg)](https://arxiv.org/abs/1612.06321)
......@@ -45,8 +44,20 @@ B. Cao*, A. Araujo* and J. Sim,
arxiv:2001.05027
```
GLDv2:
[![Paper](http://img.shields.io/badge/paper-arXiv.2004.01804-B3181B.svg)](https://arxiv.org/abs/2004.01804)
```
"Google Landmarks Dataset v2 - A Large-Scale Benchmark for Instance-Level Recognition and Retrieval",
T. Weyand*, A. Araujo*, B. Cao and J. Sim,
Proc. CVPR'20
```
## News
- [Apr'20] Check out our CVPR'20 paper: ["Google Landmarks Dataset v2 - A
Large-Scale Benchmark for Instance-Level Recognition and
Retrieval"](https://arxiv.org/abs/2004.01804)
- [Jan'20] Check out our new paper:
["Unifying Deep Local and Global Features for Image Search"](https://arxiv.org/abs/2001.05027)
- [Jun'19] DELF achieved 2nd place in
......@@ -63,7 +74,7 @@ arxiv:2001.05027
- [Mar'18] DELF is now available in
[TF-Hub](https://www.tensorflow.org/hub/modules/google/delf/1)
## Dataset
## Datasets
We have two Google-Landmarks dataset versions:
......@@ -97,23 +108,29 @@ sections for examples on how to use the models.
**DELF pre-trained on the Google-Landmarks dataset v1**
([link](http://storage.googleapis.com/delf/delf_gld_20190411.tar.gz)). Presented
in the [CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
Boosts performance by ~4% mAP compared to ICCV'17 DELF model.
in the [Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584). Boosts
performance by ~4% mAP compared to ICCV'17 DELF model.
**DELG pre-trained on the Google-Landmarks dataset v1**
([link](http://storage.googleapis.com/delf/delg_gld_20200520.tar.gz)). Presented
in the [DELG paper](https://arxiv.org/abs/2001.05027).
**RN101-ArcFace pre-trained on the Google-Landmarks dataset v2 (train-clean)**
([link](https://storage.googleapis.com/delf/rn101_af_gldv2clean_20200521.tar.gz)).
Presented in the [GLDv2 paper](https://arxiv.org/abs/2004.01804).
**DELF pre-trained on Landmarks-Clean/Landmarks-Full dataset**
([link](http://storage.googleapis.com/delf/delf_v1_20171026.tar.gz)). Presented
in the [ICCV'17 DELF paper](https://arxiv.org/abs/1612.06321), model was trained
on the dataset released by the [DIR paper](https://arxiv.org/abs/1604.01325).
in the [DELF paper](https://arxiv.org/abs/1612.06321), model was trained on the
dataset released by the [DIR paper](https://arxiv.org/abs/1604.01325).
**Faster-RCNN detector pre-trained on Google Landmark Boxes**
([link](http://storage.googleapis.com/delf/d2r_frcnn_20190411.tar.gz)).
Presented in the
[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
Presented in the [Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
**MobileNet-SSD detector pre-trained on Google Landmark Boxes**
([link](http://storage.googleapis.com/delf/d2r_mnetssd_20190411.tar.gz)).
Presented in the
[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
Presented in the [Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
Besides these, we also release pre-trained codebooks for local feature
aggregation. See the
......@@ -137,6 +154,13 @@ Please follow [these instructions](delf/python/delg/DELG_INSTRUCTIONS.md). At
the end, you should obtain image retrieval results on the Revisited Oxford/Paris
datasets.
### GLDv2 baseline
Please follow
[these instructions](delf/python/google_landmarks_dataset/README.md). At the
end, you should obtain image retrieval results on the Revisited Oxford/Paris
datasets.
### Landmark detection
Please follow [these instructions](DETECTION.md). At the end, you should obtain
......@@ -153,8 +177,8 @@ Oxford/Paris datasets.
## Code overview
DELF/D2R code is located under the `delf` directory. There are two directories
therein, `protos` and `python`.
DELF/D2R/DELG/GLD code is located under the `delf` directory. There are two
directories therein, `protos` and `python`.
### `delf/protos`
......@@ -164,7 +188,7 @@ This directory contains protobufs:
aggregation.
- `box.proto`: protobuf for serializing detected boxes.
- `datum.proto`: general-purpose protobuf for serializing float tensors.
- `delf_config.proto`: protobuf for configuring DELF extraction.
- `delf_config.proto`: protobuf for configuring DELF/DELG extraction.
- `feature.proto`: protobuf for serializing DELF features.
### `delf/python`
......@@ -191,7 +215,7 @@ feature extraction/matching, and object detection:
- `detector.py` is a module to construct an object detector function.
- `extract_boxes.py` enables object detection from a list of images.
- `extract_features.py` enables DELF extraction from a list of images.
- `extractor.py` is a module to construct a DELF local feature extraction
- `extractor.py` is a module to construct a DELF/DELG local feature extraction
function.
- `match_images.py` supports image matching using DELF features extracted
using `extract_features.py`.
......@@ -230,7 +254,8 @@ scripts/configs related to the Detect-to-Retrieve paper:
AggregationConfig's for Detect-to-Retrieve experiments.
The subdirectory `delf/python/google_landmarks_dataset` contains sample
scripts/modules for computing GLD metrics:
scripts/modules for computing GLD metrics / reproducing results from the GLDv2
paper:
- `compute_recognition_metrics.py` performs recognition metric computation
given input predictions and solution files.
......@@ -238,6 +263,9 @@ scripts/modules for computing GLD metrics:
input predictions and solution files.
- `dataset_file_io.py` is a module for dataset-related file IO.
- `metrics.py` is a module for GLD metric computation.
- `rn101_af_gldv2clean_config.pbtxt` gives the DelfConfig used in the
ResNet101-ArcFace (trained on GLDv2-train-clean) baseline used in the GLDv2
paper.
The subdirectory `delf/python/training` contains sample scripts/modules for
performing DELF training:
......@@ -262,6 +290,14 @@ André Araujo (@andrefaraujo)
## Release history
### May, 2020
- Codebase is now Python3-first
- DELG model/code released
- GLDv2 baseline model released
**Thanks to contributors**: Barbara Fusinska and André Araujo.
### April, 2020 (version 2.0)
- Initial DELF training code released.
......
research/delf/delf/python/delg/extract_features.py
View file @ 979e296a
......@@ -17,6 +17,9 @@
Note that query images are cropped before feature extraction, as required by the
evaluation protocols of these datasets.
The types of extracted features (local and/or global) depend on the input
DelfConfig.
The program checks if features already exist, and skips computation for those.
"""
......@@ -46,7 +49,8 @@ FLAGS = flags.FLAGS
flags.DEFINE_string(
'delf_config_path', '/tmp/delf_config_example.pbtxt',
'Path to DelfConfig proto text file with configuration to be used for DELG '
'extraction.')
'extraction. Local features are extracted if use_local_features is True; '
'global features are extracted if use_global_features is True.')
flags.DEFINE_string(
'dataset_file_path', '/tmp/gnd_roxford5k.mat',
'Dataset file for Revisited Oxford or Paris dataset, in .mat format.')
......@@ -133,13 +137,21 @@ def main(argv):
image_name = image_list[i]
input_image_filename = os.path.join(FLAGS.images_dir,
image_name + _IMAGE_EXTENSION)
output_global_feature_filename = os.path.join(
FLAGS.output_features_dir, image_name + _DELG_GLOBAL_EXTENSION)
output_local_feature_filename = os.path.join(
FLAGS.output_features_dir, image_name + _DELG_LOCAL_EXTENSION)
if tf.io.gfile.exists(
output_global_feature_filename) and tf.io.gfile.exists(
output_local_feature_filename):
# Compose output file name and decide if image should be skipped.
should_skip_global = True
should_skip_local = True
if config.use_global_features:
output_global_feature_filename = os.path.join(
FLAGS.output_features_dir, image_name + _DELG_GLOBAL_EXTENSION)
if not tf.io.gfile.exists(output_global_feature_filename):
should_skip_global = False
if config.use_local_features:
output_local_feature_filename = os.path.join(
FLAGS.output_features_dir, image_name + _DELG_LOCAL_EXTENSION)
if not tf.io.gfile.exists(output_local_feature_filename):
should_skip_local = False
if should_skip_global and should_skip_local:
print('Skipping %s' % image_name)
continue
......@@ -157,15 +169,17 @@ def main(argv):
# Extract and save features.
extracted_features = extractor_fn(im, resize_factor)
global_descriptor = extracted_features['global_descriptor']
locations = extracted_features['local_features']['locations']
descriptors = extracted_features['local_features']['descriptors']
feature_scales = extracted_features['local_features']['scales']
attention = extracted_features['local_features']['attention']
datum_io.WriteToFile(global_descriptor, output_global_feature_filename)
feature_io.WriteToFile(output_local_feature_filename, locations,
feature_scales, descriptors, attention)
if config.use_global_features:
global_descriptor = extracted_features['global_descriptor']
datum_io.WriteToFile(global_descriptor,
output_global_feature_filename)
if config.use_local_features:
locations = extracted_features['local_features']['locations']
descriptors = extracted_features['local_features']['descriptors']
feature_scales = extracted_features['local_features']['scales']
attention = extracted_features['local_features']['attention']
feature_io.WriteToFile(output_local_feature_filename, locations,
feature_scales, descriptors, attention)
if __name__ == '__main__':
......
research/delf/delf/python/google_landmarks_dataset/README.md 0 → 100644
View file @ 979e296a
## GLDv2 code/models
[![Paper](http://img.shields.io/badge/paper-arXiv.2004.01804-B3181B.svg)](https://arxiv.org/abs/2004.01804)
These instructions can be used to reproduce results from the
[GLDv2 paper](https://arxiv.org/abs/2004.01804). We present here results on the
Revisited Oxford/Paris datasets since they are smaller and quicker to
reproduce -- but note that a very similar procedure can be used to obtain
results on the GLDv2 retrieval or recognition datasets.
Note that this directory also contains code to compute GLDv2 metrics: see
`compute_retrieval_metrics.py`, `compute_recognition_metrics.py` and associated
file reading / metric computation modules.
For more details on the dataset, please refer to its
[website](https://github.com/cvdfoundation/google-landmark).
### Download Revisited Oxford/Paris datasets
```bash
mkdir -p ~/revisitop/data && cd ~/revisitop/data
# Oxford dataset.
wget http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/oxbuild_images.tgz
mkdir oxford5k_images
tar -xvzf oxbuild_images.tgz -C oxford5k_images/
# Paris dataset. Download and move all images to same directory.
wget http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/paris_1.tgz
wget http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/paris_2.tgz
mkdir paris6k_images_tmp
tar -xvzf paris_1.tgz -C paris6k_images_tmp/
tar -xvzf paris_2.tgz -C paris6k_images_tmp/
mkdir paris6k_images
mv paris6k_images_tmp/paris/*/*.jpg paris6k_images/
# Revisited annotations.
wget http://cmp.felk.cvut.cz/revisitop/data/datasets/roxford5k/gnd_roxford5k.mat
wget http://cmp.felk.cvut.cz/revisitop/data/datasets/rparis6k/gnd_rparis6k.mat
```
### Download model
```bash
# From models/research/delf/delf/python/google_landmarks_dataset
mkdir parameters && cd parameters
# RN101-ArcFace model trained on GLDv2-clean.
wget https://storage.googleapis.com/delf/rn101_af_gldv2clean_20200521.tar.gz
tar -xvzf rn101_af_gldv2clean_20200521.tar.gz
```
### Feature extraction
We present here commands for extraction on `roxford5k`. To extract on `rparis6k`
instead, please edit the arguments accordingly (especially the
`dataset_file_path` argument).
#### Query feature extraction
In the Revisited Oxford/Paris experimental protocol, query images must be the
cropped before feature extraction (this is done in the `extract_features`
script, when setting `image_set=query`). Note that this is specific to these
datasets, and not required for the GLDv2 retrieval/recognition datasets.
Run query feature extraction as follows:
```bash
# From models/research/delf/delf/python/google_landmarks_dataset
python3 ../delg/extract_features.py \
--delf_config_path rn101_af_gldv2clean_config.pbtxt \
--dataset_file_path ~/revisitop/data/gnd_roxford5k.mat \
--images_dir ~/revisitop/data/oxford5k_images \
--image_set query \
--output_features_dir ~/revisitop/data/oxford5k_features/query
```
#### Index feature extraction
Run index feature extraction as follows:
```bash
# From models/research/delf/delf/python/google_landmarks_dataset
python3 ../delg/extract_features.py \
--delf_config_path rn101_af_gldv2clean_config.pbtxt \
--dataset_file_path ~/revisitop/data/gnd_roxford5k.mat \
--images_dir ~/revisitop/data/oxford5k_images \
--image_set index \
--output_features_dir ~/revisitop/data/oxford5k_features/index
```
### Perform retrieval
To run retrieval on `roxford5k`, the following command can be used:
```bash
# From models/research/delf/delf/python/google_landmarks_dataset
python3 ../delg/perform_retrieval.py \
--dataset_file_path ~/revisitop/data/gnd_roxford5k.mat \
--query_features_dir ~/revisitop/data/oxford5k_features/query \
--index_features_dir ~/revisitop/data/oxford5k_features/index \
--output_dir ~/revisitop/results/oxford5k
```
A file with named `metrics.txt` will be written to the path given in
`output_dir`. The contents should look approximately like:
```
hard
mAP=55.54
mP@k[ 1 5 10] [88.57 80.86 70.14]
mR@k[ 1 5 10] [19.46 33.65 42.44]
medium
mAP=76.23
mP@k[ 1 5 10] [95.71 92.86 90.43]
mR@k[ 1 5 10] [10.17 25.96 35.29]
```
research/delf/delf/python/google_landmarks_dataset/rn101_af_gldv2clean_config.pbtxt 0 → 100644
View file @ 979e296a
use_local_features: false
use_global_features: true
model_path: "parameters/rn101_af_gldv2clean_20200521"
image_scales: 0.70710677
image_scales: 1.0
image_scales: 1.4142135
delf_global_config {
use_pca: false
}
max_image_size: 1024
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment