README.md 8.26 KB
Newer Older
1
2
3
![TensorFlow Requirement: 1.x](https://img.shields.io/badge/TensorFlow%20Requirement-1.x-brightgreen)
![TensorFlow 2 Not Supported](https://img.shields.io/badge/TensorFlow%202%20Not%20Supported-%E2%9C%95-red.svg)

Andre Araujo's avatar
Andre Araujo committed
4
5
6
# DELF: DEep Local Features

This project presents code for extracting DELF features, which were introduced
7
8
9
10
11
12
13
with the paper
["Large-Scale Image Retrieval with Attentive Deep Local Features"](https://arxiv.org/abs/1612.06321).
It also contains code for the follow-up paper
["Detect-to-Retrieve: Efficient Regional Aggregation for Image Search"](https://arxiv.org/abs/1812.01584).

We also released pre-trained models based on the
[Google Landmarks dataset](https://www.kaggle.com/google/google-landmarks-dataset).
Andre Araujo's avatar
Andre Araujo committed
14
15
16

DELF is particularly useful for large-scale instance-level image recognition. It
detects and describes semantic local features which can be geometrically
17
18
verified between images showing the same object instance. The pre-trained models
released here have been optimized for landmark recognition, so expect it to work
Andre Araujo's avatar
Andre Araujo committed
19
20
21
well in this area. We also provide tensorflow code for building the DELF model,
which could then be used to train models for other types of objects.

22
If you make use of this code, please consider citing the following papers:
Andre Araujo's avatar
Andre Araujo committed
23
24
25

```
"Large-Scale Image Retrieval with Attentive Deep Local Features",
26
H. Noh, A. Araujo, J. Sim, T. Weyand and B. Han,
Andre Araujo's avatar
Andre Araujo committed
27
28
29
Proc. ICCV'17
```

30
31
32
33
34
35
36
37
and/or

```
"Detect-to-Retrieve: Efficient Regional Aggregation for Image Search",
M. Teichmann*, A. Araujo*, M. Zhu and J. Sim,
Proc. CVPR'19
```

Andre Araujo's avatar
Andre Araujo committed
38
39
## News

40
41
42
43
44
45
-   [Apr'19] Check out our CVPR'19 paper:
    ["Detect-to-Retrieve: Efficient Regional Aggregation for Image Search"](https://arxiv.org/abs/1812.01584)
-   [Jun'18] DELF achieved state-of-the-art results in a CVPR'18 image retrieval
    paper: [Radenovic et al., "Revisiting Oxford and Paris: Large-Scale Image
    Retrieval Benchmarking"](https://arxiv.org/abs/1803.11285).
-   [Apr'18] DELF was featured in
Andre Araujo's avatar
Andre Araujo committed
46
    [ModelDepot](https://modeldepot.io/mikeshi/delf/overview)
47
-   [Mar'18] DELF is now available in
Andre Araujo's avatar
Andre Araujo committed
48
49
50
51
    [TF-Hub](https://www.tensorflow.org/hub/modules/google/delf/1)

## Dataset

52
53
54
55
56
57
58
59
60
61
62
63
64
65
We have two Google-Landmarks dataset versions:

-   Initial version (v1) can be found
    [here](https://www.kaggle.com/google/google-landmarks-dataset). In includes
    the Google Landmark Boxes which were described in the Detect-to-Retrieve
    paper.
-   Second version (v2) has been released as part of two Kaggle challenges:
    [Landmark Recognition](https://www.kaggle.com/c/landmark-recognition-2019)
    and [Landmark Retrieval](https://www.kaggle.com/c/landmark-retrieval-2019).
    It can be downloaded from CVDF
    [here](https://github.com/cvdfoundation/google-landmark).

If you make use of these datasets in your research, please consider citing the
papers mentioned above.
Andre Araujo's avatar
Andre Araujo committed
66

Andre Araujo's avatar
Andre Araujo committed
67
68
## Installation

69
70
71
72
73
To be able to use this code, please follow
[these instructions](INSTALL_INSTRUCTIONS.md) to properly install the DELF
library.

## Quick start
Andre Araujo's avatar
Andre Araujo committed
74

75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
### Pre-trained models

We release several pre-trained models. See instructions in the following
sections for examples on how to use the models.

**DELF pre-trained on the Google-Landmarks dataset v1**
([link](http://storage.googleapis.com/delf/delf_gld_20190411.tar.gz)). Presented
in the [CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
Boosts performance by ~4% mAP compared to ICCV'17 DELF model.

**DELF pre-trained on Landmarks-Clean/Landmarks-Full dataset**
([link](http://storage.googleapis.com/delf/delf_v1_20171026.tar.gz)). Presented
in the [ICCV'17 DELF paper](https://arxiv.org/abs/1612.06321), model was trained
on the dataset released by the [DIR paper](https://arxiv.org/abs/1604.01325).

**Faster-RCNN detector pre-trained on Google Landmark Boxes**
([link](http://storage.googleapis.com/delf/d2r_frcnn_20190411.tar.gz)).
Presented in the
[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).

**MobileNet-SSD detector pre-trained on Google Landmark Boxes**
([link](http://storage.googleapis.com/delf/d2r_mnetssd_20190411.tar.gz)).
Presented in the
[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).

100
101
102
103
104
Besides these, we also release pre-trained codebooks for local feature
aggregation. See the
[Detect-to-Retrieve instructions](delf/python/detect_to_retrieve/DETECT_TO_RETRIEVE_INSTRUCTIONS.md)
for details.

105
### DELF extraction and matching
Andre Araujo's avatar
Andre Araujo committed
106
107
108
109

Please follow [these instructions](EXTRACTION_MATCHING.md). At the end, you
should obtain a nice figure showing local feature matches, as:

110
111
112
113
114
115
116
117
![MatchedImagesExample](delf/python/examples/matched_images_example.jpg)

### Landmark detection

Please follow [these instructions](DETECTION.md). At the end, you should obtain
a nice figure showing a detection, as:

![DetectionExample1](delf/python/examples/detection_example_1.jpg)
Andre Araujo's avatar
Andre Araujo committed
118

119
120
### Detect-to-Retrieve

121
122
123
124
Please follow
[these instructions](delf/python/detect_to_retrieve/DETECT_TO_RETRIEVE_INSTRUCTIONS.md).
At the end, you should obtain image retrieval results on the Revisited
Oxford/Paris datasets.
125

Andre Araujo's avatar
Andre Araujo committed
126
127
## Code overview

128
DELF/D2R code is located under the `delf` directory. There are two directories
Andre Araujo's avatar
Andre Araujo committed
129
130
131
132
therein, `protos` and `python`.

### `delf/protos`

133
This directory contains protobufs:
Andre Araujo's avatar
Andre Araujo committed
134

135
136
-   `aggregation_config.proto`: protobuf for configuring local feature
    aggregation.
137
-   `box.proto`: protobuf for serializing detected boxes.
Andre Araujo's avatar
Andre Araujo committed
138
139
-   `datum.proto`: general-purpose protobuf for serializing float tensors.
-   `delf_config.proto`: protobuf for configuring DELF extraction.
140
-   `feature.proto`: protobuf for serializing DELF features.
Andre Araujo's avatar
Andre Araujo committed
141
142
143
144
145

### `delf/python`

This directory contains files for several different purposes:

146
147
-   `box_io.py`, `datum_io.py`, `feature_io.py` are helper files for reading and
    writing tensors and features.
Andre Araujo's avatar
Andre Araujo committed
148
-   `delf_v1.py` contains the code to create DELF models.
149
150
151
152
-   `feature_aggregation_extractor.py` contains a module to perform local
    feature aggregation.
-   `feature_aggregation_similarity.py` contains a module to perform similarity
    computation for aggregated local features.
Andre Araujo's avatar
Andre Araujo committed
153
154
155
156
157
158
-   `feature_extractor.py` contains the code to extract features using DELF.
    This is particularly useful for extracting features over multiple scales,
    with keypoint selection based on attention scores, and PCA/whitening
    post-processing.

The subdirectory `delf/python/examples` contains sample scripts to run DELF
159
feature extraction/matching, and object detection:
Andre Araujo's avatar
Andre Araujo committed
160

161
162
163
-   `delf_config_example.pbtxt` shows an example instantiation of the DelfConfig
    proto, used for DELF feature extraction.
-   `extract_boxes.py` enables object detection from a list of images.
Andre Araujo's avatar
Andre Araujo committed
164
165
166
-   `extract_features.py` enables DELF extraction from a list of images.
-   `match_images.py` supports image matching using DELF features extracted
    using `extract_features.py`.
167

168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
The subdirectory `delf/python/detect_to_retrieve` contains sample
scripts/configs related to the Detect-to-Retrieve paper:

-   `cluster_delf_features.py` for local feature clustering.
-   `dataset.py` for parsing/evaluating results on Revisited Oxford/Paris
    datasets.
-   `extract_aggregation.py` for aggregated local feature extraction.
-   `extract_index_boxes_and_features.py` for index image local feature
    extraction / bounding box detection on Revisited datasets.
-   `extract_query_features.py` for query image local feature extraction on
    Revisited datasets.
-   `perform_retrieval.py` for performing retrieval/evaluating methods using
    aggregated local features on Revisited datasets.
-   `delf_gld_config.pbtxt` gives the DelfConfig used in Detect-to-Retrieve
    paper.
-   `index_aggregation_config.pbtxt`, `query_aggregation_config.pbtxt` give
    AggregationConfig's for Detect-to-Retrieve experiments.

Besides these, other files in the different subdirectories contain tests for the
various modules.
Andre Araujo's avatar
Andre Araujo committed
188
189
190
191
192
193
194

## Maintainers

André Araujo (@andrefaraujo)

## Release history

195
196
### April, 2019

197
Detect-to-Retrieve code released.
198
199
200
201
202
203
204
205

Includes pre-trained models to detect landmark boxes, and DELF model pre-trained
on Google Landmarks v1 dataset.

**Thanks to contributors**: André Araujo, Marvin Teichmann, Menglong Zhu,
Jack Sim.

### October, 2017
Andre Araujo's avatar
Andre Araujo committed
206
207

Initial release containing DELF-v1 code, including feature extraction and
208
matching examples. Pre-trained DELF model from ICCV'17 paper is released.
Andre Araujo's avatar
Andre Araujo committed
209
210
211

**Thanks to contributors**: André Araujo, Hyeonwoo Noh, Youlong Cheng,
Jack Sim.