README.md 6.62 KB
Newer Older
Andre Araujo's avatar
Andre Araujo committed
1
2
3
# DELF: DEep Local Features

This project presents code for extracting DELF features, which were introduced
4
5
6
7
8
9
10
with the paper
["Large-Scale Image Retrieval with Attentive Deep Local Features"](https://arxiv.org/abs/1612.06321).
It also contains code for the follow-up paper
["Detect-to-Retrieve: Efficient Regional Aggregation for Image Search"](https://arxiv.org/abs/1812.01584).

We also released pre-trained models based on the
[Google Landmarks dataset](https://www.kaggle.com/google/google-landmarks-dataset).
Andre Araujo's avatar
Andre Araujo committed
11
12
13

DELF is particularly useful for large-scale instance-level image recognition. It
detects and describes semantic local features which can be geometrically
14
15
verified between images showing the same object instance. The pre-trained models
released here have been optimized for landmark recognition, so expect it to work
Andre Araujo's avatar
Andre Araujo committed
16
17
18
well in this area. We also provide tensorflow code for building the DELF model,
which could then be used to train models for other types of objects.

19
If you make use of this code, please consider citing the following papers:
Andre Araujo's avatar
Andre Araujo committed
20
21
22

```
"Large-Scale Image Retrieval with Attentive Deep Local Features",
23
H. Noh, A. Araujo, J. Sim, T. Weyand and B. Han,
Andre Araujo's avatar
Andre Araujo committed
24
25
26
Proc. ICCV'17
```

27
28
29
30
31
32
33
34
and/or

```
"Detect-to-Retrieve: Efficient Regional Aggregation for Image Search",
M. Teichmann*, A. Araujo*, M. Zhu and J. Sim,
Proc. CVPR'19
```

Andre Araujo's avatar
Andre Araujo committed
35
36
## News

37
38
39
40
41
42
-   [Apr'19] Check out our CVPR'19 paper:
    ["Detect-to-Retrieve: Efficient Regional Aggregation for Image Search"](https://arxiv.org/abs/1812.01584)
-   [Jun'18] DELF achieved state-of-the-art results in a CVPR'18 image retrieval
    paper: [Radenovic et al., "Revisiting Oxford and Paris: Large-Scale Image
    Retrieval Benchmarking"](https://arxiv.org/abs/1803.11285).
-   [Apr'18] DELF was featured in
Andre Araujo's avatar
Andre Araujo committed
43
    [ModelDepot](https://modeldepot.io/mikeshi/delf/overview)
44
-   [Mar'18] DELF is now available in
Andre Araujo's avatar
Andre Araujo committed
45
46
47
48
    [TF-Hub](https://www.tensorflow.org/hub/modules/google/delf/1)

## Dataset

49
50
51
52
53
54
55
56
57
58
59
60
61
62
We have two Google-Landmarks dataset versions:

-   Initial version (v1) can be found
    [here](https://www.kaggle.com/google/google-landmarks-dataset). In includes
    the Google Landmark Boxes which were described in the Detect-to-Retrieve
    paper.
-   Second version (v2) has been released as part of two Kaggle challenges:
    [Landmark Recognition](https://www.kaggle.com/c/landmark-recognition-2019)
    and [Landmark Retrieval](https://www.kaggle.com/c/landmark-retrieval-2019).
    It can be downloaded from CVDF
    [here](https://github.com/cvdfoundation/google-landmark).

If you make use of these datasets in your research, please consider citing the
papers mentioned above.
Andre Araujo's avatar
Andre Araujo committed
63

Andre Araujo's avatar
Andre Araujo committed
64
65
## Installation

66
67
68
69
70
To be able to use this code, please follow
[these instructions](INSTALL_INSTRUCTIONS.md) to properly install the DELF
library.

## Quick start
Andre Araujo's avatar
Andre Araujo committed
71

72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
### Pre-trained models

We release several pre-trained models. See instructions in the following
sections for examples on how to use the models.

**DELF pre-trained on the Google-Landmarks dataset v1**
([link](http://storage.googleapis.com/delf/delf_gld_20190411.tar.gz)). Presented
in the [CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).
Boosts performance by ~4% mAP compared to ICCV'17 DELF model.

**DELF pre-trained on Landmarks-Clean/Landmarks-Full dataset**
([link](http://storage.googleapis.com/delf/delf_v1_20171026.tar.gz)). Presented
in the [ICCV'17 DELF paper](https://arxiv.org/abs/1612.06321), model was trained
on the dataset released by the [DIR paper](https://arxiv.org/abs/1604.01325).

**Faster-RCNN detector pre-trained on Google Landmark Boxes**
([link](http://storage.googleapis.com/delf/d2r_frcnn_20190411.tar.gz)).
Presented in the
[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).

**MobileNet-SSD detector pre-trained on Google Landmark Boxes**
([link](http://storage.googleapis.com/delf/d2r_mnetssd_20190411.tar.gz)).
Presented in the
[CVPR'19 Detect-to-Retrieve paper](https://arxiv.org/abs/1812.01584).

### DELF extraction and matching
Andre Araujo's avatar
Andre Araujo committed
98
99
100
101

Please follow [these instructions](EXTRACTION_MATCHING.md). At the end, you
should obtain a nice figure showing local feature matches, as:

102
103
104
105
106
107
108
109
![MatchedImagesExample](delf/python/examples/matched_images_example.jpg)

### Landmark detection

Please follow [these instructions](DETECTION.md). At the end, you should obtain
a nice figure showing a detection, as:

![DetectionExample1](delf/python/examples/detection_example_1.jpg)
Andre Araujo's avatar
Andre Araujo committed
110

111
112
113
114
### Detect-to-Retrieve

Code release is in progress. Stay tuned!

Andre Araujo's avatar
Andre Araujo committed
115
116
## Code overview

117
DELF/D2R code is located under the `delf` directory. There are two directories
Andre Araujo's avatar
Andre Araujo committed
118
119
120
121
therein, `protos` and `python`.

### `delf/protos`

122
This directory contains protobufs:
Andre Araujo's avatar
Andre Araujo committed
123

124
-   `box.proto`: protobuf for serializing detected boxes.
Andre Araujo's avatar
Andre Araujo committed
125
126
-   `datum.proto`: general-purpose protobuf for serializing float tensors.
-   `delf_config.proto`: protobuf for configuring DELF extraction.
127
-   `feature.proto`: protobuf for serializing DELF features.
Andre Araujo's avatar
Andre Araujo committed
128
129
130
131
132

### `delf/python`

This directory contains files for several different purposes:

133
134
-   `box_io.py`, `datum_io.py`, `feature_io.py` are helper files for reading and
    writing tensors and features.
Andre Araujo's avatar
Andre Araujo committed
135
136
137
138
139
140
141
142
143
144
-   `delf_v1.py` contains the code to create DELF models.
-   `feature_extractor.py` contains the code to extract features using DELF.
    This is particularly useful for extracting features over multiple scales,
    with keypoint selection based on attention scores, and PCA/whitening
    post-processing.

Besides these, other files in this directory contain tests for different
modules.

The subdirectory `delf/python/examples` contains sample scripts to run DELF
145
feature extraction/matching, and object detection:
Andre Araujo's avatar
Andre Araujo committed
146

147
148
149
-   `delf_config_example.pbtxt` shows an example instantiation of the DelfConfig
    proto, used for DELF feature extraction.
-   `extract_boxes.py` enables object detection from a list of images.
Andre Araujo's avatar
Andre Araujo committed
150
151
152
-   `extract_features.py` enables DELF extraction from a list of images.
-   `match_images.py` supports image matching using DELF features extracted
    using `extract_features.py`.
153
154
155

The subdirectory `delf/python/detect_to_retrieve` contains sample scripts
related to the Detect-to-Retrieve paper (work in progress).
Andre Araujo's avatar
Andre Araujo committed
156
157
158
159
160
161
162

## Maintainers

André Araujo (@andrefaraujo)

## Release history

163
164
165
166
167
168
169
170
171
172
173
### April, 2019

Detect-to-Retrieve code released (work in progress).

Includes pre-trained models to detect landmark boxes, and DELF model pre-trained
on Google Landmarks v1 dataset.

**Thanks to contributors**: André Araujo, Marvin Teichmann, Menglong Zhu,
Jack Sim.

### October, 2017
Andre Araujo's avatar
Andre Araujo committed
174
175

Initial release containing DELF-v1 code, including feature extraction and
176
matching examples. Pre-trained DELF model from ICCV'17 paper is released.
Andre Araujo's avatar
Andre Araujo committed
177
178
179

**Thanks to contributors**: André Araujo, Hyeonwoo Noh, Youlong Cheng,
Jack Sim.