Skip to content

GitLab

Unverified Commit c627506f authored by André Araujo's avatar André Araujo Committed by GitHub
Browse files

DELF open-source library v2.0 (#8454) 

* Merged commit includes the following changes:
253126424  by Andre Araujo:

    Scripts to compute metrics for Google Landmarks dataset.

    Also, a small fix to metric in retrieval case: avoids duplicate predicted images.

--
253118971  by Andre Araujo:

    Metrics for Google Landmarks dataset.

--
253106953  by Andre Araujo:

    Library to read files from Google Landmarks challenges.

--
250700636  by Andre Araujo:

    Handle case of aggregation extraction with empty set of input features.

--
250516819  by Andre Araujo:

    Add minimum size for DELF extractor.

--
250435822  by Andre Araujo:

    Add max_image_size/min_image_size for open-source DELF proto / module.

--
250414606  by Andre Araujo:

    Refactor extract_aggregation to allow reuse with different datasets.

--
250356863  by Andre Araujo:

    Remove unnecessary cmd_args variable from boxes_and_features_extraction.

--
249783379  by Andre Araujo:

    Create directory for writing mapping file if it does not exist.

--
249581591  by Andre Araujo:

    Refactor scripts to extract boxes and features from images in Revisited datasets.
    Also, change tf.logging.info --> print for easier logging in open source code.

--
249511821  by Andre Araujo:

    Small change to function for file/directory handling.

--
249289499  by Andre Araujo:

    Internal change.

--

PiperOrigin-RevId: 253126424

* Updating DELF init to adjust to latest changes

* Editing init files for python packages

* Edit D2R dataset reader to work with py3.

PiperOrigin-RevId: 253135576

* DELF package: fix import ordering

* Adding new requirements to setup.py

* Adding init file for training dir

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* Adding init file for training subdirs

* Working version of DELF training

* Internal change.

PiperOrigin-RevId: 253248648

* Fix variance loading in open-source code.

PiperOrigin-RevId: 260619120

* Separate image re-ranking as a standalone library, and add metric writing to dataset library.

PiperOrigin-RevId: 260998608

* Tool to read written D2R Revisited datasets metrics file. Test is added.

Also adds a unit test for previously-existing SaveMetricsFile function.

PiperOrigin-RevId: 263361410

* Add optional resize factor for feature extraction.

PiperOrigin-RevId: 264437080

* Fix NumPy's new version spacing changes.

PiperOrigin-RevId: 265127245

* Maker image matching function visible, and add support for RANSAC seed.

PiperOrigin-RevId: 277177468

* Avoid matplotlib failure due to missing display backend.

PiperOrigin-RevId: 287316435

* Removes tf.contrib dependency.

PiperOrigin-RevId: 288842237

* Fix tf contrib removal for feature_aggregation_extractor.

PiperOrigin-RevId: 289487669

* Merged commit includes the following changes:
309118395  by Andre Araujo:

    Make DELF open-source code compatible with TF2.

--
309067582  by Andre Araujo:

    Handle image resizing rounding properly for python extraction.

    New behavior is tested with unit tests.

--
308690144  by Andre Araujo:

    Several changes to improve DELF model/training code and make it work in TF 2.1.0:
    - Rename some files for better clarity
    - Using compat.v1 versions of functions
    - Formatting changes
    - Using more appropriate TF function names

--
308689397  by Andre Araujo:

    Internal change.

--
308341315  by Andre Araujo:

    Remove old slim dependency in DELF open-source model.

    This avoids issues with requiring old TF-v1, making it compatible with latest TF.

--
306777559  by Andre Araujo:

    Internal change

--
304505811  by Andre Araujo:

    Raise error during geometric verification if local features have different dimensionalities.

--
301739992  by Andre Araujo:

    Transform some geometric verification constants into arguments, to allow custom matching.

--
301300324  by Andre Araujo:

    Apply name change(experimental_run_v2 -> run) for all callers in Tensorflow.

--
299919057  by Andre Araujo:

    Automated refactoring to make code Python 3 compatible.

--
297953698  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297521242  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297278247  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297270405  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297238741  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
297108605  by Andre Araujo:

    Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration

--
294676131  by Andre Araujo:

    Add option to resize images to square resolutions without aspect ratio preservation.

--
293849641  by Andre Araujo:

    Internal change.

--
293840896  by Andre Araujo:

    Changing Slim import to tf_slim codebase.

--
293661660  by Andre Araujo:

    Allow the delf training script to read from TFRecords dataset.

--
291755295  by Andre Araujo:

    Internal change.

--
291448508  by Andre Araujo:

    Internal change.

--
291414459  by Andre Araujo:

    Adding train script.

--
291384336  by Andre Araujo:

    Adding model export script and test.

--
291260565  by Andre Araujo:

    Adding placeholder for Google Landmarks dataset.

--
291205548  by Andre Araujo:

    Definition of DELF model using Keras ResNet50 as backbone.

--
289500793  by Andre Araujo:

    Add TFRecord building script for delf.

--

PiperOrigin-RevId: 309118395

* Updating README, dependency versions

* Updating training README

* Fixing init import of export_model

* Fixing init import of export_model_utils

* tkinter in INSTALL_INSTRUCTIONS

* Merged commit includes the following changes:

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

* INSTALL_INSTRUCTIONS mentioning different cloning options
parent 71d2680d
Hide whitespace changes
Inline Side-by-side
Showing with 724 additions and 345 deletions
research/delf/INSTALL_INSTRUCTIONS.md
View file @ c627506f
......@@ -2,17 +2,34 @@
### Tensorflow
For detailed steps to install Tensorflow, follow the [Tensorflow installation
instructions](https://www.tensorflow.org/install/). A typical user can install
Tensorflow using one of the following commands:
For detailed steps to install Tensorflow, follow the
[Tensorflow installation instructions](https://www.tensorflow.org/install/). A
typical user can install Tensorflow using one of the following commands:
```bash
# For CPU:
pip install 'tensorflow==1.14'
pip install 'tensorflow'
# For GPU:
pip install 'tensorflow-gpu==1.14'
pip install 'tensorflow-gpu'
```
### TF-Slim
Note: currently, we need to install the latest version from source, to avoid
using previous versions which relied on tf.contrib (which is now deprecated).
```bash
git clone git@github.com:google-research/tf-slim.git
cd tf-slim
pip install .
```
Note that these commands assume you are cloning using SSH. If you are using
HTTPS instead, use `git clone https://github.com/google-research/tf-slim.git`
instead. See
[this link](https://help.github.com/en/github/using-git/which-remote-url-should-i-use)
for more information.
### Protobuf
The DELF library uses [protobuf](https://github.com/google/protobuf) (the python
......@@ -33,6 +50,7 @@ Install python library dependencies:
```bash
pip install matplotlib numpy scikit-image scipy
sudo apt-get install python-tk
```
### `tensorflow/models`
......@@ -43,18 +61,20 @@ your `PYTHONPATH`, as instructed
[here](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md))
```bash
git clone https://github.com/tensorflow/models
# First, install slim's "nets" package.
cd models/research/slim/
pip install -e .
git clone git@github.com:tensorflow/models.git
# Second, setup the object_detection module by editing PYTHONPATH.
# Setup the object_detection module by editing PYTHONPATH.
cd ..
# From tensorflow/models/research/
export PYTHONPATH=$PYTHONPATH:`pwd`
```
Note that these commands assume you are cloning using SSH. If you are using
HTTPS instead, use `git clone https://github.com/tensorflow/models.git` instead.
See
[this link](https://help.github.com/en/github/using-git/which-remote-url-should-i-use)
for more information.
Then, compile DELF's protobufs. Use `PATH_TO_PROTOC` as the directory where you
downloaded the `protoc` compiler.
......@@ -63,7 +83,8 @@ downloaded the `protoc` compiler.
${PATH_TO_PROTOC?}/bin/protoc delf/protos/*.proto --python_out=.
```
Finally, install the DELF package.
Finally, install the DELF package. This may also install some other dependencies
under the hood.
```bash
# From tensorflow/models/research/delf/
......@@ -85,11 +106,21 @@ loaded successfully.
Installation issues may happen if multiple python versions are mixed. The
instructions above assume python2.7 version is used; if using python3.X, be sure
to use `pip3` instead of `pip`, and all should work.
to use `pip3` instead of `pip`, `python3-tk` instead of `python-tk`, and all
should work.
#### `pip install`
Issues might be observed if using `pip install` with `-e` option (editable
mode). You may try out to simply remove the `-e` from the commands above. Also,
depending on your machine setup, you might need to run the `sudo pip install` command,
that is with a `sudo` at the beginning.
depending on your machine setup, you might need to run the `sudo pip install`
command, that is with a `sudo` at the beginning.
#### Cloning github repositories
The default commands above assume you are cloning using SSH. If you are using
HTTPS instead, use for example `git clone
https://github.com/tensorflow/models.git` instead of `git clone
git@github.com:tensorflow/models.git`. See
[this link](https://help.github.com/en/github/using-git/which-remote-url-should-i-use)
for more information.
research/delf/README.md
View file @ c627506f
![TensorFlow Requirement: 1.x](https://img.shields.io/badge/TensorFlow%20Requirement-1.x-brightgreen)
![TensorFlow 2 Not Supported](https://img.shields.io/badge/TensorFlow%202%20Not%20Supported-%E2%9C%95-red.svg)
# DELF: DEep Local Features
This project presents code for extracting DELF features, which were introduced
......@@ -17,7 +14,7 @@ detects and describes semantic local features which can be geometrically
verified between images showing the same object instance. The pre-trained models
released here have been optimized for landmark recognition, so expect it to work
well in this area. We also provide tensorflow code for building the DELF model,
which could then be used to train models for other types of objects.
and [NEW] code for model training.
If you make use of this code, please consider citing the following papers:
......@@ -37,6 +34,10 @@ Proc. CVPR'19
## News
- [Jun'19] DELF achieved 2nd place in
[CVPR Visual Localization challenge (Local Features track)](https://sites.google.com/corp/view/ltvl2019).
See our slides
[here](https://docs.google.com/presentation/d/e/2PACX-1vTswzoXelqFqI_pCEIVl2uazeyGr7aKNklWHQCX-CbQ7MB17gaycqIaDTguuUCRm6_lXHwCdrkP7n1x/pub?start=false&loop=false&delayms=3000).
- [Apr'19] Check out our CVPR'19 paper:
["Detect-to-Retrieve: Efficient Regional Aggregation for Image Search"](https://arxiv.org/abs/1812.01584)
- [Jun'18] DELF achieved state-of-the-art results in a CVPR'18 image retrieval
......@@ -59,7 +60,9 @@ We have two Google-Landmarks dataset versions:
[Landmark Recognition](https://www.kaggle.com/c/landmark-recognition-2019)
and [Landmark Retrieval](https://www.kaggle.com/c/landmark-retrieval-2019).
It can be downloaded from CVDF
[here](https://github.com/cvdfoundation/google-landmark).
[here](https://github.com/cvdfoundation/google-landmark). See also
[the CVPR'20 paper](https://arxiv.org/abs/2004.01804) on this new dataset
version.
If you make use of these datasets in your research, please consider citing the
papers mentioned above.
......@@ -109,6 +112,10 @@ should obtain a nice figure showing local feature matches, as:
![MatchedImagesExample](delf/python/examples/matched_images_example.jpg)
### DELF training
Please follow [these instructions](delf/python/training/README.md).
### Landmark detection
Please follow [these instructions](DETECTION.md). At the end, you should obtain
......@@ -145,7 +152,7 @@ This directory contains files for several different purposes:
- `box_io.py`, `datum_io.py`, `feature_io.py` are helper files for reading and
writing tensors and features.
- `delf_v1.py` contains the code to create DELF models.
- `delf_v1.py` contains code to create DELF models.
- `feature_aggregation_extractor.py` contains a module to perform local
feature aggregation.
- `feature_aggregation_similarity.py` contains a module to perform similarity
......@@ -160,29 +167,62 @@ feature extraction/matching, and object detection:
- `delf_config_example.pbtxt` shows an example instantiation of the DelfConfig
proto, used for DELF feature extraction.
- `detector.py` is a module to construct an object detector function.
- `extract_boxes.py` enables object detection from a list of images.
- `extract_features.py` enables DELF extraction from a list of images.
- `extractor.py` is a module to construct a DELF local feature extraction
function.
- `match_images.py` supports image matching using DELF features extracted
using `extract_features.py`.
The subdirectory `delf/python/detect_to_retrieve` contains sample
scripts/configs related to the Detect-to-Retrieve paper:
- `aggregation_extraction.py` is a library to extract/save feature
aggregation.
- `boxes_and_features_extraction.py` is a library to extract/save boxes and
DELF features.
- `cluster_delf_features.py` for local feature clustering.
- `dataset.py` for parsing/evaluating results on Revisited Oxford/Paris
datasets.
- `delf_gld_config.pbtxt` gives the DelfConfig used in Detect-to-Retrieve
paper.
- `extract_aggregation.py` for aggregated local feature extraction.
- `extract_index_boxes_and_features.py` for index image local feature
extraction / bounding box detection on Revisited datasets.
- `extract_query_features.py` for query image local feature extraction on
Revisited datasets.
- `image_reranking.py` is a module to re-rank images with geometric
verification.
- `perform_retrieval.py` for performing retrieval/evaluating methods using
aggregated local features on Revisited datasets.
- `delf_gld_config.pbtxt` gives the DelfConfig used in Detect-to-Retrieve
paper.
- `index_aggregation_config.pbtxt`, `query_aggregation_config.pbtxt` give
AggregationConfig's for Detect-to-Retrieve experiments.
The subdirectory `delf/python/google_landmarks_dataset` contains sample
scripts/modules for computing GLD metrics:
- `compute_recognition_metrics.py` performs recognition metric computation
given input predictions and solution files.
- `compute_retrieval_metrics.py` performs retrieval metric computation given
input predictions and solution files.
- `dataset_file_io.py` is a module for dataset-related file IO.
- `metrics.py` is a module for GLD metric computation.
The subdirectory `delf/python/training` contains sample scripts/modules for
performing DELF training:
- `datasets/googlelandmarks.py` is the dataset module used for training.
- `model/delf_model.py` is the model module used for training.
- `model/export_model.py` is a script for exporting trained models in the
format used by the inference code.
- `model/export_model_utils.py` is a module with utilities for model
exporting.
- `model/resnet50.py` is a module with a backbone RN50 implementation.
- `build_image_dataset.py` converts downloaded dataset into TFRecords format
for training.
- `train.py` is the main training script.
Besides these, other files in the different subdirectories contain tests for the
various modules.
......@@ -192,6 +232,13 @@ André Araujo (@andrefaraujo)
## Release history
### April, 2020 (version 2.0)
- Initial DELF training code released.
- Codebase is now fully compatible with TF 2.1.
**Thanks to contributors**: Arun Mukundan, Yuewei Na and André Araujo.
### April, 2019
Detect-to-Retrieve code released.
......
research/delf/delf/__init__.py
View file @ c627506f
......@@ -33,5 +33,7 @@ from delf.python import feature_io
from delf.python.examples import detector
from delf.python.examples import extractor
from delf.python import detect_to_retrieve
from delf.python import google_landmarks_dataset
from delf.python import training
from delf.python.training import model
from delf.python.training import datasets
# pylint: enable=unused-import
research/delf/delf/python/box_io.py
View file @ c627506f
......@@ -132,7 +132,7 @@ def ReadFromFile(file_path):
scores: [N] float array with detection scores.
class_indices: [N] int array with class indices.
"""
with tf.gfile.GFile(file_path, 'rb') as f:
with tf.io.gfile.GFile(file_path, 'rb') as f:
return ParseFromString(f.read())
......@@ -147,5 +147,5 @@ def WriteToFile(file_path, boxes, scores, class_indices):
class_indices: [N] int array with class indices.
"""
serialized_data = SerializeToString(boxes, scores, class_indices)
with tf.gfile.GFile(file_path, 'w') as f:
with tf.io.gfile.GFile(file_path, 'w') as f:
f.write(serialized_data)
research/delf/delf/python/box_io_test.py
View file @ c627506f
......@@ -57,7 +57,7 @@ class BoxesIoTest(tf.test.TestCase):
def testWriteAndReadToFile(self):
boxes, scores, class_indices = self._create_data()
tmpdir = tf.test.get_temp_dir()
tmpdir = tf.compat.v1.test.get_temp_dir()
filename = os.path.join(tmpdir, 'test.boxes')
box_io.WriteToFile(filename, boxes, scores, class_indices)
data_read = box_io.ReadFromFile(filename)
......@@ -67,7 +67,7 @@ class BoxesIoTest(tf.test.TestCase):
self.assertAllEqual(class_indices, data_read[2])
def testWriteAndReadToFileEmptyFile(self):
tmpdir = tf.test.get_temp_dir()
tmpdir = tf.compat.v1.test.get_temp_dir()
filename = os.path.join(tmpdir, 'test.box')
box_io.WriteToFile(filename, np.array([]), np.array([]), np.array([]))
data_read = box_io.ReadFromFile(filename)
......
research/delf/delf/python/datum_io.py
View file @ c627506f
......@@ -179,7 +179,7 @@ def ReadFromFile(file_path):
Returns:
data: NumPy array.
"""
with tf.gfile.GFile(file_path, 'rb') as f:
with tf.io.gfile.GFile(file_path, 'rb') as f:
return ParseFromString(f.read())
......@@ -192,7 +192,7 @@ def ReadPairFromFile(file_path):
Returns:
Two NumPy arrays.
"""
with tf.gfile.GFile(file_path, 'rb') as f:
with tf.io.gfile.GFile(file_path, 'rb') as f:
return ParsePairFromString(f.read())
......@@ -204,7 +204,7 @@ def WriteToFile(data, file_path):
file_path: Path to file that will be written.
"""
serialized_data = SerializeToString(data)
with tf.gfile.GFile(file_path, 'w') as f:
with tf.io.gfile.GFile(file_path, 'w') as f:
f.write(serialized_data)
......@@ -217,5 +217,5 @@ def WritePairToFile(arr_1, arr_2, file_path):
file_path: Path to file that will be written.
"""
serialized_data = SerializePairToString(arr_1, arr_2)
with tf.gfile.GFile(file_path, 'w') as f:
with tf.io.gfile.GFile(file_path, 'w') as f:
f.write(serialized_data)
research/delf/delf/python/datum_io_test.py
View file @ c627506f
......@@ -69,7 +69,7 @@ class DatumIoTest(tf.test.TestCase):
def testWriteAndReadToFile(self):
data = np.array([[[-1.0, 125.0, -2.5], [14.5, 3.5, 0.0]],
[[20.0, 0.0, 30.0], [25.5, 36.0, 42.0]]])
tmpdir = tf.test.get_temp_dir()
tmpdir = tf.compat.v1.test.get_temp_dir()
filename = os.path.join(tmpdir, 'test.datum')
datum_io.WriteToFile(data, filename)
data_read = datum_io.ReadFromFile(filename)
......@@ -84,7 +84,7 @@ class DatumIoTest(tf.test.TestCase):
data_2 = np.array(
[[[255, 0, 5], [10, 300, 0]], [[20, 1, 100], [255, 360, 420]]],
dtype='uint32')
tmpdir = tf.test.get_temp_dir()
tmpdir = tf.compat.v1.test.get_temp_dir()
filename = os.path.join(tmpdir, 'test.datum_pair')
datum_io.WritePairToFile(data_1, data_2, filename)
data_read_1, data_read_2 = datum_io.ReadPairFromFile(filename)
......
research/delf/delf/python/delf_v1.py
View file @ c627506f
......@@ -26,10 +26,9 @@ from __future__ import division
from __future__ import print_function
import tensorflow as tf
from nets import resnet_v1
slim = tf.contrib.slim
from tf_slim import layers
from tf_slim.nets import resnet_v1
from tf_slim.ops.arg_scope import arg_scope
_SUPPORTED_TARGET_LAYER = ['resnet_v1_50/block3', 'resnet_v1_50/block4']
......@@ -68,7 +67,7 @@ class DelfV1(object):
"""
def __init__(self, target_layer_type=_SUPPORTED_TARGET_LAYER[0]):
tf.logging.info('Creating model %s ', target_layer_type)
tf.compat.v1.logging.info('Creating model %s ', target_layer_type)
self._target_layer_type = target_layer_type
if self._target_layer_type not in _SUPPORTED_TARGET_LAYER:
......@@ -89,8 +88,8 @@ class DelfV1(object):
the attention score map.
Args:
attention_feature_map: Potentially normalized feature map that will
be aggregated with attention score map.
attention_feature_map: Potentially normalized feature map that will be
aggregated with attention score map.
feature_map: Unnormalized feature map that will be used to compute
attention score map.
attention_nonlinear: Type of non-linearity that will be applied to
......@@ -105,11 +104,11 @@ class DelfV1(object):
Raises:
ValueError: If unknown attention non-linearity type is provided.
"""
with tf.variable_scope(
with tf.compat.v1.variable_scope(
'attention', values=[attention_feature_map, feature_map]):
with tf.variable_scope('compute', values=[feature_map]):
with tf.compat.v1.variable_scope('compute', values=[feature_map]):
activation_fn_conv1 = tf.nn.relu
feature_map_conv1 = slim.conv2d(
feature_map_conv1 = layers.conv2d(
feature_map,
512,
kernel,
......@@ -117,7 +116,7 @@ class DelfV1(object):
activation_fn=activation_fn_conv1,
scope='conv1')
attention_score = slim.conv2d(
attention_score = layers.conv2d(
feature_map_conv1,
1,
kernel,
......@@ -127,12 +126,12 @@ class DelfV1(object):
scope='conv2')
# Set activation of conv2 layer of attention model.
with tf.variable_scope(
with tf.compat.v1.variable_scope(
'merge', values=[attention_feature_map, attention_score]):
if attention_nonlinear not in _SUPPORTED_ATTENTION_NONLINEARITY:
raise ValueError('Unknown attention non-linearity.')
if attention_nonlinear == 'softplus':
with tf.variable_scope(
with tf.compat.v1.variable_scope(
'softplus_attention',
values=[attention_feature_map, attention_score]):
attention_prob = tf.nn.softplus(attention_score)
......@@ -169,7 +168,7 @@ class DelfV1(object):
Raises:
ValueError: If unknown attention_type is provided.
"""
with tf.variable_scope(
with tf.compat.v1.variable_scope(
_ATTENTION_VARIABLE_SCOPE,
values=[feature_map, end_points],
reuse=reuse):
......@@ -182,8 +181,9 @@ class DelfV1(object):
attention_feature_map = feature_map
end_points['attention_feature_map'] = attention_feature_map
attention_outputs = self._PerformAttention(
attention_feature_map, feature_map, attention_nonlinear, kernel)
attention_outputs = self._PerformAttention(attention_feature_map,
feature_map,
attention_nonlinear, kernel)
prelogits, attention_prob, attention_score = attention_outputs
end_points['prelogits'] = prelogits
end_points['attention_prob'] = attention_prob
......@@ -248,8 +248,8 @@ class DelfV1(object):
kernel: Convolutional kernel to use in attention layers (eg, [3, 3]).
training_resnet: Whether or not the Resnet blocks from the model are in
training mode.
training_attention: Whether or not the attention part of the model is
in training mode.
training_attention: Whether or not the attention part of the model is in
training mode.
reuse: Whether or not the layer and its variables should be reused.
use_batch_norm: Whether or not to use batch normalization.
......@@ -262,18 +262,17 @@ class DelfV1(object):
end_points: Set of activations for external use.
"""
# Construct Resnet50 features.
with slim.arg_scope(
resnet_v1.resnet_arg_scope(use_batch_norm=use_batch_norm)):
with arg_scope(resnet_v1.resnet_arg_scope(use_batch_norm=use_batch_norm)):
_, end_points = self.GetResnet50Subnetwork(
images, is_training=training_resnet, reuse=reuse)
feature_map = end_points[self._target_layer_type]
# Construct attention subnetwork on top of features.
with slim.arg_scope(
with arg_scope(
resnet_v1.resnet_arg_scope(
weight_decay=weight_decay, use_batch_norm=use_batch_norm)):
with slim.arg_scope([slim.batch_norm], is_training=training_attention):
with arg_scope([layers.batch_norm], is_training=training_attention):
(prelogits, attention_prob, attention_score,
end_points) = self._GetAttentionSubnetwork(
feature_map,
......@@ -330,13 +329,13 @@ class DelfV1(object):
training_resnet=training_resnet,
training_attention=training_attention,
reuse=reuse))
with slim.arg_scope(
with arg_scope(
resnet_v1.resnet_arg_scope(
weight_decay=weight_decay, batch_norm_scale=True)):
with slim.arg_scope([slim.batch_norm], is_training=training_attention):
with tf.variable_scope(
with arg_scope([layers.batch_norm], is_training=training_attention):
with tf.compat.v1.variable_scope(
_ATTENTION_VARIABLE_SCOPE, values=[attention_feat], reuse=reuse):
logits = slim.conv2d(
logits = layers.conv2d(
attention_feat,
num_classes, [1, 1],
activation_fn=None,
......
research/delf/delf/python/detect_to_retrieve/aggregation_extraction.py
View file @ c627506f
......@@ -59,7 +59,7 @@ def _ReadMappingBasenameToBoxNames(input_path, index_image_names):
strings (file names containing DELF features for boxes).
"""
images_to_box_feature_files = {}
with tf.gfile.GFile(input_path, 'r') as f:
with tf.io.gfile.GFile(input_path, 'r') as f:
reader = csv.DictReader(f)
for row in reader:
index_image_name = index_image_names[int(row['index_image_id'])]
......@@ -101,7 +101,7 @@ def ExtractAggregatedRepresentationsToFiles(image_names, features_dir,
# Parse AggregationConfig proto, and select output extension.
config = aggregation_config_pb2.AggregationConfig()
with tf.gfile.GFile(aggregation_config_path, 'r') as f:
with tf.io.gfile.GFile(aggregation_config_path, 'r') as f:
text_format.Merge(f.read(), config)
output_extension = '.'
if config.use_regional_aggregation:
......@@ -121,10 +121,10 @@ def ExtractAggregatedRepresentationsToFiles(image_names, features_dir,
mapping_path, image_names)
# Create output directory if necessary.
if not tf.gfile.Exists(output_aggregation_dir):
tf.gfile.MakeDirs(output_aggregation_dir)
if not tf.io.gfile.exists(output_aggregation_dir):
tf.io.gfile.makedirs(output_aggregation_dir)
with tf.Session() as sess:
with tf.compat.v1.Session() as sess:
extractor = feature_aggregation_extractor.ExtractAggregatedRepresentation(
sess, config)
......
research/delf/delf/python/detect_to_retrieve/boxes_and_features_extraction.py
View file @ c627506f
......@@ -55,7 +55,7 @@ def _PilLoader(path):
Returns:
PIL image in RGB format.
"""
with tf.gfile.GFile(path, 'rb') as f:
with tf.io.gfile.GFile(path, 'rb') as f:
img = Image.open(f)
return img.convert('RGB')
......@@ -68,7 +68,7 @@ def _WriteMappingBasenameToIds(index_names_ids_and_boxes, output_path):
ID and box ID.
output_path: Output CSV path.
"""
with tf.gfile.GFile(output_path, 'w') as f:
with tf.io.gfile.GFile(output_path, 'w') as f:
csv_writer = csv.DictWriter(
f, fieldnames=['name', 'index_image_id', 'box_id'])
csv_writer.writeheader()
......@@ -118,22 +118,22 @@ def ExtractBoxesAndFeaturesToFiles(image_names, image_paths, delf_config_path,
# Parse DelfConfig proto.
config = delf_config_pb2.DelfConfig()
with tf.gfile.GFile(delf_config_path, 'r') as f:
with tf.io.gfile.GFile(delf_config_path, 'r') as f:
text_format.Merge(f.read(), config)
# Create output directories if necessary.
if not tf.gfile.Exists(output_features_dir):
tf.gfile.MakeDirs(output_features_dir)
if not tf.gfile.Exists(output_boxes_dir):
tf.gfile.MakeDirs(output_boxes_dir)
if not tf.gfile.Exists(os.path.dirname(output_mapping)):
tf.gfile.MakeDirs(os.path.dirname(output_mapping))
if not tf.io.gfile.exists(output_features_dir):
tf.io.gfile.makedirs(output_features_dir)
if not tf.io.gfile.exists(output_boxes_dir):
tf.io.gfile.makedirs(output_boxes_dir)
if not tf.io.gfile.exists(os.path.dirname(output_mapping)):
tf.io.gfile.makedirs(os.path.dirname(output_mapping))
names_ids_and_boxes = []
with tf.Graph().as_default():
with tf.Session() as sess:
with tf.compat.v1.Session() as sess:
# Initialize variables, construct detector and DELF extractor.
init_op = tf.global_variables_initializer()
init_op = tf.compat.v1.global_variables_initializer()
sess.run(init_op)
detector_fn = detector.MakeDetector(
sess, detector_model_dir, import_scope='detector')
......@@ -161,7 +161,7 @@ def ExtractBoxesAndFeaturesToFiles(image_names, image_paths, delf_config_path,
width, height = pil_im.size
# Extract and save boxes.
if tf.gfile.Exists(output_box_filename):
if tf.io.gfile.exists(output_box_filename):
print('Skipping box computation for %s' % image_name)
(boxes_out, scores_out,
class_indices_out) = box_io.ReadFromFile(output_box_filename)
......@@ -197,7 +197,7 @@ def ExtractBoxesAndFeaturesToFiles(image_names, image_paths, delf_config_path,
names_ids_and_boxes.append([box_name, i, delf_file_ind - 1])
if tf.gfile.Exists(output_feature_filename):
if tf.io.gfile.exists(output_feature_filename):
print('Skipping DELF computation for %s' % box_name)
continue
......
research/delf/delf/python/detect_to_retrieve/cluster_delf_features.py
View file @ c627506f
......@@ -52,7 +52,7 @@ _DELF_DIM = 128
_STATUS_CHECK_ITERATIONS = 100
class _IteratorInitHook(tf.train.SessionRunHook):
class _IteratorInitHook(tf.estimator.SessionRunHook):
"""Hook to initialize data iterator after session is created."""
def __init__(self):
......@@ -70,14 +70,14 @@ def main(argv):
raise RuntimeError('Too many command-line arguments.')
# Process output directory.
if tf.gfile.Exists(cmd_args.output_cluster_dir):
if tf.io.gfile.exists(cmd_args.output_cluster_dir):
raise RuntimeError(
'output_cluster_dir = %s already exists. This may indicate that a '
'previous run already wrote checkpoints in this directory, which would '
'lead to incorrect training. Please re-run this script by specifying an'
' inexisting directory.' % cmd_args.output_cluster_dir)
else:
tf.gfile.MakeDirs(cmd_args.output_cluster_dir)
tf.io.gfile.makedirs(cmd_args.output_cluster_dir)
# Read list of index images from dataset file.
print('Reading list of index images from dataset file...')
......@@ -126,8 +126,8 @@ def main(argv):
Returns:
Tensor with the data for training.
"""
features_placeholder = tf.placeholder(tf.float32,
features_for_clustering.shape)
features_placeholder = tf.compat.v1.placeholder(
tf.float32, features_for_clustering.shape)
delf_dataset = tf.data.Dataset.from_tensor_slices((features_placeholder))
delf_dataset = delf_dataset.shuffle(1000).batch(
features_for_clustering.shape[0])
......@@ -146,7 +146,7 @@ def main(argv):
input_fn, init_hook = _get_input_fn()
kmeans = tf.estimator.experimental.KMeans(
kmeans = tf.compat.v1.estimator.experimental.KMeans(
num_clusters=cmd_args.num_clusters,
model_dir=cmd_args.output_cluster_dir,
use_mini_batch=False,
......
research/delf/delf/python/detect_to_retrieve/dataset.py
View file @ c627506f
......@@ -40,7 +40,7 @@ def ReadDatasetFile(dataset_file_path):
array of integers; additionally, it has a key 'bbx' mapping to a NumPy
array of floats with bounding box coordinates.
"""
with tf.gfile.GFile(dataset_file_path, 'rb') as f:
with tf.io.gfile.GFile(dataset_file_path, 'rb') as f:
cfg = matlab.loadmat(f)
# Parse outputs according to the specificities of the dataset file.
......@@ -314,3 +314,156 @@ def ComputeMetrics(sorted_index_ids, ground_truth, desired_pr_ranks):
return (mean_average_precision, mean_precisions, mean_recalls,
average_precisions, precisions, recalls)
def SaveMetricsFile(mean_average_precision, mean_precisions, mean_recalls,
pr_ranks, output_path):
"""Saves aggregated retrieval metrics to text file.
Args:
mean_average_precision: Dict mapping each dataset protocol to a float.
mean_precisions: Dict mapping each dataset protocol to a NumPy array of
floats with shape [len(pr_ranks)].
mean_recalls: Dict mapping each dataset protocol to a NumPy array of floats
with shape [len(pr_ranks)].
pr_ranks: List of integers.
output_path: Full file path.
"""
with tf.io.gfile.GFile(output_path, 'w') as f:
for k in sorted(mean_average_precision.keys()):
f.write('{}\n mAP={}\n mP@k{} {}\n mR@k{} {}\n'.format(
k, np.around(mean_average_precision[k] * 100, decimals=2),
np.array(pr_ranks), np.around(mean_precisions[k] * 100, decimals=2),
np.array(pr_ranks), np.around(mean_recalls[k] * 100, decimals=2)))
def _ParseSpaceSeparatedStringsInBrackets(line, prefixes, ind):
"""Parses line containing space-separated strings in brackets.
Args:
line: String, containing line in metrics file with mP@k or mR@k figures.
prefixes: Tuple/list of strings, containing valid prefixes.
ind: Integer indicating which field within brackets is parsed.
Yields:
entry: String format entry.
Raises:
ValueError: If input line does not contain a valid prefix.
"""
for prefix in prefixes:
if line.startswith(prefix):
line = line[len(prefix):]
break
else:
raise ValueError('Line %s is malformed, cannot find valid prefixes' % line)
for entry in line.split('[')[ind].split(']')[0].split():
yield entry
def _ParsePrRanks(line):
"""Parses PR ranks from mP@k line in metrics file.
Args:
line: String, containing line in metrics file with mP@k figures.
Returns:
pr_ranks: List of integers, containing used ranks.
Raises:
ValueError: If input line is malformed.
"""
return [
int(pr_rank) for pr_rank in _ParseSpaceSeparatedStringsInBrackets(
line, [' mP@k['], 0) if pr_rank
]
def _ParsePrScores(line, num_pr_ranks):
"""Parses PR scores from line in metrics file.
Args:
line: String, containing line in metrics file with mP@k or mR@k figures.
num_pr_ranks: Integer, number of scores that should be in output list.
Returns:
pr_scores: List of floats, containing scores.
Raises:
ValueError: If input line is malformed.
"""
pr_scores = [
float(pr_score) for pr_score in _ParseSpaceSeparatedStringsInBrackets(
line, (' mP@k[', ' mR@k['), 1) if pr_score
]
if len(pr_scores) != num_pr_ranks:
raise ValueError('Line %s is malformed, expected %d scores but found %d' %
(line, num_pr_ranks, len(pr_scores)))
return pr_scores
def ReadMetricsFile(metrics_path):
"""Reads aggregated retrieval metrics from text file.
Args:
metrics_path: Full file path, containing aggregated retrieval metrics.
Returns:
mean_average_precision: Dict mapping each dataset protocol to a float.
pr_ranks: List of integer ranks used in aggregated recall/precision metrics.
mean_precisions: Dict mapping each dataset protocol to a NumPy array of
floats with shape [len(`pr_ranks`)].
mean_recalls: Dict mapping each dataset protocol to a NumPy array of floats
with shape [len(`pr_ranks`)].
Raises:
ValueError: If input file is malformed.
"""
with tf.io.gfile.GFile(metrics_path, 'r') as f:
file_contents_stripped = [l.rstrip() for l in f]
if len(file_contents_stripped) % 4:
raise ValueError(
'Malformed input %s: number of lines must be a multiple of 4, '
'but it is %d' % (metrics_path, len(file_contents_stripped)))
mean_average_precision = {}
pr_ranks = []
mean_precisions = {}
mean_recalls = {}
protocols = set()
for i in range(0, len(file_contents_stripped), 4):
protocol = file_contents_stripped[i]
if protocol in protocols:
raise ValueError(
'Malformed input %s: protocol %s is found a second time' %
(metrics_path, protocol))
protocols.add(protocol)
# Parse mAP.
mean_average_precision[protocol] = float(
file_contents_stripped[i + 1].split('=')[1]) / 100.0
# Parse (or check consistency of) pr_ranks.
parsed_pr_ranks = _ParsePrRanks(file_contents_stripped[i + 2])
if not pr_ranks:
pr_ranks = parsed_pr_ranks
else:
if parsed_pr_ranks != pr_ranks:
raise ValueError('Malformed input %s: inconsistent PR ranks' %
metrics_path)
# Parse mean precisions.
mean_precisions[protocol] = np.array(
_ParsePrScores(file_contents_stripped[i + 2], len(pr_ranks)),
dtype=float) / 100.0
# Parse mean recalls.
mean_recalls[protocol] = np.array(
_ParsePrScores(file_contents_stripped[i + 3], len(pr_ranks)),
dtype=float) / 100.0
return mean_average_precision, pr_ranks, mean_precisions, mean_recalls
research/delf/delf/python/detect_to_retrieve/dataset_test.py
View file @ c627506f
......@@ -18,6 +18,8 @@ from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import numpy as np
import tensorflow as tf
......@@ -192,6 +194,92 @@ class DatasetTest(tf.test.TestCase):
self.assertAllClose(precisions, expected_precisions)
self.assertAllClose(recalls, expected_recalls)
def testSaveMetricsFileWorks(self):
# Define inputs.
mean_average_precision = {'hard': 0.7, 'medium': 0.9}
mean_precisions = {
'hard': np.array([1.0, 0.8]),
'medium': np.array([1.0, 1.0])
}
mean_recalls = {
'hard': np.array([0.5, 0.8]),
'medium': np.array([0.5, 1.0])
}
pr_ranks = [1, 5]
output_path = os.path.join(tf.compat.v1.test.get_temp_dir(), 'metrics.txt')
# Run tested function.
dataset.SaveMetricsFile(mean_average_precision, mean_precisions,
mean_recalls, pr_ranks, output_path)
# Define expected results.
expected_metrics = ('hard\n'
' mAP=70.0\n'
' mP@k[1 5] [100. 80.]\n'
' mR@k[1 5] [50. 80.]\n'
'medium\n'
' mAP=90.0\n'
' mP@k[1 5] [100. 100.]\n'
' mR@k[1 5] [ 50. 100.]\n')
# Parse actual results, and compare to expected.
with tf.io.gfile.GFile(output_path) as f:
metrics = f.read()
self.assertEqual(metrics, expected_metrics)
def testSaveAndReadMetricsWorks(self):
# Define inputs.
mean_average_precision = {'hard': 0.7, 'medium': 0.9}
mean_precisions = {
'hard': np.array([1.0, 0.8]),
'medium': np.array([1.0, 1.0])
}
mean_recalls = {
'hard': np.array([0.5, 0.8]),
'medium': np.array([0.5, 1.0])
}
pr_ranks = [1, 5]
output_path = os.path.join(tf.compat.v1.test.get_temp_dir(), 'metrics.txt')
# Run tested functions.
dataset.SaveMetricsFile(mean_average_precision, mean_precisions,
mean_recalls, pr_ranks, output_path)
(read_mean_average_precision, read_pr_ranks, read_mean_precisions,
read_mean_recalls) = dataset.ReadMetricsFile(output_path)
# Compares actual and expected metrics.
self.assertEqual(read_mean_average_precision, mean_average_precision)
self.assertEqual(read_pr_ranks, pr_ranks)
self.assertEqual(read_mean_precisions.keys(), mean_precisions.keys())
self.assertAllEqual(read_mean_precisions['hard'], mean_precisions['hard'])
self.assertAllEqual(read_mean_precisions['medium'],
mean_precisions['medium'])
self.assertEqual(read_mean_recalls.keys(), mean_recalls.keys())
self.assertAllEqual(read_mean_recalls['hard'], mean_recalls['hard'])
self.assertAllEqual(read_mean_recalls['medium'], mean_recalls['medium'])
def testReadMetricsWithRepeatedProtocolFails(self):
# Define inputs.
input_path = os.path.join(tf.compat.v1.test.get_temp_dir(), 'metrics.txt')
with tf.io.gfile.GFile(input_path, 'w') as f:
f.write('hard\n'
' mAP=70.0\n'
' mP@k[1 5] [ 100. 80.]\n'
' mR@k[1 5] [ 50. 80.]\n'
'medium\n'
' mAP=90.0\n'
' mP@k[1 5] [ 100. 100.]\n'
' mR@k[1 5] [ 50. 100.]\n'
'medium\n'
' mAP=90.0\n'
' mP@k[1 5] [ 100. 100.]\n'
' mR@k[1 5] [ 50. 100.]\n')
# Run tested functions.
with self.assertRaisesRegex(ValueError, 'Malformed input'):
dataset.ReadMetricsFile(input_path)
if __name__ == '__main__':
tf.test.main()
research/delf/delf/python/detect_to_retrieve/extract_query_features.py
View file @ c627506f
......@@ -61,7 +61,7 @@ def _PilLoader(path):
Returns:
PIL image in RGB format.
"""
with tf.gfile.GFile(path, 'rb') as f:
with tf.io.gfile.GFile(path, 'rb') as f:
img = Image.open(f)
return img.convert('RGB')
......@@ -70,28 +70,29 @@ def main(argv):
if len(argv) > 1:
raise RuntimeError('Too many command-line arguments.')
tf.logging.set_verbosity(tf.logging.INFO)
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
# Read list of query images from dataset file.
tf.logging.info('Reading list of query images and boxes from dataset file...')
tf.compat.v1.logging.info(
'Reading list of query images and boxes from dataset file...')
query_list, _, ground_truth = dataset.ReadDatasetFile(
cmd_args.dataset_file_path)
num_images = len(query_list)
tf.logging.info('done! Found %d images', num_images)
tf.compat.v1.logging.info('done! Found %d images', num_images)
# Parse DelfConfig proto.
config = delf_config_pb2.DelfConfig()
with tf.gfile.GFile(cmd_args.delf_config_path, 'r') as f:
with tf.io.gfile.GFile(cmd_args.delf_config_path, 'r') as f:
text_format.Merge(f.read(), config)
# Create output directory if necessary.
if not tf.gfile.Exists(cmd_args.output_features_dir):
tf.gfile.MakeDirs(cmd_args.output_features_dir)
if not tf.io.gfile.exists(cmd_args.output_features_dir):
tf.io.gfile.makedirs(cmd_args.output_features_dir)
with tf.Graph().as_default():
with tf.Session() as sess:
with tf.compat.v1.Session() as sess:
# Initialize variables, construct DELF extractor.
init_op = tf.global_variables_initializer()
init_op = tf.compat.v1.global_variables_initializer()
sess.run(init_op)
extractor_fn = extractor.MakeExtractor(sess, config)
......@@ -102,8 +103,8 @@ def main(argv):
query_image_name + _IMAGE_EXTENSION)
output_feature_filename = os.path.join(
cmd_args.output_features_dir, query_image_name + _DELF_EXTENSION)
if tf.gfile.Exists(output_feature_filename):
tf.logging.info('Skipping %s', query_image_name)
if tf.io.gfile.exists(output_feature_filename):
tf.compat.v1.logging.info('Skipping %s', query_image_name)
continue
# Crop query image according to bounding box.
......
research/delf/delf/python/detect_to_retrieve/image_reranking.py 0 → 100644
View file @ c627506f
# Copyright 2019 The TensorFlow Authors All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
"""Library to re-rank images based on geometric verification."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os
import numpy as np
from scipy import spatial
from skimage import measure
from skimage import transform
from delf import feature_io
# Extensions.
_DELF_EXTENSION = '.delf'
# Pace to log.
_STATUS_CHECK_GV_ITERATIONS = 10
# Re-ranking / geometric verification parameters.
_NUM_TO_RERANK = 100
_NUM_RANSAC_TRIALS = 1000
_MIN_RANSAC_SAMPLES = 3
def MatchFeatures(query_locations,
query_descriptors,
index_image_locations,
index_image_descriptors,
ransac_seed=None,
feature_distance_threshold=0.9,
ransac_residual_threshold=10.0):
"""Matches local features using geometric verification.
First, finds putative local feature matches by matching `query_descriptors`
against a KD-tree from the `index_image_descriptors`. Then, attempts to fit an
affine transformation between the putative feature corresponces using their
locations.
Args:
query_locations: Locations of local features for query image. NumPy array of
shape [#query_features, 2].
query_descriptors: Descriptors of local features for query image. NumPy
array of shape [#query_features, depth].
index_image_locations: Locations of local features for index image. NumPy
array of shape [#index_image_features, 2].
index_image_descriptors: Descriptors of local features for index image.
NumPy array of shape [#index_image_features, depth].
ransac_seed: Seed used by RANSAC. If None (default), no seed is provided.
feature_distance_threshold: Distance threshold below which a pair of
features is considered a potential match, and will be fed into RANSAC.
ransac_residual_threshold: Residual error threshold for considering matches
as inliers, used in RANSAC algorithm.
Returns:
score: Number of inliers of match. If no match is found, returns 0.
Raises:
ValueError: If local descriptors from query and index images have different
dimensionalities.
"""
num_features_query = query_locations.shape[0]
num_features_index_image = index_image_locations.shape[0]
if not num_features_query or not num_features_index_image:
return 0
local_feature_dim = query_descriptors.shape[1]
if index_image_descriptors.shape[1] != local_feature_dim:
raise ValueError(
'Local feature dimensionality is not consistent for query and index '
'images.')
# Find nearest-neighbor matches using a KD tree.
index_image_tree = spatial.cKDTree(index_image_descriptors)
_, indices = index_image_tree.query(
query_descriptors, distance_upper_bound=feature_distance_threshold)
# Select feature locations for putative matches.
query_locations_to_use = np.array([
query_locations[i,]
for i in range(num_features_query)
if indices[i] != num_features_index_image
])
index_image_locations_to_use = np.array([
index_image_locations[indices[i],]
for i in range(num_features_query)
if indices[i] != num_features_index_image
])
# If there are no putative matches, early return 0.
if not query_locations_to_use.shape[0]:
return 0
# Perform geometric verification using RANSAC.
_, inliers = measure.ransac(
(index_image_locations_to_use, query_locations_to_use),
transform.AffineTransform,
min_samples=_MIN_RANSAC_SAMPLES,
residual_threshold=ransac_residual_threshold,
max_trials=_NUM_RANSAC_TRIALS,
random_state=ransac_seed)
if inliers is None:
inliers = []
return sum(inliers)
def RerankByGeometricVerification(input_ranks, initial_scores, query_name,
index_names, query_features_dir,
index_features_dir, junk_ids):
"""Re-ranks retrieval results using geometric verification.
Args:
input_ranks: 1D NumPy array with indices of top-ranked index images, sorted
from the most to the least similar.
initial_scores: 1D NumPy array with initial similarity scores between query
and index images. Entry i corresponds to score for image i.
query_name: Name for query image (string).
index_names: List of names for index images (strings).
query_features_dir: Directory where query local feature file is located
(string).
index_features_dir: Directory where index local feature files are located
(string).
junk_ids: Set with indices of junk images which should not be considered
during re-ranking.
Returns:
output_ranks: 1D NumPy array with index image indices, sorted from the most
to the least similar according to the geometric verification and initial
scores.
Raises:
ValueError: If `input_ranks`, `initial_scores` and `index_names` do not have
the same number of entries.
"""
num_index_images = len(index_names)
if len(input_ranks) != num_index_images:
raise ValueError('input_ranks and index_names have different number of '
'elements: %d vs %d' %
(len(input_ranks), len(index_names)))
if len(initial_scores) != num_index_images:
raise ValueError('initial_scores and index_names have different number of '
'elements: %d vs %d' %
(len(initial_scores), len(index_names)))
# Filter out junk images from list that will be re-ranked.
input_ranks_for_gv = []
for ind in input_ranks:
if ind not in junk_ids:
input_ranks_for_gv.append(ind)
num_to_rerank = min(_NUM_TO_RERANK, len(input_ranks_for_gv))
# Load query image features.
query_features_path = os.path.join(query_features_dir,
query_name + _DELF_EXTENSION)
query_locations, _, query_descriptors, _, _ = feature_io.ReadFromFile(
query_features_path)
# Initialize list containing number of inliers and initial similarity scores.
inliers_and_initial_scores = []
for i in range(num_index_images):
inliers_and_initial_scores.append([0, initial_scores[i]])
# Loop over top-ranked images and get results.
print('Starting to re-rank')
for i in range(num_to_rerank):
if i > 0 and i % _STATUS_CHECK_GV_ITERATIONS == 0:
print('Re-ranking: i = %d out of %d' % (i, num_to_rerank))
index_image_id = input_ranks_for_gv[i]
# Load index image features.
index_image_features_path = os.path.join(
index_features_dir, index_names[index_image_id] + _DELF_EXTENSION)
(index_image_locations, _, index_image_descriptors, _,
_) = feature_io.ReadFromFile(index_image_features_path)
inliers_and_initial_scores[index_image_id][0] = MatchFeatures(
query_locations, query_descriptors, index_image_locations,
index_image_descriptors)
# Sort based on (inliers_score, initial_score).
def _InliersInitialScoresSorting(k):
"""Helper function to sort list based on two entries.
Args:
k: Index into `inliers_and_initial_scores`.
Returns:
Tuple containing inlier score and initial score.
"""
return (inliers_and_initial_scores[k][0], inliers_and_initial_scores[k][1])
output_ranks = sorted(
range(num_index_images), key=_InliersInitialScoresSorting, reverse=True)
return output_ranks
research/delf/delf/python/detect_to_retrieve/perform_retrieval.py
View file @ c627506f
......@@ -24,9 +24,6 @@ import sys
import time
import numpy as np
from scipy import spatial
from skimage import measure
from skimage import transform
import tensorflow as tf
from google.protobuf import text_format
......@@ -34,8 +31,8 @@ from tensorflow.python.platform import app
from delf import aggregation_config_pb2
from delf import datum_io
from delf import feature_aggregation_similarity
from delf import feature_io
from delf.python.detect_to_retrieve import dataset
from delf.python.detect_to_retrieve import image_reranking
cmd_args = None
......@@ -45,7 +42,6 @@ _ASMK = aggregation_config_pb2.AggregationConfig.ASMK
_ASMK_STAR = aggregation_config_pb2.AggregationConfig.ASMK_STAR
# Extensions.
_DELF_EXTENSION = '.delf'
_VLAD_EXTENSION_SUFFIX = 'vlad'
_ASMK_EXTENSION_SUFFIX = 'asmk'
_ASMK_STAR_EXTENSION_SUFFIX = 'asmk_star'
......@@ -55,18 +51,10 @@ _PR_RANKS = (1, 5, 10)
# Pace to log.
_STATUS_CHECK_LOAD_ITERATIONS = 50
_STATUS_CHECK_GV_ITERATIONS = 10
# Output file names.
_METRICS_FILENAME = 'metrics.txt'
# Re-ranking / geometric verification parameters.
_NUM_TO_RERANK = 100
_FEATURE_DISTANCE_THRESHOLD = 0.9
_NUM_RANSAC_TRIALS = 1000
_MIN_RANSAC_SAMPLES = 3
_RANSAC_RESIDUAL_THRESHOLD = 10
def _ReadAggregatedDescriptors(input_dir, image_list, config):
"""Reads aggregated descriptors.
......@@ -123,180 +111,6 @@ def _ReadAggregatedDescriptors(input_dir, image_list, config):
return aggregated_descriptors, visual_words
def _MatchFeatures(query_locations, query_descriptors, index_image_locations,
index_image_descriptors):
"""Matches local features using geometric verification.
First, finds putative local feature matches by matching `query_descriptors`
against a KD-tree from the `index_image_descriptors`. Then, attempts to fit an
affine transformation between the putative feature corresponces using their
locations.
Args:
query_locations: Locations of local features for query image. NumPy array of
shape [#query_features, 2].
query_descriptors: Descriptors of local features for query image. NumPy
array of shape [#query_features, depth].
index_image_locations: Locations of local features for index image. NumPy
array of shape [#index_image_features, 2].
index_image_descriptors: Descriptors of local features for index image.
NumPy array of shape [#index_image_features, depth].
Returns:
score: Number of inliers of match. If no match is found, returns 0.
"""
num_features_query = query_locations.shape[0]
num_features_index_image = index_image_locations.shape[0]
if not num_features_query or not num_features_index_image:
return 0
# Find nearest-neighbor matches using a KD tree.
index_image_tree = spatial.cKDTree(index_image_descriptors)
_, indices = index_image_tree.query(
query_descriptors, distance_upper_bound=_FEATURE_DISTANCE_THRESHOLD)
# Select feature locations for putative matches.
query_locations_to_use = np.array([
query_locations[i,]
for i in range(num_features_query)
if indices[i] != num_features_index_image
])
index_image_locations_to_use = np.array([
index_image_locations[indices[i],]
for i in range(num_features_query)
if indices[i] != num_features_index_image
])
# If there are no putative matches, early return 0.
if not query_locations_to_use.shape[0]:
return 0
# Perform geometric verification using RANSAC.
_, inliers = measure.ransac(
(index_image_locations_to_use, query_locations_to_use),
transform.AffineTransform,
min_samples=_MIN_RANSAC_SAMPLES,
residual_threshold=_RANSAC_RESIDUAL_THRESHOLD,
max_trials=_NUM_RANSAC_TRIALS)
if inliers is None:
inliers = []
return sum(inliers)
def _RerankByGeometricVerification(input_ranks, initial_scores, query_name,
index_names, query_features_dir,
index_features_dir, junk_ids):
"""Re-ranks retrieval results using geometric verification.
Args:
input_ranks: 1D NumPy array with indices of top-ranked index images, sorted
from the most to the least similar.
initial_scores: 1D NumPy array with initial similarity scores between query
and index images. Entry i corresponds to score for image i.
query_name: Name for query image (string).
index_names: List of names for index images (strings).
query_features_dir: Directory where query local feature file is located
(string).
index_features_dir: Directory where index local feature files are located
(string).
junk_ids: Set with indices of junk images which should not be considered
during re-ranking.
Returns:
output_ranks: 1D NumPy array with index image indices, sorted from the most
to the least similar according to the geometric verification and initial
scores.
Raises:
ValueError: If `input_ranks`, `initial_scores` and `index_names` do not have
the same number of entries.
"""
num_index_images = len(index_names)
if len(input_ranks) != num_index_images:
raise ValueError('input_ranks and index_names have different number of '
'elements: %d vs %d' %
(len(input_ranks), len(index_names)))
if len(initial_scores) != num_index_images:
raise ValueError('initial_scores and index_names have different number of '
'elements: %d vs %d' %
(len(initial_scores), len(index_names)))
# Filter out junk images from list that will be re-ranked.
input_ranks_for_gv = []
for ind in input_ranks:
if ind not in junk_ids:
input_ranks_for_gv.append(ind)
num_to_rerank = min(_NUM_TO_RERANK, len(input_ranks_for_gv))
# Load query image features.
query_features_path = os.path.join(query_features_dir,
query_name + _DELF_EXTENSION)
query_locations, _, query_descriptors, _, _ = feature_io.ReadFromFile(
query_features_path)
# Initialize list containing number of inliers and initial similarity scores.
inliers_and_initial_scores = []
for i in range(num_index_images):
inliers_and_initial_scores.append([0, initial_scores[i]])
# Loop over top-ranked images and get results.
print('Starting to re-rank')
for i in range(num_to_rerank):
if i > 0 and i % _STATUS_CHECK_GV_ITERATIONS == 0:
print('Re-ranking: i = %d out of %d' % (i, num_to_rerank))
index_image_id = input_ranks_for_gv[i]
# Load index image features.
index_image_features_path = os.path.join(
index_features_dir, index_names[index_image_id] + _DELF_EXTENSION)
(index_image_locations, _, index_image_descriptors, _,
_) = feature_io.ReadFromFile(index_image_features_path)
inliers_and_initial_scores[index_image_id][0] = _MatchFeatures(
query_locations, query_descriptors, index_image_locations,
index_image_descriptors)
# Sort based on (inliers_score, initial_score).
def _InliersInitialScoresSorting(k):
"""Helper function to sort list based on two entries.
Args:
k: Index into `inliers_and_initial_scores`.
Returns:
Tuple containing inlier score and initial score.
"""
return (inliers_and_initial_scores[k][0], inliers_and_initial_scores[k][1])
output_ranks = sorted(
range(num_index_images), key=_InliersInitialScoresSorting, reverse=True)
return output_ranks
def _SaveMetricsFile(mean_average_precision, mean_precisions, mean_recalls,
pr_ranks, output_path):
"""Saves aggregated retrieval metrics to text file.
Args:
mean_average_precision: Dict mapping each dataset protocol to a float.
mean_precisions: Dict mapping each dataset protocol to a NumPy array of
floats with shape [len(pr_ranks)].
mean_recalls: Dict mapping each dataset protocol to a NumPy array of floats
with shape [len(pr_ranks)].
pr_ranks: List of integers.
output_path: Full file path.
"""
with tf.gfile.GFile(output_path, 'w') as f:
for k in sorted(mean_average_precision.keys()):
f.write('{}\n mAP={}\n mP@k{} {}\n mR@k{} {}\n'.format(
k, np.around(mean_average_precision[k] * 100, decimals=2),
np.array(pr_ranks), np.around(mean_precisions[k] * 100, decimals=2),
np.array(pr_ranks), np.around(mean_recalls[k] * 100, decimals=2)))
def main(argv):
if len(argv) > 1:
raise RuntimeError('Too many command-line arguments.')
......@@ -314,10 +128,10 @@ def main(argv):
# Parse AggregationConfig protos.
query_config = aggregation_config_pb2.AggregationConfig()
with tf.gfile.GFile(cmd_args.query_aggregation_config_path, 'r') as f:
with tf.io.gfile.GFile(cmd_args.query_aggregation_config_path, 'r') as f:
text_format.Merge(f.read(), query_config)
index_config = aggregation_config_pb2.AggregationConfig()
with tf.gfile.GFile(cmd_args.index_aggregation_config_path, 'r') as f:
with tf.io.gfile.GFile(cmd_args.index_aggregation_config_path, 'r') as f:
text_format.Merge(f.read(), index_config)
# Read aggregated descriptors.
......@@ -355,11 +169,11 @@ def main(argv):
# Re-rank using geometric verification.
if cmd_args.use_geometric_verification:
medium_ranks_after_gv[i] = _RerankByGeometricVerification(
medium_ranks_after_gv[i] = image_reranking.RerankByGeometricVerification(
ranks_before_gv[i], similarities, query_list[i], index_list,
cmd_args.query_features_dir, cmd_args.index_features_dir,
set(medium_ground_truth[i]['junk']))
hard_ranks_after_gv[i] = _RerankByGeometricVerification(
hard_ranks_after_gv[i] = image_reranking.RerankByGeometricVerification(
ranks_before_gv[i], similarities, query_list[i], index_list,
cmd_args.query_features_dir, cmd_args.index_features_dir,
set(hard_ground_truth[i]['junk']))
......@@ -368,8 +182,8 @@ def main(argv):
print('done! Retrieval for query %d took %f seconds' % (i, elapsed))
# Create output directory if necessary.
if not tf.gfile.Exists(cmd_args.output_dir):
tf.gfile.MakeDirs(cmd_args.output_dir)
if not tf.io.gfile.exists(cmd_args.output_dir):
tf.io.gfile.makedirs(cmd_args.output_dir)
# Compute metrics.
medium_metrics = dataset.ComputeMetrics(ranks_before_gv, medium_ground_truth,
......@@ -403,9 +217,9 @@ def main(argv):
'medium_after_gv': medium_metrics_after_gv[2],
'hard_after_gv': hard_metrics_after_gv[2]
})
_SaveMetricsFile(mean_average_precision_dict, mean_precisions_dict,
mean_recalls_dict, _PR_RANKS,
os.path.join(cmd_args.output_dir, _METRICS_FILENAME))
dataset.SaveMetricsFile(mean_average_precision_dict, mean_precisions_dict,
mean_recalls_dict, _PR_RANKS,
os.path.join(cmd_args.output_dir, _METRICS_FILENAME))
if __name__ == '__main__':
......
research/delf/delf/python/examples/detector.py
View file @ c627506f
......@@ -32,8 +32,8 @@ def MakeDetector(sess, model_dir, import_scope=None):
Returns:
Function that receives an image and returns detection results.
"""
tf.saved_model.loader.load(
sess, [tf.saved_model.tag_constants.SERVING],
tf.compat.v1.saved_model.loader.load(
sess, [tf.compat.v1.saved_model.tag_constants.SERVING],
model_dir,
import_scope=import_scope)
import_scope_prefix = import_scope + '/' if import_scope is not None else ''
......
research/delf/delf/python/examples/extract_boxes.py
View file @ c627506f
......@@ -58,7 +58,7 @@ def _ReadImageList(list_path):
Returns:
image_paths: List of image paths.
"""
with tf.gfile.GFile(list_path, 'r') as f:
with tf.io.gfile.GFile(list_path, 'r') as f:
image_paths = f.readlines()
image_paths = [entry.rstrip() for entry in image_paths]
return image_paths
......@@ -130,46 +130,48 @@ def main(argv):
if len(argv) > 1:
raise RuntimeError('Too many command-line arguments.')
tf.logging.set_verbosity(tf.logging.INFO)
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
# Read list of images.
tf.logging.info('Reading list of images...')
tf.compat.v1.logging.info('Reading list of images...')
image_paths = _ReadImageList(cmd_args.list_images_path)
num_images = len(image_paths)
tf.logging.info('done! Found %d images', num_images)
tf.compat.v1.logging.info('done! Found %d images', num_images)
# Create output directories if necessary.
if not tf.gfile.Exists(cmd_args.output_dir):
tf.gfile.MakeDirs(cmd_args.output_dir)
if cmd_args.output_viz_dir and not tf.gfile.Exists(cmd_args.output_viz_dir):
tf.gfile.MakeDirs(cmd_args.output_viz_dir)
if not tf.io.gfile.exists(cmd_args.output_dir):
tf.io.gfile.makedirs(cmd_args.output_dir)
if cmd_args.output_viz_dir and not tf.io.gfile.exists(
cmd_args.output_viz_dir):
tf.io.gfile.makedirs(cmd_args.output_viz_dir)
# Tell TensorFlow that the model will be built into the default Graph.
with tf.Graph().as_default():
# Reading list of images.
filename_queue = tf.train.string_input_producer(image_paths, shuffle=False)
reader = tf.WholeFileReader()
filename_queue = tf.compat.v1.train.string_input_producer(
image_paths, shuffle=False)
reader = tf.compat.v1.WholeFileReader()
_, value = reader.read(filename_queue)
image_tf = tf.image.decode_jpeg(value, channels=3)
image_tf = tf.io.decode_jpeg(value, channels=3)
image_tf = tf.expand_dims(image_tf, 0)
with tf.Session() as sess:
init_op = tf.global_variables_initializer()
with tf.compat.v1.Session() as sess:
init_op = tf.compat.v1.global_variables_initializer()
sess.run(init_op)
detector_fn = detector.MakeDetector(sess, cmd_args.detector_path)
# Start input enqueue threads.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
threads = tf.compat.v1.train.start_queue_runners(sess=sess, coord=coord)
start = time.clock()
for i, image_path in enumerate(image_paths):
# Write to log-info once in a while.
if i == 0:
tf.logging.info('Starting to detect objects in images...')
tf.compat.v1.logging.info('Starting to detect objects in images...')
elif i % _STATUS_CHECK_ITERATIONS == 0:
elapsed = (time.clock() - start)
tf.logging.info(
tf.compat.v1.logging.info(
'Processing image %d out of %d, last %d '
'images took %f seconds', i, num_images, _STATUS_CHECK_ITERATIONS,
elapsed)
......@@ -183,8 +185,8 @@ def main(argv):
out_boxes_filename = base_boxes_filename + _BOX_EXT
out_boxes_fullpath = os.path.join(cmd_args.output_dir,
out_boxes_filename)
if tf.gfile.Exists(out_boxes_fullpath):
tf.logging.info('Skipping %s', image_path)
if tf.io.gfile.exists(out_boxes_fullpath):
tf.compat.v1.logging.info('Skipping %s', image_path)
continue
# Extract and save boxes.
......
research/delf/delf/python/examples/extract_features.py
View file @ c627506f
......@@ -27,6 +27,7 @@ import os
import sys
import time
from six.moves import range
import tensorflow as tf
from google.protobuf import text_format
......@@ -53,55 +54,57 @@ def _ReadImageList(list_path):
Returns:
image_paths: List of image paths.
"""
with tf.gfile.GFile(list_path, 'r') as f:
with tf.io.gfile.GFile(list_path, 'r') as f:
image_paths = f.readlines()
image_paths = [entry.rstrip() for entry in image_paths]
return image_paths
def main(unused_argv):
tf.logging.set_verbosity(tf.logging.INFO)
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
# Read list of images.
tf.logging.info('Reading list of images...')
tf.compat.v1.logging.info('Reading list of images...')
image_paths = _ReadImageList(cmd_args.list_images_path)
num_images = len(image_paths)
tf.logging.info('done! Found %d images', num_images)
tf.compat.v1.logging.info('done! Found %d images', num_images)
# Parse DelfConfig proto.
config = delf_config_pb2.DelfConfig()
with tf.gfile.FastGFile(cmd_args.config_path, 'r') as f:
with tf.io.gfile.GFile(cmd_args.config_path, 'r') as f:
text_format.Merge(f.read(), config)
# Create output directory if necessary.
if not tf.gfile.Exists(cmd_args.output_dir):
tf.gfile.MakeDirs(cmd_args.output_dir)
if not tf.io.gfile.exists(cmd_args.output_dir):
tf.io.gfile.makedirs(cmd_args.output_dir)
# Tell TensorFlow that the model will be built into the default Graph.
with tf.Graph().as_default():
# Reading list of images.
filename_queue = tf.train.string_input_producer(image_paths, shuffle=False)
reader = tf.WholeFileReader()
filename_queue = tf.compat.v1.train.string_input_producer(
image_paths, shuffle=False)
reader = tf.compat.v1.WholeFileReader()
_, value = reader.read(filename_queue)
image_tf = tf.image.decode_jpeg(value, channels=3)
image_tf = tf.io.decode_jpeg(value, channels=3)
with tf.Session() as sess:
init_op = tf.global_variables_initializer()
with tf.compat.v1.Session() as sess:
init_op = tf.compat.v1.global_variables_initializer()
sess.run(init_op)
extractor_fn = extractor.MakeExtractor(sess, config)
# Start input enqueue threads.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess=sess, coord=coord)
threads = tf.compat.v1.train.start_queue_runners(sess=sess, coord=coord)
start = time.clock()
for i in range(num_images):
# Write to log-info once in a while.
if i == 0:
tf.logging.info('Starting to extract DELF features from images...')
tf.compat.v1.logging.info(
'Starting to extract DELF features from images...')
elif i % _STATUS_CHECK_ITERATIONS == 0:
elapsed = (time.clock() - start)
tf.logging.info(
tf.compat.v1.logging.info(
'Processing image %d out of %d, last %d '
'images took %f seconds', i, num_images, _STATUS_CHECK_ITERATIONS,
elapsed)
......@@ -114,8 +117,8 @@ def main(unused_argv):
out_desc_filename = os.path.splitext(os.path.basename(
image_paths[i]))[0] + _DELF_EXT
out_desc_fullpath = os.path.join(cmd_args.output_dir, out_desc_filename)
if tf.gfile.Exists(out_desc_fullpath):
tf.logging.info('Skipping %s', image_paths[i])
if tf.io.gfile.exists(out_desc_fullpath):
tf.compat.v1.logging.info('Skipping %s', image_paths[i])
continue
# Extract and save features.
......
research/delf/delf/python/examples/extractor.py
View file @ c627506f
......@@ -30,44 +30,70 @@ _MIN_HEIGHT = 10
_MIN_WIDTH = 10
def ResizeImage(image, config):
def ResizeImage(image, config, resize_factor=1.0, square_output=False):
"""Resizes image according to config.
Args:
image: Uint8 array with shape (height, width, 3).
config: DelfConfig proto containing the model configuration.
resize_factor: Optional float resize factor for the input image. If given,
the maximum and minimum allowed image sizes in `config` are scaled by this
factor. Must be non-negative.
square_output: If True, the output image's aspect ratio is potentially
distorted and a square image (ie, height=width) is returned. The image is
resized such that the largest image side is used in both dimensions.
Returns:
resized_image: Uint8 array with resized image.
scale_factor: Float with factor used for resizing (If upscaling, larger than
1; if downscaling, smaller than 1).
scale_factors: 2D float array, with factors used for resizing along height
and width (If upscaling, larger than 1; if downscaling, smaller than 1).
Raises:
ValueError: If `image` has incorrect number of dimensions/channels.
"""
if resize_factor < 0.0:
raise ValueError('negative resize_factor is not allowed: %f' %
resize_factor)
if image.ndim != 3:
raise ValueError('image has incorrect number of dimensions: %d' %
image.ndims)
height, width, channels = image.shape
# Take into account resize factor.
max_image_size = resize_factor * config.max_image_size
min_image_size = resize_factor * config.min_image_size
if channels != 3:
raise ValueError('image has incorrect number of channels: %d' % channels)
if config.max_image_size != -1 and (width > config.max_image_size or
height > config.max_image_size):
scale_factor = config.max_image_size / max(width, height)
elif config.min_image_size != -1 and (width < config.min_image_size and
height < config.min_image_size):
scale_factor = config.min_image_size / max(width, height)
largest_side = max(width, height)
if max_image_size >= 0 and largest_side > max_image_size:
scale_factor = max_image_size / largest_side
elif min_image_size >= 0 and largest_side < min_image_size:
scale_factor = min_image_size / largest_side
elif square_output and (height != width):
scale_factor = 1.0
else:
# No resizing needed, early return.
return image, 1.0
return image, np.ones(2, dtype=float)
# Note that new_shape is in (width, height) format (PIL convention), while
# scale_factors are in (height, width) convention (NumPy convention).
if square_output:
new_shape = (int(round(largest_side * scale_factor)),
int(round(largest_side * scale_factor)))
else:
new_shape = (int(round(width * scale_factor)),
int(round(height * scale_factor)))
scale_factors = np.array([new_shape[1] / height, new_shape[0] / width],
dtype=float)
new_shape = (int(width * scale_factor), int(height * scale_factor))
pil_image = Image.fromarray(image)
resized_image = np.array(pil_image.resize(new_shape, resample=Image.BILINEAR))
return resized_image, scale_factor
return resized_image, scale_factors
def MakeExtractor(sess, config, import_scope=None):
......@@ -81,8 +107,8 @@ def MakeExtractor(sess, config, import_scope=None):
Returns:
Function that receives an image and returns features.
"""
tf.saved_model.loader.load(
sess, [tf.saved_model.tag_constants.SERVING],
tf.compat.v1.saved_model.loader.load(
sess, [tf.compat.v1.saved_model.tag_constants.SERVING],
config.model_path,
import_scope=import_scope)
import_scope_prefix = import_scope + '/' if import_scope is not None else ''
......@@ -118,7 +144,7 @@ def MakeExtractor(sess, config, import_scope=None):
Returns:
Tuple (locations, descriptors, feature_scales, attention)
"""
resized_image, scale_factor = ResizeImage(image, config)
resized_image, scale_factors = ResizeImage(image, config)
# If the image is too small, returns empty features.
if resized_image.shape[0] < _MIN_HEIGHT or resized_image.shape[
......@@ -134,7 +160,7 @@ def MakeExtractor(sess, config, import_scope=None):
input_image_scales: list(config.image_scales),
input_max_feature_num: config.delf_local_config.max_feature_num
})
rescaled_locations_out = locations_out / scale_factor
rescaled_locations_out = locations_out / scale_factors
return (rescaled_locations_out, descriptors_out, feature_scales_out,
attention_out)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment