Merge pull request #4734 from AlexeyKurakin/master

Releasing code for Adversarial Logit Pairing

CODEOWNERS

 * @tensorflow/tf-garden-team
 /official/ @tensorflow/tf-garden-team @karmel
 /research/adversarial_crypto/ @dave-andersen
+/research/adversarial_logit_pairing/ @AlexeyKurakin
 /research/adversarial_text/ @rsepassi @a-dai
 /research/adv_imagenet_models/ @AlexeyKurakin
 /research/attention_ocr/ @alexgorban

research/adversarial_logit_pairing/README.md

+
+# Adversarial logit pairing
+
+This directory contains implementation of
+[Adversarial logit pairing](https://arxiv.org/abs/1803.06373) paper as well as
+few models pre-trained on ImageNet and Tiny ImageNet.
+
+Please contact [Alexey Kurakin](https://github.com/AlexeyKurakin) regarding
+this code.
+
+## Pre-requesites
+
+Code dependencies:
+
+* TensorFlow 1.8 and Python 2.7 (other versions may work, but were not tested)
+* [Abseil Python](https://github.com/abseil/abseil-py).
+* Script which converts Tiny Imagenet dataset into TFRecord format also
+  depends on [Pandas](https://pandas.pydata.org/).
+
+## Datasets
+
+To use this code you need to download datasets. You only need to download
+those datasets which you're going to use. Following list of datasets is
+supported:
+
+* [ImageNet](http://www.image-net.org/). Follow
+  [Preparing the datasets](https://github.com/tensorflow/models/tree/master/research/slim#Data)
+  instructions in TF-Slim documentation to download and convert ImageNet dataset
+  to TFRecord format.
+
+* [Tiny ImageNet](https://tiny-imagenet.herokuapp.com/).
+  To obtain Tiny ImageNet dataset do following:
+
+  ```
+  # Download zip archive with TinyImagenet
+  curl -O http://cs231n.stanford.edu/tiny-imagenet-200.zip
+
+  # Extract archive
+  unzip tiny-imagenet-200.zip
+
+  # Convert dataset to TFRecord format
+  mkdir tiny-imagenet-tfrecord
+  python tiny_imagenet_converter/converter.py \
+    --input_dir=tiny-imagenet-200 \
+    --output_dir=tiny-imagenet-tfrecord
+  ```
+
+## Running the code
+
+NOTE: Provided code supports distributed training on multiple machines,
+and all provided checkpoints were trained in a distributed way. However it is
+beyond the scope of this document to describe how to do distributed training.
+Readed should refer to
+[other material](https://www.tensorflow.org/deploy/distributed) to learn
+about it.
+
+### Training
+
+Following command runs training:
+
+```
+# Following arguments has to be specified for training:
+# - MAX_NUMBER_OF_TRAINING_STEPS - maximum number of training steps,
+#     omit this flag or set it to -1 to have unlimited number of training steps.
+# - MODEL_NAME - name of the model, now only "resnet_v2_50" is supported.
+# - MOVING_AVG_DECAY - decay rate for exponential moving average of the
+#     trainable variables. Training with exponential moving average usually
+#     leads to better accuracy. Default of 0.9999. -1 disable exponential moving
+#     average. Default works well, so typically you set it only if you want
+#     to disable this feature.
+# - HYPERPARAMETERS - string with hyperparameters,
+#     see model_lib.py for full list of hyperparameters.
+# - DATASET - dataset, either "imagenet" or "tiny_imagenet".
+# - IMAGE_SIZE - size of the image (single number).
+# - OUTPUT_DIRECTORY - directory where to write results.
+# - IMAGENET_DIR - directory with ImageNet dataset in TFRecord format.
+# - TINY_IMAGENET_DIR - directory with Tiny ImageNet dataset in TFRecord format.
+#
+# Note that only one of IMAGENET_DIR or TINY_IMAGENET_DIR has to be provided
+# depending on which dataset you use.
+#
+python train.py \
+  --max_steps="${MAX_NUMBER_OF_TRAINING_STEPS}" \
+  --model_name="${MODEL_NAME}" \
+  --moving_average_decay="${MOVING_AVG_DECAY}" \
+  --hparams="${HYPERPARAMETERS}" \
+  --dataset="${DATASET}" \
+  --dataset_image_size="${IMAGE_SIZE}" \
+  --output_dir="${OUTPUT_DIRECTORY}" \
+  --imagenet_data_dir="${IMAGENET_DIR}" \
+  --tiny_imagenet_data_dir="${TINY_IMAGENET_DIR}"
+```
+
+Full list of training hyperparameters could be found in `model_lib.py`.
+These hyperparameters control learning rate schedule, optimizer, weight decay,
+label smoothing and adversarial training.
+
+Adversarial training is controlled by following hyperparameters:
+
+* `train_adv_method` - method which is used to craft adversarial examples during
+  training. Could be one of the following:
+
+  * `clean` - perform regular training with clean examples;
+  * `pgd_EPS_STEP_NITER` - use non targeted PGD with maximum size of
+    perturbation equal to `EPS`, step size equal to `STEP`
+    and number of iterations equal to `NITER`. Size of perturbation and step
+    size are expected to be integers between 1 and 255.
+  * `pgdll_EPS_STEP_NITER` - use targeted PGD, where target class is least
+    likely prediction of the network.
+  * `pgdrnd_EPS_STEP_NITER` - use targeted PGD, where target class is chosen
+    randomly.
+
+* `train_lp_weight` - weight of adversarial logit pairing loss. If zero or
+  negarive, then no logit pairing is performed and training is done using
+  mixed minibatch PGD. If positive then adversarial logit pairing term is added
+  to the loss.
+
+Below is example of how to run training with adversarial logit pairing on
+ImageNet 64x64:
+
+```
+python train.py \
+  --model_name="resnet_v2_50" \
+  --hparams="train_adv_method=pgdll_16_2_10,train_lp_weight=0.5" \
+  --dataset="imagenet" \
+  --dataset_image_size=64 \
+  --output_dir="/tmp/adv_train" \
+  --imagenet_data_dir="${IMAGENET_DIR}"
+```
+
+### Fine tuning
+
+Provided trainin script could be used to fine tune pre-trained checkpoint.
+Following command does this:
+
+```
+# Fine tuning adds following additional arguments:
+# - SCOPES_DO_NOT_LOAD_FROM_CHECKPOINT - comma separates list of scopes of
+#     variables, which should not be loadeded from checkpoint (and default
+#     initialization should be used instead).
+#     SCOPES_DO_NOT_LOAD_FROM_CHECKPOINT should be either same or a subset of
+#     LIST_OF_SCOPES_OF_TRAINABLE_VARS.
+# - LIST_OF_SCOPES_OF_TRAINABLE_VARS - comma separated list of scopes of
+#     trainable variables. Only variables which are prefixed with these scopes
+#     will be trained.
+# - PATH_TO_PRETRAINED_CHECKPOINT - directory with pretrained checkpoint which
+#     is used as initialization for fine tuning.
+#
+python train.py \
+  --max_steps="${MAX_NUMBER_OF_TRAINING_STEPS}" \
+  --model_name="${MODEL_NAME}" \
+  --moving_average_decay="${MOVING_AVG_DECAY}" \
+  --hparams="${HYPERPARAMETERS}" \
+  --dataset="${DATASET}" \
+  --dataset_image_size="${IMAGE_SIZE}" \
+  --output_dir="${OUTPUT_DIRECTORY}" \
+  --imagenet_data_dir="${IMAGENET_DIR}" \
+  --tiny_imagenet_data_dir="${TINY_IMAGENET_DIR}" \
+  --finetune_exclude_pretrained_scopes="${SCOPES_DO_NOT_LOAD_FROM_CHECKPOINT}" \
+  --finetune_trainable_scopes="${LIST_OF_SCOPES_OF_TRAINABLE_VARS}" \
+  --finetune_checkpoint_path="${PATH_TO_PRETRAINED_CHECKPOINT}"
+```
+
+Below is an example of how to fine tune last few layers of the model on
+Tiny Imagenet dataset:
+
+```
+python train.py \
+  --model_name="resnet_v2_50" \
+  --hparams="train_adv_method=pgdll_16_2_10,train_lp_weight=0.5,learning_rate=0.02" \
+  --dataset="tiny_imagenet" \
+  --dataset_image_size=64 \
+  --output_dir="/tmp/adv_finetune" \
+  --tiny_imagenet_data_dir="${TINY_IMAGENET_DIR}" \
+  --finetune_exclude_pretrained_scopes="resnet_v2_50/logits" \
+  --finetune_trainable_scopes="resnet_v2_50/logits,resnet_v2_50/postnorm" \
+  --finetune_checkpoint_path="/tmp/adv_train"
+```
+
+### Evaluation
+
+Following command runs evaluation:
+
+```
+# Following arguments should be provided for eval:
+# - TRAINING_DIRECTORY - directory where training checkpoints are saved.
+# - TRAINABLE_SCOPES - when loading checkpoint which was obtained by fine tuning
+#     this argument should be the same as LIST_OF_SCOPES_OF_TRAINABLE_VARS
+#     during training. Otherwise it should be empty.
+#     This is needed to properly load exponential moving average variables.
+#     If exponential moving averages are disabled then this flag could be
+#     omitted.
+# - EVAL_SUBDIR_NAME - name of the subdirectory inside TRAINING_DIRECTORY
+#     where evaluation code will be saving event files.
+# - DATASET - name of the dataset.
+# - IMAGE_SIZE - size of the image in the dataset.
+# - DATSET_SPLIT_NAME - name of the split in the dataset,
+#     either 'train' or 'validation'. Default is 'validation'.
+# - MODEL_NAME - name of the model.
+# - MOVING_AVG_DECAY - decay rate for exponential moving average.
+# - ADV_METHOD_FOR_EVAL - should be "clean" to evaluate on clean example or
+#     description of the adversarial method to evaluate on adversarial examples.
+# - HYPERPARAMETERS - hyperparameters, only "eval_batch_size" matters for eval
+# - NUMBER_OF_EXAMPLES - how many examples from the dataset use for evaluation,
+#     specify -1 to use all examples.
+# - EVAL_ONCE - if True then evaluate only once, otherwise keep evaluation
+#     running repeatedly on new checkpoints. Repeated evaluation might be useful
+#     when running concurrent with training.
+# - IMAGENET_DIR - directory with ImageNet dataset in TFRecord format.
+# - TINY_IMAGENET_DIR - directory with Tiny ImageNet dataset in TFRecord format.
+#
+python eval.py \
+  --train_dir="${TRAINING_DIRECTORY} \
+  --trainable_scopes="${TRAINABLE_SCOPES}" \
+  --eval_name="${EVAL_SUBDIR_NAME}" \
+  --dataset="${DATASET}" \
+  --dataset_image_size="${IMAGE_SIZE}" \
+  --split_name="${DATSET_SPLIT_NAME}" \
+  --model_name="${MODEL_NAME}" \
+  --moving_average_decay="${MOVING_AVG_DECAY}" \
+  --adv_method="${ADV_METHOD_FOR_EVAL}" \
+  --hparams="${HYPERPARAMETERS}" \
+  --num_examples="${NUMBER_OF_EXAMPLES}" \
+  --eval_once="${EVAL_ONCE}" \
+  --imagenet_data_dir="${IMAGENET_DIR}" \
+  --tiny_imagenet_data_dir="${TINY_IMAGENET_DIR}"
+```
+
+Example of running evaluation on 10000 of clean examples from ImageNet
+training set:
+
+```
+python eval.py \
+  --train_dir=/tmp/adv_train \
+  --dataset=imagenet \
+  --dataset_image_size=64 \
+  --split_name=train \
+  --adv_method=clean \
+  --hparams="eval_batch_size=50" \
+  --num_examples=10000 \
+  --eval_once=True \
+  --imagenet_data_dir="${IMAGENET_DIR}"
+```
+
+Example of running evaluatin on adversarial images generated from Tiny ImageNet
+validation set using fine-tuned checkpoint:
+
+```
+python eval.py \
+  --train_dir=tmp/adv_finetune \
+  --trainable_scopes="resnet_v2_50/logits,resnet_v2_50/postnorm" \
+  --dataset=tiny_imagenet \
+  --dataset_image_size=64 \
+  --adv_method=pgdrnd_16_2_10 \
+  --hparams="eval_batch_size=50" \
+  --eval_once=True \
+  --tiny_imagenet_data_dir="${TINY_IMAGENET_DIR}"
+```
+
+### Pre-trained models
+
+Following set of pre-trained checkpoints released with this code:
+
+| Model       |    Dataset   |  Accuracy on<br>clean images | Accuracy on<br>`pgdll_16_1_20` | Accuracy on<br>`pgdll_16_2_10` |
+| ----------- | ------------ | --------------- | --------------------------- | -------------- |
+| [Baseline ResNet-v2-50](http://download.tensorflow.org/models/adversarial_logit_pairing/imagenet64_base_2018_06_26.ckpt.tar.gz) | ImageNet 64x64 | 60.5% | 1.8% | 3.5% |
+| [ALP-trained ResNet-v2-50](http://download.tensorflow.org/models/adversarial_logit_pairing/imagenet64_alp025_2018_06_26.ckpt.tar.gz) | ImageNet 64x64 | 55.7% | 27.5% | 27.8% |
+| [Baseline ResNet-v2-50](http://download.tensorflow.org/models/adversarial_logit_pairing/tiny_imagenet_base_2018_06_26.ckpt.tar.gz) | Tiny ImageNet | 69.2% | 0.1% | 0.3% |
+| [ALP-trained ResNet-v2-50](http://download.tensorflow.org/models/adversarial_logit_pairing/tiny_imagenet_alp05_2018_06_26.ckpt.tar.gz) | Tiny ImageNet | 72.0% | 41.3% | 40.8% |
+
+
+* All provided checkpoints were initially trained with exponential moving
+  average. However for ease of use they were re-saved without it.
+  So to load and use provided checkpoints you need to specify
+  `--moving_average_decay=-1` flag.
+* All ALP models were trained with `pgdll_16_2_10` adversarial examples.
+* All Tiny Imagenet models were obtained by fine tuning corresponding
+  ImageNet 64x64 models. ALP-trained models were fine tuned with ALP.

research/adversarial_logit_pairing/adversarial_attack.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Library with adversarial attacks.
+
+This library designed to be self-contained and have no dependencies other
+than TensorFlow. It only contains PGD / Iterative FGSM attacks,
+see https://arxiv.org/abs/1706.06083 and https://arxiv.org/abs/1607.02533
+for details.
+
+For wider set of adversarial attacks refer to Cleverhans library:
+https://github.com/tensorflow/cleverhans
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import tensorflow as tf
+
+
+def generate_pgd_common(x,
+                        bounds,
+                        model_fn,
+                        attack_params,
+                        one_hot_labels,
+                        perturbation_multiplier):
+  """Common code for generating PGD adversarial examples.
+
+  Args:
+    x: original examples.
+    bounds: tuple with bounds of image values, bounds[0] < bounds[1].
+    model_fn: model function with signature model_fn(images).
+    attack_params: parameters of the attack.
+    one_hot_labels: one hot label vector to use in the loss.
+    perturbation_multiplier: multiplier of adversarial perturbation,
+      either +1.0 or -1.0.
+
+  Returns:
+    Tensor with adversarial examples.
+
+  Raises:
+    ValueError: if attack parameters are invalid.
+  """
+  # parse attack_params
+  # Format of attack_params: 'EPS_STEP_NITER'
+  # where EPS - epsilon, STEP - step size, NITER - number of iterations
+  params_list = attack_params.split('_')
+  if len(params_list) != 3:
+    raise ValueError('Invalid parameters of PGD attack: %s' % attack_params)
+  epsilon = int(params_list[0])
+  step_size = int(params_list[1])
+  niter = int(params_list[2])
+
+  # rescale epsilon and step size to image bounds
+  epsilon = float(epsilon) / 255.0 * (bounds[1] - bounds[0])
+  step_size = float(step_size) / 255.0 * (bounds[1] - bounds[0])
+
+  # clipping boundaries
+  clip_min = tf.maximum(x - epsilon, bounds[0])
+  clip_max = tf.minimum(x + epsilon, bounds[1])
+
+  # compute starting point
+  start_x = x + tf.random_uniform(tf.shape(x), -epsilon, epsilon)
+  start_x = tf.clip_by_value(start_x, clip_min, clip_max)
+
+  # main iteration of PGD
+  loop_vars = [0, start_x]
+
+  def loop_cond(index, _):
+    return index < niter
+
+  def loop_body(index, adv_images):
+    logits = model_fn(adv_images)
+    loss = tf.reduce_sum(
+        tf.nn.softmax_cross_entropy_with_logits_v2(
+            labels=one_hot_labels,
+            logits=logits))
+    perturbation = step_size * tf.sign(tf.gradients(loss, adv_images)[0])
+    new_adv_images = adv_images + perturbation_multiplier * perturbation
+    new_adv_images = tf.clip_by_value(new_adv_images, clip_min, clip_max)
+    return index + 1, new_adv_images
+
+  with tf.control_dependencies([start_x]):
+    _, result = tf.while_loop(
+        loop_cond,
+        loop_body,
+        loop_vars,
+        back_prop=False,
+        parallel_iterations=1)
+    return result
+
+
+def generate_pgd_ll(x, bounds, model_fn, attack_params):
+  # pylint: disable=g-doc-args
+  """Generats targeted PGD adversarial examples with least likely target class.
+
+  See generate_pgd_common for description of arguments.
+
+  Returns:
+    Tensor with adversarial examples.
+  """
+  # pylint: enable=g-doc-args
+
+  # compute one hot least likely class
+  logits = model_fn(x)
+  num_classes = tf.shape(logits)[1]
+  one_hot_labels = tf.one_hot(tf.argmin(model_fn(x), axis=1), num_classes)
+
+  return generate_pgd_common(x, bounds, model_fn, attack_params,
+                             one_hot_labels=one_hot_labels,
+                             perturbation_multiplier=-1.0)
+
+
+def generate_pgd_rand(x, bounds, model_fn, attack_params):
+  # pylint: disable=g-doc-args
+  """Generats targeted PGD adversarial examples with random target class.
+
+  See generate_pgd_common for description of arguments.
+
+  Returns:
+    Tensor with adversarial examples.
+  """
+  # pylint: enable=g-doc-args
+
+  # compute one hot random class
+  logits = model_fn(x)
+  batch_size = tf.shape(logits)[0]
+  num_classes = tf.shape(logits)[1]
+  random_labels = tf.random_uniform(shape=[batch_size],
+                                    minval=0,
+                                    maxval=num_classes,
+                                    dtype=tf.int32)
+  one_hot_labels = tf.one_hot(random_labels, num_classes)
+
+  return generate_pgd_common(x, bounds, model_fn, attack_params,
+                             one_hot_labels=one_hot_labels,
+                             perturbation_multiplier=-1.0)
+
+
+def generate_pgd(x, bounds, model_fn, attack_params):
+  # pylint: disable=g-doc-args
+  """Generats non-targeted PGD adversarial examples.
+
+  See generate_pgd_common for description of arguments.
+
+  Returns:
+    tensor with adversarial examples.
+  """
+  # pylint: enable=g-doc-args
+
+  # compute one hot predicted class
+  logits = model_fn(x)
+  num_classes = tf.shape(logits)[1]
+  one_hot_labels = tf.one_hot(tf.argmax(model_fn(x), axis=1), num_classes)
+
+  return generate_pgd_common(x, bounds, model_fn, attack_params,
+                             one_hot_labels=one_hot_labels,
+                             perturbation_multiplier=1.0)
+
+
+def generate_adversarial_examples(x, bounds, model_fn, attack_description):
+  """Generates adversarial examples.
+
+  Args:
+    x: original examples.
+    bounds: tuple with bounds of image values, bounds[0] < bounds[1]
+    model_fn: model function with signature model_fn(images).
+    attack_description: string which describes an attack, see notes below for
+      details.
+
+  Returns:
+    Tensor with adversarial examples.
+
+  Raises:
+    ValueError: if attack description is invalid.
+
+
+  Attack description could be one of the following strings:
+  - "clean" - no attack, return original images.
+  - "pgd_EPS_STEP_NITER" - non-targeted PGD attack.
+  - "pgdll_EPS_STEP_NITER" - tageted PGD attack with least likely target class.
+  - "pgdrnd_EPS_STEP_NITER" - targetd PGD attack with random target class.
+
+  Meaning of attack parameters is following:
+  - EPS - maximum size of adversarial perturbation, between 0 and 255.
+  - STEP - step size of one iteration of PGD, between 0 and 255.
+  - NITER - number of iterations.
+  """
+  if attack_description == 'clean':
+    return x
+  idx = attack_description.find('_')
+  if idx < 0:
+    raise ValueError('Invalid value of attack description %s'
+                     % attack_description)
+  attack_name = attack_description[:idx]
+  attack_params = attack_description[idx+1:]
+  if attack_name == 'pgdll':
+    return generate_pgd_ll(x, bounds, model_fn, attack_params)
+  elif attack_name == 'pgdrnd':
+    return generate_pgd_rand(x, bounds, model_fn, attack_params)
+  elif attack_name == 'pgd':
+    return generate_pgd(x, bounds, model_fn, attack_params)
+  else:
+    raise ValueError('Invalid value of attack description %s'
+                     % attack_description)
+

research/adversarial_logit_pairing/datasets/dataset_factory.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Library which creates datasets."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from datasets import imagenet_input
+from datasets import tiny_imagenet_input
+
+
+def get_dataset(dataset_name, split, batch_size, image_size, is_training):
+  """Returns dataset.
+
+  Args:
+    dataset_name: name of the dataset, "imagenet" or "tiny_imagenet".
+    split: name of the split, "train" or "validation".
+    batch_size: size of the minibatch.
+    image_size: size of the one side of the image. Output images will be
+      resized to square shape image_size*image_size.
+    is_training: if True then training preprocessing is done, otherwise eval
+      preprocessing is done.
+
+  Raises:
+    ValueError: if dataset_name is invalid.
+
+  Returns:
+    dataset: instance of tf.data.Dataset with the dataset.
+    num_examples: number of examples in given split of the dataset.
+    num_classes: number of classes in the dataset.
+    bounds: tuple with bounds of image values. All returned image pixels
+      are between bounds[0] and bounds[1].
+  """
+  if dataset_name == 'tiny_imagenet':
+    dataset = tiny_imagenet_input.tiny_imagenet_input(
+        split, batch_size, image_size, is_training)
+    num_examples = tiny_imagenet_input.num_examples_per_epoch(split)
+    num_classes = 200
+    bounds = (-1, 1)
+  elif dataset_name == 'imagenet':
+    dataset = imagenet_input.imagenet_input(
+        split, batch_size, image_size, is_training)
+    num_examples = imagenet_input.num_examples_per_epoch(split)
+    num_classes = 1001
+    bounds = (-1, 1)
+  else:
+    raise ValueError('Invalid dataset %s' % dataset_name)
+  return dataset, num_examples, num_classes, bounds

research/adversarial_logit_pairing/datasets/imagenet_input.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Imagenet input."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+from absl import flags
+import tensorflow as tf
+
+FLAGS = flags.FLAGS
+
+
+flags.DEFINE_string('imagenet_data_dir', None,
+                    'Directory with Imagenet dataset in TFRecord format.')
+
+
+def _decode_and_random_crop(image_buffer, bbox, image_size):
+  """Randomly crops image and then scales to target size."""
+  with tf.name_scope('distorted_bounding_box_crop',
+                     values=[image_buffer, bbox]):
+    sample_distorted_bounding_box = tf.image.sample_distorted_bounding_box(
+        tf.image.extract_jpeg_shape(image_buffer),
+        bounding_boxes=bbox,
+        min_object_covered=0.1,
+        aspect_ratio_range=[0.75, 1.33],
+        area_range=[0.08, 1.0],
+        max_attempts=10,
+        use_image_if_no_bounding_boxes=True)
+    bbox_begin, bbox_size, _ = sample_distorted_bounding_box
+
+    # Crop the image to the specified bounding box.
+    offset_y, offset_x, _ = tf.unstack(bbox_begin)
+    target_height, target_width, _ = tf.unstack(bbox_size)
+    crop_window = tf.stack([offset_y, offset_x, target_height, target_width])
+    image = tf.image.decode_and_crop_jpeg(image_buffer, crop_window, channels=3)
+    image = tf.image.convert_image_dtype(
+        image, dtype=tf.float32)
+
+    image = tf.image.resize_bicubic([image],
+                                    [image_size, image_size])[0]
+
+    return image
+
+
+def _decode_and_center_crop(image_buffer, image_size):
+  """Crops to center of image with padding then scales to target size."""
+  shape = tf.image.extract_jpeg_shape(image_buffer)
+  image_height = shape[0]
+  image_width = shape[1]
+
+  padded_center_crop_size = tf.cast(
+      0.875 * tf.cast(tf.minimum(image_height, image_width), tf.float32),
+      tf.int32)
+
+  offset_height = ((image_height - padded_center_crop_size) + 1) // 2
+  offset_width = ((image_width - padded_center_crop_size) + 1) // 2
+  crop_window = tf.stack([offset_height, offset_width,
+                          padded_center_crop_size, padded_center_crop_size])
+  image = tf.image.decode_and_crop_jpeg(image_buffer, crop_window, channels=3)
+  image = tf.image.convert_image_dtype(
+      image, dtype=tf.float32)
+
+  image = tf.image.resize_bicubic([image],
+                                  [image_size, image_size])[0]
+
+  return image
+
+
+def _normalize(image):
+  """Rescale image to [-1, 1] range."""
+  return tf.multiply(tf.subtract(image, 0.5), 2.0)
+
+
+def image_preprocessing(image_buffer, bbox, image_size, is_training):
+  """Does image decoding and preprocessing.
+
+  Args:
+    image_buffer: string tensor with encoded image.
+    bbox: bounding box of the object at the image.
+    image_size: image size.
+    is_training: whether to do training or eval preprocessing.
+
+  Returns:
+    Tensor with the image.
+  """
+  if is_training:
+    image = _decode_and_random_crop(image_buffer, bbox, image_size)
+    image = _normalize(image)
+    image = tf.image.random_flip_left_right(image)
+  else:
+    image = _decode_and_center_crop(image_buffer, image_size)
+    image = _normalize(image)
+  image = tf.reshape(image, [image_size, image_size, 3])
+  return image
+
+
+def imagenet_parser(value, image_size, is_training):
+  """Parse an ImageNet record from a serialized string Tensor.
+
+  Args:
+    value: encoded example.
+    image_size: size of the output image.
+    is_training: if True then do training preprocessing,
+      otherwise do eval preprocessing.
+
+  Returns:
+    image: tensor with the image.
+    label: true label of the image.
+  """
+  keys_to_features = {
+      'image/encoded':
+          tf.FixedLenFeature((), tf.string, ''),
+      'image/format':
+          tf.FixedLenFeature((), tf.string, 'jpeg'),
+      'image/class/label':
+          tf.FixedLenFeature([], tf.int64, -1),
+      'image/class/text':
+          tf.FixedLenFeature([], tf.string, ''),
+      'image/object/bbox/xmin':
+          tf.VarLenFeature(dtype=tf.float32),
+      'image/object/bbox/ymin':
+          tf.VarLenFeature(dtype=tf.float32),
+      'image/object/bbox/xmax':
+          tf.VarLenFeature(dtype=tf.float32),
+      'image/object/bbox/ymax':
+          tf.VarLenFeature(dtype=tf.float32),
+      'image/object/class/label':
+          tf.VarLenFeature(dtype=tf.int64),
+  }
+
+  parsed = tf.parse_single_example(value, keys_to_features)
+
+  image_buffer = tf.reshape(parsed['image/encoded'], shape=[])
+
+  xmin = tf.expand_dims(parsed['image/object/bbox/xmin'].values, 0)
+  ymin = tf.expand_dims(parsed['image/object/bbox/ymin'].values, 0)
+  xmax = tf.expand_dims(parsed['image/object/bbox/xmax'].values, 0)
+  ymax = tf.expand_dims(parsed['image/object/bbox/ymax'].values, 0)
+  # Note that ordering is (y, x)
+  bbox = tf.concat([ymin, xmin, ymax, xmax], 0)
+  # Force the variable number of bounding boxes into the shape
+  # [1, num_boxes, coords].
+  bbox = tf.expand_dims(bbox, 0)
+  bbox = tf.transpose(bbox, [0, 2, 1])
+
+  image = image_preprocessing(
+      image_buffer=image_buffer,
+      bbox=bbox,
+      image_size=image_size,
+      is_training=is_training
+  )
+
+  # Labels are in [1, 1000] range
+  label = tf.cast(
+      tf.reshape(parsed['image/class/label'], shape=[]), dtype=tf.int32)
+
+  return image, label
+
+
+def imagenet_input(split, batch_size, image_size, is_training):
+  """Returns ImageNet dataset.
+
+  Args:
+    split: name of the split, "train" or "validation".
+    batch_size: size of the minibatch.
+    image_size: size of the one side of the image. Output images will be
+      resized to square shape image_size*image_size.
+    is_training: if True then training preprocessing is done, otherwise eval
+      preprocessing is done.
+
+  Raises:
+    ValueError: if name of the split is incorrect.
+
+  Returns:
+    Instance of tf.data.Dataset with the dataset.
+  """
+  if split.lower().startswith('train'):
+    file_pattern = os.path.join(FLAGS.imagenet_data_dir, 'train-*')
+  elif split.lower().startswith('validation'):
+    file_pattern = os.path.join(FLAGS.imagenet_data_dir, 'validation-*')
+  else:
+    raise ValueError('Invalid split: %s' % split)
+
+  dataset = tf.data.Dataset.list_files(file_pattern, shuffle=is_training)
+
+  if is_training:
+    dataset = dataset.repeat()
+
+  def fetch_dataset(filename):
+    return tf.data.TFRecordDataset(filename, buffer_size=8*1024*1024)
+
+  # Read the data from disk in parallel
+  dataset = dataset.apply(
+      tf.contrib.data.parallel_interleave(
+          fetch_dataset, cycle_length=4, sloppy=True))
+  dataset = dataset.shuffle(1024)
+
+  # Parse, preprocess, and batch the data in parallel
+  dataset = dataset.apply(
+      tf.contrib.data.map_and_batch(
+          lambda value: imagenet_parser(value, image_size, is_training),
+          batch_size=batch_size,
+          num_parallel_batches=4,
+          drop_remainder=True))
+
+  def set_shapes(images, labels):
+    """Statically set the batch_size dimension."""
+    images.set_shape(images.get_shape().merge_with(
+        tf.TensorShape([batch_size, None, None, None])))
+    labels.set_shape(labels.get_shape().merge_with(
+        tf.TensorShape([batch_size])))
+    return images, labels
+
+  # Assign static batch size dimension
+  dataset = dataset.map(set_shapes)
+
+  # Prefetch overlaps in-feed with training
+  dataset = dataset.prefetch(tf.contrib.data.AUTOTUNE)
+  return dataset
+
+
+def num_examples_per_epoch(split):
+  """Returns the number of examples in the data set.
+
+  Args:
+    split: name of the split, "train" or "validation".
+
+  Raises:
+    ValueError: if split name is incorrect.
+
+  Returns:
+    Number of example in the split.
+  """
+  if split.lower().startswith('train'):
+    return 1281167
+  elif split.lower().startswith('validation'):
+    return 50000
+  else:
+    raise ValueError('Invalid split: %s' % split)

research/adversarial_logit_pairing/datasets/tiny_imagenet_input.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Tiny imagenet input."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+from absl import flags
+import tensorflow as tf
+
+FLAGS = flags.FLAGS
+
+
+flags.DEFINE_string('tiny_imagenet_data_dir', None,
+                    'Directory with Tiny Imagenet dataset in TFRecord format.')
+
+
+def tiny_imagenet_parser(value, image_size, is_training):
+  """Parses tiny imagenet example.
+
+  Args:
+    value: encoded example.
+    image_size: size of the image.
+    is_training: if True then do training preprocessing (which includes
+      random cropping), otherwise do eval preprocessing.
+
+  Returns:
+    image: tensor with the image.
+    label: true label of the image.
+  """
+  keys_to_features = {
+      'image/encoded': tf.FixedLenFeature((), tf.string, ''),
+      'label/tiny_imagenet': tf.FixedLenFeature([], tf.int64, -1),
+  }
+
+  parsed = tf.parse_single_example(value, keys_to_features)
+
+  image_buffer = tf.reshape(parsed['image/encoded'], shape=[])
+  image = tf.image.decode_image(image_buffer, channels=3)
+  image = tf.image.convert_image_dtype(
+      image, dtype=tf.float32)
+
+  # Crop image
+  if is_training:
+    bbox_begin, bbox_size, _ = tf.image.sample_distorted_bounding_box(
+        tf.shape(image),
+        bounding_boxes=tf.constant([0.0, 0.0, 1.0, 1.0],
+                                   dtype=tf.float32,
+                                   shape=[1, 1, 4]),
+        min_object_covered=0.5,
+        aspect_ratio_range=[0.75, 1.33],
+        area_range=[0.5, 1.0],
+        max_attempts=20,
+        use_image_if_no_bounding_boxes=True)
+    image = tf.slice(image, bbox_begin, bbox_size)
+
+  # resize image
+  image = tf.image.resize_bicubic([image], [image_size, image_size])[0]
+
+  # Rescale image to [-1, 1] range.
+  image = tf.multiply(tf.subtract(image, 0.5), 2.0)
+
+  image = tf.reshape(image, [image_size, image_size, 3])
+
+  # Labels are in [0, 199] range
+  label = tf.cast(
+      tf.reshape(parsed['label/tiny_imagenet'], shape=[]), dtype=tf.int32)
+
+  return image, label
+
+
+def tiny_imagenet_input(split, batch_size, image_size, is_training):
+  """Returns Tiny Imagenet Dataset.
+
+  Args:
+    split: name of the split, "train" or "validation".
+    batch_size: size of the minibatch.
+    image_size: size of the one side of the image. Output images will be
+      resized to square shape image_size*image_size.
+    is_training: if True then training preprocessing is done, otherwise eval
+      preprocessing is done.instance of tf.data.Dataset with the dataset.
+
+  Raises:
+    ValueError: if name of the split is incorrect.
+
+  Returns:
+    Instance of tf.data.Dataset with the dataset.
+  """
+  if split.lower().startswith('train'):
+    filepath = os.path.join(FLAGS.tiny_imagenet_data_dir, 'train.tfrecord')
+  elif split.lower().startswith('validation'):
+    filepath = os.path.join(FLAGS.tiny_imagenet_data_dir, 'validation.tfrecord')
+  else:
+    raise ValueError('Invalid split: %s' % split)
+
+  dataset = tf.data.TFRecordDataset(filepath, buffer_size=8*1024*1024)
+
+  if is_training:
+    dataset = dataset.shuffle(10000)
+    dataset = dataset.repeat()
+
+  dataset = dataset.apply(
+      tf.contrib.data.map_and_batch(
+          lambda value: tiny_imagenet_parser(value, image_size, is_training),
+          batch_size=batch_size,
+          num_parallel_batches=4,
+          drop_remainder=True))
+
+  def set_shapes(images, labels):
+    """Statically set the batch_size dimension."""
+    images.set_shape(images.get_shape().merge_with(
+        tf.TensorShape([batch_size, None, None, None])))
+    labels.set_shape(labels.get_shape().merge_with(
+        tf.TensorShape([batch_size])))
+    return images, labels
+
+  # Assign static batch size dimension
+  dataset = dataset.map(set_shapes)
+
+  dataset = dataset.prefetch(tf.contrib.data.AUTOTUNE)
+
+  return dataset
+
+
+def num_examples_per_epoch(split):
+  """Returns the number of examples in the data set.
+
+  Args:
+    split: name of the split, "train" or "validation".
+
+  Raises:
+    ValueError: if split name is incorrect.
+
+  Returns:
+    Number of example in the split.
+  """
+  if split.lower().startswith('train'):
+    return 100000
+  elif split.lower().startswith('validation'):
+    return 10000
+  else:
+    raise ValueError('Invalid split: %s' % split)

research/adversarial_logit_pairing/eval.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Program which runs evaluation of Imagenet 64x64 and TinyImagenet models."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+
+from absl import app
+from absl import flags
+
+import tensorflow as tf
+
+import adversarial_attack
+import model_lib
+from datasets import dataset_factory
+
+FLAGS = flags.FLAGS
+
+
+flags.DEFINE_string('train_dir', None,
+                    'Training directory. If specified then this program '
+                    'runs in continuous evaluation mode.')
+
+flags.DEFINE_string('checkpoint_path', None,
+                    'Path to the file with checkpoint. If specified then '
+                    'this program evaluates only provided checkpoint one time.')
+
+flags.DEFINE_string('output_file', None,
+                    'Name of output file. Used only in single evaluation mode.')
+
+flags.DEFINE_string('eval_name', 'default', 'Name for eval subdirectory.')
+
+flags.DEFINE_string('master', '', 'Tensorflow master.')
+
+flags.DEFINE_string('model_name', 'resnet_v2_50', 'Name of the model.')
+
+flags.DEFINE_string('adv_method', 'clean',
+                    'Method which is used to generate adversarial examples.')
+
+flags.DEFINE_string('dataset', 'imagenet',
+                    'Dataset: "tiny_imagenet" or "imagenet".')
+
+flags.DEFINE_integer('dataset_image_size', 64,
+                     'Size of the images in the dataset.')
+
+flags.DEFINE_string('hparams', '', 'Hyper parameters.')
+
+flags.DEFINE_string('split_name', 'validation', 'Name of the split.')
+
+flags.DEFINE_float('moving_average_decay', 0.9999,
+                   'The decay to use for the moving average.')
+
+flags.DEFINE_integer('eval_interval_secs', 120,
+                     'The frequency, in seconds, with which evaluation is run.')
+
+flags.DEFINE_integer(
+    'num_examples', -1,
+    'If positive - maximum number of example to use for evaluation.')
+
+flags.DEFINE_bool('eval_once', False,
+                  'If true then evaluate model only once.')
+
+flags.DEFINE_string('trainable_scopes', None,
+                    'If set then it defines list of variable scopes for '
+                    'trainable variables.')
+
+
+def main(_):
+  if not FLAGS.train_dir and not FLAGS.checkpoint_path:
+    print('Either --train_dir or --checkpoint_path flags has to be provided.')
+  if FLAGS.train_dir and FLAGS.checkpoint_path:
+    print('Only one of --train_dir or --checkpoint_path should be provided.')
+  params = model_lib.default_hparams()
+  params.parse(FLAGS.hparams)
+  tf.logging.info('User provided hparams: %s', FLAGS.hparams)
+  tf.logging.info('All hyper parameters: %s', params)
+  batch_size = params.eval_batch_size
+  graph = tf.Graph()
+  with graph.as_default():
+    # dataset
+    dataset, num_examples, num_classes, bounds = dataset_factory.get_dataset(
+        FLAGS.dataset,
+        FLAGS.split_name,
+        batch_size,
+        FLAGS.dataset_image_size,
+        is_training=False)
+    dataset_iterator = dataset.make_one_shot_iterator()
+    images, labels = dataset_iterator.get_next()
+    if FLAGS.num_examples > 0:
+      num_examples = min(num_examples, FLAGS.num_examples)
+
+    # setup model
+    global_step = tf.train.get_or_create_global_step()
+    model_fn_two_args = model_lib.get_model(FLAGS.model_name, num_classes)
+    model_fn = lambda x: model_fn_two_args(x, is_training=False)
+    if not FLAGS.adv_method or FLAGS.adv_method == 'clean':
+      logits = model_fn(images)
+    else:
+      adv_examples = adversarial_attack.generate_adversarial_examples(
+          images, bounds, model_fn, FLAGS.adv_method)
+      logits = model_fn(adv_examples)
+
+    # update trainable variables if fine tuning is used
+    model_lib.filter_trainable_variables(FLAGS.trainable_scopes)
+
+    # Setup the moving averages
+    if FLAGS.moving_average_decay and (FLAGS.moving_average_decay > 0):
+      variable_averages = tf.train.ExponentialMovingAverage(
+          FLAGS.moving_average_decay, global_step)
+      variables_to_restore = variable_averages.variables_to_restore(
+          tf.contrib.framework.get_model_variables())
+      variables_to_restore[global_step.op.name] = global_step
+    else:
+      variables_to_restore = tf.contrib.framework.get_variables_to_restore()
+
+    # Setup evaluation metric
+    with tf.name_scope('Eval'):
+      names_to_values, names_to_updates = (
+          tf.contrib.metrics.aggregate_metric_map({
+              'Accuracy': tf.metrics.accuracy(labels, tf.argmax(logits, 1)),
+              'Top5': tf.metrics.recall_at_k(tf.to_int64(labels), logits, 5)
+          }))
+
+      for name, value in names_to_values.iteritems():
+        tf.summary.scalar(name, value)
+
+    # Run evaluation
+    num_batches = int(num_examples / batch_size)
+    if FLAGS.train_dir:
+      output_dir = os.path.join(FLAGS.train_dir, FLAGS.eval_name)
+      if not tf.gfile.Exists(output_dir):
+        tf.gfile.MakeDirs(output_dir)
+      tf.contrib.training.evaluate_repeatedly(
+          FLAGS.train_dir,
+          master=FLAGS.master,
+          scaffold=tf.train.Scaffold(
+              saver=tf.train.Saver(variables_to_restore)),
+          eval_ops=names_to_updates.values(),
+          eval_interval_secs=FLAGS.eval_interval_secs,
+          hooks=[
+              tf.contrib.training.StopAfterNEvalsHook(num_batches),
+              tf.contrib.training.SummaryAtEndHook(output_dir),
+              tf.train.LoggingTensorHook(names_to_values, at_end=True),
+          ],
+          max_number_of_evaluations=1 if FLAGS.eval_once else None)
+    else:
+      result = tf.contrib.training.evaluate_once(
+          FLAGS.checkpoint_path,
+          master=FLAGS.master,
+          scaffold=tf.train.Scaffold(
+              saver=tf.train.Saver(variables_to_restore)),
+          eval_ops=names_to_updates.values(),
+          final_ops=names_to_values,
+          hooks=[
+              tf.contrib.training.StopAfterNEvalsHook(num_batches),
+              tf.train.LoggingTensorHook(names_to_values, at_end=True),
+          ])
+      if FLAGS.output_file:
+        with tf.gfile.Open(FLAGS.output_file, 'a') as f:
+          f.write('%s,%.3f,%.3f\n'
+                  % (FLAGS.eval_name, result['Accuracy'], result['Top5']))
+
+
+if __name__ == '__main__':
+  app.run(main)

research/adversarial_logit_pairing/model_lib.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Library with common functions for training and eval."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import six
+
+import tensorflow as tf
+
+from tensorflow.contrib.slim.nets import resnet_v2
+
+
+def default_hparams():
+  """Returns default hyperparameters."""
+  return tf.contrib.training.HParams(
+      # Batch size for training and evaluation.
+      batch_size=32,
+      eval_batch_size=50,
+
+      # General training parameters.
+      weight_decay=0.0001,
+      label_smoothing=0.1,
+
+      # Parameters of the adversarial training.
+      train_adv_method='clean',  # adversarial training method
+      train_lp_weight=0.0,  # Weight of adversarial logit pairing loss
+
+      # Parameters of the optimizer.
+      optimizer='rms',  # possible values are: 'rms', 'momentum', 'adam'
+      momentum=0.9,  # momentum
+      rmsprop_decay=0.9,  # Decay term for RMSProp
+      rmsprop_epsilon=1.0,  # Epsilon term for RMSProp
+
+      # Parameters of learning rate schedule.
+      lr_schedule='exp_decay',  # Possible values: 'exp_decay', 'step', 'fixed'
+      learning_rate=0.045,
+      lr_decay_factor=0.94,  # Learning exponential decay
+      lr_num_epochs_per_decay=2.0,  # Number of epochs per lr decay
+      lr_list=[1.0 / 6, 2.0 / 6, 3.0 / 6,
+               4.0 / 6, 5.0 / 6, 1.0, 0.1, 0.01,
+               0.001, 0.0001],
+      lr_decay_epochs=[1, 2, 3, 4, 5, 30, 60, 80,
+                       90])
+
+
+def get_lr_schedule(hparams, examples_per_epoch, replicas_to_aggregate=1):
+  """Returns TensorFlow op which compute learning rate.
+
+  Args:
+    hparams: hyper parameters.
+    examples_per_epoch: number of training examples per epoch.
+    replicas_to_aggregate: number of training replicas running in parallel.
+
+  Raises:
+    ValueError: if learning rate schedule specified in hparams is incorrect.
+
+  Returns:
+    learning_rate: tensor with learning rate.
+    steps_per_epoch: number of training steps per epoch.
+  """
+  global_step = tf.train.get_or_create_global_step()
+  steps_per_epoch = float(examples_per_epoch) / float(hparams.batch_size)
+  if replicas_to_aggregate > 0:
+    steps_per_epoch /= replicas_to_aggregate
+
+  if hparams.lr_schedule == 'exp_decay':
+    decay_steps = long(steps_per_epoch * hparams.lr_num_epochs_per_decay)
+    learning_rate = tf.train.exponential_decay(
+        hparams.learning_rate,
+        global_step,
+        decay_steps,
+        hparams.lr_decay_factor,
+        staircase=True)
+  elif hparams.lr_schedule == 'step':
+    lr_decay_steps = [long(epoch * steps_per_epoch)
+                      for epoch in hparams.lr_decay_epochs]
+    learning_rate = tf.train.piecewise_constant(
+        global_step, lr_decay_steps, hparams.lr_list)
+  elif hparams.lr_schedule == 'fixed':
+    learning_rate = hparams.learning_rate
+  else:
+    raise ValueError('Invalid value of lr_schedule: %s' % hparams.lr_schedule)
+
+  if replicas_to_aggregate > 0:
+    learning_rate *= replicas_to_aggregate
+
+  return learning_rate, steps_per_epoch
+
+
+def get_optimizer(hparams, learning_rate):
+  """Returns optimizer.
+
+  Args:
+    hparams: hyper parameters.
+    learning_rate: learning rate tensor.
+
+  Raises:
+    ValueError: if type of optimizer specified in hparams is incorrect.
+
+  Returns:
+    Instance of optimizer class.
+  """
+  if hparams.optimizer == 'rms':
+    optimizer = tf.train.RMSPropOptimizer(learning_rate,
+                                          hparams.rmsprop_decay,
+                                          hparams.momentum,
+                                          hparams.rmsprop_epsilon)
+  elif hparams.optimizer == 'momentum':
+    optimizer = tf.train.MomentumOptimizer(learning_rate,
+                                           hparams.momentum)
+  elif hparams.optimizer == 'adam':
+    optimizer = tf.train.AdamOptimizer(learning_rate)
+  else:
+    raise ValueError('Invalid value of optimizer: %s' % hparams.optimizer)
+  return optimizer
+
+
+RESNET_MODELS = {'resnet_v2_50': resnet_v2.resnet_v2_50}
+
+
+def get_model(model_name, num_classes):
+  """Returns function which creates model.
+
+  Args:
+    model_name: Name of the model.
+    num_classes: Number of classes.
+
+  Raises:
+    ValueError: If model_name is invalid.
+
+  Returns:
+    Function, which creates model when called.
+  """
+  if model_name.startswith('resnet'):
+    def resnet_model(images, is_training, reuse=tf.AUTO_REUSE):
+      with tf.contrib.framework.arg_scope(resnet_v2.resnet_arg_scope()):
+        resnet_fn = RESNET_MODELS[model_name]
+        logits, _ = resnet_fn(images, num_classes, is_training=is_training,
+                              reuse=reuse)
+        logits = tf.reshape(logits, [-1, num_classes])
+      return logits
+    return resnet_model
+  else:
+    raise ValueError('Invalid model: %s' % model_name)
+
+
+def filter_trainable_variables(trainable_scopes):
+  """Keep only trainable variables which are prefixed with given scopes.
+
+  Args:
+    trainable_scopes: either list of trainable scopes or string with comma
+      separated list of trainable scopes.
+
+  This function removes all variables which are not prefixed with given
+  trainable_scopes from collection of trainable variables.
+  Useful during network fine tuning, when you only need to train subset of
+  variables.
+  """
+  if not trainable_scopes:
+    return
+  if isinstance(trainable_scopes, six.string_types):
+    trainable_scopes = [scope.strip() for scope in trainable_scopes.split(',')]
+  trainable_scopes = {scope for scope in trainable_scopes if scope}
+  if not trainable_scopes:
+    return
+  trainable_collection = tf.get_collection_ref(
+      tf.GraphKeys.TRAINABLE_VARIABLES)
+  non_trainable_vars = [
+      v for v in trainable_collection
+      if not any([v.op.name.startswith(s) for s in trainable_scopes])
+  ]
+  for v in non_trainable_vars:
+    trainable_collection.remove(v)

research/adversarial_logit_pairing/tiny_imagenet_converter/converter.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Converts Tiny Imagenet dataset into TFRecord format.
+
+As an output this program generates following files in TFRecord format:
+- train.tfrecord
+- validation.tfrecord
+- test.tfrecord
+
+Generated train and validation files will contain tf.Example entries with
+following features:
+- image/encoded - encoded image
+- image/format - image format
+- label/wnid - label WordNet ID
+- label/imagenet - imagenet label [1 ... 1000]
+- label/tiny_imagenet - tiny imagenet label [0 ... 199]
+- bbox/xmin
+- bbox/ymin
+- bbox/xmax
+- bbox/ymax
+
+Test file will contain entries with 'image/encoded' and 'image/format' features.
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from collections import namedtuple
+import os
+import random
+
+from absl import app
+from absl import flags
+from absl import logging
+
+import pandas as pd
+
+import tensorflow as tf
+
+
+FLAGS = flags.FLAGS
+
+flags.DEFINE_string('input_dir', '', 'Input directory')
+flags.DEFINE_string('output_dir', '', 'Output directory')
+
+flags.DEFINE_string('imagenet_synsets_path', '',
+                    'Optional path to /imagenet_lsvrc_2015_synsets.txt')
+
+
+ImageMetadata = namedtuple('ImageMetadata', ['label', 'x1', 'y1', 'x2', 'y2'])
+
+
+class WnIdToNodeIdConverter(object):
+  """Converts WordNet IDs to numerical labels."""
+
+  def __init__(self, wnids_path, background_class):
+    self._wnid_to_node_id = {}
+    self._node_id_to_wnid = {}
+    with tf.gfile.Open(wnids_path) as f:
+      wnids_sequence = [wnid.strip() for wnid in f.readlines() if wnid.strip()]
+    node_id_offset = 1 if background_class else 0
+    for i, label in enumerate(wnids_sequence):
+      self._wnid_to_node_id[label] = i + node_id_offset
+      self._node_id_to_wnid[i + node_id_offset] = label
+
+  def to_node_id(self, wnid):
+    return self._wnid_to_node_id[wnid]
+
+  def to_wnid(self, node_id):
+    return self._node_id_to_wnid[node_id]
+
+  def all_wnids(self):
+    return self._wnid_to_node_id.keys()
+
+
+def read_tiny_imagenet_annotations(annotations_filename,
+                                   images_dir,
+                                   one_label=None):
+  """Reads one file with Tiny Imagenet annotations."""
+  result = []
+  if one_label:
+    column_names = ['filename', 'x1', 'y1', 'x2', 'y2']
+  else:
+    column_names = ['filename', 'label', 'x1', 'y1', 'x2', 'y2']
+  with tf.gfile.Open(annotations_filename) as f:
+    data = pd.read_csv(f, sep='\t', names=column_names)
+  for row in data.itertuples():
+    label = one_label if one_label else getattr(row, 'label')
+    full_filename = os.path.join(images_dir, getattr(row, 'filename'))
+    result.append((full_filename,
+                   ImageMetadata(label=label,
+                                 x1=getattr(row, 'x1'),
+                                 y1=getattr(row, 'y1'),
+                                 x2=getattr(row, 'x2'),
+                                 y2=getattr(row, 'y2'))))
+  return result
+
+
+def read_validation_annotations(validation_dir):
+  """Reads validation data annotations."""
+  return read_tiny_imagenet_annotations(
+      os.path.join(validation_dir, 'val_annotations.txt'),
+      os.path.join(validation_dir, 'images'))
+
+
+def read_training_annotations(training_dir):
+  """Reads training data annotations."""
+  result = []
+  sub_dirs = tf.gfile.ListDirectory(training_dir)
+  for sub_dir in sub_dirs:
+    if not sub_dir.startswith('n'):
+      logging.warning('Found non-class directory in training dir: %s', sub_dir)
+      continue
+    sub_dir_results = read_tiny_imagenet_annotations(
+        os.path.join(training_dir, sub_dir, sub_dir + '_boxes.txt'),
+        os.path.join(training_dir, sub_dir, 'images'),
+        one_label=sub_dir)
+    result.extend(sub_dir_results)
+  return result
+
+
+def read_test_annotations(test_dir):
+  """Reads test data annotations."""
+  files = tf.gfile.ListDirectory(os.path.join(test_dir, 'images'))
+  return [(os.path.join(test_dir, 'images', f), None)
+          for f in files if f.endswith('.JPEG')]
+
+
+def get_image_format(filename):
+  """Returns image format from filename."""
+  filename = filename.lower()
+  if filename.endswith('jpeg') or filename.endswith('jpg'):
+    return 'jpeg'
+  elif filename.endswith('png'):
+    return 'png'
+  else:
+    raise ValueError('Unrecognized file format: %s' % filename)
+
+
+class TinyImagenetWriter(object):
+  """Helper class which writes Tiny Imagenet dataset into TFRecord file."""
+
+  def __init__(self, tiny_imagenet_wnid_conveter, imagenet_wnid_converter):
+    self.tiny_imagenet_wnid_conveter = tiny_imagenet_wnid_conveter
+    self.imagenet_wnid_converter = imagenet_wnid_converter
+
+  def write_tf_record(self,
+                      annotations,
+                      output_file):
+    """Generates TFRecord file from given list of annotations."""
+    with tf.python_io.TFRecordWriter(output_file) as writer:
+      for image_filename, image_metadata in annotations:
+        with tf.gfile.Open(image_filename) as f:
+          image_buffer = f.read()
+        image_format = get_image_format(image_filename)
+        features = {
+            'image/encoded': tf.train.Feature(
+                bytes_list=tf.train.BytesList(value=[image_buffer])),
+            'image/format': tf.train.Feature(
+                bytes_list=tf.train.BytesList(value=[image_format]))
+        }
+        if image_metadata:
+          # bounding box features
+          features['bbox/xmin'] = tf.train.Feature(
+              int64_list=tf.train.Int64List(value=[image_metadata.x1]))
+          features['bbox/ymin'] = tf.train.Feature(
+              int64_list=tf.train.Int64List(value=[image_metadata.y1]))
+          features['bbox/xmax'] = tf.train.Feature(
+              int64_list=tf.train.Int64List(value=[image_metadata.x2]))
+          features['bbox/ymax'] = tf.train.Feature(
+              int64_list=tf.train.Int64List(value=[image_metadata.y2]))
+          # tiny imagenet label, from [0, 200) iterval
+          tiny_imagenet_label = self.tiny_imagenet_wnid_conveter.to_node_id(
+              image_metadata.label)
+          features['label/wnid'] = tf.train.Feature(
+              bytes_list=tf.train.BytesList(value=image_metadata.label))
+          features['label/tiny_imagenet'] = tf.train.Feature(
+              int64_list=tf.train.Int64List(value=[tiny_imagenet_label]))
+          # full imagenet label, from [1, 1001) interval
+          if self.imagenet_wnid_converter:
+            imagenet_label = self.imagenet_wnid_converter.to_node_id(
+                image_metadata.label)
+            features['label/imagenet'] = tf.train.Feature(
+                int64_list=tf.train.Int64List(value=[imagenet_label]))
+        example = tf.train.Example(features=tf.train.Features(feature=features))
+        writer.write(example.SerializeToString())
+
+
+def main(_):
+  assert FLAGS.input_dir, 'Input directory must be provided'
+  assert FLAGS.output_dir, 'Output directory must be provided'
+
+  # Create WordNet ID conveters for tiny imagenet and possibly for imagenet
+  tiny_imagenet_wnid_conveter = WnIdToNodeIdConverter(
+      os.path.join(FLAGS.input_dir, 'wnids.txt'),
+      background_class=False)
+  if FLAGS.imagenet_synsets_path:
+    imagenet_wnid_converter = WnIdToNodeIdConverter(FLAGS.imagenet_synsets_path,
+                                                    background_class=True)
+  else:
+    imagenet_wnid_converter = None
+
+  # read tiny imagenet annotations
+  train_annotations = read_training_annotations(
+      os.path.join(FLAGS.input_dir, 'train'))
+  random.shuffle(train_annotations)
+  val_annotations = read_validation_annotations(
+      os.path.join(FLAGS.input_dir, 'val'))
+  test_filenames = read_test_annotations(os.path.join(FLAGS.input_dir, 'test'))
+
+  # Generate TFRecord files
+  writer = TinyImagenetWriter(tiny_imagenet_wnid_conveter,
+                              imagenet_wnid_converter)
+  tf.logging.info('Converting %d training images', len(train_annotations))
+  writer.write_tf_record(train_annotations,
+                         os.path.join(FLAGS.output_dir, 'train.tfrecord'))
+  tf.logging.info('Converting %d validation images ', len(val_annotations))
+  writer.write_tf_record(val_annotations,
+                         os.path.join(FLAGS.output_dir, 'validation.tfrecord'))
+  tf.logging.info('Converting %d test images', len(test_filenames))
+  writer.write_tf_record(test_filenames,
+                         os.path.join(FLAGS.output_dir, 'test.tfrecord'))
+  tf.logging.info('All files are converted')
+
+
+if __name__ == '__main__':
+  app.run(main)

research/adversarial_logit_pairing/train.py

+# Copyright 2018 Google Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Program which train models."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from absl import app
+from absl import flags
+
+import tensorflow as tf
+
+import adversarial_attack
+import model_lib
+from datasets import dataset_factory
+
+FLAGS = flags.FLAGS
+
+
+flags.DEFINE_integer('max_steps', -1, 'Number of steps to stop at.')
+
+flags.DEFINE_string('output_dir', None,
+                    'Training directory where checkpoints will be saved.')
+
+flags.DEFINE_integer('ps_tasks', 0, 'Number of parameter servers.')
+
+flags.DEFINE_integer('task', 0, 'Task ID for running distributed training.')
+
+flags.DEFINE_string('master', '', 'Tensorflow master.')
+
+flags.DEFINE_string('model_name', 'resnet_v2_50', 'Name of the model.')
+
+flags.DEFINE_string('dataset', 'imagenet',
+                    'Dataset: "tiny_imagenet" or "imagenet".')
+
+flags.DEFINE_integer('dataset_image_size', 64,
+                     'Size of the images in the dataset.')
+
+flags.DEFINE_integer('num_summary_images', 3,
+                     'Number of images to display in Tensorboard.')
+
+flags.DEFINE_integer(
+    'save_summaries_steps', 100,
+    'The frequency with which summaries are saved, in steps.')
+
+flags.DEFINE_integer(
+    'save_summaries_secs', None,
+    'The frequency with which summaries are saved, in seconds.')
+
+flags.DEFINE_integer(
+    'save_model_steps', 500,
+    'The frequency with which the model is saved, in steps.')
+
+flags.DEFINE_string('hparams', '', 'Hyper parameters.')
+
+flags.DEFINE_integer('replicas_to_aggregate', 1,
+                     'Number of gradients to collect before param updates.')
+
+flags.DEFINE_integer('worker_replicas', 1, 'Number of worker replicas.')
+
+flags.DEFINE_float('moving_average_decay', 0.9999,
+                   'The decay to use for the moving average.')
+
+# Flags to control fine tuning
+
+flags.DEFINE_string('finetune_checkpoint_path', None,
+                    'Path to checkpoint for fine tuning. '
+                    'If None then no fine tuning is done.')
+
+flags.DEFINE_string('finetune_exclude_pretrained_scopes', '',
+                    'Variable scopes to exclude when loading checkpoint for  '
+                    'fine tuning.')
+
+flags.DEFINE_string('finetune_trainable_scopes', None,
+                    'If set then it defines list of variable scopes for '
+                    'trainable variables.')
+
+
+def _get_finetuning_init_fn(variable_averages):
+  """Returns an init functions, used for fine tuning."""
+  if not FLAGS.finetune_checkpoint_path:
+    return None
+
+  if tf.train.latest_checkpoint(FLAGS.output_dir):
+    return None
+
+  if tf.gfile.IsDirectory(FLAGS.finetune_checkpoint_path):
+    checkpoint_path = tf.train.latest_checkpoint(FLAGS.finetune_checkpoint_path)
+  else:
+    checkpoint_path = FLAGS.finetune_checkpoint_path
+
+  if not checkpoint_path:
+    tf.logging.warning('Not doing fine tuning, can not find checkpoint in %s',
+                       FLAGS.finetune_checkpoint_path)
+    return None
+
+  tf.logging.info('Fine-tuning from %s', checkpoint_path)
+
+  if FLAGS.finetune_exclude_pretrained_scopes:
+    exclusions = {
+        scope.strip()
+        for scope in FLAGS.finetune_exclude_pretrained_scopes.split(',')
+    }
+  else:
+    exclusions = set()
+
+  filtered_model_variables = [
+      v for v in tf.contrib.framework.get_model_variables()
+      if not any([v.op.name.startswith(e) for e in exclusions])
+  ]
+
+  if variable_averages:
+    variables_to_restore = {}
+    for v in filtered_model_variables:
+      # variables_to_restore[variable_averages.average_name(v)] = v
+      if v in tf.trainable_variables():
+        variables_to_restore[variable_averages.average_name(v)] = v
+      else:
+        variables_to_restore[v.op.name] = v
+  else:
+    variables_to_restore = {v.op.name: v for v in filtered_model_variables}
+
+  assign_fn = tf.contrib.framework.assign_from_checkpoint_fn(
+      checkpoint_path,
+      variables_to_restore)
+  if assign_fn:
+    return lambda _, sess: assign_fn(sess)
+  else:
+    return None
+
+
+def main(_):
+  assert FLAGS.output_dir, '--output_dir has to be provided'
+  if not tf.gfile.Exists(FLAGS.output_dir):
+    tf.gfile.MakeDirs(FLAGS.output_dir)
+  params = model_lib.default_hparams()
+  params.parse(FLAGS.hparams)
+  tf.logging.info('User provided hparams: %s', FLAGS.hparams)
+  tf.logging.info('All hyper parameters: %s', params)
+  batch_size = params.batch_size
+  graph = tf.Graph()
+  with graph.as_default():
+    with tf.device(tf.train.replica_device_setter(ps_tasks=FLAGS.ps_tasks)):
+      # dataset
+      dataset, examples_per_epoch, num_classes, bounds = (
+          dataset_factory.get_dataset(
+              FLAGS.dataset,
+              'train',
+              batch_size,
+              FLAGS.dataset_image_size,
+              is_training=True))
+      dataset_iterator = dataset.make_one_shot_iterator()
+      images, labels = dataset_iterator.get_next()
+      one_hot_labels = tf.one_hot(labels, num_classes)
+
+      # set up model
+      global_step = tf.train.get_or_create_global_step()
+      model_fn = model_lib.get_model(FLAGS.model_name, num_classes)
+      if params.train_adv_method == 'clean':
+        logits = model_fn(images, is_training=True)
+        adv_examples = None
+      else:
+        model_fn_eval_mode = lambda x: model_fn(x, is_training=False)
+        adv_examples = adversarial_attack.generate_adversarial_examples(
+            images, bounds, model_fn_eval_mode, params.train_adv_method)
+        all_examples = tf.concat([images, adv_examples], axis=0)
+        logits = model_fn(all_examples, is_training=True)
+        one_hot_labels = tf.concat([one_hot_labels, one_hot_labels], axis=0)
+
+      # update trainable variables if fine tuning is used
+      model_lib.filter_trainable_variables(
+          FLAGS.finetune_trainable_scopes)
+
+      # set up losses
+      total_loss = tf.losses.softmax_cross_entropy(
+          onehot_labels=one_hot_labels,
+          logits=logits,
+          label_smoothing=params.label_smoothing)
+      tf.summary.scalar('loss_xent', total_loss)
+
+      if params.train_lp_weight > 0:
+        images1, images2 = tf.split(logits, 2)
+        loss_lp = tf.losses.mean_squared_error(
+            images1, images2, weights=params.train_lp_weight)
+        tf.summary.scalar('loss_lp', loss_lp)
+        total_loss += loss_lp
+
+      if params.weight_decay > 0:
+        loss_wd = (
+            params.weight_decay
+            * tf.add_n([tf.nn.l2_loss(v) for v in tf.trainable_variables()])
+        )
+        tf.summary.scalar('loss_wd', loss_wd)
+        total_loss += loss_wd
+
+      # Setup the moving averages:
+      if FLAGS.moving_average_decay and (FLAGS.moving_average_decay > 0):
+        with tf.name_scope('moving_average'):
+          moving_average_variables = tf.contrib.framework.get_model_variables()
+          variable_averages = tf.train.ExponentialMovingAverage(
+              FLAGS.moving_average_decay, global_step)
+      else:
+        moving_average_variables = None
+        variable_averages = None
+
+      # set up optimizer and training op
+      learning_rate, steps_per_epoch = model_lib.get_lr_schedule(
+          params, examples_per_epoch, FLAGS.replicas_to_aggregate)
+
+      optimizer = model_lib.get_optimizer(params, learning_rate)
+
+      optimizer = tf.train.SyncReplicasOptimizer(
+          opt=optimizer,
+          replicas_to_aggregate=FLAGS.replicas_to_aggregate,
+          total_num_replicas=FLAGS.worker_replicas,
+          variable_averages=variable_averages,
+          variables_to_average=moving_average_variables)
+
+      train_op = tf.contrib.training.create_train_op(
+          total_loss, optimizer,
+          update_ops=tf.get_collection(tf.GraphKeys.UPDATE_OPS))
+
+      tf.summary.image('images', images[0:FLAGS.num_summary_images])
+      if adv_examples is not None:
+        tf.summary.image('adv_images', adv_examples[0:FLAGS.num_summary_images])
+      tf.summary.scalar('total_loss', total_loss)
+      tf.summary.scalar('learning_rate', learning_rate)
+      tf.summary.scalar('current_epoch',
+                        tf.to_double(global_step) / steps_per_epoch)
+
+      # Training
+      is_chief = FLAGS.task == 0
+
+      scaffold = tf.train.Scaffold(
+          init_fn=_get_finetuning_init_fn(variable_averages))
+      hooks = [
+          tf.train.LoggingTensorHook({'total_loss': total_loss,
+                                      'global_step': global_step},
+                                     every_n_iter=1),
+          tf.train.NanTensorHook(total_loss),
+      ]
+      chief_only_hooks = [
+          tf.train.SummarySaverHook(save_steps=FLAGS.save_summaries_steps,
+                                    save_secs=FLAGS.save_summaries_secs,
+                                    output_dir=FLAGS.output_dir,
+                                    scaffold=scaffold),
+          tf.train.CheckpointSaverHook(FLAGS.output_dir,
+                                       save_steps=FLAGS.save_model_steps,
+                                       scaffold=scaffold),
+      ]
+
+      if FLAGS.max_steps > 0:
+        hooks.append(
+            tf.train.StopAtStepHook(last_step=FLAGS.max_steps))
+
+      # hook for sync replica training
+      hooks.append(optimizer.make_session_run_hook(is_chief))
+
+      with tf.train.MonitoredTrainingSession(
+          master=FLAGS.master,
+          is_chief=is_chief,
+          checkpoint_dir=FLAGS.output_dir,
+          scaffold=scaffold,
+          hooks=hooks,
+          chief_only_hooks=chief_only_hooks,
+          save_checkpoint_secs=None,
+          save_summaries_steps=None,
+          save_summaries_secs=None) as session:
+        while not session.should_stop():
+          session.run([train_op])
+
+
+if __name__ == '__main__':
+  app.run(main)