Merge pull request #7 from tensorflow/master

updated

CODEOWNERS

-* @tensorflow/tf-garden-team
+* @tensorflow/tf-garden-team @tensorflow/tf-model-garden-team
 /official/ @rachellj218 @saberkun
 /official/bert @saberkun @hongjunChoi @rachellj218
+/research/adv_imagenet_models/ @alexeykurakin
 /research/adversarial_crypto/ @dave-andersen
-/research/adversarial_logit_pairing/ @AlexeyKurakin
+/research/adversarial_logit_pairing/ @alexeykurakin
 /research/adversarial_text/ @rsepassi @a-dai
-/research/adv_imagenet_models/ @AlexeyKurakin
 /research/attention_ocr/ @alexgorban
 /research/audioset/ @plakal @dpwe
 /research/autoaugment/* @barretzoph
@@ -14,10 +14,13 @@
 /research/compression/ @nmjohn
 /research/cvt_text/ @clarkkev @lmthang
 /research/deep_contextual_bandits/ @rikel
+/research/deep_speech/ @yhliang2018
 /research/deeplab/ @aquariusjay @yknzhu @gpapan
 /research/delf/ @andrefaraujo
 /research/domain_adaptation/ @bousmalis @dmrd
 /research/efficient-hrl/ @ofirnachum
+/research/feelvos/ @pvoigtlaender @yuningchai @aquariusjay
+/research/fivo/ @dieterichlawson
 /research/global_objectives/ @mackeya-google
 /research/im2txt/ @cshallue
 /research/inception/ @shlens @vincentvanhoucke
@@ -26,7 +29,7 @@
 /research/learning_to_remember_rare_events/ @lukaszkaiser @ofirnachum
 /research/learning_unsupervised_learning/ @lukemetz @nirum
 /research/lexnet_nc/ @vered1986 @waterson
-/research/lfads/ @jazcollins @susillo
+/research/lfads/ @jazcollins @sussillo
 /research/lm_1b/ @oriolvinyals @panyx0718
 /research/lm_commonsense/ @thtrieu
 /research/lstm_object_detection/ @dreamdragon @masonliuw @yinxiaoli @yongzhe2160
@@ -39,9 +42,10 @@
 /research/object_detection/ @jch1 @tombstone @derekjchow @jesu9 @dreamdragon @pkulzc
 /research/pcl_rl/ @ofirnachum
 /research/ptn/ @xcyan @arkanath @hellojas @honglaklee
+/research/qa_kg/ @yuyuz
 /research/real_nvp/ @laurent-dinh
 /research/rebar/ @gjtucker
-/research/resnet/ @panyx0718
+/research/sentiment_analysis/ @sculd
 /research/seq2species/ @apbusia @depristo
 /research/skip_thoughts/ @cshallue
 /research/slim/ @sguada @nathansilberman
@@ -50,15 +54,7 @@
 /research/struct2depth/ @aneliaangelova
 /research/swivel/ @waterson
 /research/tcn/ @coreylynch @sermanet
-/research/tensorrt/ @karmel
 /research/textsum/ @panyx0718 @peterjliu
 /research/transformer/ @daviddao
 /research/vid2depth/ @rezama
 /research/video_prediction/ @cbfinn
-/research/fivo/ @dieterichlawson
-/samples/ @MarkDaoust @lamberta
-/samples/languages/java/ @asimshankar
-/tutorials/embedding/ @zffchen78 @a-dai
-/tutorials/image/ @sherrym @shlens
-/tutorials/image/cifar10_estimator/ @protoget
-/tutorials/rnn/ @lukaszkaiser @ebrevdo

README.md

-# TensorFlow Models
+![Logo](https://storage.googleapis.com/model_garden_artifacts/TF_Model_Garden.png)
 
-This repository contains a number of different models implemented in [TensorFlow](https://www.tensorflow.org):
+# Welcome to the Model Garden for TensorFlow
 
-The [official models](official) are a collection of example models that use TensorFlow 2's high-level APIs. They are intended to be well-maintained, tested, and kept up to date with the latest stable TensorFlow API. They should also be reasonably optimized for fast performance while still being easy to read. We especially recommend newer TensorFlow users to start here.
+The TensorFlow Model Garden is a repository with a number of different implementations of state-of-the-art (SOTA) models and modeling solutions for TensorFlow users. We aim to demonstrate the best practices for modeling so that TensorFlow users can take full advantage of TensorFlow for their research and product development.
 
-The [research models](https://github.com/tensorflow/models/tree/master/research) are a large collection of models implemented in TensorFlow by researchers. They are not officially supported or available in release branches; it is up to the individual researchers to maintain the models and/or provide support on issues and pull requests.
+## Structure
+
+| Folder | Description |
+|-----------|-------------|
+| [official](official) | • **A collection of example implementations for SOTA models using the latest TensorFlow 2's high-level APIs**<br />• Officially maintained, supported, and kept up to date with the latest TensorFlow 2 APIs<br />• Reasonably optimized for fast performance while still being easy to read |
+| [research](research) | • A collection of research model implementations in TensorFlow 1 or 2 by researchers<br />• Up to the individual researchers to maintain the model implementations and/or provide support on issues and pull requests |
 
 ## Contribution guidelines
 
-If you want to contribute to models, be sure to review the [contribution guidelines](CONTRIBUTING.md).
+If you want to contribute to models, please review the [contribution guidelines](CONTRIBUTING.md).
 
 ## License
 

official/README-TPU.md

-# Offically Supported TensorFlow 2.1 Models on Cloud TPU
+# Offically Supported TensorFlow 2.1+ Models on Cloud TPU
 
 ## Natural Language Processing
 
 *   [bert](nlp/bert): A powerful pre-trained language representation model:
     BERT, which stands for Bidirectional Encoder Representations from
     Transformers.
-    [BERT FineTuning with Cloud TPU](https://cloud.google.com/tpu/docs/tutorials/bert-2.x) provides step by step instructions on Cloud TPU training. You can look [Bert MNLI Tensorboard.dev metrics](https://tensorboard.dev/experiment/mIah5lppTASvrHqWrdr6NA) for MNLI fine tuning task.
+    [BERT FineTuning with Cloud TPU](https://cloud.google.com/tpu/docs/tutorials/bert-2.x) provides step by step instructions on Cloud TPU training. You can look [Bert MNLI Tensorboard.dev metrics](https://tensorboard.dev/experiment/LijZ1IrERxKALQfr76gndA) for MNLI fine tuning task.
 *   [transformer](nlp/transformer): A transformer model to translate the WMT
     English to German dataset.
         [Training transformer on Cloud TPU](https://cloud.google.com/tpu/docs/tutorials/transformer-2.x) for step by step instructions on Cloud TPU training.
 
 ## Computer Vision
 
+*   [efficientnet](vision/image_classification): A family of convolutional
+    neural networks that scale by balancing network depth, width, and
+    resolution and can be used to classify ImageNet's dataset of 1000 classes.
+    See [Tensorboard.dev training metrics](https://tensorboard.dev/experiment/KnaWjrq5TXGfv0NW5m7rpg/#scalars).
 *   [mnist](vision/image_classification): A basic model to classify digits
     from the MNIST dataset. See [Running MNIST on Cloud TPU](https://cloud.google.com/tpu/docs/tutorials/mnist-2.x) tutorial and [Tensorboard.dev metrics](https://tensorboard.dev/experiment/mIah5lppTASvrHqWrdr6NA).
+*   [mask-rcnn](vision/detection): An object detection and instance segmentation model. See [Tensorboard.dev training metrics](https://tensorboard.dev/experiment/LH7k0fMsRwqUAcE09o9kPA).
 *   [resnet](vision/image_classification): A deep residual network that can
     be used to classify ImageNet's dataset of 1000 classes.
     See [Training ResNet on Cloud TPU](https://cloud.google.com/tpu/docs/tutorials/resnet-2.x) tutorial and [Tensorboard.dev metrics](https://tensorboard.dev/experiment/CxlDK8YMRrSpYEGtBRpOhg).

official/README.md

-# TensorFlow Official Models
+![Logo](https://storage.googleapis.com/model_garden_artifacts/TF_Model_Garden.png)
 
-The TensorFlow official models are a collection of models that use
-TensorFlow's high-level APIs. They are intended to be well-maintained, tested,
-and kept up to date with the latest TensorFlow API. They should also be
-reasonably optimized for fast performance while still being easy to read.
+# TensorFlow Official Models
 
-These models are used as end-to-end tests, ensuring that the models run with the
-same or improved speed and performance with each new TensorFlow build.
+The TensorFlow official models are a collection of models
+that use TensorFlow’s high-level APIs.
+They are intended to be well-maintained, tested, and kept up to date
+with the latest TensorFlow API.
+They should also be reasonably optimized for fast performance while still
+being easy to read.
+These models are used as end-to-end tests, ensuring that the models run
+with the same or improved speed and performance with each new TensorFlow build.
 
-## Tensorflow releases
+## Model Implementations
 
-The master branch of the models are **in development** with TensorFlow 2.x, and
-they target the
-[nightly binaries](https://github.com/tensorflow/tensorflow#installation) built
-from the
-[master branch of TensorFlow](https://github.com/tensorflow/tensorflow/tree/master).
-You may start from installing with pip:
+### Natural Language Processing
 
-```shell
-pip3 install tf-nightly
-```
+| Model | Description | Reference |
+| ----- | ----------- | --------- |
+| [ALBERT](nlp/albert) | A Lite BERT for Self-supervised Learning of Language Representations | [arXiv:1909.11942](https://arxiv.org/abs/1909.11942) |
+| [BERT](nlp/bert) | A powerful pre-trained language representation model: BERT (Bidirectional Encoder Representations from Transformers) | [arXiv:1810.04805](https://arxiv.org/abs/1810.04805) |
+| [NHNet](nlp/nhnet) | A transformer-based multi-sequence to sequence model: Generating Representative Headlines for News Stories | [arXiv:2001.09386](https://arxiv.org/abs/2001.09386) |
+| [Transformer](nlp/transformer) | A transformer model to translate the WMT English to German dataset | [arXiv:1706.03762](https://arxiv.org/abs/1706.03762) |
+| [XLNet](nlp/xlnet) | XLNet: Generalized Autoregressive Pretraining for Language Understanding | [arXiv:1906.08237](https://arxiv.org/abs/1906.08237) |
 
-**Stable versions** of the official models targeting releases of TensorFlow are
-available as tagged branches or
-[downloadable releases](https://github.com/tensorflow/models/releases). Model
-repository version numbers match the target TensorFlow release, such that
-[release v2.1.0](https://github.com/tensorflow/models/releases/tag/v2.1.0) are
-compatible with
-[TensorFlow v2.1.0](https://github.com/tensorflow/tensorflow/releases/tag/v2.1.0).
+### Computer Vision
 
-If you are on a version of TensorFlow earlier than 1.4, please
-[update your installation](https://www.tensorflow.org/install/).
+| Model | Description | Reference |
+| ----- | ----------- | --------- |
+| [MNIST](vision/image_classification) | A basic model to classify digits from the MNIST dataset | [Link](http://yann.lecun.com/exdb/mnist/) |
+| [ResNet](vision/image_classification) | A deep residual network for image recognition | [arXiv:1512.03385](https://arxiv.org/abs/1512.03385) |
+| [RetinaNet](vision/detection) | A fast and powerful object detector | [arXiv:1708.02002](https://arxiv.org/abs/1708.02002) |
+| [Mask R-CNN](vision/detection) | An object detection and instance segmentation model | [arXiv:1703.06870](https://arxiv.org/abs/1703.06870) |
 
-## Requirements
+### Other models
 
-Please follow the below steps before running models in this repo:
+| Model | Description | Reference |
+| ----- | ----------- | --------- |
+| [NCF](recommendation) | Neural Collaborative Filtering model for recommendation tasks | [arXiv:1708.05031](https://arxiv.org/abs/1708.05031) |
 
-1.  TensorFlow
-    [nightly binaries](https://github.com/tensorflow/tensorflow#installation)
+---
 
-2.  If users would like to clone this repo but do not care about change history,
-please consider:
+## How to get started with the Model Garden official models
 
-  ```shell
-  export repo_version="master"
-  git clone -b ${repo_version} https://github.com/tensorflow/models.git --depth=1
-  ```
+* The models in the master branch are developed using TensorFlow 2,
+and they target the TensorFlow [nightly binaries](https://github.com/tensorflow/tensorflow#installation)
+built from the
+[master branch of TensorFlow](https://github.com/tensorflow/tensorflow/tree/master).
+* The stable versions targeting releases of TensorFlow are available
+as tagged branches or [downloadable releases](https://github.com/tensorflow/models/releases).
+* Model repository version numbers match the target TensorFlow release,
+such that
+[release v2.1.0](https://github.com/tensorflow/models/releases/tag/v2.1.0)
+are compatible with
+[TensorFlow v2.1.0](https://github.com/tensorflow/tensorflow/releases/tag/v2.1.0).
 
-3.  Add the top-level ***/models*** folder to the Python path with the command:
+Please follow the below steps before running models in this repository.
 
-  ```shell
-  export PYTHONPATH=$PYTHONPATH:/path/to/models
-  ```
+### Requirements
 
-  Using Colab:
+* The latest TensorFlow Model Garden release and TensorFlow 2
+  * If you are on a version of TensorFlow earlier than 2.1, please
+upgrade your TensorFlow to [the latest TensorFlow 2](https://www.tensorflow.org/install/).
 
-  ```python
-  import os
-  os.environ['PYTHONPATH'] += ":/path/to/models"
-  ```
+```shell
+pip3 install tf-nightly
+```
 
-4.  Install dependencies:
+### Installation
 
-  ```shell
-  pip3 install --user -r official/requirements.txt
-  ```
+#### Method 1: Install the TensorFlow Model Garden pip package
 
+**tf-models-nightly** is the nightly Model Garden package
+created daily automatically. pip will install all models
+and dependencies automatically.
 
-To make Official Models easier to use, we are planning to create a pip
-installable Official Models package. This is being tracked in
-[#917](https://github.com/tensorflow/models/issues/917).
+```shell
+pip install tf-models-nightly
+```
 
-## Available models
+Please check out our [example](colab/bert.ipynb)
+to learn how to use a PIP package.
 
-**NOTE: For Officially Supported TPU models please check [README-TPU](README-TPU.md).**
+#### Method 2: Clone the source
 
-**NOTE:** Please make sure to follow the steps in the
-[Requirements](#requirements) section.
+1. Clone the GitHub repository:
 
-### Natural Language Processing
+```shell
+git clone https://github.com/tensorflow/models.git
+```
 
-*   [albert](nlp/albert): A Lite BERT for Self-supervised Learning of Language
-    Representations.
-*   [bert](nlp/bert): A powerful pre-trained language representation model:
-    BERT, which stands for Bidirectional Encoder Representations from
-    Transformers.
-*   [transformer](nlp/transformer): A transformer model to translate the WMT English
-    to German dataset.
-*   [xlnet](nlp/xlnet): XLNet: Generalized Autoregressive Pretraining for
-    Language Understanding.
+2. Add the top-level ***/models*** folder to the Python path.
 
-### Computer Vision
+```shell
+export PYTHONPATH=$PYTHONPATH:/path/to/models
+```
 
-*   [mnist](vision/image_classification): A basic model to classify digits from
-    the MNIST dataset.
-*   [resnet](vision/image_classification): A deep residual network that can be
-    used to classify both CIFAR-10 and ImageNet's dataset of 1000 classes.
-*   [retinanet](vision/detection): A fast and powerful object detector.
+If you are using a Colab notebook, please set the Python path with os.environ.
 
-### Others
+```python
+import os
+os.environ['PYTHONPATH'] += ":/path/to/models"
+```
 
-*   [ncf](recommendation): Neural Collaborative Filtering model for
-    recommendation tasks.
+3. Install other dependencies
 
-Models that will not update to TensorFlow 2.x stay inside R1 directory:
+```shell
+pip3 install --user -r official/requirements.txt
+```
 
-*   [boosted_trees](r1/boosted_trees): A Gradient Boosted Trees model to
-    classify higgs boson process from HIGGS Data Set.
-*   [wide_deep](r1/wide_deep): A model that combines a wide model and deep
-    network to classify census income data.
+---
 
 ## More models to come!
 
-We are in the progress to revamp official model garden with TensorFlow 2.0 and
-Keras. In the near future, we will bring:
+The team is actively developing new models.
+In the near future, we will add:
 
-*   State-of-the-art language understanding models: XLNet, GPT2, and more
-    members in Transformer family.
-*   Start-of-the-art image classification models: EfficientNet, MnasNet and
-    variants.
-*   A set of excellent objection detection models.
+- State-of-the-art language understanding models:
+  More members in Transformer family
+- Start-of-the-art image classification models:
+  EfficientNet, MnasNet and variants.
+- A set of excellent objection detection models.
 
 If you would like to make any fixes or improvements to the models, please
 [submit a pull request](https://github.com/tensorflow/models/compare).
 
-## New Models
+---
+
+## Contributions
 
-The team is actively working to add new models to the repository. Every model
-should follow the following guidelines, to uphold the our objectives of
-readable, usable, and maintainable code.
+Every model should follow our guidelines to uphold our objectives of readable,
+usable, and maintainable code.
 
-**General guidelines**
+### General Guidelines
 
-* Code should be well documented and tested.
-* Runnable from a blank environment with relative ease.
-* Trainable on: single GPU/CPU (baseline), multiple GPUs, TPU
-* Compatible with Python 3 (using [six](https://pythonhosted.org/six/) when
-  being compatible with Python 2 is necessary)
-* Conform to [Google Python Style Guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md)
+- Code should be well documented and tested.
+- Runnable from a blank environment with ease.
+- Trainable on: single GPU/CPU (baseline), multiple GPUs & TPUs
+- Compatible with Python 3 (using [six](https://pythonhosted.org/six/)
+when being compatible with Python 2 is necessary)
+- Conform to
+  [Google Python Style Guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md)
 
-**Implementation guidelines**
+### Implementation Guidelines
 
-These guidelines exist so the model implementations are consistent for better
-readability and maintainability.
+These guidelines are to ensure consistent model implementations for
+better readability and maintainability.
 
-*   Use [common utility functions](utils)
-*   Export SavedModel at the end of training.
-*   Consistent flags and flag-parsing library
-    ([read more here](utils/flags/guidelines.md))
-*   Produce benchmarks and logs ([read more here](utils/logs/guidelines.md))
+- Use [common utility functions](utils)
+- Export SavedModel at the end of the training.
+- Consistent flags and flag-parsing library ([read more here](utils/flags/guidelines.md))

official/utils/testing/benchmark_wrappers.py → official/benchmark/benchmark_wrappers.py

 # Lint as: python3
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
 """Utils to annotate and trace benchmarks."""
 
 from __future__ import absolute_import

official/benchmark/bert_benchmark.py

@@ -34,7 +34,7 @@ from official.benchmark import bert_benchmark_utils as benchmark_utils
 from official.nlp.bert import configs
 from official.nlp.bert import run_classifier
 from official.utils.misc import distribution_utils
-from official.utils.testing import benchmark_wrappers
+from official.benchmark import benchmark_wrappers
 
 # pylint: disable=line-too-long
 PRETRAINED_CHECKPOINT_PATH = 'gs://cloud-tpu-checkpoints/bert/keras_bert/uncased_L-24_H-1024_A-16/bert_model.ckpt'
@@ -56,6 +56,7 @@ class BertClassifyBenchmarkBase(benchmark_utils.BertBenchmarkBase):
     self.num_epochs = None
     self.num_steps_per_epoch = None
     self.tpu = tpu
+    FLAGS.steps_per_loop = 50
 
   @flagsaver.flagsaver
   def _run_bert_classifier(self, callbacks=None, use_ds=True):
@@ -81,8 +82,6 @@ class BertClassifyBenchmarkBase(benchmark_utils.BertBenchmarkBase):
           distribution_strategy='mirrored' if use_ds else 'off',
           num_gpus=self.num_gpus)
 
-    steps_per_loop = 50
-
     max_seq_length = input_meta_data['max_seq_length']
     train_input_fn = run_classifier.get_dataset_fn(
         FLAGS.train_data_path,
@@ -101,7 +100,7 @@ class BertClassifyBenchmarkBase(benchmark_utils.BertBenchmarkBase):
         FLAGS.model_dir,
         epochs,
         steps_per_epoch,
-        steps_per_loop,
+        FLAGS.steps_per_loop,
         eval_steps,
         warmup_steps,
         FLAGS.learning_rate,

official/benchmark/bert_benchmark_utils.py

@@ -23,11 +23,11 @@ import time
 # pylint: disable=g-bad-import-order
 import numpy as np
 from absl import flags
-import tensorflow.compat.v2 as tf
+import tensorflow as tf
 # pylint: enable=g-bad-import-order
 
 from official.utils.flags import core as flags_core
-from official.utils.testing.perfzero_benchmark import PerfZeroBenchmark
+from official.benchmark.perfzero_benchmark import PerfZeroBenchmark
 
 FLAGS = flags.FLAGS
 

official/benchmark/bert_squad_benchmark.py

@@ -33,7 +33,7 @@ from official.benchmark import bert_benchmark_utils as benchmark_utils
 from official.nlp.bert import run_squad
 from official.utils.misc import distribution_utils
 from official.utils.misc import keras_utils
-from official.utils.testing import benchmark_wrappers
+from official.benchmark import benchmark_wrappers
 
 
 # pylint: disable=line-too-long
@@ -104,7 +104,6 @@ class BertSquadBenchmarkBase(benchmark_utils.BertBenchmarkBase):
   @flagsaver.flagsaver
   def _train_squad(self, run_eagerly=False, ds_type='mirrored'):
     """Runs BERT SQuAD training. Uses mirrored strategy by default."""
-    assert tf.version.VERSION.startswith('2.')
     self._init_gpu_and_data_threads()
     input_meta_data = self._read_input_meta_data_from_file()
     strategy = self._get_distribution_strategy(ds_type)
@@ -118,7 +117,6 @@ class BertSquadBenchmarkBase(benchmark_utils.BertBenchmarkBase):
   @flagsaver.flagsaver
   def _evaluate_squad(self, ds_type='mirrored'):
     """Runs BERT SQuAD evaluation. Uses mirrored strategy by default."""
-    assert tf.version.VERSION.startswith('2.')
     self._init_gpu_and_data_threads()
     input_meta_data = self._read_input_meta_data_from_file()
     strategy = self._get_distribution_strategy(ds_type)
@@ -128,7 +126,7 @@ class BertSquadBenchmarkBase(benchmark_utils.BertBenchmarkBase):
     eval_metrics = run_squad.eval_squad(strategy=strategy,
                                         input_meta_data=input_meta_data)
     # Use F1 score as reported evaluation metric.
-    self.eval_metrics = eval_metrics['f1']
+    self.eval_metrics = eval_metrics['final_f1']
 
 
 class BertSquadBenchmarkReal(BertSquadBenchmarkBase):
@@ -254,7 +252,7 @@ class BertSquadBenchmarkReal(BertSquadBenchmarkBase):
     self._setup()
     self.num_gpus = 8
     FLAGS.model_dir = self._get_model_dir('benchmark_8_gpu_squad')
-    FLAGS.train_batch_size = 32
+    FLAGS.train_batch_size = 24
     FLAGS.tf_gpu_thread_mode = 'gpu_private'
 
     self._run_and_report_benchmark()

official/benchmark/keras_benchmark.py

@@ -19,9 +19,8 @@ from __future__ import division
 from __future__ import print_function
 
 import tensorflow as tf
-
+from official.benchmark.perfzero_benchmark import PerfZeroBenchmark
 from official.utils.flags import core as flags_core
-from official.utils.testing.perfzero_benchmark import PerfZeroBenchmark
 
 
 class KerasBenchmark(PerfZeroBenchmark):
@@ -32,7 +31,6 @@ class KerasBenchmark(PerfZeroBenchmark):
                default_flags=None,
                flag_methods=None,
                tpu=None):
-    assert tf.version.VERSION.startswith('2.')
     super(KerasBenchmark, self).__init__(
         output_dir=output_dir,
         default_flags=default_flags,

official/benchmark/keras_cifar_benchmark.py

@@ -23,7 +23,7 @@ from absl import flags
 import tensorflow as tf  # pylint: disable=g-bad-import-order
 
 from official.benchmark import keras_benchmark
-from official.utils.testing import benchmark_wrappers
+from official.benchmark import benchmark_wrappers
 from official.benchmark.models import resnet_cifar_main
 
 MIN_TOP_1_ACCURACY = 0.929

official/benchmark/models/resnet_cifar_main.py

@@ -23,12 +23,13 @@ from absl import flags
 from absl import logging
 import numpy as np
 import tensorflow as tf
+from official.benchmark.models import cifar_preprocessing
 from official.benchmark.models import resnet_cifar_model
+from official.benchmark.models import synthetic_util
 from official.utils.flags import core as flags_core
 from official.utils.logs import logger
 from official.utils.misc import distribution_utils
 from official.utils.misc import keras_utils
-from official.vision.image_classification.resnet import cifar_preprocessing
 from official.vision.image_classification.resnet import common
 
 
@@ -159,7 +160,7 @@ def run(flags_obj):
   strategy_scope = distribution_utils.get_strategy_scope(strategy)
 
   if flags_obj.use_synthetic_data:
-    distribution_utils.set_up_synthetic_data()
+    synthetic_util.set_up_synthetic_data()
     input_fn = common.get_synth_input_fn(
         height=cifar_preprocessing.HEIGHT,
         width=cifar_preprocessing.WIDTH,
@@ -168,7 +169,7 @@ def run(flags_obj):
         dtype=flags_core.get_tf_dtype(flags_obj),
         drop_remainder=True)
   else:
-    distribution_utils.undo_set_up_synthetic_data()
+    synthetic_util.undo_set_up_synthetic_data()
     input_fn = cifar_preprocessing.input_fn
 
   train_input_dataset = input_fn(

official/benchmark/models/resnet_cifar_test.py

@@ -24,10 +24,10 @@ import tensorflow as tf
 
 from tensorflow.python.eager import context
 from tensorflow.python.platform import googletest
+from official.benchmark.models import cifar_preprocessing
 from official.benchmark.models import resnet_cifar_main
 from official.utils.misc import keras_utils
 from official.utils.testing import integration
-from official.vision.image_classification.resnet import cifar_preprocessing
 
 
 class KerasCifarTest(googletest.TestCase):

official/vision/image_classification/resnet/resnet_imagenet_main.py → official/benchmark/models/resnet_imagenet_main.py

-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -98,7 +98,6 @@ def run(flags_obj):
 
   # pylint: disable=protected-access
   if flags_obj.use_synthetic_data:
-    distribution_utils.set_up_synthetic_data()
     input_fn = common.get_synth_input_fn(
         height=imagenet_preprocessing.DEFAULT_IMAGE_SIZE,
         width=imagenet_preprocessing.DEFAULT_IMAGE_SIZE,
@@ -107,7 +106,6 @@ def run(flags_obj):
         dtype=dtype,
         drop_remainder=True)
   else:
-    distribution_utils.undo_set_up_synthetic_data()
     input_fn = imagenet_preprocessing.input_fn
 
   # When `enable_xla` is True, we always drop the remainder of the batches

official/benchmark/models/resnet_imagenet_test.py

+# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Test the keras ResNet model with ImageNet data."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from absl.testing import parameterized
+import tensorflow as tf
+
+from tensorflow.python.eager import context
+from official.benchmark.models import resnet_imagenet_main
+from official.utils.misc import keras_utils
+from official.utils.testing import integration
+from official.vision.image_classification.resnet import imagenet_preprocessing
+
+
+@parameterized.parameters(
+    "resnet",
+    # "resnet_polynomial_decay",  b/151854314
+    "mobilenet",
+    # "mobilenet_polynomial_decay"  b/151854314
+)
+class KerasImagenetTest(tf.test.TestCase):
+  """Unit tests for Keras Models with ImageNet."""
+  _default_flags_dict = [
+      "-batch_size", "4",
+      "-train_steps", "1",
+      "-use_synthetic_data", "true",
+      "-data_format", "channels_last",
+  ]
+  _extra_flags_dict = {
+      "resnet": [
+          "-model", "resnet50_v1.5",
+          "-optimizer", "resnet50_default",
+      ],
+      "resnet_polynomial_decay": [
+          "-model", "resnet50_v1.5",
+          "-optimizer", "resnet50_default",
+          "-pruning_method", "polynomial_decay",
+      ],
+      "mobilenet": [
+          "-model", "mobilenet",
+          "-optimizer", "mobilenet_default",
+      ],
+      "mobilenet_polynomial_decay": [
+          "-model", "mobilenet",
+          "-optimizer", "mobilenet_default",
+          "-pruning_method", "polynomial_decay",
+      ],
+  }
+  _tempdir = None
+
+  @classmethod
+  def setUpClass(cls):  # pylint: disable=invalid-name
+    super(KerasImagenetTest, cls).setUpClass()
+    resnet_imagenet_main.define_imagenet_keras_flags()
+
+  def setUp(self):
+    super(KerasImagenetTest, self).setUp()
+    imagenet_preprocessing.NUM_IMAGES["validation"] = 4
+    self.policy = \
+        tf.compat.v2.keras.mixed_precision.experimental.global_policy()
+
+  def tearDown(self):
+    super(KerasImagenetTest, self).tearDown()
+    tf.io.gfile.rmtree(self.get_temp_dir())
+    tf.compat.v2.keras.mixed_precision.experimental.set_policy(self.policy)
+
+  def get_extra_flags_dict(self, flags_key):
+    return self._extra_flags_dict[flags_key] + self._default_flags_dict
+
+  def test_end_to_end_no_dist_strat(self, flags_key):
+    """Test Keras model with 1 GPU, no distribution strategy."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    extra_flags = [
+        "-distribution_strategy", "off",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  def test_end_to_end_graph_no_dist_strat(self, flags_key):
+    """Test Keras model in legacy graph mode with 1 GPU, no dist strat."""
+    extra_flags = [
+        "-enable_eager", "false",
+        "-distribution_strategy", "off",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  def test_end_to_end_1_gpu(self, flags_key):
+    """Test Keras model with 1 GPU."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    if context.num_gpus() < 1:
+      self.skipTest(
+          "{} GPUs are not available for this test. {} GPUs are available".
+          format(1, context.num_gpus()))
+
+    extra_flags = [
+        "-num_gpus", "1",
+        "-distribution_strategy", "mirrored",
+        "-enable_checkpoint_and_export", "1",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  def test_end_to_end_1_gpu_fp16(self, flags_key):
+    """Test Keras model with 1 GPU and fp16."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    if context.num_gpus() < 1:
+      self.skipTest(
+          "{} GPUs are not available for this test. {} GPUs are available"
+          .format(1, context.num_gpus()))
+
+    extra_flags = [
+        "-num_gpus", "1",
+        "-dtype", "fp16",
+        "-distribution_strategy", "mirrored",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    if "polynomial_decay" in extra_flags:
+      self.skipTest("Pruning with fp16 is not currently supported.")
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  def test_end_to_end_2_gpu(self, flags_key):
+    """Test Keras model with 2 GPUs."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    if context.num_gpus() < 2:
+      self.skipTest(
+          "{} GPUs are not available for this test. {} GPUs are available".
+          format(2, context.num_gpus()))
+
+    extra_flags = [
+        "-num_gpus", "2",
+        "-distribution_strategy", "mirrored",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  def test_end_to_end_xla_2_gpu(self, flags_key):
+    """Test Keras model with XLA and 2 GPUs."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    if context.num_gpus() < 2:
+      self.skipTest(
+          "{} GPUs are not available for this test. {} GPUs are available".
+          format(2, context.num_gpus()))
+
+    extra_flags = [
+        "-num_gpus", "2",
+        "-enable_xla", "true",
+        "-distribution_strategy", "mirrored",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  def test_end_to_end_2_gpu_fp16(self, flags_key):
+    """Test Keras model with 2 GPUs and fp16."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    if context.num_gpus() < 2:
+      self.skipTest(
+          "{} GPUs are not available for this test. {} GPUs are available".
+          format(2, context.num_gpus()))
+
+    extra_flags = [
+        "-num_gpus", "2",
+        "-dtype", "fp16",
+        "-distribution_strategy", "mirrored",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    if "polynomial_decay" in extra_flags:
+      self.skipTest("Pruning with fp16 is not currently supported.")
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  def test_end_to_end_xla_2_gpu_fp16(self, flags_key):
+    """Test Keras model with XLA, 2 GPUs and fp16."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    if context.num_gpus() < 2:
+      self.skipTest(
+          "{} GPUs are not available for this test. {} GPUs are available".
+          format(2, context.num_gpus()))
+
+    extra_flags = [
+        "-num_gpus", "2",
+        "-dtype", "fp16",
+        "-enable_xla", "true",
+        "-distribution_strategy", "mirrored",
+    ]
+    extra_flags = extra_flags + self.get_extra_flags_dict(flags_key)
+
+    if "polynomial_decay" in extra_flags:
+      self.skipTest("Pruning with fp16 is not currently supported.")
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+
+if __name__ == "__main__":
+  tf.test.main()

official/benchmark/models/resnet_imagenet_test_tpu.py

+# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Test the keras ResNet model with ImageNet data on TPU."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from absl.testing import parameterized
+import tensorflow as tf
+from official.benchmark.models import resnet_imagenet_main
+from official.utils.misc import keras_utils
+from official.utils.testing import integration
+from official.vision.image_classification.resnet import imagenet_preprocessing
+
+
+class KerasImagenetTest(tf.test.TestCase, parameterized.TestCase):
+  """Unit tests for Keras Models with ImageNet."""
+
+  _extra_flags_dict = {
+      "resnet": [
+          "-batch_size", "4",
+          "-train_steps", "1",
+          "-use_synthetic_data", "true"
+          "-model", "resnet50_v1.5",
+          "-optimizer", "resnet50_default",
+      ],
+      "resnet_polynomial_decay": [
+          "-batch_size", "4",
+          "-train_steps", "1",
+          "-use_synthetic_data", "true",
+          "-model", "resnet50_v1.5",
+          "-optimizer", "resnet50_default",
+          "-pruning_method", "polynomial_decay",
+      ],
+  }
+  _tempdir = None
+
+  @classmethod
+  def setUpClass(cls):  # pylint: disable=invalid-name
+    super(KerasImagenetTest, cls).setUpClass()
+    resnet_imagenet_main.define_imagenet_keras_flags()
+
+  def setUp(self):
+    super(KerasImagenetTest, self).setUp()
+    imagenet_preprocessing.NUM_IMAGES["validation"] = 4
+    self.policy = \
+        tf.compat.v2.keras.mixed_precision.experimental.global_policy()
+
+  def tearDown(self):
+    super(KerasImagenetTest, self).tearDown()
+    tf.io.gfile.rmtree(self.get_temp_dir())
+    tf.compat.v2.keras.mixed_precision.experimental.set_policy(self.policy)
+
+  @parameterized.parameters([
+      "resnet",
+      # "resnet_polynomial_decay"  b/151854314
+  ])
+  def test_end_to_end_tpu(self, flags_key):
+    """Test Keras model with TPU distribution strategy."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    extra_flags = [
+        "-distribution_strategy", "tpu",
+        "-data_format", "channels_last",
+        "-enable_checkpoint_and_export", "1",
+    ]
+    extra_flags = extra_flags + self._extra_flags_dict[flags_key]
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+  @parameterized.parameters(["resnet"])
+  def test_end_to_end_tpu_bf16(self, flags_key):
+    """Test Keras model with TPU and bfloat16 activation."""
+    config = keras_utils.get_config_proto_v1()
+    tf.compat.v1.enable_eager_execution(config=config)
+
+    extra_flags = [
+        "-distribution_strategy", "tpu",
+        "-data_format", "channels_last",
+        "-dtype", "bf16",
+    ]
+    extra_flags = extra_flags + self._extra_flags_dict[flags_key]
+
+    integration.run_synthetic(
+        main=resnet_imagenet_main.run,
+        tmp_root=self.get_temp_dir(),
+        extra_flags=extra_flags
+    )
+
+
+if __name__ == "__main__":
+  tf.test.main()

official/staging/shakespeare/shakespeare_main.py → official/benchmark/models/shakespeare/shakespeare_main.py

@@ -47,7 +47,6 @@ def define_flags():
                          epochs_between_evals=False,
                          stop_threshold=False,
                          num_gpu=True,
-                         hooks=False,
                          export_dir=False,
                          run_eagerly=True,
                          distribution_strategy=True)