GitLab

249500988  by hongkuny<hongkuny@google.com>:

    Lints

--

PiperOrigin-RevId: 249500988

* Add big tests.

* fix super

* Add fp16, increase 8xGPU batch-sizes

* Adding the rest of the fp16 tests.

* Big accuracy test batch_perf_gpu

* fix docstrings

* add _run_and_report

* Edited docstrings

* Merged commit includes the following changes:
249218656  by tianlin<tianlin@google.com>:

    Deal with imports, fix a typo and make unit tests fast.

--
249198645  by tianlin<tianlin@google.com>:

    Trivial: Remove one empty line before "import tensorflow"

--
249195490  by tianlin<tianlin@google.com>:

    Initialize Transformer TF V2 Model with Keras subclassing implementation. (Compatible with TF V1)

--
249195008  by tianlin<tianlin@google.com>:

    Internal change

249173564  by hongkuny<hongkuny@google.com>:

    Internal change

249079258  by hongkuny<hongkuny@google.com>:

    Internal change

247691534  by haoyuzhang<haoyuzhang@google.com>:

    Internal change

247533725  by haoyuzhang<haoyuzhang@google.com>:

    Internal change

247509295  by haoyuzhang<haoyuzhang@google.com>:

    Internal change

247311355  by wangtz<wangtz@google.com>:

    Internal change

247303127  by wangtz<wangtz@google.com>:

  ...

249377254  by hongkuny<hongkuny@google.com>:

    Internal change

249373328  by hongkuny<hongkuny@google.com>:

    Clean up tf import

--
249333938  by hongkuny<hongkuny@google.com>:

    Fix tf1 import

--
249325089  by hongkuny<hongkuny@google.com>:

    BERT 2.0

--
249173564  by hongkuny<hongkuny@google.com>:

    Internal change

PiperOrigin-RevId: 249377254

* Delete accuracy if exists in eval results.

* get global_step only if it exists in eval results

This will allow one to easily reproduce a benchmark by running with the flags.

* Set the --clone_model_in_keras_dist_strat to None.  Remove the separate no_cloning benchmarks and add a couple of cloning ones.  Fixes the learning rate schedule to cache its ops per graph.

* Added 'tfdata_exp' version of all benchmarks which set
FLAGS.tf_data_experimental_slack = True. Renamed
`data_prefetch_with_slack` to `data_delay_prefetch` (haoyu's change)
to make the names more distinct.

* Add flag to resnet input pipeline and surface through
keras_imagenet_main.py

- Test passes locally python3 and test is already
    skipped for python2.

* Add FP16 and benchmarks.

* add missing run and report.

* Add loss_scale as option not included with dtype.

* move loss_scale validation under dtype conditional.

* add loss_scale to flags tested.

* Fix trivial model to work properly with fp16

* Add comment on manual casting

Previously we had one dense layer in trivial model. The weight was [224*224*3, num_classes].
Using two dense layers, the weights are [224*224*3, 1] and [1, num_classes].

* Do not report metrics in performance benchmarks

* Rename flag

* Modified tweaked tests to use tensor learning rate

* Add first benchmark and return stats.

* Remove print statements update training steps.

* Revert print T: in print statement.

* Remove print(stats)

* add 2 gpu accuracy test for base.

* Fixed total_batch_size when using gpu + gFile deprecations.

* 8 GPU test name fix

* Add 4 and 8 GPU tests.

* typo fixes.

* Clean up test names and methods.

* bleu uncased.  docstring format fix.

* Add learning rate tensor. This makes training slower

* Improve LearningRateSchedule with better efficiency

* Fix lint error

* Replace constant definition with existing one

* Enable CuDNN BatchNorm spatial persistent by default; Remove 2nd zero padding layer

* Apply scale=False and fused=True consistently to BatchNorm layers

* Undo remove padding layer

* Replace zero padding with padding attribute in max pooling for better performance

* Resolve comments

* Revert "Replace zero padding with padding attribute in max pooling for better performance"

This reverts commit ad49db057c800ecac008eec1057005bd2c08ac73.

This options allows the new tf.train.experimental.enable_mixed_precision_graph_rewrite() function to be used for fp16, instead of manual casts.

Replace per_device with per_replica and PerDevice with PerReplica, because the PerDevice concept was renamed and doesn't exist anymore. (#6693)

* Replace per_device with per_replica and PerDevice with PerReplica, because the PerReplica concept was renamed and doesn't exist anymore.

* Add accuracy check.

* Avoid double flag init, move data_dir to real data.

* Comment on lower accuracy target.

* bug fix

* bug fix

* Add benchmarks with the --cloning flag to Resnet and NFC.

* Renamed cloning to clone_model_in_keras_dist_strat. Dropped a few tests that aren't essential.

* Fixed up the formatting after re-naming the flag to a much longer  name.  Thanks, lint.
* Fixed the lint error in nfc_common.py

* Combined imagenet and cifar-10 benchmarks

* Comments and epochs_between_evals.

* Added tuned tests and cleaned up benchmark flags

* Fix names.

* Return results and add images/sec hook.

* updated doc strings for return values.

* 128 to 256 batch for FP16 test

* added more doc strings to fix lint.

* Add num_packs flag for MirroredStrategy's cross device ops.

* fix parens

* Fix lint errors and make all_reduce_alg more robust.

* Set default num_packs to 1

tf.test.is_gpu_available() should not be called in flags since this is
called before app.main() and the runtime has not yet been initialized.

* Remove contrib AllReduceCrossDeviceOps and update all_reduce_alg options with MirroredStrategy.

* cleanup

Reason: test failures because contrib is not available in V2

This reverts commit 325dd761.