GitLab

* Shard input for distribution strategy.

* Pass in input_context from real input_fn.

* Pass in input_context from real input_fn.

* Make pipeline id base 1 for better readability.

* Add support for TPUEstimator to data processing pipeline and add the ability to store epochs in user specified location.

* Fix ncf test for keras

* add a todo for batch_size and eval_batch_size for ncf keras

* lint fix

* fix typos

* Lint fix

* fix lint

* resolve pr comment

* resolve pr comment

* xla to bs=128 for num_gpu=8

* remove todo

* optimizer back to compat.v1

* add doc string to fix lint

* Adding LARS to ResNet

* Fixes for the LARS patch

* Fixes for the LARS patch

* more fixes

* 1 more fix

* s/CollectiveAllReduceStrategy/MultiWorkerMirroredStrategy

* More s/contrib.distribute/distribute.experimental

* Collective communication options in MultiWorkerMirroredStrategy.

* Minor fixes

* No checkpointing if multi worker.

* turn off checkpointing

* fix lint

* Mixed precision support

* Add TODOs

* fix resnet breakage and add keras end-to-end tests

* delint

* address PR comments

* tmp commit

* tmp commit

* first attempt (without eval)

* Bug fixes

* bug fixes

* training done

* Loss NAN, no eval

* Loss weight problem solved

* resolve the NAN loss problem

* Problem solved. Clean up needed

* Added a todo

* Remove debug prints

* Extract get_optimizer to ncf_common

* Move metrics computation back to neumf; use DS.scope api

* Extract DS.scope code to utils

* lint fixes

* Move obtaining DS above producer.start to avoid race condition

* move pt 1

* move pt 2

* Update the run script

* Wrap keras_model related code into functions

* Update the doc for softmax_logitfy and change the method name

* Resolve PR comments

* working version with: eager, DS, batch and no masks

* Remove git conflict indicator

* move reshape to neumf_model

* working version, not converge

* converged

* fix a test

* more lint fix

* more lint fix

* more lint fixes

* more lint fix

* Removed unused imports

* fix test

* dummy commit for kicking of checks

* fix lint issue

* dummy input to kick off checks

* dummy input to kick off checks

* add collective to dist strat

* addressed review comments

* add a doc string

* Added XLA test with a monkey-patched op to avoid OOM

* Added doc strings in Keras benchmarks to avoid Lint error

* s/CollectiveAllReduceStrategy/MultiWorkerMirroredStrategy

* More s/contrib.distribute/distribute.experimental

* Add benchmarks for thread tuning.

* Address comment/

* Add a comment.

Add root_data_dir to constructor of Resnet50KerasBenchmarkSynth and Resnet50KerasBenchmarkReal (#6259)

* Remove isintance change for contrib strategy

Replace it with class name check instead which should work regardless

* Add quotes for string

* fix quote type

This is needed to avoid breaking benchmark execution if PerfZero provides more
Named arguments before  the benchmark class constructor is updated.

* Allow user to specify root_data_dir in the benchmark class constructor

* Address comments

* Update official resnet for multi worker training with distribution strategies.

* Fixes for multi worker training.

* Fix call to `get_distribution_strategy`.

* Undo test change.

* Fix spacing.

* Move cluster configuration to distribution_utils.

* Move train_and_evaluate out of loop.  Also, update docstrings for multi-worker flags and add use_train_and_evaluate flag.

* Update distribution_strategy flag to match exported name for collective strategy.

* Add flag to enable XLA in Keras models

* Fix lint errors (some of them are old errors)

* Workarond for memory issue in eager mode.

* Add a TODO

* Fix typo

* Address comments

* remove patch which appear hacky.

* fix typo

* Add exp_per_second history to hook.

- Add tracking exp_per_second to benchmark tests.

* remove turn off dist strat.

* Average all results.

* One device from contrib to core.

* remove test code.

* Manually scale the loss in Resnet DS model

* Update resnet_run_loop.py

* Add a flag to specify distribution strategies.

* Fix a small error.

* Address comments.

* Address comments.

* Fix typos.

* fix test benchmark_graph_1_gpu_no_dist_strat failing

- Failure only occurs when all 1_gpu tests are run
together with the error:
tensorflow.python.framework.errors_impl.NotFoundError:
Resource localhost/logdir:/tmp/cifar10_model/
N10tensorflow22SummaryWriterInterfaceE does not exist.
[Op:WriteScalarSummary] name: epoch_loss/

Another fix might be to generate a different model_dir
in the core code, but that has other draw backs such as
restarting from the checkpoint.

* Model_dir for all tests.