GitLab

* intermediate commit

finish replacing spillover with resampled padding

intermediate commit

* resolve merge conflict

* intermediate commit

* further consolidate the data pipeline

* complete first pass at data pipeline refactor

* remove some leftover code

* fix test

* remove resampling, and move train padding logic into neumf.py

* small tweaks

* fix weight bug

* address PR comments

* fix dict zip. (Reed led me astray)

* delint

* make data test deterministic and delint

* Reed didn't lead me astray. I just can't read.

* more delinting

* even more delinting

* use resampling for last batch padding

* pad last batch with unique data

* Revert "pad last batch with unique data"

This reverts commit cbdf46efcd5c7907038a24105b88d38e7f1d6da2.

* move padded batch to the beginning

* delint

* fix step check for synthetic data

Since we plan on deleting this method, it is only used in distribution_utils_test.py.

* move flagfile into the cache_dir

* remove duplicate code

* delint

* Use data_dir instead of flags.FLAGS.data_dir in data_preprocessing.py.

* Use data_dir instead of flags.FLAGS.data_dir in data_preprocessing.py.

* Replace multiprocess pool with popen_helper.get_pool() in data_preprocessing.

* Use data_dir instead of flags.FLAGS.data_dir in data_preprocessing.py.

* Use data_dir instead of flags.FLAGS.data_dir in data_preprocessing.py.

Add comments, exit async process after waiting for flagfile for too long and make directory for data_dir in case it does not exist.

* Add --use_synthetic_data option to NCF.

* Add comment to _SYNTHETIC_BATCHES_PER_EPOCH

* Fix test

* Hopefully fix lint issue

* improve default handling for eval_batch_size

* return eval_batch_size default to None

* fix syntax error

* Update resnet README with new checkpoints and SavedModels

* add more detail on channels_first vs channels_last

* fix typo

* add disclaimer about checkpoints

* set strip_default_attrs=True for SavedModel exports

* specify dtype in resnet export

* another dtype fix

* fix another dtype issue, and set --image_bytes_as_serving_input to default to False

* move evaluation from numpy to tensorflow

fix syntax error

don't use sigmoid to convert logits. there is too much precision loss.

WIP: add logit metrics

continue refactor of NCF evaluation

fix syntax error

fix bugs in eval loss calculation

fix eval loss reweighting

remove numpy based metric calculations

fix logging hooks

fix sigmoid to softmax bug

fix comment

catch rare PIPE error and address some PR comments

* fix metric test and address PR comments

* delint and fix python2

* fix test and address PR comments

* extend eval to TPUs

Some changes specific to prediction. Removing traces of expected results, as this is just prediction.

with serving signature that accepts JPEG image bytes instead
of a fixed size [HxWxC] image tensor.

Passing JPEG image bytes is easier for inference/serving use
cases. The model internally resizes/crops the JPEG image to
required [HxWxC] tensor before passing it on for actual model
inference.

This change aligns with Cloud TPU/ResNet-50 model that offers a
similar interface (jpeg bytes) for inferencing here:

https://github.com/tensorflow/tpu/tree/master/models/official/resnet

NOTE: This flag is set to `True` by default for ImageNet, and is
disallowed for CIFAR (as it does not apply to CIFAR).

Right now we don't have input data for prediction. So using top 10
entries of test data as input.

* bug fixes and add seed

* more random corrections

* make cleanup more robust

* return cleanup fn

* delint and address PR comments.

* delint and fix tests

* delinting is never done

* add pipeline hashing

* delint