GitLab

PiperOrigin-RevId: 336795303

PiperOrigin-RevId: 326286926

PiperOrigin-RevId: 326286926

Move mock_task to utils/testing/

PiperOrigin-RevId: 325275356

Move mock_task to utils/testing/

PiperOrigin-RevId: 325275356

PiperOrigin-RevId: 309079916

PiperOrigin-RevId: 306521269

PiperOrigin-RevId: 305709689

PiperOrigin-RevId: 300858086

PiperOrigin-RevId: 292178802

PiperOrigin-RevId: 284874717

PiperOrigin-RevId: 281846531

PiperOrigin-RevId: 281192912

change benchmark's log verbosity to logging.INFO. it seems to me that DEBUG map to ---v=1 internally, which is way to verbose for the purpose of benchmarking.

PiperOrigin-RevId: 274040907

--clean, --train_epochs, and --epochs_between_evals have been unexposed from models which do not use them

PiperOrigin-RevId: 267065651

PiperOrigin-RevId: 264527204

Only the V1 resnet model uses --max_train_steps. This unexposes the flag in the keras_application_models, mnist, keras resnet, CTL resnet Models. Before this change, such models allowed the flag to be specified, but ignored it.

I also removed the "max_train" argument from the run_synthetic function, since this only had any meaning for the V1 resnet model. Instead, the V1 resnet model now directly passes --max_train_steps=1 to run_synthetic.

PiperOrigin-RevId: 264269836

* Update to use py3 lint

* Update model_saving_utils.py

Testing. To be reverted

* Update model_saving_utils.py

old lint is no longer used.

Only care about errors and output into an error file.

* Update pylint.rcfile

* Update pylint.rcfile

* Update pylint.rcfile

* add new sanity check script for lint to replace current lint script.

* Revert "Update pylint.rcfile"

This reverts commit f6036cd7e7c4b9e3eeb47bb56a63927a040a2761.

* Revert "Update pylint.rcfile"

This reverts commit e3af497342e26bbbbecfc8c8f79cb0e24a2ef960.

* Revert "Update pylint.rcfile"

This reverts commit 6136636eee6e90fd191ebbb4ccaa9fb89c0290f4.

* update scripts

* disable trailing-newlines

* Fix unit tests failures.

* 96% of TF 2.0 tests on GPU are passing.

* Currently all passing GPU and CPU TF 2.0

* Address code comments.

* use tf 2.0 cast.

* Comment about working on TF 2.0 CPU

* Uses contrib turn off for TF 2.0.

* Fix wide_deep and add keras_common_tests.

* use context to get num_gpus.

* Switch to tf.keras.metrics

* Moved common keras code to utils.

* Initial 1 gpu benchmark

- Aligned flags with resnet example
- removed code/features that are not super useful
- eval as part of train if bleu source/ref provided
- add exp_per_second hook

* Rename benchmark classes, pass batch-size and log_steps.

* fix docstring

* Predict done with checkpoints inline

- perfzero baseclass

* steps not epochs with smoother training loop.

* do not initialize history outside loop.

* 5000 between eval not 500

* estimator to keras.

* remove epochs var.

* use range not xrange.

* 200K steps for 1 gpu

* fix global step

- Test passes locally python3 and test is already
    skipped for python2.

* Remove contrib thread pool.

* Remove commented out contrib import.

* Fix lint issues.

* move tf.data.options higher. Tweak line breaks.

This reverts commit 57e07520.

This reverts commit d6b2b83c.

* Add resnet56 short tests. (#6101)

* Add resnet56 short tests.
- created base benchmark module
- renamed accuracy test class to contain the word Accuracy
which will result in a need to update all the jobs
and a loss of history but is worth it.
- short tests are mostly copied from shining with oss refactor

* Address feedback.

* Move flag_methods to init
- Address setting default flags repeatedly.

* Rename accuracy tests.

* Lint errors resolved.

* fix model_dir set to flags.data_dir.

* fixed not fulling pulling out flag_methods.

* Use core mirrored strategy in official models (#6126)

* Imagenet short tests (#6132)

* Add short imagenet tests (taken from seemuch)
- also rename to match go forward naming

* fix method name

* Update doc strings.

* Fixe gpu number.

* points default data_dir to child folder. (#6131)

Failed test is python2  and was a kokoro failure

* Imagenet short tests (#6136)

* Add short imagenet tests (taken from seemuch)
- also rename to match go forward naming

* fix method name

* Update doc strings.

* Fixe gpu number.

* Add fill_objects

* fixed calling wrong class in super.

* fix lint issue.

* Flag (#6121)

* Fix the turn_off_ds flag problem

* add param names to all args

* Export benchmark stats using tf.test.Benchmark.report_benchmark() (#6103)

* Export benchmark stats using tf.test.Benchmark.report_benchmark()

* Fix python style using pyformat

* Typos. (#6120)

* log verbosity=2 logs every epoch no progress bars (#6142)

* tf_upgrade_v2 on resnet and utils folder.

* tf_upgrade_v2 on resnet and utils folder.

Add bisection based producer for increased scalability, enable fully deterministic data production, and use the materialized and bisection producer to check each other (via expected output md5's)

* intermediate commit

* ncf now working

* reorder pipeline

* allow batched decode for file backed dataset

* fix bug

* more tweaks

* parallize false negative generation

* shared pool hack

* workers ignore sigint

* intermediate commit

* simplify buffer backed dataset creation to fixed length record approach only. (more cleanup needed)

* more tweaks

* simplify pipeline

* fix misplaced cleanup() calls. (validation works\!)

* more tweaks

* sixify memoryview usage

* more sixification

* fix bug

* add future imports

* break up training input pipeline

* more pipeline tuning

* first pass at moving negative generation to async

* refactor async pipeline to use files instead of ipc

* refactor async pipeline

* move expansion and concatenation from reduce worker to generation workers

* abandon complete async due to interactions with the tensorflow threadpool

* cleanup

* remove performance_comparison.py

* experiment with rough generator + interleave pipeline

* yet more pipeline tuning

* update on-the-fly pipeline

* refactor preprocessing, and move train generation behind a GRPC server

* fix leftover call

* intermediate commit

* intermediate commit

* fix index error in data pipeline, and add logging to train data server

* make sharding more robust to imbalance

* correctly sample with replacement

* file buffers are no longer needed for this branch

* tweak sampling methods

* add README for data pipeline

* fix eval sampling, and vectorize eval metrics

* add spillover and static training batch sizes

* clean up cruft from earlier iterations

* rough delint

* delint 2 / n

* add type annotations

* update run script

* make run.sh a bit nicer

* change embedding initializer to match reference

* rough pass at pure estimator model_fn

* impose static shape hack (revisit later)

* refinements

* fix dir error in run.sh

* add documentation

* add more docs and fix an assert

* old data test is no longer valid. Keeping it around as reference for the new one

* rough draft of data pipeline validation script

* don't rely on shuffle default

* tweaks and documentation

* add separate eval batch size for performance

* initial commit

* terrible hacking

* mini hacks

* missed a bug

* messing about trying to get TPU running

* TFRecords based TPU attempt

* bug fixes

* don't log remotely

* more bug fixes

* TPU tweaks and bug fixes

* more tweaks

* more adjustments

* rework model definition

* tweak data pipeline

* refactor async TFRecords generation

* temp commit to run.sh

* update log behavior

* fix logging bug

* add check for subprocess start to avoid cryptic hangs

* unify deserialize and make it TPU compliant

* delint

* remove gRPC pipeline code

* fix logging bug

* delint and remove old test files

* add unit tests for NCF pipeline

* delint

* clean up run.sh, and add run_tpu.sh

* forgot the most important line

* fix run.sh bugs

* yet more bash debugging

* small tweak to add keras summaries to model_fn

* Clean up sixification issues

* address PR comments

* delinting is never over

* Using BenchmarkLogger

* Using BenchmarkLogger

* Fixing tests

* Linting fixes.

* Adding comments

* Moving mock logger

* Moving mock logger

* Glinting

* Responding to CR

* Reverting assertEmpty

* squash of modular absl usage commits

* delint

* address PR comments

* change hooks to comma separated list, as absl behavior for space separated lists is not as expected

* change reference_data.py to use tf.gfile

* simplify json treatment

* Update reference files to account for a superficial change in batch_norm

* Add presubmit testing script for local testing.

* Update the test script to be more modularized.

1. Check the script file location and cd into repo root dir.
2. Allow caller to call differnt tests.

* add end-to-end tests for wide_deep

delint

* address PR comments

* Add benchmark upload util to bigquery.

Also update the benchmark logger and bigquery schema for the
errors found during the integration test.

* Fix lint error.

* Update test to clear all the env vars during test.

This was causing error since the Kokoro test has TF_PKG=tf-nightly
injected during test.

* Update lintrc to ignore google related package.

* Another attempt to fix lint import error.

* Address the review comment.

* Fix lint error.

* Another fix for lint.

* Update test comment for env var clean up.