GitLab

Instead of needing to ensure variables are float32, casting inputs to float32, etc, instead dtype="float32" is passed to the layer constructor, which will do all that logic automatically.

The only difference is the output of LayerNorm is now float32 instead of float16, so an extra cast is needed elsewhere.

PiperOrigin-RevId: 273833286

PiperOrigin-RevId: 273371605

PiperOrigin-RevId: 269914707

Refactor basic utils to modeling/

PiperOrigin-RevId: 269600561

PiperOrigin-RevId: 269376599

PiperOrigin-RevId: 268057019

--stop_threshold, --num_gpu, --hooks, --export_dir, and --distribution_strategy have been unexposed from models which do not use them

PiperOrigin-RevId: 268032080

PiperOrigin-RevId: 267435985

--clean, --train_epochs, and --epochs_between_evals have been unexposed from models which do not use them

PiperOrigin-RevId: 267065651

PiperOrigin-RevId: 266848625

I soon plan on making mixed precision policies exposed in TF2 only.

PiperOrigin-RevId: 266433372

PiperOrigin-RevId: 266413847

The transformer model requires TF2.

PiperOrigin-RevId: 266026991

PiperOrigin-RevId: 265994446

PiperOrigin-RevId: 265856020

PiperOrigin-RevId: 265771605

PiperOrigin-RevId: 265499806

PiperOrigin-RevId: 265122309

PiperOrigin-RevId: 264958330

PiperOrigin-RevId: 264867908

PiperOrigin-RevId: 264853703

PiperOrigin-RevId: 264776396

The old infer_float32_policies policy will be removed from TensorFlow soon.

PiperOrigin-RevId: 264300408

An earlier change made it so running this file would cause an error. The unit tests still passed, as the unit tests do not directly call the main() function

PiperOrigin-RevId: 264268400

Small tweak to the range based on seeing values
slightly over 29 and slightly under 28. This avoids
false positives that even if valid are not actionable.

PiperOrigin-RevId: 264174941

PiperOrigin-RevId: 263863438

PiperOrigin-RevId: 263812316

PiperOrigin-RevId: 263669433

PiperOrigin-RevId: 263485329

PiperOrigin-RevId: 263463300

Also, do Transformer inference in fp16, as well as training, when --dtype=fp16. In TF 2, layers now cannot run in multiple different dtypes, so we must use the same dtype for training and inference.

262039434  by A. Unique TensorFlower<gardener@tensorflow.org>:

    Internal change

262024241  by hongkuny<hongkuny@google.com>:

    Adds __init__.py

--
262021128  by isaprykin<isaprykin@google.com>:

    Internal change

PiperOrigin-RevId: 262039434

* Fix the ValueError: Error when checking model input on the new codepath

Fixes the following error:

  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 2428, in _standardize_user_data
    exception_prefix='input')
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_utils.py", line 530, in standardize_input_data
    str(len(data)) + ' arrays: ' + str(data)[:200] + '...')
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [<tf.Tensor 'cond_8/Identity:0' shape=(None, None) dtype=int64>]..

Tested and reproduced by running trasformer_main_test (thanks to whoever wrote it, phew!)

* Remove the unnecessary TODO.