GitLab

Before, it returned None. But almost every use of get_distribution_strategy() assumes an actual strategy is returned and crashes when None is returned. Returning the default strategy fixes these issues and is equivalent to using no strategy, as the default strategy is always in effect when no other strategy is used.

PiperOrigin-RevId: 380951055

Also improve error message when distribution=off is specified without properly quoting "off"

PiperOrigin-RevId: 359395329

PiperOrigin-RevId: 350170448

PiperOrigin-RevId: 349347368

PiperOrigin-RevId: 331359058

PiperOrigin-RevId: 326286926

PiperOrigin-RevId: 326286926

PiperOrigin-RevId: 303767122

PiperOrigin-RevId: 294997928

Replace per_device with per_replica and PerDevice with PerReplica, because the PerDevice concept was renamed and doesn't exist anymore. (#6693)

* Replace per_device with per_replica and PerDevice with PerReplica, because the PerReplica concept was renamed and doesn't exist anymore.

* Add a flag to specify distribution strategies.

* Fix a small error.

* Address comments.

* Address comments.

* Fix typos.

`num_replicas` is being replaced by `num_replicas_in_sync` and something that returns the number of steps running concurrently.

To match new terminology in DistributionStrategy.

Since we plan on deleting this method, it is only used in distribution_utils_test.py.

* Add DistributionStrategy to transformer model

* add num_gpu flag

* Calculate per device batch size for transformer

* remove reference to flags_core

* Add synthetic data option to transformer

* fix typo

* add import back in

* Use hierarchical copy

* address PR comments

* lint

* fix spaces

* group train op together to fix single GPU error

* Fix translate bug (sorted_keys is a dict, not a list)

* Change params to a default dict (translate.py was throwing errors because params didn't have the TPU parameters.)

* Address PR comments. Removed multi gpu flag + more

* fix lint

* fix more lints

* add todo for Synthetic dataset

* Update docs