GitLab

PiperOrigin-RevId: 463429471

PiperOrigin-RevId: 463166393

Move `output_range` argument to call(). After evaluating several use cases, `output_range` is better to be specified at `call` time which allows us the train with full sequence and serve with certain tokens.

PiperOrigin-RevId: 462885854

Fix attention dropout rate bug. The output_dropout rate was used for attention dropout by mistake.

PiperOrigin-RevId: 462470287

PiperOrigin-RevId: 461063108

PiperOrigin-RevId: 460756776

Adding FAVOR++ mechanism from the paper: "Chefs' Random Tables: Non-Trigonometric Random Features" to TF MODEL GARDEN.

PiperOrigin-RevId: 460625574

PiperOrigin-RevId: 458268967

PiperOrigin-RevId: 458012017

PiperOrigin-RevId: 457383160

PiperOrigin-RevId: 456159384

PiperOrigin-RevId: 456100154

PiperOrigin-RevId: 453982185

PiperOrigin-RevId: 453749903

PiperOrigin-RevId: 453490507

PiperOrigin-RevId: 450124488

PiperOrigin-RevId: 448695305

PiperOrigin-RevId: 446536595

PiperOrigin-RevId: 446219773

Also, refactor uvit builder.

PiperOrigin-RevId: 443105970

PiperOrigin-RevId: 441337748

PiperOrigin-RevId: 437812893

PiperOrigin-RevId: 436824079

PiperOrigin-RevId: 434027325

PiperOrigin-RevId: 433920851

PiperOrigin-RevId: 433892706

PiperOrigin-RevId: 432311587

PiperOrigin-RevId: 432292375

PiperOrigin-RevId: 424187808

PiperOrigin-RevId: 422917125

PiperOrigin-RevId: 422612138

PiperOrigin-RevId: 419617435

This layer replaces the weight matrix of the output_dense layer with a block diagonal matrix to save layer parameters and FLOPs. A linear mixing layer can be added optionally to improve layer expressibility.

PiperOrigin-RevId: 418828099

PiperOrigin-RevId: 417646384

PiperOrigin-RevId: 416671104

PiperOrigin-RevId: 414027254

PiperOrigin-RevId: 413261350

PiperOrigin-RevId: 412185559

PiperOrigin-RevId: 408977858