Commit 0c48b89f authored by Jeremiah Liu's avatar Jeremiah Liu Committed by A. Unique TensorFlower
Browse files

Update layer documentation.

PiperOrigin-RevId: 364328905
parent 572fafb8
...@@ -33,6 +33,10 @@ assemble new `tf.keras` layers or models. ...@@ -33,6 +33,10 @@ assemble new `tf.keras` layers or models.
of self multi-head attention, cross multi-head attention and feedforward of self multi-head attention, cross multi-head attention and feedforward
network. network.
* [RandomFeatureGaussianProcess](gaussian_process.py) implements random
feature-based Gaussian process described in ["Random Features for
Large-Scale Kernel Machines"](https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf).
* [ReZeroTransformer](rezero_transformer.py) implements Transformer with * [ReZeroTransformer](rezero_transformer.py) implements Transformer with
ReZero described in ReZero described in
["ReZero is All You Need: Fast Convergence at Large Depth"](https://arxiv.org/abs/2003.04887). ["ReZero is All You Need: Fast Convergence at Large Depth"](https://arxiv.org/abs/2003.04887).
...@@ -47,6 +51,11 @@ assemble new `tf.keras` layers or models. ...@@ -47,6 +51,11 @@ assemble new `tf.keras` layers or models.
* [SelfAttentionMask](self_attention_mask.py) creates a 3D attention mask from * [SelfAttentionMask](self_attention_mask.py) creates a 3D attention mask from
a 2D tensor mask. a 2D tensor mask.
* [SpectralNormalization](spectral_normalization.py) implements a tf.Wrapper
that applies spectral normalization regularization to a given layer. See
[Spectral Norm Regularization for Improving the Generalizability of
Deep Learning](https://arxiv.org/abs/1705.10941)
* [MaskedSoftmax](masked_softmax.py) implements a softmax with an optional * [MaskedSoftmax](masked_softmax.py) implements a softmax with an optional
masking input. If no mask is provided to this layer, it performs a standard masking input. If no mask is provided to this layer, it performs a standard
softmax; however, if a mask tensor is applied (which should be 1 in softmax; however, if a mask tensor is applied (which should be 1 in
...@@ -60,6 +69,11 @@ assemble new `tf.keras` layers or models. ...@@ -60,6 +69,11 @@ assemble new `tf.keras` layers or models.
* [ClassificationHead](cls_head.py) A pooling head over a sequence of * [ClassificationHead](cls_head.py) A pooling head over a sequence of
embeddings, commonly used by classification tasks. embeddings, commonly used by classification tasks.
* [GaussianProcessClassificationHead](cls_head.py) A spectral-normalized
neural Gaussian process (SNGP)-based classification head as described in
["Simple and Principled Uncertainty Estimation with Deterministic Deep
Learning via Distance Awareness"](https://arxiv.org/abs/2006.10108).
* [GatedFeedforward](gated_feedforward.py) implements the gated linear layer * [GatedFeedforward](gated_feedforward.py) implements the gated linear layer
feedforward as described in feedforward as described in
["GLU Variants Improve Transformer"](https://arxiv.org/abs/2002.05202). ["GLU Variants Improve Transformer"](https://arxiv.org/abs/2002.05202).
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment