"src/targets/vscode:/vscode.git/clone" did not exist on "548783c84db8dc728cd6423ef6a6c5b23803db20"
README.md 2.66 KB
Newer Older
1
# Layers
Hongkun Yu's avatar
Hongkun Yu committed
2

3
4
5
Layers are the fundamental building blocks for NLP models. They can be used to
assemble new layers, networks, or models.

Hongkun Yu's avatar
Hongkun Yu committed
6
*   [MultiHeadAttention](attention.py) implements an optionally masked attention
7
    between query, key, value tensors as described in
Hongkun Yu's avatar
Hongkun Yu committed
8
9
    ["Attention Is All You Need"](https://arxiv.org/abs/1706.03762). If
    `from_tensor` and `to_tensor` are the same, then this is self-attention.
10

Hongkun Yu's avatar
Hongkun Yu committed
11
12
*   [CachedAttention](attention.py) implements an attention layer with cache
    used for auto-agressive decoding.
13

14
15
16
17
*   [MultiChannelAttention](multi_channel_attention.py) implements an variant of
    multi-head attention which can be used to merge multiple streams for
    cross-attentions.

Hongkun Yu's avatar
Hongkun Yu committed
18
19
20
*   [TalkingHeadsAttention](talking_heads_attention.py) implements the talking
    heads attention, as decribed in
    ["Talking-Heads Attention"](https://arxiv.org/abs/2003.02436).
21

Hongkun Yu's avatar
Hongkun Yu committed
22
23
24
*   [Transformer](transformer.py) implements an optionally masked transformer as
    described in
    ["Attention Is All You Need"](https://arxiv.org/abs/1706.03762).
25

Hongkun Yu's avatar
Hongkun Yu committed
26
27
28
29
*   [TransformerDecoderLayer](transformer.py) TransformerDecoderLayer is made up
    of self multi-head attention, cross multi-head attention and
    feedforward network.

Hongkun Yu's avatar
Hongkun Yu committed
30
31
32
*   [ReZeroTransformer](rezero_transformer.py) implements Transformer with
    ReZero described in
    ["ReZero is All You Need: Fast Convergence at Large Depth"](https://arxiv.org/abs/2003.04887).
33

Hongkun Yu's avatar
Hongkun Yu committed
34
35
*   [OnDeviceEmbedding](on_device_embedding.py) implements efficient embedding
    lookups designed for TPU-based models.
Le Hou's avatar
Le Hou committed
36

Hongkun Yu's avatar
Hongkun Yu committed
37
38
39
*   [PositionalEmbedding](position_embedding.py) creates a positional embedding
    as described in ["BERT: Pre-training of Deep Bidirectional Transformers for
    Language Understanding"](https://arxiv.org/abs/1810.04805).
40

Hongkun Yu's avatar
Hongkun Yu committed
41
42
*   [SelfAttentionMask](self_attention_mask.py) creates a 3D attention mask from
    a 2D tensor mask.
43

Hongkun Yu's avatar
Hongkun Yu committed
44
45
46
47
48
49
*   [MaskedSoftmax](masked_softmax.py) implements a softmax with an optional
    masking input. If no mask is provided to this layer, it performs a standard
    softmax; however, if a mask tensor is applied (which should be 1 in
    positions where the data should be allowed through, and 0 where the data
    should be masked), the output will have masked positions set to
    approximately zero.
50

Hongkun Yu's avatar
Hongkun Yu committed
51
52
*   [`MaskedLM`](masked_lm.py) implements a masked language model. It assumes
    the embedding table variable is passed to it.
Hongkun Yu's avatar
Hongkun Yu committed
53

Hongkun Yu's avatar
Hongkun Yu committed
54
55
*   [ClassificationHead](cls_head.py) A pooling head over a sequence of
    embeddings, commonly used by classification tasks.
Chen Chen's avatar
Chen Chen committed
56
57
58
59

*   [GatedFeedforward](gated_feedforward.py) implements the gated linear layer
    feedforward as described in
    ["GLU Variants Improve Transformer"](https://arxiv.org/abs/2002.05202).