xlnet.rst 4.5 KB
Newer Older
1
2
XLNet
----------------------------------------------------
3

Lysandre's avatar
Lysandre committed
4
5
6
7
Overview
~~~~~~~~~~~~~~~~~~~~~

The XLNet model was proposed in `XLNet: Generalized Autoregressive Pretraining for Language Understanding <https://arxiv.org/abs/1906.08237>`_
Lysandre's avatar
Lysandre committed
8
9
10
11
12
by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method
to learn bidirectional contexts by maximizing the expected likelihood over all permutations
of the input sequence factorization order.

Lysandre's avatar
Lysandre committed
13
14
15
16
17
18
19
20
21
22
23
The abstract from the paper is the following:

*With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves
better performance than pretraining approaches based on autoregressive language modeling. However, relying on
corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a
pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive
pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over
all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model,
into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by
a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.*
Lysandre's avatar
Lysandre committed
24

Lysandre's avatar
Lysandre committed
25
Tips:
Lysandre's avatar
Lysandre committed
26

Lysandre's avatar
Lysandre committed
27
28
29
30
31
- The specific attention pattern can be controlled at training and test time using the `perm_mask` input.
- Due to the difficulty of training a fully auto-regressive model over various factorization order,
  XLNet is pretrained using only a sub-set of the output tokens as target which are selected
  with the `target_mapping` input.
- To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the `perm_mask` and
32
  `target_mapping` inputs to control the attention span and outputs (see examples in `examples/text-generation/run_generation.py`)
Lysandre's avatar
Lysandre committed
33
- XLNet is one of the few models that has no sequence length limit.
Lysandre's avatar
Lysandre committed
34

35
36
The original code can be found `here <https://github.com/zihangdai/xlnet/>`_.

Lysandre's avatar
Lysandre committed
37

Lysandre's avatar
Lysandre committed
38
XLNetConfig
39
40
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

41
.. autoclass:: transformers.XLNetConfig
42
43
44
    :members:


Lysandre's avatar
Lysandre committed
45
XLNetTokenizer
46
47
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

48
.. autoclass:: transformers.XLNetTokenizer
Lysandre Debut's avatar
Lysandre Debut committed
49
50
    :members: build_inputs_with_special_tokens, get_special_tokens_mask,
        create_token_type_ids_from_sequences, save_vocabulary
51
52


Lysandre's avatar
Lysandre committed
53
XLNetModel
54
55
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

56
.. autoclass:: transformers.XLNetModel
57
58
59
    :members:


Lysandre's avatar
Lysandre committed
60
XLNetLMHeadModel
61
62
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

63
.. autoclass:: transformers.XLNetLMHeadModel
64
65
66
    :members:


Lysandre's avatar
Lysandre committed
67
XLNetForSequenceClassification
68
69
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

70
.. autoclass:: transformers.XLNetForSequenceClassification
71
72
73
    :members:


Sylvain Gugger's avatar
Sylvain Gugger committed
74
XLNetForMultipleChoice
Lysandre's avatar
Lysandre committed
75
76
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sylvain Gugger's avatar
Sylvain Gugger committed
77
.. autoclass:: transformers.XLNetForMultipleChoice
Lysandre's avatar
Lysandre committed
78
79
80
    :members:


Sylvain Gugger's avatar
Sylvain Gugger committed
81
XLNetForTokenClassification
Lysandre's avatar
Lysandre committed
82
83
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sylvain Gugger's avatar
Sylvain Gugger committed
84
.. autoclass:: transformers.XLNetForTokenClassification
Lysandre's avatar
Lysandre committed
85
86
87
    :members:


Lysandre's avatar
Lysandre committed
88
XLNetForQuestionAnsweringSimple
Lysandre's avatar
Lysandre committed
89
90
91
92
93
94
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.XLNetForQuestionAnsweringSimple
    :members:


Lysandre's avatar
Lysandre committed
95
XLNetForQuestionAnswering
96
97
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

98
.. autoclass:: transformers.XLNetForQuestionAnswering
99
    :members:
LysandreJik's avatar
LysandreJik committed
100
101


Lysandre's avatar
Lysandre committed
102
TFXLNetModel
LysandreJik's avatar
LysandreJik committed
103
104
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

105
.. autoclass:: transformers.TFXLNetModel
LysandreJik's avatar
LysandreJik committed
106
107
108
    :members:


Lysandre's avatar
Lysandre committed
109
TFXLNetLMHeadModel
LysandreJik's avatar
LysandreJik committed
110
111
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

112
.. autoclass:: transformers.TFXLNetLMHeadModel
LysandreJik's avatar
LysandreJik committed
113
114
115
    :members:


Lysandre's avatar
Lysandre committed
116
TFXLNetForSequenceClassification
LysandreJik's avatar
LysandreJik committed
117
118
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

119
.. autoclass:: transformers.TFXLNetForSequenceClassification
LysandreJik's avatar
LysandreJik committed
120
121
122
    :members:


Sylvain Gugger's avatar
Sylvain Gugger committed
123
124
125
126
127
128
129
130
131
132
133
134
135
136
TFLNetForMultipleChoice
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.TFXLNetForMultipleChoice
    :members:


TFXLNetForTokenClassification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.TFXLNetForTokenClassification
    :members:


Lysandre's avatar
Lysandre committed
137
TFXLNetForQuestionAnsweringSimple
LysandreJik's avatar
LysandreJik committed
138
139
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

140
.. autoclass:: transformers.TFXLNetForQuestionAnsweringSimple
LysandreJik's avatar
LysandreJik committed
141
    :members: