funnel.rst 8 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
.. 
    Copyright 2020 The HuggingFace Team. All rights reserved.

    Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
    an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
    specific language governing permissions and limitations under the License.

Sylvain Gugger's avatar
Sylvain Gugger committed
13
Funnel Transformer
Sylvain Gugger's avatar
Sylvain Gugger committed
14
-----------------------------------------------------------------------------------------------------------------------
Sylvain Gugger's avatar
Sylvain Gugger committed
15
16

Overview
Sylvain Gugger's avatar
Sylvain Gugger committed
17
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
18

Sylvain Gugger's avatar
Sylvain Gugger committed
19
20
21
22
The Funnel Transformer model was proposed in the paper `Funnel-Transformer: Filtering out Sequential Redundancy for
Efficient Language Processing <https://arxiv.org/abs/2006.03236>`__. It is a bidirectional transformer model, like
BERT, but with a pooling operation after each block of layers, a bit like in traditional convolutional neural networks
(CNN) in computer vision.
Sylvain Gugger's avatar
Sylvain Gugger committed
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

The abstract from the paper is the following:

*With the success of language pretraining, it is highly desirable to develop more efficient architectures of good
scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the
much-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only
require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which
gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More
importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further
improve the model capacity. In addition, to perform token-level predictions as required by common pretraining
objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence
via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on
a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading
comprehension.*

Tips:

- Since Funnel Transformer uses pooling, the sequence length of the hidden states changes after each block of layers.
  The base model therefore has a final sequence length that is a quarter of the original one. This model can be used
  directly for tasks that just require a sentence summary (like sequence classification or multiple choice). For other
  tasks, the full model is used; this full model has a decoder that upsamples the final hidden states to the same
  sequence length as the input.
Sylvain Gugger's avatar
Sylvain Gugger committed
45
46
- The Funnel Transformer checkpoints are all available with a full version and a base version. The first ones should be
  used for :class:`~transformers.FunnelModel`, :class:`~transformers.FunnelForPreTraining`,
Sylvain Gugger's avatar
Sylvain Gugger committed
47
48
49
50
51
  :class:`~transformers.FunnelForMaskedLM`, :class:`~transformers.FunnelForTokenClassification` and
  class:`~transformers.FunnelForQuestionAnswering`. The second ones should be used for
  :class:`~transformers.FunnelBaseModel`, :class:`~transformers.FunnelForSequenceClassification` and
  :class:`~transformers.FunnelForMultipleChoice`.

Sylvain Gugger's avatar
Sylvain Gugger committed
52
The original code can be found `here <https://github.com/laiguokun/Funnel-Transformer>`__.
Sylvain Gugger's avatar
Sylvain Gugger committed
53
54
55


FunnelConfig
Sylvain Gugger's avatar
Sylvain Gugger committed
56
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
57
58
59
60
61
62

.. autoclass:: transformers.FunnelConfig
    :members:


FunnelTokenizer
Sylvain Gugger's avatar
Sylvain Gugger committed
63
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
64
65
66
67
68
69
70

.. autoclass:: transformers.FunnelTokenizer
    :members: build_inputs_with_special_tokens, get_special_tokens_mask,
        create_token_type_ids_from_sequences, save_vocabulary


FunnelTokenizerFast
Sylvain Gugger's avatar
Sylvain Gugger committed
71
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
72
73
74
75
76
77

.. autoclass:: transformers.FunnelTokenizerFast
    :members:


Funnel specific outputs
Sylvain Gugger's avatar
Sylvain Gugger committed
78
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
79

Sylvain Gugger's avatar
Sylvain Gugger committed
80
.. autoclass:: transformers.models.funnel.modeling_funnel.FunnelForPreTrainingOutput
Sylvain Gugger's avatar
Sylvain Gugger committed
81
82
    :members:

Sylvain Gugger's avatar
Sylvain Gugger committed
83
.. autoclass:: transformers.models.funnel.modeling_tf_funnel.TFFunnelForPreTrainingOutput
84
85
    :members:

Sylvain Gugger's avatar
Sylvain Gugger committed
86
87

FunnelBaseModel
Sylvain Gugger's avatar
Sylvain Gugger committed
88
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
89
90

.. autoclass:: transformers.FunnelBaseModel
Sylvain Gugger's avatar
Sylvain Gugger committed
91
    :members: forward
Sylvain Gugger's avatar
Sylvain Gugger committed
92
93
94


FunnelModel
Sylvain Gugger's avatar
Sylvain Gugger committed
95
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
96
97

.. autoclass:: transformers.FunnelModel
Sylvain Gugger's avatar
Sylvain Gugger committed
98
    :members: forward
Sylvain Gugger's avatar
Sylvain Gugger committed
99
100
101


FunnelModelForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
102
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
103
104

.. autoclass:: transformers.FunnelForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
105
    :members: forward
Sylvain Gugger's avatar
Sylvain Gugger committed
106
107
108


FunnelForMaskedLM
Sylvain Gugger's avatar
Sylvain Gugger committed
109
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
110
111

.. autoclass:: transformers.FunnelForMaskedLM
Sylvain Gugger's avatar
Sylvain Gugger committed
112
    :members: forward
Sylvain Gugger's avatar
Sylvain Gugger committed
113
114
115


FunnelForSequenceClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
116
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
117
118

.. autoclass:: transformers.FunnelForSequenceClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
119
    :members: forward
Sylvain Gugger's avatar
Sylvain Gugger committed
120
121
122


FunnelForMultipleChoice
Sylvain Gugger's avatar
Sylvain Gugger committed
123
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
124
125

.. autoclass:: transformers.FunnelForMultipleChoice
Sylvain Gugger's avatar
Sylvain Gugger committed
126
    :members: forward
Sylvain Gugger's avatar
Sylvain Gugger committed
127
128
129


FunnelForTokenClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
130
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
131
132

.. autoclass:: transformers.FunnelForTokenClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
133
    :members: forward
Sylvain Gugger's avatar
Sylvain Gugger committed
134
135
136


FunnelForQuestionAnswering
Sylvain Gugger's avatar
Sylvain Gugger committed
137
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
138
139

.. autoclass:: transformers.FunnelForQuestionAnswering
Sylvain Gugger's avatar
Sylvain Gugger committed
140
    :members: forward
141
142
143


TFFunnelBaseModel
Sylvain Gugger's avatar
Sylvain Gugger committed
144
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
145
146

.. autoclass:: transformers.TFFunnelBaseModel
Sylvain Gugger's avatar
Sylvain Gugger committed
147
    :members: call
148
149
150


TFFunnelModel
Sylvain Gugger's avatar
Sylvain Gugger committed
151
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
152
153

.. autoclass:: transformers.TFFunnelModel
Sylvain Gugger's avatar
Sylvain Gugger committed
154
    :members: call
155
156
157


TFFunnelModelForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
158
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
159
160

.. autoclass:: transformers.TFFunnelForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
161
    :members: call
162
163
164


TFFunnelForMaskedLM
Sylvain Gugger's avatar
Sylvain Gugger committed
165
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
166
167

.. autoclass:: transformers.TFFunnelForMaskedLM
Sylvain Gugger's avatar
Sylvain Gugger committed
168
    :members: call
169
170
171


TFFunnelForSequenceClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
172
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
173
174

.. autoclass:: transformers.TFFunnelForSequenceClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
175
    :members: call
176
177
178


TFFunnelForMultipleChoice
Sylvain Gugger's avatar
Sylvain Gugger committed
179
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
180
181

.. autoclass:: transformers.TFFunnelForMultipleChoice
Sylvain Gugger's avatar
Sylvain Gugger committed
182
    :members: call
183
184
185


TFFunnelForTokenClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
186
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
187
188

.. autoclass:: transformers.TFFunnelForTokenClassification
Sylvain Gugger's avatar
Sylvain Gugger committed
189
    :members: call
190
191
192


TFFunnelForQuestionAnswering
Sylvain Gugger's avatar
Sylvain Gugger committed
193
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
194
195

.. autoclass:: transformers.TFFunnelForQuestionAnswering
Sylvain Gugger's avatar
Sylvain Gugger committed
196
    :members: call