"comfy/ldm/modules/vscode:/vscode.git/clone" did not exist on "3d2f60b315c2f55819781a13447dc87f56e4b1db"
lxmert.rst 5.55 KB
Newer Older
1
LXMERT
Sylvain Gugger's avatar
Sylvain Gugger committed
2
-----------------------------------------------------------------------------------------------------------------------
3
4

Overview
Sylvain Gugger's avatar
Sylvain Gugger committed
5
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6

Sylvain Gugger's avatar
Sylvain Gugger committed
7
8
9
10
11
12
13
The LXMERT model was proposed in `LXMERT: Learning Cross-Modality Encoder Representations from Transformers
<https://arxiv.org/abs/1908.07490>`__ by Hao Tan & Mohit Bansal. It is a series of bidirectional transformer encoders
(one for the vision modality, one for the language modality, and then one to fuse both modalities) pretrained using a
combination of masked language modeling, visual-language text alignment, ROI-feature regression, masked
visual-attribute modeling, masked visual-object modeling, and visual-question answering objectives.
The pretraining consists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering,
VQA 2.0, and GQA.
14
15
16

The abstract from the paper is the following:

Sylvain Gugger's avatar
Sylvain Gugger committed
17
18
19
20
21
22
23
24
25
26
27
28
29
*Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly,
the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality
Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we
build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language
encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language
semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative
pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification),
cross-modality matching, and image question answering. These tasks help in learning both intra-modality and
cross-modality relationships. After fine-tuning from our pretrained parameters, our model achieves the state-of-the-art
results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our
pretrained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR, and improve the previous
best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel
model components and pretraining strategies significantly contribute to our strong results; and also present several
30
31
32
33
attention visualizations for the different encoders*

Tips:

Sylvain Gugger's avatar
Sylvain Gugger committed
34
35
36
37
38
39
40
41
42
- Bounding boxes are not necessary to be used in the visual feature embeddings, any kind of visual-spacial features
  will work.
- Both the language hidden states and the visual hidden states that LXMERT outputs are passed through the
  cross-modality layer, so they contain information from both modalities. To access a modality that only attends to
  itself, select the vision/language hidden states from the first input in the tuple.
- The bidirectional cross-modality encoder attention only returns attention values when the language modality is used
  as the input and the vision modality is used as the context vector. Further, while the cross-modality encoder
  contains self-attention for each respective modality and cross-attention, only the cross attention is returned and
  both self attention outputs are disregarded.
43

Sylvain Gugger's avatar
Sylvain Gugger committed
44
The original code can be found `here <https://github.com/airsplay/lxmert>`__.
45
46
47


LxmertConfig
Sylvain Gugger's avatar
Sylvain Gugger committed
48
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
49
50
51
52
53
54

.. autoclass:: transformers.LxmertConfig
    :members:


LxmertTokenizer
Sylvain Gugger's avatar
Sylvain Gugger committed
55
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
56
57

.. autoclass:: transformers.LxmertTokenizer
Sylvain Gugger's avatar
Sylvain Gugger committed
58
59
60
61
62
63
64
65
    :members:


LxmertTokenizerFast
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.LxmertTokenizerFast
    :members:
66
67
68


Lxmert specific outputs
Sylvain Gugger's avatar
Sylvain Gugger committed
69
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87

.. autoclass:: transformers.modeling_lxmert.LxmertModelOutput
    :members:

.. autoclass:: transformers.modeling_lxmert.LxmertForPreTrainingOutput
    :members:

.. autoclass:: transformers.modeling_lxmert.LxmertForQuestionAnsweringOutput
    :members:

.. autoclass:: transformers.modeling_tf_lxmert.TFLxmertModelOutput
    :members:

.. autoclass:: transformers.modeling_tf_lxmert.TFLxmertForPreTrainingOutput
    :members:


LxmertModel
Sylvain Gugger's avatar
Sylvain Gugger committed
88
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
89
90

.. autoclass:: transformers.LxmertModel
Sylvain Gugger's avatar
Sylvain Gugger committed
91
    :members: forward
92
93

LxmertForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
94
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
95
96

.. autoclass:: transformers.LxmertForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
97
    :members: forward
98
99

LxmertForQuestionAnswering
Sylvain Gugger's avatar
Sylvain Gugger committed
100
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101
102

.. autoclass:: transformers.LxmertForQuestionAnswering
Sylvain Gugger's avatar
Sylvain Gugger committed
103
    :members: forward
104
105
106


TFLxmertModel
Sylvain Gugger's avatar
Sylvain Gugger committed
107
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
108
109

.. autoclass:: transformers.TFLxmertModel
Sylvain Gugger's avatar
Sylvain Gugger committed
110
    :members: call
111
112

TFLxmertForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
113
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
114
115

.. autoclass:: transformers.TFLxmertForPreTraining
Sylvain Gugger's avatar
Sylvain Gugger committed
116
    :members: call