xlm.rst 3.56 KB
Newer Older
1
2
XLM
----------------------------------------------------
3

Lysandre's avatar
Lysandre committed
4
5
6
7
Overview
~~~~~~~~~~~~~~~~~~~~~

The XLM model was proposed in `Cross-lingual Language Model Pretraining <https://arxiv.org/abs/1901.07291>`_
Lysandre's avatar
Lysandre committed
8
9
by Guillaume Lample*, Alexis Conneau*. It's a transformer pre-trained using one of the following objectives:

Lysandre's avatar
Lysandre committed
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
- a causal language modeling (CLM) objective (next token prediction),
- a masked language modeling (MLM) objective (Bert-like), or
- a Translation Language Modeling (TLM) object (extension of Bert's MLM to multiple language inputs)

The abstract from the paper is the following:

*Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding.
In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining.
We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual
data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain
state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI,
our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation,
we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On
supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming
the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.*
Lysandre's avatar
Lysandre committed
25

Lysandre's avatar
Lysandre committed
26
Tips:
Lysandre's avatar
Lysandre committed
27

Lysandre's avatar
Lysandre committed
28
29
30
31
- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to
  select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).
- XLM has multilingual checkpoints which leverage a specific `lang` parameter. Check out the
  `multi-lingual <../multilingual.html>`__ page for more information.
Lysandre's avatar
Lysandre committed
32
33
34


XLMConfig
35
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
36

37
.. autoclass:: transformers.XLMConfig
38
39
    :members:

Lysandre's avatar
Lysandre committed
40
XLMTokenizer
41
42
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

43
.. autoclass:: transformers.XLMTokenizer
Lysandre Debut's avatar
Lysandre Debut committed
44
45
    :members: build_inputs_with_special_tokens, get_special_tokens_mask,
        create_token_type_ids_from_sequences, save_vocabulary
46

Lysandre's avatar
Lysandre committed
47
XLMModel
48
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
49

50
.. autoclass:: transformers.XLMModel
51
52
53
    :members:


Lysandre's avatar
Lysandre committed
54
XLMWithLMHeadModel
55
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
56

57
.. autoclass:: transformers.XLMWithLMHeadModel
58
59
60
    :members:


Lysandre's avatar
Lysandre committed
61
XLMForSequenceClassification
62
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
63

64
.. autoclass:: transformers.XLMForSequenceClassification
65
66
67
    :members:


Lysandre's avatar
Lysandre committed
68
XLMForQuestionAnsweringSimple
Lysandre's avatar
Lysandre committed
69
70
71
72
73
74
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.XLMForQuestionAnsweringSimple
    :members:


Lysandre's avatar
Lysandre committed
75
XLMForQuestionAnswering
76
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
77

78
.. autoclass:: transformers.XLMForQuestionAnswering
79
    :members:
LysandreJik's avatar
LysandreJik committed
80
81


Lysandre's avatar
Lysandre committed
82
TFXLMModel
LysandreJik's avatar
LysandreJik committed
83
84
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

thomwolf's avatar
thomwolf committed
85
.. autoclass:: transformers.TFXLMModel
LysandreJik's avatar
LysandreJik committed
86
87
88
    :members:


Lysandre's avatar
Lysandre committed
89
TFXLMWithLMHeadModel
LysandreJik's avatar
LysandreJik committed
90
91
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

92
.. autoclass:: transformers.TFXLMWithLMHeadModel
LysandreJik's avatar
LysandreJik committed
93
94
95
    :members:


Lysandre's avatar
Lysandre committed
96
TFXLMForSequenceClassification
LysandreJik's avatar
LysandreJik committed
97
98
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

99
.. autoclass:: transformers.TFXLMForSequenceClassification
LysandreJik's avatar
LysandreJik committed
100
101
102
    :members:


Lysandre's avatar
Lysandre committed
103
TFXLMForQuestionAnsweringSimple
LysandreJik's avatar
LysandreJik committed
104
105
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

106
.. autoclass:: transformers.TFXLMForQuestionAnsweringSimple
LysandreJik's avatar
LysandreJik committed
107
    :members: