index.rst 15.1 KB
Newer Older
1
Transformers
Sylvain Gugger's avatar
Sylvain Gugger committed
2
=======================================================================================================================
3

4
State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.
thomwolf's avatar
thomwolf committed
5

6
7
8
馃 Transformers (formerly known as `pytorch-transformers` and `pytorch-pretrained-bert`) provides general-purpose
architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural
Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
9
10
11
TensorFlow 2.0 and PyTorch.

This is the documentation of our repository `transformers <https://github.com/huggingface/transformers>`_.
12

LysandreJik's avatar
LysandreJik committed
13
Features
Sylvain Gugger's avatar
Sylvain Gugger committed
14
-----------------------------------------------------------------------------------------------------------------------
LysandreJik's avatar
LysandreJik committed
15
16
17
18

- High performance on NLU and NLG tasks
- Low barrier to entry for educators and practitioners

LysandreJik's avatar
LysandreJik committed
19
20
State-of-the-art NLP for everyone:

LysandreJik's avatar
LysandreJik committed
21
22
23
24
- Deep learning researchers
- Hands-on practitioners
- AI/ML/NLP teachers and educators

LysandreJik's avatar
LysandreJik committed
25
26
Lower compute costs, smaller carbon footprint:

LysandreJik's avatar
LysandreJik committed
27
28
29
30
- Researchers can share trained models instead of always retraining
- Practitioners can reduce compute time and production costs
- 8 architectures with over 30 pretrained models, some in more than 100 languages

LysandreJik's avatar
LysandreJik committed
31
32
Choose the right framework for every part of a model's lifetime:

LysandreJik's avatar
LysandreJik committed
33
34
35
36
37
38
- Train state-of-the-art models in 3 lines of code
- Deep interoperability between TensorFlow 2.0 and PyTorch models
- Move a single model between TF2.0/PyTorch frameworks at will
- Seamlessly pick the right framework for training, evaluation, production

Contents
Sylvain Gugger's avatar
Sylvain Gugger committed
39
-----------------------------------------------------------------------------------------------------------------------
LysandreJik's avatar
LysandreJik committed
40

Sylvain Gugger's avatar
Sylvain Gugger committed
41
42
43
44
The documentation is organized in five parts:

- **GET STARTED** contains a quick tour, the installation instructions and some useful information about our philosophy
  and a glossary.
Sylvain Gugger's avatar
Sylvain Gugger committed
45
- **USING 馃 TRANSFORMERS** contains general tutorials on how to use the library.
Sylvain Gugger's avatar
Sylvain Gugger committed
46
47
48
- **ADVANCED GUIDES** contains more advanced guides that are more specific to a given script or part of the library.
- **RESEARCH** focuses on tutorials that have less to do with how to use the library but more about general resarch in
  transformers model
49
50
51
52
- The three last section contain the documentation of each public class and function, grouped in:
    - **MAIN CLASSES** for the main classes exposing the important APIs of the library.
    - **MODELS** for the classes and functions related to each model implemented in the library.
    - **INTERNAL HELPERS** for the classes and functions we use internally.
Sylvain Gugger's avatar
Sylvain Gugger committed
53

54
55
56
The library currently contains PyTorch and Tensorflow implementations, pre-trained model weights, usage scripts and
conversion utilities for the following models:

57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
..
    This list is updated automatically from the README with `make fix-copies`. Do not update manually!

1. `ALBERT <https://huggingface.co/transformers/model_doc/albert.html>`__ (from Google Research and the Toyota
   Technological Institute at Chicago) released with the paper `ALBERT: A Lite BERT for Self-supervised Learning of
   Language Representations <https://arxiv.org/abs/1909.11942>`__, by Zhenzhong Lan, Mingda Chen, Sebastian Goodman,
   Kevin Gimpel, Piyush Sharma, Radu Soricut.
2. `BART <https://huggingface.co/transformers/model_doc/bart.html>`__ (from Facebook) released with the paper `BART:
   Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
   <https://arxiv.org/pdf/1910.13461.pdf>`__ by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman
   Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer.
3. `BERT <https://huggingface.co/transformers/model_doc/bert.html>`__ (from Google) released with the paper `BERT:
   Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`__ by
   Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
4. `BERT For Sequence Generation <https://tfhub.dev/s?module-type=text-generation&subtype=module,placeholder>`__ (from
   Google) released with the paper `Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
   <https://arxiv.org/abs/1907.12461>`__ by Sascha Rothe, Shashi Narayan, Aliaksei Severyn.
5. `CamemBERT <https://huggingface.co/transformers/model_doc/camembert.html>`__ (from Inria/Facebook/Sorbonne) released
   with the paper `CamemBERT: a Tasty French Language Model <https://arxiv.org/abs/1911.03894>`__ by Louis Martin*,
   Benjamin Muller*, Pedro Javier Ortiz Su谩rez*, Yoann Dupont, Laurent Romary, 脡ric Villemonte de la Clergerie, Djam茅
   Seddah and Beno卯t Sagot.
6. `CTRL <https://huggingface.co/transformers/model_doc/ctrl.html>`__ (from Salesforce) released with the paper `CTRL:
   A Conditional Transformer Language Model for Controllable Generation <https://arxiv.org/abs/1909.05858>`__ by Nitish
   Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Richard Socher.
7. `DeBERTa <https://huggingface.co/transformers/model_doc/deberta.html>`__ (from Microsoft Research) released with the
   paper `DeBERTa: Decoding-enhanced BERT with Disentangled Attention <https://arxiv.org/abs/2006.03654>`__ by
   Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen.
8. `DialoGPT <https://huggingface.co/transformers/model_doc/dialogpt.html>`__ (from Microsoft Research) released with
   the paper `DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
   <https://arxiv.org/abs/1911.00536>`__ by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang
   Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.
9. `DistilBERT <https://huggingface.co/transformers/model_doc/distilbert.html>`__ (from HuggingFace), released together
89
   with the paper `DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
90
91
92
93
94
95
96
97
   <https://arxiv.org/abs/1910.01108>`__ by Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been
   applied to compress GPT2 into `DistilGPT2
   <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__, RoBERTa into `DistilRoBERTa
   <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__, Multilingual BERT into
   `DistilmBERT <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__ and a German version
   of DistilBERT.
10. `DPR <https://github.com/facebookresearch/DPR>`__ (from Facebook) released with the paper `Dense Passage Retrieval
    for Open-Domain Question Answering <https://arxiv.org/abs/2004.04906>`__ by Vladimir Karpukhin, Barlas O臒uz, Sewon
98
    Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
11. `ELECTRA <https://huggingface.co/transformers/model_doc/electra.html>`__ (from Google Research/Stanford University)
    released with the paper `ELECTRA: Pre-training text encoders as discriminators rather than generators
    <https://arxiv.org/abs/2003.10555>`__ by Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
12. `FlauBERT <https://huggingface.co/transformers/model_doc/flaubert.html>`__ (from CNRS) released with the paper
    `FlauBERT: Unsupervised Language Model Pre-training for French <https://arxiv.org/abs/1912.05372>`__ by Hang Le,
    Lo茂c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Beno卯t Crabb茅,
    Laurent Besacier, Didier Schwab.
13. `Funnel Transformer <https://github.com/laiguokun/Funnel-Transformer>`__ (from CMU/Google Brain) released with the
    paper `Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
    <https://arxiv.org/abs/2006.03236>`__ by Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le.
14. `GPT <https://huggingface.co/transformers/model_doc/gpt.html>`__ (from OpenAI) released with the paper `Improving
    Language Understanding by Generative Pre-Training <https://blog.openai.com/language-unsupervised/>`__ by Alec
    Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.
15. `GPT-2 <https://huggingface.co/transformers/model_doc/gpt2.html>`__ (from OpenAI) released with the paper `Language
    Models are Unsupervised Multitask Learners <https://blog.openai.com/better-language-models/>`__ by Alec Radford*,
    Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.
16. `LayoutLM <https://github.com/microsoft/unilm/tree/master/layoutlm>`__ (from Microsoft Research Asia) released with
116
    the paper `LayoutLM: Pre-training of Text and Layout for Document Image Understanding
117
118
119
120
121
122
123
124
125
126
127
128
    <https://arxiv.org/abs/1912.13318>`__ by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou.
17. `Longformer <https://huggingface.co/transformers/model_doc/longformer.html>`__ (from AllenAI) released with the
    paper `Longformer: The Long-Document Transformer <https://arxiv.org/abs/2004.05150>`__ by Iz Beltagy, Matthew E.
    Peters, Arman Cohan.
18. `LXMERT <https://github.com/airsplay/lxmert>`__ (from UNC Chapel Hill) released with the paper `LXMERT: Learning
    Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answering
    <https://arxiv.org/abs/1908.07490>`__ by Hao Tan and Mohit Bansal.
19. `MarianMT <https://huggingface.co/transformers/model_doc/marian.html>`__ Machine translation models trained using
    `OPUS <http://opus.nlpl.eu/>`__ data by J枚rg Tiedemann. The `Marian Framework <https://marian-nmt.github.io/>`__ is
    being developed by the Microsoft Translator Team.
20. `MBart <https://github.com/pytorch/fairseq/tree/master/examples/mbart>`__ (from Facebook) released with the paper 
    `Multilingual Denoising Pre-training for Neural Machine Translation <https://arxiv.org/abs/2001.08210>`__ by Yinhan
129
    Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
130
131
132
133
134
21. `MMBT <https://github.com/facebookresearch/mmbt/>`__ (from Facebook), released together with the paper a
    `Supervised Multimodal Bitransformers for Classifying Images and Text <https://arxiv.org/pdf/1909.02950.pdf>`__ by
    Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine.
22. `Pegasus <https://github.com/google-research/pegasus>`__ (from Google) released with the paper `PEGASUS:
    Pre-training with Extracted Gap-sentences for Abstractive Summarization <https://arxiv.org/abs/1912.08777>`__> by
135
    Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu.
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
23. `Reformer <https://huggingface.co/transformers/model_doc/reformer.html>`__ (from Google Research) released with the
    paper `Reformer: The Efficient Transformer <https://arxiv.org/abs/2001.04451>`__ by Nikita Kitaev, 艁ukasz Kaiser,
    Anselm Levskaya.
24. `RoBERTa <https://huggingface.co/transformers/model_doc/roberta.html>`__ (from Facebook), released together with
    the paper a `Robustly Optimized BERT Pretraining Approach <https://arxiv.org/abs/1907.11692>`__ by Yinhan Liu, Myle
    Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
    ultilingual BERT into `DistilmBERT
    <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__ and a German version of
    DistilBERT.
25. `SqueezeBert <https://huggingface.co/transformers/model_doc/squeezebert.html>`__ released with the paper
    `SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
    <https://arxiv.org/abs/2006.11316>`__ by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, and Kurt W. Keutzer.
26. `T5 <https://huggingface.co/transformers/model_doc/t5.html>`__ (from Google AI) released with the paper `Exploring
    the Limits of Transfer Learning with a Unified Text-to-Text Transformer <https://arxiv.org/abs/1910.10683>`__ by
    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi
    Zhou and Wei Li and Peter J. Liu.
27. `Transformer-XL <https://huggingface.co/transformers/model_doc/transformerxl.html>`__ (from Google/CMU) released
    with the paper `Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
    <https://arxiv.org/abs/1901.02860>`__ by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le,
    Ruslan Salakhutdinov.
28. `XLM <https://huggingface.co/transformers/model_doc/xlm.html>`__ (from Facebook) released together with the paper
    `Cross-lingual Language Model Pretraining <https://arxiv.org/abs/1901.07291>`__ by Guillaume Lample and Alexis
    Conneau.
29. `XLM-RoBERTa <https://huggingface.co/transformers/model_doc/xlmroberta.html>`__ (from Facebook AI), released
    together with the paper `Unsupervised Cross-lingual Representation Learning at Scale
    <https://arxiv.org/abs/1911.02116>`__ by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary,
    Guillaume Wenzek, Francisco Guzm谩n, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
30. `XLNet <https://huggingface.co/transformers/model_doc/xlnet.html>`__ (from Google/CMU) released with the paper
    `鈥媂LNet: Generalized Autoregressive Pretraining for Language Understanding <https://arxiv.org/abs/1906.08237>`__ by
    Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
31. `Other community models <https://huggingface.co/models>`__, contributed by the `community
    <https://huggingface.co/users>`__.
LysandreJik's avatar
LysandreJik committed
168

169
170
.. toctree::
    :maxdepth: 2
171
    :caption: Get started
172

Sylvain Gugger's avatar
Sylvain Gugger committed
173
    quicktour
174
    installation
Sylvain Gugger's avatar
Sylvain Gugger committed
175
    philosophy
Lysandre's avatar
Lysandre committed
176
    glossary
177
178
179

.. toctree::
    :maxdepth: 2
Sylvain Gugger's avatar
Sylvain Gugger committed
180
    :caption: Using 馃 Transformers
181

Sylvain Gugger's avatar
Sylvain Gugger committed
182
183
    task_summary
    model_summary
Sylvain Gugger's avatar
Sylvain Gugger committed
184
    preprocessing
185
    training
186
    model_sharing
Sylvain Gugger's avatar
Sylvain Gugger committed
187
    tokenizer_summary
188
189
190
191
192
193
194
    multilingual

.. toctree::
    :maxdepth: 2
    :caption: Advanced guides

    pretrained_models
195
    examples
196
    custom_datasets
197
    notebooks
198
    converting_tensorflow_models
199
    migration
200
    contributing
201
    testing
Funtowicz Morgan's avatar
Funtowicz Morgan committed
202
    serialization
203
204
205
206
207
208

.. toctree::
    :maxdepth: 2
    :caption: Research

    bertology
209
    perplexity
210
    benchmarks
211

thomwolf's avatar
thomwolf committed
212
213
.. toctree::
    :maxdepth: 2
214
    :caption: Main Classes
thomwolf's avatar
thomwolf committed
215

Sylvain Gugger's avatar
Sylvain Gugger committed
216
    main_classes/callback
thomwolf's avatar
thomwolf committed
217
    main_classes/configuration
218
    main_classes/logging
thomwolf's avatar
thomwolf committed
219
220
    main_classes/model
    main_classes/optimizer_schedules
221
222
    main_classes/output
    main_classes/pipelines
LysandreJik's avatar
LysandreJik committed
223
    main_classes/processors
224
225
226
227
228
229
230
231
    main_classes/tokenizer
    main_classes/trainer

.. toctree::
    :maxdepth: 2
    :caption: Models

    model_doc/albert
thomwolf's avatar
thomwolf committed
232
    model_doc/auto
233
    model_doc/bart
234
    model_doc/bert
235
    model_doc/bertgeneration
Sam Shleifer's avatar
Sam Shleifer committed
236
    model_doc/blenderbot
Lysandre's avatar
Lysandre committed
237
    model_doc/camembert
238
    model_doc/ctrl
Pengcheng He's avatar
Pengcheng He committed
239
    model_doc/deberta
240
    model_doc/dialogpt
241
    model_doc/distilbert
Quentin Lhoest's avatar
Quentin Lhoest committed
242
    model_doc/dpr
243
244
245
    model_doc/electra
    model_doc/encoderdecoder
    model_doc/flaubert
246
    model_doc/fsmt
Sylvain Gugger's avatar
Sylvain Gugger committed
247
    model_doc/funnel
Minghao Li's avatar
Minghao Li committed
248
    model_doc/layoutlm
249
250
251
252
253
254
255
256
    model_doc/longformer
    model_doc/lxmert
    model_doc/marian
    model_doc/mbart
    model_doc/mobilebert
    model_doc/gpt
    model_doc/gpt2
    model_doc/pegasus
Sylvain Gugger's avatar
Sylvain Gugger committed
257
    model_doc/rag
258
259
260
    model_doc/reformer
    model_doc/retribert
    model_doc/roberta
261
    model_doc/squeezebert
262
263
264
265
266
267
268
269
270
271
    model_doc/t5
    model_doc/transformerxl
    model_doc/xlm
    model_doc/xlmroberta
    model_doc/xlnet

.. toctree::
    :maxdepth: 2
    :caption: Internal Helpers

Sylvain Gugger's avatar
Sylvain Gugger committed
272
    internal/modeling_utils
273
    internal/pipelines_utils
274
    internal/tokenization_utils
Sylvain Gugger's avatar
Sylvain Gugger committed
275
    internal/trainer_utils