README.md 39.9 KB
Newer Older
thomwolf's avatar
thomwolf committed
1
2
<p align="center">
    <br>
thomwolf's avatar
thomwolf committed
3
    <img src="https://raw.githubusercontent.com/huggingface/transformers/master/docs/source/imgs/transformers_logo_name.png" width="400"/>
thomwolf's avatar
thomwolf committed
4
5
6
    <br>
<p>
<p align="center">
Lysandre Debut's avatar
Lysandre Debut committed
7
    <a href="https://circleci.com/gh/huggingface/transformers">
thomwolf's avatar
thomwolf committed
8
        <img alt="Build" src="https://img.shields.io/circleci/build/github/huggingface/transformers/master">
thomwolf's avatar
thomwolf committed
9
10
    </a>
    <a href="https://github.com/huggingface/transformers/blob/master/LICENSE">
thomwolf's avatar
thomwolf committed
11
        <img alt="GitHub" src="https://img.shields.io/github/license/huggingface/transformers.svg?color=blue">
thomwolf's avatar
thomwolf committed
12
13
    </a>
    <a href="https://huggingface.co/transformers/index.html">
thomwolf's avatar
thomwolf committed
14
        <img alt="Documentation" src="https://img.shields.io/website/http/huggingface.co/transformers/index.html.svg?down_color=red&down_message=offline&up_message=online">
thomwolf's avatar
thomwolf committed
15
16
    </a>
    <a href="https://github.com/huggingface/transformers/releases">
thomwolf's avatar
thomwolf committed
17
        <img alt="GitHub release" src="https://img.shields.io/github/release/huggingface/transformers.svg">
thomwolf's avatar
thomwolf committed
18
19
20
    </a>
</p>

thomwolf's avatar
thomwolf committed
21
22
23
<h3 align="center">
<p>State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch
</h3>
thomwolf's avatar
thomwolf committed
24

25
🤗 Transformers (formerly known as `pytorch-transformers` and `pytorch-pretrained-bert`) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, T5, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over thousands of pretrained models in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch.
thomwolf's avatar
thomwolf committed
26

Clement's avatar
Clement committed
27
28
[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/0)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/0)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/1)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/1)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/2)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/2)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/3)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/3)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/4)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/4)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/5)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/5)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/6)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/6)[![](https://sourcerer.io/fame/clmnt/huggingface/transformers/images/7)](https://sourcerer.io/fame/clmnt/huggingface/transformers/links/7)

thomwolf's avatar
thomwolf committed
29
### Features
thomwolf's avatar
thomwolf committed
30
31
32
33
34
35
36
37
38
39
40
- High performance on NLU and NLG tasks
- Low barrier to entry for educators and practitioners

State-of-the-art NLP for everyone
- Deep learning researchers
- Hands-on practitioners
- AI/ML/NLP teachers and educators

Lower compute costs, smaller carbon footprint
- Researchers can share trained models instead of always retraining
- Practitioners can reduce compute time and production costs
41
- Dozens of architectures with over 1,000 pretrained models, some in more than 100 languages
thomwolf's avatar
thomwolf committed
42
43
44

Choose the right framework for every part of a model's lifetime
- Train state-of-the-art models in 3 lines of code
thomwolf's avatar
thomwolf committed
45
46
- Deep interoperability between TensorFlow 2.0 and PyTorch models
- Move a single model between TF2.0/PyTorch frameworks at will
thomwolf's avatar
thomwolf committed
47
- Seamlessly pick the right framework for training, evaluation, production
Julien Chaumond's avatar
Julien Chaumond committed
48

thomwolf's avatar
indeed  
thomwolf committed
49

thomwolf's avatar
thomwolf committed
50
51
52
| Section | Description |
|-|-|
| [Installation](#installation) | How to install the package |
thomwolf's avatar
thomwolf committed
53
| [Model architectures](#model-architectures) | Architectures (with pretrained weights) |
thomwolf's avatar
thomwolf committed
54
55
| [Online demo](#online-demo) | Experimenting with this repo’s text generation capabilities |
| [Quick tour: Usage](#quick-tour) | Tokenizers & models usage: Bert and GPT-2 |
wangfei's avatar
wangfei committed
56
| [Quick tour: TF 2.0 and PyTorch ](#Quick-tour-TF-20-training-and-PyTorch-interoperability) | Train a TF 2.0 model in 10 lines of code, load it in PyTorch |
thomwolf's avatar
thomwolf committed
57
| [Quick tour: pipelines](#quick-tour-of-pipelines) | Using Pipelines: Wrapper around tokenizer and models to use finetuned models |
thomwolf's avatar
thomwolf committed
58
| [Quick tour: Fine-tuning/usage scripts](#quick-tour-of-the-fine-tuningusage-scripts) | Using provided scripts: GLUE, SQuAD and Text generation |
59
| [Quick tour: Share your models ](#Quick-tour-of-model-sharing) | Upload and share your fine-tuned models with the community |
60
| [Migrating from pytorch-transformers to transformers](#Migrating-from-pytorch-transformers-to-transformers) | Migrating your code from pytorch-transformers to transformers |
thomwolf's avatar
thomwolf committed
61
| [Migrating from pytorch-pretrained-bert to pytorch-transformers](#Migrating-from-pytorch-pretrained-bert-to-transformers) | Migrating your code from pytorch-pretrained-bert to transformers |
62
| [Documentation][(v2.5.0)](https://huggingface.co/transformers/v2.5.0)[(v2.4.0/v2.4.1)](https://huggingface.co/transformers/v2.4.0)[(v2.3.0)](https://huggingface.co/transformers/v2.3.0)[(v2.2.0/v2.2.1/v2.2.2)](https://huggingface.co/transformers/v2.2.0) [(v2.1.1)](https://huggingface.co/transformers/v2.1.1) [(v2.0.0)](https://huggingface.co/transformers/v2.0.0) [(v1.2.0)](https://huggingface.co/transformers/v1.2.0) [(v1.1.0)](https://huggingface.co/transformers/v1.1.0) [(v1.0.0)](https://huggingface.co/transformers/v1.0.0) [(master)](https://huggingface.co/transformers) | Full API documentation and more |
thomwolf's avatar
thomwolf committed
63

thomwolf's avatar
thomwolf committed
64
## Installation
VictorSanh's avatar
VictorSanh committed
65

66
This repo is tested on Python 3.6+, PyTorch 1.0.0+ and TensorFlow 2.0.0-rc1
VictorSanh's avatar
VictorSanh committed
67

68
69
70
71
72
73
You should install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).

Create a virtual environment with the version of Python you're going to use and activate it.

Now, if you want to use 🤗 Transformers, you can install it with pip. If you'd like to play with the examples, you must install it from source.

thomwolf's avatar
thomwolf committed
74
### With pip
thomwolf's avatar
thomwolf committed
75

76
First you need to install one of, or both, TensorFlow 2.0 and PyTorch.
Christopher Goh's avatar
Christopher Goh committed
77
Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available) and/or [PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform.
78
79

When TensorFlow 2.0 and/or PyTorch has been installed, 🤗 Transformers can be installed using pip as follows:
thomwolf's avatar
thomwolf committed
80

thomwolf's avatar
thomwolf committed
81
```bash
82
pip install transformers
thomwolf's avatar
thomwolf committed
83
```
VictorSanh's avatar
VictorSanh committed
84

thomwolf's avatar
thomwolf committed
85
### From source
thomwolf's avatar
thomwolf committed
86

87
Here also, you first need to install one of, or both, TensorFlow 2.0 and PyTorch.
Christopher Goh's avatar
Christopher Goh committed
88
Please refer to [TensorFlow installation page](https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available) and/or [PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform.
89

DenysNahurnyi's avatar
DenysNahurnyi committed
90
When TensorFlow 2.0 and/or PyTorch has been installed, you can install from source by cloning the repository and running:
thomwolf's avatar
thomwolf committed
91

thomwolf's avatar
thomwolf committed
92
```bash
93
94
git clone https://github.com/huggingface/transformers
cd transformers
95
pip install .
thomwolf's avatar
thomwolf committed
96
```
VictorSanh's avatar
VictorSanh committed
97

98
99
100
101
When you update the repository, you should upgrade the transformers installation and its dependencies as follows:

```bash
git pull
102
pip install --upgrade .
103
104
```

105
106
107
108
### Run the examples

Examples are included in the repository but are not shipped with the library.

109
Therefore, in order to run the latest versions of the examples, you need to install from source, as described above.
110

111
Look at the [README](https://github.com/huggingface/transformers/blob/master/examples/README.md) for how to run examples.
thomwolf's avatar
thomwolf committed
112

113
### Tests
thomwolf's avatar
thomwolf committed
114

115
A series of tests are included for the library and for some example scripts. Library tests can be found in the [tests folder](https://github.com/huggingface/transformers/tree/master/tests) and examples tests in the [examples folder](https://github.com/huggingface/transformers/tree/master/examples).
thomwolf's avatar
thomwolf committed
116

117
118
Depending on which framework is installed (TensorFlow 2.0 and/or PyTorch), the irrelevant tests will be skipped. Ensure that both frameworks are installed if you want to execute all tests.

119
Here's the easiest way to run tests for the library:
thomwolf's avatar
thomwolf committed
120

121
```bash
122
pip install -e ".[testing]"
123
make test
124
125
```

126
and for the examples:
127

thomwolf's avatar
thomwolf committed
128
```bash
129
pip install -e ".[testing]"
130
131
pip install -r examples/requirements.txt
make test-examples
thomwolf's avatar
thomwolf committed
132
```
thomwolf's avatar
thomwolf committed
133

134
For details, refer to the [contributing guide](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#tests).
135

136
137
138
139
### Do you want to run a Transformer model on a mobile device?

You should check out our [`swift-coreml-transformers`](https://github.com/huggingface/swift-coreml-transformers) repo.

140
It contains a set of tools to convert PyTorch or TensorFlow 2.0 trained Transformer models (currently contains `GPT-2`, `DistilGPT-2`, `BERT`, and `DistilBERT`) to CoreML models that run on iOS devices.
141

142
At some point in the future, you'll be able to seamlessly move from pre-training or fine-tuning models to productizing them in CoreML, or prototype a model or an app in CoreML then research its hyperparameters or architecture from TensorFlow 2.0 and/or PyTorch. Super exciting!
143

thomwolf's avatar
thomwolf committed
144
145
## Model architectures

146
🤗 Transformers currently provides the following NLU/NLG architectures:
thomwolf's avatar
thomwolf committed
147

148
149
150
151
152
153
154
155
156
157
158
159
160
1. **[BERT](https://huggingface.co/transformers/model_doc/bert.html)** (from Google) released with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
2. **[GPT](https://huggingface.co/transformers/model_doc/gpt.html)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.
3. **[GPT-2](https://huggingface.co/transformers/model_doc/gpt2.html)** (from OpenAI) released with the paper [Language Models are Unsupervised Multitask Learners](https://blog.openai.com/better-language-models/) by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.
4. **[Transformer-XL](https://huggingface.co/transformers/model_doc/transformerxl.html)** (from Google/CMU) released with the paper [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
5. **[XLNet](https://huggingface.co/transformers/model_doc/xlnet.html)** (from Google/CMU) released with the paper [​XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
6. **[XLM](https://huggingface.co/transformers/model_doc/xlm.html)** (from Facebook) released together with the paper [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau.
7. **[RoBERTa](https://huggingface.co/transformers/model_doc/roberta.html)** (from Facebook), released together with the paper a [Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
8. **[DistilBERT](https://huggingface.co/transformers/model_doc/distilbert.html)** (from HuggingFace), released together with the paper [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108) by Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 into [DistilGPT2](https://github.com/huggingface/transformers/tree/master/examples/distillation), RoBERTa into [DistilRoBERTa](https://github.com/huggingface/transformers/tree/master/examples/distillation), Multilingual BERT into [DistilmBERT](https://github.com/huggingface/transformers/tree/master/examples/distillation) and a German version of DistilBERT.
9. **[CTRL](https://huggingface.co/transformers/model_doc/ctrl.html)** (from Salesforce) released with the paper [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://arxiv.org/abs/1909.05858) by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Richard Socher.
10. **[CamemBERT](https://huggingface.co/transformers/model_doc/camembert.html)** (from Inria/Facebook/Sorbonne) released with the paper [CamemBERT: a Tasty French Language Model](https://arxiv.org/abs/1911.03894) by Louis Martin*, Benjamin Muller*, Pedro Javier Ortiz Suárez*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.
11. **[ALBERT](https://huggingface.co/transformers/model_doc/albert.html)** (from Google Research and the Toyota Technological Institute at Chicago) released with the paper [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942), by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.
12. **[T5](https://huggingface.co/transformers/model_doc/t5.html)** (from Google AI) released with the paper [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.
13. **[XLM-RoBERTa](https://huggingface.co/transformers/model_doc/xlmroberta.html)** (from Facebook AI), released together with the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.
thomwolf's avatar
thomwolf committed
161
14. **[MMBT](https://github.com/facebookresearch/mmbt/)** (from Facebook), released together with the paper a [Supervised Multimodal Bitransformers for Classifying Images and Text](https://arxiv.org/pdf/1909.02950.pdf) by Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine.
162
163
164
15. **[FlauBERT](https://huggingface.co/transformers/model_doc/flaubert.html)** (from CNRS) released with the paper [FlauBERT: Unsupervised Language Model Pre-training for French](https://arxiv.org/abs/1912.05372) by Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.
16. **[BART](https://huggingface.co/transformers/model_doc/bart.html)** (from Facebook) released with the paper [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://arxiv.org/pdf/1910.13461.pdf) by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer.
17. **[ELECTRA](https://huggingface.co/transformers/model_doc/electra.html)** (from Google Research/Stanford University) released with the paper [ELECTRA: Pre-training text encoders as discriminators rather than generators](https://arxiv.org/abs/2003.10555) by Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.
Patrick von Platen's avatar
Patrick von Platen committed
165
18. **[DialoGPT](https://huggingface.co/transformers/model_doc/dialogpt.html)** (from Microsoft Research) released with the paper [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://arxiv.org/abs/1911.00536) by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.
Lysandre Debut's avatar
Lysandre Debut committed
166
167
18. **[Other community models](https://huggingface.co/models)**, contributed by the [community](https://huggingface.co/users).
19. Want to contribute a new model? We have added a **detailed guide and templates** to guide you in the process of adding a new model. You can find them in the [`templates`](./templates) folder of the repository. Be sure to check the [contributing guidelines](./CONTRIBUTING.md) and contact the maintainers or open an issue to collect feedbacks before starting your PR.
thomwolf's avatar
thomwolf committed
168
169
170

These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Peason R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/transformers/examples.html).

171
172
173
174
175
176
177
## Online demo

**[Write With Transformer](https://transformer.huggingface.co)**, built by the Hugging Face team at transformer.huggingface.co, is the official demo of this repo’s text generation capabilities.
You can use it to experiment with completions generated by `GPT2Model`, `TransfoXLModel`, and `XLNetModel`.

> “🦄 Write with transformer is to writing what calculators are to calculus.”

Julien Chaumond's avatar
Julien Chaumond committed
178
![write_with_transformer](https://transformer.huggingface.co/front/assets/thumbnail-large.png)
179

thomwolf's avatar
thomwolf committed
180
## Quick tour
thomwolf's avatar
thomwolf committed
181

thomwolf's avatar
thomwolf committed
182
Let's do a very quick overview of the model architectures in 🤗 Transformers. Detailed examples for each model architecture (Bert, GPT, GPT-2, Transformer-XL, XLNet and XLM) can be found in the [full documentation](https://huggingface.co/transformers/).
thomwolf's avatar
thomwolf committed
183
184
185

```python
import torch
186
from transformers import *
thomwolf's avatar
thomwolf committed
187

188
# Transformers has a unified API
189
# for 10 transformer architectures and 30 pretrained weights.
thomwolf's avatar
thomwolf committed
190
#          Model          | Tokenizer          | Pretrained weights shortcut
thomwolf's avatar
thomwolf committed
191
192
193
MODELS = [(BertModel,       BertTokenizer,       'bert-base-uncased'),
          (OpenAIGPTModel,  OpenAIGPTTokenizer,  'openai-gpt'),
          (GPT2Model,       GPT2Tokenizer,       'gpt2'),
keskarnitish's avatar
keskarnitish committed
194
          (CTRLModel,       CTRLTokenizer,       'ctrl'),
thomwolf's avatar
thomwolf committed
195
196
197
          (TransfoXLModel,  TransfoXLTokenizer,  'transfo-xl-wt103'),
          (XLNetModel,      XLNetTokenizer,      'xlnet-base-cased'),
          (XLMModel,        XLMTokenizer,        'xlm-mlm-enfr-1024'),
198
          (DistilBertModel, DistilBertTokenizer, 'distilbert-base-cased'),
199
200
201
          (RobertaModel,    RobertaTokenizer,    'roberta-base'),
          (XLMRobertaModel, XLMRobertaTokenizer, 'xlm-roberta-base'),
         ]
thomwolf's avatar
thomwolf committed
202

thomwolf's avatar
thomwolf committed
203
204
# To use TensorFlow 2.0 versions of the models, simply prefix the class names with 'TF', e.g. `TFRobertaModel` is the TF 2.0 counterpart of the PyTorch model `RobertaModel`

thomwolf's avatar
thomwolf committed
205
206
207
208
209
210
211
# Let's encode some text in a sequence of hidden-states using each model:
for model_class, tokenizer_class, pretrained_weights in MODELS:
    # Load pretrained model/tokenizer
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights)

    # Encode text
212
    input_ids = torch.tensor([tokenizer.encode("Here is some text to encode", add_special_tokens=True)])  # Add special tokens takes care of adding [CLS], [SEP], <s>... tokens in the right way for each model.
Thomas Wolf's avatar
Thomas Wolf committed
213
214
    with torch.no_grad():
        last_hidden_states = model(input_ids)[0]  # Models outputs are now tuples
thomwolf's avatar
thomwolf committed
215
216
217

# Each architecture is provided with several class for fine-tuning on down-stream tasks, e.g.
BERT_MODEL_CLASSES = [BertModel, BertForPreTraining, BertForMaskedLM, BertForNextSentencePrediction,
thomwolf's avatar
thomwolf committed
218
                      BertForSequenceClassification, BertForTokenClassification, BertForQuestionAnswering]
thomwolf's avatar
thomwolf committed
219

thomwolf's avatar
thomwolf committed
220
221
222
# All the classes for an architecture can be initiated from pretrained weights for this architecture
# Note that additional weights added for fine-tuning are only initialized
# and need to be trained on the down-stream task
223
224
pretrained_weights = 'bert-base-uncased'
tokenizer = BertTokenizer.from_pretrained(pretrained_weights)
thomwolf's avatar
thomwolf committed
225
226
for model_class in BERT_MODEL_CLASSES:
    # Load pretrained model/tokenizer
227
    model = model_class.from_pretrained(pretrained_weights)
thomwolf's avatar
thomwolf committed
228

Santosh Gupta's avatar
Santosh Gupta committed
229
230
231
232
233
234
    # Models can return full list of hidden-states & attentions weights at each layer
    model = model_class.from_pretrained(pretrained_weights,
                                        output_hidden_states=True,
                                        output_attentions=True)
    input_ids = torch.tensor([tokenizer.encode("Let's see all hidden-states and attentions on this text")])
    all_hidden_states, all_attentions = model(input_ids)[-2:]
thomwolf's avatar
thomwolf committed
235

Santosh Gupta's avatar
Santosh Gupta committed
236
237
238
    # Models are compatible with Torchscript
    model = model_class.from_pretrained(pretrained_weights, torchscript=True)
    traced_model = torch.jit.trace(model, (input_ids,))
thomwolf's avatar
thomwolf committed
239

Santosh Gupta's avatar
Santosh Gupta committed
240
241
242
243
244
    # Simple serialization for models and tokenizers
    model.save_pretrained('./directory/to/save/')  # save
    model = model_class.from_pretrained('./directory/to/save/')  # re-load
    tokenizer.save_pretrained('./directory/to/save/')  # save
    tokenizer = BertTokenizer.from_pretrained('./directory/to/save/')  # re-load
thomwolf's avatar
thomwolf committed
245

Santosh Gupta's avatar
Santosh Gupta committed
246
    # SOTA examples for GLUE, SQUAD, text generation...
thomwolf's avatar
thomwolf committed
247
248
```

thomwolf's avatar
thomwolf committed
249
250
251
252
253
254
255
## Quick tour TF 2.0 training and PyTorch interoperability

Let's do a quick example of how a TensorFlow 2.0 model can be trained in 12 lines of code with 🤗 Transformers and then loaded in PyTorch for fast inspection/tests.

```python
import tensorflow as tf
import tensorflow_datasets
thomwolf's avatar
thomwolf committed
256
from transformers import *
thomwolf's avatar
thomwolf committed
257
258
259
260
261
262
263

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

# Prepare dataset for GLUE as a tf.data.Dataset instance
thomwolf's avatar
thomwolf committed
264
265
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
thomwolf's avatar
thomwolf committed
266
267
268
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

269
# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
thomwolf's avatar
thomwolf committed
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)

# Load the TensorFlow model in PyTorch for inspection
model.save_pretrained('./save/')
pytorch_model = BertForSequenceClassification.from_pretrained('./save/', from_tf=True)

# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the task
sentence_0 = "This research was consistent with his findings."
sentence_1 = "His findings were compatible with this research."
sentence_2 = "His findings were not compatible with this research."
inputs_1 = tokenizer.encode_plus(sentence_0, sentence_1, add_special_tokens=True, return_tensors='pt')
inputs_2 = tokenizer.encode_plus(sentence_0, sentence_2, add_special_tokens=True, return_tensors='pt')

290
291
292
pred_1 = pytorch_model(inputs_1['input_ids'], token_type_ids=inputs_1['token_type_ids'])[0].argmax().item()
pred_2 = pytorch_model(inputs_2['input_ids'], token_type_ids=inputs_2['token_type_ids'])[0].argmax().item()

thomwolf's avatar
thomwolf committed
293
294
295
296
print("sentence_1 is", "a paraphrase" if pred_1 else "not a paraphrase", "of sentence_0")
print("sentence_2 is", "a paraphrase" if pred_2 else "not a paraphrase", "of sentence_0")
```

thomwolf's avatar
thomwolf committed
297
## Quick tour of the fine-tuning/usage scripts
thomwolf's avatar
thomwolf committed
298

299
**Important**
300
301
302
303
Before running the fine-tuning scripts, please read the
[instructions](#run-the-examples) on how to
setup your environment to run the examples.

thomwolf's avatar
thomwolf committed
304
The library comprises several example scripts with SOTA performances for NLU and NLG tasks:
thomwolf's avatar
thomwolf committed
305

Julien Chaumond's avatar
Julien Chaumond committed
306
307
308
- `run_glue.py`: an example fine-tuning sequence classification models on nine different GLUE tasks (*sequence-level classification*)
- `run_squad.py`: an example fine-tuning question answering models on the question answering dataset SQuAD 2.0 (*token-level classification*)
- `run_ner.py`: an example fine-tuning token classification models on named entity recognition (*token-level classification*)
keskarnitish's avatar
keskarnitish committed
309
- `run_generation.py`: an example using GPT, GPT-2, CTRL, Transformer-XL and XLNet for conditional language generation
thomwolf's avatar
thomwolf committed
310
- other model-specific examples (see the documentation).
thomwolf's avatar
thomwolf committed
311

thomwolf's avatar
thomwolf committed
312
Here are three quick usage examples for these scripts:
thomwolf's avatar
thomwolf committed
313

thomwolf's avatar
thomwolf committed
314
### `run_glue.py`: Fine-tuning on GLUE tasks for sequence classification
thomwolf's avatar
thomwolf committed
315

thomwolf's avatar
thomwolf committed
316
The [General Language Understanding Evaluation (GLUE) benchmark](https://gluebenchmark.com/) is a collection of nine sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.
thomwolf's avatar
thomwolf committed
317

Julien Chaumond's avatar
Julien Chaumond committed
318
Before running any of these GLUE tasks you should download the
thomwolf's avatar
thomwolf committed
319
320
321
[GLUE data](https://gluebenchmark.com/tasks) by running
[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
and unpack it to some directory `$GLUE_DIR`.
thomwolf's avatar
thomwolf committed
322

323
324
325
326
327
328
You should also install the additional packages required by the examples:

```shell
pip install -r ./examples/requirements.txt
```

thomwolf's avatar
thomwolf committed
329
330
331
```shell
export GLUE_DIR=/path/to/glue
export TASK_NAME=MRPC
thomwolf's avatar
thomwolf committed
332

333
334
335
336
337
338
339
340
341
342
343
344
python ./examples/run_glue.py \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/$TASK_NAME/
thomwolf's avatar
thomwolf committed
345
346
```

thomwolf's avatar
thomwolf committed
347
where task name can be one of CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI.
thomwolf's avatar
thomwolf committed
348

thomwolf's avatar
thomwolf committed
349
The dev set results will be present within the text file 'eval_results.txt' in the specified output_dir. In case of MNLI, since there are two separate dev sets, matched and mismatched, there will be a separate output folder called '/tmp/MNLI-MM/' in addition to '/tmp/MNLI/'.
thomwolf's avatar
thomwolf committed
350

thomwolf's avatar
thomwolf committed
351
#### Fine-tuning XLNet model on the STS-B regression task
thomwolf's avatar
thomwolf committed
352

thomwolf's avatar
thomwolf committed
353
This example code fine-tunes XLNet on the STS-B corpus using parallel training on a server with 4 V100 GPUs.
354
Parallel training is a simple way to use several GPUs (but is slower and less flexible than distributed training, see below).
thomwolf's avatar
thomwolf committed
355

thomwolf's avatar
thomwolf committed
356
357
```shell
export GLUE_DIR=/path/to/glue
thomwolf's avatar
thomwolf committed
358

thomwolf's avatar
thomwolf committed
359
360
361
python ./examples/run_glue.py \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
362
    --do_eval   \
thomwolf's avatar
thomwolf committed
363
364
365
366
367
368
369
370
371
372
373
374
    --task_name=sts-b     \
    --data_dir=${GLUE_DIR}/STS-B  \
    --output_dir=./proc_data/sts-b-110   \
    --max_seq_length=128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --gradient_accumulation_steps=1 \
    --max_steps=1200  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --warmup_steps=120
thomwolf's avatar
thomwolf committed
375
376
```

Anthony MOI's avatar
Anthony MOI committed
377
On this machine we thus have a batch size of 32, please increase `gradient_accumulation_steps` to reach the same batch size if you have a smaller machine. These hyper-parameters should result in a Pearson correlation coefficient of `+0.917` on the development set.
thomwolf's avatar
thomwolf committed
378

thomwolf's avatar
thomwolf committed
379
#### Fine-tuning Bert model on the MRPC classification task
thomwolf's avatar
thomwolf committed
380

thomwolf's avatar
thomwolf committed
381
This example code fine-tunes the Bert Whole Word Masking model on the Microsoft Research Paraphrase Corpus (MRPC) corpus using distributed training on 8 V100 GPUs to reach a F1 > 92.
thomwolf's avatar
thomwolf committed
382

thomwolf's avatar
thomwolf committed
383
```bash
384
python -m torch.distributed.launch --nproc_per_node 8 ./examples/run_glue.py   \
thomwolf's avatar
thomwolf committed
385
386
387
388
389
390
391
392
393
394
395
396
397
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --task_name MRPC \
    --do_train   \
    --do_eval   \
    --data_dir $GLUE_DIR/MRPC/   \
    --max_seq_length 128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5   \
    --num_train_epochs 3.0  \
    --output_dir /tmp/mrpc_output/ \
    --overwrite_output_dir   \
    --overwrite_cache \
thomwolf's avatar
thomwolf committed
398
399
```

thomwolf's avatar
thomwolf committed
400
Training with these hyper-parameters gave us the following results:
thomwolf's avatar
thomwolf committed
401

thomwolf's avatar
thomwolf committed
402
403
404
405
406
407
408
```bash
  acc = 0.8823529411764706
  acc_and_f1 = 0.901702786377709
  eval_loss = 0.3418912578906332
  f1 = 0.9210526315789473
  global_step = 174
  loss = 0.07231863956341798
thomwolf's avatar
thomwolf committed
409
410
```

thomwolf's avatar
thomwolf committed
411
### `run_squad.py`: Fine-tuning on SQuAD for question-answering
thomwolf's avatar
thomwolf committed
412

thomwolf's avatar
thomwolf committed
413
This example code fine-tunes BERT on the SQuAD dataset using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD:
thomwolf's avatar
thomwolf committed
414

thomwolf's avatar
thomwolf committed
415
```bash
416
python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
thomwolf's avatar
thomwolf committed
417
418
419
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
thomwolf's avatar
thomwolf committed
420
    --do_eval \
thomwolf's avatar
thomwolf committed
421
422
423
424
425
426
427
428
429
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \
thomwolf's avatar
thomwolf committed
430
431
```

thomwolf's avatar
thomwolf committed
432
Training with these hyper-parameters gave us the following results:
thomwolf's avatar
thomwolf committed
433

thomwolf's avatar
thomwolf committed
434
435
436
```bash
python $SQUAD_DIR/evaluate-v1.1.py $SQUAD_DIR/dev-v1.1.json ../models/wwm_uncased_finetuned_squad/predictions.json
{"exact_match": 86.91579943235573, "f1": 93.1532499015869}
thomwolf's avatar
thomwolf committed
437
438
```

thomwolf's avatar
thomwolf committed
439
This is the model provided as `bert-large-uncased-whole-word-masking-finetuned-squad`.
440

keskarnitish's avatar
keskarnitish committed
441
### `run_generation.py`: Text generation with GPT, GPT-2, CTRL, Transformer-XL and XLNet
442

thomwolf's avatar
thomwolf committed
443
A conditional generation script is also included to generate text from a prompt.
DenysNahurnyi's avatar
DenysNahurnyi committed
444
The generation script includes the [tricks](https://github.com/rusiaaman/XLNet-gen#methodology) proposed by Aman Rusia to get high-quality generation with memory models like Transformer-XL and XLNet (include a predefined text to make short inputs longer).
445

thomwolf's avatar
thomwolf committed
446
Here is how to run the script with the small version of OpenAI GPT-2 model:
447

thomwolf's avatar
thomwolf committed
448
```shell
thomwolf's avatar
thomwolf committed
449
python ./examples/run_generation.py \
thomwolf's avatar
thomwolf committed
450
451
452
    --model_type=gpt2 \
    --length=20 \
    --model_name_or_path=gpt2 \
453
454
```

455
and from the Salesforce CTRL model:
keskarnitish's avatar
keskarnitish committed
456
457
458
459
```shell
python ./examples/run_generation.py \
    --model_type=ctrl \
    --length=20 \
460
    --model_name_or_path=ctrl \
keskarnitish's avatar
keskarnitish committed
461
462
463
464
    --temperature=0 \
    --repetition_penalty=1.2 \
```

465
466
## Quick tour of model sharing

Julien Chaumond's avatar
Julien Chaumond committed
467
Starting with `v2.2.2`, you can now upload and share your fine-tuned models with the community, using the <abbr title="Command-line interface">CLI</abbr> that's built-in to the library.
468

469
**First, create an account on [https://huggingface.co/join](https://huggingface.co/join)**. Optionally, join an existing organization or create a new one. Then:
470
471
472
473
474
475
476
477
478
479
480
481

```shell
transformers-cli login
# log in using the same credentials as on huggingface.co
```
Upload your model:
```shell
transformers-cli upload ./path/to/pretrained_model/

# ^^ Upload folder containing weights/tokenizer/config
# saved via `.save_pretrained()`

Julien Chaumond's avatar
Julien Chaumond committed
482
transformers-cli upload ./config.json [--filename folder/foobar.json]
483
484

# ^^ Upload a single file
Julien Chaumond's avatar
Julien Chaumond committed
485
# (you can optionally override its filename, which can be nested inside a folder)
486
487
```

488
489
490
491
492
493
If you want your model to be namespaced by your organization name rather than your username, add the following flag to any command:
```shell
--organization organization_name
```

Your model will then be accessible through its identifier, a concatenation of your username (or organization name) and the folder name above:
494
```python
Julien Chaumond's avatar
Julien Chaumond committed
495
496
497
"username/pretrained_model"
# or if an org:
"organization_name/pretrained_model"
498
499
```

500
**Please add a README.md model card** to the repo under `model_cards/` with: model description, training params (dataset, preprocessing, hardware used, hyperparameters), evaluation results, intended uses & limitations, etc.
501
502
503

Your model now has a page on huggingface.co/models 🔥

504
505
Anyone can load it from code:
```python
506
507
tokenizer = AutoTokenizer.from_pretrained("namespace/pretrained_model")
model = AutoModel.from_pretrained("namespace/pretrained_model")
508
509
```

510
List all your files on S3:
511
```shell
Julien Chaumond's avatar
Julien Chaumond committed
512
transformers-cli s3 ls
513
514
```

515
You can also delete unneeded files:
Julien Chaumond's avatar
Julien Chaumond committed
516
517
518
519
520

```shell
transformers-cli s3 rm
```

521
522
523
## Quick tour of pipelines

New in version `v2.3`: `Pipeline` are high-level objects which automatically handle tokenization, running your data through a transformers model
524
and outputting the result in a structured object.
525
526

You can create `Pipeline` objects for the following down-stream tasks:
thomwolf's avatar
thomwolf committed
527

528
529
 - `feature-extraction`: Generates a tensor representation for the input sequence
 - `ner`: Generates named entity mapping for each word in the input sequence.
530
 - `sentiment-analysis`: Gives the polarity (positive / negative) of the whole input sequence.
Julien Chaumond's avatar
Julien Chaumond committed
531
532
533
 - `text-classification`: Initialize a `TextClassificationPipeline` directly, or see `sentiment-analysis` for an example.
 - `question-answering`: Provided some context and a question refering to the context, it will extract the answer to the question in the context.
 - `fill-mask`: Takes an input sequence containing a masked token (e.g. `<mask>`) and return list of most probable filled sequences, with their probabilities.
534
535
 - `summarization`
 - `translation_xx_to_yy`
536
537
538
539
540
541
542
543
544
545
546
547

```python
from transformers import pipeline

# Allocate a pipeline for sentiment-analysis
nlp = pipeline('sentiment-analysis')
nlp('We are very happy to include pipeline into the transformers repository.')
>>> {'label': 'POSITIVE', 'score': 0.99893874}

# Allocate a pipeline for question-answering
nlp = pipeline('question-answering')
nlp({
548
    'question': 'What is the name of the repository ?',
549
550
551
552
553
    'context': 'Pipeline have been included in the huggingface/transformers repository'
})
>>> {'score': 0.28756016668193496, 'start': 35, 'end': 59, 'answer': 'huggingface/transformers'}
```

thomwolf's avatar
thomwolf committed
554
555
556
557
558
559
560
561
562
563
564
565
566
## Migrating from pytorch-transformers to transformers

Here is a quick summary of what you should take care of when migrating from `pytorch-transformers` to `transformers`.

### Positional order of some models' keywords inputs (`attention_mask`, `token_type_ids`...) changed

To be able to use Torchscript (see #1010, #1204 and #1195) the specific order of some models **keywords inputs** (`attention_mask`, `token_type_ids`...) has been changed.

If you used to call the models with keyword names for keyword arguments, e.g. `model(inputs_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)`, this should not cause any change.

If you used to call the models with positional inputs for keyword arguments, e.g. `model(inputs_ids, attention_mask, token_type_ids)`, you may have to double check the exact order of input arguments.


567
## Migrating from pytorch-pretrained-bert to transformers
thomwolf's avatar
thomwolf committed
568

thomwolf's avatar
thomwolf committed
569
Here is a quick summary of what you should take care of when migrating from `pytorch-pretrained-bert` to `transformers`.
thomwolf's avatar
thomwolf committed
570
571
572

### Models always output `tuples`

573
The main breaking change when migrating from `pytorch-pretrained-bert` to `transformers` is that every model's forward method always outputs a `tuple` with various elements depending on the model and the configuration parameters.
thomwolf's avatar
thomwolf committed
574

DenysNahurnyi's avatar
DenysNahurnyi committed
575
The exact content of the tuples for each model is detailed in the models' docstrings and the [documentation](https://huggingface.co/transformers/).
thomwolf's avatar
thomwolf committed
576
577
578

In pretty much every case, you will be fine by taking the first element of the output as the output you previously used in `pytorch-pretrained-bert`.

579
Here is a `pytorch-pretrained-bert` to `transformers` conversion example for a `BertForSequenceClassification` classification model:
thomwolf's avatar
thomwolf committed
580
581
582
583
584
585
586
587

```python
# Let's load our model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# If you used to have this line in pytorch-pretrained-bert:
loss = model(input_ids, labels=labels)

588
# Now just use this line in transformers to extract the loss from the output tuple:
thomwolf's avatar
thomwolf committed
589
590
591
outputs = model(input_ids, labels=labels)
loss = outputs[0]

592
# In transformers you can also have access to the logits:
thomwolf's avatar
thomwolf committed
593
594
loss, logits = outputs[:2]

595
# And even the attention weights if you configure the model to output them (and other outputs too, see the docstrings and documentation)
thomwolf's avatar
thomwolf committed
596
597
598
599
600
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', output_attentions=True)
outputs = model(input_ids, labels=labels)
loss, logits, attentions = outputs
```

601
602
603
604
### Using hidden states

By enabling the configuration option `output_hidden_states`, it was possible to retrieve the last hidden states of the encoder. In `pytorch-transformers` as well as `transformers` the return value has changed slightly: `all_hidden_states` now also includes the hidden state of the embeddings in addition to those of the encoding layers. This allows users to easily access the embeddings final state.

thomwolf's avatar
thomwolf committed
605
606
### Serialization

DenysNahurnyi's avatar
DenysNahurnyi committed
607
Breaking change in the `from_pretrained()` method:
608

Christopher Goh's avatar
Christopher Goh committed
609
1. Models are now set in evaluation mode by default when instantiated with the `from_pretrained()` method. To train them, don't forget to set them back in training mode (`model.train()`) to activate the dropout modules.
610

Christopher Goh's avatar
Christopher Goh committed
611
2. The additional `*input` and `**kwargs` arguments supplied to the `from_pretrained()` method used to be directly passed to the underlying model's class `__init__()` method. They are now used to update the model configuration attribute instead, which can break derived model classes built based on the previous `BertForSequenceClassification` examples. We are working on a way to mitigate this breaking change in [#866](https://github.com/huggingface/transformers/pull/866) by forwarding the the model's `__init__()` method (i) the provided positional arguments and (ii) the keyword arguments which do not match any configuration class attributes.
612

thomwolf's avatar
typos  
thomwolf committed
613
Also, while not a breaking change, the serialization methods have been standardized and you probably should switch to the new method `save_pretrained(save_directory)` if you were using any other serialization method before.
thomwolf's avatar
thomwolf committed
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639

Here is an example:

```python
### Let's load a model and tokenizer
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

### Do some stuff to our model and tokenizer
# Ex: add new tokens to the vocabulary and embeddings of our model
tokenizer.add_tokens(['[SPECIAL_TOKEN_1]', '[SPECIAL_TOKEN_2]'])
model.resize_token_embeddings(len(tokenizer))
# Train our model
train(model)

### Now let's save our model and tokenizer to a directory
model.save_pretrained('./my_saved_model_directory/')
tokenizer.save_pretrained('./my_saved_model_directory/')

### Reload the model and the tokenizer
model = BertForSequenceClassification.from_pretrained('./my_saved_model_directory/')
tokenizer = BertTokenizer.from_pretrained('./my_saved_model_directory/')
```

### Optimizers: BertAdam & OpenAIAdam are now AdamW, schedules are standard PyTorch schedules

640
641
642
643
644
645
646
The two optimizers previously included, `BertAdam` and `OpenAIAdam`, have been replaced by a single `AdamW` optimizer which has a few differences:

- it only implements weights decay correction,
- schedules are now externals (see below),
- gradient clipping is now also external (see below).

The new optimizer `AdamW` matches PyTorch `Adam` optimizer API and let you use standard PyTorch or apex methods for the schedule and clipping.
thomwolf's avatar
thomwolf committed
647
648
649
650
651
652
653
654

The schedules are now standard [PyTorch learning rate schedulers](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate) and not part of the optimizer anymore.

Here is a conversion examples from `BertAdam` with a linear warmup and decay schedule to `AdamW` and the same schedule:

```python
# Parameters:
lr = 1e-3
655
max_grad_norm = 1.0
656
num_training_steps = 1000
thomwolf's avatar
thomwolf committed
657
num_warmup_steps = 100
658
warmup_proportion = float(num_warmup_steps) / float(num_training_steps)  # 0.1
thomwolf's avatar
thomwolf committed
659
660

### Previously BertAdam optimizer was instantiated like this:
661
optimizer = BertAdam(model.parameters(), lr=lr, schedule='warmup_linear', warmup=warmup_proportion, t_total=num_training_steps)
thomwolf's avatar
thomwolf committed
662
663
664
665
666
667
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    optimizer.step()

668
### In Transformers, optimizer and schedules are splitted and instantiated like this:
thomwolf's avatar
thomwolf committed
669
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
670
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps)  # PyTorch scheduler
thomwolf's avatar
thomwolf committed
671
672
### and used like this:
for batch in train_data:
673
    model.train()
thomwolf's avatar
thomwolf committed
674
675
    loss = model(batch)
    loss.backward()
676
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
thomwolf's avatar
thomwolf committed
677
    optimizer.step()
thomwolf's avatar
thomwolf committed
678
    scheduler.step()
thomwolf's avatar
thomwolf committed
679
    optimizer.zero_grad()
thomwolf's avatar
thomwolf committed
680
681
```

thomwolf's avatar
thomwolf committed
682
## Citation
thomwolf's avatar
thomwolf committed
683

thomwolf's avatar
thomwolf committed
684
We now have a paper you can cite for the 🤗 Transformers library:
685
```bibtex
thomwolf's avatar
thomwolf committed
686
687
688
689
690
691
@article{Wolf2019HuggingFacesTS,
  title={HuggingFace's Transformers: State-of-the-art Natural Language Processing},
  author={Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and R'emi Louf and Morgan Funtowicz and Jamie Brew},
  journal={ArXiv},
  year={2019},
  volume={abs/1910.03771}
thomwolf's avatar
thomwolf committed
692
693
}
```