README.md 21 KB
Newer Older
thomwolf's avatar
thomwolf committed
1
# 👾 PyTorch-Transformers
VictorSanh's avatar
VictorSanh committed
2

thomwolf's avatar
thomwolf committed
3
[![CircleCI](https://circleci.com/gh/huggingface/pytorch-transformers.svg?style=svg)](https://circleci.com/gh/huggingface/pytorch-transformers)
Julien Chaumond's avatar
Julien Chaumond committed
4

Peiqin Lin's avatar
typos  
Peiqin Lin committed
5
PyTorch-Transformers (formerly known as `pytorch-pretrained-bert`) is a library of state-of-the-art pre-trained models for Natural Language Processing (NLP).
thomwolf's avatar
indeed  
thomwolf committed
6
7

The library currently contains PyTorch implementations, pre-trained model weights, usage scripts and conversion utilities for the following models:
VictorSanh's avatar
VictorSanh committed
8

thomwolf's avatar
thomwolf committed
9
10
11
12
13
14
1. **[BERT](https://github.com/google-research/bert)** (from Google) released with the paper [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
2. **[GPT](https://github.com/openai/finetune-transformer-lm)** (from OpenAI) released with the paper [Improving Language Understanding by Generative Pre-Training](https://blog.openai.com/language-unsupervised/) by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.
3. **[GPT-2](https://blog.openai.com/better-language-models/)** (from OpenAI) released with the paper [Language Models are Unsupervised Multitask Learners](https://blog.openai.com/better-language-models/) by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.
4. **[Transformer-XL](https://github.com/kimiyoung/transformer-xl)** (from Google/CMU) released with the paper [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://arxiv.org/abs/1901.02860) by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
5. **[XLNet](https://github.com/zihangdai/xlnet/)** (from Google/CMU) released with the paper [​XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://arxiv.org/abs/1906.08237) by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.
6. **[XLM](https://github.com/facebookresearch/XLM/)** (from Facebook) released together with the paper [Cross-lingual Language Model Pretraining](https://arxiv.org/abs/1901.07291) by Guillaume Lample and Alexis Conneau.
thomwolf's avatar
thomwolf committed
15
7. **[RoBERTa](https://github.com/pytorch/fairseq/tree/master/examples/roberta)** (from Facebook), released together with the paper a [Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
LysandreJik's avatar
LysandreJik committed
16
8. **[DistilBERT](https://github.com/huggingface/pytorch-transformers/tree/master/examples/distillation)** (from HuggingFace), released together with the blogpost [Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT](https://medium.com/huggingface/distilbert-8cf3380435b5
thomwolf's avatar
thomwolf committed
17
) by Victor Sanh, Lysandre Debut and Thomas Wolf.
thomwolf's avatar
thomwolf committed
18

thomwolf's avatar
thomwolf committed
19
These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Peason R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of the [documentation](https://huggingface.co/pytorch-transformers/examples.html).
20

thomwolf's avatar
thomwolf committed
21
| Section | Description |
thomwolf's avatar
thomwolf committed
22
|-|-|
thomwolf's avatar
thomwolf committed
23
| [Installation](#installation) | How to install the package |
24
| [Online demo](#online-demo) | Experimenting with this repo’s text generation capabilities |
Grégory Châtel's avatar
Grégory Châtel committed
25
| [Quick tour: Usage](#quick-tour) | Tokenizers & models usage: Bert and GPT-2 |
Praateek Mahajan's avatar
Praateek Mahajan committed
26
| [Quick tour: Fine-tuning/usage scripts](#quick-tour-of-the-fine-tuningusage-scripts) | Using provided scripts: GLUE, SQuAD and Text generation |
thomwolf's avatar
thomwolf committed
27
| [Migrating from pytorch-pretrained-bert to pytorch-transformers](#Migrating-from-pytorch-pretrained-bert-to-pytorch-transformers) | Migrating your code from pytorch-pretrained-bert to pytorch-transformers |
thomwolf's avatar
thomwolf committed
28
| [Documentation](https://huggingface.co/pytorch-transformers/) | Full API documentation and more |
thomwolf's avatar
thomwolf committed
29

thomwolf's avatar
thomwolf committed
30
## Installation
VictorSanh's avatar
VictorSanh committed
31

32
This repo is tested on Python 2.7 and 3.5+ (examples are tested only on python 3.5+) and PyTorch 1.0.0+
VictorSanh's avatar
VictorSanh committed
33

thomwolf's avatar
thomwolf committed
34
### With pip
thomwolf's avatar
thomwolf committed
35

thomwolf's avatar
thomwolf committed
36
PyTorch-Transformers can be installed by pip as follows:
thomwolf's avatar
thomwolf committed
37

thomwolf's avatar
thomwolf committed
38
```bash
thomwolf's avatar
thomwolf committed
39
pip install pytorch-transformers
thomwolf's avatar
thomwolf committed
40
```
VictorSanh's avatar
VictorSanh committed
41

thomwolf's avatar
thomwolf committed
42
### From source
thomwolf's avatar
thomwolf committed
43
44

Clone the repository and run:
thomwolf's avatar
thomwolf committed
45

thomwolf's avatar
thomwolf committed
46
47
48
```bash
pip install [--editable] .
```
VictorSanh's avatar
VictorSanh committed
49

thomwolf's avatar
thomwolf committed
50
### Tests
thomwolf's avatar
thomwolf committed
51

thomwolf's avatar
thomwolf committed
52
A series of tests is included for the library and the example scripts. Library tests can be found in the [tests folder](https://github.com/huggingface/pytorch-transformers/tree/master/pytorch_transformers/tests) and examples tests in the [examples folder](https://github.com/huggingface/pytorch-transformers/tree/master/examples).
thomwolf's avatar
thomwolf committed
53

thomwolf's avatar
thomwolf committed
54
These tests can be run using `pytest` (install pytest if needed with `pip install pytest`).
thomwolf's avatar
thomwolf committed
55

thomwolf's avatar
thomwolf committed
56
You can run the tests from the root of the cloned repository with the commands:
thomwolf's avatar
thomwolf committed
57

thomwolf's avatar
thomwolf committed
58
59
60
61
```bash
python -m pytest -sv ./pytorch_transformers/tests/
python -m pytest -sv ./examples/
```
thomwolf's avatar
thomwolf committed
62

63
64
65
66
67
68
69
70
71
### Do you want to run a Transformer model on a mobile device?

You should check out our [`swift-coreml-transformers`](https://github.com/huggingface/swift-coreml-transformers) repo.

It contains an example of a conversion script from a Pytorch trained Transformer model (here, `GPT-2`) to a CoreML model that runs on iOS devices.

At some point in the future, you'll be able to seamlessly move from pre-training or fine-tuning models in PyTorch to productizing them in CoreML,
or prototype a model or an app in CoreML then research its hyperparameters or architecture from PyTorch. Super exciting!

72
73
74
75
76
77
78
## Online demo

**[Write With Transformer](https://transformer.huggingface.co)**, built by the Hugging Face team at transformer.huggingface.co, is the official demo of this repo’s text generation capabilities.
You can use it to experiment with completions generated by `GPT2Model`, `TransfoXLModel`, and `XLNetModel`.

> “🦄 Write with transformer is to writing what calculators are to calculus.”

Julien Chaumond's avatar
Julien Chaumond committed
79
![write_with_transformer](https://transformer.huggingface.co/front/assets/thumbnail-large.png)
80

thomwolf's avatar
thomwolf committed
81
## Quick tour
thomwolf's avatar
thomwolf committed
82

rish-16's avatar
rish-16 committed
83
Let's do a very quick overview of PyTorch-Transformers. Detailed examples for each model architecture (Bert, GPT, GPT-2, Transformer-XL, XLNet and XLM) can be found in the [full documentation](https://huggingface.co/pytorch-transformers/).
thomwolf's avatar
thomwolf committed
84
85
86

```python
import torch
thomwolf's avatar
thomwolf committed
87
88
89
from pytorch_transformers import *

# PyTorch-Transformers has a unified API
90
# for 7 transformer architectures and 30 pretrained weights.
thomwolf's avatar
thomwolf committed
91
92
93
94
95
96
#          Model          | Tokenizer          | Pretrained weights shortcut
MODELS = [(BertModel,       BertTokenizer,      'bert-base-uncased'),
          (OpenAIGPTModel,  OpenAIGPTTokenizer, 'openai-gpt'),
          (GPT2Model,       GPT2Tokenizer,      'gpt2'),
          (TransfoXLModel,  TransfoXLTokenizer, 'transfo-xl-wt103'),
          (XLNetModel,      XLNetTokenizer,     'xlnet-base-cased'),
LysandreJik's avatar
LysandreJik committed
97
98
          (XLMModel,        XLMTokenizer,       'xlm-mlm-enfr-1024'),
          (RobertaModel,    RobertaTokenizer,   'roberta-base')]
thomwolf's avatar
thomwolf committed
99
100
101
102
103
104
105
106

# Let's encode some text in a sequence of hidden-states using each model:
for model_class, tokenizer_class, pretrained_weights in MODELS:
    # Load pretrained model/tokenizer
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights)

    # Encode text
107
    input_ids = torch.tensor([tokenizer.encode("Here is some text to encode", add_special_tokens=True)])  # Add special tokens takes care of adding [CLS], [SEP], <s>... tokens in the right way for each model.
Thomas Wolf's avatar
Thomas Wolf committed
108
109
    with torch.no_grad():
        last_hidden_states = model(input_ids)[0]  # Models outputs are now tuples
thomwolf's avatar
thomwolf committed
110
111
112
113
114
115

# Each architecture is provided with several class for fine-tuning on down-stream tasks, e.g.
BERT_MODEL_CLASSES = [BertModel, BertForPreTraining, BertForMaskedLM, BertForNextSentencePrediction,
                      BertForSequenceClassification, BertForMultipleChoice, BertForTokenClassification,
                      BertForQuestionAnswering]

thomwolf's avatar
thomwolf committed
116
117
118
# All the classes for an architecture can be initiated from pretrained weights for this architecture
# Note that additional weights added for fine-tuning are only initialized
# and need to be trained on the down-stream task
thomwolf's avatar
thomwolf committed
119
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
thomwolf's avatar
thomwolf committed
120
121
122
123
124
for model_class in BERT_MODEL_CLASSES:
    # Load pretrained model/tokenizer
    model = model_class.from_pretrained('bert-base-uncased')

# Models can return full list of hidden-states & attentions weights at each layer
thomwolf's avatar
thomwolf committed
125
126
127
model = model_class.from_pretrained(pretrained_weights,
                                    output_hidden_states=True,
                                    output_attentions=True)
thomwolf's avatar
thomwolf committed
128
129
130
131
132
133
134
135
136
137
138
input_ids = torch.tensor([tokenizer.encode("Let's see all hidden-states and attentions on this text")])
all_hidden_states, all_attentions = model(input_ids)[-2:]

# Models are compatible with Torchscript
model = model_class.from_pretrained(pretrained_weights, torchscript=True)
traced_model = torch.jit.trace(model, (input_ids,))

# Simple serialization for models and tokenizers
model.save_pretrained('./directory/to/save/')  # save
model = model_class.from_pretrained('./directory/to/save/')  # re-load
tokenizer.save_pretrained('./directory/to/save/')  # save
carefree0910's avatar
carefree0910 committed
139
tokenizer = tokenizer_class.from_pretrained('./directory/to/save/')  # re-load
thomwolf's avatar
thomwolf committed
140
141

# SOTA examples for GLUE, SQUAD, text generation...
thomwolf's avatar
thomwolf committed
142
143
```

thomwolf's avatar
thomwolf committed
144
## Quick tour of the fine-tuning/usage scripts
thomwolf's avatar
thomwolf committed
145

thomwolf's avatar
thomwolf committed
146
The library comprises several example scripts with SOTA performances for NLU and NLG tasks:
thomwolf's avatar
thomwolf committed
147

thomwolf's avatar
thomwolf committed
148
149
150
151
- `run_glue.py`: an example fine-tuning Bert, XLNet and XLM on nine different GLUE tasks (*sequence-level classification*)
- `run_squad.py`: an example fine-tuning Bert, XLNet and XLM on the question answering dataset SQuAD 2.0 (*token-level classification*)
- `run_generation.py`: an example using GPT, GPT-2, Transformer-XL and XLNet for conditional language generation
- other model-specific examples (see the documentation).
thomwolf's avatar
thomwolf committed
152

thomwolf's avatar
thomwolf committed
153
Here are three quick usage examples for these scripts:
thomwolf's avatar
thomwolf committed
154

thomwolf's avatar
thomwolf committed
155
### `run_glue.py`: Fine-tuning on GLUE tasks for sequence classification
thomwolf's avatar
thomwolf committed
156

thomwolf's avatar
thomwolf committed
157
The [General Language Understanding Evaluation (GLUE) benchmark](https://gluebenchmark.com/) is a collection of nine sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.
thomwolf's avatar
thomwolf committed
158

thomwolf's avatar
thomwolf committed
159
160
161
162
Before running anyone of these GLUE tasks you should download the
[GLUE data](https://gluebenchmark.com/tasks) by running
[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
and unpack it to some directory `$GLUE_DIR`.
thomwolf's avatar
thomwolf committed
163

164
165
166
167
168
169
You should also install the additional packages required by the examples:

```shell
pip install -r ./examples/requirements.txt
```

thomwolf's avatar
thomwolf committed
170
171
172
```shell
export GLUE_DIR=/path/to/glue
export TASK_NAME=MRPC
thomwolf's avatar
thomwolf committed
173

174
175
176
177
178
179
180
181
182
183
184
185
186
187
python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/$TASK_NAME/
thomwolf's avatar
thomwolf committed
188
189
```

thomwolf's avatar
thomwolf committed
190
where task name can be one of CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI.
thomwolf's avatar
thomwolf committed
191

thomwolf's avatar
thomwolf committed
192
The dev set results will be present within the text file 'eval_results.txt' in the specified output_dir. In case of MNLI, since there are two separate dev sets, matched and mismatched, there will be a separate output folder called '/tmp/MNLI-MM/' in addition to '/tmp/MNLI/'.
thomwolf's avatar
thomwolf committed
193

thomwolf's avatar
thomwolf committed
194
#### Fine-tuning XLNet model on the STS-B regression task
thomwolf's avatar
thomwolf committed
195

thomwolf's avatar
thomwolf committed
196
This example code fine-tunes XLNet on the STS-B corpus using parallel training on a server with 4 V100 GPUs.
197
Parallel training is a simple way to use several GPUs (but is slower and less flexible than distributed training, see below).
thomwolf's avatar
thomwolf committed
198

thomwolf's avatar
thomwolf committed
199
200
```shell
export GLUE_DIR=/path/to/glue
thomwolf's avatar
thomwolf committed
201

thomwolf's avatar
thomwolf committed
202
203
204
205
python ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
206
    --do_eval   \
thomwolf's avatar
thomwolf committed
207
208
209
210
211
212
213
214
215
216
217
218
    --task_name=sts-b     \
    --data_dir=${GLUE_DIR}/STS-B  \
    --output_dir=./proc_data/sts-b-110   \
    --max_seq_length=128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --gradient_accumulation_steps=1 \
    --max_steps=1200  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --warmup_steps=120
thomwolf's avatar
thomwolf committed
219
220
```

Anthony MOI's avatar
Anthony MOI committed
221
On this machine we thus have a batch size of 32, please increase `gradient_accumulation_steps` to reach the same batch size if you have a smaller machine. These hyper-parameters should result in a Pearson correlation coefficient of `+0.917` on the development set.
thomwolf's avatar
thomwolf committed
222

thomwolf's avatar
thomwolf committed
223
#### Fine-tuning Bert model on the MRPC classification task
thomwolf's avatar
thomwolf committed
224

thomwolf's avatar
thomwolf committed
225
This example code fine-tunes the Bert Whole Word Masking model on the Microsoft Research Paraphrase Corpus (MRPC) corpus using distributed training on 8 V100 GPUs to reach a F1 > 92.
thomwolf's avatar
thomwolf committed
226

thomwolf's avatar
thomwolf committed
227
```bash
228
python -m torch.distributed.launch --nproc_per_node 8 ./examples/run_glue.py   \
thomwolf's avatar
thomwolf committed
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --task_name MRPC \
    --do_train   \
    --do_eval   \
    --do_lower_case   \
    --data_dir $GLUE_DIR/MRPC/   \
    --max_seq_length 128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5   \
    --num_train_epochs 3.0  \
    --output_dir /tmp/mrpc_output/ \
    --overwrite_output_dir   \
    --overwrite_cache \
thomwolf's avatar
thomwolf committed
244
245
```

thomwolf's avatar
thomwolf committed
246
Training with these hyper-parameters gave us the following results:
thomwolf's avatar
thomwolf committed
247

thomwolf's avatar
thomwolf committed
248
249
250
251
252
253
254
```bash
  acc = 0.8823529411764706
  acc_and_f1 = 0.901702786377709
  eval_loss = 0.3418912578906332
  f1 = 0.9210526315789473
  global_step = 174
  loss = 0.07231863956341798
thomwolf's avatar
thomwolf committed
255
256
```

thomwolf's avatar
thomwolf committed
257
### `run_squad.py`: Fine-tuning on SQuAD for question-answering
thomwolf's avatar
thomwolf committed
258

thomwolf's avatar
thomwolf committed
259
This example code fine-tunes BERT on the SQuAD dataset using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD:
thomwolf's avatar
thomwolf committed
260

thomwolf's avatar
thomwolf committed
261
```bash
262
python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
thomwolf's avatar
thomwolf committed
263
264
265
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
thomwolf's avatar
thomwolf committed
266
    --do_eval \
thomwolf's avatar
thomwolf committed
267
268
269
270
271
272
273
274
275
276
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/wwm_uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \
thomwolf's avatar
thomwolf committed
277
278
```

thomwolf's avatar
thomwolf committed
279
Training with these hyper-parameters gave us the following results:
thomwolf's avatar
thomwolf committed
280

thomwolf's avatar
thomwolf committed
281
282
283
```bash
python $SQUAD_DIR/evaluate-v1.1.py $SQUAD_DIR/dev-v1.1.json ../models/wwm_uncased_finetuned_squad/predictions.json
{"exact_match": 86.91579943235573, "f1": 93.1532499015869}
thomwolf's avatar
thomwolf committed
284
285
```

thomwolf's avatar
thomwolf committed
286
This is the model provided as `bert-large-uncased-whole-word-masking-finetuned-squad`.
287

thomwolf's avatar
thomwolf committed
288
### `run_generation.py`: Text generation with GPT, GPT-2, Transformer-XL and XLNet
289

thomwolf's avatar
thomwolf committed
290
A conditional generation script is also included to generate text from a prompt.
Julien Chaumond's avatar
Julien Chaumond committed
291
The generation script includes the [tricks](https://github.com/rusiaaman/XLNet-gen#methodology) proposed by Aman Rusia to get high quality generation with memory models like Transformer-XL and XLNet (include a predefined text to make short inputs longer).
292

thomwolf's avatar
thomwolf committed
293
Here is how to run the script with the small version of OpenAI GPT-2 model:
294

thomwolf's avatar
thomwolf committed
295
```shell
thomwolf's avatar
thomwolf committed
296
python ./examples/run_generation.py \
thomwolf's avatar
thomwolf committed
297
298
299
    --model_type=gpt2 \
    --length=20 \
    --model_name_or_path=gpt2 \
300
301
```

thomwolf's avatar
thomwolf committed
302
303
304
305
306
307
308
309
## Migrating from pytorch-pretrained-bert to pytorch-transformers

Here is a quick summary of what you should take care of when migrating from `pytorch-pretrained-bert` to `pytorch-transformers`

### Models always output `tuples`

The main breaking change when migrating from `pytorch-pretrained-bert` to `pytorch-transformers` is that the models forward method always outputs a `tuple` with various elements depending on the model and the configuration parameters.

Pierric Cistac's avatar
Pierric Cistac committed
310
The exact content of the tuples for each model are detailed in the models' docstrings and the [documentation](https://huggingface.co/pytorch-transformers/).
thomwolf's avatar
thomwolf committed
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329

In pretty much every case, you will be fine by taking the first element of the output as the output you previously used in `pytorch-pretrained-bert`.

Here is a `pytorch-pretrained-bert` to `pytorch-transformers` conversion example for a `BertForSequenceClassification` classification model:

```python
# Let's load our model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# If you used to have this line in pytorch-pretrained-bert:
loss = model(input_ids, labels=labels)

# Now just use this line in pytorch-transformers to extract the loss from the output tuple:
outputs = model(input_ids, labels=labels)
loss = outputs[0]

# In pytorch-transformers you can also have access to the logits:
loss, logits = outputs[:2]

330
# And even the attention weights if you configure the model to output them (and other outputs too, see the docstrings and documentation)
thomwolf's avatar
thomwolf committed
331
332
333
334
335
336
337
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', output_attentions=True)
outputs = model(input_ids, labels=labels)
loss, logits, attentions = outputs
```

### Serialization

338
339
340
341
Breaking change in the `from_pretrained()`method:

1. Models are now set in evaluation mode by default when instantiated with the `from_pretrained()` method. To train them don't forget to set them back in training mode (`model.train()`) to activate the dropout modules.

342
2. The additional `*input` and `**kwargs` arguments supplied to the `from_pretrained()` method used to be directly passed to the underlying model's class `__init__()` method. They are now used to update the model configuration attribute instead which can break derived model classes build based on the previous `BertForSequenceClassification` examples. We are working on a way to mitigate this breaking change in [#866](https://github.com/huggingface/pytorch-transformers/pull/866) by forwarding the the model `__init__()` method (i) the provided positional arguments and (ii) the keyword arguments which do not match any configuration class attributes.
343

thomwolf's avatar
typos  
thomwolf committed
344
Also, while not a breaking change, the serialization methods have been standardized and you probably should switch to the new method `save_pretrained(save_directory)` if you were using any other serialization method before.
thomwolf's avatar
thomwolf committed
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370

Here is an example:

```python
### Let's load a model and tokenizer
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

### Do some stuff to our model and tokenizer
# Ex: add new tokens to the vocabulary and embeddings of our model
tokenizer.add_tokens(['[SPECIAL_TOKEN_1]', '[SPECIAL_TOKEN_2]'])
model.resize_token_embeddings(len(tokenizer))
# Train our model
train(model)

### Now let's save our model and tokenizer to a directory
model.save_pretrained('./my_saved_model_directory/')
tokenizer.save_pretrained('./my_saved_model_directory/')

### Reload the model and the tokenizer
model = BertForSequenceClassification.from_pretrained('./my_saved_model_directory/')
tokenizer = BertTokenizer.from_pretrained('./my_saved_model_directory/')
```

### Optimizers: BertAdam & OpenAIAdam are now AdamW, schedules are standard PyTorch schedules

371
372
373
374
375
376
377
The two optimizers previously included, `BertAdam` and `OpenAIAdam`, have been replaced by a single `AdamW` optimizer which has a few differences:

- it only implements weights decay correction,
- schedules are now externals (see below),
- gradient clipping is now also external (see below).

The new optimizer `AdamW` matches PyTorch `Adam` optimizer API and let you use standard PyTorch or apex methods for the schedule and clipping.
thomwolf's avatar
thomwolf committed
378
379
380
381
382
383
384
385

The schedules are now standard [PyTorch learning rate schedulers](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate) and not part of the optimizer anymore.

Here is a conversion examples from `BertAdam` with a linear warmup and decay schedule to `AdamW` and the same schedule:

```python
# Parameters:
lr = 1e-3
386
max_grad_norm = 1.0
thomwolf's avatar
thomwolf committed
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
num_total_steps = 1000
num_warmup_steps = 100
warmup_proportion = float(num_warmup_steps) / float(num_total_steps)  # 0.1

### Previously BertAdam optimizer was instantiated like this:
optimizer = BertAdam(model.parameters(), lr=lr, schedule='warmup_linear', warmup=warmup_proportion, t_total=num_total_steps)
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    optimizer.step()

### In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
406
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # Gradient clipping is not in AdamW anymore (so you can use amp without issue)
thomwolf's avatar
thomwolf committed
407
    optimizer.step()
thomwolf's avatar
thomwolf committed
408
    scheduler.step()
thomwolf's avatar
thomwolf committed
409
    optimizer.zero_grad()
thomwolf's avatar
thomwolf committed
410
411
```

thomwolf's avatar
thomwolf committed
412
## Citation
thomwolf's avatar
thomwolf committed
413

thomwolf's avatar
thomwolf committed
414
At the moment, there is no paper associated to PyTorch-Transformers but we are working on preparing one. In the meantime, please include a mention of the library and a link to the present repository if you use this work in a published or open-source project.