task_summary.rst 46.6 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
.. 
    Copyright 2020 The HuggingFace Team. All rights reserved.

    Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
    an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
    specific language governing permissions and limitations under the License.

Sylvain Gugger's avatar
Sylvain Gugger committed
13
Summary of the tasks
Sylvain Gugger's avatar
Sylvain Gugger committed
14
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15
16

This page shows the most frequent use-cases when using the library. The models available allow for many different
Sylvain Gugger's avatar
Sylvain Gugger committed
17
18
configurations and a great versatility in use-cases. The most simple ones are presented here, showcasing usage for
tasks such as question answering, sequence classification, named entity recognition and others.
19
20
21

These examples leverage auto-models, which are classes that will instantiate a model according to a given checkpoint,
automatically selecting the correct model architecture. Please check the :class:`~transformers.AutoModel` documentation
Sylvain Gugger's avatar
Sylvain Gugger committed
22
for more information. Feel free to modify the code to be more specific and adapt it to your specific use-case.
23
24
25
26
27
28

In order for a model to perform well on a task, it must be loaded from a checkpoint corresponding to that task. These
checkpoints are usually pre-trained on a large corpus of data and fine-tuned on a specific task. This means the
following:

- Not all models were fine-tuned on all tasks. If you want to fine-tune a model on a specific task, you can leverage
Sylvain Gugger's avatar
Sylvain Gugger committed
29
30
31
32
33
34
  one of the `run_$TASK.py` scripts in the `examples
  <https://github.com/huggingface/transformers/tree/master/examples>`__ directory.
- Fine-tuned models were fine-tuned on a specific dataset. This dataset may or may not overlap with your use-case and
  domain. As mentioned previously, you may leverage the `examples
  <https://github.com/huggingface/transformers/tree/master/examples>`__ scripts to fine-tune your model, or you may
  create your own training script.
35
36
37
38

In order to do an inference on a task, several mechanisms are made available by the library:

- Pipelines: very easy-to-use abstractions, which require as little as two lines of code.
Sylvain Gugger's avatar
Sylvain Gugger committed
39
40
- Direct model use: Less abstractions, but more flexibility and power via a direct access to a tokenizer
  (PyTorch/TensorFlow) and full inference capacity.
41
42
43
44
45
46
47
48
49
50
51
52

Both approaches are showcased here.

.. note::

    All tasks presented here leverage pre-trained checkpoints that were fine-tuned on specific tasks. Loading a
    checkpoint that was not fine-tuned on a specific task would load only the base transformer layers and not the
    additional head that is used for the task, initializing the weights of that head randomly.

    This would produce random output.

Sequence Classification
Sylvain Gugger's avatar
Sylvain Gugger committed
53
-----------------------------------------------------------------------------------------------------------------------
54

Sylvain Gugger's avatar
Sylvain Gugger committed
55
56
Sequence classification is the task of classifying sequences according to a given number of classes. An example of
sequence classification is the GLUE dataset, which is entirely based on that task. If you would like to fine-tune a
57
model on a GLUE sequence classification task, you may leverage the :prefix_link:`run_glue.py
Sylvain Gugger's avatar
Sylvain Gugger committed
58
59
60
61
<examples/pytorch/text-classification/run_glue.py>`, :prefix_link:`run_tf_glue.py
<examples/tensorflow/text-classification/run_tf_glue.py>`, :prefix_link:`run_tf_text_classification.py
<examples/tensorflow/text-classification/run_tf_text_classification.py>` or :prefix_link:`run_xnli.py
<examples/pytorch/text-classification/run_xnli.py>` scripts.
62

Sylvain Gugger's avatar
Sylvain Gugger committed
63
64
Here is an example of using pipelines to do sentiment analysis: identifying if a sequence is positive or negative. It
leverages a fine-tuned model on sst2, which is a GLUE task.
65

66
This returns a label ("POSITIVE" or "NEGATIVE") alongside a score, as follows:
67

68
.. code-block::
69

70
    >>> from transformers import pipeline
71

72
    >>> nlp = pipeline("sentiment-analysis")
73

74
75
76
    >>> result = nlp("I hate you")[0]
    >>> print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
    label: NEGATIVE, with score: 0.9991
77

78
79
80
    >>> result = nlp("I love you")[0]
    >>> print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
    label: POSITIVE, with score: 0.9999
81
82


Sylvain Gugger's avatar
Sylvain Gugger committed
83
84
Here is an example of doing a sequence classification using a model to determine if two sequences are paraphrases of
each other. The process is the following:
85

Sylvain Gugger's avatar
Sylvain Gugger committed
86
87
88
89
90
91
92
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it
   with the weights stored in the checkpoint.
2. Build a sequence from the two sentences, with the correct model-specific separators token type ids and attention
   masks (:func:`~transformers.PreTrainedTokenizer.encode` and :func:`~transformers.PreTrainedTokenizer.__call__` take
   care of this).
3. Pass this sequence through the model so that it is classified in one of the two available classes: 0 (not a
   paraphrase) and 1 (is a paraphrase).
93
94
4. Compute the softmax of the result to get probabilities over the classes.
5. Print the results.
95

96
.. code-block::
97

98
99
100
    >>> ## PYTORCH CODE
    >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification
    >>> import torch
101

102
    >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased-finetuned-mrpc")
103
    >>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mrpc")
104

105
    >>> classes = ["not paraphrase", "is paraphrase"]
106

107
108
109
    >>> sequence_0 = "The company HuggingFace is based in New York City"
    >>> sequence_1 = "Apples are especially bad for your health"
    >>> sequence_2 = "HuggingFace's headquarters are situated in Manhattan"
110

111
112
    >>> paraphrase = tokenizer(sequence_0, sequence_2, return_tensors="pt")
    >>> not_paraphrase = tokenizer(sequence_0, sequence_1, return_tensors="pt")
113

114
115
    >>> paraphrase_classification_logits = model(**paraphrase).logits
    >>> not_paraphrase_classification_logits = model(**not_paraphrase).logits
116

117
118
    >>> paraphrase_results = torch.softmax(paraphrase_classification_logits, dim=1).tolist()[0]
    >>> not_paraphrase_results = torch.softmax(not_paraphrase_classification_logits, dim=1).tolist()[0]
119

120
121
122
123
124
    >>> # Should be paraphrase
    >>> for i in range(len(classes)):
    ...     print(f"{classes[i]}: {int(round(paraphrase_results[i] * 100))}%")
    not paraphrase: 10%
    is paraphrase: 90%
125

126
127
128
129
130
131
132
133
    >>> # Should not be paraphrase
    >>> for i in range(len(classes)):
    ...     print(f"{classes[i]}: {int(round(not_paraphrase_results[i] * 100))}%")
    not paraphrase: 94%
    is paraphrase: 6%
    >>> ## TENSORFLOW CODE
    >>> from transformers import AutoTokenizer, TFAutoModelForSequenceClassification
    >>> import tensorflow as tf
134

135
    >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased-finetuned-mrpc")
136
    >>> model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mrpc")
137

138
    >>> classes = ["not paraphrase", "is paraphrase"]
139

140
141
142
    >>> sequence_0 = "The company HuggingFace is based in New York City"
    >>> sequence_1 = "Apples are especially bad for your health"
    >>> sequence_2 = "HuggingFace's headquarters are situated in Manhattan"
143

144
145
    >>> paraphrase = tokenizer(sequence_0, sequence_2, return_tensors="tf")
    >>> not_paraphrase = tokenizer(sequence_0, sequence_1, return_tensors="tf")
146

147
148
    >>> paraphrase_classification_logits = model(paraphrase)[0]
    >>> not_paraphrase_classification_logits = model(not_paraphrase)[0]
149

150
151
    >>> paraphrase_results = tf.nn.softmax(paraphrase_classification_logits, axis=1).numpy()[0]
    >>> not_paraphrase_results = tf.nn.softmax(not_paraphrase_classification_logits, axis=1).numpy()[0]
152

153
154
155
    >>> # Should be paraphrase
    >>> for i in range(len(classes)):
    ...     print(f"{classes[i]}: {int(round(paraphrase_results[i] * 100))}%")
156
157
158
    not paraphrase: 10%
    is paraphrase: 90%

159
160
161
    >>> # Should not be paraphrase
    >>> for i in range(len(classes)):
    ...     print(f"{classes[i]}: {int(round(not_paraphrase_results[i] * 100))}%")
162
163
164
165
    not paraphrase: 94%
    is paraphrase: 6%

Extractive Question Answering
Sylvain Gugger's avatar
Sylvain Gugger committed
166
-----------------------------------------------------------------------------------------------------------------------
167
168

Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a
Sylvain Gugger's avatar
Sylvain Gugger committed
169
question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune a
170
model on a SQuAD task, you may leverage the `run_qa.py
Sylvain Gugger's avatar
Sylvain Gugger committed
171
172
173
174
<https://github.com/huggingface/transformers/tree/master/examples/pytorch/question-answering/run_qa.py>`__ and
`run_tf_squad.py
<https://github.com/huggingface/transformers/tree/master/examples/tensorflow/question-answering/run_tf_squad.py>`__
scripts.
175

176

Sylvain Gugger's avatar
Sylvain Gugger committed
177
178
Here is an example of using pipelines to do question answering: extracting an answer from a text given a question. It
leverages a fine-tuned model on SQuAD.
179

180
.. code-block::
181

182
    >>> from transformers import pipeline
183

184
    >>> nlp = pipeline("question-answering")
185

186
187
188
    >>> context = r"""
    ... Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a
    ... question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
Sylvain Gugger's avatar
Sylvain Gugger committed
189
    ... a model on a SQuAD task, you may leverage the examples/pytorch/question-answering/run_squad.py script.
190
    ... """
191

Sylvain Gugger's avatar
Sylvain Gugger committed
192
193
This returns an answer extracted from the text, a confidence score, alongside "start" and "end" values, which are the
positions of the extracted answer in the text.
194

195
196
197
198
199
.. code-block::

    >>> result = nlp(question="What is extractive question answering?", context=context)
    >>> print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")
    Answer: 'the task of extracting an answer from a text given a question.', score: 0.6226, start: 34, end: 96
200

201
202
203
    >>> result = nlp(question="What is a good example of a question answering dataset?", context=context)
    >>> print(f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}")
    Answer: 'SQuAD dataset,', score: 0.5053, start: 147, end: 161
204
205
206
207


Here is an example of question answering using a model and a tokenizer. The process is the following:

Sylvain Gugger's avatar
Sylvain Gugger committed
208
209
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it
   with the weights stored in the checkpoint.
210
2. Define a text and a few questions.
Sylvain Gugger's avatar
Sylvain Gugger committed
211
212
213
214
3. Iterate over the questions and build a sequence from the text and the current question, with the correct
   model-specific separators token type ids and attention masks.
4. Pass this sequence through the model. This outputs a range of scores across the entire sequence tokens (question and
   text), for both the start and end positions.
215
216
217
5. Compute the softmax of the result to get probabilities over the tokens.
6. Fetch the tokens from the identified start and stop values, convert those tokens to a string.
7. Print the results.
218

219
220
221
222
223
224
225
.. code-block::

    >>> ## PYTORCH CODE
    >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering
    >>> import torch

    >>> tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
226
    >>> model = AutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241

    >>> text = r"""
    ... 🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose
    ... architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU) and Natural
    ... Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
    ... TensorFlow 2.0 and PyTorch.
    ... """

    >>> questions = [
    ...     "How many pretrained models are available in 🤗 Transformers?",
    ...     "What does 🤗 Transformers provide?",
    ...     "🤗 Transformers provides interoperability between which frameworks?",
    ... ]

    >>> for question in questions:
242
    ...     inputs = tokenizer(question, text, add_special_tokens=True, return_tensors="pt")
243
244
    ...     input_ids = inputs["input_ids"].tolist()[0]
    ...
245
246
247
    ...     outputs = model(**inputs)
    ...     answer_start_scores = outputs.start_logits
    ...     answer_end_scores = outputs.end_logits
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
    ...
    ...     answer_start = torch.argmax(
    ...         answer_start_scores
    ...     )  # Get the most likely beginning of answer with the argmax of the score
    ...     answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score
    ...
    ...     answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))
    ...
    ...     print(f"Question: {question}")
    ...     print(f"Answer: {answer}")
    Question: How many pretrained models are available in 🤗 Transformers?
    Answer: over 32 +
    Question: What does 🤗 Transformers provide?
    Answer: general - purpose architectures
    Question: 🤗 Transformers provides interoperability between which frameworks?
    Answer: tensorflow 2 . 0 and pytorch
    >>> ## TENSORFLOW CODE
    >>> from transformers import AutoTokenizer, TFAutoModelForQuestionAnswering
    >>> import tensorflow as tf

    >>> tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
269
    >>> model = TFAutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284

    >>> text = r"""
    ... 🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose
    ... architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU) and Natural
    ... Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between
    ... TensorFlow 2.0 and PyTorch.
    ... """

    >>> questions = [
    ...     "How many pretrained models are available in 🤗 Transformers?",
    ...     "What does 🤗 Transformers provide?",
    ...     "🤗 Transformers provides interoperability between which frameworks?",
    ... ]

    >>> for question in questions:
285
    ...     inputs = tokenizer(question, text, add_special_tokens=True, return_tensors="tf")
286
287
    ...     input_ids = inputs["input_ids"].numpy()[0]
    ...
288
289
290
    ...     outputs = model(inputs)
    ...     answer_start_scores = outputs.start_logits
    ...     answer_end_scores = outputs.end_logits
291
292
293
294
295
296
297
298
299
300
301
    ...
    ...     answer_start = tf.argmax(
    ...         answer_start_scores, axis=1
    ...     ).numpy()[0]  # Get the most likely beginning of answer with the argmax of the score
    ...     answer_end = (
    ...         tf.argmax(answer_end_scores, axis=1) + 1
    ...     ).numpy()[0]  # Get the most likely end of answer with the argmax of the score
    ...     answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))
    ...
    ...     print(f"Question: {question}")
    ...     print(f"Answer: {answer}")
Sylvain Gugger's avatar
Sylvain Gugger committed
302
    Question: How many pretrained models are available in 🤗 Transformers?
303
    Answer: over 32 +
Sylvain Gugger's avatar
Sylvain Gugger committed
304
    Question: What does 🤗 Transformers provide?
305
    Answer: general - purpose architectures
Sylvain Gugger's avatar
Sylvain Gugger committed
306
    Question: 🤗 Transformers provides interoperability between which frameworks?
307
308
309
310
311
    Answer: tensorflow 2 . 0 and pytorch



Language Modeling
Sylvain Gugger's avatar
Sylvain Gugger committed
312
-----------------------------------------------------------------------------------------------------------------------
313

Sylvain Gugger's avatar
Sylvain Gugger committed
314
315
316
Language modeling is the task of fitting a model to a corpus, which can be domain specific. All popular
transformer-based models are trained using a variant of language modeling, e.g. BERT with masked language modeling,
GPT-2 with causal language modeling.
317

318
Language modeling can be useful outside of pretraining as well, for example to shift the model distribution to be
Sylvain Gugger's avatar
Sylvain Gugger committed
319
320
domain-specific: using a language model trained over a very large corpus, and then fine-tuning it to a news dataset or
on scientific papers e.g. `LysandreJik/arxiv-nlp <https://huggingface.co/lysandre/arxiv-nlp>`__.
321
322

Masked Language Modeling
Sylvain Gugger's avatar
Sylvain Gugger committed
323
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
324
325
326

Masked language modeling is the task of masking tokens in a sequence with a masking token, and prompting the model to
fill that mask with an appropriate token. This allows the model to attend to both the right context (tokens on the
Sylvain Gugger's avatar
Sylvain Gugger committed
327
right of the mask) and the left context (tokens on the left of the mask). Such a training creates a strong basis for
328
downstream tasks requiring bi-directional context, such as SQuAD (question answering, see `Lewis, Lui, Goyal et al.
329
<https://arxiv.org/abs/1910.13461>`__, part 4.2). If you would like to fine-tune a model on a masked language modeling
Sylvain Gugger's avatar
Sylvain Gugger committed
330
task, you may leverage the :prefix_link:`run_mlm.py <examples/pytorch/language-modeling/run_mlm.py>` script.
331
332
333

Here is an example of using pipelines to replace a mask from a sequence:

334
.. code-block::
335

336
    >>> from transformers import pipeline
337

338
    >>> nlp = pipeline("fill-mask")
339

Sylvain Gugger's avatar
Sylvain Gugger committed
340
This outputs the sequences with the mask filled, the confidence score, and the token id in the tokenizer vocabulary:
341

342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
.. code-block::

    >>> from pprint import pprint
    >>> pprint(nlp(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks."))
    [{'score': 0.1792745739221573,
      'sequence': '<s>HuggingFace is creating a tool that the community uses to '
                  'solve NLP tasks.</s>',
      'token': 3944,
      'token_str': 'Ġtool'},
     {'score': 0.11349421739578247,
      'sequence': '<s>HuggingFace is creating a framework that the community uses '
                  'to solve NLP tasks.</s>',
      'token': 7208,
      'token_str': 'Ġframework'},
     {'score': 0.05243554711341858,
      'sequence': '<s>HuggingFace is creating a library that the community uses to '
                  'solve NLP tasks.</s>',
      'token': 5560,
      'token_str': 'Ġlibrary'},
     {'score': 0.03493533283472061,
      'sequence': '<s>HuggingFace is creating a database that the community uses '
                  'to solve NLP tasks.</s>',
      'token': 8503,
      'token_str': 'Ġdatabase'},
     {'score': 0.02860250137746334,
      'sequence': '<s>HuggingFace is creating a prototype that the community uses '
                  'to solve NLP tasks.</s>',
      'token': 17715,
      'token_str': 'Ġprototype'}]
371

372
Here is an example of doing masked language modeling using a model and a tokenizer. The process is the following:
373

Sylvain Gugger's avatar
Sylvain Gugger committed
374
375
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a DistilBERT model and
   loads it with the weights stored in the checkpoint.
376
377
2. Define a sequence with a masked token, placing the :obj:`tokenizer.mask_token` instead of a word.
3. Encode that sequence into a list of IDs and find the position of the masked token in that list.
Sylvain Gugger's avatar
Sylvain Gugger committed
378
379
4. Retrieve the predictions at the index of the mask token: this tensor has the same size as the vocabulary, and the
   values are the scores attributed to each token. The model gives higher score to tokens it deems probable in that
380
381
382
   context.
5. Retrieve the top 5 tokens using the PyTorch :obj:`topk` or TensorFlow :obj:`top_k` methods.
6. Replace the mask token by the tokens and print the results
383

384
.. code-block::
385

386
387
388
    >>> ## PYTORCH CODE
    >>> from transformers import AutoModelWithLMHead, AutoTokenizer
    >>> import torch
389

390
    >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
391
    >>> model = AutoModelWithLMHead.from_pretrained("distilbert-base-cased")
392

393
    >>> sequence = f"Distilled models are smaller than the models they mimic. Using them instead of the large versions would help {tokenizer.mask_token} our carbon footprint."
394

395
396
    >>> input = tokenizer.encode(sequence, return_tensors="pt")
    >>> mask_token_index = torch.where(input == tokenizer.mask_token_id)[1]
397

398
    >>> token_logits = model(input).logits
399
    >>> mask_token_logits = token_logits[0, mask_token_index, :]
400

401
402
403
404
    >>> top_5_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()
    >>> ## TENSORFLOW CODE
    >>> from transformers import TFAutoModelWithLMHead, AutoTokenizer
    >>> import tensorflow as tf
405

406
    >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
407
    >>> model = TFAutoModelWithLMHead.from_pretrained("distilbert-base-cased")
408

409
    >>> sequence = f"Distilled models are smaller than the models they mimic. Using them instead of the large versions would help {tokenizer.mask_token} our carbon footprint."
410

411
412
    >>> input = tokenizer.encode(sequence, return_tensors="tf")
    >>> mask_token_index = tf.where(input == tokenizer.mask_token_id)[0, 1]
413

414
415
    >>> token_logits = model(input)[0]
    >>> mask_token_logits = token_logits[0, mask_token_index, :]
416

417
    >>> top_5_tokens = tf.math.top_k(mask_token_logits, 5).indices.numpy()
418
419
420
421


This prints five sequences, with the top 5 tokens predicted by the model:

422
.. code-block::
423

424
425
    >>> for token in top_5_tokens:
    ...     print(sequence.replace(tokenizer.mask_token, tokenizer.decode([token])))
426
427
428
429
430
431
432
433
    Distilled models are smaller than the models they mimic. Using them instead of the large versions would help reduce our carbon footprint.
    Distilled models are smaller than the models they mimic. Using them instead of the large versions would help increase our carbon footprint.
    Distilled models are smaller than the models they mimic. Using them instead of the large versions would help decrease our carbon footprint.
    Distilled models are smaller than the models they mimic. Using them instead of the large versions would help offset our carbon footprint.
    Distilled models are smaller than the models they mimic. Using them instead of the large versions would help improve our carbon footprint.


Causal Language Modeling
Sylvain Gugger's avatar
Sylvain Gugger committed
434
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
435
436
437

Causal language modeling is the task of predicting the token following a sequence of tokens. In this situation, the
model only attends to the left context (tokens on the left of the mask). Such a training is particularly interesting
438
for generation tasks. If you would like to fine-tune a model on a causal language modeling task, you may leverage the
Sylvain Gugger's avatar
Sylvain Gugger committed
439
:prefix_link:`run_clm.py <examples/pytorch/language-modeling/run_clm.py>` script.
440

Sylvain Gugger's avatar
Sylvain Gugger committed
441
442
Usually, the next token is predicted by sampling from the logits of the last hidden state the model produces from the
input sequence.
443

Sylvain Gugger's avatar
Sylvain Gugger committed
444
445
446
Here is an example of using the tokenizer and model and leveraging the
:func:`~transformers.PreTrainedModel.top_k_top_p_filtering` method to sample the next token following an input sequence
of tokens.
447

448
.. code-block::
449

450
451
452
453
    >>> ## PYTORCH CODE
    >>> from transformers import AutoModelWithLMHead, AutoTokenizer, top_k_top_p_filtering
    >>> import torch
    >>> from torch.nn import functional as F
454

455
    >>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
456
    >>> model = AutoModelWithLMHead.from_pretrained("gpt2")
457

458
    >>> sequence = f"Hugging Face is based in DUMBO, New York City, and"
459

460
    >>> input_ids = tokenizer.encode(sequence, return_tensors="pt")
461

462
    >>> # get logits of last hidden state
463
    >>> next_token_logits = model(input_ids).logits[:, -1, :]
464

465
466
    >>> # filter
    >>> filtered_next_token_logits = top_k_top_p_filtering(next_token_logits, top_k=50, top_p=1.0)
467

468
469
470
    >>> # sample
    >>> probs = F.softmax(filtered_next_token_logits, dim=-1)
    >>> next_token = torch.multinomial(probs, num_samples=1)
471

472
    >>> generated = torch.cat([input_ids, next_token], dim=-1)
473

474
475
476
477
    >>> resulting_string = tokenizer.decode(generated.tolist()[0])
    >>> ## TENSORFLOW CODE
    >>> from transformers import TFAutoModelWithLMHead, AutoTokenizer, tf_top_k_top_p_filtering
    >>> import tensorflow as tf
478

479
    >>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
480
    >>> model = TFAutoModelWithLMHead.from_pretrained("gpt2")
481

482
    >>> sequence = f"Hugging Face is based in DUMBO, New York City, and "
483

484
    >>> input_ids = tokenizer.encode(sequence, return_tensors="tf")
485

486
487
    >>> # get logits of last hidden state
    >>> next_token_logits = model(input_ids)[0][:, -1, :]
488

489
490
    >>> # filter
    >>> filtered_next_token_logits = tf_top_k_top_p_filtering(next_token_logits, top_k=50, top_p=1.0)
491

492
493
    >>> # sample
    >>> next_token = tf.random.categorical(filtered_next_token_logits, dtype=tf.int32, num_samples=1)
494

495
    >>> generated = tf.concat([input_ids, next_token], axis=1)
496

497
    >>> resulting_string = tokenizer.decode(generated.numpy().tolist()[0])
498
499


500
This outputs a (hopefully) coherent next token following the original sequence, which in our case is the word *has*:
501

502
.. code-block::
503

Sylvain Gugger's avatar
Sylvain Gugger committed
504
    >>> print(resulting_string)
505
506
    Hugging Face is based in DUMBO, New York City, and has

Sylvain Gugger's avatar
Sylvain Gugger committed
507
508
In the next section, we show how this functionality is leveraged in :func:`~transformers.PreTrainedModel.generate` to
generate multiple tokens up to a user-defined length.
509
510

Text Generation
Sylvain Gugger's avatar
Sylvain Gugger committed
511
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
512

Sylvain Gugger's avatar
Sylvain Gugger committed
513
514
515
516
In text generation (*a.k.a* *open-ended text generation*) the goal is to create a coherent portion of text that is a
continuation from the given context. The following example shows how *GPT-2* can be used in pipelines to generate text.
As a default all models apply *Top-K* sampling when used in pipelines, as configured in their respective configurations
(see `gpt-2 config <https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json>`__ for example).
517

518
519
520
.. code-block::

    >>> from transformers import pipeline
521

522
523
524
    >>> text_generator = pipeline("text-generation")
    >>> print(text_generator("As far as I am concerned, I will", max_length=50, do_sample=False))
    [{'generated_text': 'As far as I am concerned, I will be the first to admit that I am not a fan of the idea of a "free market." I think that the idea of a free market is a bit of a stretch. I think that the idea'}]
525
526
527



Sylvain Gugger's avatar
Sylvain Gugger committed
528
Here, the model generates a random text with a total maximal length of *50* tokens from context *"As far as I am
529
530
concerned, I will"*. The default arguments of ``PreTrainedModel.generate()`` can be directly overridden in the
pipeline, as is shown above for the argument ``max_length``.
531

532
Here is an example of text generation using ``XLNet`` and its tokenizer.
533

534
.. code-block::
535

536
537
    >>> ## PYTORCH CODE
    >>> from transformers import AutoModelWithLMHead, AutoTokenizer
538

539
    >>> model = AutoModelWithLMHead.from_pretrained("xlnet-base-cased")
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
    >>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")

    >>> # Padding text helps XLNet with short prompts - proposed by Aman Rusia in https://github.com/rusiaaman/XLNet-gen#methodology
    >>> PADDING_TEXT = """In 1991, the remains of Russian Tsar Nicholas II and his family
    ... (except for Alexei and Maria) are discovered.
    ... The voice of Nicholas's young son, Tsarevich Alexei Nikolaevich, narrates the
    ... remainder of the story. 1883 Western Siberia,
    ... a young Grigori Rasputin is asked by his father and a group of men to perform magic.
    ... Rasputin has a vision and denounces one of the men as a horse thief. Although his
    ... father initially slaps him for making such an accusation, Rasputin watches as the
    ... man is chased outside and beaten. Twenty years later, Rasputin sees a vision of
    ... the Virgin Mary, prompting him to become a priest. Rasputin quickly becomes famous,
    ... with people, even a bishop, begging for his blessing. <eod> </s> <eos>"""

    >>> prompt = "Today the weather is really nice and I am planning on "
    >>> inputs = tokenizer.encode(PADDING_TEXT + prompt, add_special_tokens=False, return_tensors="pt")

    >>> prompt_length = len(tokenizer.decode(inputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
    >>> outputs = model.generate(inputs, max_length=250, do_sample=True, top_p=0.95, top_k=60)
    >>> generated = prompt + tokenizer.decode(outputs[0])[prompt_length:]

    >>> ## TENSORFLOW CODE
    >>> from transformers import TFAutoModelWithLMHead, AutoTokenizer

564
    >>> model = TFAutoModelWithLMHead.from_pretrained("xlnet-base-cased")
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
    >>> tokenizer = AutoTokenizer.from_pretrained("xlnet-base-cased")

    >>> # Padding text helps XLNet with short prompts - proposed by Aman Rusia in https://github.com/rusiaaman/XLNet-gen#methodology
    >>> PADDING_TEXT = """In 1991, the remains of Russian Tsar Nicholas II and his family
    ... (except for Alexei and Maria) are discovered.
    ... The voice of Nicholas's young son, Tsarevich Alexei Nikolaevich, narrates the
    ... remainder of the story. 1883 Western Siberia,
    ... a young Grigori Rasputin is asked by his father and a group of men to perform magic.
    ... Rasputin has a vision and denounces one of the men as a horse thief. Although his
    ... father initially slaps him for making such an accusation, Rasputin watches as the
    ... man is chased outside and beaten. Twenty years later, Rasputin sees a vision of
    ... the Virgin Mary, prompting him to become a priest. Rasputin quickly becomes famous,
    ... with people, even a bishop, begging for his blessing. <eod> </s> <eos>"""

    >>> prompt = "Today the weather is really nice and I am planning on "
    >>> inputs = tokenizer.encode(PADDING_TEXT + prompt, add_special_tokens=False, return_tensors="tf")

    >>> prompt_length = len(tokenizer.decode(inputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
    >>> outputs = model.generate(inputs, max_length=250, do_sample=True, top_p=0.95, top_k=60)
    >>> generated = prompt + tokenizer.decode(outputs[0])[prompt_length:]

.. code-block::
587

588
589
    >>> print(generated)
    Today the weather is really nice and I am planning on anning on taking a nice...... of a great time!<eop>...............
590

Sylvain Gugger's avatar
Sylvain Gugger committed
591
592
593
594
Text generation is currently possible with *GPT-2*, *OpenAi-GPT*, *CTRL*, *XLNet*, *Transfo-XL* and *Reformer* in
PyTorch and for most models in Tensorflow as well. As can be seen in the example above *XLNet* and *Transfo-XL* often
need to be padded to work well. GPT-2 is usually a good choice for *open-ended text generation* because it was trained
on millions of webpages with a causal language modeling objective.
595

Sylvain Gugger's avatar
Sylvain Gugger committed
596
597
For more information on how to apply different decoding strategies for text generation, please also refer to our text
generation blog post `here <https://huggingface.co/blog/how-to-generate>`__.
598
599
600


Named Entity Recognition
Sylvain Gugger's avatar
Sylvain Gugger committed
601
-----------------------------------------------------------------------------------------------------------------------
602

Sylvain Gugger's avatar
Sylvain Gugger committed
603
604
605
Named Entity Recognition (NER) is the task of classifying tokens according to a class, for example, identifying a token
as a person, an organisation or a location. An example of a named entity recognition dataset is the CoNLL-2003 dataset,
which is entirely based on that task. If you would like to fine-tune a model on an NER task, you may leverage the
Sylvain Gugger's avatar
Sylvain Gugger committed
606
:prefix_link:`run_ner.py <examples/pytorch/token-classification/run_ner.py>` script.
607

Sylvain Gugger's avatar
Sylvain Gugger committed
608
609
Here is an example of using pipelines to do named entity recognition, specifically, trying to identify tokens as
belonging to one of 9 classes:
610
611
612
613
614
615
616
617
618
619
620

- O, Outside of a named entity
- B-MIS, Beginning of a miscellaneous entity right after another miscellaneous entity
- I-MIS, Miscellaneous entity
- B-PER, Beginning of a person's name right after another person's name
- I-PER, Person's name
- B-ORG, Beginning of an organisation right after another organisation
- I-ORG, Organisation
- B-LOC, Beginning of a location right after another location
- I-LOC, Location

Sylvain Gugger's avatar
Sylvain Gugger committed
621
622
It leverages a fine-tuned model on CoNLL-2003, fine-tuned by `@stefan-it <https://github.com/stefan-it>`__ from `dbmdz
<https://github.com/dbmdz>`__.
623

624
.. code-block::
625

626
    >>> from transformers import pipeline
627

628
    >>> nlp = pipeline("ner")
629

630
631
    >>> sequence = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very"
    ...            "close to the Manhattan Bridge which is visible from the window."
632
633


Sylvain Gugger's avatar
Sylvain Gugger committed
634
635
This outputs a list of all words that have been identified as one of the entities from the 9 classes defined above.
Here are the expected results:
636

637
638
.. code-block::

Sylvain Gugger's avatar
Sylvain Gugger committed
639
    >>> print(nlp(sequence))
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
    [
        {'word': 'Hu', 'score': 0.9995632767677307, 'entity': 'I-ORG'},
        {'word': '##gging', 'score': 0.9915938973426819, 'entity': 'I-ORG'},
        {'word': 'Face', 'score': 0.9982671737670898, 'entity': 'I-ORG'},
        {'word': 'Inc', 'score': 0.9994403719902039, 'entity': 'I-ORG'},
        {'word': 'New', 'score': 0.9994346499443054, 'entity': 'I-LOC'},
        {'word': 'York', 'score': 0.9993270635604858, 'entity': 'I-LOC'},
        {'word': 'City', 'score': 0.9993864893913269, 'entity': 'I-LOC'},
        {'word': 'D', 'score': 0.9825621843338013, 'entity': 'I-LOC'},
        {'word': '##UM', 'score': 0.936983048915863, 'entity': 'I-LOC'},
        {'word': '##BO', 'score': 0.8987102508544922, 'entity': 'I-LOC'},
        {'word': 'Manhattan', 'score': 0.9758241176605225, 'entity': 'I-LOC'},
        {'word': 'Bridge', 'score': 0.990249514579773, 'entity': 'I-LOC'}
    ]

655
Note how the tokens of the sequence "Hugging Face" have been identified as an organisation, and "New York City",
Sylvain Gugger's avatar
Sylvain Gugger committed
656
"DUMBO" and "Manhattan Bridge" have been identified as locations.
657

658
659
Here is an example of doing named entity recognition, using a model and a tokenizer. The process is the following:

Sylvain Gugger's avatar
Sylvain Gugger committed
660
661
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it
   with the weights stored in the checkpoint.
662
663
2. Define the label list with which the model was trained on.
3. Define a sequence with known entities, such as "Hugging Face" as an organisation and "New York City" as a location.
Sylvain Gugger's avatar
Sylvain Gugger committed
664
665
4. Split words into tokens so that they can be mapped to predictions. We use a small hack by, first, completely
   encoding and decoding the sequence, so that we're left with a string that contains the special tokens.
666
5. Encode that sequence into IDs (special tokens are added automatically).
Sylvain Gugger's avatar
Sylvain Gugger committed
667
668
669
6. Retrieve the predictions by passing the input to the model and getting the first output. This results in a
   distribution over the 9 possible classes for each token. We take the argmax to retrieve the most likely class for
   each token.
670
7. Zip together each token with its prediction and print it.
671

672
673
674
675
676
677
.. code-block::

    >>> ## PYTORCH CODE
    >>> from transformers import AutoModelForTokenClassification, AutoTokenizer
    >>> import torch

678
    >>> model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
    >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

    >>> label_list = [
    ...     "O",       # Outside of a named entity
    ...     "B-MISC",  # Beginning of a miscellaneous entity right after another miscellaneous entity
    ...     "I-MISC",  # Miscellaneous entity
    ...     "B-PER",   # Beginning of a person's name right after another person's name
    ...     "I-PER",   # Person's name
    ...     "B-ORG",   # Beginning of an organisation right after another organisation
    ...     "I-ORG",   # Organisation
    ...     "B-LOC",   # Beginning of a location right after another location
    ...     "I-LOC"    # Location
    ... ]

    >>> sequence = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very" \
    ...            "close to the Manhattan Bridge."

    >>> # Bit of a hack to get the tokens with the special tokens
    >>> tokens = tokenizer.tokenize(tokenizer.decode(tokenizer.encode(sequence)))
    >>> inputs = tokenizer.encode(sequence, return_tensors="pt")

700
    >>> outputs = model(inputs).logits
701
702
703
704
705
    >>> predictions = torch.argmax(outputs, dim=2)
    >>> ## TENSORFLOW CODE
    >>> from transformers import TFAutoModelForTokenClassification, AutoTokenizer
    >>> import tensorflow as tf

706
    >>> model = TFAutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
    >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

    >>> label_list = [
    ...     "O",       # Outside of a named entity
    ...     "B-MISC",  # Beginning of a miscellaneous entity right after another miscellaneous entity
    ...     "I-MISC",  # Miscellaneous entity
    ...     "B-PER",   # Beginning of a person's name right after another person's name
    ...     "I-PER",   # Person's name
    ...     "B-ORG",   # Beginning of an organisation right after another organisation
    ...     "I-ORG",   # Organisation
    ...     "B-LOC",   # Beginning of a location right after another location
    ...     "I-LOC"    # Location
    ... ]

    >>> sequence = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very" \
    ...            "close to the Manhattan Bridge."

    >>> # Bit of a hack to get the tokens with the special tokens
    >>> tokens = tokenizer.tokenize(tokenizer.decode(tokenizer.encode(sequence)))
    >>> inputs = tokenizer.encode(sequence, return_tensors="tf")

    >>> outputs = model(inputs)[0]
    >>> predictions = tf.argmax(outputs, axis=2)
730
731


Sylvain Gugger's avatar
Sylvain Gugger committed
732
733
734
This outputs a list of each token mapped to its corresponding prediction. Differently from the pipeline, here every
token has a prediction as we didn't remove the "0"th class, which means that no particular entity was found on that
token. The following array should be the output:
735

736
.. code-block::
737

738
    >>> print([(token, label_list[prediction]) for token, prediction in zip(tokens, predictions[0].numpy())])
739
740
    [('[CLS]', 'O'), ('Hu', 'I-ORG'), ('##gging', 'I-ORG'), ('Face', 'I-ORG'), ('Inc', 'I-ORG'), ('.', 'O'), ('is', 'O'), ('a', 'O'), ('company', 'O'), ('based', 'O'), ('in', 'O'), ('New', 'I-LOC'), ('York', 'I-LOC'), ('City', 'I-LOC'), ('.', 'O'), ('Its', 'O'), ('headquarters', 'O'), ('are', 'O'), ('in', 'O'), ('D', 'I-LOC'), ('##UM', 'I-LOC'), ('##BO', 'I-LOC'), (',', 'O'), ('therefore', 'O'), ('very', 'O'), ('##c', 'O'), ('##lose', 'O'), ('to', 'O'), ('the', 'O'), ('Manhattan', 'I-LOC'), ('Bridge', 'I-LOC'), ('.', 'O'), ('[SEP]', 'O')]

741
Summarization
Sylvain Gugger's avatar
Sylvain Gugger committed
742
-----------------------------------------------------------------------------------------------------------------------
743

744
Summarization is the task of summarizing a document or an article into a shorter text. If you would like to fine-tune a
745
model on a summarization task, you may leverage the `run_summarization.py
Sylvain Gugger's avatar
Sylvain Gugger committed
746
747
<https://github.com/huggingface/transformers/tree/master/examples/pytorch/summarization/run_summarization.py>`__
script.
748

Sylvain Gugger's avatar
Sylvain Gugger committed
749
750
An example of a summarization dataset is the CNN / Daily Mail dataset, which consists of long news articles and was
created for the task of summarization. If you would like to fine-tune a model on a summarization task, various
Sylvain Gugger's avatar
Sylvain Gugger committed
751
approaches are described in this :prefix_link:`document <examples/pytorch/summarization/README.md>`.
752

Sylvain Gugger's avatar
Sylvain Gugger committed
753
754
Here is an example of using the pipelines to do summarization. It leverages a Bart model that was fine-tuned on the CNN
/ Daily Mail data set.
755

756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
.. code-block::

    >>> from transformers import pipeline

    >>> summarizer = pipeline("summarization")

    >>> ARTICLE = """ New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County, New York.
    ... A year later, she got married again in Westchester County, but to a different man and without divorcing her first husband.
    ... Only 18 days after that marriage, she got hitched yet again. Then, Barrientos declared "I do" five more times, sometimes only within two weeks of each other.
    ... In 2010, she married once more, this time in the Bronx. In an application for a marriage license, she stated it was her "first and only" marriage.
    ... Barrientos, now 39, is facing two criminal counts of "offering a false instrument for filing in the first degree," referring to her false statements on the
    ... 2010 marriage license application, according to court documents.
    ... Prosecutors said the marriages were part of an immigration scam.
    ... On Friday, she pleaded not guilty at State Supreme Court in the Bronx, according to her attorney, Christopher Wright, who declined to comment further.
    ... After leaving court, Barrientos was arrested and charged with theft of service and criminal trespass for allegedly sneaking into the New York subway through an emergency exit, said Detective
    ... Annette Markowski, a police spokeswoman. In total, Barrientos has been married 10 times, with nine of her marriages occurring between 1999 and 2002.
    ... All occurred either in Westchester County, Long Island, New Jersey or the Bronx. She is believed to still be married to four men, and at one time, she was married to eight men at once, prosecutors say.
    ... Prosecutors said the immigration scam involved some of her husbands, who filed for permanent residence status shortly after the marriages.
    ... Any divorces happened only after such filings were approved. It was unclear whether any of the men will be prosecuted.
    ... The case was referred to the Bronx District Attorney\'s Office by Immigration and Customs Enforcement and the Department of Homeland Security\'s
    ... Investigation Division. Seven of the men are from so-called "red-flagged" countries, including Egypt, Turkey, Georgia, Pakistan and Mali.
    ... Her eighth husband, Rashid Rajput, was deported in 2006 to his native Pakistan after an investigation by the Joint Terrorism Task Force.
    ... If convicted, Barrientos faces up to four years in prison.  Her next court appearance is scheduled for May 18.
    ... """
780

Sylvain Gugger's avatar
Sylvain Gugger committed
781
782
783
Because the summarization pipeline depends on the ``PreTrainedModel.generate()`` method, we can override the default
arguments of ``PreTrainedModel.generate()`` directly in the pipeline for ``max_length`` and ``min_length`` as shown
below. This outputs the following summary:
784

785
786
787
788
.. code-block::

    >>> print(summarizer(ARTICLE, max_length=130, min_length=30, do_sample=False))
    [{'summary_text': 'Liana Barrientos, 39, is charged with two counts of "offering a false instrument for filing in the first degree" In total, she has been married 10 times, with nine of her marriages occurring between 1999 and 2002. She is believed to still be married to four men.'}]
789

790
Here is an example of doing summarization using a model and a tokenizer. The process is the following:
791

Sylvain Gugger's avatar
Sylvain Gugger committed
792
793
1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder
   model, such as ``Bart`` or ``T5``.
794
795
2. Define the article that should be summarized.
3. Add the T5 specific prefix "summarize: ".
796
4. Use the ``PreTrainedModel.generate()`` method to generate the summary.
797

Sylvain Gugger's avatar
Sylvain Gugger committed
798
In this example we use Google's T5 model. Even though it was pre-trained only on a multi-task mixed dataset (including
Sylvain Gugger's avatar
Sylvain Gugger committed
799
CNN / Daily Mail), it yields very good results.
800

801
.. code-block::
802

803
804
    >>> ## PYTORCH CODE
    >>> from transformers import AutoModelWithLMHead, AutoTokenizer
805

806
    >>> model = AutoModelWithLMHead.from_pretrained("t5-base")
807
    >>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
808

809
810
811
812
813
    >>> # T5 uses a max_length of 512 so we cut the article to 512 tokens.
    >>> inputs = tokenizer.encode("summarize: " + ARTICLE, return_tensors="pt", max_length=512)
    >>> outputs = model.generate(inputs, max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
    >>> ## TENSORFLOW CODE
    >>> from transformers import TFAutoModelWithLMHead, AutoTokenizer
814

815
    >>> model = TFAutoModelWithLMHead.from_pretrained("t5-base")
816
    >>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
817

818
819
820
    >>> # T5 uses a max_length of 512 so we cut the article to 512 tokens.
    >>> inputs = tokenizer.encode("summarize: " + ARTICLE, return_tensors="tf", max_length=512)
    >>> outputs = model.generate(inputs, max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
821

822
Translation
Sylvain Gugger's avatar
Sylvain Gugger committed
823
-----------------------------------------------------------------------------------------------------------------------
824

825
Translation is the task of translating a text from one language to another. If you would like to fine-tune a model on a
826
translation task, you may leverage the `run_translation.py
Sylvain Gugger's avatar
Sylvain Gugger committed
827
<https://github.com/huggingface/transformers/tree/master/examples/pytorch/translation/run_translation.py>`__ script.
828

Sylvain Gugger's avatar
Sylvain Gugger committed
829
830
An example of a translation dataset is the WMT English to German dataset, which has sentences in English as the input
data and the corresponding sentences in German as the target data. If you would like to fine-tune a model on a
Sylvain Gugger's avatar
Sylvain Gugger committed
831
832
translation task, various approaches are described in this :prefix_link:`document
<examples/pytorch.translation/README.md>`.
833

Sylvain Gugger's avatar
Sylvain Gugger committed
834
835
Here is an example of using the pipelines to do translation. It leverages a T5 model that was only pre-trained on a
multi-task mixture dataset (including WMT), yet, yielding impressive translation results.
836

837
.. code-block::
838

839
    >>> from transformers import pipeline
840

841
842
843
    >>> translator = pipeline("translation_en_to_de")
    >>> print(translator("Hugging Face is a technology company based in New York and Paris", max_length=40))
    [{'translation_text': 'Hugging Face ist ein Technologieunternehmen mit Sitz in New York und Paris.'}]
844

Sylvain Gugger's avatar
Sylvain Gugger committed
845
846
Because the translation pipeline depends on the ``PreTrainedModel.generate()`` method, we can override the default
arguments of ``PreTrainedModel.generate()`` directly in the pipeline as is shown for ``max_length`` above.
847

848
849
Here is an example of doing translation using a model and a tokenizer. The process is the following:

Sylvain Gugger's avatar
Sylvain Gugger committed
850
851
1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder
   model, such as ``Bart`` or ``T5``.
852
2. Define the article that should be summarized.
853
3. Add the T5 specific prefix "translate English to German: "
854
4. Use the ``PreTrainedModel.generate()`` method to perform the translation.
855

856
.. code-block::
857

858
859
    >>> ## PYTORCH CODE
    >>> from transformers import AutoModelWithLMHead, AutoTokenizer
860

861
    >>> model = AutoModelWithLMHead.from_pretrained("t5-base")
862
    >>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
863

864
865
866
867
    >>> inputs = tokenizer.encode("translate English to German: Hugging Face is a technology company based in New York and Paris", return_tensors="pt")
    >>> outputs = model.generate(inputs, max_length=40, num_beams=4, early_stopping=True)
    >>> ## TENSORFLOW CODE
    >>> from transformers import TFAutoModelWithLMHead, AutoTokenizer
868

869
    >>> model = TFAutoModelWithLMHead.from_pretrained("t5-base")
870
    >>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
871

872
873
    >>> inputs = tokenizer.encode("translate English to German: Hugging Face is a technology company based in New York and Paris", return_tensors="tf")
    >>> outputs = model.generate(inputs, max_length=40, num_beams=4, early_stopping=True)
874

875
876
877
878
879
880
As with the pipeline example, we get the same translation:

.. code-block::

    >>> print(tokenizer.decode(outputs[0]))
    Hugging Face ist ein Technologieunternehmen mit Sitz in New York und Paris.