"examples/pytorch/summarization/README.md" did not exist on "eadd870b2f503047dd81b8dcd9d115dc1b4a9196"
serialization.mdx 26 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
<!--Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

13
# Export 馃 Transformers Models
Sylvain Gugger's avatar
Sylvain Gugger committed
14

lewtun's avatar
lewtun committed
15
16
17
18
If you need to deploy 馃 Transformers models in production environments, we
recommend exporting them to a serialized format that can be loaded and executed
on specialized runtimes and hardware. In this guide, we'll show you how to
export 馃 Transformers models in two widely used formats: ONNX and TorchScript.
Sylvain Gugger's avatar
Sylvain Gugger committed
19

lewtun's avatar
lewtun committed
20
21
22
23
Once exported, a model can optimized for inference via techniques such as
quantization and pruning. If you are interested in optimizing your models to run
with maximum efficiency, check out the [馃 Optimum
library](https://github.com/huggingface/optimum).
Sylvain Gugger's avatar
Sylvain Gugger committed
24

lewtun's avatar
lewtun committed
25
## ONNX
Sylvain Gugger's avatar
Sylvain Gugger committed
26

lewtun's avatar
lewtun committed
27
28
29
30
31
32
33
The [ONNX (Open Neural Network eXchange)](http://onnx.ai) project is an open
standard that defines a common set of operators and a common file format to
represent deep learning models in a wide variety of frameworks, including
PyTorch and TensorFlow. When a model is exported to the ONNX format, these
operators are used to construct a computational graph (often called an
_intermediate representation_) which represents the flow of data through the
neural network.
Sylvain Gugger's avatar
Sylvain Gugger committed
34

lewtun's avatar
lewtun committed
35
36
37
By exposing a graph with standardized operators and data types, ONNX makes it
easy to switch between frameworks. For example, a model trained in PyTorch can
be exported to ONNX format and then imported in TensorFlow (and vice versa).
Sylvain Gugger's avatar
Sylvain Gugger committed
38

lewtun's avatar
lewtun committed
39
40
41
42
馃 Transformers provides a `transformers.onnx` package that enables you to
convert model checkpoints to an ONNX graph by leveraging configuration objects.
These configuration objects come ready made for a number of model architectures,
and are designed to be easily extendable to other architectures.
Sylvain Gugger's avatar
Sylvain Gugger committed
43

lewtun's avatar
lewtun committed
44
Ready-made configurations include the following architectures:
Sylvain Gugger's avatar
Sylvain Gugger committed
45

46
<!--This table is automatically generated by `make fix-copies`, do not fill manually!-->
Sylvain Gugger's avatar
Sylvain Gugger committed
47
48
49

- ALBERT
- BART
Jim Rohrer's avatar
Jim Rohrer committed
50
- BEiT
Sylvain Gugger's avatar
Sylvain Gugger committed
51
- BERT
52
- BigBird
53
- BigBird-Pegasus
54
55
- Blenderbot
- BlenderbotSmall
Sylvain Gugger's avatar
Sylvain Gugger committed
56
- CamemBERT
57
- ConvBERT
58
- ConvNeXT
59
- Data2VecText
60
- Data2VecVision
61
- DeiT
Sylvain Gugger's avatar
Sylvain Gugger committed
62
- DistilBERT
63
- ELECTRA
64
- FlauBERT
Sylvain Gugger's avatar
Sylvain Gugger committed
65
- GPT Neo
66
- GPT-J
67
- I-BERT
Sylvain Gugger's avatar
Sylvain Gugger committed
68
- LayoutLM
Daniel Stancl's avatar
Daniel Stancl committed
69
- LongT5
70
- M2M100
71
- Marian
Sylvain Gugger's avatar
Sylvain Gugger committed
72
- mBART
73
- MobileBERT
Sylvain Gugger's avatar
Sylvain Gugger committed
74
- OpenAI GPT-2
75
- Perceiver
Gunjan Chhablani's avatar
Gunjan Chhablani committed
76
- PLBart
regisss's avatar
regisss committed
77
- ResNet
Sylvain Gugger's avatar
Sylvain Gugger committed
78
- RoBERTa
79
- RoFormer
80
- SqueezeBERT
Sylvain Gugger's avatar
Sylvain Gugger committed
81
- T5
lewtun's avatar
lewtun committed
82
- ViT
Ritik Nandwal's avatar
Ritik Nandwal committed
83
- XLM
Sylvain Gugger's avatar
Sylvain Gugger committed
84
- XLM-RoBERTa
85
- XLM-RoBERTa-XL
Sylvain Gugger's avatar
Sylvain Gugger committed
86

lewtun's avatar
lewtun committed
87
In the next two sections, we'll show you how to:
Sylvain Gugger's avatar
Sylvain Gugger committed
88

lewtun's avatar
lewtun committed
89
90
* Export a supported model using the `transformers.onnx` package.
* Export a custom model for an unsupported architecture.
Sylvain Gugger's avatar
Sylvain Gugger committed
91

lewtun's avatar
lewtun committed
92
### Exporting a model to ONNX
Sylvain Gugger's avatar
Sylvain Gugger committed
93

lewtun's avatar
lewtun committed
94
95
To export a 馃 Transformers model to ONNX, you'll first need to install some
extra dependencies:
Sylvain Gugger's avatar
Sylvain Gugger committed
96

lewtun's avatar
lewtun committed
97
98
99
100
101
```bash
pip install transformers[onnx]
```

The `transformers.onnx` package can then be used as a Python module:
Sylvain Gugger's avatar
Sylvain Gugger committed
102
103
104
105

```bash
python -m transformers.onnx --help

lewtun's avatar
lewtun committed
106
usage: Hugging Face Transformers ONNX exporter [-h] -m MODEL [--feature {causal-lm, ...}] [--opset OPSET] [--atol ATOL] output
Sylvain Gugger's avatar
Sylvain Gugger committed
107
108
109
110
111
112
113

positional arguments:
  output                Path indicating where to store generated ONNX model.

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
lewtun's avatar
lewtun committed
114
115
116
117
118
                        Model ID on huggingface.co or path on disk to load model from.
  --feature {causal-lm, ...}
                        The type of features to export the model with.
  --opset OPSET         ONNX opset version to export the model with.
  --atol ATOL           Absolute difference tolerence when validating the model.
Sylvain Gugger's avatar
Sylvain Gugger committed
119
120
121
122
123
```

Exporting a checkpoint using a ready-made configuration can be done as follows:

```bash
lewtun's avatar
lewtun committed
124
python -m transformers.onnx --model=distilbert-base-uncased onnx/
Sylvain Gugger's avatar
Sylvain Gugger committed
125
126
```

lewtun's avatar
lewtun committed
127
which should show the following logs:
Sylvain Gugger's avatar
Sylvain Gugger committed
128
129
130

```bash
Validating ONNX model...
131
        -[鉁揮 ONNX model output names match reference model ({'last_hidden_state'})
lewtun's avatar
lewtun committed
132
133
134
135
        - Validating ONNX Model output "last_hidden_state":
                -[鉁揮 (2, 8, 768) matches (2, 8, 768)
                -[鉁揮 all values close (atol: 1e-05)
All good, model saved at: onnx/model.onnx
Sylvain Gugger's avatar
Sylvain Gugger committed
136
137
```

lewtun's avatar
lewtun committed
138
This exports an ONNX graph of the checkpoint defined by the `--model` argument.
139
140
In this example it is `distilbert-base-uncased`, but it can be any checkpoint on
the Hugging Face Hub or one that's stored locally.
Sylvain Gugger's avatar
Sylvain Gugger committed
141

lewtun's avatar
lewtun committed
142
143
144
145
The resulting `model.onnx` file can then be run on one of the [many
accelerators](https://onnx.ai/supported-tools.html#deployModel) that support the
ONNX standard. For example, we can load and run the model with [ONNX
Runtime](https://onnxruntime.ai/) as follows:
Sylvain Gugger's avatar
Sylvain Gugger committed
146

lewtun's avatar
lewtun committed
147
148
149
150
151
152
153
154
155
156
```python
>>> from transformers import AutoTokenizer
>>> from onnxruntime import InferenceSession

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
>>> session = InferenceSession("onnx/model.onnx")
>>> # ONNX Runtime expects NumPy arrays as input
>>> inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
>>> outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```
Sylvain Gugger's avatar
Sylvain Gugger committed
157

lewtun's avatar
lewtun committed
158
159
160
The required output names (i.e. `["last_hidden_state"]`) can be obtained by
taking a look at the ONNX configuration of each model. For example, for
DistilBERT we have:
Sylvain Gugger's avatar
Sylvain Gugger committed
161

lewtun's avatar
lewtun committed
162
163
```python
>>> from transformers.models.distilbert import DistilBertConfig, DistilBertOnnxConfig
Sylvain Gugger's avatar
Sylvain Gugger committed
164

lewtun's avatar
lewtun committed
165
166
167
168
>>> config = DistilBertConfig()
>>> onnx_config = DistilBertOnnxConfig(config)
>>> print(list(onnx_config.outputs.keys()))
["last_hidden_state"]
Sylvain Gugger's avatar
Sylvain Gugger committed
169
170
```

171
172
173
174
175
176
177
178
179
180
181
182
The process is identical for TensorFlow checkpoints on the Hub. For example, we
can export a pure TensorFlow checkpoint from the [Keras
organization](https://huggingface.co/keras-io) as follows:

```bash
python -m transformers.onnx --model=keras-io/transformers-qa onnx/
```

To export a model that's stored locally, you'll need to have the model's weights
and tokenizer files stored in a directory. For example, we can load and save a
checkpoint as follows:

Sylvain Gugger's avatar
Sylvain Gugger committed
183
184
<frameworkcontent>
<pt>
185
186
187
188
```python
>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification

>>> # Load tokenizer and PyTorch weights form the Hub
189
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
190
191
192
193
>>> pt_model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased")
>>> # Save to disk
>>> tokenizer.save_pretrained("local-pt-checkpoint")
>>> pt_model.save_pretrained("local-pt-checkpoint")
Sylvain Gugger's avatar
Sylvain Gugger committed
194
195
196
197
198
199
200
201
202
203
204
```

Once the checkpoint is saved, we can export it to ONNX by pointing the `--model`
argument of the `transformers.onnx` package to the desired directory:

```bash
python -m transformers.onnx --model=local-pt-checkpoint onnx/
```
</pt>
<tf>
```python
205
206
207
>>> from transformers import AutoTokenizer, TFAutoModelForSequenceClassification

>>> # Load tokenizer and TensorFlow weights from the Hub
208
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
209
210
211
212
213
214
215
216
217
218
219
220
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased")
>>> # Save to disk
>>> tokenizer.save_pretrained("local-tf-checkpoint")
>>> tf_model.save_pretrained("local-tf-checkpoint")
```

Once the checkpoint is saved, we can export it to ONNX by pointing the `--model`
argument of the `transformers.onnx` package to the desired directory:

```bash
python -m transformers.onnx --model=local-tf-checkpoint onnx/
```
Sylvain Gugger's avatar
Sylvain Gugger committed
221
222
</tf>
</frameworkcontent>
223

lewtun's avatar
lewtun committed
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
### Selecting features for different model topologies

Each ready-made configuration comes with a set of _features_ that enable you to
export models for different types of topologies or tasks. As shown in the table
below, each feature is associated with a different auto class:

| Feature                              | Auto Class                           |
| ------------------------------------ | ------------------------------------ |
| `causal-lm`, `causal-lm-with-past`   | `AutoModelForCausalLM`               |
| `default`, `default-with-past`       | `AutoModel`                          |
| `masked-lm`                          | `AutoModelForMaskedLM`               |
| `question-answering`                 | `AutoModelForQuestionAnswering`      |
| `seq2seq-lm`, `seq2seq-lm-with-past` | `AutoModelForSeq2SeqLM`              |
| `sequence-classification`            | `AutoModelForSequenceClassification` |
| `token-classification`               | `AutoModelForTokenClassification`    |

For each configuration, you can find the list of supported features via the
`FeaturesManager`. For example, for DistilBERT we have:
Sylvain Gugger's avatar
Sylvain Gugger committed
242
243

```python
lewtun's avatar
lewtun committed
244
>>> from transformers.onnx.features import FeaturesManager
Sylvain Gugger's avatar
Sylvain Gugger committed
245

lewtun's avatar
lewtun committed
246
247
248
>>> distilbert_features = list(FeaturesManager.get_supported_features_for_model_type("distilbert").keys())
>>> print(distilbert_features)
["default", "masked-lm", "causal-lm", "sequence-classification", "token-classification", "question-answering"]
Sylvain Gugger's avatar
Sylvain Gugger committed
249
250
```

lewtun's avatar
lewtun committed
251
252
253
You can then pass one of these features to the `--feature` argument in the
`transformers.onnx` package. For example, to export a text-classification model
we can pick a fine-tuned model from the Hub and run:
Sylvain Gugger's avatar
Sylvain Gugger committed
254

lewtun's avatar
lewtun committed
255
256
257
258
```bash
python -m transformers.onnx --model=distilbert-base-uncased-finetuned-sst-2-english \
                            --feature=sequence-classification onnx/
```
Sylvain Gugger's avatar
Sylvain Gugger committed
259

lewtun's avatar
lewtun committed
260
261
262
263
which will display the following logs:

```bash
Validating ONNX model...
264
        -[鉁揮 ONNX model output names match reference model ({'logits'})
lewtun's avatar
lewtun committed
265
266
267
268
        - Validating ONNX Model output "logits":
                -[鉁揮 (2, 2) matches (2, 2)
                -[鉁揮 all values close (atol: 1e-05)
All good, model saved at: onnx/model.onnx
Sylvain Gugger's avatar
Sylvain Gugger committed
269
270
```

lewtun's avatar
lewtun committed
271
272
273
274
275
276
277
278
279
280
281
282
Notice that in this case, the output names from the fine-tuned model are
`logits` instead of the `last_hidden_state` we saw with the
`distilbert-base-uncased` checkpoint earlier. This is expected since the
fine-tuned model has a sequence classification head.

<Tip>

The features that have a `with-past` suffix (e.g. `causal-lm-with-past`)
correspond to model topologies with precomputed hidden states (key and values
in the attention blocks) that can be used for fast autoregressive decoding.

</Tip>
Sylvain Gugger's avatar
Sylvain Gugger committed
283
284


lewtun's avatar
lewtun committed
285
### Exporting a model for an unsupported architecture
Sylvain Gugger's avatar
Sylvain Gugger committed
286

lewtun's avatar
lewtun committed
287
288
If you wish to export a model whose architecture is not natively supported by
the library, there are three main steps to follow:
Sylvain Gugger's avatar
Sylvain Gugger committed
289

lewtun's avatar
lewtun committed
290
291
292
1. Implement a custom ONNX configuration.
2. Export the model to ONNX.
3. Validate the outputs of the PyTorch and exported models.
Sylvain Gugger's avatar
Sylvain Gugger committed
293

lewtun's avatar
lewtun committed
294
295
In this section, we'll look at how DistilBERT was implemented to show what's
involved with each step.
Sylvain Gugger's avatar
Sylvain Gugger committed
296

lewtun's avatar
lewtun committed
297
#### Implementing a custom ONNX configuration
Sylvain Gugger's avatar
Sylvain Gugger committed
298

lewtun's avatar
lewtun committed
299
300
301
Let's start with the ONNX configuration object. We provide three abstract
classes that you should inherit from, depending on the type of model
architecture you wish to export:
Sylvain Gugger's avatar
Sylvain Gugger committed
302

303
304
305
* Encoder-based models inherit from [`~onnx.config.OnnxConfig`]
* Decoder-based models inherit from [`~onnx.config.OnnxConfigWithPast`]
* Encoder-decoder models inherit from [`~onnx.config.OnnxSeq2SeqConfigWithPast`]
Sylvain Gugger's avatar
Sylvain Gugger committed
306
307
308

<Tip>

lewtun's avatar
lewtun committed
309
310
A good way to implement a custom ONNX configuration is to look at the existing
implementation in the `configuration_<model_name>.py` file of a similar architecture.
Sylvain Gugger's avatar
Sylvain Gugger committed
311
312
313

</Tip>

lewtun's avatar
lewtun committed
314
315
Since DistilBERT is an encoder-based model, its configuration inherits from
`OnnxConfig`:
Sylvain Gugger's avatar
Sylvain Gugger committed
316

lewtun's avatar
lewtun committed
317
318
319
320
321
322
323
324
325
326
327
328
329
330
```python
>>> from typing import Mapping, OrderedDict
>>> from transformers.onnx import OnnxConfig


>>> class DistilBertOnnxConfig(OnnxConfig):
...     @property
...     def inputs(self) -> Mapping[str, Mapping[int, str]]:
...         return OrderedDict(
...             [
...                 ("input_ids", {0: "batch", 1: "sequence"}),
...                 ("attention_mask", {0: "batch", 1: "sequence"}),
...             ]
...         )
Sylvain Gugger's avatar
Sylvain Gugger committed
331
332
```

lewtun's avatar
lewtun committed
333
334
335
336
337
338
Every configuration object must implement the `inputs` property and return a
mapping, where each key corresponds to an expected input, and each value
indicates the axis of that input. For DistilBERT, we can see that two inputs are
required: `input_ids` and `attention_mask`. These inputs have the same shape of
`(batch_size, sequence_length)` which is why we see the same axes used in the
configuration.
Sylvain Gugger's avatar
Sylvain Gugger committed
339
340
341

<Tip>

lewtun's avatar
lewtun committed
342
343
344
345
346
Notice that `inputs` property for `DistilBertOnnxConfig` returns an
`OrderedDict`. This ensures that the inputs are matched with their relative
position within the `PreTrainedModel.forward()` method when tracing the graph.
We recommend using an `OrderedDict` for the `inputs` and `outputs` properties
when implementing custom ONNX configurations.
Sylvain Gugger's avatar
Sylvain Gugger committed
347
348
349

</Tip>

lewtun's avatar
lewtun committed
350
351
Once you have implemented an ONNX configuration, you can instantiate it by
providing the base model's configuration as follows:
Sylvain Gugger's avatar
Sylvain Gugger committed
352

lewtun's avatar
lewtun committed
353
354
```python
>>> from transformers import AutoConfig
Sylvain Gugger's avatar
Sylvain Gugger committed
355

lewtun's avatar
lewtun committed
356
357
358
>>> config = AutoConfig.from_pretrained("distilbert-base-uncased")
>>> onnx_config = DistilBertOnnxConfig(config)
```
Sylvain Gugger's avatar
Sylvain Gugger committed
359

lewtun's avatar
lewtun committed
360
361
The resulting object has several useful properties. For example you can view the
ONNX operator set that will be used during the export:
Sylvain Gugger's avatar
Sylvain Gugger committed
362

lewtun's avatar
lewtun committed
363
364
365
366
```python
>>> print(onnx_config.default_onnx_opset)
11
```
Sylvain Gugger's avatar
Sylvain Gugger committed
367

lewtun's avatar
lewtun committed
368
You can also view the outputs associated with the model as follows:
Sylvain Gugger's avatar
Sylvain Gugger committed
369

lewtun's avatar
lewtun committed
370
371
372
373
```python
>>> print(onnx_config.outputs)
OrderedDict([("last_hidden_state", {0: "batch", 1: "sequence"})])
```
Sylvain Gugger's avatar
Sylvain Gugger committed
374

lewtun's avatar
lewtun committed
375
376
377
378
379
380
381
382
383
Notice that the outputs property follows the same structure as the inputs; it
returns an `OrderedDict` of named outputs and their shapes. The output structure
is linked to the choice of feature that the configuration is initialised with.
By default, the ONNX configuration is initialized with the `default` feature
that corresponds to exporting a model loaded with the `AutoModel` class. If you
want to export a different model topology, just provide a different feature to
the `task` argument when you initialize the ONNX configuration. For example, if
we wished to export DistilBERT with a sequence classification head, we could
use:
Sylvain Gugger's avatar
Sylvain Gugger committed
384

lewtun's avatar
lewtun committed
385
386
```python
>>> from transformers import AutoConfig
Sylvain Gugger's avatar
Sylvain Gugger committed
387

lewtun's avatar
lewtun committed
388
389
390
391
392
>>> config = AutoConfig.from_pretrained("distilbert-base-uncased")
>>> onnx_config_for_seq_clf = DistilBertOnnxConfig(config, task="sequence-classification")
>>> print(onnx_config_for_seq_clf.outputs)
OrderedDict([('logits', {0: 'batch'})])
```
Sylvain Gugger's avatar
Sylvain Gugger committed
393
394
395

<Tip>

396
397
398
All of the base properties and methods associated with [`~onnx.config.OnnxConfig`] and the
other configuration classes can be overriden if needed. Check out
[`BartOnnxConfig`] for an advanced example.
Sylvain Gugger's avatar
Sylvain Gugger committed
399
400
401

</Tip>

lewtun's avatar
lewtun committed
402
#### Exporting the model
Sylvain Gugger's avatar
Sylvain Gugger committed
403

lewtun's avatar
lewtun committed
404
405
406
407
Once you have implemented the ONNX configuration, the next step is to export the
model. Here we can use the `export()` function provided by the
`transformers.onnx` package. This function expects the ONNX configuration, along
with the base model and tokenizer, and the path to save the exported file:
Sylvain Gugger's avatar
Sylvain Gugger committed
408

lewtun's avatar
lewtun committed
409
410
411
412
```python
>>> from pathlib import Path
>>> from transformers.onnx import export
>>> from transformers import AutoTokenizer, AutoModel
Sylvain Gugger's avatar
Sylvain Gugger committed
413

lewtun's avatar
lewtun committed
414
415
416
417
>>> onnx_path = Path("model.onnx")
>>> model_ckpt = "distilbert-base-uncased"
>>> base_model = AutoModel.from_pretrained(model_ckpt)
>>> tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
Sylvain Gugger's avatar
Sylvain Gugger committed
418

lewtun's avatar
lewtun committed
419
420
>>> onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, onnx_config.default_onnx_opset, onnx_path)
```
Sylvain Gugger's avatar
Sylvain Gugger committed
421

lewtun's avatar
lewtun committed
422
423
424
425
The `onnx_inputs` and `onnx_outputs` returned by the `export()` function are
lists of the keys defined in the `inputs` and `outputs` properties of the
configuration. Once the model is exported, you can test that the model is well
formed as follows:
Sylvain Gugger's avatar
Sylvain Gugger committed
426

lewtun's avatar
lewtun committed
427
428
```python
>>> import onnx
Sylvain Gugger's avatar
Sylvain Gugger committed
429

lewtun's avatar
lewtun committed
430
431
432
>>> onnx_model = onnx.load("model.onnx")
>>> onnx.checker.check_model(onnx_model)
```
Sylvain Gugger's avatar
Sylvain Gugger committed
433
434
435

<Tip>

lewtun's avatar
lewtun committed
436
437
438
439
440
441
If your model is larger than 2GB, you will see that many additional files are
created during the export. This is _expected_ because ONNX uses [Protocol
Buffers](https://developers.google.com/protocol-buffers/) to store the model and
these have a size limit of 2GB. See the [ONNX
documentation](https://github.com/onnx/onnx/blob/master/docs/ExternalData.md)
for instructions on how to load models with external data.
Sylvain Gugger's avatar
Sylvain Gugger committed
442
443
444

</Tip>

lewtun's avatar
lewtun committed
445
#### Validating the model outputs
Sylvain Gugger's avatar
Sylvain Gugger committed
446

lewtun's avatar
lewtun committed
447
448
449
450
The final step is to validate that the outputs from the base and exported model
agree within some absolute tolerance. Here we can use the
`validate_model_outputs()` function provided by the `transformers.onnx` package
as follows:
Sylvain Gugger's avatar
Sylvain Gugger committed
451

lewtun's avatar
lewtun committed
452
453
```python
>>> from transformers.onnx import validate_model_outputs
Sylvain Gugger's avatar
Sylvain Gugger committed
454

lewtun's avatar
lewtun committed
455
456
457
>>> validate_model_outputs(
...     onnx_config, tokenizer, base_model, onnx_path, onnx_outputs, onnx_config.atol_for_validation
... )
Sylvain Gugger's avatar
Sylvain Gugger committed
458
459
```

lewtun's avatar
lewtun committed
460
461
462
463
This function uses the `OnnxConfig.generate_dummy_inputs()` method to generate
inputs for the base and exported model, and the absolute tolerance can be
defined in the configuration. We generally find numerical agreement in the 1e-6
to 1e-4 range, although anything smaller than 1e-3 is likely to be OK.
Sylvain Gugger's avatar
Sylvain Gugger committed
464

lewtun's avatar
lewtun committed
465
### Contributing a new configuration to 馃 Transformers
Sylvain Gugger's avatar
Sylvain Gugger committed
466

lewtun's avatar
lewtun committed
467
468
469
We are looking to expand the set of ready-made configurations and welcome
contributions from the community! If you would like to contribute your addition
to the library, you will need to:
Sylvain Gugger's avatar
Sylvain Gugger committed
470

lewtun's avatar
lewtun committed
471
472
* Implement the ONNX configuration in the corresponding `configuration_<model_name>.py`
file
473
474
* Include the model architecture and corresponding features in [`~onnx.features.FeatureManager`]
* Add your model architecture to the tests in `test_onnx_v2.py`
Sylvain Gugger's avatar
Sylvain Gugger committed
475

lewtun's avatar
lewtun committed
476
477
478
Check out how the configuration for [IBERT was
contributed](https://github.com/huggingface/transformers/pull/14868/files) to
get an idea of what's involved.
Sylvain Gugger's avatar
Sylvain Gugger committed
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559

## TorchScript

<Tip>

This is the very beginning of our experiments with TorchScript and we are still exploring its capabilities with
variable-input-size models. It is a focus of interest to us and we will deepen our analysis in upcoming releases,
with more code examples, a more flexible implementation, and benchmarks comparing python-based codes with compiled
TorchScript.

</Tip>

According to Pytorch's documentation: "TorchScript is a way to create serializable and optimizable models from PyTorch
code". Pytorch's two modules [JIT and TRACE](https://pytorch.org/docs/stable/jit.html) allow the developer to export
their model to be re-used in other programs, such as efficiency-oriented C++ programs.

We have provided an interface that allows the export of 馃 Transformers models to TorchScript so that they can be reused
in a different environment than a Pytorch-based python program. Here we explain how to export and use our models using
TorchScript.

Exporting a model requires two things:

- a forward pass with dummy inputs.
- model instantiation with the `torchscript` flag.

These necessities imply several things developers should be careful about. These are detailed below.

### TorchScript flag and tied weights

This flag is necessary because most of the language models in this repository have tied weights between their
`Embedding` layer and their `Decoding` layer. TorchScript does not allow the export of models that have tied
weights, therefore it is necessary to untie and clone the weights beforehand.

This implies that models instantiated with the `torchscript` flag have their `Embedding` layer and `Decoding`
layer separate, which means that they should not be trained down the line. Training would de-synchronize the two
layers, leading to unexpected results.

This is not the case for models that do not have a Language Model head, as those do not have tied weights. These models
can be safely exported without the `torchscript` flag.

### Dummy inputs and standard lengths

The dummy inputs are used to do a model forward pass. While the inputs' values are propagating through the layers,
Pytorch keeps track of the different operations executed on each tensor. These recorded operations are then used to
create the "trace" of the model.

The trace is created relatively to the inputs' dimensions. It is therefore constrained by the dimensions of the dummy
input, and will not work for any other sequence length or batch size. When trying with a different size, an error such
as:

`The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2`

will be raised. It is therefore recommended to trace the model with a dummy input size at least as large as the largest
input that will be fed to the model during inference. Padding can be performed to fill the missing values. As the model
will have been traced with a large input size however, the dimensions of the different matrix will be large as well,
resulting in more calculations.

It is recommended to be careful of the total number of operations done on each input and to follow performance closely
when exporting varying sequence-length models.

### Using TorchScript in Python

Below is an example, showing how to save, load models as well as how to use the trace for inference.

#### Saving a model

This snippet shows how to use TorchScript to export a `BertModel`. Here the `BertModel` is instantiated according
to a `BertConfig` class and then saved to disk under the filename `traced_bert.pt`

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch

enc = BertTokenizer.from_pretrained("bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
Sylvain Gugger's avatar
Sylvain Gugger committed
560
tokenized_text[masked_index] = "[MASK]"
Sylvain Gugger's avatar
Sylvain Gugger committed
561
562
563
564
565
566
567
568
569
570
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.
Sylvain Gugger's avatar
Sylvain Gugger committed
571
572
573
574
575
576
577
578
config = BertConfig(
    vocab_size_or_config_json_file=32000,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    torchscript=True,
)
Sylvain Gugger's avatar
Sylvain Gugger committed
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612

# Instantiating the model
model = BertModel(config)

# The model needs to be in evaluation mode
model.eval()

# If you are instantiating the model with *from_pretrained* you can also easily set the TorchScript flag
model = BertModel.from_pretrained("bert-base-uncased", torchscript=True)

# Creating the trace
traced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])
torch.jit.save(traced_model, "traced_bert.pt")
```

#### Loading a model

This snippet shows how to load the `BertModel` that was previously saved to disk under the name `traced_bert.pt`.
We are re-using the previously initialised `dummy_input`.

```python
loaded_model = torch.jit.load("traced_bert.pt")
loaded_model.eval()

all_encoder_layers, pooled_output = loaded_model(*dummy_input)
```

#### Using a traced model for inference

Using the traced model for inference is as simple as using its `__call__` dunder method:

```python
traced_model(tokens_tensor, segments_tensors)
```
613
614
615

### Deploying HuggingFace TorchScript models on AWS using the Neuron SDK

616
617
618
619
620
621
AWS introduced the [Amazon EC2 Inf1](https://aws.amazon.com/ec2/instance-types/inf1/)
instance family for low cost, high performance machine learning inference in the cloud.
The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware accelerator,
specializing in deep learning inferencing workloads.
[AWS Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#)
is the SDK for Inferentia that supports tracing and optimizing transformers models for
622
623
624
625
626
627
628
629
630
631
deployment on Inf1. The Neuron SDK provides:


1. Easy-to-use API with one line of code change to trace and optimize a TorchScript model for inference in the cloud.
2. Out of the box performance optimizations for [improved cost-performance](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/benchmark/>)
3. Support for HuggingFace transformers models built with either [PyTorch](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert.html)
   or [TensorFlow](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html).

#### Implications

632
633
634
Transformers Models based on the [BERT (Bidirectional Encoder Representations from Transformers)](https://huggingface.co/docs/transformers/main/model_doc/bert)
architecture, or its variants such as [distilBERT](https://huggingface.co/docs/transformers/main/model_doc/distilbert)
 and [roBERTa](https://huggingface.co/docs/transformers/main/model_doc/roberta)
635
 will run best on Inf1 for non-generative tasks such as Extractive Question Answering,
636
 Sequence Classification, Token Classification. Alternatively, text generation
637
638
tasks can be adapted to run on Inf1, according to this [AWS Neuron MarianMT tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/transformers-marianmt.html).
More information about models that can be converted out of the box on Inferentia can be
639
640
641
642
643
644
645
646
647
648
649
found in the [Model Architecture Fit section of the Neuron documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/models/models-inferentia.html#models-inferentia).

#### Dependencies

Using AWS Neuron to convert models requires the following dependencies and environment:

* A [Neuron SDK environment](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-frameworks/pytorch-neuron/index.html#installation-guide),
  which comes pre-configured on [AWS Deep Learning AMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-inferentia-launching.html).

#### Converting a Model for AWS Neuron

650
Using the same script as in [Using TorchScript in Python](https://huggingface.co/docs/transformers/main/en/serialization#using-torchscript-in-python)
651
to trace a "BertModel", you import `torch.neuron` framework extension to access
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
the components of the Neuron SDK through a Python API.

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch
import torch.neuron
```
And only modify the tracing line of code

from:

```python
torch.jit.trace(model, [tokens_tensor, segments_tensors])
```

to:

```python
torch.neuron.trace(model, [token_tensor, segments_tensors])
```

This change enables Neuron SDK to trace the model and optimize it to run in Inf1 instances.

675
To learn more about AWS Neuron SDK features, tools, example tutorials and latest updates,
NielsRogge's avatar
NielsRogge committed
676
please see the [AWS NeuronSDK documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html).