pipelines.mdx 14.4 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<!--Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

# Pipelines

The pipelines are a great and easy way to use models for inference. These pipelines are objects that abstract most of
the complex code from the library, offering a simple API dedicated to several tasks, including Named Entity
Recognition, Masked Language Modeling, Sentiment Analysis, Feature Extraction and Question Answering. See the
[task summary](../task_summary) for examples of use.

There are two categories of pipeline abstractions to be aware about:

- The [`pipeline`] which is the most powerful object encapsulating all other pipelines.
23
- Task-specific pipelines are available for [audio](#audio), [computer vision](#computer-vision), [natural language processing](#natural-language-processing), and [multimodal](#multimodal) tasks.
Sylvain Gugger's avatar
Sylvain Gugger committed
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

## The pipeline abstraction

The *pipeline* abstraction is a wrapper around all the other available pipelines. It is instantiated as any other
pipeline but can provide additional quality of life.

Simple call on one item:

```python
>>> pipe = pipeline("text-classification")
>>> pipe("This restaurant is awesome")
[{'label': 'POSITIVE', 'score': 0.9998743534088135}]
```

If you want to use a specific model from the [hub](https://huggingface.co) you can ignore the task if the model on
the hub already defines it:

```python
>>> pipe = pipeline(model="roberta-large-mnli")
>>> pipe("This restaurant is awesome")
Samuel Xu's avatar
Samuel Xu committed
44
[{'label': 'NEUTRAL', 'score': 0.7313136458396912}]
Sylvain Gugger's avatar
Sylvain Gugger committed
45
46
```

Samuel Xu's avatar
Samuel Xu committed
47
To call a pipeline on many items, you can call it with a *list*.
Sylvain Gugger's avatar
Sylvain Gugger committed
48
49
50

```python
>>> pipe = pipeline("text-classification")
Samuel Xu's avatar
Samuel Xu committed
51
>>> pipe(["This restaurant is awesome", "This restaurant is awful"])
Sylvain Gugger's avatar
Sylvain Gugger committed
52
53
54
55
[{'label': 'POSITIVE', 'score': 0.9998743534088135},
 {'label': 'NEGATIVE', 'score': 0.9996669292449951}]
```

Samuel Xu's avatar
Samuel Xu committed
56
To iterate over full datasets it is recommended to use a `dataset` directly. This means you don't need to allocate
Sylvain Gugger's avatar
Sylvain Gugger committed
57
58
59
60
61
62
the whole dataset at once, nor do you need to do batching yourself. This should work just as fast as custom loops on
GPU. If it doesn't don't hesitate to create an issue.

```python
import datasets
from transformers import pipeline
63
from transformers.pipelines.pt_utils import KeyDataset
64
from tqdm.auto import tqdm
Sylvain Gugger's avatar
Sylvain Gugger committed
65
66
67
68
69

pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h", device=0)
dataset = datasets.load_dataset("superb", name="asr", split="test")

# KeyDataset (only *pt*) will simply return the item in the dict returned by the dataset item
Rohit Gupta's avatar
Rohit Gupta committed
70
# as we're not interested in the *target* part of the dataset. For sentence pair use KeyPairDataset
71
for out in tqdm(pipe(KeyDataset(dataset, "file"))):
Sylvain Gugger's avatar
Sylvain Gugger committed
72
73
74
75
76
77
    print(out)
    # {"text": "NUMBER TEN FRESH NELLY IS WAITING ON YOU GOOD NIGHT HUSBAND"}
    # {"text": ....}
    # ....
```

78
79
80
81
82
83
84
85
For ease of use, a generator is also possible:


```python
from transformers import pipeline

pipe = pipeline("text-classification")

Sylvain Gugger's avatar
Sylvain Gugger committed
86

87
88
89
90
91
92
93
94
95
def data():
    while True:
        # This could come from a dataset, a database, a queue or HTTP request
        # in a server
        # Caveat: because this is iterative, you cannot use `num_workers > 1` variable
        # to use multiple threads to preprocess data. You can still have 1 thread that
        # does the preprocessing while the main runs the big inference
        yield "This is a test"

Sylvain Gugger's avatar
Sylvain Gugger committed
96

97
98
99
100
101
102
103
for out in pipe(data()):
    print(out)
    # {"text": "NUMBER TEN FRESH NELLY IS WAITING ON YOU GOOD NIGHT HUSBAND"}
    # {"text": ....}
    # ....
```

Sylvain Gugger's avatar
Sylvain Gugger committed
104
105
106
107
[[autodoc]] pipeline

## Pipeline batching

108
109
All pipelines can use batching. This will work
whenever the pipeline uses its streaming ability (so when passing lists or `Dataset` or `generator`).
Sylvain Gugger's avatar
Sylvain Gugger committed
110
111

```python
Sylvain Gugger's avatar
Sylvain Gugger committed
112
from transformers import pipeline
113
from transformers.pipelines.pt_utils import KeyDataset
Sylvain Gugger's avatar
Sylvain Gugger committed
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
import datasets

dataset = datasets.load_dataset("imdb", name="plain_text", split="unsupervised")
pipe = pipeline("text-classification", device=0)
for out in pipe(KeyDataset(dataset, "text"), batch_size=8, truncation="only_first"):
    print(out)
    # [{'label': 'POSITIVE', 'score': 0.9998743534088135}]
    # Exactly the same output as before, but the content are passed
    # as batches to the model
```

<Tip warning={true}>

However, this is not automatically a win for performance. It can be either a 10x speedup or 5x slowdown depending
on hardware, data and the actual model being used.

130
Example where it's mostly a speedup:
Sylvain Gugger's avatar
Sylvain Gugger committed
131
132
133
134

</Tip>

```python
Sylvain Gugger's avatar
Sylvain Gugger committed
135
136
from transformers import pipeline
from torch.utils.data import Dataset
137
from tqdm.auto import tqdm
Sylvain Gugger's avatar
Sylvain Gugger committed
138

Sylvain Gugger's avatar
Sylvain Gugger committed
139
pipe = pipeline("text-classification", device=0)
Sylvain Gugger's avatar
Sylvain Gugger committed
140
141


Sylvain Gugger's avatar
Sylvain Gugger committed
142
143
144
class MyDataset(Dataset):
    def __len__(self):
        return 5000
Sylvain Gugger's avatar
Sylvain Gugger committed
145

Sylvain Gugger's avatar
Sylvain Gugger committed
146
147
    def __getitem__(self, i):
        return "This is a test"
Sylvain Gugger's avatar
Sylvain Gugger committed
148
149


Sylvain Gugger's avatar
Sylvain Gugger committed
150
dataset = MyDataset()
Sylvain Gugger's avatar
Sylvain Gugger committed
151
152

for batch_size in [1, 8, 64, 256]:
Sylvain Gugger's avatar
Sylvain Gugger committed
153
154
    print("-" * 30)
    print(f"Streaming batch_size={batch_size}")
155
    for out in tqdm(pipe(dataset, batch_size=batch_size), total=len(dataset)):
Sylvain Gugger's avatar
Sylvain Gugger committed
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
        pass
```

```
# On GTX 970
------------------------------
Streaming no batching
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 5000/5000 [00:26<00:00, 187.52it/s]
------------------------------
Streaming batch_size=8
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻坾 5000/5000 [00:04<00:00, 1205.95it/s]
------------------------------
Streaming batch_size=64
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻坾 5000/5000 [00:02<00:00, 2478.24it/s]
------------------------------
Streaming batch_size=256
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻坾 5000/5000 [00:01<00:00, 2554.43it/s]
(diminishing returns, saturated the GPU)
```

Example where it's most a slowdown:

```python
Sylvain Gugger's avatar
Sylvain Gugger committed
179
180
181
182
183
184
185
186
187
class MyDataset(Dataset):
    def __len__(self):
        return 5000

    def __getitem__(self, i):
        if i % 64 == 0:
            n = 100
        else:
            n = 1
Sylvain Gugger's avatar
Sylvain Gugger committed
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
        return "This is a test" * n
```

This is a occasional very long sentence compared to the other. In that case, the **whole** batch will need to be 400
tokens long, so the whole batch will be [64, 400] instead of [64, 4], leading to the high slowdown. Even worse, on
bigger batches, the program simply crashes.


```
------------------------------
Streaming no batching
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻坾 1000/1000 [00:05<00:00, 183.69it/s]
------------------------------
Streaming batch_size=8
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻坾 1000/1000 [00:03<00:00, 265.74it/s]
------------------------------
Streaming batch_size=64
100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 1000/1000 [00:26<00:00, 37.80it/s]
------------------------------
Streaming batch_size=256
  0%|                                                                                 | 0/1000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/nicolas/src/transformers/test.py", line 42, in <module>
211
    for out in tqdm(pipe(dataset, batch_size=256), total=len(dataset)):
Sylvain Gugger's avatar
Sylvain Gugger committed
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
....
    q = q / math.sqrt(dim_per_head)  # (bs, n_heads, q_length, dim_per_head)
RuntimeError: CUDA out of memory. Tried to allocate 376.00 MiB (GPU 0; 3.95 GiB total capacity; 1.72 GiB already allocated; 354.88 MiB free; 2.46 GiB reserved in total by PyTorch)
```

There are no good (general) solutions for this problem, and your mileage may vary depending on your use cases. Rule of
thumb:

For users, a rule of thumb is:

- **Measure performance on your load, with your hardware. Measure, measure, and keep measuring. Real numbers are the
  only way to go.**
- If you are latency constrained (live product doing inference), don't batch
- If you are using CPU, don't batch.
- If you are using throughput (you want to run your model on a bunch of static data), on GPU, then:

  - If you have no clue about the size of the sequence_length ("natural" data), by default don't batch, measure and
    try tentatively to add it, add OOM checks to recover when it will fail (and it will at some point if you don't
    control the sequence_length.)
  - If your sequence_length is super regular, then batching is more likely to be VERY interesting, measure and push
    it until you get OOMs.
  - The larger the GPU the more likely batching is going to be more interesting
- As soon as you enable batching, make sure you can handle OOMs nicely.

236
237
238
## Pipeline chunk batching

`zero-shot-classification` and `question-answering` are slightly specific in the sense, that a single input might yield
Kamal Raj's avatar
Kamal Raj committed
239
multiple forward pass of a model. Under normal circumstances, this would yield issues with `batch_size` argument.
240
241
242
243
244
245
246
247

In order to circumvent this issue, both of these pipelines are a bit specific, they are `ChunkPipeline` instead of
regular `Pipeline`. In short:


```python
preprocessed = pipe.preprocess(inputs)
model_outputs = pipe.forward(preprocessed)
Kamal Raj's avatar
Kamal Raj committed
248
outputs = pipe.postprocess(model_outputs)
249
250
251
252
253
254
255
256
257
258
```

Now becomes:


```python
all_model_outputs = []
for preprocessed in pipe.preprocess(inputs):
    model_outputs = pipe.forward(preprocessed)
    all_model_outputs.append(model_outputs)
Kamal Raj's avatar
Kamal Raj committed
259
outputs = pipe.postprocess(all_model_outputs)
260
261
262
263
264
265
266
```

This should be very transparent to your code because the pipelines are used in
the same way.

This is a simplified view, since the pipeline can handle automatically the batch to ! Meaning you don't have to care
about how many forward passes you inputs are actually going to trigger, you can optimize the `batch_size`
Kamal Raj's avatar
Kamal Raj committed
267
independently of the inputs. The caveats from the previous section still apply.
268

Sylvain Gugger's avatar
Sylvain Gugger committed
269
270
271
272
273
274
275
276
277
278
279
280
281
282
## Pipeline custom code

If you want to override a specific pipeline.

Don't hesitate to create an issue for your task at hand, the goal of the pipeline is to be easy to use and support most
cases, so `transformers` could maybe support your use case.


If you want to try simply you can:

- Subclass your pipeline of choice

```python
class MyPipeline(TextClassificationPipeline):
Sylvain Gugger's avatar
Sylvain Gugger committed
283
284
    def postprocess():
        # Your code goes here
Sylvain Gugger's avatar
Sylvain Gugger committed
285
        scores = scores * 100
Sylvain Gugger's avatar
Sylvain Gugger committed
286
287
        # And here

Sylvain Gugger's avatar
Sylvain Gugger committed
288
289
290
291
292
293
294
295
296
297
298
299
300

my_pipeline = MyPipeline(model=model, tokenizer=tokenizer, ...)
# or if you use *pipeline* function, then:
my_pipeline = pipeline(model="xxxx", pipeline_class=MyPipeline)
```

That should enable you to do all the custom code you want.


## Implementing a pipeline

[Implementing a new pipeline](../add_new_pipeline)

301
## Audio
Sylvain Gugger's avatar
Sylvain Gugger committed
302

303
Pipelines available for audio tasks include the following.
Sylvain Gugger's avatar
Sylvain Gugger committed
304
305
306
307
308
309
310
311
312
313
314
315
316

### AudioClassificationPipeline

[[autodoc]] AudioClassificationPipeline
    - __call__
    - all

### AutomaticSpeechRecognitionPipeline

[[autodoc]] AutomaticSpeechRecognitionPipeline
    - __call__
    - all

317
## Computer vision
Sylvain Gugger's avatar
Sylvain Gugger committed
318

319
Pipelines available for computer vision tasks include the following.
Sylvain Gugger's avatar
Sylvain Gugger committed
320

321
322
### DepthEstimationPipeline
[[autodoc]] DepthEstimationPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
323
324
325
    - __call__
    - all

326
327
328
### ImageClassificationPipeline

[[autodoc]] ImageClassificationPipeline
329
    - __call__
330
    - all
331

332
### ImageSegmentationPipeline
333

334
[[autodoc]] ImageSegmentationPipeline
335
336
    - __call__
    - all
Sylvain Gugger's avatar
Sylvain Gugger committed
337

338
339
340
### ObjectDetectionPipeline

[[autodoc]] ObjectDetectionPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
341
342
343
    - __call__
    - all

344
345
346
347
348
349
### VideoClassificationPipeline

[[autodoc]] VideoClassificationPipeline
    - __call__
    - all

350
### ZeroShotImageClassificationPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
351

352
[[autodoc]] ZeroShotImageClassificationPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
353
354
355
    - __call__
    - all

356
### ZeroShotObjectDetectionPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
357

358
[[autodoc]] ZeroShotObjectDetectionPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
359
360
361
    - __call__
    - all

362
## Natural Language Processing
Sylvain Gugger's avatar
Sylvain Gugger committed
363

364
365
366
367
368
369
370
Pipelines available for natural language processing tasks include the following.

### ConversationalPipeline

[[autodoc]] Conversation

[[autodoc]] ConversationalPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
371
372
373
    - __call__
    - all

374
### FillMaskPipeline
375

376
[[autodoc]] FillMaskPipeline
377
378
379
    - __call__
    - all

Sylvain Gugger's avatar
Sylvain Gugger committed
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
### NerPipeline

[[autodoc]] NerPipeline

See [`TokenClassificationPipeline`] for all details.

### QuestionAnsweringPipeline

[[autodoc]] QuestionAnsweringPipeline
    - __call__
    - all

### SummarizationPipeline

[[autodoc]] SummarizationPipeline
    - __call__
    - all

### TableQuestionAnsweringPipeline

[[autodoc]] TableQuestionAnsweringPipeline
    - __call__

### TextClassificationPipeline

[[autodoc]] TextClassificationPipeline
    - __call__
    - all

### TextGenerationPipeline

[[autodoc]] TextGenerationPipeline
    - __call__
    - all

### Text2TextGenerationPipeline

[[autodoc]] Text2TextGenerationPipeline
    - __call__
    - all

### TokenClassificationPipeline

[[autodoc]] TokenClassificationPipeline
    - __call__
    - all

### TranslationPipeline

[[autodoc]] TranslationPipeline
    - __call__
    - all

433
### ZeroShotClassificationPipeline
434

435
[[autodoc]] ZeroShotClassificationPipeline
436
437
438
    - __call__
    - all

439
## Multimodal
Sylvain Gugger's avatar
Sylvain Gugger committed
440

441
442
443
444
445
Pipelines available for multimodal tasks include the following.

### DocumentQuestionAnsweringPipeline

[[autodoc]] DocumentQuestionAnsweringPipeline
Sylvain Gugger's avatar
Sylvain Gugger committed
446
447
448
    - __call__
    - all

449
### FeatureExtractionPipeline
450

451
[[autodoc]] FeatureExtractionPipeline
452
453
454
    - __call__
    - all

455
### ImageToTextPipeline
456

457
458
459
460
461
462
463
[[autodoc]] ImageToTextPipeline
    - __call__
    - all

### VisualQuestionAnsweringPipeline

[[autodoc]] VisualQuestionAnsweringPipeline
464
465
466
    - __call__
    - all

Sylvain Gugger's avatar
Sylvain Gugger committed
467
468
469
## Parent class: `Pipeline`

[[autodoc]] Pipeline