sequence_classification.md 14.3 KB
Newer Older
Steven Liu's avatar
Steven Liu committed
1
2
3
4
5
6
7
8
9
10
<!--Copyright 2022 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
11
12
13
14

鈿狅笍 Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

Steven Liu's avatar
Steven Liu committed
15
16
17
18
-->

# Text classification

19
20
[[open-in-colab]]

Steven Liu's avatar
Steven Liu committed
21
22
<Youtube id="leNG9fN9FQU"/>

amyeroberts's avatar
amyeroberts committed
23
Text classification is a common NLP task that assigns a label or class to text. Some of the largest companies run text classification in production for a wide range of practical applications. One of the most popular forms of text classification is sentiment analysis, which assigns a label like 馃檪 positive, 馃檨 negative, or 馃槓 neutral to a sequence of text.
24
25

This guide will show you how to:
Steven Liu's avatar
Steven Liu committed
26

27
1. Finetune [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on the [IMDb](https://huggingface.co/datasets/imdb) dataset to determine whether a movie review is positive or negative.
28
2. Use your finetuned model for inference.
Steven Liu's avatar
Steven Liu committed
29
30
31

<Tip>

32
To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/text-classification).
Steven Liu's avatar
Steven Liu committed
33
34
35

</Tip>

36
37
38
Before you begin, make sure you have all the necessary libraries installed:

```bash
39
pip install transformers datasets evaluate accelerate
40
41
42
43
44
45
46
47
48
49
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Steven Liu's avatar
Steven Liu committed
50
51
## Load IMDb dataset

52
Start by loading the IMDb dataset from the 馃 Datasets library:
Steven Liu's avatar
Steven Liu committed
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

```py
>>> from datasets import load_dataset

>>> imdb = load_dataset("imdb")
```

Then take a look at an example:

```py
>>> imdb["test"][0]
{
    "label": 0,
    "text": "I love sci-fi and am willing to put up with a lot. Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi' setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV. It's not. It's clich茅d and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It's really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it's rubbish as they have to always say \"Gene Roddenberry's Earth...\" otherwise people would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.",
}
```

amyeroberts's avatar
amyeroberts committed
70
There are two fields in this dataset:
Steven Liu's avatar
Steven Liu committed
71

72
73
- `text`: the movie review text.
- `label`: a value that is either `0` for a negative review or `1` for a positive review.
Steven Liu's avatar
Steven Liu committed
74
75
76

## Preprocess

77
The next step is to load a DistilBERT tokenizer to preprocess the `text` field:
Steven Liu's avatar
Steven Liu committed
78
79
80
81

```py
>>> from transformers import AutoTokenizer

82
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
Steven Liu's avatar
Steven Liu committed
83
84
85
86
87
88
89
90
91
```

Create a preprocessing function to tokenize `text` and truncate sequences to be no longer than DistilBERT's maximum input length:

```py
>>> def preprocess_function(examples):
...     return tokenizer(examples["text"], truncation=True)
```

92
To apply the preprocessing function over the entire dataset, use 馃 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once:
Steven Liu's avatar
Steven Liu committed
93
94
95
96
97

```py
tokenized_imdb = imdb.map(preprocess_function, batched=True)
```

98
Now create a batch of examples using [`DataCollatorWithPadding`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.
Steven Liu's avatar
Steven Liu committed
99

Sylvain Gugger's avatar
Sylvain Gugger committed
100
101
<frameworkcontent>
<pt>
Steven Liu's avatar
Steven Liu committed
102
103
104
105
```py
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
Sylvain Gugger's avatar
Sylvain Gugger committed
106
107
108
109
```
</pt>
<tf>
```py
Steven Liu's avatar
Steven Liu committed
110
111
112
113
>>> from transformers import DataCollatorWithPadding

>>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```
Sylvain Gugger's avatar
Sylvain Gugger committed
114
115
</tf>
</frameworkcontent>
Steven Liu's avatar
Steven Liu committed
116

117
## Evaluate
Steven Liu's avatar
Steven Liu committed
118

119
Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 馃 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 馃 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):
Steven Liu's avatar
Steven Liu committed
120
121

```py
122
>>> import evaluate
Steven Liu's avatar
Steven Liu committed
123

124
>>> accuracy = evaluate.load("accuracy")
Steven Liu's avatar
Steven Liu committed
125
126
```

127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

Before you start training your model, create a map of the expected ids to their labels with `id2label` and `label2id`:

```py
>>> id2label = {0: "NEGATIVE", 1: "POSITIVE"}
>>> label2id = {"NEGATIVE": 0, "POSITIVE": 1}
```

<frameworkcontent>
<pt>
Steven Liu's avatar
Steven Liu committed
152
153
<Tip>

154
If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!
Steven Liu's avatar
Steven Liu committed
155
156

</Tip>
157

158
159
160
161
162
163
You're ready to start training your model now! Load DistilBERT with [`AutoModelForSequenceClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

>>> model = AutoModelForSequenceClassification.from_pretrained(
164
...     "distilbert/distilbert-base-uncased", num_labels=2, id2label=id2label, label2id=label2id
165
166
... )
```
Steven Liu's avatar
Steven Liu committed
167
168
169

At this point, only three steps remain:

170
171
172
1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.
Steven Liu's avatar
Steven Liu committed
173
174
175

```py
>>> training_args = TrainingArguments(
176
...     output_dir="my_awesome_model",
Steven Liu's avatar
Steven Liu committed
177
178
179
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
180
...     num_train_epochs=2,
Steven Liu's avatar
Steven Liu committed
181
...     weight_decay=0.01,
182
...     eval_strategy="epoch",
183
184
185
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     push_to_hub=True,
Steven Liu's avatar
Steven Liu committed
186
187
188
189
190
191
192
193
194
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_imdb["train"],
...     eval_dataset=tokenized_imdb["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
195
...     compute_metrics=compute_metrics,
Steven Liu's avatar
Steven Liu committed
196
197
198
199
200
201
202
... )

>>> trainer.train()
```

<Tip>

203
[`Trainer`] applies dynamic padding by default when you pass `tokenizer` to it. In this case, you don't need to specify a data collator explicitly.
Steven Liu's avatar
Steven Liu committed
204
205

</Tip>
206
207
208
209
210
211

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
212
213
</pt>
<tf>
214
<Tip>
Matt's avatar
Matt committed
215

216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer
>>> import tensorflow as tf

>>> batch_size = 16
>>> num_epochs = 5
>>> batches_per_epoch = len(tokenized_imdb["train"]) // batch_size
>>> total_train_steps = int(batches_per_epoch * num_epochs)
>>> optimizer, schedule = create_optimizer(init_lr=2e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load DistilBERT with [`TFAutoModelForSequenceClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained(
238
...     "distilbert/distilbert-base-uncased", num_labels=2, id2label=id2label, label2id=label2id
239
240
241
242
... )
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:
Steven Liu's avatar
Steven Liu committed
243
244

```py
Matt's avatar
Matt committed
245
246
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_imdb["train"],
Steven Liu's avatar
Steven Liu committed
247
248
249
250
251
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

Matt's avatar
Matt committed
252
253
>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_imdb["test"],
Steven Liu's avatar
Steven Liu committed
254
255
256
257
258
259
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

260
Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:
261

262
263
```py
>>> import tensorflow as tf
264

265
>>> model.compile(optimizer=optimizer)  # No loss argument!
266
267
```

amyeroberts's avatar
amyeroberts committed
268
The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).
269

270
Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:
Steven Liu's avatar
Steven Liu committed
271
272

```py
273
>>> from transformers.keras_callbacks import KerasMetricCallback
Steven Liu's avatar
Steven Liu committed
274

275
>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
Steven Liu's avatar
Steven Liu committed
276
277
```

278
Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:
Steven Liu's avatar
Steven Liu committed
279
280

```py
281
>>> from transformers.keras_callbacks import PushToHubCallback
Steven Liu's avatar
Steven Liu committed
282

283
284
285
286
>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
Steven Liu's avatar
Steven Liu committed
287
288
```

289
Then bundle your callbacks together:
Steven Liu's avatar
Steven Liu committed
290
291

```py
292
>>> callbacks = [metric_callback, push_to_hub_callback]
Steven Liu's avatar
Steven Liu committed
293
294
```

295
Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:
Steven Liu's avatar
Steven Liu committed
296
297

```py
298
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=callbacks)
Steven Liu's avatar
Steven Liu committed
299
```
300
301

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
302
303
</tf>
</frameworkcontent>
Steven Liu's avatar
Steven Liu committed
304
305
306

<Tip>

307
For a more in-depth example of how to finetune a model for text classification, take a look at the corresponding
308
309
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/text_classification-tf.ipynb).
Steven Liu's avatar
Steven Liu committed
310

311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Grab some text you'd like to run inference on:

```py
>>> text = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for sentiment analysis with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis", model="stevhliu/my_awesome_model")
>>> classifier(text)
[{'label': 'POSITIVE', 'score': 0.9994940757751465}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_model")
>>> inputs = tokenizer(text, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("stevhliu/my_awesome_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_class_id = logits.argmax().item()
>>> model.config.id2label[predicted_class_id]
'POSITIVE'
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("stevhliu/my_awesome_model")
>>> inputs = tokenizer(text, return_tensors="tf")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained("stevhliu/my_awesome_model")
>>> logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a text label:

```py
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
>>> model.config.id2label[predicted_class_id]
'POSITIVE'
```
</tf>
391
</frameworkcontent>