Skip to content

GitLab

Unverified Commit c157c7e3 authored by Steven Liu's avatar Steven Liu Committed by GitHub
Browse files

Update fine-tune docs (#15259) 

* add fine-tune tutorial

* make edits, fix style

* 📝 make edits

* 🖍 fix code format links to external libraries

* 🔄revert code formatting

* 🖍 use DefaultDataCollator instead of DataCollatorWithPadding
parent d0b5ed11
Hide whitespace changes
Inline Side-by-side
Showing with 217 additions and 244 deletions
docs/source/training.mdx
View file @ c157c7e3
<!--Copyright 2020 The HuggingFace Team. All rights reserved. <!--Copyright 2022 The HuggingFace Team. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at the License. You may obtain a copy of the License at
...@@ -10,385 +10,358 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o ...@@ -10,385 +10,358 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o
specific language governing permissions and limitations under the License. specific language governing permissions and limitations under the License.
--> -->
# Fine-tuning a pretrained model # Fine-tune a pretrained model
[[open-in-colab]] [[open-in-colab]]
In this tutorial, we will show you how to fine-tune a pretrained model from the Transformers library. In TensorFlow, There are significant benefits to using a pretrained model. It reduces computation costs, your carbon footprint, and allows you to use state-of-the-art models without having to train one from scratch. 🤗 Transformers provides access to thousands of pretrained models for a wide range of tasks. When you use a pretrained model, you train it on a dataset specific to your task. This is known as fine-tuning, an incredibly powerful training technique. In this tutorial, you will fine-tune a pretrained model with a deep learning framework of your choice:
models can be directly trained using Keras and the `fit` method. In PyTorch, there is no generic training loop so
the 🤗 Transformers library provides an API with the class [`Trainer`] to let you fine-tune or train
a model from scratch easily. Then we will show you how to alternatively write the whole training loop in PyTorch.
Before we can fine-tune a model, we need a dataset. In this tutorial, we will show you how to fine-tune BERT on the * Fine-tune a pretrained model with 🤗 Transformers [`Trainer`].
[IMDB dataset](https://www.imdb.com/interfaces/): the task is to classify whether movie reviews are positive or * Fine-tune a pretrained model in TensorFlow with Keras.
negative. For examples of other tasks, refer to the [additional-resources](#additional-resources) section! * Fine-tune a pretrained model in native PyTorch.
<a id='data-processing'></a> <a id='data-processing'></a>
## Preparing the datasets ## Prepare a dataset
<Youtube id="_BZearw7f0w"/> <Youtube id="_BZearw7f0w"/>
We will use the [🤗 Datasets](https://github.com/huggingface/datasets/) library to download and preprocess the IMDB Before you can fine-tune a pretrained model, download a dataset and prepare it for training. The previous tutorial showed you how to process data for training, and now you get an opportunity to put those skills to the test!
datasets. We will go over this part pretty quickly. Since the focus of this tutorial is on training, you should refer
to the 🤗 Datasets [documentation](https://huggingface.co/docs/datasets/) or the [preprocessing](preprocessing) tutorial for
more information.
First, we can use the `load_dataset` function to download and cache the dataset: Begin by loading the [Yelp Reviews](https://huggingface.co/datasets/yelp_review_full) dataset:
```python ```py
from datasets import load_dataset >>> from datasets import load_dataset
raw_datasets = load_dataset("imdb") >>> dataset = load_dataset("yelp_review_full")
>>> dataset[100]
{'label': 0,
'text': 'My expectations for McDonalds are t rarely high. But for one to still fail so spectacularly...that takes something special!\\nThe cashier took my friends\'s order, then promptly ignored me. I had to force myself in front of a cashier who opened his register to wait on the person BEHIND me. I waited over five minutes for a gigantic order that included precisely one kid\'s meal. After watching two people who ordered after me be handed their food, I asked where mine was. The manager started yelling at the cashiers for \\"serving off their orders\\" when they didn\'t have their food. But neither cashier was anywhere near those controls, and the manager was the one serving food to customers and clearing the boards.\\nThe manager was rude when giving me my order. She didn\'t make sure that I had everything ON MY RECEIPT, and never even had the decency to apologize that I felt I was getting poor service.\\nI\'ve eaten at various McDonalds restaurants for over 30 years. I\'ve worked at more than one location. I expect bad days, bad moods, and the occasional mistake. But I have yet to have a decent experience at this store. It will remain a place I avoid unless someone in my party needs to avoid illness from low blood sugar. Perhaps I should go back to the racially biased service of Steak n Shake instead!'}
``` ```
This works like the `from_pretrained` method we saw for the models and tokenizers (except the cache directory is As you now know, you need a tokenizer to process the text and include a padding and truncation strategy to handle any variable sequence lengths. To process your dataset in one step, use 🤗 Datasets [`map`](https://huggingface.co/docs/datasets/process.html#map) method to apply a preprocessing function over the entire dataset:
_~/.cache/huggingface/dataset_ by default).
The `raw_datasets` object is a dictionary with three keys: `"train"`, `"test"` and `"unsupervised"` ```py
(which correspond to the three splits of that dataset). We will use the `"train"` split for training and the >>> from transformers import AutoTokenizer
`"test"` split for validation.
To preprocess our data, we will need a tokenizer: >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased") >>> def tokenize_function(examples):
``` ... return tokenizer(examples["text"], padding="max_length", truncation=True)
As we saw in [preprocessing](preprocessing), we can prepare the text inputs for the model with the following command (this is an
example, not a command you can execute):
```python >>> tokenized_datasets = dataset.map(tokenize_function, batched=True)
inputs = tokenizer(sentences, padding="max_length", truncation=True)
``` ```
This will make all the samples have the maximum length the model can accept (here 512), either by padding or truncating If you like, you can create a smaller subset of the full dataset to fine-tune on to reduce the time it takes:
them.
However, we can instead apply these preprocessing steps to all the splits of our dataset at once by using the
`map` method:
```python
def tokenize_function(examples):
return tokenizer(examples["text"], padding="max_length", truncation=True)
```py
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True) >>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
``` ```
You can learn more about the map method or the other ways to preprocess the data in the 🤗 Datasets [documentation](https://huggingface.co/docs/datasets/).
Next we will generate a small subset of the training and validation set, to enable faster training:
```python
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
full_train_dataset = tokenized_datasets["train"]
full_eval_dataset = tokenized_datasets["test"]
```
In all the examples below, we will always use `small_train_dataset` and `small_eval_dataset`. Just replace
them by their _full_ equivalent to train or evaluate on the full dataset.
<a id='trainer'></a> <a id='trainer'></a>
## Fine-tuning in PyTorch with the Trainer API ## Fine-tune with `Trainer`
<Youtube id="nvBXf7s7vTI"/> <Youtube id="nvBXf7s7vTI"/>
Since PyTorch does not provide a training loop, the 🤗 Transformers library provides a [`Trainer`] 🤗 Transformers provides a [`Trainer`] class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision.
API that is optimized for 🤗 Transformers models, with a wide range of training options and with built-in features like
logging, gradient accumulation, and mixed precision.
First, let's define our model: Start by loading your model and specify the number of expected labels. From the Yelp Review [dataset card](https://huggingface.co/datasets/yelp_review_full#data-fields), you know there are five labels:
```python ```py
from transformers import AutoModelForSequenceClassification >>> from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2) >>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)
``` ```
This will issue a warning about some of the pretrained weights not being used and some weights being randomly <Tip>
initialized. That's because we are throwing away the pretraining head of the BERT model to replace it with a
classification head which is randomly initialized. We will fine-tune this model on our task, transferring the knowledge
of the pretrained model to it (which is why doing this is called transfer learning).
Then, to define our [`Trainer`], we will need to instantiate a You will see a warning about some of the pretrained weights not being used and some weights being randomly
[`TrainingArguments`]. This class contains all the hyperparameters we can tune for the initialized. Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.
[`Trainer`] or the flags to activate the different training options it supports. Let's begin by
using all the defaults, the only thing we then have to provide is a directory in which the checkpoints will be saved:
```python </Tip>
from transformers import TrainingArguments
training_args = TrainingArguments("test_trainer") ### Training hyperparameters
```
Then we can instantiate a [`Trainer`] like this:
```python Next, create a [`TrainingArguments`] class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.
from transformers import Trainer
trainer = Trainer(model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset) Specify where to save the checkpoints from your training:
```
To fine-tune our model, we just need to call ```py
>>> from transformers import TrainingArguments
```python >>> training_args = TrainingArguments(output_dir="test_trainer")
trainer.train()
``` ```
which will start a training that you can follow with a progress bar, which should take a couple of minutes to complete ### Metrics
(as long as you have access to a GPU). It won't actually tell you anything useful about how well (or badly) your model
is performing however as by default, there is no evaluation during training, and we didn't tell the
[`Trainer`] to compute any metrics. Let's have a look on how to do that now!
To have the [`Trainer`] compute and report metrics, we need to give it a `compute_metrics` [`Trainer`] does not automatically evaluate model performance during training. You will need to pass [`Trainer`] a function to compute and report metrics. The 🤗 Datasets library provides a simple [`accuracy`](https://huggingface.co/metrics/accuracy) function you can load with the `load_metric` (see this [tutorial](https://huggingface.co/docs/datasets/metrics.html) for more information) function:
function that takes predictions and labels (grouped in a namedtuple called [`EvalPrediction`]) and
return a dictionary with string items (the metric names) and float values (the metric values).
The 🤗 Datasets library provides an easy way to get the common metrics used in NLP with the `load_metric` function. ```py
here we simply use accuracy. Then we define the `compute_metrics` function that just convert logits to predictions >>> import numpy as np
(remember that all 🤗 Transformers models return the logits) and feed them to `compute` method of this metric. >>> from datasets import load_metric
```python >>> metric = load_metric("accuracy")
import numpy as np ```
from datasets import load_metric
metric = load_metric("accuracy")
Call `compute` on `metric` to calculate the accuracy of your predictions. Before passing your predictions to `compute`, you need to convert the predictions to logits (remember all 🤗 Transformers models return logits):
def compute_metrics(eval_pred): ```py
logits, labels = eval_pred >>> def compute_metrics(eval_pred):
predictions = np.argmax(logits, axis=-1) ... logits, labels = eval_pred
return metric.compute(predictions=predictions, references=labels) ... predictions = np.argmax(logits, axis=-1)
... return metric.compute(predictions=predictions, references=labels)
``` ```
The compute function needs to receive a tuple (with logits and labels) and has to return a dictionary with string keys If you'd like to monitor your evaluation metrics during fine-tuning, specify the `evaluation_strategy` parameter in your training arguments to report the evaluation metric at the end of each epoch:
(the name of the metric) and float values. It will be called at the end of each evaluation phase on the whole arrays of
predictions/labels. ```py
>>> from transformers import TrainingArguments
To check if this works on practice, let's create a new [`Trainer`] with our fine-tuned model:
```python
trainer = Trainer(
model=model,
args=training_args,
train_dataset=small_train_dataset,
eval_dataset=small_eval_dataset,
compute_metrics=compute_metrics,
)
trainer.evaluate()
```
which showed an accuracy of 87.5% in our case. >>> training_args = TrainingArguments(output_dir="test_trainer", evaluation_strategy="epoch")
```
If you want to fine-tune your model and regularly report the evaluation metrics (for instance at the end of each ### Trainer
epoch), here is how you should define your training arguments:
```python Create a [`Trainer`] object with your model, training arguments, training and test datasets, and evaluation function:
from transformers import TrainingArguments
training_args = TrainingArguments("test_trainer", evaluation_strategy="epoch") ```py
>>> trainer = Trainer(
... model=model,
... args=training_args,
... train_dataset=small_train_dataset,
... eval_dataset=small_eval_dataset,
... compute_metrics=compute_metrics,
... )
``` ```
See the documentation of [`TrainingArguments`] for more options. Then fine-tune your model by calling [`~transformers.Trainer.train`]:
```py
>>> trainer.train()
```
<a id='keras'></a> <a id='keras'></a>
## Fine-tuning with Keras ## Fine-tune with Keras
<Youtube id="rnTGBy2ax1c"/> <Youtube id="rnTGBy2ax1c"/>
Models can also be trained natively in TensorFlow using the Keras API. First, let's define our model: 🤗 Transformers models also supports training in TensorFlow with the Keras API. You only need to make a few changes before you can fine-tune.
```python ### Convert dataset to TensorFlow format
import tensorflow as tf
from transformers import TFAutoModelForSequenceClassification
model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2) The [`DefaultDataCollator`] assembles tensors into a batch for the model to train on. Make sure you specify `return_tensors` to return TensorFlow tensors:
```
Then we will need to convert our datasets from before in standard `tf.data.Dataset`. Since we have fixed shapes, ```py
it can easily be done like this. First we remove the _"text"_ column from our datasets and set them in TensorFlow >>> from transformers import DefaultDataCollator
format:
```python >>> data_collator = DefaultDataCollator(return_tensors="tf")
tf_train_dataset = small_train_dataset.remove_columns(["text"]).with_format("tensorflow")
tf_eval_dataset = small_eval_dataset.remove_columns(["text"]).with_format("tensorflow")
``` ```
Then we convert everything in big tensors and use the `tf.data.Dataset.from_tensor_slices` method: <Tip>
[`Trainer`] uses [`DataCollatorWithPadding`] by default so you don't need to explicitly specify a data collator.
</Tip>
```python Next, convert the tokenized datasets to TensorFlow datasets with the [`to_tf_dataset`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.to_tf_dataset) method. Specify your inputs in `columns`, and your label in `label_cols`:
train_features = {x: tf_train_dataset[x] for x in tokenizer.model_input_names}
train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset["label"]))
train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)
eval_features = {x: tf_eval_dataset[x] for x in tokenizer.model_input_names} ```py
eval_tf_dataset = tf.data.Dataset.from_tensor_slices((eval_features, tf_eval_dataset["label"])) >>> tf_train_dataset = small_train_dataset.to_tf_dataset(
eval_tf_dataset = eval_tf_dataset.batch(8) ... columns=["attention_mask", "input_ids", "token_type_ids"],
... label_cols=["labels"],
... shuffle=True,
... collate_fn=data_collator,
... batch_size=8,
... )
>>> tf_validation_dataset = small_eval_dataset.to_tf_dataset(
... columns=["attention_mask", "input_ids", "token_type_ids"],
... label_cols=["labels"],
... shuffle=False,
... collate_fn=data_collator,
... batch_size=8,
... )
``` ```
With this done, the model can then be compiled and trained as any Keras model: ### Compile and fit
Load a TensorFlow model with the expected number of labels:
```python ```py
model.compile( >>> import tensorflow as tf
optimizer=tf.keras.optimizers.Adam(learning_rate=5e-5), >>> from transformers import TFAutoModelForSequenceClassification
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=tf.metrics.SparseCategoricalAccuracy(),
)
model.fit(train_tf_dataset, validation_data=eval_tf_dataset, epochs=3) >>> model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)
``` ```
With the tight interoperability between TensorFlow and PyTorch models, you can even save the model and then reload it Then compile and fine-tune your model with [`fit`](https://keras.io/api/models/model_training_apis/) as you would with any other Keras model:
as a PyTorch model (or vice-versa):
```python ```py
from transformers import AutoModelForSequenceClassification >>> model.compile(
... optimizer=tf.keras.optimizers.Adam(learning_rate=5e-5),
... loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
... metrics=tf.metrics.SparseCategoricalAccuracy(),
... )
model.save_pretrained("my_imdb_model") >>> model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3)
pytorch_model = AutoModelForSequenceClassification.from_pretrained("my_imdb_model", from_tf=True)
``` ```
<a id='pytorch_native'></a> <a id='pytorch_native'></a>
## Fine-tuning in native PyTorch ## Fine-tune in native PyTorch
<Youtube id="Dh9CL8fyG80"/> <Youtube id="Dh9CL8fyG80"/>
You might need to restart your notebook at this stage to free some memory, or execute the following code: [`Trainer`] takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch.
```python At this point, you may need to restart your notebook or execute the following code to free some memory:
```py
del model del model
del pytorch_model del pytorch_model
del trainer del trainer
torch.cuda.empty_cache() torch.cuda.empty_cache()
``` ```
Let's now see how to achieve the same results as in [trainer section](#trainer) in PyTorch. First we need to Next, manually postprocess `tokenized_dataset` to prepare it for training.
define the dataloaders, which we will use to iterate over batches. We just need to apply a bit of post-processing to
our `tokenized_datasets` before doing that to: 1. Remove the `text` column because the model does not accept raw text as an input:
```py
>>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])
```
2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:
- remove the columns corresponding to values the model does not expect (here the `"text"` column) ```py
- rename the column `"label"` to `"labels"` (because the model expect the argument to be named `labels`) >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
- set the format of the datasets so they return PyTorch Tensors instead of lists. ```
Our _tokenized_datasets_ has one method for each of those steps: 3. Set the format of the dataset to return PyTorch tensors instead of lists:
```python ```py
tokenized_datasets = tokenized_datasets.remove_columns(["text"]) >>> tokenized_datasets.set_format("torch")
tokenized_datasets = tokenized_datasets.rename_column("label", "labels") ```
tokenized_datasets.set_format("torch")
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000)) Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
```py
>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
``` ```
Now that this is done, we can easily define our dataloaders: ### DataLoader
Create a `DataLoader` for your training and test datasets so you can iterate over batches of data:
```python ```py
from torch.utils.data import DataLoader >>> from torch.utils.data import DataLoader
train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8) >>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
eval_dataloader = DataLoader(small_eval_dataset, batch_size=8) >>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)
``` ```
Next, we define our model: Load your model with the number of expected labels:
```python ```py
from transformers import AutoModelForSequenceClassification >>> from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2) >>> model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5)
``` ```
We are almost ready to write our training loop, the only two things are missing are an optimizer and a learning rate ### Optimizer and learning rate scheduler
scheduler. The default optimizer used by the [`Trainer`] is [`AdamW`]:
```python Create an optimizer and learning rate scheduler to fine-tune the model. Let's use the [`AdamW`](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html) optimizer from PyTorch:
from transformers import AdamW
optimizer = AdamW(model.parameters(), lr=5e-5) ```py
>>> from torch.optim import AdamW
>>> optimizer = AdamW(model.parameters(), lr=5e-5)
``` ```
Finally, the learning rate scheduler used by default is just a linear decay from the maximum value (5e-5 here) to 0: Create the default learning rate scheduler from [`Trainer`]:
```python ```py
from transformers import get_scheduler >>> from transformers import get_scheduler
num_epochs = 3 >>> num_epochs = 3
num_training_steps = num_epochs * len(train_dataloader) >>> num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler("linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps) >>> lr_scheduler = get_scheduler(
... name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... )
``` ```
One last thing, we will want to use the GPU if we have access to one (otherwise training might take several hours Lastly, specify `device` to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.
instead of a couple of minutes). To do this, we define a `device` we will put our model and our batches on.
```python ```py
import torch >>> import torch
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu") >>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model.to(device) >>> model.to(device)
``` ```
We now are ready to train! To get some sense of when it will be finished, we add a progress bar over our number of <Tip>
training steps, using the _tqdm_ library.
Get free access to a cloud GPU if you don't have one with a hosted notebook like [Colaboratory](https://colab.research.google.com/) or [SageMaker StudioLab](https://studiolab.sagemaker.aws/).
</Tip>
Great, now you are ready to train! 🥳
### Training loop
To keep track of your training progress, use the [tqdm](https://tqdm.github.io/) library to add a progress bar over the number of training steps:
```python ```py
from tqdm.auto import tqdm >>> from tqdm.auto import tqdm
progress_bar = tqdm(range(num_training_steps)) >>> progress_bar = tqdm(range(num_training_steps))
model.train() >>> model.train()
for epoch in range(num_epochs): >>> for epoch in range(num_epochs):
for batch in train_dataloader: ... for batch in train_dataloader:
batch = {k: v.to(device) for k, v in batch.items()} ... batch = {k: v.to(device) for k, v in batch.items()}
outputs = model(**batch) ... outputs = model(**batch)
loss = outputs.loss ... loss = outputs.loss
loss.backward() ... loss.backward()
optimizer.step() ... optimizer.step()
lr_scheduler.step() ... lr_scheduler.step()
optimizer.zero_grad() ... optimizer.zero_grad()
progress_bar.update(1) ... progress_bar.update(1)
``` ```
Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a ### Metrics
bit strange, as we are directly fine-tuning the whole model without taking any precaution. It actually works better
this way for Transformers model (so this is not an oversight on our side). If you're not familiar with what "freezing
the body" of the model means, forget you read this paragraph.
Now to check the results, we need to write the evaluation loop. Like in the [trainer section](#trainer) we will Just like how you need to add an evaluation function to [`Trainer`], you need to do the same when you write your own training loop. But instead of calculating and reporting the metric at the end of each epoch, this time you will accumulate all the batches with [`add_batch`](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=add_batch#datasets.Metric.add_batch) and calculate the metric at the very end.
use a metric from the datasets library. Here we accumulate the predictions at each batch before computing the final
result when the loop is finished.
```python ```py
metric = load_metric("accuracy") >>> metric = load_metric("accuracy")
model.eval() >>> model.eval()
for batch in eval_dataloader: >>> for batch in eval_dataloader:
batch = {k: v.to(device) for k, v in batch.items()} ... batch = {k: v.to(device) for k, v in batch.items()}
with torch.no_grad(): ... with torch.no_grad():
outputs = model(**batch) ... outputs = model(**batch)
logits = outputs.logits ... logits = outputs.logits
predictions = torch.argmax(logits, dim=-1) ... predictions = torch.argmax(logits, dim=-1)
metric.add_batch(predictions=predictions, references=batch["labels"]) ... metric.add_batch(predictions=predictions, references=batch["labels"])
metric.compute() >>> metric.compute()
``` ```
<a id='additional-resources'></a> <a id='additional-resources'></a>
## Additional resources ## Additional resources
To look at more fine-tuning examples you can refer to: For more fine-tuning examples, refer to:
- [🤗 Transformers Examples](https://github.com/huggingface/transformers/tree/master/examples) which includes scripts - [🤗 Transformers Examples](https://github.com/huggingface/transformers/tree/master/examples) includes scripts
to train on all common NLP tasks in PyTorch and TensorFlow. to train common NLP tasks in PyTorch and TensorFlow.
- [🤗 Transformers Notebooks](notebooks) which contains various notebooks and in particular one per task (look for - [🤗 Transformers Notebooks](notebooks) contains various notebooks on how to fine-tune a model for specific tasks in PyTorch and TensorFlow.
the _how to finetune a model on xxx_).
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment