Unverified Commit fa8ee8e8 authored by Patrick von Platen's avatar Patrick von Platen Committed by GitHub
Browse files

fix torchscript docs (#6740)

parent 64c7c2bc
...@@ -130,13 +130,12 @@ Pytorch's two modules `JIT and TRACE <https://pytorch.org/docs/stable/jit.html>` ...@@ -130,13 +130,12 @@ Pytorch's two modules `JIT and TRACE <https://pytorch.org/docs/stable/jit.html>`
their model to be re-used in other programs, such as efficiency-oriented C++ programs. their model to be re-used in other programs, such as efficiency-oriented C++ programs.
We have provided an interface that allows the export of 🤗 Transformers models to TorchScript so that they can We have provided an interface that allows the export of 🤗 Transformers models to TorchScript so that they can
be reused in a different environment than a Pytorch-based python program. Here we explain how to use our models so that be reused in a different environment than a Pytorch-based python program. Here we explain how to export and use our models using TorchScript.
they can be exported, and what to be mindful of when using these models with TorchScript.
Exporting a model needs two things: Exporting a model requires two things:
* dummy inputs to execute a model forward pass. * a forward pass with dummy inputs.
* the model needs to be instantiated with the ``torchscript`` flag. * model instantiation with the ``torchscript`` flag.
These necessities imply several things developers should be careful about. These are detailed below. These necessities imply several things developers should be careful about. These are detailed below.
...@@ -147,8 +146,8 @@ Implications ...@@ -147,8 +146,8 @@ Implications
TorchScript flag and tied weights TorchScript flag and tied weights
------------------------------------------------ ------------------------------------------------
This flag is necessary because most of the language models in this repository have tied weights between their This flag is necessary because most of the language models in this repository have tied weights between their
``Embedding`` layer and their ``Decoding`` layer. TorchScript does not allow the export of models that have tied weights, ``Embedding`` layer and their ``Decoding`` layer. TorchScript does not allow the export of models that have tied weights, therefore
it is therefore necessary to untie the weights beforehand. it is necessary to untie and clone the weights beforehand.
This implies that models instantiated with the ``torchscript`` flag have their ``Embedding`` layer and ``Decoding`` layer This implies that models instantiated with the ``torchscript`` flag have their ``Embedding`` layer and ``Decoding`` layer
separate, which means that they should not be trained down the line. Training would de-synchronize the two layers, separate, which means that they should not be trained down the line. Training would de-synchronize the two layers,
...@@ -181,7 +180,7 @@ when exporting varying sequence-length models. ...@@ -181,7 +180,7 @@ when exporting varying sequence-length models.
Using TorchScript in Python Using TorchScript in Python
------------------------------------------------- -------------------------------------------------
Below are examples of using the Python to save, load models as well as how to use the trace for inference. Below is an example, showing how to save, load models as well as how to use the trace for inference.
Saving a model Saving a model
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...@@ -237,10 +236,10 @@ We are re-using the previously initialised ``dummy_input``. ...@@ -237,10 +236,10 @@ We are re-using the previously initialised ``dummy_input``.
.. code-block:: python .. code-block:: python
loaded_model = torch.jit.load("traced_model.pt") loaded_model = torch.jit.load("traced_bert.pt")
loaded_model.eval() loaded_model.eval()
all_encoder_layers, pooled_output = loaded_model(dummy_input) all_encoder_layers, pooled_output = loaded_model(*dummy_input)
Using a traced model for inference Using a traced model for inference
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment