"tools/vscode:/vscode.git/clone" did not exist on "8d32c6b88793273c60312e6d3dab251f0ea31e0b"
README.md 3.87 KB
Newer Older
thomwolf's avatar
thomwolf committed
1
2
3
4
5
6
7
8
9
# How to add a new model in 馃Transformers

This folder describes the process to add a new model in 馃Transformers and provide templates for the required files.

The library is designed to incorporate a variety of models and code bases. As such the process for adding a new model usually mostly consists in copy-pasting to relevant original code in the various sections of the templates included in the present repository.

One important point though is that the library has the following goals impacting the way models are incorporated:

- one specific feature of the API is the capability to run the model and tokenizer inline. The tokenization code thus often have to be slightly adapted to allow for running in the python interpreter.
Julien Chaumond's avatar
Julien Chaumond committed
10
- the package is also designed to be as self-consistent and with a small and reliable set of packages dependencies. In consequence, additional dependencies are usually not allowed when adding a model but can be allowed for the inclusion of a new tokenizer (recent examples of dependencies added for tokenizer specificities include `sentencepiece` and `sacremoses`). Please make sure to check the existing dependencies when possible before adding a new one.
thomwolf's avatar
thomwolf committed
11
12
13
14
15
16
17
18
19
20

For a quick overview of the library organization, please check the [QuickStart section of the documentation](https://huggingface.co/transformers/quickstart.html).

# Typical workflow for including a model

Here an overview of the general workflow: 

- [ ] add model/configuration/tokenization classes
- [ ] add conversion scripts
- [ ] add tests
21
- [ ] add @slow integration test
thomwolf's avatar
thomwolf committed
22
23
- [ ] finalize

Julien Chaumond's avatar
Julien Chaumond committed
24
Let's detail what should be done at each step
thomwolf's avatar
thomwolf committed
25
26
27
28
29
30
31

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [ ] copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model name,
- [ ] edit the files to replace `XXX` (with various casing) with your model name
Julien Chaumond's avatar
Julien Chaumond committed
32
33
34
- [ ] copy-paste or create a simple configuration class for your model in the `configuration_...` file
- [ ] copy-paste or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0)
- [ ] copy-paste or create a tokenizer class for your model in the `tokenization_...` file
thomwolf's avatar
thomwolf committed
35
36
37
38
39
40

# Adding conversion scripts

Here is the workflow for the conversion scripts:

- [ ] copy the conversion script (`convert_...`) from the present folder to the main folder.
Julien Chaumond's avatar
Julien Chaumond committed
41
- [ ] edit this script to convert your original checkpoint weights to the current pytorch ones.
thomwolf's avatar
thomwolf committed
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

# Adding tests:

Here is the workflow for the adding tests:

- [ ] copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main folder and rename them, replacing `xxx` with your model name,
- [ ] edit the tests files to replace `XXX` (with various casing) with your model name
- [ ] edit the tests code as needed

# Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [ ] add import for all the relevant classes in `__init__.py`
- [ ] add your configuration in `configuration_auto.py`
- [ ] add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`
- [ ] add your tokenizer in `tokenization_auto.py`
- [ ] add your models and tokenizer to `pipeline.py`
Julien Chaumond's avatar
Julien Chaumond committed
60
- [ ] add a link to your conversion script in the main conversion utility (in `commands/convert.py`)
thomwolf's avatar
thomwolf committed
61
- [ ] edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py` file
Julien Chaumond's avatar
Julien Chaumond committed
62
- [ ] add a mention of your model in the doc: `README.md` and the documentation itself at `docs/source/pretrained_models.rst`.
Julien Chaumond's avatar
Julien Chaumond committed
63
- [ ] upload the pretrained weights, configurations and vocabulary files.
64
- [ ] create model card(s) for your models on huggingface.co. For those last two steps, check the [model sharing documentation](https://github.com/huggingface/transformers#quick-tour-of-model-sharing).