README.md 12.7 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<!---
Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

17
18
# Generating the documentation

19
To generate the documentation, you first have to build it. Several packages are necessary to build the doc,
20
you can install them with the following command, at the root of the code repository:
21
22

```bash
23
pip install -e ".[docs]"
24
```
25

26
27
28
---
**NOTE**

29
You only need to generate the documentation to inspect it locally (if you're planning changes and want to
30
31
32
33
check how they look like before committing for instance). You don't have to commit the built documentation.

---

34
35
## Packages installed

36
Here's an overview of all the packages installed. If you ran the previous command installing all packages from
37
38
`requirements.txt`, you do not need to run the following commands.

39
Building it requires the package `sphinx` that you can
40
41
42
43
44
45
install using:

```bash
pip install -U sphinx
```

46
You would also need the custom installed [theme](https://github.com/readthedocs/sphinx_rtd_theme) by
47
48
49
50
51
52
[Read The Docs](https://readthedocs.org/). You can install it using the following command:

```bash
pip install sphinx_rtd_theme
```

53
54
55
56
57
58
59
60
The third necessary package is the `recommonmark` package to accept Markdown as well as Restructured text:

```bash
pip install recommonmark
```

## Building the documentation

61
62
63
64
65
66
Once you have setup `sphinx`, you can build the documentation by running the following command in the `/docs` folder:

```bash
make html
```

67
A folder called ``_build/html`` should have been created. You can now open the file ``_build/html/index.html`` in your
68
browser.
69

70
71
72
---
**NOTE**

Dima Veselov's avatar
Dima Veselov committed
73
If you are adding/removing elements from the toc-tree or from any structural item, it is recommended to clean the build
74
75
76
77
78
79
80
81
82
83
84
85
directory before rebuilding. Run the following command to clean and build:

```bash
make clean && make html
```

---

It should build the static app that will be available under `/docs/_build/html`

## Adding a new element to the tree (toc-tree)

LysandreJik's avatar
LysandreJik committed
86
Accepted files are reStructuredText (.rst) and Markdown (.md). Create a file with its extension and put it
thomwolf's avatar
thomwolf committed
87
in the source directory. You can then link it to the toc-tree by putting the filename without the extension.
88

89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
## Renaming section headers and moving sections

It helps to keep the old links working when renaming section header and/or moving sections from one document to another. This is because the old links are likely to be used in Issues, Forums and Social media and it'd be make for a much more superior user experience if users reading those months later could still easily navigate to the originally intended information.

Therefore we simply keep a little map of moved sections at the end of the document where the original section was. The key is to preserve the original anchor.

So if you renamed a section from: "Section A" to "Section B", then you can add at the end of the file:

```
Sections that were moved:

[ <a href="#section-b">Section A</a><a id="section-a"></a> ]
```
and of course if you moved it to another file, then:

```
Sections that were moved:

[ <a href="../new-file#section-b">Section A</a><a id="section-a"></a> ]
```

Use the relative style to link to the new file so that the versioned docs continue to work.

For an example of a rich moved sections set please see the very end of [the Trainer doc](https://github.com/huggingface/transformers/blob/master/docs/source/main_classes/trainer.mdx).


115
116
117
118
119
120
121
122
123
## Preview the documentation in a pull request

Once you have made your pull request, you can check what the documentation will look like after it's merged by
following these steps:

- Look at the checks at the bottom of the conversation page of your PR (you may need to click on "show all checks" to
  expand them).
- Click on "details" next to the `ci/circleci: build_doc` check.
- In the new window, click on the "Artifacts" tab.
124
- Locate the file "docs/_build/html/index.html" (or any specific page you want to check) and click on it to get a
125
126
  preview.

127
128
129
130
## Writing Documentation - Specification

The `huggingface/transformers` documentation follows the
[Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style. It is
131
132
mostly written in ReStructuredText
([Sphinx simple documentation](https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html),
133
[Sourceforge complete documentation](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html)).
134
135


136
137
138
### Adding a new tutorial

Adding a new tutorial or section is done in two steps:
139
140
141
142

- Add a new file under `./source`. This file can either be ReStructuredText (.rst) or Markdown (.md).
- Link that file in `./source/index.rst` on the correct toc-tree.

143
144
145
146
Make sure to put your new file under the proper section. It's unlikely to go in the first section (*Get Started*), so
depending on the intended targets (beginners, more advanced users or researchers) it should go in section two, three or
four.

147
148
149
### Adding a new model

When adding a new model:
150
151

- Create a file `xxx.rst` under `./source/model_doc` (don't hesitate to copy an existing file as template).
152
153
154
155
156
157
158
- Link that file in `./source/index.rst` on the `model_doc` toc-tree.
- Write a short overview of the model:
    - Overview with paper & authors
    - Paper abstract
    - Tips and tricks and how to use it best
- Add the classes that should be linked in the model. This generally includes the configuration, the tokenizer, and
  every model of that class (the base model, alongside models with additional heads), both in PyTorch and TensorFlow.
159
160
  The order is generally:
    - Configuration,
161
162
163
164
165
166
167
168
169
    - Tokenizer
    - PyTorch base model
    - PyTorch head models
    - TensorFlow base model
    - TensorFlow head models

These classes should be added using the RST syntax. Usually as follows:
```
XXXConfig
170
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
171
172
173
174
175

.. autoclass:: transformers.XXXConfig
    :members:
```

176
177
This will include every public method of the configuration that is documented. If for some reason you wish for a method
not to be displayed in the documentation, you can do so by specifying which methods should be in the docs:
178
179
180

```
XXXTokenizer
181
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
182
183
184
185
186
187
188
189
190

.. autoclass:: transformers.XXXTokenizer
    :members: build_inputs_with_special_tokens, get_special_tokens_mask,
        create_token_type_ids_from_sequences, save_vocabulary

```

### Writing source documentation

191
Values that should be put in `code` should either be surrounded by double backticks: \`\`like so\`\` or be written as
192
193
an object using the :obj: syntax: :obj:\`like so\`. Note that argument names and objects like True, None or any strings
should usually be put in `code`.
194

195
When mentioning a class, it is recommended to use the :class: syntax as the mentioned class will be automatically
196
linked by Sphinx: :class:\`~transformers.XXXClass\`
197

198
199
200
201
202
When mentioning a function, it is recommended to use the :func: syntax as the mentioned function will be automatically
linked by Sphinx: :func:\`~transformers.function\`.

When mentioning a method, it is recommended to use the :meth: syntax as the mentioned method will be automatically
linked by Sphinx: :meth:\`~transformers.XXXClass.method\`.
203
204
205
206
207

Links should be done as so (note the double underscore at the end): \`text for the link <./local-link-or-global-link#loc>\`__

#### Defining arguments in a method

208
Arguments should be defined with the `Args:` prefix, followed by a line return and an indentation.
209
210
211
212
213
214
215
216
217
218
The argument should be followed by its type, with its shape if it is a tensor, and a line return.
Another indentation is necessary before writing the description of the argument.

Here's an example showcasing everything so far:

```
    Args:
        input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`):
            Indices of input sequence tokens in the vocabulary.

219
220
221
            Indices can be obtained using :class:`~transformers.AlbertTokenizer`.
            See :meth:`~transformers.PreTrainedTokenizer.encode` and
            :meth:`~transformers.PreTrainedTokenizer.__call__` for details.
222
223
224
225

            `What are input IDs? <../glossary.html#input-ids>`__
```

226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
For optional arguments or arguments with defaults we follow the following syntax: imagine we have a function with the
following signature:

```
def my_function(x: str = None, a: float = 1):
```

then its documentation should look like this:

```
    Args:
        x (:obj:`str`, `optional`):
            This argument controls ...
        a (:obj:`float`, `optional`, defaults to 1):
            This argument is used to ...
```

Note that we always omit the "defaults to :obj:\`None\`" when None is the default for any argument. Also note that even
if the first line describing your argument type and its default gets long, you can't break it on several lines. You can
245
however write as many lines as you want in the indented description (see the example above with `input_ids`).
246

247
#### Writing a multi-line code block
248
249
250
251
252
253
254
255
256
257
258
259
260

Multi-line code blocks can be useful for displaying examples. They are done like so:

```
Example::

    # first line of code
    # second line
    # etc
```

The `Example` string at the beginning can be replaced by anything as long as there are two semicolons following it.

261
262
263
We follow the [doctest](https://docs.python.org/3/library/doctest.html) syntax for the examples to automatically test
the results stay consistent with the library.

264
265
#### Writing a return block

266
Arguments should be defined with the `Args:` prefix, followed by a line return and an indentation.
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
The first line should be the type of the return, followed by a line return. No need to indent further for the elements
building the return.

Here's an example for tuple return, comprising several objects:

```
    Returns:
        :obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs:
        loss (`optional`, returned when ``masked_lm_labels`` is provided) ``torch.FloatTensor`` of shape ``(1,)``:
            Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss.
        prediction_scores (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, config.vocab_size)`)
            Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
```

Here's an example for a single value return:

```
    Returns:
285
        :obj:`List[int]`: A list of integers in the range [0, 1] --- 1 for a special token, 0 for a sequence token.
286
```
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326

#### Adding a new section

In ReST section headers are designated as such with the help of a line of underlying characters, e.g.,:

```
Section 1
^^^^^^^^^^^^^^^^^^

Sub-section 1
~~~~~~~~~~~~~~~~~~
```

ReST allows the use of any characters to designate different section levels, as long as they are used consistently within the same document. For details see [sections doc](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections). Because there is no standard different documents often end up using different characters for the same levels which makes it very difficult to know which character to use when creating a new section.

Specifically, if when running `make docs` you get an error like:
```
docs/source/main_classes/trainer.rst:127:Title level inconsistent:
```
you picked an inconsistent character for some of the levels.

But how do you know which characters you must use for an already existing level or when adding a new level?

You can use this helper script:
```
perl -ne '/^(.)\1{100,}/ && do { $h{$1}=++$c if !$h{$1} }; END { %h = reverse %h ; print "$_ $h{$_}\n" for sort keys %h}' docs/source/main_classes/trainer.rst
1 -
2 ~
3 ^
4 =
5 "
```

This tells you which characters have already been assigned for each level.

So using this particular example's output -- if your current section's header uses `=` as its underline character, you now know you're at level 4, and if you want to add a sub-section header you know you want `"` as it'd level 5.

If you needed to add yet another sub-level, then pick a character that is not used already. That is you must pick a character that is not in the output of that script.

Here is the full list of characters that can be used in this context: `= - ` : ' " ~ ^ _ * + # < >`