ctrl.rst 4.54 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
.. 
    Copyright 2020 The HuggingFace Team. All rights reserved.

    Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
    an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
    specific language governing permissions and limitations under the License.

LysandreJik's avatar
LysandreJik committed
13
CTRL
Sylvain Gugger's avatar
Sylvain Gugger committed
14
-----------------------------------------------------------------------------------------------------------------------
LysandreJik's avatar
LysandreJik committed
15

Sylvain Gugger's avatar
Sylvain Gugger committed
16
Overview
Sylvain Gugger's avatar
Sylvain Gugger committed
17
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
18

Sylvain Gugger's avatar
Sylvain Gugger committed
19
20
CTRL model was proposed in `CTRL: A Conditional Transformer Language Model for Controllable Generation
<https://arxiv.org/abs/1909.05858>`_ by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and
Sylvain Gugger's avatar
Sylvain Gugger committed
21
22
Richard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus
of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).
Lysandre's avatar
Lysandre committed
23

Lysandre's avatar
Lysandre committed
24
25
26
27
28
The abstract from the paper is the following:

*Large-scale language models show promising text generation capabilities, but users cannot easily control particular
aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,
trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were
Sylvain Gugger's avatar
Sylvain Gugger committed
29
30
31
32
derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while
providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the
training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data
via model-based source attribution.*
Lysandre's avatar
Lysandre committed
33
34
35
36

Tips:

- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences
Sylvain Gugger's avatar
Sylvain Gugger committed
37
38
39
40
  or links to generate coherent text. Refer to the `original implementation <https://github.com/salesforce/ctrl>`__ for
  more information.
- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
Lysandre's avatar
Lysandre committed
41
- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
Sylvain Gugger's avatar
Sylvain Gugger committed
42
43
  token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be
  observed in the `run_generation.py` example script.
Lysandre's avatar
Lysandre committed
44
- The PyTorch models can take the `past` as input, which is the previously computed key/value attention pairs. Using
Sylvain Gugger's avatar
Sylvain Gugger committed
45
46
47
  this `past` value prevents the model from re-computing pre-computed values in the context of text generation. See
  `reusing the past in generative models <../quickstart.html#using-the-past>`__ for more information on the usage of
  this argument.
Lysandre's avatar
Lysandre committed
48

Sylvain Gugger's avatar
Sylvain Gugger committed
49
The original code can be found `here <https://github.com/salesforce/ctrl>`__.
50

Lysandre's avatar
Lysandre committed
51
52

CTRLConfig
Sylvain Gugger's avatar
Sylvain Gugger committed
53
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
54
55
56
57
58

.. autoclass:: transformers.CTRLConfig
    :members:


Lysandre's avatar
Lysandre committed
59
CTRLTokenizer
Sylvain Gugger's avatar
Sylvain Gugger committed
60
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
61
62

.. autoclass:: transformers.CTRLTokenizer
Lysandre Debut's avatar
Lysandre Debut committed
63
    :members: save_vocabulary
LysandreJik's avatar
LysandreJik committed
64
65


Lysandre's avatar
Lysandre committed
66
CTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
67
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
68
69

.. autoclass:: transformers.CTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
70
    :members: forward
LysandreJik's avatar
LysandreJik committed
71
72


Lysandre's avatar
Lysandre committed
73
CTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
74
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
75
76

.. autoclass:: transformers.CTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
77
    :members: forward
LysandreJik's avatar
LysandreJik committed
78
79


80
81
82
83
84
85
86
CTRLForSequenceClassification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.CTRLForSequenceClassification
    :members: forward


Lysandre's avatar
Lysandre committed
87
TFCTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
88
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
89
90

.. autoclass:: transformers.TFCTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
91
    :members: call
LysandreJik's avatar
LysandreJik committed
92
93


Lysandre's avatar
Lysandre committed
94
TFCTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
95
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
96
97

.. autoclass:: transformers.TFCTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
98
    :members: call
LysandreJik's avatar
LysandreJik committed
99