ctrl.rst 4.85 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
.. 
    Copyright 2020 The HuggingFace Team. All rights reserved.

    Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
    the License. You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
    an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
    specific language governing permissions and limitations under the License.

LysandreJik's avatar
LysandreJik committed
13
CTRL
Sylvain Gugger's avatar
Sylvain Gugger committed
14
-----------------------------------------------------------------------------------------------------------------------
LysandreJik's avatar
LysandreJik committed
15

Sylvain Gugger's avatar
Sylvain Gugger committed
16
Overview
Sylvain Gugger's avatar
Sylvain Gugger committed
17
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sylvain Gugger's avatar
Sylvain Gugger committed
18

Sylvain Gugger's avatar
Sylvain Gugger committed
19
20
CTRL model was proposed in `CTRL: A Conditional Transformer Language Model for Controllable Generation
<https://arxiv.org/abs/1909.05858>`_ by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and
Sylvain Gugger's avatar
Sylvain Gugger committed
21
22
Richard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus
of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).
Lysandre's avatar
Lysandre committed
23

Lysandre's avatar
Lysandre committed
24
25
26
27
28
The abstract from the paper is the following:

*Large-scale language models show promising text generation capabilities, but users cannot easily control particular
aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,
trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were
Sylvain Gugger's avatar
Sylvain Gugger committed
29
30
31
32
derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while
providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the
training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data
via model-based source attribution.*
Lysandre's avatar
Lysandre committed
33
34
35
36

Tips:

- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences
Sylvain Gugger's avatar
Sylvain Gugger committed
37
38
39
40
  or links to generate coherent text. Refer to the `original implementation <https://github.com/salesforce/ctrl>`__ for
  more information.
- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
Lysandre's avatar
Lysandre committed
41
- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
Sylvain Gugger's avatar
Sylvain Gugger committed
42
43
  token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be
  observed in the `run_generation.py` example script.
Lysandre's avatar
Lysandre committed
44
- The PyTorch models can take the `past` as input, which is the previously computed key/value attention pairs. Using
Sylvain Gugger's avatar
Sylvain Gugger committed
45
46
47
  this `past` value prevents the model from re-computing pre-computed values in the context of text generation. See
  `reusing the past in generative models <../quickstart.html#using-the-past>`__ for more information on the usage of
  this argument.
Lysandre's avatar
Lysandre committed
48

49
50
This model was contributed by `keskarnitishr <https://huggingface.co/keskarnitishr>`__. The original code can be found
`here <https://github.com/salesforce/ctrl>`__.
51

Lysandre's avatar
Lysandre committed
52
53

CTRLConfig
Sylvain Gugger's avatar
Sylvain Gugger committed
54
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
55
56
57
58
59

.. autoclass:: transformers.CTRLConfig
    :members:


Lysandre's avatar
Lysandre committed
60
CTRLTokenizer
Sylvain Gugger's avatar
Sylvain Gugger committed
61
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
62
63

.. autoclass:: transformers.CTRLTokenizer
Lysandre Debut's avatar
Lysandre Debut committed
64
    :members: save_vocabulary
LysandreJik's avatar
LysandreJik committed
65
66


Lysandre's avatar
Lysandre committed
67
CTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
68
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
69
70

.. autoclass:: transformers.CTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
71
    :members: forward
LysandreJik's avatar
LysandreJik committed
72
73


Lysandre's avatar
Lysandre committed
74
CTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
75
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
76
77

.. autoclass:: transformers.CTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
78
    :members: forward
LysandreJik's avatar
LysandreJik committed
79
80


81
82
83
84
85
86
87
CTRLForSequenceClassification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.CTRLForSequenceClassification
    :members: forward


Lysandre's avatar
Lysandre committed
88
TFCTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
89
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
90
91

.. autoclass:: transformers.TFCTRLModel
Sylvain Gugger's avatar
Sylvain Gugger committed
92
    :members: call
LysandreJik's avatar
LysandreJik committed
93
94


Lysandre's avatar
Lysandre committed
95
TFCTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
96
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LysandreJik's avatar
LysandreJik committed
97
98

.. autoclass:: transformers.TFCTRLLMHeadModel
Sylvain Gugger's avatar
Sylvain Gugger committed
99
    :members: call
LysandreJik's avatar
LysandreJik committed
100

101
102
103
104
105
TFCTRLForSequenceClassification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autoclass:: transformers.TFCTRLForSequenceClassification
    :members: call