"...lm-evaluation-harness.git" did not exist on "4ad6cd9fa22309058d827ecbbcad965c2fc618ca"
optimizer_schedules.rst 1.3 KB
Newer Older
thomwolf's avatar
thomwolf committed
1
2
3
Optimizer
----------------------------------------------------

4
5
6
7
8
The ``.optimization`` module provides:

- an optimizer with weight decay fixed that can be used to fine-tuned models, and
- several schedules in the form of schedule objects that inherit from ``_LRSchedule``:

thomwolf's avatar
thomwolf committed
9
10
11
12
13
14
15
16
17
``AdamW``
~~~~~~~~~~~~~~~~

.. autoclass:: pytorch_transformers.AdamW
    :members:

Schedules
----------------------------------------------------

18
19
20
Learning Rate Schedules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

thomwolf's avatar
thomwolf committed
21
22
23
.. autoclass:: pytorch_transformers.ConstantLRSchedule
    :members:

24

thomwolf's avatar
thomwolf committed
25
26
27
.. autoclass:: pytorch_transformers.WarmupConstantSchedule
    :members:

28
29
30
31
32
.. image:: /imgs/warmup_constant_schedule.png
    :target: /imgs/warmup_constant_schedule.png
    :alt:


thomwolf's avatar
thomwolf committed
33
34
35
.. autoclass:: pytorch_transformers.WarmupCosineSchedule
    :members:

36
37
38
39
40
.. image:: /imgs/warmup_cosine_schedule.png
    :target: /imgs/warmup_cosine_schedule.png
    :alt:


thomwolf's avatar
thomwolf committed
41
42
43
.. autoclass:: pytorch_transformers.WarmupCosineWithHardRestartsSchedule
    :members:

44
45
46
47
48
49
.. image:: /imgs/warmup_cosine_hard_restarts_schedule.png
    :target: /imgs/warmup_cosine_hard_restarts_schedule.png
    :alt:



thomwolf's avatar
thomwolf committed
50
51
.. autoclass:: pytorch_transformers.WarmupLinearSchedule
    :members:
52
53
54
55

.. image:: /imgs/warmup_linear_schedule.png
    :target: /imgs/warmup_linear_schedule.png
    :alt: