Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
fp16_utils.rst 1.32 KB
Newer Older
Christian Sarofeen's avatar
Initial release  
Christian Sarofeen committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50 
.. role:: hidden
    :class: hidden-section

apex.fp16_utils
===================================

This submodule contains utilities designed to streamline the mixed precision training recipe 
presented by NVIDIA `on Parallel Forall`_ and in GTC 2018 Sessions 
`Training Neural Networks with Mixed Precision: Theory and Practice`_ and 
`Training Neural Networks with Mixed Precision: Real Examples`_.
For Pytorch users, Real Examples in particular is recommended.

.. _`on Parallel Forall`:
    https://devblogs.nvidia.com/mixed-precision-training-deep-neural-networks/
.. _`Training Neural Networks with Mixed Precision: Theory and Practice`:
    http://on-demand.gputechconf.com/gtc/2018/video/S8923/
.. _`Training Neural Networks with Mixed Precision: Real Examples`:
    http://on-demand.gputechconf.com/gtc/2018/video/S81012/

.. automodule:: apex.fp16_utils
.. currentmodule:: apex.fp16_utils

.. FusedNorm
   ----------

.. autofunction:: prep_param_lists

.. autofunction:: master_params_to_model_params

.. autofunction:: model_grads_to_master_grads

.. autoclass:: FP16_Optimizer
    :members:

.. autoclass:: Fused_Weight_Norm
    :members:

.. .. automodule:: apex.fp16_utils.loss_scaler

.. autoclass:: LossScaler
    :members:

.. autoclass:: DynamicLossScaler
    :members:

.. .. automodule:: apex.fp16_utils.fp16util
    :members: