DeepSpeed
=========

Model Setup
-----------

.. toctree::
   :maxdepth: 2

   initialize

Training API
------------

.. toctree::
   :maxdepth: 2

   training


Checkpointing API
-----------------
.. toctree::
   :maxdepth: 2

   model-checkpointing
   activation-checkpointing


ZeRO API
--------
.. toctree::
   :maxdepth: 2

   zero3



Transformer Kernel API
----------------------
.. toctree::
   :maxdepth: 2

   kernel

Pipeline Parallelism
--------------------
.. toctree::
   :maxdepth: 2

   pipeline

Optimizers
--------------------
.. toctree::
   :maxdepth: 2

   optimizers

Learning Rate Schedulers
--------------------
.. toctree::
   :maxdepth: 2

   schedulers

Flops Profiler
--------------------
.. toctree::
   :maxdepth: 2

   flops-profiler

Indices and tables
------------------

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`