DeepSpeed
=========

Model Setup
-----------

.. toctree::
   :maxdepth: 2

   initialize
   inference-init

Training API
------------

.. toctree::
   :maxdepth: 2

   training

Inference API
-------------

.. toctree::
   :maxdepth: 2

   inference-engine

Checkpointing API
-----------------
.. toctree::
   :maxdepth: 2

   model-checkpointing
   activation-checkpointing


ZeRO API
--------
.. toctree::
   :maxdepth: 2

   zero3

Mixture of Experts (MoE)
------------------------
.. toctree::
   :maxdepth: 2

   moe

Transformer Kernel API
----------------------
.. toctree::
   :maxdepth: 2

   kernel

Pipeline Parallelism
--------------------
.. toctree::
   :maxdepth: 2

   pipeline

Optimizers
----------
.. toctree::
   :maxdepth: 2

   optimizers

Learning Rate Schedulers
--------------------
.. toctree::
   :maxdepth: 2

   schedulers

Flops Profiler
--------------------
.. toctree::
   :maxdepth: 2

   flops-profiler

Autotuning
--------------------
.. toctree::
   :maxdepth: 2

   autotuning

Memory Usage
------------------
.. toctree::
   :maxdepth: 2

   memory

Monitoring
----------
.. toctree::
   :maxdepth: 2

   monitor

Indices and tables
------------------

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`