Transformer kernel release (#242)

* Transformer kernels release
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

docs/code-docs/source/kernel.rst

+Transformer Kernels
+===================
+
+The transformer kernel API in DeepSpeed can be used to create BERT transformer layer for
+more efficient pre-training and fine-tuning, it includes the transformer layer configurations and
+transformer layer module initialization.
+
+Here we present the transformer kernel API.
+Please see the `BERT pre-training tutorial <https://www.deepspeed.ai/tutorials/bert-pretraining/>`_ for usage details.
+
+DeepSpeed Transformer Config
+----------------------------
+.. autoclass:: deepspeed.DeepSpeedTransformerConfig
+
+DeepSpeed Transformer Layer
+----------------------------
+.. autoclass:: deepspeed.DeepSpeedTransformerLayer

requirements.txt

@@ -8,3 +8,4 @@ tensorflow-gpu==1.15.2
 pytest
 pytest-forked
 pre-commit
+clang-format