syncbn.rst

Implementing Synchronized Multi-GPU Batch Normalization
=======================================================
We will release the implementation detail of Multi-GPU Batch Normalization in later version.

Why Synchronize?
----------------

- Standard Implementation

How to Synchronize?
-------------------

- Forward and Backward Pass

- Synchronized DataParallel

- Cross GPU Autograd

Comparing Performance 
---------------------