Merge pull request #70 from laekov/v0.2.1-rc1

Release v0.2.1

doc/release-note.md

+## v0.2.1
+
+## Load balancing
+
+* Fix gradient for balance loss.
+
+## Misc
+
+* Typos.
+* Update benchmark interface.
+* Remove some redundant code for performance improvement.
+* Enable `USE_NCCL` by default.
+* Compatibility for PyTorch `<1.8.0` and `>=1.8.0`.
+
+## Megatron adaption
+
+* Patch for numerical correctness of gradient clipping.
+* Support to pipeline parallelism.
+
+## v0.2.0
+
+## Load balancing
+
+* A brand new gate module with capacity-related utilities.
+* GShard's and Switch Transformer's balance strategies are implemented as integrated gates.
+* Balance loss is enabled.
+* Balance monitor is provided.
+
+## Checkpointing
+
+* MoE models can be loaded and saved by fmoe's checkpointing module.
+
+## Performance
+
+* FP16 training performance is improved.
+
+## Misc
+
+* CUDA code directory is reconstructed.
+* More tests are added.
+
 ## v0.1.2
 
 ### Compilation

setup.py

@@ -15,7 +15,7 @@ authors = [
         'Qin Li',
 ]
 
-if os.environ.get('USE_NCCL', '0') == '1':
+if os.environ.get('USE_NCCL', '1') == '1':
     cxx_flags.append('-DFMOE_USE_NCCL')
     ext_libs.append('nccl')
 
@@ -23,7 +23,7 @@ if os.environ.get('USE_NCCL', '0') == '1':
 if __name__ == '__main__':
     setuptools.setup(
         name='fastmoe',
-        version='0.2.0',
+        version='0.2.1',
         description='An efficient Mixture-of-Experts system for PyTorch',
         author=', '.join(authors),
         author_email='hja20@mails.tsinghua.edu.cn',