Commits · b97483a460d47d53e576a46aa4e9ed23512271e2 · OpenDAS / FastMoE

23 Feb, 2021 1 commit
- megatron support for specific expert parallelism · b97483a4
  Rick Ho authored Feb 23, 2021
  
  b97483a4
22 Feb, 2021 2 commits
- customized top_k and add test for localddp · f39f411a
  Sengxian authored Feb 22, 2021
  
  f39f411a
- customized dp-comm and hidden-hidden-size · 27c89b5a
  Rick Ho authored Feb 22, 2021
  
  27c89b5a
07 Feb, 2021 2 commits
- fix typo · d6169eb1
  Rick Ho authored Feb 07, 2021
  
  d6169eb1
- support arbitrary module as expert · b3380ec2
  Rick Ho authored Feb 07, 2021
  
  b3380ec2
05 Feb, 2021 3 commits
- sync in the whole world instead of mp world in megatron · d6e7a429
  Rick Ho authored Feb 05, 2021
  
  d6e7a429
- pass pylint · f2040d9f
  Rick Ho authored Feb 05, 2021
  
  f2040d9f
- add functions to support checkpointing in megatron ddp · 481f5c4f
  Rick Ho authored Feb 05, 2021
  
  481f5c4f
04 Feb, 2021 4 commits
- adapt with pytorch 1.8.0 (deprecated 1.6.0) · 15f98a10
  Rick Ho authored Feb 04, 2021
  
  15f98a10
- fix no grad after all-gather bug · 56c1bd63
  Rick Ho authored Feb 04, 2021
  
  56c1bd63
- use parallel label in gate · d83234b0
  Rick Ho authored Feb 04, 2021
  
  d83234b0
- ddp module for sophiscated hybrid parallel · 67c667f2
  Rick Ho authored Feb 04, 2021
  
  67c667f2
03 Feb, 2021 2 commits
- fix pure data parallel · ae2c434e
  Rick Ho authored Feb 03, 2021
  
  ae2c434e
- fmoefy · 6b8d2f2e
  Rick Ho authored Feb 03, 2021
  
  6b8d2f2e
02 Feb, 2021 3 commits
- fix replica condition and minor optimizations · dc3db673
  Rick Ho authored Feb 02, 2021
  
  dc3db673
- Optimize redundancy communication · 01ae2d72
  Sengxian authored Feb 02, 2021
  
  01ae2d72
- Format using black and add model_parallel_rank · fdbac1df
  Sengxian authored Feb 02, 2021
  
  fdbac1df
29 Jan, 2021 2 commits
- fix python bugs · 6900f1de
  Rick Ho authored Jan 29, 2021
  
  6900f1de
- reconstruct fmoe nn module · 437afda2
  Rick Ho authored Jan 29, 2021
  
  437afda2
26 Jan, 2021 1 commit
- initial version to run with megatron · 58e949cf
  Rick Ho authored Jan 26, 2021
  
  58e949cf
25 Jan, 2021 1 commit
- basic megatron support frame · d0f07ff7
  Rick Ho authored Jan 25, 2021
  
  d0f07ff7