Commits · f6afdbeeae6d7b44cd45651906c02760605a6eb7 · OpenDAS / FastMoE

24 Feb, 2021 1 commit
- add megatron back · f6afdbee
  Rick Ho authored Feb 24, 2021
  
  f6afdbee
23 Feb, 2021 6 commits
- optional residual · 72e40c74
  Rick Ho authored Feb 23, 2021
  
  72e40c74
- fix tensor initialization bug · e86dea53
  Rick Ho authored Feb 23, 2021
  
  e86dea53
- optional layer-norm in transformer mlp module · 9e67148c
  Rick Ho authored Feb 23, 2021
  
  9e67148c
- megatron support for specific expert parallelism · b97483a4
  Rick Ho authored Feb 23, 2021
  
  b97483a4
- use nn.modulelist to wrap up customized experts · 8f67b530
  Rick Ho authored Feb 23, 2021
  
  8f67b530
- Fix input grad in mp group · 704092b1
  Sengxian authored Feb 23, 2021
  
  704092b1
22 Feb, 2021 2 commits
- customized top_k and add test for localddp · f39f411a
  Sengxian authored Feb 22, 2021
  
  f39f411a
- customized dp-comm and hidden-hidden-size · 27c89b5a
  Rick Ho authored Feb 22, 2021
  
  27c89b5a
21 Feb, 2021 4 commits
- follow pylint to remove lambda · 3d8e8a43
  Jiezhong Qiu authored Feb 21, 2021
  
  3d8e8a43
- change expert_fn structure · c0a3a425
  Rick Ho authored Feb 21, 2021
  
  c0a3a425
- test bias, dp still not passed · a88d1124
  Jiezhong Qiu authored Feb 21, 2021
  
  a88d1124
- add and initialize bias term in FMoELinear · 3c24222c
  Jiezhong Qiu authored Feb 21, 2021
  
  3c24222c
20 Feb, 2021 3 commits
- fix pylint issues · 269f3fd4
  Jiezhong Qiu authored Feb 20, 2021
  
  269f3fd4
- fix import error and dtype/device problem in MoE · da4a8e5e
  Jiezhong Qiu authored Feb 20, 2021
  
  da4a8e5e
- init FMoELinear param using mp_ank and numpy rng · 5f5ccd47
  Jiezhong Qiu authored Feb 20, 2021
  
  5f5ccd47
19 Feb, 2021 1 commit
- broadcast distributed parameters · da11cb76
  Rick Ho authored Feb 19, 2021
  
  da11cb76
08 Feb, 2021 2 commits
- Add test for arbitrary expert in FMoE · 5ead59db
  Sengxian authored Feb 08, 2021
  
  5ead59db
- newer test with top-k bug fixed · c5556037
  Rick Ho authored Feb 08, 2021
  
  c5556037
07 Feb, 2021 3 commits
- fix typo · d6169eb1
  Rick Ho authored Feb 07, 2021
  
  d6169eb1
- support arbitrary module as expert · b3380ec2
  Rick Ho authored Feb 07, 2021
  
  b3380ec2
- separate gates file · 8328c794
  Rick Ho authored Feb 07, 2021
  
  8328c794
05 Feb, 2021 6 commits
- do not require comm in non-nccl environment · 5e9bb2e9
  Rick Ho authored Feb 05, 2021
  
  5e9bb2e9
- update instructions for megatron · 59b27103
  Rick Ho authored Feb 05, 2021
  
  59b27103
- sync in the whole world instead of mp world in megatron · d6e7a429
  Rick Ho authored Feb 05, 2021
  
  d6e7a429
- pass pylint · f2040d9f
  Rick Ho authored Feb 05, 2021
  
  f2040d9f
- support multiple pytorch versions prviate apis · bf2fd0c0
  Rick Ho authored Feb 05, 2021
  
  bf2fd0c0
- add functions to support checkpointing in megatron ddp · 481f5c4f
  Rick Ho authored Feb 05, 2021
  
  481f5c4f
04 Feb, 2021 5 commits
- adapt with pytorch 1.8.0 (deprecated 1.6.0) · 15f98a10
  Rick Ho authored Feb 04, 2021
  
  15f98a10
- setup pylint and write docs for functions · 585604fe
  Rick Ho authored Feb 04, 2021
  
  585604fe
- fix no grad after all-gather bug · 56c1bd63
  Rick Ho authored Feb 04, 2021
  
  56c1bd63
- use parallel label in gate · d83234b0
  Rick Ho authored Feb 04, 2021
  
  d83234b0
- ddp module for sophiscated hybrid parallel · 67c667f2
  Rick Ho authored Feb 04, 2021
  
  67c667f2
03 Feb, 2021 2 commits
- fix pure data parallel · ae2c434e
  Rick Ho authored Feb 03, 2021
  
  ae2c434e
- fmoefy · 6b8d2f2e
  Rick Ho authored Feb 03, 2021
  
  6b8d2f2e
02 Feb, 2021 5 commits
- fix bmm out shape · 4b650671
  Rick Ho authored Feb 02, 2021
  
  4b650671
- fix replica condition and minor optimizations · dc3db673
  Rick Ho authored Feb 02, 2021
  
  dc3db673
- remove debug output and todo for replicated mp input · a8ecd3d7
  Rick Ho authored Feb 02, 2021
  
  a8ecd3d7
- Optimize redundancy communication · 01ae2d72
  Sengxian authored Feb 02, 2021
  
  01ae2d72
- Format using black and add model_parallel_rank · fdbac1df
  Sengxian authored Feb 02, 2021
  
  fdbac1df