Commits · b44de4cd1b39d0d1fbf3286203773ed00e37d8a1 · OpenDAS / FastMoE

25 Feb, 2021 7 commits
- add TODO for bias issue · b44de4cd
  Jiezhong Qiu authored Feb 25, 2021
  
  b44de4cd
- fix bug in wrong gradient of bias in FMoELinear · 08e08319
  Jiezhong Qiu authored Feb 25, 2021
  
  08e08319
- fix bug · 804999f1
  Jiezhong Qiu authored Feb 25, 2021
  
  804999f1
- put dropout after relu · 96ed8976
  Jiezhong Qiu authored Feb 25, 2021
```
although order of dp and relu doesn't matter
```
  96ed8976
- Merge pull request #6 from xfmr-xl · 03b2a725
  Rick Ho authored Feb 25, 2021
```
Test Transformer-XL
```
  03b2a725
- allow user to indicate whether to use MoE · 0a942e3f
  Jiezhong Qiu authored Feb 25, 2021
  
  0a942e3f
- remove unused code · bdb64914
  Jiezhong Qiu authored Feb 25, 2021
  
  bdb64914
24 Feb, 2021 7 commits
- use nn.Sequential as activation · 5dc62b41
  Jiezhong Qiu authored Feb 24, 2021
```
activation passed by func can not be saved
```
  5dc62b41
- add final dropout before residual · 61e4533f
  Jiezhong Qiu authored Feb 24, 2021
  
  61e4533f
- update gitignore · 5f4441ee
  Jiezhong Qiu authored Feb 24, 2021
  
  5f4441ee
- fix layernorm and remove unnecessary bias · 1f48a6b3
  Jiezhong Qiu authored Feb 24, 2021
  
  1f48a6b3
- allow user to indicate number of experts and topk · fa023f32
  Jiezhong Qiu authored Feb 24, 2021
  
  fa023f32
- update gitignore · 3bdfae96
  Jiezhong Qiu authored Feb 24, 2021
  
  3bdfae96
- fixes pytorch/pytorch/issues/36035 · d7706f0d
  Jiezhong Qiu authored Feb 24, 2021
```
follow suggestions from https://github.com/pytorch/pytorch/issues/36035#issuecomment-770960405
```
  d7706f0d
23 Feb, 2021 5 commits
- fix tensor initialization bug · e86dea53
  Rick Ho authored Feb 23, 2021
  
  e86dea53
- optional layer-norm in transformer mlp module · 9e67148c
  Rick Ho authored Feb 23, 2021
  
  9e67148c
- megatron support for specific expert parallelism · b97483a4
  Rick Ho authored Feb 23, 2021
  
  b97483a4
- use nn.modulelist to wrap up customized experts · 8f67b530
  Rick Ho authored Feb 23, 2021
  
  8f67b530
- Fix input grad in mp group · 704092b1
  Sengxian authored Feb 23, 2021
  
  704092b1
22 Feb, 2021 5 commits
- fix unpack bug · 092c8d67
  Sengxian authored Feb 22, 2021
  
  092c8d67
- Fix conflict · 05728524
  Sengxian authored Feb 22, 2021
  
  05728524
- customized top_k and add test for localddp · f39f411a
  Sengxian authored Feb 22, 2021
  
  f39f411a
- add inspection of input.grad in tests · 5cb4e63c
  Rick Ho authored Feb 22, 2021
  
  5cb4e63c
- customized dp-comm and hidden-hidden-size · 27c89b5a
  Rick Ho authored Feb 22, 2021
  
  27c89b5a
21 Feb, 2021 9 commits
- Merge pull request #5 from laekov/bias · 87dad9d5
  Rick Ho authored Feb 21, 2021
```
merge Bias
```
  87dad9d5
- Merge remote-tracking branch 'origin/master' into bias · 01464726
  Jiezhong Qiu authored Feb 21, 2021
  
  01464726
- follow pylint to remove lambda · 3d8e8a43
  Jiezhong Qiu authored Feb 21, 2021
  
  3d8e8a43
- test bias grad · e36c3c4c
  Jiezhong Qiu authored Feb 21, 2021
  
  e36c3c4c
- new test structure according to new expert_fn structure · 6acc3e41
  Rick Ho authored Feb 21, 2021
  
  6acc3e41
- change expert_fn structure · c0a3a425
  Rick Ho authored Feb 21, 2021
  
  c0a3a425
- test bias, dp still not passed · a88d1124
  Jiezhong Qiu authored Feb 21, 2021
  
  a88d1124
- update nccl installation method and write readme · ed9277f9
  Rick Ho authored Feb 21, 2021
  
  ed9277f9
- add and initialize bias term in FMoELinear · 3c24222c
  Jiezhong Qiu authored Feb 21, 2021
  
  3c24222c
20 Feb, 2021 7 commits
- Merge pull request #4 from laekov/init_expert · 406955e7
  Rick Ho authored Feb 20, 2021
```
merge init expert
```
  406955e7
- fix pylint issues · 269f3fd4
  Jiezhong Qiu authored Feb 20, 2021
  
  269f3fd4
- remove nccl from lib list · 1a6073b5
  Jiezhong Qiu authored Feb 20, 2021
  
  1a6073b5
- Merge remote-tracking branch 'origin/master' into init_expert · fa6872cd
  Jiezhong Qiu authored Feb 20, 2021
  
  fa6872cd
- add nccl to lib list · 02ead75d
  Rick Ho authored Feb 20, 2021
  
  02ead75d
- fix import error and dtype/device problem in MoE · da4a8e5e
  Jiezhong Qiu authored Feb 20, 2021
  
  da4a8e5e
- init FMoELinear param using mp_ank and numpy rng · 5f5ccd47
  Jiezhong Qiu authored Feb 20, 2021
  
  5f5ccd47