Commits · 53b5b8c38c6dd0edb52b87999e553c1f9fcd4d4a · OpenDAS / FastMoE

"configs/_base_/datasets/nuim-instance.py" did not exist on "583c4accbbf8e37b15638820b7b781f4475c6bde"

01 Mar, 2021 1 commit
- update megatron example · 53b5b8c3
  Rick Ho authored Mar 01, 2021
  
  53b5b8c3
28 Feb, 2021 10 commits
- bigger logo in title · 2d067240
  Rick Ho authored Feb 28, 2021
  
  2d067240
- update logo in readme · fcd7ee54
  Rick Ho authored Feb 28, 2021
  
  fcd7ee54
- add logo · 4c9bc502
  Rick Ho authored Feb 28, 2021
  
  4c9bc502
- fix pytorch docker link in readme · 2628cedb
  Rick Ho authored Feb 28, 2021
  
  2628cedb
- Chinese readme and release note · dd7cbfc4
  Rick Ho authored Feb 28, 2021
  
  dd7cbfc4
- add slack link to readme · 23d1aa66
  Rick Ho authored Feb 28, 2021
  
  23d1aa66
- adjust figure sizes and positions · 38d34c9c
  Jiezhong Qiu authored Feb 28, 2021
  
  38d34c9c
- Update README.md · 6d2a7bc3
  Jiezhong Qiu authored Feb 28, 2021
  
  6d2a7bc3
- add figures for moe dp mode and mp mode · 1844f496
  Jiezhong Qiu authored Feb 28, 2021
  
  1844f496
- clean scripts · e1ed5d12
  Jiezhong Qiu authored Feb 28, 2021
```
remove author related info
```
  e1ed5d12
26 Feb, 2021 9 commits
- fix rank of megatron dp · c0cfcc47
  Rick Ho authored Feb 26, 2021
  
  c0cfcc47
- Merge pull request #10 from laekov/megatron-mlp-init · 593feab2
  Rick Ho authored Feb 26, 2021
```
use magatron's init method for ffn
```
  593feab2
- use magatron's init method for ffn · 8ddd246f
  Jiezhong Qiu authored Feb 26, 2021
  
  8ddd246f
- Merge pull request #9 from laekov/laekov/accfix · 5e5b4044
  Rick Ho authored Feb 26, 2021
```
Laekov/accfix
```
  5e5b4044
- fix lint · ba878d29
  Rick Ho authored Feb 26, 2021
  
  ba878d29
- let fmoe.megatron use correct mpu · 66f7166d
  Rick Ho authored Feb 26, 2021
  
  66f7166d
- Merge branch 'master' into laekov/accfix · b0990e4b
  Rick Ho authored Feb 26, 2021
  
  b0990e4b
- zero gate update · 89de2153
  Rick Ho authored Feb 26, 2021
  
  89de2153
- Merge pull request #8 from laekov/simple_mlp · 1cfc5462
  Rick Ho authored Feb 26, 2021
```
Simple mlp
```
  1cfc5462
25 Feb, 2021 12 commits
- seems that solution 1 is still the fatest · 1dc7e73e
  Jiezhong Qiu authored Feb 26, 2021
```
i.e., expand bias using torch.repeat_interleave directly
```
  1dc7e73e
- propose and discuss 3 solutions to expand bias · c65039da
  Jiezhong Qiu authored Feb 26, 2021
  
  c65039da
- return a zero bias wo grad for megatron · 63f6ebbf
  Jiezhong Qiu authored Feb 26, 2021
```
the true bias has been added in FMoeLinear
```
  63f6ebbf
- simplify TransformerMLP · fe2009b1
  Jiezhong Qiu authored Feb 26, 2021
```
move away layernorm/residual/dropout
```
  fe2009b1
- Merge pull request #7 from laekov/reproducibility · b56c8043
  Rick Ho authored Feb 25, 2021
```
Reproducibility
```
  b56c8043
- add TODO for bias issue · b44de4cd
  Jiezhong Qiu authored Feb 25, 2021
  
  b44de4cd
- fix bug in wrong gradient of bias in FMoELinear · 08e08319
  Jiezhong Qiu authored Feb 25, 2021
  
  08e08319
- fix bug · 804999f1
  Jiezhong Qiu authored Feb 25, 2021
  
  804999f1
- put dropout after relu · 96ed8976
  Jiezhong Qiu authored Feb 25, 2021
```
although order of dp and relu doesn't matter
```
  96ed8976
- Merge pull request #6 from xfmr-xl · 03b2a725
  Rick Ho authored Feb 25, 2021
```
Test Transformer-XL
```
  03b2a725
- allow user to indicate whether to use MoE · 0a942e3f
  Jiezhong Qiu authored Feb 25, 2021
  
  0a942e3f
- remove unused code · bdb64914
  Jiezhong Qiu authored Feb 25, 2021
  
  bdb64914
24 Feb, 2021 8 commits
- use nn.Sequential as activation · 5dc62b41
  Jiezhong Qiu authored Feb 24, 2021
```
activation passed by func can not be saved
```
  5dc62b41
- add final dropout before residual · 61e4533f
  Jiezhong Qiu authored Feb 24, 2021
  
  61e4533f
- update gitignore · 5f4441ee
  Jiezhong Qiu authored Feb 24, 2021
  
  5f4441ee
- reset parameters in megatron · 94eca783
  Rick Ho authored Feb 24, 2021
  
  94eca783
- fix layernorm and remove unnecessary bias · 1f48a6b3
  Jiezhong Qiu authored Feb 24, 2021
  
  1f48a6b3
- allow user to indicate number of experts and topk · fa023f32
  Jiezhong Qiu authored Feb 24, 2021
  
  fa023f32
- update gitignore · 3bdfae96
  Jiezhong Qiu authored Feb 24, 2021
  
  3bdfae96
- fixes pytorch/pytorch/issues/36035 · d7706f0d
  Jiezhong Qiu authored Feb 24, 2021
```
follow suggestions from https://github.com/pytorch/pytorch/issues/36035#issuecomment-770960405
```
  d7706f0d