Commits · 82103edbc4b0b4e9505de0dbe196fa39c8f46c89 · OpenDAS / FastMoE

26 Apr, 2021 1 commit
- reconstruct gate structure · 82103edb
  Rick Ho authored Apr 26, 2021
  
  82103edb
22 Mar, 2021 19 commits
- add balance to init · 2bceb76b
  Rick Ho authored Mar 22, 2021
  
  2bceb76b
- initial split · b690f245
  Rick Ho authored Mar 22, 2021
  
  b690f245
- Revise variable name · 1b7817a5
  Sengxian authored Mar 22, 2021
  
  1b7817a5
- remove debug info · bedaa47d
  Jiezhong Qiu authored Mar 22, 2021
  
  bedaa47d
- format with black · 6b5a34cc
  Jiezhong Qiu authored Mar 22, 2021
  
  6b5a34cc
- fix pylint · 68997976
  Jiezhong Qiu authored Mar 22, 2021
  
  68997976
- load/save fp32 main weight when fp16 training · 90e394bc
  Jiezhong Qiu authored Mar 22, 2021
  
  90e394bc
- fp16/fp32 optimizer has different data structure · bcfeaf3b
  Jiezhong Qiu authored Mar 17, 2021
  
  bcfeaf3b
- remove unnecessary states when dp rank>0 · 12b23ae3
  Jiezhong Qiu authored Mar 17, 2021
  
  12b23ae3
- Create Megatron.LICENSE · bc655118
  Jiezhong Qiu authored Mar 17, 2021
  
  bc655118
- load checkpoint for expert parallel · 24541d6a
  Jiezhong Qiu authored Mar 17, 2021
  
  24541d6a
- avoid saveing duplicated optim state in dp rank>0 · f5a5d31a
  Jiezhong Qiu authored Mar 17, 2021
  
  f5a5d31a
- fix pylint · c844413b
  Jiezhong Qiu authored Mar 16, 2021
  
  c844413b
- save experts separately in each data parallel rank · 49a4678c
  Jiezhong Qiu authored Mar 16, 2021
  
  49a4678c
- Revise variable name · e028f2ec
  Sengxian authored Mar 22, 2021
  
  e028f2ec
- Add gshard and noisy gate balance strategy · 89d6c794
  Sengxian authored Mar 22, 2021
  
  89d6c794
- remove debug info · f49a9ec4
  Jiezhong Qiu authored Mar 22, 2021
  
  f49a9ec4
- format with black · d30b746a
  Jiezhong Qiu authored Mar 22, 2021
  
  d30b746a
- fix pylint · a372eb66
  Jiezhong Qiu authored Mar 22, 2021
  
  a372eb66
21 Mar, 2021 1 commit
- load/save fp32 main weight when fp16 training · d8124b80
  Jiezhong Qiu authored Mar 22, 2021
  
  d8124b80
17 Mar, 2021 5 commits
- fp16/fp32 optimizer has different data structure · 59f10bf3
  Jiezhong Qiu authored Mar 17, 2021
  
  59f10bf3
- remove unnecessary states when dp rank>0 · f224b69b
  Jiezhong Qiu authored Mar 17, 2021
  
  f224b69b
- Create Megatron.LICENSE · 87c1e6bb
  Jiezhong Qiu authored Mar 17, 2021
  
  87c1e6bb
- load checkpoint for expert parallel · 6ca840b5
  Jiezhong Qiu authored Mar 17, 2021
  
  6ca840b5
- avoid saveing duplicated optim state in dp rank>0 · 1c69da9c
  Jiezhong Qiu authored Mar 17, 2021
  
  1c69da9c
16 Mar, 2021 2 commits
- fix pylint · a54546a4
  Jiezhong Qiu authored Mar 16, 2021
  
  a54546a4
- save experts separately in each data parallel rank · 9c902b82
  Jiezhong Qiu authored Mar 16, 2021
  
  9c902b82
13 Mar, 2021 1 commit
- fix lint · 3b82e379
  Rick Ho authored Mar 13, 2021
  
  3b82e379
09 Mar, 2021 1 commit
- Format using black · 4d48209d
  Sengxian authored Mar 09, 2021
  
  4d48209d
08 Mar, 2021 1 commit
- fix numerical issue in tests and typo · 527a8cc9
  Rick Ho authored Mar 08, 2021
  
  527a8cc9
26 Feb, 2021 5 commits
- fix rank of megatron dp · c0cfcc47
  Rick Ho authored Feb 26, 2021
  
  c0cfcc47
- use magatron's init method for ffn · 8ddd246f
  Jiezhong Qiu authored Feb 26, 2021
  
  8ddd246f
- fix lint · ba878d29
  Rick Ho authored Feb 26, 2021
  
  ba878d29
- let fmoe.megatron use correct mpu · 66f7166d
  Rick Ho authored Feb 26, 2021
  
  66f7166d
- zero gate update · 89de2153
  Rick Ho authored Feb 26, 2021
  
  89de2153
25 Feb, 2021 4 commits
- seems that solution 1 is still the fatest · 1dc7e73e
  Jiezhong Qiu authored Feb 26, 2021
```
i.e., expand bias using torch.repeat_interleave directly
```
  1dc7e73e
- propose and discuss 3 solutions to expand bias · c65039da
  Jiezhong Qiu authored Feb 26, 2021
  
  c65039da
- return a zero bias wo grad for megatron · 63f6ebbf
  Jiezhong Qiu authored Feb 26, 2021
```
the true bias has been added in FMoeLinear
```
  63f6ebbf
- simplify TransformerMLP · fe2009b1
  Jiezhong Qiu authored Feb 26, 2021
```
move away layernorm/residual/dropout
```
  fe2009b1