Commits · fa5f45f0a2a69b726b87e4d88d9a954eba804b5e · OpenDAS / FastMoE

29 May, 2021 1 commit
- fix bugs to run megatron with gshard gate · fa5f45f0
  Rick Ho authored May 29, 2021
  
  fa5f45f0
24 May, 2021 2 commits
- add random routing in gshard gate · e58e7b3b
  Rich Ho authored May 24, 2021
  
  e58e7b3b
- mask and experts list (#2) · ff7333c7
  Colin authored May 24, 2021
```
- mask some tensors of tokens for fmoe forward
- pass a list of expert classes to specify what experts in what order want to use
```
  ff7333c7
23 May, 2021 1 commit
- mask and experts list · 28ba2d28
  Colin authored May 23, 2021
  
  28ba2d28
21 May, 2021 1 commit
- update accoding to comments · 8d14dd29
  Rich Ho authored May 21, 2021
  
  8d14dd29
19 May, 2021 3 commits
- fix scatter bug across gpus · 90c4bccf
  Rich Ho authored May 19, 2021
  
  90c4bccf
- pass test on single gpu · 670d70ed
  Rick Ho authored May 19, 2021
  
  670d70ed
- remove repeate interleave and local scatter · 414a2f86
  Rick Ho authored May 19, 2021
  
  414a2f86
18 May, 2021 2 commits
- use customized pos assignment kernel to support -1 · 980cf4b6
  Rick Ho authored May 18, 2021
  
  980cf4b6
- fix bugs to run on multiple gpus · a468db2b
  Rick Ho authored May 18, 2021
  
  a468db2b
13 May, 2021 3 commits
- test switch gate · 38b334cc
  Rich Ho authored May 13, 2021
  
  38b334cc
- gshard gate test · ddfaaf49
  Rich Ho authored May 13, 2021
  
  ddfaaf49
- add test but cannot pass · 5a0ba835
  Rick Ho authored May 13, 2021
  
  5a0ba835
27 Apr, 2021 3 commits
- gshard prune gate cuda code and test · 56cb8c15
  Rick Ho authored Apr 27, 2021
  
  56cb8c15
- compute capacity in gshard · 3e8c263c
  Rick Ho authored Apr 27, 2021
  
  3e8c263c
- balancing cuda code · 1aced6d8
  Rick Ho authored Apr 27, 2021
  
  1aced6d8
26 Apr, 2021 4 commits
- fix zero gate and constant gate · 450549a4
  Rick Ho authored Apr 26, 2021
  
  450549a4
- reconstruct fmoe cuda code · bb3c8966
  Rick Ho authored Apr 26, 2021
  
  bb3c8966
- add gshard and switch gate with loss · 4c90e6e8
  Rick Ho authored Apr 26, 2021
  
  4c90e6e8
- reconstruct gate structure · 82103edb
  Rick Ho authored Apr 26, 2021
  
  82103edb
23 Mar, 2021 2 commits
- Bias now being calculated directly in MOELinear layer. Added corresponding... · 303d0e93
  TiagoMAntunes authored Mar 23, 2021
```
Bias now being calculated directly in MOELinear layer. Added corresponding CUDA changes. Updated forward and backward functions of MOELinear
```
  303d0e93
- Updated arguments for MOELinear.apply · 8bac18dc
  TiagoMAntunes authored Mar 23, 2021
  
  8bac18dc
22 Mar, 2021 18 commits
- add balance to init · 2bceb76b
  Rick Ho authored Mar 22, 2021
  
  2bceb76b
- initial split · b690f245
  Rick Ho authored Mar 22, 2021
  
  b690f245
- Revise variable name · 1b7817a5
  Sengxian authored Mar 22, 2021
  
  1b7817a5
- remove debug info · bedaa47d
  Jiezhong Qiu authored Mar 22, 2021
  
  bedaa47d
- format with black · 6b5a34cc
  Jiezhong Qiu authored Mar 22, 2021
  
  6b5a34cc
- fix pylint · 68997976
  Jiezhong Qiu authored Mar 22, 2021
  
  68997976
- load/save fp32 main weight when fp16 training · 90e394bc
  Jiezhong Qiu authored Mar 22, 2021
  
  90e394bc
- fp16/fp32 optimizer has different data structure · bcfeaf3b
  Jiezhong Qiu authored Mar 17, 2021
  
  bcfeaf3b
- remove unnecessary states when dp rank>0 · 12b23ae3
  Jiezhong Qiu authored Mar 17, 2021
  
  12b23ae3
- Create Megatron.LICENSE · bc655118
  Jiezhong Qiu authored Mar 17, 2021
  
  bc655118
- load checkpoint for expert parallel · 24541d6a
  Jiezhong Qiu authored Mar 17, 2021
  
  24541d6a
- avoid saveing duplicated optim state in dp rank>0 · f5a5d31a
  Jiezhong Qiu authored Mar 17, 2021
  
  f5a5d31a
- fix pylint · c844413b
  Jiezhong Qiu authored Mar 16, 2021
  
  c844413b
- save experts separately in each data parallel rank · 49a4678c
  Jiezhong Qiu authored Mar 16, 2021
  
  49a4678c
- Revise variable name · e028f2ec
  Sengxian authored Mar 22, 2021
  
  e028f2ec
- Add gshard and noisy gate balance strategy · 89d6c794
  Sengxian authored Mar 22, 2021
  
  89d6c794
- remove debug info · f49a9ec4
  Jiezhong Qiu authored Mar 22, 2021
  
  f49a9ec4
- format with black · d30b746a
  Jiezhong Qiu authored Mar 22, 2021
  
  d30b746a