Commits · c1c19f3e1a4dfd7354edcd485452c51d1535428c · OpenDAS / FastMoE

21 Aug, 2023 1 commit
- Merge pull request #171 from laekov/xptree/fix-bf16 · c1c19f3e
  Rick Ho authored Aug 21, 2023
```
fix cublas gemm call for bf16 input
```
  c1c19f3e
20 Aug, 2023 1 commit
- fix cublas gemm call for bf16 input · a7975801
  xptree authored Aug 20, 2023
  
  a7975801
30 Jul, 2023 2 commits
- Merge pull request #167 from laekov/zms-update-readme · cd8372b3
  Rick Ho authored Jul 30, 2023
```
Update outdated README
```
  cd8372b3
- Update outdated README · d3e41415
  Mingshu Zhai authored Jul 30, 2023
```
As mentioned in https://github.com/laekov/fastmoe/issues/166
```
  d3e41415
24 Jul, 2023 4 commits
- Merge pull request #165 from laekov/parallel-doc · de4ef388
  Rick Ho authored Jul 24, 2023
```
Document for process groups
```
  de4ef388
- spell check · 8bb58982
  Rick Ho authored Jul 24, 2023
  
  8bb58982
- Parallelism Document · e70726a9
  Rick Ho authored Jul 24, 2023
  
  e70726a9
- update readme and inline docstring · 02341276
  Rick Ho authored Jul 24, 2023
  
  02341276
04 Jul, 2023 1 commit
- Megatron v3.0.2 Patch (#159) · a6d202a6
  Jiezhong Qiu authored Jul 04, 2023
```
* support megatron v3.0.2

* keep num_experts for lower version of megatron
```
  a6d202a6
06 Jun, 2023 2 commits
- Merge pull request #160 from laekov/nvbf16 · d56522bc
  Mingshu Zhai authored Jun 06, 2023
```
bf16 support
```
  d56522bc
- bf16 support with test · 0bb60881
  Rick Ho authored Jun 06, 2023
  
  0bb60881
30 May, 2023 2 commits
- release note v1.0.1 · c9ccc0eb
  Rick Ho authored May 30, 2023
  
  c9ccc0eb
- Merge pull request #157 from laekov/pytorch2-compat · 1e8e455a
  Rick Ho authored May 30, 2023
```
Fix ProcessGroupNCCL mismatch in pytorch2
```
  1e8e455a
29 May, 2023 1 commit
- fix processgroupnccl mismatch in pytorch2 · 267eb9cc
  Rick Ho authored May 29, 2023
  
  267eb9cc
21 May, 2023 5 commits
- Merge pull request #155 from santurini/patch-3 · 9fe65ec0
  Rick Ho authored May 21, 2023
```
Added link to installation guide
```
  9fe65ec0
- Merge pull request #154 from santurini/patch-2 · 2ec2f14a
  Rick Ho authored May 21, 2023
```
Create installation-guide.md
```
  2ec2f14a
- Added link to installation guide · 7817fa15
  Arturo Ghinassi authored May 21, 2023
  
  7817fa15
- Reviewed following @laekov suggestions · 3eabb02f
  Arturo Ghinassi authored May 21, 2023
  
  3eabb02f
- Create installation-guide.md · cb74e2f6
  Arturo Ghinassi authored May 21, 2023
```
Added installation tutorial in the documentation
```
  cb74e2f6
19 May, 2023 6 commits
- Merge pull request #152 from santurini/patch-1 · 50ac47f9
  Rick Ho authored May 19, 2023
```
Cast input to weights type for AMP support
```
  50ac47f9
- Cast input to weights type for AMP support · af8c4737
  Arturo Ghinassi authored May 19, 2023
  
  af8c4737
- Merge pull request #151 from laekov/revert-150-patch-1 · a31fcc55
  Rick Ho authored May 19, 2023
```
Revert "convert input to same type as weight for mixed precision training"
```
  a31fcc55
- Revert "convert input to same type as weight for mixed precision training" · f1a6e6fd
  Rick Ho authored May 19, 2023
  
  f1a6e6fd
- Merge pull request #150 from santurini/patch-1 · 3a0eb3e6
  Rick Ho authored May 19, 2023
```
convert input to same type as weight for mixed precision training
```
  3a0eb3e6
- changed .float() to .as_type(self.weight) · 1f7c04c9
  Arturo Ghinassi authored May 19, 2023
```
AMP support
```
  1f7c04c9
18 May, 2023 1 commit
- convert input to same type as weight for mixed precision training · 2196b766
  Arturo Ghinassi authored May 18, 2023
```
When using CUDA AMP FMoE Linear throws type error as input is half() and weights are float()
```
  2196b766
21 Mar, 2023 3 commits
- Merge pull request #146 from laekov/fastermoe_multi_experts · df715c9f
  Rick Ho authored Mar 21, 2023
```
make FasterMoE more general
```
  df715c9f
- support n_expert > 1 for FasterMoE smart scheduling and expert shadowing · 14878015
  zms1999 authored Mar 21, 2023
  
  14878015
- Merge pull request #145 from laekov/patch_megatron_v2_5 · 698a12ae
  Rick Ho authored Mar 21, 2023
```
FastMoE with Megatron-LM v2.5
```
  698a12ae
20 Mar, 2023 1 commit
- patch for megatron v2.5 · b31b7e3f
  zms1999 authored Mar 21, 2023
  
  b31b7e3f
13 Feb, 2023 2 commits
- Merge pull request #144 from Fragile-azalea/master · b8d0e81f
  Rick Ho authored Feb 13, 2023
```
remove synchronize
```
  b8d0e81f
- remove synchronize · 6ebec59c
  Fragile-azalea authored Feb 13, 2023
  
  6ebec59c
28 Dec, 2022 3 commits
- Merge pull request #142 from laekov/torch-1.13-compat · 9303ac18
  Rick Ho authored Dec 28, 2022
```
Compatibility to older cuda and torch 1.13
```
  9303ac18
- cudaStreamWaitEvent compat · 25e9324f
  Rick Ho authored Dec 28, 2022
  
  25e9324f
- update header file processgroupnccl · c7e6a3db
  Rick Ho authored Dec 28, 2022
  
  c7e6a3db
14 Dec, 2022 3 commits
- Merge pull request #141 from laekov/gate-fix · 2cb96b40
  Rick Ho authored Dec 14, 2022
```
Diverge gshard gate
```
  2cb96b40
- capacity numerical fix · a762d33c
  Rick Ho authored Dec 14, 2022
  
  a762d33c
- diverge gshard gate · 6cb550fd
  Rick Ho authored Dec 14, 2022
  
  6cb550fd
14 Nov, 2022 2 commits
- Merge pull request #138 from laekov/clip-bug · cdc140f1
  Rick Ho authored Nov 14, 2022
```
fix bug: add proper comm group
```
  cdc140f1
- fix bug: add proper comm group · 66d27e9a
  zms1999 authored Nov 14, 2022
  
  66d27e9a