Commits · f1a6e6fd6c74bef5c4bad13d91d5d1e99cc068ad · OpenDAS / FastMoE

19 May, 2023 3 commits
- Revert "convert input to same type as weight for mixed precision training" · f1a6e6fd
  Rick Ho authored May 19, 2023
  
  f1a6e6fd
- Merge pull request #150 from santurini/patch-1 · 3a0eb3e6
  Rick Ho authored May 19, 2023
```
convert input to same type as weight for mixed precision training
```
  3a0eb3e6
- changed .float() to .as_type(self.weight) · 1f7c04c9
  Arturo Ghinassi authored May 19, 2023
```
AMP support
```
  1f7c04c9
18 May, 2023 1 commit
- convert input to same type as weight for mixed precision training · 2196b766
  Arturo Ghinassi authored May 18, 2023
```
When using CUDA AMP FMoE Linear throws type error as input is half() and weights are float()
```
  2196b766
21 Mar, 2023 3 commits
- Merge pull request #146 from laekov/fastermoe_multi_experts · df715c9f
  Rick Ho authored Mar 21, 2023
```
make FasterMoE more general
```
  df715c9f
- support n_expert > 1 for FasterMoE smart scheduling and expert shadowing · 14878015
  zms1999 authored Mar 21, 2023
  
  14878015
- Merge pull request #145 from laekov/patch_megatron_v2_5 · 698a12ae
  Rick Ho authored Mar 21, 2023
```
FastMoE with Megatron-LM v2.5
```
  698a12ae
20 Mar, 2023 1 commit
- patch for megatron v2.5 · b31b7e3f
  zms1999 authored Mar 21, 2023
  
  b31b7e3f
13 Feb, 2023 2 commits
- Merge pull request #144 from Fragile-azalea/master · b8d0e81f
  Rick Ho authored Feb 13, 2023
```
remove synchronize
```
  b8d0e81f
- remove synchronize · 6ebec59c
  Fragile-azalea authored Feb 13, 2023
  
  6ebec59c
28 Dec, 2022 3 commits
- Merge pull request #142 from laekov/torch-1.13-compat · 9303ac18
  Rick Ho authored Dec 28, 2022
```
Compatibility to older cuda and torch 1.13
```
  9303ac18
- cudaStreamWaitEvent compat · 25e9324f
  Rick Ho authored Dec 28, 2022
  
  25e9324f
- update header file processgroupnccl · c7e6a3db
  Rick Ho authored Dec 28, 2022
  
  c7e6a3db
14 Dec, 2022 3 commits
- Merge pull request #141 from laekov/gate-fix · 2cb96b40
  Rick Ho authored Dec 14, 2022
```
Diverge gshard gate
```
  2cb96b40
- capacity numerical fix · a762d33c
  Rick Ho authored Dec 14, 2022
  
  a762d33c
- diverge gshard gate · 6cb550fd
  Rick Ho authored Dec 14, 2022
  
  6cb550fd
14 Nov, 2022 2 commits
- Merge pull request #138 from laekov/clip-bug · cdc140f1
  Rick Ho authored Nov 14, 2022
```
fix bug: add proper comm group
```
  cdc140f1
- fix bug: add proper comm group · 66d27e9a
  zms1999 authored Nov 14, 2022
  
  66d27e9a
30 Sep, 2022 2 commits
- Merge pull request #136 from laekov/update-versions · 53844b65
  Rick Ho authored Sep 30, 2022
```
Update version requirement in the documents
```
  53844b65
- update version requirement in the documents · d9b6359d
  Rick Ho authored Sep 30, 2022
  
  d9b6359d
19 Sep, 2022 1 commit
- Merge pull request #135 from laekov/example-doc · 53bac444
  Rick Ho authored Sep 19, 2022
```
Document for examples
```
  53bac444
16 Sep, 2022 1 commit
- document for examples · 8cbe171d
  Rick Ho authored Sep 16, 2022
  
  8cbe171d
04 Aug, 2022 2 commits
- Merge pull request #130 from Fragile-azalea/master · 40f5fd36
  Rick Ho authored Aug 04, 2022
```
Fix GshardGate top1_idx
```
  40f5fd36
- fix GshardGate top1_idx · 5f8e3999
  Fragile-azalea authored Aug 04, 2022
  
  5f8e3999
23 Jul, 2022 2 commits
- Merge pull request #127 from heheda12345/master · f2e0d1d5
  Rick Ho authored Jul 23, 2022
```
update readme: enable NCCL by default
```
  f2e0d1d5
- update readme: enable NCCL by default · 069da0e8
  heheda authored Jul 23, 2022
  
  069da0e8
18 Jul, 2022 2 commits
- Merge pull request #123 from laekov/torch-v1.12.0-compat · 5a5dfa18
  Rick Ho authored Jul 18, 2022
```
Fix nccl uid bcast for torch v1.12.0
```
  5a5dfa18
- fix nccl uid bcast · a0b1e31a
  Rick Ho authored Jul 18, 2022
  
  a0b1e31a
12 Jun, 2022 1 commit
- Merge pull request #120 from laekov/ddp-bcast-fix · 9a01f8fa
  Rick Ho authored Jun 12, 2022
```
Fix Broadcast rank bug in DGDP
```
  9a01f8fa
10 Jun, 2022 1 commit
- update ddp with get first rank and tests · dd68fd78
  Rick Ho authored Jun 10, 2022
  
  dd68fd78
01 Jun, 2022 4 commits
- Merge pull request #117 from laekov/distributed-doc · 670e1407
  Rick Ho authored Jun 01, 2022
```
Add a hint for DGDP synchronization
```
  670e1407
- add an option of init sync · 039b4550
  Rick Ho authored Jun 01, 2022
  
  039b4550
- fix bug for sync_params · 6fe415e9
  Rick Ho authored Jun 01, 2022
  
  6fe415e9
- add a hint for DGDP synchronization · 21a331fb
  Rick Ho authored Jun 01, 2022
  
  21a331fb
27 May, 2022 2 commits
- Merge pull request #115 from laekov/gate-test-fix · 527e66af
  Rick Ho authored May 27, 2022
```
Fix gshard gate test
```
  527e66af
- fix gshard gate test · e01c8b5f
  Rick Ho authored May 27, 2022
  
  e01c8b5f
26 May, 2022 2 commits
- Merge pull request #113 from laekov/compat-fix · d3f71f21
  Rick Ho authored May 26, 2022
```
Fix smart schedule with older PyTorch
```
  d3f71f21
- fix schedule with older autograd · db247e85
  Rick Ho authored May 26, 2022
  
  db247e85
30 Apr, 2022 2 commits
- Merge pull request #108 from laekov/faster-bug · 5083a736
  Rick Ho authored Apr 30, 2022
```
Fix type mismatch, shape mismatch and lack of condition in FasterMoE's expert shadowing
```
  5083a736
- add condition of skipping pop params · 4682c1d0
  zms1999 authored Apr 30, 2022
  
  4682c1d0