Commits · cf7efd4f5adeb676ec30d2ff3bf2149da58ec4a3 · OpenDAS / Megatron-LM

02 Sep, 2021 1 commit
- t # This is a combination of 2 commits. · cf7efd4f
  Sangkug Lym authored Aug 30, 2021
```
allreduce overlap with wgrad gemm

change custom delay to dummy add
```
  cf7efd4f
02 Jul, 2021 1 commit
- fix typo · e515f026
  hwijeen authored Jul 02, 2021
  
  e515f026
22 Mar, 2021 1 commit
- Cherry-pick fix from development repo. · 8aa4619f
  Jared Casper authored Mar 22, 2021
  
  8aa4619f
26 Jan, 2021 1 commit
- added params norms · 57d1356e
  mohammad authored Jan 25, 2021
  
  57d1356e
25 Jan, 2021 1 commit
- Address comments, fix argument bug. · bdd47d64
  Jared Casper authored Jan 25, 2021
  
  bdd47d64
22 Jan, 2021 1 commit
- Fixing merge_mp_partitions · 78066ab0
  Jared Casper authored Jan 20, 2021
  
  78066ab0
25 Dec, 2020 1 commit
- fp32 is also working · 2eaa3ccc
  mohammad authored Dec 24, 2020
  
  2eaa3ccc
22 Dec, 2020 1 commit
- Add the option for fp32 residual connection (fp32 residual connection... · 62632d39
  mohammad authored Dec 21, 2020
```
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)
```
  62632d39
12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
03 Sep, 2020 1 commit
- debuged the cpu init for mp=1 · 04cd82bd
  mohammad authored Sep 03, 2020
  
  04cd82bd
27 Aug, 2020 1 commit
- Addressing code review comments · 1eafa861
  Boris Fomitchev authored Aug 26, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  1eafa861
26 Aug, 2020 1 commit
- Making --lazy_mpu_init act as _USE_CPU_INITIALIZATION · e10760ea
  Boris Fomitchev authored Aug 26, 2020
  
  e10760ea
03 Aug, 2020 1 commit
- added gpu initialization and option to avoid master values · 0403b808
  mohammad authored Aug 03, 2020
  
  0403b808
31 Jul, 2020 1 commit
- Code review comments - changing parallel test condition · 35bea728
  Boris Fomitchev authored Jul 30, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  35bea728
30 Jul, 2020 1 commit
- Changes for NeMo/lightning compatibility · 417c7f6a
  Boris Fomitchev authored Jul 30, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  417c7f6a
26 Jul, 2020 1 commit
- Complete fix for APEX absence in NeMo · aa0ee72e
  Boris Fomitchev authored Jul 25, 2020
  
  aa0ee72e
24 Jul, 2020 1 commit
- ONNX export fix, including one for gelu() · 158a99c3
  Boris Fomitchev authored Jul 23, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  158a99c3
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
14 Apr, 2020 1 commit
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
10 Feb, 2020 1 commit
- Model parallel merger · 57c2060f
  Mohammad Shoeybi authored Feb 10, 2020
  
  57c2060f
08 Oct, 2019 1 commit
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb
30 Jul, 2019 1 commit

large update including model parallelism and gpt2 · abe36e2e

Raul Puri authored Jul 29, 2019


Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

abe36e2e