Commits · eec218d8dea655e92cd02f8cb09e6874e4b1f8ad · OpenDAS / Megatron-LM

30 Mar, 2022 1 commit
- sequence parallelism for embedding dropout and last linear layer + memory optimizations · eec218d8
  Vijay Korthikanti authored Mar 30, 2022
  
  eec218d8
09 Mar, 2022 1 commit
- bug fixes · dd96d402
  Vijay Korthikanti authored Mar 08, 2022
  
  dd96d402
07 Mar, 2022 2 commits
- fixes to main merge · 269f28f7
  Vijay Korthikanti authored Mar 07, 2022
  
  269f28f7
- refactor to help merge with main · 0d77c0e9
  Vijay Korthikanti authored Mar 07, 2022
  
  0d77c0e9
03 Mar, 2022 1 commit
- column parallel linear with sequence parallelism · 02bb1f5c
  Vijay Korthikanti authored Mar 02, 2022
  
  02bb1f5c
02 Mar, 2022 1 commit
- layernorm grad sync + name chnages · c0f10643
  Vijay Korthikanti authored Mar 02, 2022
  
  c0f10643
19 Feb, 2022 1 commit
- tensor model parallelism memory optmization · 5d4689c4
  Vijay Korthikanti authored Feb 18, 2022
  
  5d4689c4
18 Feb, 2022 1 commit
- support fp32 training and fix embedding update · b5726555
  Sangkug Lym authored Feb 18, 2022
  
  b5726555
16 Feb, 2022 1 commit

Sangkug Lym authored Feb 13, 2022

remove redundant linear layer class definition

add fuse_gradient_accumulation attribute to weights for simple targetting

reflect feedback and clean up the codes

arg change

83b1e42f

03 Sep, 2021 1 commit
- reflect feedback · 4df8b7a2
  slym authored Sep 02, 2021
  
  4df8b7a2
02 Sep, 2021 3 commits
- reflect feedback · 3f652469
  slym authored Sep 02, 2021
  
  3f652469
- minor changes · 16c90445
  slym authored Sep 02, 2021
  
  16c90445
- t # This is a combination of 2 commits. · cf7efd4f
  Sangkug Lym authored Aug 30, 2021
```
allreduce overlap with wgrad gemm

change custom delay to dummy add
```
  cf7efd4f
31 Aug, 2021 1 commit
- fix a typo · 30abf2c5
  vycezhong authored Aug 31, 2021
  
  30abf2c5
02 Jul, 2021 1 commit
- fix typo · e515f026
  hwijeen authored Jul 02, 2021
  
  e515f026
22 Mar, 2021 1 commit
- Cherry-pick fix from development repo. · 8aa4619f
  Jared Casper authored Mar 22, 2021
  
  8aa4619f
26 Jan, 2021 1 commit
- added params norms · 57d1356e
  mohammad authored Jan 25, 2021
  
  57d1356e
25 Jan, 2021 1 commit
- Address comments, fix argument bug. · bdd47d64
  Jared Casper authored Jan 25, 2021
  
  bdd47d64
22 Jan, 2021 1 commit
- Fixing merge_mp_partitions · 78066ab0
  Jared Casper authored Jan 20, 2021
  
  78066ab0
25 Dec, 2020 1 commit
- fp32 is also working · 2eaa3ccc
  mohammad authored Dec 24, 2020
  
  2eaa3ccc
22 Dec, 2020 1 commit
- Add the option for fp32 residual connection (fp32 residual connection... · 62632d39
  mohammad authored Dec 21, 2020
```
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)
```
  62632d39
12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

11 Sep, 2020 1 commit
- Various speed optimizations. · 51a2e6b0
  Vijay Korthikanti authored Sep 11, 2020
  
  51a2e6b0
03 Sep, 2020 1 commit
- debuged the cpu init for mp=1 · 04cd82bd
  mohammad authored Sep 03, 2020
  
  04cd82bd
27 Aug, 2020 1 commit
- Addressing code review comments · 1eafa861
  Boris Fomitchev authored Aug 26, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  1eafa861
26 Aug, 2020 1 commit
- Making --lazy_mpu_init act as _USE_CPU_INITIALIZATION · e10760ea
  Boris Fomitchev authored Aug 26, 2020
  
  e10760ea
03 Aug, 2020 1 commit
- added gpu initialization and option to avoid master values · 0403b808
  mohammad authored Aug 03, 2020
  
  0403b808
31 Jul, 2020 1 commit
- Code review comments - changing parallel test condition · 35bea728
  Boris Fomitchev authored Jul 30, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  35bea728
30 Jul, 2020 1 commit
- Changes for NeMo/lightning compatibility · 417c7f6a
  Boris Fomitchev authored Jul 30, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  417c7f6a
26 Jul, 2020 1 commit
- Complete fix for APEX absence in NeMo · aa0ee72e
  Boris Fomitchev authored Jul 25, 2020
  
  aa0ee72e
24 Jul, 2020 1 commit
- ONNX export fix, including one for gelu() · 158a99c3
  Boris Fomitchev authored Jul 23, 2020
```
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
```
  158a99c3
16 Apr, 2020 1 commit
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
14 Apr, 2020 1 commit
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
10 Feb, 2020 1 commit
- Model parallel merger · 57c2060f
  Mohammad Shoeybi authored Feb 10, 2020
  
  57c2060f
08 Oct, 2019 1 commit
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb
30 Jul, 2019 1 commit

large update including model parallelism and gpt2 · abe36e2e

Raul Puri authored Jul 29, 2019


Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

abe36e2e