Commits · 57bfa7caba01bf96befd292e02ffe79fc744f1bd · OpenDAS / Megatron-LM

30 Sep, 2022 1 commit
- Perform distributed optimizer's all-gather in param dtype (instead of grad dtype) · 57bfa7ca
  Lawrence McAfee authored Sep 30, 2022
  
  57bfa7ca
16 Aug, 2022 2 commits
- fixed grad scalar warning for bf16 · aaa5715a
  Mohammad Shoeybi authored Aug 16, 2022
  
  aaa5715a
- fixed grad scalar warning so it only prints it for fp16 · 27bc1338
  mshoeybi authored Aug 15, 2022
  
  27bc1338
10 Aug, 2022 1 commit
- Timing levels · 77efcccb
  Mohammad Shoeybi authored Aug 10, 2022
  
  77efcccb
08 Jun, 2022 2 commits
- re-added grad dtype assertion in clip_grads. · ef63acd2
  Lawrence McAfee authored Jun 08, 2022
  
  ef63acd2
- removed count-zeros debuggables. · 9044bc42
  Lawrence McAfee authored Jun 08, 2022
  
  9044bc42
01 Jun, 2022 1 commit
- testing fix w/ clip_grads. · 7fccd6a1
  Lawrence McAfee authored Jun 01, 2022
  
  7fccd6a1
24 May, 2022 2 commits
- allreduce layernorm fixes. · de593298
  Lawrence McAfee authored May 24, 2022
  
  de593298
- moved SP's layernorm all-reduce into optimizer.py. · ee57e086
  Lawrence McAfee authored May 24, 2022
  
  ee57e086
17 May, 2022 1 commit
- address review comments · 356eb36a
  Vijay Korthikanti authored May 16, 2022
  
  356eb36a
28 Apr, 2022 2 commits
- reverting pre allocation of optimizer states; it does not seem to help with fragmentation · 5d2e13a6
  Vijay Korthikanti authored Apr 28, 2022
  
  5d2e13a6
- address review comments · cfd2e216
  Vijay Korthikanti authored Apr 28, 2022
  
  cfd2e216
20 Apr, 2022 1 commit
- missing import. · 1cc21017
  Lawrence McAfee authored Apr 20, 2022
  
  1cc21017
30 Mar, 2022 1 commit
- sequence parallelism for embedding dropout and last linear layer + memory optimizations · eec218d8
  Vijay Korthikanti authored Mar 30, 2022
  
  eec218d8
29 Mar, 2022 8 commits
- commented distrib_optimizer.py. · b42c3052
  Lawrence McAfee authored Mar 29, 2022
  
  b42c3052
- partially commented distrib_optimizer.py. · fb3a1345
  Lawrence McAfee authored Mar 29, 2022
  
  fb3a1345
- replace triple single w/ triple double quote. · fe2d623e
  Lawrence McAfee authored Mar 29, 2022
  
  fe2d623e
- renamed: full_ -> model_. · 7ed649ed
  Lawrence McAfee authored Mar 29, 2022
  
  7ed649ed
- commented optimizer.py. · 82e6730c
  Lawrence McAfee authored Mar 29, 2022
  
  82e6730c
- commented clip_grads.py. · 03ba8eb1
  Lawrence McAfee authored Mar 29, 2022
  
  03ba8eb1
- removed debug_base/main/model methods. · 087cbff1
  Lawrence McAfee authored Mar 29, 2022
  
  087cbff1
- fixed shared weight attribute for fp32. · be8de1b3
  Lawrence McAfee authored Mar 29, 2022
  
  be8de1b3
28 Mar, 2022 2 commits
- some fixes. · 2c9ed910
  Lawrence McAfee authored Mar 28, 2022
  
  2c9ed910
- correctly copying tensor parallel attributes for fp32_groups. · b1781f31
  Lawrence McAfee authored Mar 28, 2022
  
  b1781f31
25 Mar, 2022 7 commits
- debugging fp32 training. · d0925652
  Lawrence McAfee authored Mar 25, 2022
  
  d0925652
- working: checkpoint save/load. · 9f64f5f4
  Lawrence McAfee authored Mar 25, 2022
  
  9f64f5f4
- more cleanup; timing tests vs. 'dupe' dist-opt. · 82491e4b
  Lawrence McAfee authored Mar 24, 2022
  
  82491e4b
- cleanup. · 91f3579e
  Lawrence McAfee authored Mar 24, 2022
  
  91f3579e
- losses match. · 4b9a218e
  Lawrence McAfee authored Mar 24, 2022
  
  4b9a218e
- training; loss not yet checked. · 41ad0c5d
  Lawrence McAfee authored Mar 24, 2022
  
  41ad0c5d
- working: copy grads. · fe3cfd86
  Lawrence McAfee authored Mar 24, 2022
  
  fe3cfd86
24 Mar, 2022 3 commits
- stalling in copy_grads(). · 5e29905f
  Lawrence McAfee authored Mar 24, 2022
  
  5e29905f
- many edits; working towards first draft. · 2c3cb9fc
  Lawrence McAfee authored Mar 24, 2022
  
  2c3cb9fc
- getting started with no-grad-duplication implementation. · 867105c2
  Lawrence McAfee authored Mar 23, 2022
  
  867105c2
23 Mar, 2022 5 commits
- fixed clip_grads for when len(grads_for_norm) == 0. · c52c9dbf
  Lawrence McAfee authored Mar 23, 2022
  
  c52c9dbf
- added timers for fp32 optimizer. · dcf2ccc9
  Lawrence McAfee authored Mar 23, 2022
  
  dcf2ccc9
- added timing for count-zeros & inner-step. · 80c91145
  Lawrence McAfee authored Mar 23, 2022
  
  80c91145
- moved 'get_main_grads_for_grad_norm()'; for fp16/fp32 sharing. · e858d338
  Lawrence McAfee authored Mar 23, 2022
  
  e858d338
- removed mem savings factor. +cleanup. · 11581195
  Lawrence McAfee authored Mar 23, 2022
  
  11581195
22 Mar, 2022 1 commit
- running & saving memory w/ _reduce_scatter_base/_all_gather_base. · 41038d54
  Lawrence McAfee authored Mar 22, 2022
  
  41038d54