Commits · 57bfa7caba01bf96befd292e02ffe79fc744f1bd · OpenDAS / Megatron-LM · GitLab

30 Sep, 2022 1 commit
- Perform distributed optimizer's all-gather in param dtype (instead of grad dtype) · 57bfa7ca
  Lawrence McAfee authored Sep 30, 2022
  
  57bfa7ca
16 Aug, 2022 2 commits
- fixed grad scalar warning for bf16 · aaa5715a
  Mohammad Shoeybi authored Aug 16, 2022
  
  aaa5715a
- fixed grad scalar warning so it only prints it for fp16 · 27bc1338
  mshoeybi authored Aug 15, 2022
  
  27bc1338
10 Aug, 2022 1 commit
- Timing levels · 77efcccb
  Mohammad Shoeybi authored Aug 10, 2022
  
  77efcccb
24 May, 2022 2 commits
- allreduce layernorm fixes. · de593298
  Lawrence McAfee authored May 24, 2022
  
  de593298
- moved SP's layernorm all-reduce into optimizer.py. · ee57e086
  Lawrence McAfee authored May 24, 2022
  
  ee57e086
28 Apr, 2022 1 commit
- address review comments · cfd2e216
  Vijay Korthikanti authored Apr 28, 2022
  
  cfd2e216
30 Mar, 2022 1 commit
- sequence parallelism for embedding dropout and last linear layer + memory optimizations · eec218d8
  Vijay Korthikanti authored Mar 30, 2022
  
  eec218d8
29 Mar, 2022 3 commits
- replace triple single w/ triple double quote. · fe2d623e
  Lawrence McAfee authored Mar 29, 2022
  
  fe2d623e
- commented optimizer.py. · 82e6730c
  Lawrence McAfee authored Mar 29, 2022
  
  82e6730c
- removed debug_base/main/model methods. · 087cbff1
  Lawrence McAfee authored Mar 29, 2022
  
  087cbff1
28 Mar, 2022 2 commits
- some fixes. · 2c9ed910
  Lawrence McAfee authored Mar 28, 2022
  
  2c9ed910
- correctly copying tensor parallel attributes for fp32_groups. · b1781f31
  Lawrence McAfee authored Mar 28, 2022
  
  b1781f31
25 Mar, 2022 2 commits
- more cleanup; timing tests vs. 'dupe' dist-opt. · 82491e4b
  Lawrence McAfee authored Mar 24, 2022
  
  82491e4b
- working: copy grads. · fe3cfd86
  Lawrence McAfee authored Mar 24, 2022
  
  fe3cfd86
24 Mar, 2022 3 commits
- stalling in copy_grads(). · 5e29905f
  Lawrence McAfee authored Mar 24, 2022
  
  5e29905f
- many edits; working towards first draft. · 2c3cb9fc
  Lawrence McAfee authored Mar 24, 2022
  
  2c3cb9fc
- getting started with no-grad-duplication implementation. · 867105c2
  Lawrence McAfee authored Mar 23, 2022
  
  867105c2
23 Mar, 2022 4 commits
- added timers for fp32 optimizer. · dcf2ccc9
  Lawrence McAfee authored Mar 23, 2022
  
  dcf2ccc9
- added timing for count-zeros & inner-step. · 80c91145
  Lawrence McAfee authored Mar 23, 2022
  
  80c91145
- moved 'get_main_grads_for_grad_norm()'; for fp16/fp32 sharing. · e858d338
  Lawrence McAfee authored Mar 23, 2022
  
  e858d338
- removed mem savings factor. +cleanup. · 11581195
  Lawrence McAfee authored Mar 23, 2022
  
  11581195
22 Mar, 2022 2 commits
- grad norm 'matches' (not bitwise equal). · 6728a780
  Lawrence McAfee authored Mar 22, 2022
  
  6728a780
- implemented 'get_grad_views_for_grad_norm()'. · c6d20c05
  Lawrence McAfee authored Mar 22, 2022
  
  c6d20c05
21 Mar, 2022 1 commit
- grad norm debugging. · 0481f58e
  Lawrence McAfee authored Mar 21, 2022
  
  0481f58e
18 Mar, 2022 2 commits
- mem savings factor training; need to check loss. · ed90a1b2
  Lawrence McAfee authored Mar 18, 2022
  
  ed90a1b2
- split reduce_grads/gather_params using gbuf sub-views. · 12d91733
  Lawrence McAfee authored Mar 18, 2022
  
  12d91733
14 Mar, 2022 1 commit
- partially cleaned optimizer.py. · efa3cbcf
  Lawrence McAfee authored Mar 14, 2022
  
  efa3cbcf
11 Mar, 2022 1 commit
- guard 'gather_params()' with 'if update_successful:' · f6811e28
  Lawrence McAfee authored Mar 11, 2022
  
  f6811e28
10 Mar, 2022 2 commits
- small fixes. · 862d70fc
  Lawrence McAfee authored Mar 10, 2022
  
  862d70fc
- updated args for allreduce_embeddings · 08ee8ea2
  Lawrence McAfee authored Mar 10, 2022
  
  08ee8ea2
09 Mar, 2022 9 commits
- updated FP32Optimizer for latest changes. · c88bc979
  Lawrence McAfee authored Mar 09, 2022
  
  c88bc979
- removed faulty (already commented out) gather_model_params param iterator. · 94a90215
  Lawrence McAfee authored Mar 09, 2022
  
  94a90215
- renamed reduce_grads/gather_params -> reduce_model_grads/gather_model_params · 64b9d28c
  Lawrence McAfee authored Mar 09, 2022
  
  64b9d28c
- passing args, timers from train_step to optimizer methods. · d58d1762
  Lawrence McAfee authored Mar 09, 2022
  
  d58d1762
- consolidated reduce_grad's sub-methods (i.e., allreduce_embedding_grads) · 45b364b1
  Lawrence McAfee authored Mar 09, 2022
  
  45b364b1
- passing 'model_parallel_group' to clip_grads, count_zeros · 9546d8f0
  Lawrence McAfee authored Mar 09, 2022
  
  9546d8f0
- moved 'reduce_grads()' to MegatronOptimizer. · e46230dc
  Lawrence McAfee authored Mar 09, 2022
  
  e46230dc
- encapsulated 'get_model_parallel_group()'; renamed BaseFloat16Optimizer -> MixedPrecisionOptimizer · 772a4a2d
  Lawrence McAfee authored Mar 09, 2022
  
  772a4a2d
- removed clip_grad_norm shortcut. · 6e888151
  Lawrence McAfee authored Mar 09, 2022
  
  6e888151