Commits · aff99e32b48582af674fd0ffbd3452252d4576a3 · OpenDAS / Megatron-LM

24 May, 2022 1 commit
- added timer for layernorm all-reduce. · aff99e32
  Lawrence McAfee authored May 24, 2022
  
  aff99e32
09 May, 2022 1 commit
- (1)support pyt20.12 compatibility, (2) arg naming update · f602ac56
  Sangkug Lym authored Apr 11, 2022
```
change dummy_handler to nullcontext
```
  f602ac56
28 Apr, 2022 1 commit
- address review comments · cfd2e216
  Vijay Korthikanti authored Apr 28, 2022
  
  cfd2e216
30 Mar, 2022 1 commit
- sequence parallelism for embedding dropout and last linear layer + memory optimizations · eec218d8
  Vijay Korthikanti authored Mar 30, 2022
  
  eec218d8
25 Mar, 2022 3 commits
- debugging fp32 training. · d0925652
  Lawrence McAfee authored Mar 25, 2022
  
  d0925652
- working: checkpoint save/load. · 9f64f5f4
  Lawrence McAfee authored Mar 25, 2022
  
  9f64f5f4
- jit functions warmpups to (1) match fprop and recompute results and (2) remove grad_enable · c7fef593
  Sangkug Lym authored Mar 24, 2022
  
  c7fef593
23 Mar, 2022 3 commits
- renamed timers. · 2706edc4
  Lawrence McAfee authored Mar 23, 2022
  
  2706edc4
- added timers for reduce-grads/gather-params. · cad2c80a
  Lawrence McAfee authored Mar 23, 2022
  
  cad2c80a
- added timing for count-zeros & inner-step. · 80c91145
  Lawrence McAfee authored Mar 23, 2022
  
  80c91145
14 Mar, 2022 1 commit
- cleaned training.py · 787882a0
  Lawrence McAfee authored Mar 14, 2022
  
  787882a0
11 Mar, 2022 2 commits
- guard 'gather_params()' with 'if update_successful:' · f6811e28
  Lawrence McAfee authored Mar 11, 2022
  
  f6811e28
- tiny change. · c64098ef
  Lawrence McAfee authored Mar 10, 2022
  
  c64098ef
10 Mar, 2022 1 commit
- small fixes. · 862d70fc
  Lawrence McAfee authored Mar 10, 2022
  
  862d70fc
09 Mar, 2022 3 commits
- renamed reduce_grads/gather_params -> reduce_model_grads/gather_model_params · 64b9d28c
  Lawrence McAfee authored Mar 09, 2022
  
  64b9d28c
- passing args, timers from train_step to optimizer methods. · d58d1762
  Lawrence McAfee authored Mar 09, 2022
  
  d58d1762
- consolidated reduce_grad's sub-methods (i.e., allreduce_embedding_grads) · 45b364b1
  Lawrence McAfee authored Mar 09, 2022
  
  45b364b1
03 Mar, 2022 1 commit
- get sequence parallelism to work with pipeline parallelism · 6658158b
  Vijay Korthikanti authored Mar 02, 2022
  
  6658158b
02 Mar, 2022 2 commits
- layernorm grad sync + name chnages · c0f10643
  Vijay Korthikanti authored Mar 02, 2022
  
  c0f10643
- fixed gather params; now copying all params, not just local dp subset. · f8fd34e3
  Lawrence McAfee authored Mar 02, 2022
  
  f8fd34e3
01 Mar, 2022 1 commit
- fixed 'count zeros' for distrib opt · cbcd5579
  Lawrence McAfee authored Mar 01, 2022
  
  cbcd5579
23 Feb, 2022 1 commit
- isolated grad discrepency to BaseFloat16Optimizer.step() · 371a8828
  Lawrence McAfee authored Feb 23, 2022
  
  371a8828
22 Feb, 2022 1 commit
- debugging; localized issue to gather_params() · c13c0a3e
  Lawrence McAfee authored Feb 22, 2022
  
  c13c0a3e
15 Feb, 2022 4 commits
- adress review comments · 488f8c02
  Vijay Korthikanti authored Feb 11, 2022
  
  488f8c02
- minor fixes · 6f3bf9c0
  Vijay Korthikanti authored Feb 01, 2022
  
  6f3bf9c0
- vision third phase merge: pretraining methods + mit,swin backbones · 48c2a144
  Vijay Korthikanti authored Jan 31, 2022
  
  48c2a144
- wip; switching to grad-buffer-centric design · cb6f96b6
  Lawrence McAfee authored Feb 15, 2022
  
  cb6f96b6
11 Feb, 2022 1 commit
- adress review comments · 4554c3fe
  Vijay Korthikanti authored Feb 11, 2022
  
  4554c3fe
10 Feb, 2022 2 commits
- more work on Float16DistributedOptimizer · 49cca4d9
  Lawrence McAfee authored Feb 10, 2022
  
  49cca4d9
- working on Float16DistributedOptimizer · 329fe582
  Lawrence McAfee authored Feb 10, 2022
  
  329fe582
01 Feb, 2022 3 commits
- comments, cleanup. · b93bef00
  Lawrence McAfee authored Feb 01, 2022
  
  b93bef00
- minor fixes · e1f9c3a5
  Vijay Korthikanti authored Feb 01, 2022
  
  e1f9c3a5
- vision third phase merge: pretraining methods + mit,swin backbones · 01a82723
  Vijay Korthikanti authored Jan 31, 2022
  
  01a82723
31 Jan, 2022 1 commit
- working for t5 [ encoder embedding only ] · 3af6725d
  Lawrence McAfee authored Jan 31, 2022
  
  3af6725d
29 Jan, 2022 1 commit
- narrowed issue to pipeline rank 0, virtual pipeline rank >= 1 · 5bc9f889
  Lawrence McAfee authored Jan 28, 2022
  
  5bc9f889
28 Jan, 2022 2 commits
- more naming cleanup · 641408f5
  Vijay Korthikanti authored Jan 28, 2022
  
  641408f5
- changing class name AnnealingLR to OptimizerParamScheduler · 04ecc834
  Vijay Korthikanti authored Jan 28, 2022
  
  04ecc834
27 Jan, 2022 1 commit
- address review comments · 53931b8b
  Vijay Korthikanti authored Jan 27, 2022
  
  53931b8b
26 Jan, 2022 1 commit
- address review comments · 8acbbe25
  Vijay Korthikanti authored Jan 26, 2022
  
  8acbbe25
24 Jan, 2022 1 commit
- added args.transformer_pipeline_model_parallel_size · a06af061
  Lawrence McAfee authored Jan 24, 2022
  
  a06af061