Commits · fe3cfd86e5c25418000c4c90f7b1ea9d3448c90b · OpenDAS / Megatron-LM

25 Mar, 2022 1 commit
- working: copy grads. · fe3cfd86
  Lawrence McAfee authored Mar 24, 2022
  
  fe3cfd86
24 Mar, 2022 3 commits
- stalling in copy_grads(). · 5e29905f
  Lawrence McAfee authored Mar 24, 2022
  
  5e29905f
- many edits; working towards first draft. · 2c3cb9fc
  Lawrence McAfee authored Mar 24, 2022
  
  2c3cb9fc
- getting started with no-grad-duplication implementation. · 867105c2
  Lawrence McAfee authored Mar 23, 2022
  
  867105c2
23 Mar, 2022 7 commits
- fixed clip_grads for when len(grads_for_norm) == 0. · c52c9dbf
  Lawrence McAfee authored Mar 23, 2022
  
  c52c9dbf
- added timers for fp32 optimizer. · dcf2ccc9
  Lawrence McAfee authored Mar 23, 2022
  
  dcf2ccc9
- renamed timers. · 2706edc4
  Lawrence McAfee authored Mar 23, 2022
  
  2706edc4
- added timers for reduce-grads/gather-params. · cad2c80a
  Lawrence McAfee authored Mar 23, 2022
  
  cad2c80a
- added timing for count-zeros & inner-step. · 80c91145
  Lawrence McAfee authored Mar 23, 2022
  
  80c91145
- moved 'get_main_grads_for_grad_norm()'; for fp16/fp32 sharing. · e858d338
  Lawrence McAfee authored Mar 23, 2022
  
  e858d338
- removed mem savings factor. +cleanup. · 11581195
  Lawrence McAfee authored Mar 23, 2022
  
  11581195
22 Mar, 2022 4 commits
- running & saving memory w/ _reduce_scatter_base/_all_gather_base. · 41038d54
  Lawrence McAfee authored Mar 22, 2022
  
  41038d54
- setup code to try _reduce_scatter_base, _all_gather_base. · 64b94f00
  Lawrence McAfee authored Mar 22, 2022
  
  64b94f00
- grad norm 'matches' (not bitwise equal). · 6728a780
  Lawrence McAfee authored Mar 22, 2022
  
  6728a780
- implemented 'get_grad_views_for_grad_norm()'. · c6d20c05
  Lawrence McAfee authored Mar 22, 2022
  
  c6d20c05
21 Mar, 2022 1 commit
- grad norm debugging. · 0481f58e
  Lawrence McAfee authored Mar 21, 2022
  
  0481f58e
18 Mar, 2022 6 commits
- cleanup. · 3d433e8a
  Lawrence McAfee authored Mar 18, 2022
  
  3d433e8a
- using smaller chunk_numel_min. · 19730725
  Lawrence McAfee authored Mar 18, 2022
  
  19730725
- mem savings factor training; need to check loss. · ed90a1b2
  Lawrence McAfee authored Mar 18, 2022
  
  ed90a1b2
- added arg '--distrib-opt-comm-mem-savings'. · 06b9ebe0
  Lawrence McAfee authored Mar 18, 2022
  
  06b9ebe0
- after experiments, before new arg 'distrib-opt-comm-buf-1m'. · 34fc1e37
  Lawrence McAfee authored Mar 18, 2022
  
  34fc1e37
- split reduce_grads/gather_params using gbuf sub-views. · 12d91733
  Lawrence McAfee authored Mar 18, 2022
  
  12d91733
14 Mar, 2022 11 commits
- cleaned arguments.py. · e30ad67e
  Lawrence McAfee authored Mar 14, 2022
  
  e30ad67e
- cleaned distributed.py. · 14e60427
  Lawrence McAfee authored Mar 14, 2022
  
  14e60427
- cleaned distrib_optimizer.py. · 2c1660e7
  Lawrence McAfee authored Mar 14, 2022
  
  2c1660e7
- partially cleaned optimizer.py. · efa3cbcf
  Lawrence McAfee authored Mar 14, 2022
  
  efa3cbcf
- cleaned optimizer/__init__.py. · e6120623
  Lawrence McAfee authored Mar 14, 2022
  
  e6120623
- cleaned clip_grads.py. · a4afb428
  Lawrence McAfee authored Mar 14, 2022
  
  a4afb428
- cleaned training.py · 787882a0
  Lawrence McAfee authored Mar 14, 2022
  
  787882a0
- cleaned/commented checkpointing.py. · 0f2a9f19
  Lawrence McAfee authored Mar 14, 2022
  
  0f2a9f19
- more specific formatting of model/optim checkpoint paths. · 9c86abd9
  Lawrence McAfee authored Mar 14, 2022
  
  9c86abd9
- optimizer saves list(group), not list(param). · af2b136f
  Lawrence McAfee authored Mar 14, 2022
  
  af2b136f
- started updating checkpointing.py. · 37ca7859
  Lawrence McAfee authored Mar 14, 2022
  
  37ca7859
11 Mar, 2022 3 commits
- guard 'gather_params()' with 'if update_successful:' · f6811e28
  Lawrence McAfee authored Mar 11, 2022
  
  f6811e28
- tiny change. · c64098ef
  Lawrence McAfee authored Mar 10, 2022
  
  c64098ef
- Merge branch 'main' into lmcafee/distrib-opt · 5fb0ff9a
  Lawrence McAfee authored Mar 10, 2022
  
  5fb0ff9a
10 Mar, 2022 2 commits
- small fixes. · 862d70fc
  Lawrence McAfee authored Mar 10, 2022
  
  862d70fc
- updated args for allreduce_embeddings · 08ee8ea2
  Lawrence McAfee authored Mar 10, 2022
  
  08ee8ea2
09 Mar, 2022 2 commits
- updated FP32Optimizer for latest changes. · c88bc979
  Lawrence McAfee authored Mar 09, 2022
  
  c88bc979
- removed old (pre-gbuf-padding) get_model_grad_buffer_dp_views(). · 7ac342b7
  Lawrence McAfee authored Mar 09, 2022
  
  7ac342b7