Commits · 787882a01098ea353d69f9b7d675b11dc696d547 · OpenDAS / Megatron-LM · GitLab

14 Mar, 2022 5 commits
- cleaned training.py · 787882a0
  Lawrence McAfee authored Mar 14, 2022
  
  787882a0
- cleaned/commented checkpointing.py. · 0f2a9f19
  Lawrence McAfee authored Mar 14, 2022
  
  0f2a9f19
- more specific formatting of model/optim checkpoint paths. · 9c86abd9
  Lawrence McAfee authored Mar 14, 2022
  
  9c86abd9
- optimizer saves list(group), not list(param). · af2b136f
  Lawrence McAfee authored Mar 14, 2022
  
  af2b136f
- started updating checkpointing.py. · 37ca7859
  Lawrence McAfee authored Mar 14, 2022
  
  37ca7859
11 Mar, 2022 3 commits
- guard 'gather_params()' with 'if update_successful:' · f6811e28
  Lawrence McAfee authored Mar 11, 2022
  
  f6811e28
- tiny change. · c64098ef
  Lawrence McAfee authored Mar 10, 2022
  
  c64098ef
- Merge branch 'main' into lmcafee/distrib-opt · 5fb0ff9a
  Lawrence McAfee authored Mar 10, 2022
  
  5fb0ff9a
10 Mar, 2022 2 commits
- small fixes. · 862d70fc
  Lawrence McAfee authored Mar 10, 2022
  
  862d70fc
- updated args for allreduce_embeddings · 08ee8ea2
  Lawrence McAfee authored Mar 10, 2022
  
  08ee8ea2
09 Mar, 2022 14 commits
- updated FP32Optimizer for latest changes. · c88bc979
  Lawrence McAfee authored Mar 09, 2022
  
  c88bc979
- removed old (pre-gbuf-padding) get_model_grad_buffer_dp_views(). · 7ac342b7
  Lawrence McAfee authored Mar 09, 2022
  
  7ac342b7
- more cleanup · 0528bf77
  Lawrence McAfee authored Mar 09, 2022
  
  0528bf77
- removed faulty (already commented out) gather_model_params param iterator. · 94a90215
  Lawrence McAfee authored Mar 09, 2022
  
  94a90215
- renamed reduce_grads/gather_params -> reduce_model_grads/gather_model_params · 64b9d28c
  Lawrence McAfee authored Mar 09, 2022
  
  64b9d28c
- passing args, timers from train_step to optimizer methods. · d58d1762
  Lawrence McAfee authored Mar 09, 2022
  
  d58d1762
- consolidated reduce_grad's sub-methods (i.e., allreduce_embedding_grads) · 45b364b1
  Lawrence McAfee authored Mar 09, 2022
  
  45b364b1
- finished cleaning clip_grads.py · a9b1fc0a
  Lawrence McAfee authored Mar 09, 2022
  
  a9b1fc0a
- passing 'model_parallel_group' to clip_grads, count_zeros · 9546d8f0
  Lawrence McAfee authored Mar 09, 2022
  
  9546d8f0
- moved 'reduce_grads()' to MegatronOptimizer. · e46230dc
  Lawrence McAfee authored Mar 09, 2022
  
  e46230dc
- encapsulated 'get_model_parallel_group()'; renamed BaseFloat16Optimizer -> MixedPrecisionOptimizer · 772a4a2d
  Lawrence McAfee authored Mar 09, 2022
  
  772a4a2d
- removed clip_grad_norm shortcut. · 6e888151
  Lawrence McAfee authored Mar 09, 2022
  
  6e888151
- moved distrib opt to own file. · 28b63ec7
  Lawrence McAfee authored Mar 09, 2022
  
  28b63ec7
- padded ddp's grad_buffer to multiple of data parallel world size · 55695f81
  Lawrence McAfee authored Mar 09, 2022
  
  55695f81
07 Mar, 2022 1 commit
- deleted old distributed_fused_adam · 4f2356dc
  Lawrence McAfee authored Mar 07, 2022
  
  4f2356dc
02 Mar, 2022 3 commits
- working w/ mixed parallelism [ dp, tp, pp ]. · e5bda3c9
  Lawrence McAfee authored Mar 02, 2022
  
  e5bda3c9
- copied t5 embed sync to distrib opt · 2f99b4f4
  Lawrence McAfee authored Mar 02, 2022
  
  2f99b4f4
- fixed gather params; now copying all params, not just local dp subset. · f8fd34e3
  Lawrence McAfee authored Mar 02, 2022
  
  f8fd34e3
01 Mar, 2022 2 commits
- fixed 'count zeros' for distrib opt · cbcd5579
  Lawrence McAfee authored Mar 01, 2022
  
  cbcd5579
- more debugging; some cleanup · 67e23459
  Lawrence McAfee authored Feb 28, 2022
  
  67e23459
28 Feb, 2022 1 commit
- clip grad reduce across world. [ not just model group ] · a58653dc
  Lawrence McAfee authored Feb 28, 2022
  
  a58653dc
24 Feb, 2022 2 commits
- Merge branch 'switch' into 'main' · 53f3efc4
  Lawrence McAfee authored Feb 24, 2022
```
Switch Transformer

See merge request ADLR/megatron-lm!373
```
  53f3efc4
- Fixing merge conflict · 458d7785
  rprenger authored Feb 23, 2022
  
  458d7785
23 Feb, 2022 4 commits
- isolated grad discrepency to BaseFloat16Optimizer.step() · 371a8828
  Lawrence McAfee authored Feb 23, 2022
  
  371a8828
- renamed reduce_gradients -> reduce_grads [ matches gather_params ] · e7f0cdee
  Lawrence McAfee authored Feb 23, 2022
  
  e7f0cdee
- fixed param_world_shard bug. · 4b843668
  Lawrence McAfee authored Feb 23, 2022
  
  4b843668
- Merge branch 'slym/grad_accum_fusion' into 'main' · 9c5a830f
  Jared Casper authored Feb 22, 2022
```
Gradient accumulation fusion

See merge request ADLR/megatron-lm!394
```
  9c5a830f
22 Feb, 2022 3 commits
- debugging; localized issue to gather_params() · c13c0a3e
  Lawrence McAfee authored Feb 22, 2022
  
  c13c0a3e
- more cleanup of main params/grads · 9b7854e4
  Lawrence McAfee authored Feb 22, 2022
  
  9b7854e4
- removed zero-size optimizer group shards. · 291592e4
  Lawrence McAfee authored Feb 22, 2022
  
  291592e4