Commits · 67e23459169231d3a0c6dd43b4a875c030db0c41 · OpenDAS / Megatron-LM

"example/13_pool2d_fwd/README.md" did not exist on "5d37d7bff4e631c3b94112c31a52f209ca39dfe2"

01 Mar, 2022 1 commit
- more debugging; some cleanup · 67e23459
  Lawrence McAfee authored Feb 28, 2022
  
  67e23459
28 Feb, 2022 1 commit
- clip grad reduce across world. [ not just model group ] · a58653dc
  Lawrence McAfee authored Feb 28, 2022
  
  a58653dc
23 Feb, 2022 3 commits
- isolated grad discrepency to BaseFloat16Optimizer.step() · 371a8828
  Lawrence McAfee authored Feb 23, 2022
  
  371a8828
- renamed reduce_gradients -> reduce_grads [ matches gather_params ] · e7f0cdee
  Lawrence McAfee authored Feb 23, 2022
  
  e7f0cdee
- fixed param_world_shard bug. · 4b843668
  Lawrence McAfee authored Feb 23, 2022
  
  4b843668
22 Feb, 2022 4 commits
- debugging; localized issue to gather_params() · c13c0a3e
  Lawrence McAfee authored Feb 22, 2022
  
  c13c0a3e
- more cleanup of main params/grads · 9b7854e4
  Lawrence McAfee authored Feb 22, 2022
  
  9b7854e4
- removed zero-size optimizer group shards. · 291592e4
  Lawrence McAfee authored Feb 22, 2022
  
  291592e4
- set main_param_shard==None if local group_size==0 [ e.g., word embedding params · 23f9238d
  Lawrence McAfee authored Feb 22, 2022
  
  23f9238d
18 Feb, 2022 5 commits
- small fixes; training & learning! · ac5ef637
  Lawrence McAfee authored Feb 18, 2022
  
  ac5ef637
- copy model params to main params at very start · a4f41882
  Lawrence McAfee authored Feb 18, 2022
  
  a4f41882
- some cleanup · ae6cd9b5
  Lawrence McAfee authored Feb 18, 2022
  
  ae6cd9b5
- modularized reduce_gradients, gather params; trainin runs, but loss==nan · e5db0fda
  Lawrence McAfee authored Feb 18, 2022
  
  e5db0fda
- fixed clip_grad_norm bug · a7782b21
  Lawrence McAfee authored Feb 18, 2022
  
  a7782b21
16 Feb, 2022 1 commit
- small changes · d1f5776a
  Lawrence McAfee authored Feb 15, 2022
  
  d1f5776a
15 Feb, 2022 7 commits
- reduce scatter working · ff4c4273
  Lawrence McAfee authored Feb 15, 2022
  
  ff4c4273
- added 'all' world shards to gbuf map · bf64c85c
  Lawrence McAfee authored Feb 15, 2022
  
  bf64c85c
- updated zero_grad · eaa0c1df
  Lawrence McAfee authored Feb 15, 2022
  
  eaa0c1df
- allocated main params/grads · 33626179
  Lawrence McAfee authored Feb 15, 2022
  
  33626179
- created optimizer group shards · 525a8351
  Lawrence McAfee authored Feb 15, 2022
  
  525a8351
- modularized shard indexing · f498a6fe
  Lawrence McAfee authored Feb 15, 2022
  
  f498a6fe
- wip; switching to grad-buffer-centric design · cb6f96b6
  Lawrence McAfee authored Feb 15, 2022
  
  cb6f96b6
14 Feb, 2022 8 commits
- todo; align shards with model's contiguous buffer · a3f3c3ad
  Lawrence McAfee authored Feb 14, 2022
  
  a3f3c3ad
- copying model grad slices to main grad · 3f0bc681
  Lawrence McAfee authored Feb 14, 2022
  
  3f0bc681
- fix zero_grad; set_to_none = False · 6875dff5
  Lawrence McAfee authored Feb 14, 2022
  
  6875dff5
- tweaked slice index naming convention · 1215c420
  Lawrence McAfee authored Feb 14, 2022
  
  1215c420
- map param to originating virtual model; eventually move this to constructor · c5f93269
  Lawrence McAfee authored Feb 14, 2022
  
  c5f93269
- included original param index in map · 3ded2425
  Lawrence McAfee authored Feb 14, 2022
  
  3ded2425
- built local shard param index map · a74e245c
  Lawrence McAfee authored Feb 14, 2022
  
  a74e245c
- collect param offsets for contiguous grad buffer · f7232502
  Lawrence McAfee authored Feb 14, 2022
  
  f7232502
11 Feb, 2022 2 commits
- bit more progress · 5706ba42
  Lawrence McAfee authored Feb 11, 2022
  
  5706ba42
- studied float16 optimizer; more updates · f48e1f29
  Lawrence McAfee authored Feb 11, 2022
  
  f48e1f29
10 Feb, 2022 2 commits
- more work on Float16DistributedOptimizer · 49cca4d9
  Lawrence McAfee authored Feb 10, 2022
  
  49cca4d9
- working on Float16DistributedOptimizer · 329fe582
  Lawrence McAfee authored Feb 10, 2022
  
  329fe582
09 Feb, 2022 1 commit
- feb 9 alpha · 7dc8c475
  Lawrence McAfee authored Feb 09, 2022
  
  7dc8c475
26 Jan, 2022 1 commit
- address review comments · 8acbbe25
  Vijay Korthikanti authored Jan 26, 2022
  
  8acbbe25
22 Jan, 2022 1 commit
- second phase of vision code merge · d8c85650
  Vijay Korthikanti authored Jan 21, 2022
  
  d8c85650
12 Jan, 2022 1 commit
- Phase1 merge: vit optimizations + dataset enhancements + scaled_softmax kernel · 7a77abd9
  Vijay Korthikanti authored Jan 12, 2022
  
  7a77abd9
19 Aug, 2021 1 commit
- made contiguous buffer in local ddp default · e8fb052f
  mshoeybi authored Aug 18, 2021
  
  e8fb052f
11 Aug, 2021 1 commit
- added asserts/checks for local ddp and params_have_main_grad · f597f02e
  Lawrence McAfee authored Aug 11, 2021
  
  f597f02e