Commits · e8fb052f810ab799ff947cbeb16f962ae32fc3b4 · OpenDAS / Megatron-LM

19 Aug, 2021 1 commit
- made contiguous buffer in local ddp default · e8fb052f
  mshoeybi authored Aug 18, 2021
  
  e8fb052f
11 Aug, 2021 1 commit
- added asserts/checks for local ddp and params_have_main_grad · f597f02e
  Lawrence McAfee authored Aug 11, 2021
  
  f597f02e
16 Jul, 2021 1 commit
- (conditionally) release grad/main_grad memory in copy_model_grads_to_main_grads, after copy · 6f2bff5c
  Lawrence McAfee authored Jul 16, 2021
  
  6f2bff5c
12 Jul, 2021 1 commit
- added comment explaining why fp32_from_float16_groups should be zeroed here · 4e64903d
  Lawrence McAfee authored Jul 12, 2021
  
  4e64903d
09 Jul, 2021 1 commit
- fixed zero_grad for fp32_from_float16_groups · 4fd6432d
  Lawrence McAfee authored Jul 09, 2021
  
  4fd6432d
08 Mar, 2021 1 commit
- Bfloat with fp32 grad acc · b4bc51b1
  Mohammad Shoeybi authored Mar 08, 2021
  
  b4bc51b1
04 Mar, 2021 1 commit
- Update argument names and fix merge error · 0aff3629
  Rewon Child authored Mar 04, 2021
  
  0aff3629
23 Feb, 2021 1 commit
- Add option to log zeros · 4e77e7c6
  Rewon Child authored Feb 23, 2021
  
  4e77e7c6
24 Jan, 2021 1 commit
- added grad norm to logging and tensorboard · 00764415
  mohammad authored Jan 23, 2021
  
  00764415
05 Jan, 2021 4 commits
- addressed Jareds and Deepaks comments · db88a27b
  mohammad authored Jan 05, 2021
  
  db88a27b
- some small fixes · 9eedf896
  mohammad authored Jan 04, 2021
  
  9eedf896
- further refactor, tested, and changed master to main · 22fa9bac
  mohammad authored Jan 04, 2021
  
  22fa9bac
- added reload model params for finetuning · 160ba680
  mohammad authored Jan 04, 2021
  
  160ba680
31 Dec, 2020 1 commit
- addressed deepaks comments · 43529f78
  mohammad authored Dec 30, 2020
  
  43529f78
30 Dec, 2020 5 commits
- done with refactoring and checks · b77d9062
  mohammad authored Dec 29, 2020
  
  b77d9062
- add multi-tensor-apply to clip grad · 9a010310
  mohammad authored Dec 29, 2020
  
  9a010310
- moved grad scaling to multi-tensor-apply · 345f5d0d
  mohammad authored Dec 29, 2020
  
  345f5d0d
- further refactor, matching old results · e68ea218
  mohammad authored Dec 29, 2020
  
  e68ea218
- slight refactor of clip grads · b0a3fdfe
  mohammad authored Dec 29, 2020
  
  b0a3fdfe
28 Dec, 2020 1 commit
- found a bug in l2 norm calculation · 6191ff59
  Mohammad authored Dec 27, 2020
  
  6191ff59
27 Dec, 2020 1 commit
- load and save state dicts added · 97ba5c0e
  mohammad authored Dec 26, 2020
  
  97ba5c0e
25 Dec, 2020 7 commits
- further refactoring · 0888a3e1
  mohammad authored Dec 25, 2020
  
  0888a3e1
- grads is removed from mpu · dfd8ed47
  mohammad authored Dec 25, 2020
  
  dfd8ed47
- clip grad fixed and moved to optimizer · 46879674
  mohammad authored Dec 25, 2020
  
  46879674
- moved entire optimizer build and tested · 28062e14
  mohammad authored Dec 24, 2020
  
  28062e14
- megatron optimizer tested, before working on clip grad · fb218c9d
  mohammad authored Dec 24, 2020
  
  fb218c9d
- fp32 is also working · 2eaa3ccc
  mohammad authored Dec 24, 2020
  
  2eaa3ccc
- working on the new optimizer · c6a58e41
  mohammad authored Dec 24, 2020
  
  c6a58e41