Commits · 748a9a7db35cf04d53d814a5975f79f78ea4661d · OpenDAS / Megatron-LM

12 Jul, 2022 1 commit
- Add error checking by wrapping all weights and biases into named messages with named tensors. · 748a9a7d
  Jared Casper authored Jul 11, 2022
  
  748a9a7d
24 Jun, 2022 6 commits
- Merge branch 'main' into checkpoint_util · 2755bcb8
  Jared Casper authored Jun 24, 2022
  
  2755bcb8
- Merge branch 'ci' into 'main' · 3f4e71df
  Jared Casper authored Jun 24, 2022
```
Update ci image.

See merge request ADLR/megatron-lm!425
```
  3f4e71df
- Update ci image. · e075d830
  Jared Casper authored Jun 24, 2022
  
  e075d830
- Fixing up checkpointing. · 2725dc0b
  Jared Casper authored Jun 24, 2022
  
  2725dc0b
- Remove unused function. · 4eb802c4
  Jared Casper authored Jun 24, 2022
  
  4eb802c4
- Merge branch 'main' into checkpoint_util · 80389ef6
  Jared Casper authored Jun 24, 2022
  
  80389ef6
08 Jun, 2022 6 commits
- Merge branch 'lmcafee/distrib-opt-nodupe' into 'main' · d07d29df
  Jared Casper authored Jun 08, 2022
```
Distributed optimizer

See merge request ADLR/megatron-lm!408
```
  d07d29df
- fixed load_checkpoint(). · 74ba3797
  Lawrence McAfee authored Jun 08, 2022
  
  74ba3797
- error fixes & tested. · b178e6fc
  Lawrence McAfee authored Jun 08, 2022
  
  b178e6fc
- added backwards compatibility to checkpointing.py. · 977efdfb
  Lawrence McAfee authored Jun 08, 2022
  
  977efdfb
- re-added grad dtype assertion in clip_grads. · ef63acd2
  Lawrence McAfee authored Jun 08, 2022
  
  ef63acd2
- removed count-zeros debuggables. · 9044bc42
  Lawrence McAfee authored Jun 08, 2022
  
  9044bc42
01 Jun, 2022 1 commit
- testing fix w/ clip_grads. · 7fccd6a1
  Lawrence McAfee authored Jun 01, 2022
  
  7fccd6a1
26 May, 2022 3 commits
- Merge branch 'dp_src_rank' into 'main' · 55ba1f7b
  Jared Casper authored May 26, 2022
```
bug fix for get_data_parallel_src_rank

See merge request ADLR/megatron-lm!420
```
  55ba1f7b
- bug fix for get_data_parallel_src_rank · 9ad1944d
  Vijay Korthikanti authored May 26, 2022
  
  9ad1944d
- Merge branch 'beam_search' into 'main' · bb618c02
  Jared Casper authored May 25, 2022
```
Beam search

See merge request ADLR/megatron-lm!396
```
  bb618c02
25 May, 2022 2 commits
- separate file for license · da11c982
  Peng Xu authored May 25, 2022
  
  da11c982
- Merge branch 'sequence_parallel' into 'main' · d898a899
  Jared Casper authored May 25, 2022
```
preallocating global buffer to avoid memory fragmentation

See merge request ADLR/megatron-lm!419
```
  d898a899
24 May, 2022 9 commits
- resolved review comments · 739cb43d
  Vijay Korthikanti authored May 24, 2022
  
  739cb43d
- allreduce layernorm fixes. · de593298
  Lawrence McAfee authored May 24, 2022
  
  de593298
- fixed for new jit warmup changes. · cf1c7848
  Lawrence McAfee authored May 24, 2022
  
  cf1c7848
- preallocating global buffer to avoid memory fragmentation · 9dc3c42a
  Vijay Korthikanti authored May 24, 2022
  
  9dc3c42a
- added timer for layernorm all-reduce. · aff99e32
  Lawrence McAfee authored May 24, 2022
  
  aff99e32
- moved SP's layernorm all-reduce into optimizer.py. · ee57e086
  Lawrence McAfee authored May 24, 2022
  
  ee57e086
- Merge branch 'main' into lmcafee/distrib-opt-nodupe · c9a59554
  Lawrence McAfee authored May 24, 2022
```
merge sequence parallelism's layernorm all-reduce into distributed optimizer.
```
  c9a59554
- Merge branch 'sequence_parallel' into 'main' · 877f6a1b
  Jared Casper authored May 24, 2022
```
fix for sequence parallelism in bert pooling

See merge request ADLR/megatron-lm!418
```
  877f6a1b
- fix for sequence parallelism in bert pooling · 8474e6e5
  Vijay Korthikanti authored May 24, 2022
  
  8474e6e5
20 May, 2022 4 commits
- Merge branch 'readme-update' into 'main' · 32132c6e
  Jared Casper authored May 20, 2022
```
Update readme with sequence parallelism and selective activation recompute.

See merge request ADLR/megatron-lm!417
```
  32132c6e
- Merge branch 'sequence_parallel' into 'main' · 9aad9203
  Jared Casper authored May 20, 2022
```
Sequence parallelism + attention checkpoint

See merge request ADLR/megatron-lm!413
```
  9aad9203
- avoiding sequence parallelism on the pooler · 3f91f09b
  Vijay Korthikanti authored May 20, 2022
  
  3f91f09b
- bert regression fixes · 6aaafee6
  Vijay Korthikanti authored May 20, 2022
  
  6aaafee6
19 May, 2022 4 commits
- More discussion of utilization numbers and data parallelism. · 841f789f
  Jared Casper authored May 19, 2022
  
  841f789f
- Adding papers. · cc61aa9b
  Jared Casper authored May 19, 2022
  
  cc61aa9b
- Update numbers in README · 2e90a342
  Jared Casper authored May 19, 2022
  
  2e90a342
- Update readme with sequence parallelism and selective activation recompute. · b83477a3
  Jared Casper authored May 19, 2022
  
  b83477a3
18 May, 2022 1 commit
- t5 regression fixes · 5d647381
  Vijay Korthikanti authored May 18, 2022
  
  5d647381
17 May, 2022 3 commits
- Merge branch 'testing-preprocessing' into 'main' · 667c2bcb
  Jared Casper authored May 17, 2022
```
Data preprocessing testing changes + fixes

See merge request ADLR/megatron-lm!416
```
  667c2bcb
- address review comment · a7a12f82
  Vijay Korthikanti authored May 17, 2022
  
  a7a12f82
- address review comments · d19e8ad7
  Vijay Korthikanti authored May 16, 2022
  
  d19e8ad7