Commits · 05b808ef7559575e8b463511abdd8aa400381a26 · OpenDAS / Megatron-LM

06 Apr, 2023 2 commits
- Expand on apply-layernorm-1p description a bit. · 05b808ef
  Jared Casper authored Apr 05, 2023
  
  05b808ef
- A bit of cleanup. · 93bed794
  Jared Casper authored Apr 05, 2023
  
  93bed794
05 Apr, 2023 2 commits
- addressing the comments · 5ed304e4
  Mostofa Patwary authored Apr 05, 2023
  
  5ed304e4
- addressing the comments · 0760822b
  Mostofa Patwary authored Apr 05, 2023
  
  0760822b
04 Apr, 2023 1 commit
- addressing comments · bf1da27e
  Mostofa Patwary authored Apr 04, 2023
  
  bf1da27e
02 Apr, 2023 1 commit
- layernorm1p added · 5e079c87
  Mostofa Patwary authored Apr 02, 2023
  
  5e079c87
23 Mar, 2023 2 commits
- Move pipeline parallel functionality into core with associated changes. · 3c92fa93
  Jared Casper authored Mar 23, 2023
  
  3c92fa93
- Add contiguous to grad_output in layers.py · 09c2f6e4
  Markel Ausin authored Mar 22, 2023
  
  09c2f6e4
22 Feb, 2023 1 commit
- Retro · 17a6044d
  Lawrence McAfee authored Feb 21, 2023
  
  17a6044d
04 Jan, 2023 1 commit
- Remove FA's check for headdim <= 128 · 9200e43a
  Tri Dao authored Jan 03, 2023
  
  9200e43a
22 Dec, 2022 1 commit
- Transformer Engine Integration Rebase · 3499542e
  John Kamalu authored Dec 21, 2022
  
  3499542e
09 Dec, 2022 1 commit
- Integrate FlashAttention into Megatron-LM · d693034e
  Tri Dao authored Dec 08, 2022
  
  d693034e
29 Nov, 2022 1 commit
- bug fixes in partitioned data preprocessor · 84a43b13
  Mike Chrzanowski authored Nov 29, 2022
  
  84a43b13
24 Nov, 2022 1 commit
- Open sourcing lm detoxification code · d48d95ab
  Boxin Wang authored Nov 23, 2022
  
  d48d95ab
22 Nov, 2022 1 commit
- Fix DropPath for hidden shape [s, b, h] · 6d45a903
  Yu Yao authored Nov 22, 2022
  
  6d45a903
17 Nov, 2022 1 commit
- Data Preprocessing Optimizations · 7fc96111
  Vijay Korthikanti authored Nov 17, 2022
  
  7fc96111
11 Nov, 2022 1 commit
- Support for variable sequence lengths across micro-batches · c3e688d3
  Vijay Korthikanti authored Nov 10, 2022
  
  c3e688d3
10 Nov, 2022 1 commit
- ViT Backbone Tensor Shape Fix · fabd3e4e
  Yu Yao authored Nov 10, 2022
  
  fabd3e4e
08 Nov, 2022 1 commit
- Fix merge error. · e0a12fe1
  Jared Casper authored Nov 08, 2022
  
  e0a12fe1
27 Oct, 2022 1 commit
- Sending in prompts with the wrong type hangs the server. This is a check to make sure it's a list · 2fdd54ec
  rprenger authored Oct 27, 2022
  
  2fdd54ec
20 Oct, 2022 1 commit
- Disable newline after colon · 544e2502
  Peng Xu authored Oct 19, 2022
  
  544e2502
14 Oct, 2022 2 commits
- Remove noop used to try to force scheduling and check for environment variable instead. · bdd97312
  Jared Casper authored Oct 14, 2022
  
  bdd97312
- inverse_square_root learning param schedule · dbed5e07
  Vijay Korthikanti authored Oct 14, 2022
  
  dbed5e07
13 Oct, 2022 1 commit
- Adding proper test cases · 8b94a160
  Shanmugam Ramasamy authored Oct 13, 2022
  
  8b94a160
11 Oct, 2022 1 commit
- address review comments · 69f3249f
  Vijay Korthikanti authored Oct 11, 2022
  
  69f3249f
10 Oct, 2022 1 commit
- removed assert for the case of evaluation only without training · 45cd4e0b
  Anmol Gupta authored Oct 10, 2022
  
  45cd4e0b
06 Oct, 2022 2 commits
- Adding some basic unit tests · b69e2195
  shanmugamr authored Oct 06, 2022
  
  b69e2195
- Setter for pipeline parallel split rank, remove print · 6defe188
  Eric Harper authored Oct 05, 2022
  
  6defe188
05 Oct, 2022 2 commits
- fixed the timer issue for the case with no pipelining · 2514892c
  mshoeybi authored Oct 05, 2022
  
  2514892c
- support for separate dataset files for train, valid and test · 63e59949
  Anmol Gupta authored Oct 05, 2022
  
  63e59949
04 Oct, 2022 1 commit
- different encoder/decoder num-layers support · cbf8250b
  Vijay Korthikanti authored Oct 03, 2022
  
  cbf8250b
30 Sep, 2022 1 commit
- Perform distributed optimizer's all-gather in param dtype (instead of grad dtype) · 57bfa7ca
  Lawrence McAfee authored Sep 30, 2022
  
  57bfa7ca
27 Sep, 2022 4 commits
- Support gradient accumulation fusion in fp16. · 07916bf2
  Jared Casper authored Sep 27, 2022
  
  07916bf2
- Error, not warn, if gradient_accumulation_fusion is requested but not available. · 2366716f
  Jared Casper authored Sep 27, 2022
  
  2366716f
- Correct some merge errors. · 55817ec9
  Jared Casper authored Sep 27, 2022
  
  55817ec9
- Small fixes. · 77753d0a
  Jared Casper authored Sep 27, 2022
  
  77753d0a
26 Sep, 2022 2 commits
- Check for pipeline_parallel > 2 when using interleaving. · e7e99721
  Jared Casper authored Sep 26, 2022
  
  e7e99721
- Update exports. · cbf780d0
  Jared Casper authored Sep 26, 2022
  
  cbf780d0
24 Sep, 2022 2 commits
- Improve docstrings, destory global memory buffer. · e00a1ca1
  Jared Casper authored Sep 23, 2022
  
  e00a1ca1
- Move get_num_layers into transformer.py. · a94d0a6e
  Jared Casper authored Sep 23, 2022
  
  a94d0a6e