Commits · eec218d8dea655e92cd02f8cb09e6874e4b1f8ad · OpenDAS / Megatron-LM

30 Mar, 2022 1 commit
- sequence parallelism for embedding dropout and last linear layer + memory optimizations · eec218d8
  Vijay Korthikanti authored Mar 30, 2022
  
  eec218d8
11 Mar, 2022 1 commit
- checkpoint support for core attention · 53718d4c
  Vijay Korthikanti authored Mar 11, 2022
  
  53718d4c
09 Mar, 2022 1 commit
- bug fixes · dd96d402
  Vijay Korthikanti authored Mar 08, 2022
  
  dd96d402
07 Mar, 2022 1 commit
- fixes to main merge · 269f28f7
  Vijay Korthikanti authored Mar 07, 2022
  
  269f28f7
03 Mar, 2022 1 commit
- get sequence parallelism to work with pipeline parallelism · 6658158b
  Vijay Korthikanti authored Mar 02, 2022
  
  6658158b
02 Mar, 2022 1 commit
- layernorm grad sync + name chnages · c0f10643
  Vijay Korthikanti authored Mar 02, 2022
  
  c0f10643
19 Feb, 2022 1 commit
- tensor model parallelism memory optmization · 5d4689c4
  Vijay Korthikanti authored Feb 18, 2022
  
  5d4689c4
04 Feb, 2022 1 commit
- renamed argument; 'embed' -> 'embedding' · c04c4977
  Lawrence McAfee authored Feb 04, 2022
  
  c04c4977
03 Feb, 2022 1 commit
- Removing unnecessary permutes and scatter/gather · a0bea425
  rprenger authored Feb 03, 2022
  
  a0bea425
01 Feb, 2022 2 commits
- comments, cleanup. · b93bef00
  Lawrence McAfee authored Feb 01, 2022
  
  b93bef00
- found root source of t5 issue (fast layer norm) · bea16fa3
  Lawrence McAfee authored Feb 01, 2022
  
  bea16fa3
31 Jan, 2022 1 commit
- added 'no-op' layer, to replace transformer layer when num_layers == 0. · 1fa6990c
  Lawrence McAfee authored Jan 31, 2022
  
  1fa6990c
29 Jan, 2022 1 commit
- narrowed issue to pipeline rank 0, virtual pipeline rank >= 1 · 5bc9f889
  Lawrence McAfee authored Jan 28, 2022
  
  5bc9f889
27 Jan, 2022 1 commit
- address review comments · 53931b8b
  Vijay Korthikanti authored Jan 27, 2022
  
  53931b8b
26 Jan, 2022 1 commit
- further clarified viewless tensor comment in transformer.py · d16e2a24
  Lawrence McAfee authored Jan 26, 2022
  
  d16e2a24
25 Jan, 2022 1 commit
- limit 'make_viewless_tensor()' to case of micro_batch_size == 1; added comment · 24369dd6
  Lawrence McAfee authored Jan 25, 2022
  
  24369dd6
24 Jan, 2022 2 commits
- fixed args.virtual_pipeline_model_parallel_size · c2b7d0b3
  Lawrence McAfee authored Jan 24, 2022
  
  c2b7d0b3
- minor fixes · 7e810e41
  Vijay Korthikanti authored Jan 24, 2022
  
  7e810e41
22 Jan, 2022 1 commit
- second phase of vision code merge · d8c85650
  Vijay Korthikanti authored Jan 21, 2022
  
  d8c85650
11 Jan, 2022 2 commits
- added comments · a1fe4805
  Lawrence McAfee authored Jan 11, 2022
  
  a1fe4805
- partially cleaned · 806422e5
  Lawrence McAfee authored Jan 11, 2022
  
  806422e5
10 Jan, 2022 1 commit
- loss matches; memory savings for multi-node (tested n3, n16) · 270d6412
  Lawrence McAfee authored Jan 10, 2022
  
  270d6412
08 Jan, 2022 1 commit
- more iterating on 'viewless tensor' methods · ed0c8714
  Lawrence McAfee authored Jan 07, 2022
  
  ed0c8714
07 Jan, 2022 1 commit
- debugging make_standalone_tensor(), safely_set_tensor_data_attr() · 5422d23a
  Lawrence McAfee authored Jan 07, 2022
  
  5422d23a
17 Dec, 2021 3 commits
- minor fixes · f2bf5a56
  Vijay Korthikanti authored Dec 17, 2021
  
  f2bf5a56
- Cleaning up Switch Transformer code · be136206
  rprenger authored Dec 17, 2021
  
  be136206
- pipeline_fixes · 17843605
  Vijay Korthikanti authored Dec 17, 2021
  
  17843605
10 Dec, 2021 1 commit
- Faster Switch code · 4018d92c
  rprenger authored Dec 02, 2021
  
  4018d92c
22 Nov, 2021 1 commit
- removed distribute_checkpointed_activations_helper() · 5993f04b
  Lawrence McAfee authored Nov 22, 2021
  
  5993f04b
11 Nov, 2021 1 commit

Sangkug Lym authored Oct 08, 2021

fix the guard to fall back to the baseline fused layer norm kernel

Persisten ln: move the guard for supported hidden sizes to layer norm module

a2fdcdf0

29 Oct, 2021 1 commit
- made model stateless with respect to inference · efc750b6
  mshoeybi authored Oct 29, 2021
  
  efc750b6
12 Oct, 2021 1 commit
- fixed a bug · b10ca211
  mshoeybi authored Oct 12, 2021
  
  b10ca211
08 Oct, 2021 1 commit
- pipelining works · a7539b0f
  mshoeybi authored Oct 07, 2021
  
  a7539b0f
06 Oct, 2021 1 commit
- jit function argument type specification · 838af7d9
  Sangkug Lym authored Oct 05, 2021
  
  838af7d9
01 Oct, 2021 1 commit
- simple pipelining works · 8f160844
  mshoeybi authored Oct 01, 2021
  
  8f160844
29 Sep, 2021 1 commit
- added multi-batch inference · 390ddef8
  mshoeybi authored Sep 29, 2021
  
  390ddef8
27 Sep, 2021 1 commit
- refactoring, tested and working · 25f9c3f0
  mshoeybi authored Sep 26, 2021
  
  25f9c3f0
20 Sep, 2021 1 commit
- Inference context optimization · 8b9fe87b
  Mohammad Shoeybi authored Sep 20, 2021
  
  8b9fe87b
27 Aug, 2021 1 commit
- Use nvfuser at pytorch >= 1.10 · dba2506d
  Sangkug Lym authored Aug 27, 2021
  
  dba2506d
21 Aug, 2021 1 commit
- some cleanup · c61dc22f
  mshoeybi authored Aug 21, 2021
  
  c61dc22f