Commits · 5fb0ff9ae3c024d9c60906f0e1707610df491e72 · OpenDAS / Megatron-LM

10 Mar, 2022 1 commit
- small fixes. · 862d70fc
  Lawrence McAfee authored Mar 10, 2022
  
  862d70fc
09 Mar, 2022 1 commit
- padded ddp's grad_buffer to multiple of data parallel world size · 55695f81
  Lawrence McAfee authored Mar 09, 2022
  
  55695f81
18 Feb, 2022 1 commit
- support fp32 training and fix embedding update · b5726555
  Sangkug Lym authored Feb 18, 2022
  
  b5726555
16 Feb, 2022 1 commit

Sangkug Lym authored Feb 13, 2022

remove redundant linear layer class definition

add fuse_gradient_accumulation attribute to weights for simple targetting

reflect feedback and clean up the codes

arg change

83b1e42f

15 Feb, 2022 4 commits
- adress review comments · 488f8c02
  Vijay Korthikanti authored Feb 11, 2022
  
  488f8c02
- more minor fixes · 1cd3650d
  Vijay Korthikanti authored Feb 01, 2022
  
  1cd3650d
- minor fixes · 6f3bf9c0
  Vijay Korthikanti authored Feb 01, 2022
  
  6f3bf9c0
- vision third phase merge: pretraining methods + mit,swin backbones · 48c2a144
  Vijay Korthikanti authored Jan 31, 2022
  
  48c2a144
14 Feb, 2022 4 commits
- todo; align shards with model's contiguous buffer · a3f3c3ad
  Lawrence McAfee authored Feb 14, 2022
  
  a3f3c3ad
- tweaked slice index naming convention · 1215c420
  Lawrence McAfee authored Feb 14, 2022
  
  1215c420
- map param to originating virtual model; eventually move this to constructor · c5f93269
  Lawrence McAfee authored Feb 14, 2022
  
  c5f93269
- collect param offsets for contiguous grad buffer · f7232502
  Lawrence McAfee authored Feb 14, 2022
  
  f7232502
11 Feb, 2022 1 commit
- adress review comments · 4554c3fe
  Vijay Korthikanti authored Feb 11, 2022
  
  4554c3fe
04 Feb, 2022 1 commit
- renamed argument; 'embed' -> 'embedding' · c04c4977
  Lawrence McAfee authored Feb 04, 2022
  
  c04c4977
03 Feb, 2022 1 commit
- Removing unnecessary permutes and scatter/gather · a0bea425
  rprenger authored Feb 03, 2022
  
  a0bea425
01 Feb, 2022 5 commits
- comments, cleanup. · b93bef00
  Lawrence McAfee authored Feb 01, 2022
  
  b93bef00
- more minor fixes · 3f1a728a
  Vijay Korthikanti authored Feb 01, 2022
  
  3f1a728a
- found root source of t5 issue (fast layer norm) · bea16fa3
  Lawrence McAfee authored Feb 01, 2022
  
  bea16fa3
- minor fixes · e1f9c3a5
  Vijay Korthikanti authored Feb 01, 2022
  
  e1f9c3a5
- vision third phase merge: pretraining methods + mit,swin backbones · 01a82723
  Vijay Korthikanti authored Jan 31, 2022
  
  01a82723
31 Jan, 2022 1 commit
- added 'no-op' layer, to replace transformer layer when num_layers == 0. · 1fa6990c
  Lawrence McAfee authored Jan 31, 2022
  
  1fa6990c
29 Jan, 2022 1 commit
- narrowed issue to pipeline rank 0, virtual pipeline rank >= 1 · 5bc9f889
  Lawrence McAfee authored Jan 28, 2022
  
  5bc9f889
27 Jan, 2022 1 commit
- address review comments · 53931b8b
  Vijay Korthikanti authored Jan 27, 2022
  
  53931b8b
26 Jan, 2022 1 commit
- further clarified viewless tensor comment in transformer.py · d16e2a24
  Lawrence McAfee authored Jan 26, 2022
  
  d16e2a24
25 Jan, 2022 1 commit
- limit 'make_viewless_tensor()' to case of micro_batch_size == 1; added comment · 24369dd6
  Lawrence McAfee authored Jan 25, 2022
  
  24369dd6
24 Jan, 2022 2 commits
- fixed args.virtual_pipeline_model_parallel_size · c2b7d0b3
  Lawrence McAfee authored Jan 24, 2022
  
  c2b7d0b3
- minor fixes · 7e810e41
  Vijay Korthikanti authored Jan 24, 2022
  
  7e810e41
22 Jan, 2022 1 commit
- second phase of vision code merge · d8c85650
  Vijay Korthikanti authored Jan 21, 2022
  
  d8c85650
13 Jan, 2022 1 commit
- minor fixes · 3b29f0e7
  Vijay Korthikanti authored Jan 13, 2022
  
  3b29f0e7
12 Jan, 2022 1 commit
- Phase1 merge: vit optimizations + dataset enhancements + scaled_softmax kernel · 7a77abd9
  Vijay Korthikanti authored Jan 12, 2022
  
  7a77abd9
11 Jan, 2022 2 commits
- added comments · a1fe4805
  Lawrence McAfee authored Jan 11, 2022
  
  a1fe4805
- partially cleaned · 806422e5
  Lawrence McAfee authored Jan 11, 2022
  
  806422e5
10 Jan, 2022 1 commit
- loss matches; memory savings for multi-node (tested n3, n16) · 270d6412
  Lawrence McAfee authored Jan 10, 2022
  
  270d6412
08 Jan, 2022 1 commit
- more iterating on 'viewless tensor' methods · ed0c8714
  Lawrence McAfee authored Jan 07, 2022
  
  ed0c8714
07 Jan, 2022 1 commit
- debugging make_standalone_tensor(), safely_set_tensor_data_attr() · 5422d23a
  Lawrence McAfee authored Jan 07, 2022
  
  5422d23a
04 Jan, 2022 1 commit
- update comment · d439b417
  Vijay Korthikanti authored Jan 04, 2022
  
  d439b417
31 Dec, 2021 1 commit
- simplify code · 9da6e975
  Vijay Korthikanti authored Dec 30, 2021
  
  9da6e975
30 Dec, 2021 2 commits
- avoid allocation of word embedding for head in T5 pp=2 case · ade99d61
  Vijay Korthikanti authored Dec 30, 2021
  
  ade99d61
- fixing the corner case pp=2 · 26ea8314
  Vijay Korthikanti authored Dec 30, 2021
  
  26ea8314
17 Dec, 2021 1 commit
- minor fixes · f2bf5a56
  Vijay Korthikanti authored Dec 17, 2021
  
  f2bf5a56