Commits · 80389ef6e28cfe6cb85fe98e5313d6876c4391e4 · OpenDAS / Megatron-LM

26 May, 2022 1 commit
- bug fix for get_data_parallel_src_rank · 9ad1944d
  Vijay Korthikanti authored May 26, 2022
  
  9ad1944d
24 May, 2022 3 commits
- resolved review comments · 739cb43d
  Vijay Korthikanti authored May 24, 2022
  
  739cb43d
- preallocating global buffer to avoid memory fragmentation · 9dc3c42a
  Vijay Korthikanti authored May 24, 2022
  
  9dc3c42a
- fix for sequence parallelism in bert pooling · 8474e6e5
  Vijay Korthikanti authored May 24, 2022
  
  8474e6e5
18 May, 2022 1 commit
- t5 regression fixes · 5d647381
  Vijay Korthikanti authored May 18, 2022
  
  5d647381
17 May, 2022 3 commits
- address review comments · d19e8ad7
  Vijay Korthikanti authored May 16, 2022
  
  d19e8ad7
- address review comments · bbab79f8
  Vijay Korthikanti authored May 16, 2022
  
  bbab79f8
- address review comments · 356eb36a
  Vijay Korthikanti authored May 16, 2022
  
  356eb36a
09 May, 2022 1 commit
- (1)support pyt20.12 compatibility, (2) arg naming update · f602ac56
  Sangkug Lym authored Apr 11, 2022
```
change dummy_handler to nullcontext
```
  f602ac56
28 Apr, 2022 1 commit
- address review comments · cfd2e216
  Vijay Korthikanti authored Apr 28, 2022
  
  cfd2e216
30 Mar, 2022 1 commit
- sequence parallelism for embedding dropout and last linear layer + memory optimizations · eec218d8
  Vijay Korthikanti authored Mar 30, 2022
  
  eec218d8
09 Mar, 2022 1 commit
- bug fixes · dd96d402
  Vijay Korthikanti authored Mar 08, 2022
  
  dd96d402
07 Mar, 2022 2 commits
- fixes to main merge · 269f28f7
  Vijay Korthikanti authored Mar 07, 2022
  
  269f28f7
- refactor to help merge with main · 0d77c0e9
  Vijay Korthikanti authored Mar 07, 2022
  
  0d77c0e9
03 Mar, 2022 2 commits
- column parallel linear with sequence parallelism · 02bb1f5c
  Vijay Korthikanti authored Mar 02, 2022
  
  02bb1f5c
- get sequence parallelism to work with pipeline parallelism · 6658158b
  Vijay Korthikanti authored Mar 02, 2022
  
  6658158b
02 Mar, 2022 1 commit
- layernorm grad sync + name chnages · c0f10643
  Vijay Korthikanti authored Mar 02, 2022
  
  c0f10643
19 Feb, 2022 1 commit
- tensor model parallelism memory optmization · 5d4689c4
  Vijay Korthikanti authored Feb 18, 2022
  
  5d4689c4
18 Feb, 2022 1 commit
- support fp32 training and fix embedding update · b5726555
  Sangkug Lym authored Feb 18, 2022
  
  b5726555
17 Feb, 2022 2 commits
- addressed Jared and Patrick comments. · 90ce932d
  mshoeybi authored Feb 17, 2022
  
  90ce932d
- changed all_gather to _all_gather_base in distributed checkpointing · 37181ef4
  mshoeybi authored Feb 17, 2022
  
  37181ef4
16 Feb, 2022 1 commit

Sangkug Lym authored Feb 13, 2022

remove redundant linear layer class definition

add fuse_gradient_accumulation attribute to weights for simple targetting

reflect feedback and clean up the codes

arg change

83b1e42f

08 Feb, 2022 1 commit
- fixed t5 'get_num_layers()' · 2fadaa50
  Lawrence McAfee authored Feb 08, 2022
  
  2fadaa50
04 Feb, 2022 1 commit
- renamed argument; 'embed' -> 'embedding' · c04c4977
  Lawrence McAfee authored Feb 04, 2022
  
  c04c4977
01 Feb, 2022 1 commit
- comments, cleanup. · b93bef00
  Lawrence McAfee authored Feb 01, 2022
  
  b93bef00
31 Jan, 2022 1 commit
- working for t5 [ encoder embedding only ] · 3af6725d
  Lawrence McAfee authored Jan 31, 2022
  
  3af6725d
25 Jan, 2022 1 commit
- working with interleaving · 804ed2e6
  Lawrence McAfee authored Jan 24, 2022
  
  804ed2e6
24 Jan, 2022 3 commits
- added args.transformer_pipeline_model_parallel_size · a06af061
  Lawrence McAfee authored Jan 24, 2022
  
  a06af061
- fixed args.virtual_pipeline_model_parallel_size · c2b7d0b3
  Lawrence McAfee authored Jan 24, 2022
  
  c2b7d0b3
- working when no interleaving · 33dc8e9c
  Lawrence McAfee authored Jan 24, 2022
  
  33dc8e9c
12 Jan, 2022 1 commit
- Phase1 merge: vit optimizations + dataset enhancements + scaled_softmax kernel · 7a77abd9
  Vijay Korthikanti authored Jan 12, 2022
  
  7a77abd9
11 Jan, 2022 3 commits
- added comments · a1fe4805
  Lawrence McAfee authored Jan 11, 2022
  
  a1fe4805
- partially cleaned · 806422e5
  Lawrence McAfee authored Jan 11, 2022
  
  806422e5
- jan 11 alpha · 05042081
  Lawrence McAfee authored Jan 11, 2022
  
  05042081
10 Jan, 2022 1 commit
- loss matches; memory savings for multi-node (tested n3, n16) · 270d6412
  Lawrence McAfee authored Jan 10, 2022
  
  270d6412
08 Jan, 2022 1 commit
- more iterating on 'viewless tensor' methods · ed0c8714
  Lawrence McAfee authored Jan 07, 2022
  
  ed0c8714
07 Jan, 2022 1 commit
- debugging make_standalone_tensor(), safely_set_tensor_data_attr() · 5422d23a
  Lawrence McAfee authored Jan 07, 2022
  
  5422d23a
17 Dec, 2021 2 commits
- minor fixes · f2bf5a56
  Vijay Korthikanti authored Dec 17, 2021
  
  f2bf5a56
- pipeline_fixes · 17843605
  Vijay Korthikanti authored Dec 17, 2021
  
  17843605
22 Nov, 2021 1 commit
- removed unused 'get_args' import · 941a793f
  Lawrence McAfee authored Nov 22, 2021
  
  941a793f