Commits · eec218d8dea655e92cd02f8cb09e6874e4b1f8ad · OpenDAS / Megatron-LM

30 Mar, 2022 1 commit
- sequence parallelism for embedding dropout and last linear layer + memory optimizations · eec218d8
  Vijay Korthikanti authored Mar 30, 2022
  
  eec218d8
11 Mar, 2022 1 commit
- checkpoint support for core attention · 53718d4c
  Vijay Korthikanti authored Mar 11, 2022
  
  53718d4c
09 Mar, 2022 1 commit
- bug fixes · dd96d402
  Vijay Korthikanti authored Mar 08, 2022
  
  dd96d402
07 Mar, 2022 3 commits
- fixes to main merge · 269f28f7
  Vijay Korthikanti authored Mar 07, 2022
  
  269f28f7
- Merge branch 'main' into allreduce-split · 6fdbf26b
  Vijay Korthikanti authored Mar 07, 2022
  
  6fdbf26b
- refactor to help merge with main · 0d77c0e9
  Vijay Korthikanti authored Mar 07, 2022
  
  0d77c0e9
03 Mar, 2022 2 commits
- column parallel linear with sequence parallelism · 02bb1f5c
  Vijay Korthikanti authored Mar 02, 2022
  
  02bb1f5c
- get sequence parallelism to work with pipeline parallelism · 6658158b
  Vijay Korthikanti authored Mar 02, 2022
  
  6658158b
02 Mar, 2022 1 commit
- layernorm grad sync + name chnages · c0f10643
  Vijay Korthikanti authored Mar 02, 2022
  
  c0f10643
24 Feb, 2022 2 commits
- Merge branch 'switch' into 'main' · 53f3efc4
  Lawrence McAfee authored Feb 24, 2022
```
Switch Transformer

See merge request ADLR/megatron-lm!373
```
  53f3efc4
- Fixing merge conflict · 458d7785
  rprenger authored Feb 23, 2022
  
  458d7785
23 Feb, 2022 1 commit
- Merge branch 'slym/grad_accum_fusion' into 'main' · 9c5a830f
  Jared Casper authored Feb 22, 2022
```
Gradient accumulation fusion

See merge request ADLR/megatron-lm!394
```
  9c5a830f
19 Feb, 2022 1 commit
- tensor model parallelism memory optmization · 5d4689c4
  Vijay Korthikanti authored Feb 18, 2022
  
  5d4689c4
18 Feb, 2022 1 commit
- support fp32 training and fix embedding update · b5726555
  Sangkug Lym authored Feb 18, 2022
  
  b5726555
17 Feb, 2022 6 commits
- Merge branch 'all_gather_base' into 'main' · 0ed2f6ac
  Jared Casper authored Feb 17, 2022
```
changed all_gather to _all_gather_base in distributed checkpointing

See merge request ADLR/megatron-lm!395
```
  0ed2f6ac
- Don't need an example for Switch networks until it's more polished · d7bf1ab5
  rprenger authored Feb 17, 2022
  
  d7bf1ab5
- addressed Jared and Patrick comments. · 90ce932d
  mshoeybi authored Feb 17, 2022
  
  90ce932d
- changed all_gather to _all_gather_base in distributed checkpointing · 37181ef4
  mshoeybi authored Feb 17, 2022
  
  37181ef4
- removed slurm commands; added 'num-experts' · 3237cc33
  Lawrence McAfee authored Feb 17, 2022
  
  3237cc33
- moved run.sh -> examples/ · c8e4cdae
  Lawrence McAfee authored Feb 17, 2022
  
  c8e4cdae
16 Feb, 2022 1 commit

Sangkug Lym authored Feb 13, 2022

remove redundant linear layer class definition

add fuse_gradient_accumulation attribute to weights for simple targetting

reflect feedback and clean up the codes

arg change

83b1e42f

15 Feb, 2022 5 commits
- adress review comments · 488f8c02
  Vijay Korthikanti authored Feb 11, 2022
  
  488f8c02
- more minor fixes · 1cd3650d
  Vijay Korthikanti authored Feb 01, 2022
  
  1cd3650d
- minor fixes · 6f3bf9c0
  Vijay Korthikanti authored Feb 01, 2022
  
  6f3bf9c0
- vision third phase merge: pretraining methods + mit,swin backbones · 48c2a144
  Vijay Korthikanti authored Jan 31, 2022
  
  48c2a144
- Merge branch 'vision-merge' into 'main' · 8c8063eb
  Jared Casper authored Feb 14, 2022
```
vision third phase merge: pretraining methods + mit,swin backbones

See merge request ADLR/megatron-lm!384
```
  8c8063eb
11 Feb, 2022 5 commits
- adress review comments · 4554c3fe
  Vijay Korthikanti authored Feb 11, 2022
  
  4554c3fe
- Merge branch 'readme-no-versions' into 'main' · f00d0a3f
  Mohammad Shoeybi authored Feb 10, 2022
```
Remove specific versions of pytorch, etc. from README so it doesn't go out of date.

See merge request ADLR/megatron-lm!392
```
  f00d0a3f
- Remove specific versions of pytorch, etc. from README so it doesn't go out of date. · d50e89f1
  Jared Casper authored Feb 10, 2022
  
  d50e89f1
- Merge branch 'stop_tokens' into 'main' · d5fe59fe
  Mohammad Shoeybi authored Feb 10, 2022
```
Adding several things to the text_generation_server that were necessary for the demos

See merge request ADLR/megatron-lm!350
```
  d5fe59fe
- Addressing comments · 1d391bba
  rprenger authored Feb 10, 2022
  
  1d391bba
08 Feb, 2022 2 commits
- Merge branch 'lmcafee/embed-standalone' into 'main' · 10c6ad06
  Jared Casper authored Feb 08, 2022
```
Standalone embedding stage

See merge request ADLR/megatron-lm!385
```
  10c6ad06
- fixed t5 'get_num_layers()' · 2fadaa50
  Lawrence McAfee authored Feb 08, 2022
  
  2fadaa50
04 Feb, 2022 3 commits
- renamed argument; 'embed' -> 'embedding' · c04c4977
  Lawrence McAfee authored Feb 04, 2022
  
  c04c4977
- Adding message to ValueError · b0c824d9
  rprenger authored Feb 04, 2022
  
  b0c824d9
- Adding the web interface · 42982fc3
  rprenger authored Feb 03, 2022
  
  42982fc3
03 Feb, 2022 1 commit
- Removing unnecessary permutes and scatter/gather · a0bea425
  rprenger authored Feb 03, 2022
  
  a0bea425
01 Feb, 2022 3 commits
- comments, cleanup. · b93bef00
  Lawrence McAfee authored Feb 01, 2022
  
  b93bef00
- more minor fixes · 3f1a728a
  Vijay Korthikanti authored Feb 01, 2022
  
  3f1a728a
- found root source of t5 issue (fast layer norm) · bea16fa3
  Lawrence McAfee authored Feb 01, 2022
  
  bea16fa3