Commits · 91d4a60510a7037be6856c2557e581a4639ada19 · OpenDAS / Megatron-LM

"example/49_fpAintB_gemm/fp16int8_gemm_wmma.cpp" did not exist on "058300535b4ac4f7872dd4b93accd0ee2088e063"

03 Dec, 2020 1 commit
- Merge branch 'main' into pipeline_parallel_main · 91d4a605
  Jared Casper authored Dec 02, 2020
  
  91d4a605
02 Dec, 2020 8 commits
- Merge branch 'megatron_sampler' into 'main' · 75bd9b54
  Jared Casper authored Dec 02, 2020
```
Simplified sampler (will be needed later for batch size increase) and removed deprecated data stuff

See merge request ADLR/megatron-lm!177
```
  75bd9b54
- Merge branch 'blendable_dataset' into 'megatron_sampler' · fac6718a
  Jared Casper authored Dec 02, 2020
```
Blendable dataset

See merge request ADLR/megatron-lm!178
```
  fac6718a
- Merge branch 'refactor_learning_rate' into 'blendable_dataset' · 1eda0a17
  Jared Casper authored Dec 02, 2020
```
Refactor learning rate so it is easier to make learning rate based on consumed samples

See merge request ADLR/megatron-lm!179
```
  1eda0a17
- addressed Jareds comments · fa80af26
  mohammad authored Dec 02, 2020
  
  fa80af26
- Merge branch 'blendable_dataset' into refactor_learning_rate · 45504541
  mohammad authored Dec 02, 2020
  
  45504541
- addressed Jareds comments · 98989693
  mohammad authored Dec 02, 2020
  
  98989693
- Merge branch 'megatron_sampler' into blendable_dataset · bc56e4a5
  mohammad authored Dec 02, 2020
  
  bc56e4a5
- addrressed jareds comments · cebd3b8b
  mohammad authored Dec 02, 2020
  
  cebd3b8b
30 Nov, 2020 2 commits
- refactored learning rate scheduler so addition of variable batch size is easier · ff12df6b
  mohammad authored Nov 29, 2020
  
  ff12df6b
- added refactored learning rate · 16193619
  mohammad authored Nov 29, 2020
  
  16193619
29 Nov, 2020 3 commits
- implemented blending datasets · 65290033
  mohammad authored Nov 28, 2020
  
  65290033
- Merge branch 'megatron_sampler' into blendable_dataset · 9a0808c9
  mohammad authored Nov 28, 2020
  
  9a0808c9
- added blendable dataset · d3bb1a06
  mohammad authored Nov 28, 2020
  
  d3bb1a06
28 Nov, 2020 1 commit
- added consumed tokens to checkpoints and some refactoring · f0a445fa
  mohammad authored Nov 27, 2020
  
  f0a445fa
26 Nov, 2020 1 commit
- simplified sampler · 4311b695
  mohammad authored Nov 25, 2020
  
  4311b695
19 Nov, 2020 1 commit
- Merge branch 'main' into pipeline_parallel_main · 63c340ec
  Jared Casper authored Nov 19, 2020
  
  63c340ec
18 Nov, 2020 9 commits
- Merge branch 'update-norm' into 'main' · ea81d62f
  Mohammad Shoeybi authored Nov 17, 2020
```
Replace deprecated torch.norm with torch.linalg.norm.

See merge request ADLR/megatron-lm!175
```
  ea81d62f
- Merge branch 'community-fixes' into 'main' · ac837a4e
  Mohammad Shoeybi authored Nov 17, 2020
```
Community fixes

See merge request ADLR/megatron-lm!176
```
  ac837a4e
- Merge branch 'fix/help-title-dist' of https://github.com/lazykyama/Megatron-LM into community-fixes · 356f8771
  Jared Casper authored Nov 17, 2020
  
  356f8771
- Merge branch 'patch-1' of https://github.com/sublee/Megatron-LM into community-fixes · 24ccd8aa
  Jared Casper authored Nov 17, 2020
  
  24ccd8aa
- Merge branch 'patch-2' of https://github.com/nakosung/Megatron-LM into community-fixes · cec3dbe6
  Jared Casper authored Nov 17, 2020
  
  cec3dbe6
- Merge branch 'patch-1' of https://github.com/nakosung/Megatron-LM into community-fixes · 8d0da202
  Jared Casper authored Nov 17, 2020
  
  8d0da202
- Merge branch 'patch-1' of https://github.com/dweekly/Megatron-LM into community-fixes · 5e934043
  Jared Casper authored Nov 17, 2020
  
  5e934043
- Merge branch 'patch-1' of https://github.com/akhileshgotmare/Megatron-LM into community-fixes · 642c644a
  Jared Casper authored Nov 17, 2020
  
  642c644a
- Replace deprecated torch.norm with torch.linalg.norm. · 17035d6c
  Jared Casper authored Nov 17, 2020
  
  17035d6c
17 Nov, 2020 3 commits
- Merge branch 'main' into pipeline_parallel_main · 6fae152a
  Jared Casper authored Nov 16, 2020
  
  6fae152a
- Merge branch 'finetune_fix' into 'main' · e7c7a78f
  Mohammad Shoeybi authored Nov 16, 2020
```
Update code used for finetuning to latest API.

See merge request ADLR/megatron-lm!174
```
  e7c7a78f
- Update code used for finetuning to latest API. · b219ff00
  Jared Casper authored Nov 16, 2020
  
  b219ff00
13 Nov, 2020 1 commit
- New example scripts showing how to use MP, and some notes to main README · a5a41922
  Deepak Narayanan authored Nov 13, 2020
  
  a5a41922
12 Nov, 2020 10 commits
- Make sure dataloader state is the same after checkpoint is loaded · cd4822f1
  Deepak Narayanan authored Nov 12, 2020
  
  cd4822f1
- Move division of loss tensor by number of microbatches to training.py · c671de3e
  Deepak Narayanan authored Nov 12, 2020
  
  c671de3e
- Small bugfix in bert_model.py: make sure word_embeddings is initialized before... · 69a546be
  Deepak Narayanan authored Nov 12, 2020
```
Small bugfix in bert_model.py: make sure word_embeddings is initialized before instantiating lm_head
```
  69a546be
- Refactor code according to Jared's comments: move pipelining and... · 1979c242
  Deepak Narayanan authored Nov 12, 2020
```
Refactor code according to Jared's comments: move pipelining and non-pipelining training loops into separate methods

Also, use mpu.get_*_model_parallel_size() instead of args.*_model_parallel_size
```
  1979c242
- Allocate tensor in `communicate()` method directly on GPU (instead of... · 9ff6f473
  mshoeybi authored Nov 11, 2020
```
Allocate tensor in `communicate()` method directly on GPU (instead of allocating on CPU and then moving to GPU)
```
  9ff6f473
- Remove timing labels that don't make sense · fdcaeba0
  Deepak Narayanan authored Nov 10, 2020
  
  fdcaeba0
- Better communication timing · 7d367b1d
  Deepak Narayanan authored Nov 10, 2020
  
  7d367b1d
- Divide gradient by number of microbatches in minibatch · 3d7194c4
  Deepak Narayanan authored Nov 03, 2020
  
  3d7194c4
- Better 'forward' and 'backward' timing in megatron/training.py · a6756bf8
  Deepak Narayanan authored Nov 03, 2020
  
  a6756bf8
- Move training schedule to 1F1B for memory efficiency · 3e6898e6
  Deepak Narayanan authored Oct 24, 2020
  
  3e6898e6