Commits · 52a5f2f272e5ef242eb271227da712f7dfc55da3 · OpenDAS / Megatron-LM

12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

30 Jul, 2020 1 commit
- Remove spurious imports · 33a666d4
  Neel Kant authored Jul 30, 2020
  
  33a666d4
23 Jul, 2020 1 commit
- Address comments from last week · 11f76cd3
  Neel Kant authored Jul 22, 2020
  
  11f76cd3
14 Jul, 2020 1 commit
- Use the new allgather implementation · 628bf0dd
  Neel Kant authored Jul 13, 2020
  
  628bf0dd
10 Jul, 2020 1 commit
- added allgather and allreduce · 98feae4e
  mohammad authored Jul 09, 2020
  
  98feae4e
07 Jul, 2020 2 commits
- Made topk accuracy reporting optional · de6640be
  Neel Kant authored Jul 07, 2020
  
  de6640be
- Addressed Jared's comments · 8d7f508a
  Neel Kant authored Jul 06, 2020
  
  8d7f508a
29 Jun, 2020 1 commit
- Create tensors on cuda rather than copying · 76928caa
  Neel Kant authored Jun 29, 2020
  
  76928caa
26 Jun, 2020 1 commit
- Cosmetic changes · 2a3b445d
  Neel Kant authored Jun 26, 2020
  
  2a3b445d
24 Jun, 2020 1 commit
- More ict_merge changes and interactive testing · 3354081f
  Neel Kant authored Jun 23, 2020
  
  3354081f
22 Jun, 2020 1 commit
- Address most of comments from code review w/ Mohammad · 6495927e
  Neel Kant authored Jun 22, 2020
  
  6495927e
05 Jun, 2020 1 commit
- Prune changes to only be related to ICT · 32bb4edc
  Neel Kant authored Jun 05, 2020
  
  32bb4edc
27 May, 2020 1 commit
- Correct CrossEntropyLoss · 0e8f4331
  Neel Kant authored May 27, 2020
  
  0e8f4331
07 May, 2020 1 commit
- Add REALMAnswerSpanModel and MLM features · f2094783
  Neel Kant authored May 06, 2020
  
  f2094783
03 May, 2020 1 commit
- Move get_train_val... to dataset_utils · 16a64c41
  Neel Kant authored May 03, 2020
  
  16a64c41
30 Apr, 2020 1 commit
- Whitening code · 9d225b44
  Neel Kant authored Apr 29, 2020
  
  9d225b44
29 Apr, 2020 1 commit
- Update and test misc functionality · 6c0a5bd8
  Neel Kant authored Apr 28, 2020
  
  6c0a5bd8
24 Apr, 2020 1 commit
- Write pretrain_realm.py and misc dataset_type left from earlier · f7f730e1
  Neel Kant authored Apr 23, 2020
  
  f7f730e1
21 Apr, 2020 1 commit
- Simplify batch and forward for ICT dataset and model · 5235ed87
  Neel Kant authored Apr 21, 2020
  
  5235ed87
17 Apr, 2020 1 commit
- Refactor before merge request · 4abd7ce2
  Neel Kant authored Apr 17, 2020
  
  4abd7ce2
15 Apr, 2020 2 commits
- Debug hash dump · 662dc982
  Neel Kant authored Apr 15, 2020
  
  662dc982
- Implement reformer hashing scheme · 81c71789
  Neel Kant authored Apr 15, 2020
  
  81c71789
14 Apr, 2020 1 commit
- Re-adjust dataloader properties · 064a6881
  Neel Kant authored Apr 13, 2020
  
  064a6881
07 Apr, 2020 1 commit
- Fix issue with validation dataloader · 064bdc46
  Neel Kant authored Apr 07, 2020
  
  064bdc46
03 Apr, 2020 2 commits
- Implement InverseClozeDataset with IndexedDataset · 4558e42f
  Neel Kant authored Apr 03, 2020
  
  4558e42f
- Fix InverseClozeDataset behavior (with commented out test code) · ca6b6687
  Neel Kant authored Apr 03, 2020
  
  ca6b6687
31 Mar, 2020 1 commit
- Remove debug statements and correct dataloader · 6f56b909
  Neel Kant authored Mar 31, 2020
  
  6f56b909
30 Mar, 2020 1 commit
- Add debug statements · f1b2524b
  Neel Kant authored Mar 30, 2020
  
  f1b2524b
27 Mar, 2020 1 commit
- pretrain_bert_icy.py compiles and runs · 1dd51c0e
  Neel Kant authored Mar 27, 2020
  
  1dd51c0e
26 Mar, 2020 1 commit
- Modify pretrain_bert_ict.py to work with ICTBertModel · b1efc33d
  Neel Kant authored Mar 26, 2020
  
  b1efc33d
24 Mar, 2020 1 commit
- Correct some args and create pretrain_bert_ict.py · 21a916b1
  Neel Kant authored Mar 24, 2020
  
  21a916b1
17 Mar, 2020 2 commits
- moved albert to bert · 09e05c6f
  Mohammad Shoeybi authored Mar 16, 2020
  
  09e05c6f
- moved pretrain albert to pretrain bert · 3e4e1ab2
  Mohammad Shoeybi authored Mar 16, 2020
  
  3e4e1ab2
14 Jan, 2020 1 commit
- added query-key layer scaling and softmax fp32 option · 691747b1
  Mohammad Shoeybi authored Jan 13, 2020
  
  691747b1
21 Nov, 2019 1 commit
- fixed padding issue · f6a6811f
  Mohammad Shoeybi authored Nov 20, 2019
  
  f6a6811f
19 Nov, 2019 1 commit
- Plumbing for new albert dataset, including mods to arguments for data loaders. · 1a1cfcff
  Jared Casper authored Nov 18, 2019
  
  1a1cfcff
31 Oct, 2019 1 commit
- refacotred for code reuse · 34be7dd3
  Mohammad Shoeybi authored Oct 30, 2019
  
  34be7dd3
29 Oct, 2019 1 commit
- Major refactoring, combining gpt2 and bert · 73af1290
  Mohammad Shoeybi authored Oct 29, 2019
  
  73af1290
09 Oct, 2019 1 commit
- moved padding to utils · 2e6d5ed9
  Mohammad Shoeybi authored Oct 09, 2019
  
  2e6d5ed9