Commits · 4635bdf761b75d97c24ac37113da5e3082cf4659 · OpenDAS / Megatron-LM

12 Jan, 2021 1 commit
- Readme update + change gpt2 to gpt · 152aab30
  Mohammad Shoeybi authored Jan 11, 2021
  
  152aab30
30 Dec, 2020 1 commit
- moved compile helper to initialize · 242770dd
  mshoeybi authored Dec 29, 2020
  
  242770dd
19 Dec, 2020 4 commits
- Added a comment to justify 80 percent · 25c07e14
  mohammad authored Dec 14, 2020
  
  25c07e14
- Some bugfixes · 29a69547
  mshoeybi authored Dec 11, 2020
  
  29a69547
- Last epoch should not be globally shuffled · 39181113
  mshoeybi authored Dec 11, 2020
  
  39181113
- Fix some bugs, add exit-duration capability · a31833ce
  mshoeybi authored Dec 11, 2020
  
  a31833ce
29 Nov, 2020 1 commit
- implemented blending datasets · 65290033
  mohammad authored Nov 28, 2020
  
  65290033
12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

14 Jul, 2020 1 commit
- Fix gpt2_dataset import error · 5247f24c
  Neel Kant authored Jul 14, 2020
  
  5247f24c
24 Jun, 2020 1 commit
- More ict_merge changes and interactive testing · 3354081f
  Neel Kant authored Jun 23, 2020
  
  3354081f
10 Jun, 2020 1 commit
- gpt2 mmap sample map · dd76c914
  Raul Puri authored Jun 10, 2020
  
  dd76c914
16 Apr, 2020 2 commits
- changed licence 2019 to 2020 · 463d1257
  Mohammad authored Apr 16, 2020
  
  463d1257
- added runtime compilation for helpers · c6882114
  Mohammad authored Apr 15, 2020
  
  c6882114
14 Apr, 2020 2 commits
- Lint whole repo · b7f1b050
  Neel Kant authored Apr 14, 2020
  
  b7f1b050
- added comment to build_index_mappings · d8bb57ad
  Mohammad authored Apr 13, 2020
  
  d8bb57ad
12 Apr, 2020 1 commit
- fixed a bug related to last index in the shuffled_idx · eb74fa34
  Mohammad authored Apr 12, 2020
  
  eb74fa34
10 Apr, 2020 1 commit
- addressed Jareds comments · 1c1a55da
  Mohammad authored Apr 09, 2020
  
  1c1a55da
09 Apr, 2020 1 commit
- moved current gpt2 dataloader to old · 898fcb94
  Mohammad authored Apr 08, 2020
  
  898fcb94
30 Mar, 2020 1 commit
- both bert and gpt are working · 5f8623db
  Mohammad authored Mar 29, 2020
  
  5f8623db
27 Dec, 2019 1 commit
- checked and bert, gpt, and albert albert run · f86bb671
  Mohammad Shoeybi authored Dec 26, 2019
  
  f86bb671
19 Nov, 2019 1 commit
- Plumbing for new albert dataset, including mods to arguments for data loaders. · 1a1cfcff
  Jared Casper authored Nov 18, 2019
  
  1a1cfcff
08 Oct, 2019 1 commit
- created megatron package · b886b7bb
  Mohammad Shoeybi authored Oct 08, 2019
  
  b886b7bb
13 Aug, 2019 1 commit
- eval+numeric update · a0368ddf
  Raul Puri authored Aug 12, 2019
  
  a0368ddf
30 Jul, 2019 1 commit

large update including model parallelism and gpt2 · abe36e2e

Raul Puri authored Jul 29, 2019


Co-authored-by: shoeybi <shoeybim@gmail.com>
Co-authored-by: raulpuric <raulpuric@berkeley.edu>
Co-authored-by: jaredcasper <jaredcasper@gmail.com>
Co-authored-by: mpatwary <mostofa.patwary@gmail.com>
Co-authored-by: plegresl <plegresl@gmail.com>

abe36e2e