Commits · 4635bdf761b75d97c24ac37113da5e3082cf4659 · OpenDAS / Megatron-LM

12 Jan, 2021 1 commit
- Readme update + change gpt2 to gpt · 152aab30
  Mohammad Shoeybi authored Jan 11, 2021
  
  152aab30
12 Nov, 2020 2 commits

Intra-layer MP -> Tensor MP, Inter-layer MP -> Pipeline MP · 52a5f2f2
Deepak Narayanan authored Oct 20, 2020

52a5f2f2

Pipeline parallelism implementation with periodic full-pipeline syncs · 7abd3e90

Deepak Narayanan authored Aug 29, 2020

Also includes following changes for inter-layer model-parallel implementation:
- Refactoring of model implementations
- Training loop changes to support inter-layer communication using `ring_exchange`
- New groups for inter-layer communication
- Checkpoint changes
- Command line arguments

7abd3e90

28 Apr, 2020 1 commit
- updated the script · 25234f8a
  Mostofa Patwary authored Apr 27, 2020
  
  25234f8a
16 Apr, 2020 3 commits
- Fix embeddings and seq length · 475a6b99
  Steven Steinke authored Apr 16, 2020
  
  475a6b99
- updated scripts · ca19a8b5
  Mohammad authored Apr 16, 2020
  
  ca19a8b5
- moved steves branch · 78cb1781
  Mohammad authored Apr 16, 2020
  
  78cb1781