Commits · aed2f75e209e525c842aec7c044af7acae2a4614 · OpenDAS / Megatron-LM

03 Mar, 2021 2 commits
- Bring back call to ring_exchange() in _communicate() · cbad126a
  Deepak Narayanan authored Mar 03, 2021
  
  cbad126a
- Get PyTorch batched communication API working for interleaved schedule · 78cf869f
  Deepak Narayanan authored Feb 25, 2021
  
  78cf869f
13 Feb, 2021 1 commit
- More comments and some cleanup (e.g., better variable names) · 5489bda9
  Deepak Narayanan authored Feb 13, 2021
  
  5489bda9
09 Feb, 2021 3 commits

Compute tensor chunk size more cleanly, and add assertion for global batch size · e3e5ea89
Deepak Narayanan authored Jan 20, 2021

e3e5ea89
Break up tensors sent between pipeline stages into smaller chunks that can be all-gathered · 27fc4689
Deepak Narayanan authored Jan 20, 2021

27fc4689

Interleaved pipeline execution and code refactoring · dd889062

Deepak Narayanan authored Dec 12, 2020

- Split a model's computation into multiple virtual stages as needed,
and schedule communication correctly between these virtual stages
- Move schedule code into `schedules.py` and communication code into
`p2p_communication.py`
- Use hyphens instead of spaces in all time logging for consistency
- Factor out code in megatron/training.py into helper functions
- Refactor evaluate() function: make it use forward_backward_schedule
functions

dd889062