Skip to content

GitLab

Commit 51315905 authored by Jared Casper's avatar Jared Casper Committed by Deepak Narayanan
Browse files

Add comment describing _PIPELINE_GLOBAL_RANKS

parent 1d4e8760
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
megatron/mpu/initialize.py
View file @ 51315905
...@@ -38,6 +38,8 @@ _MPU_PIPELINE_MODEL_PARALLEL_WORLD_SIZE = None ...@@ -38,6 +38,8 @@ _MPU_PIPELINE_MODEL_PARALLEL_WORLD_SIZE = None
_MPU_TENSOR_MODEL_PARALLEL_RANK = None _MPU_TENSOR_MODEL_PARALLEL_RANK = None
_MPU_PIPELINE_MODEL_PARALLEL_RANK = None _MPU_PIPELINE_MODEL_PARALLEL_RANK = None
# A list of global ranks for each pipeline group to ease calculation of the source
# rank when broadcasting from the first or last pipeline stage
_PIPELINE_GLOBAL_RANKS = None _PIPELINE_GLOBAL_RANKS = None
def is_unitialized(): def is_unitialized():
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment