Skip to content

GitLab

Unverified Commit 2abef1ef authored by Shaden Smith's avatar Shaden Smith Committed by GitHub
Browse files

Updating MPU docs (#92)

parent bca23057
Show whitespace changes
Inline Side-by-side
Showing with 3 additions and 2 deletions
deepspeed/__init__.py
View file @ 2abef1ef
...@@ -54,7 +54,7 @@ def initialize(args, ...@@ -54,7 +54,7 @@ def initialize(args,
step(), state_dict(), and load_state_dict() methods step(), state_dict(), and load_state_dict() methods
mpu: Optional: A model parallelism unit object that implements mpu: Optional: A model parallelism unit object that implements
get_model/data_parallel_group/rank/size() get_{model,data}_parallel_{rank,group,world_size}()
dist_init_required: Optional: Initializes torch.distributed dist_init_required: Optional: Initializes torch.distributed
......
docs/features.md
View file @ 2abef1ef
...@@ -68,10 +68,11 @@ mpu.get_model_parallel_rank() ...@@ -68,10 +68,11 @@ mpu.get_model_parallel_rank()
mpu.get_model_parallel_group() mpu.get_model_parallel_group()
mpu.get_model_parallel_world_size() mpu.get_model_parallel_world_size()
mpu.get_data_parallel_rank/group/world_size() mpu.get_data_parallel_rank()
mpu.get_data_parallel_group() mpu.get_data_parallel_group()
mpu.get_data_parallel_world_size() mpu.get_data_parallel_world_size()
``` ```
### Integration with Megatron-LM ### Integration with Megatron-LM
DeepSpeed is fully compatible with [Megatron](https://github.com/NVIDIA/Megatron-LM). DeepSpeed is fully compatible with [Megatron](https://github.com/NVIDIA/Megatron-LM).
Please see the [Megatron-LM tutorial](tutorials/MegatronGPT2Tutorial.md) for details. Please see the [Megatron-LM tutorial](tutorials/MegatronGPT2Tutorial.md) for details.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment