Unverified Commit 28751958 authored by Peter Götz's avatar Peter Götz Committed by GitHub
Browse files

[`docs`] Improve visualization for vertical parallelism (#28583)

The documentation says "We refer to this Model parallelism as “Vertical” because of how models are typically visualized.", but then visualizes the model horizontally. This change visualizes the model indeed vertically.
parent 4cbd876e
...@@ -285,10 +285,19 @@ following diagram shows an 8-layer model split vertically into two slices, placi ...@@ -285,10 +285,19 @@ following diagram shows an 8-layer model split vertically into two slices, placi
GPU0 and 4-7 to GPU1: GPU0 and 4-7 to GPU1:
``` ```
=================== =================== ================
| 0 | 1 | 2 | 3 | | 4 | 5 | 6 | 7 | | Layer | |
=================== =================== | 0 | |
GPU0 GPU1 | 1 | GPU0 |
| 2 | |
| 3 | |
================
| Layer | |
| 4 | |
| 5 | GPU1 |
| 6 | |
| 7 | |
================
``` ```
In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment