"...ssh:/git@developer.sourcefind.cn:2222/OpenDAS/dynamo.git" did not exist on "fb4432e154b96fc93c12a71c8d14e32944db3619"
Unverified Commit 44012a24 authored by Ziqi Fan's avatar Ziqi Fan Committed by GitHub
Browse files

docs: add vLLM KVBM 2P2D example due to a special ENV VAR to note (#3526)


Signed-off-by: default avatarZiqi Fan <ziqif@nvidia.com>
parent 5f7c1f7e
#!/bin/bash
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
set -e
trap 'echo Cleaning up...; kill 0' EXIT
# run ingress with KV router
python -m dynamo.frontend --router-mode kv --http-port=8000 &
# run decode workers on GPU 0 and 1, without enabling KVBM
# NOTE: remove --enforce-eager for production use
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --connector nixl --enforce-eager &
CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --connector nixl --enforce-eager &
# run prefill workers on GPU 2 and 3 with KVBM enabled using 20GB of CPU cache
# NOTE: use different barrier id prefixes for each prefill worker to avoid conflicts
# NOTE: remove --enforce-eager for production use
DYN_KVBM_BARRIER_ID_PREFIX=kvbm_0 \
DYN_KVBM_CPU_CACHE_GB=20 \
CUDA_VISIBLE_DEVICES=2 \
python3 -m dynamo.vllm \
--model Qwen/Qwen3-0.6B \
--is-prefill-worker \
--connector kvbm nixl \
--enforce-eager &
DYN_KVBM_BARRIER_ID_PREFIX=kvbm_1 \
DYN_KVBM_CPU_CACHE_GB=20 \
CUDA_VISIBLE_DEVICES=3 \
python3 -m dynamo.vllm \
--model Qwen/Qwen3-0.6B \
--is-prefill-worker \
--connector kvbm nixl \
--enforce-eager
...@@ -43,11 +43,17 @@ cd $DYNAMO_HOME/components/backends/vllm ...@@ -43,11 +43,17 @@ cd $DYNAMO_HOME/components/backends/vllm
./launch/agg_kvbm.sh ./launch/agg_kvbm.sh
``` ```
### Disaggregated Serving with KVBM (1P1D) ### Disaggregated Serving with KVBM
```bash ```bash
# 1P1D - one prefill worker and one decode worker
# NOTE: need at least 2 GPUs # NOTE: need at least 2 GPUs
cd $DYNAMO_HOME/components/backends/vllm cd $DYNAMO_HOME/components/backends/vllm
./launch/disagg_kvbm.sh ./launch/disagg_kvbm.sh
# 2P2D - two prefill workers and two decode workers
# NOTE: need at least 4 GPUs
cd $DYNAMO_HOME/components/backends/vllm
./launch/disagg_kvbm_2p2d.sh
``` ```
> [!NOTE] > [!NOTE]
> To tune the size of CPU or disk cache, set `DYN_KVBM_CPU_CACHE_GB` and `DYN_KVBM_DISK_CACHE_GB` accordingly. We only set `DYN_KVBM_CPU_CACHE_GB=20` in both scripts above. > To tune the size of CPU or disk cache, set `DYN_KVBM_CPU_CACHE_GB` and `DYN_KVBM_DISK_CACHE_GB` accordingly. We only set `DYN_KVBM_CPU_CACHE_GB=20` in both scripts above.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment