Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
dynamo
Commits
761f67e0
"deploy/operator/internal/controller_common/scale.go" did not exist on "a97602a2c9e18ea9979aa8cbf391b5f5154f17d0"
Unverified
Commit
761f67e0
authored
May 28, 2025
by
Hongkuan Zhou
Committed by
GitHub
May 28, 2025
Browse files
fix: update kv-router usage (#1238)
parent
cc40af70
Changes
3
Show whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
14 additions
and
5 deletions
+14
-5
docs/guides/planner_benchmark/benchmark_planner.md
docs/guides/planner_benchmark/benchmark_planner.md
+2
-2
docs/guides/planner_benchmark/disagg_1p1d.yml
docs/guides/planner_benchmark/disagg_1p1d.yml
+6
-2
docs/guides/planner_benchmark/disagg_2p2d.yaml
docs/guides/planner_benchmark/disagg_2p2d.yaml
+6
-1
No files found.
docs/guides/planner_benchmark/benchmark_planner.md
View file @
761f67e0
...
...
@@ -48,7 +48,7 @@ To measure the performance of dynamo with planner, we start from a 1p1d deployme
```
bash
cd
examples/llm
dynamo serve graphs.disagg:Frontend
-f
disagg_1p1d.yml
dynamo serve graphs.disagg
_router
:Frontend
-f
disagg_1p1d.yml
# in terminal 2
genai-perf profile
\
...
...
@@ -82,7 +82,7 @@ In this example, we use a fixed 2p2d engine as baseline. Planner provides a `--n
```
bash
# in terminal 1
dynamo serve graphs.disagg:Frontend
-f
disagg_2p2d.yml
dynamo serve graphs.disagg
_router
:Frontend
-f
disagg_2p2d.yml
# in terminal 2
genai-perf profile
--tokenizer
deepseek-ai/DeepSeek-R1-Distill-Llama-8B
-m
deepseek-ai/DeepSeek-R1-Distill-Llama-8B
--service-kind
openai
--endpoint-type
chat
--url
http://localhost:8000
--streaming
--input-file
payload:sin_b512_t600_rr5.0-20.0-150.0_io3000150-3000150-0.2-0.8-10.jsonl
...
...
docs/guides/planner_benchmark/disagg_1p1d.yml
View file @
761f67e0
...
...
@@ -17,6 +17,7 @@ Common:
block-size
:
64
max-model-len
:
16384
kv-transfer-config
:
'
{"kv_connector":"DynamoNixlConnector"}'
router
:
kv-load
Frontend
:
served_model_name
:
deepseek-ai/DeepSeek-R1-Distill-Llama-8B
...
...
@@ -24,8 +25,11 @@ Frontend:
port
:
8000
Processor
:
router
:
kv-load
common-configs
:
[
model
,
block-size
]
common-configs
:
[
model
,
block-size
,
router
]
Router
:
min-workers
:
1
common-configs
:
[
model
,
block-size
,
router
]
VllmWorker
:
remote-prefill
:
true
...
...
docs/guides/planner_benchmark/disagg_2p2d.yaml
View file @
761f67e0
...
...
@@ -17,6 +17,7 @@ Common:
block-size
:
64
max-model-len
:
16384
kv-transfer-config
:
'
{"kv_connector":"DynamoNixlConnector"}'
router
:
kv-load
Frontend
:
served_model_name
:
deepseek-ai/DeepSeek-R1-Distill-Llama-8B
...
...
@@ -25,7 +26,11 @@ Frontend:
Processor
:
router
:
kv-load
common-configs
:
[
model
,
block-size
]
common-configs
:
[
model
,
block-size
,
router
]
Router
:
min-workers
:
1
common-configs
:
[
model
,
block-size
,
router
]
VllmWorker
:
remote-prefill
:
true
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment