Unverified Commit db54ca2f authored by Yan Ru Pei's avatar Yan Ru Pei Committed by GitHub
Browse files

docs(crates): add readmes for mocker and kv-router [DYN-2456] (#7687)


Signed-off-by: default avatarPeaBrane <yanrpei@gmail.com>
parent f0d3ce63
...@@ -8,7 +8,9 @@ version.workspace = true ...@@ -8,7 +8,9 @@ version.workspace = true
edition.workspace = true edition.workspace = true
authors.workspace = true authors.workspace = true
license.workspace = true license.workspace = true
homepage.workspace = true
repository.workspace = true repository.workspace = true
readme = "README.md"
[features] [features]
default = [] default = []
......
# dynamo-kv-router
`dynamo-kv-router` provides the core KV-aware routing data structures and scheduling primitives used
by Dynamo to steer requests toward workers with the best cache overlap.
## What This Crate Provides
- `RadixTree` and `ConcurrentRadixTree` for prefix-overlap indexing
- `ThreadPoolIndexer` and `PositionalIndexer` for higher-throughput index backends
- `KvRouterConfig`, `RouterQueuePolicy`, and `LocalScheduler` for request routing
- Protocol and hashing helpers such as `RouterEvent`, `WorkerId`,
`compute_block_hash_for_seq`, and `compute_seq_hash_for_block`
## Basic Rust Usage
```rust
use dynamo_kv_router::{
KvRouterConfig, RadixTree, compute_block_hash_for_seq, compute_seq_hash_for_block,
};
use dynamo_kv_router::protocols::BlockHashOptions;
let prompt_tokens = vec![1_u32, 2, 3, 4, 5, 6, 7, 8];
let local_hashes = compute_block_hash_for_seq(&prompt_tokens, 4, BlockHashOptions::default());
let seq_hashes = compute_seq_hash_for_block(&local_hashes);
let router_config = KvRouterConfig::default();
let index = RadixTree::new();
let scores = index.find_matches(local_hashes, false);
assert!(router_config.use_kv_events);
assert_eq!(seq_hashes.len(), 2);
assert!(scores.scores.is_empty());
```
For end-to-end routing, pair the indexers with `LocalScheduler` and the worker/config protocol
types re-exported from the crate root.
## Features
- `metrics`: Prometheus metrics for router internals
- `runtime-protocols`: integration points with `dynamo-runtime`
- `standalone-indexer`: standalone indexer service support
- `bench`: internal benchmarking helpers
## Further Reading
- Router guide: <https://docs.nvidia.com/dynamo/components/router>
- Indexer internals:
<https://github.com/ai-dynamo/dynamo/blob/main/lib/kv-router/src/indexer/README.md>
- Dynamo repository: <https://github.com/ai-dynamo/dynamo>
...@@ -8,7 +8,9 @@ version.workspace = true ...@@ -8,7 +8,9 @@ version.workspace = true
edition.workspace = true edition.workspace = true
authors.workspace = true authors.workspace = true
license.workspace = true license.workspace = true
homepage.workspace = true
repository.workspace = true repository.workspace = true
readme = "README.md"
[dependencies] [dependencies]
# repo # repo
......
# dynamo-mocker
`dynamo-mocker` is a GPU-free simulation crate for Dynamo's LLM scheduling and KV-cache behavior.
It is used for testing, replay, and benchmarking workflows where you want realistic scheduler and
cache behavior without running a real inference engine.
## What This Crate Provides
- `MockEngineArgs` for configuring a simulated engine
- `engine::create_engine` for building a vLLM-style or SGLang-style mock scheduler
- `KvEventPublishers` hooks for emitting router-visible KV cache events
- `loadgen` and `replay` modules for synthetic and trace-driven experiments
## Basic Rust Usage
```rust
use dynamo_mocker::common::protocols::{
DirectRequest, KvEventPublishers, MockEngineArgs,
};
use dynamo_mocker::engine::create_engine;
let args = MockEngineArgs::builder()
.block_size(16)
.num_gpu_blocks(1024)
.max_num_seqs(Some(32))
.max_num_batched_tokens(Some(4096))
.build()
.unwrap();
let engine = create_engine(args, 0, None, KvEventPublishers::default(), None);
engine.receive(DirectRequest {
tokens: vec![1, 2, 3, 4],
max_output_tokens: 16,
uuid: None,
dp_rank: 0,
arrival_timestamp_ms: None,
});
```
This crate is also the foundation for Dynamo's higher-level mocker CLI and replay tooling. In many
deployments you will interact with it indirectly through the Python entry points rather than
embedding it directly as a standalone Rust dependency.
## Further Reading
- Mocker guide:
<https://github.com/ai-dynamo/dynamo/blob/main/docs/mocker/mocker.md>
- Trace replay guide:
<https://github.com/ai-dynamo/dynamo/blob/main/docs/benchmarks/mocker-trace-replay.md>
- Python component README:
<https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/mocker/README.md>
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment