Commit 146f2eef authored by jon-chuang's avatar jon-chuang Committed by GitHub
Browse files

feat: improve serve commands and expose `DYNAMO_HOME` env var (#436)


Co-authored-by: default avatarRyan Olson <ryanolson@users.noreply.github.com>
parent c4106e6a
......@@ -37,6 +37,7 @@ RUN wget --tries=3 --waitretry=5 "https://static.rust-lang.org/rustup/archive/1.
WORKDIR /workspace
ENV DYNAMO_HOME=/workspace
COPY . /workspace/
......
......@@ -140,6 +140,7 @@ RUN pip install dist/ai_dynamo_runtime*cp312*.whl && \
# Tell TRTLLM worker to use the Dynamo LLM C API for KV Cache Routing
ENV DYNAMO_KV_CAPI_PATH="/opt/dynamo/bindings/lib/libdynamo_llm_capi.so"
ENV DYNAMO_HOME=/workspace
# FIXME: Copy more specific folders in for dev/debug after directory restructure
COPY . /workspace
......
......@@ -362,6 +362,7 @@ CMD []
FROM ${RUNTIME_IMAGE}:${RUNTIME_IMAGE_TAG} AS runtime
WORKDIR /workspace
ENV DYNAMO_HOME=/workspace
ENV VIRTUAL_ENV=/opt/dynamo/venv
# Copy NIXL
......
......@@ -105,28 +105,29 @@ This figure shows an overview of the major components to deploy:
```
### Example architectures
_Note_: For a non-dockerized deployment, first export `DYNAMO_HOME` to point to the dynamo repository root, e.g. `export DYNAMO_HOME=$(pwd)`
#### Aggregated serving
```bash
cd /workspace/examples/llm
cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml
```
#### Aggregated serving with KV Routing
```bash
cd /workspace/examples/llm
cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.agg_router:Frontend -f ./configs/agg_router.yaml
```
#### Disaggregated serving
```bash
cd /workspace/examples/llm
cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.disagg:Frontend -f ./configs/disagg.yaml
```
#### Disaggregated serving with KV Routing
```bash
cd /workspace/examples/llm
cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.disagg_router:Frontend -f ./configs/disagg_router.yaml
```
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment