"git@developer.sourcefind.cn:OpenDAS/dynamo.git" did not exist on "eb022ec9091c11fa9bd098e64f7c43d06b06b8d9"
Commit 146f2eef authored by jon-chuang's avatar jon-chuang Committed by GitHub
Browse files

feat: improve serve commands and expose `DYNAMO_HOME` env var (#436)


Co-authored-by: default avatarRyan Olson <ryanolson@users.noreply.github.com>
parent c4106e6a
...@@ -37,6 +37,7 @@ RUN wget --tries=3 --waitretry=5 "https://static.rust-lang.org/rustup/archive/1. ...@@ -37,6 +37,7 @@ RUN wget --tries=3 --waitretry=5 "https://static.rust-lang.org/rustup/archive/1.
WORKDIR /workspace WORKDIR /workspace
ENV DYNAMO_HOME=/workspace
COPY . /workspace/ COPY . /workspace/
......
...@@ -140,6 +140,7 @@ RUN pip install dist/ai_dynamo_runtime*cp312*.whl && \ ...@@ -140,6 +140,7 @@ RUN pip install dist/ai_dynamo_runtime*cp312*.whl && \
# Tell TRTLLM worker to use the Dynamo LLM C API for KV Cache Routing # Tell TRTLLM worker to use the Dynamo LLM C API for KV Cache Routing
ENV DYNAMO_KV_CAPI_PATH="/opt/dynamo/bindings/lib/libdynamo_llm_capi.so" ENV DYNAMO_KV_CAPI_PATH="/opt/dynamo/bindings/lib/libdynamo_llm_capi.so"
ENV DYNAMO_HOME=/workspace
# FIXME: Copy more specific folders in for dev/debug after directory restructure # FIXME: Copy more specific folders in for dev/debug after directory restructure
COPY . /workspace COPY . /workspace
......
...@@ -362,6 +362,7 @@ CMD [] ...@@ -362,6 +362,7 @@ CMD []
FROM ${RUNTIME_IMAGE}:${RUNTIME_IMAGE_TAG} AS runtime FROM ${RUNTIME_IMAGE}:${RUNTIME_IMAGE_TAG} AS runtime
WORKDIR /workspace WORKDIR /workspace
ENV DYNAMO_HOME=/workspace
ENV VIRTUAL_ENV=/opt/dynamo/venv ENV VIRTUAL_ENV=/opt/dynamo/venv
# Copy NIXL # Copy NIXL
......
...@@ -105,28 +105,29 @@ This figure shows an overview of the major components to deploy: ...@@ -105,28 +105,29 @@ This figure shows an overview of the major components to deploy:
``` ```
### Example architectures ### Example architectures
_Note_: For a non-dockerized deployment, first export `DYNAMO_HOME` to point to the dynamo repository root, e.g. `export DYNAMO_HOME=$(pwd)`
#### Aggregated serving #### Aggregated serving
```bash ```bash
cd /workspace/examples/llm cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml
``` ```
#### Aggregated serving with KV Routing #### Aggregated serving with KV Routing
```bash ```bash
cd /workspace/examples/llm cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.agg_router:Frontend -f ./configs/agg_router.yaml dynamo serve graphs.agg_router:Frontend -f ./configs/agg_router.yaml
``` ```
#### Disaggregated serving #### Disaggregated serving
```bash ```bash
cd /workspace/examples/llm cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.disagg:Frontend -f ./configs/disagg.yaml dynamo serve graphs.disagg:Frontend -f ./configs/disagg.yaml
``` ```
#### Disaggregated serving with KV Routing #### Disaggregated serving with KV Routing
```bash ```bash
cd /workspace/examples/llm cd $DYNAMO_HOME/examples/llm
dynamo serve graphs.disagg_router:Frontend -f ./configs/disagg_router.yaml dynamo serve graphs.disagg_router:Frontend -f ./configs/disagg_router.yaml
``` ```
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment