Unverified Commit d51580a4 authored by ishandhanani's avatar ishandhanani Committed by GitHub
Browse files

docs: add sglang installation variants (#2400)

parent 4d24d03b
...@@ -62,21 +62,68 @@ Start using [Docker Compose](../../../deploy/docker-compose.yml) ...@@ -62,21 +62,68 @@ Start using [Docker Compose](../../../deploy/docker-compose.yml)
docker compose -f deploy/docker-compose.yml up -d docker compose -f deploy/docker-compose.yml up -d
``` ```
### Build container ### Install `ai-dynamo[sglang]`
#### Install latest release
We suggest using uv to install the latest release of ai-dynamo[sglang]. You can install it with `curl -LsSf https://astral.sh/uv/install.sh | sh`
```bash
# create a virtual env
uv venv --python 3.12 --seed
# install the latest release
uv pip install "ai-dynamo[sglang]"
```
#### Installing editable version for development
<details>
<summary>Instructions</summary>
This requires having rust installed. We also recommend having a proper installation of the cuda toolkit as sglang requires `nvcc` to be available.
```bash
# create a virtual env
uv venv --python 3.12 --seed
# build dynamo runtime bindings
uv pip install maturin
cd $DYNAMO_HOME/lib/bindings/python
maturin develop --uv
cd $DYNAMO_HOME
uv pip install .
export PYTHONPATH="${PYTHONPATH}:$(pwd)/components/backends/sglang/src"
# install target sglang version (you can choose any version)
# we include the prerelease flag in order to install flashinfer rc versions
uv pip install --prerelease=allow sglang[all]==0.4.9.post6
```
</details>
#### Using prebuilt docker containers
<details>
<summary>Instructions</summary>
```bash ```bash
# pull our pre-build sglang runtime container
docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2 docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2
# or build from source
./container/build.sh --framework sglang
``` ```
### Run container </details>
#### Building docker container from source
<details>
<summary>Instructions</summary>
```bash ```bash
./container/run.sh -it --framework sglang ./container/build.sh --framework sglang
# run container using prebuild wheel
./container/run.sh --framework sglang -it
# mount workspace for development
./container/run.sh --framework sglang --mount-workspace
``` ```
</details>
## Run Single Node Examples ## Run Single Node Examples
> [!IMPORTANT] > [!IMPORTANT]
......
...@@ -25,4 +25,5 @@ python3 -m dynamo.sglang.worker \ ...@@ -25,4 +25,5 @@ python3 -m dynamo.sglang.worker \
--page-size 16 \ --page-size 16 \
--tp 1 \ --tp 1 \
--trust-remote-code \ --trust-remote-code \
--skip-tokenizer-init --skip-tokenizer-init \
--kv-events-config '{"publisher": "zmq", "topic": "kv-events"}'
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment