Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
dynamo
Commits
d51580a4
Unverified
Commit
d51580a4
authored
Aug 11, 2025
by
ishandhanani
Committed by
GitHub
Aug 11, 2025
Browse files
docs: add sglang installation variants (#2400)
parent
4d24d03b
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
55 additions
and
7 deletions
+55
-7
components/backends/sglang/README.md
components/backends/sglang/README.md
+53
-6
components/backends/sglang/launch/agg_router.sh
components/backends/sglang/launch/agg_router.sh
+2
-1
No files found.
components/backends/sglang/README.md
View file @
d51580a4
...
@@ -62,21 +62,68 @@ Start using [Docker Compose](../../../deploy/docker-compose.yml)
...
@@ -62,21 +62,68 @@ Start using [Docker Compose](../../../deploy/docker-compose.yml)
docker compose
-f
deploy/docker-compose.yml up
-d
docker compose
-f
deploy/docker-compose.yml up
-d
```
```
### Build container
### Install `ai-dynamo[sglang]`
#### Install latest release
We suggest using uv to install the latest release of ai-dynamo[sglang]. You can install it with
`curl -LsSf https://astral.sh/uv/install.sh | sh`
```
bash
# create a virtual env
uv venv
--python
3.12
--seed
# install the latest release
uv pip
install
"ai-dynamo[sglang]"
```
#### Installing editable version for development
<details>
<summary>
Instructions
</summary>
This requires having rust installed. We also recommend having a proper installation of the cuda toolkit as sglang requires
`nvcc`
to be available.
```
bash
# create a virtual env
uv venv
--python
3.12
--seed
# build dynamo runtime bindings
uv pip
install
maturin
cd
$DYNAMO_HOME
/lib/bindings/python
maturin develop
--uv
cd
$DYNAMO_HOME
uv pip
install
.
export
PYTHONPATH
=
"
${
PYTHONPATH
}
:
$(
pwd
)
/components/backends/sglang/src"
# install target sglang version (you can choose any version)
# we include the prerelease flag in order to install flashinfer rc versions
uv pip
install
--prerelease
=
allow sglang[all]
==
0.4.9.post6
```
</details>
#### Using prebuilt docker containers
<details>
<summary>
Instructions
</summary>
```
bash
```
bash
# pull our pre-build sglang runtime container
docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2
docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2
# or build from source
./container/build.sh
--framework
sglang
```
```
### Run container
</details>
#### Building docker container from source
<details>
<summary>
Instructions
</summary>
```
bash
```
bash
./container/run.sh
-it
--framework
sglang
./container/build.sh
--framework
sglang
# run container using prebuild wheel
./container/run.sh
--framework
sglang
-it
# mount workspace for development
./container/run.sh
--framework
sglang
--mount-workspace
```
```
</details>
## Run Single Node Examples
## Run Single Node Examples
> [!IMPORTANT]
> [!IMPORTANT]
...
...
components/backends/sglang/launch/agg_router.sh
View file @
d51580a4
...
@@ -25,4 +25,5 @@ python3 -m dynamo.sglang.worker \
...
@@ -25,4 +25,5 @@ python3 -m dynamo.sglang.worker \
--page-size
16
\
--page-size
16
\
--tp
1
\
--tp
1
\
--trust-remote-code
\
--trust-remote-code
\
--skip-tokenizer-init
--skip-tokenizer-init
\
--kv-events-config
'{"publisher": "zmq", "topic": "kv-events"}'
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment