Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
dynamo
Commits
d51580a4
"...git@developer.sourcefind.cn:2222/OpenDAS/vllm_cscc.git" did not exist on "59d7ffc17f4c948b4d25d014e4f90d0cd4c20990"
Unverified
Commit
d51580a4
authored
Aug 11, 2025
by
ishandhanani
Committed by
GitHub
Aug 11, 2025
Browse files
docs: add sglang installation variants (#2400)
parent
4d24d03b
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
55 additions
and
7 deletions
+55
-7
components/backends/sglang/README.md
components/backends/sglang/README.md
+53
-6
components/backends/sglang/launch/agg_router.sh
components/backends/sglang/launch/agg_router.sh
+2
-1
No files found.
components/backends/sglang/README.md
View file @
d51580a4
...
@@ -62,21 +62,68 @@ Start using [Docker Compose](../../../deploy/docker-compose.yml)
...
@@ -62,21 +62,68 @@ Start using [Docker Compose](../../../deploy/docker-compose.yml)
docker compose
-f
deploy/docker-compose.yml up
-d
docker compose
-f
deploy/docker-compose.yml up
-d
```
```
### Build container
### Install `ai-dynamo[sglang]`
#### Install latest release
We suggest using uv to install the latest release of ai-dynamo[sglang]. You can install it with
`curl -LsSf https://astral.sh/uv/install.sh | sh`
```
bash
# create a virtual env
uv venv
--python
3.12
--seed
# install the latest release
uv pip
install
"ai-dynamo[sglang]"
```
#### Installing editable version for development
<details>
<summary>
Instructions
</summary>
This requires having rust installed. We also recommend having a proper installation of the cuda toolkit as sglang requires
`nvcc`
to be available.
```
bash
# create a virtual env
uv venv
--python
3.12
--seed
# build dynamo runtime bindings
uv pip
install
maturin
cd
$DYNAMO_HOME
/lib/bindings/python
maturin develop
--uv
cd
$DYNAMO_HOME
uv pip
install
.
export
PYTHONPATH
=
"
${
PYTHONPATH
}
:
$(
pwd
)
/components/backends/sglang/src"
# install target sglang version (you can choose any version)
# we include the prerelease flag in order to install flashinfer rc versions
uv pip
install
--prerelease
=
allow sglang[all]
==
0.4.9.post6
```
</details>
#### Using prebuilt docker containers
<details>
<summary>
Instructions
</summary>
```
bash
```
bash
# pull our pre-build sglang runtime container
docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2
docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2
# or build from source
./container/build.sh
--framework
sglang
```
```
### Run container
</details>
#### Building docker container from source
<details>
<summary>
Instructions
</summary>
```
bash
```
bash
./container/run.sh
-it
--framework
sglang
./container/build.sh
--framework
sglang
# run container using prebuild wheel
./container/run.sh
--framework
sglang
-it
# mount workspace for development
./container/run.sh
--framework
sglang
--mount-workspace
```
```
</details>
## Run Single Node Examples
## Run Single Node Examples
> [!IMPORTANT]
> [!IMPORTANT]
...
...
components/backends/sglang/launch/agg_router.sh
View file @
d51580a4
...
@@ -25,4 +25,5 @@ python3 -m dynamo.sglang.worker \
...
@@ -25,4 +25,5 @@ python3 -m dynamo.sglang.worker \
--page-size
16
\
--page-size
16
\
--tp
1
\
--tp
1
\
--trust-remote-code
\
--trust-remote-code
\
--skip-tokenizer-init
--skip-tokenizer-init
\
--kv-events-config
'{"publisher": "zmq", "topic": "kv-events"}'
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment