Unverified Commit 8c6ab977 authored by ishandhanani's avatar ishandhanani Committed by GitHub
Browse files

fix(deps): sglang install must be done manually (#1019)


Signed-off-by: default avatarishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: default avatarRyan McCormick <rmccormick@nvidia.com>
parent 4564a387
...@@ -19,6 +19,9 @@ limitations under the License. ...@@ -19,6 +19,9 @@ limitations under the License.
This directory contains examples and reference implementations for deploying Large Language Models (LLMs) in various configurations using SGLang. SGLang internally uses ZMQ to communicate between the ingress and the engine processes. For Dynamo, we leverage the runtime to communicate directly with the engine processes and handle ingress and pre/post processing on our end. This directory contains examples and reference implementations for deploying Large Language Models (LLMs) in various configurations using SGLang. SGLang internally uses ZMQ to communicate between the ingress and the engine processes. For Dynamo, we leverage the runtime to communicate directly with the engine processes and handle ingress and pre/post processing on our end.
> [!IMPORTANT]
> In order to run these examples, you will need to install sglang using `uv pip install "sglang[all]>=0.4.6.post2"`. Additionally, SGLang currently does not have pre-built wheels for ARM. If you are on an ARM machine - you will need to install SGLang from source.
## Deployment Architectures ## Deployment Architectures
See [deployment architectures](../llm/README.md#deployment-architectures) to learn about the general idea of the architecture. SGLang currently support only aggregated serving but routing and disaggregation support are coming very soon! See [deployment architectures](../llm/README.md#deployment-architectures) to learn about the general idea of the architecture. SGLang currently support only aggregated serving but routing and disaggregation support are coming very soon!
...@@ -32,6 +35,7 @@ See [deployment architectures](../llm/README.md#deployment-architectures) to lea ...@@ -32,6 +35,7 @@ See [deployment architectures](../llm/README.md#deployment-architectures) to lea
### Prerequisites ### Prerequisites
Start required services (etcd and NATS) using [Docker Compose](../../deploy/docker-compose.yml) Start required services (etcd and NATS) using [Docker Compose](../../deploy/docker-compose.yml)
```bash ```bash
docker compose -f deploy/docker-compose.yml up -d docker compose -f deploy/docker-compose.yml up -d
``` ```
...@@ -57,4 +61,3 @@ docker compose -f deploy/docker-compose.yml up -d ...@@ -57,4 +61,3 @@ docker compose -f deploy/docker-compose.yml up -d
cd /workspace/examples/sglang cd /workspace/examples/sglang
dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml
``` ```
...@@ -34,7 +34,6 @@ dependencies = [ ...@@ -34,7 +34,6 @@ dependencies = [
"distro", "distro",
"typer", "typer",
"circus>=0.17.0", "circus>=0.17.0",
"sglang[all]==0.4.6.post2"
] ]
classifiers = [ classifiers = [
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment