"lib/llm/src/discovery/watcher.rs" did not exist on "dddebc0df2a6952a508cd1a127c7aff0bc60d934"
quickstart.md 4.18 KB
Newer Older
1
2
3
4
5
---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
---

6
This guide covers running Dynamo **using the CLI on your local machine or VM**.
7

8
9
10
11
12
<Info>
**Looking to deploy on Kubernetes instead?**
See the [Kubernetes Installation Guide](../kubernetes/installation-guide.md)
and [Kubernetes Quickstart](../kubernetes/README.md) for cluster deployments.
</Info>
13

14
## Install Dynamo
15

16
**Option A: Containers (Recommended)**
17

18
Containers have all dependencies pre-installed. No setup required.
19

20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
```bash
# SGLang
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.8.1

# TensorRT-LLM
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.8.1

# vLLM
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.8.1
```

<Tip>
To run frontend and worker in the same container, either:

- Run processes in background with `&` (see Run Dynamo section below), or
- Open a second terminal and use `docker exec -it <container_id> bash`
</Tip>

See [Release Artifacts](../reference/release-artifacts.md#container-images) for available
versions and backend guides for run instructions: [SGLang](../backends/sglang/README.md) |
[TensorRT-LLM](../backends/trtllm/README.md) | [vLLM](../backends/vllm/README.md)

**Option B: Install from PyPI**
43
44
45
46
47

```bash
# Install uv (recommended Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

48
# Create virtual environment
49
50
uv venv venv
source venv/bin/activate
51
uv pip install pip
52
53
```

54
55
56
Install system dependencies and the Dynamo wheel for your chosen backend:

**SGLang**
57
58

```bash
59
60
sudo apt install python3-dev
uv pip install --prerelease=allow "ai-dynamo[sglang]"
61
62
```

63
64
65
66
67
68
<Note>
For CUDA 13 (B300/GB300), the container is recommended. See
[SGLang install docs](https://docs.sglang.io/get_started/install.html) for details.
</Note>

**TensorRT-LLM**
69
70

```bash
71
72
73
74
75
76
77
78
79
80
81
sudo apt install python3-dev
pip install torch==2.9.0 torchvision --index-url https://download.pytorch.org/whl/cu130
pip install --pre --extra-index-url https://pypi.nvidia.com "ai-dynamo[trtllm]"
```

<Note>
TensorRT-LLM requires `pip` due to a transitive Git URL dependency that
`uv` doesn't resolve. We recommend using the TensorRT-LLM container for
broader compatibility. See the [TRT-LLM backend guide](../backends/trtllm/README.md)
for details.
</Note>
82

83
84
85
86
87
**vLLM**

```bash
sudo apt install python3-dev libxcb1
uv pip install --prerelease=allow "ai-dynamo[vllm]"
88
89
```

90
91
92
93
94
95
96
97
98
99
100
101
102
## Run Dynamo

<Tip>
**(Optional)** Before running Dynamo, verify your system configuration:
`python3 deploy/sanity_check.py`
</Tip>

Start the frontend, then start a worker for your chosen backend.

<Tip>
To run in a single terminal (useful in containers), append `> logfile.log 2>&1 &`
to run processes in background. Example: `python3 -m dynamo.frontend --store-kv file > dynamo.frontend.log 2>&1 &`
</Tip>
103
104

```bash
105
106
107
108
109
110
111
112
113
114
115
# Start the OpenAI compatible frontend (default port is 8000)
# --store-kv file avoids needing etcd (frontend and workers must share a disk)
python3 -m dynamo.frontend --store-kv file
```

In another terminal (or same terminal if using background mode), start a worker:

**SGLang**

```bash
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --store-kv file
116
117
```

118
**TensorRT-LLM**
119

120
121
122
```bash
python3 -m dynamo.trtllm --model-path Qwen/Qwen3-0.6B --store-kv file
```
123

124
**vLLM**
125

126
127
128
129
```bash
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --store-kv file \
  --kv-events-config '{"enable_kv_cache_events": false}'
```
130

131
132
<Note>
For dependency-free local development, disable KV event publishing (avoids NATS):
133

134
135
136
- **vLLM:** Add `--kv-events-config '{"enable_kv_cache_events": false}'`
- **SGLang:** No flag needed (KV events disabled by default)
- **TensorRT-LLM:** No flag needed (KV events disabled by default)
137

138
139
140
**TensorRT-LLM only:** The warning `Cannot connect to ModelExpress server/transport error. Using direct download.`
is expected and can be safely ignored.
</Note>
141

142
## Test Your Deployment
143

144
145
146
147
148
149
150
```bash
curl localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B",
       "messages": [{"role": "user", "content": "Hello!"}],
       "max_tokens": 50}'
```