get_started.md 6.45 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<!--
SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES.
All rights reserved.
SPDX-License-Identifier: Apache-2.0

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Getting Started

21

22
23
## Development Environment

24
This section describes how to set up your development environment.
25

26
### Recommended Setup: Using Dev Container
27

28
We recommend using our pre-configured development container:
29

30
1. Install prerequisites:
31
32
33
34

   - [Docker](https://www.docker.com/products/docker-desktop)
   - [Visual Studio Code](https://code.visualstudio.com/)
   - [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
35

36
2. Get the code:
37

38
39
40
41
   ```bash
   git clone https://github.com/ai-dynamo/dynamo.git
   cd dynamo
   ```
42

43
3. Open in Visual Studio Code:
44
45
46
47

   1. Launch Visual Studio Code
   2. Click the button in the bottom-left corner
   3. Select **Reopen in Container**
48

49
Visual Studio Code builds and starts a container with all necessary dependencies for Dynamo development.
50

51
52
53
54
55
### Alternative Setup: Manual Installation

If you don't want to use the dev container, you can set the environment up manually:

1. Ensure you have:
56
57
58
59
60

   - Ubuntu 24.04 (recommended)
   - x86_64 CPU
   - Python 3.x
   - Git
61

62
   See [Support Matrix](support_matrix.md) for more information.
63

64
2. **If you plan to use vLLM or SGLang**, you must also install:
65
66
   - etcd
   - NATS.io
67
68
69
70

   Before starting dyanmo, run both etcd and NATS.io in seperate processes.

3. Install required system packages:
71
72
73
74
75
   ```bash
   apt-get update
   DEBIAN_FRONTEND=noninteractive apt-get install -yq python3-dev python3-pip python3-venv libucx0
   ```

76
4. Set up the Python environment:
77
78
79
80
81
   ```bash
   python3 -m venv venv
   source venv/bin/activate
   ```

82
5. Install Dynamo:
83
84
85
   ```bash
   pip install "ai-dynamo[all]"
   ```
86

87
88
89
> [!Important]
> To ensure compatibility, use the examples in the release branch or tag that matches the version you installed.

90
91
92

## Building the Dynamo Base Image

93
94
95
Deploying your Dynamo pipelines to Kubernetes requires you to build and push a Dynamo base image to your container registry.
You can use any private container registry of your choice, including:

96
97
98
- [Docker Hub](https://hub.docker.com/)
- [NVIDIA NGC Container Registry](https://catalog.ngc.nvidia.com/)

99
100
101
102
103
104
105
106
107
108
109

To build it:

```bash
./container/build.sh
docker tag dynamo:latest-vllm <your-registry>/dynamo-base:latest-vllm
docker login <your-registry>
docker push <your-registry>/dynamo-base:latest-vllm
```

This documentation describes these frameworks:
110
111
112
113
114
115

- `--framework vllm` build:
   See [LLM Deployment Examples](examples/llm_deployment.md).

- `--framework tensorrtllm` build:
   See [TRTLLM Deployment Examples](examples/trtllm.md).
116
117

After building, use this image by setting the `DYNAMO_IMAGE` environment variable to point to your built image:
118

119
120
121
122
```bash
export DYNAMO_IMAGE=<your-registry>/dynamo-base:latest-vllm
```

123

124
125
## Running and Interacting with an LLM Locally

126
127
To run a model and interact with it locally, call `dynamo run` with a Hugging Face model.
`dynamo run` supports several backends, including `mistralrs`, `sglang`, `vllm`, and `tensorrtllm`.
128
129
130

### Example Command

131
```bash
132
133
134
dynamo run out=vllm deepseek-ai/DeepSeek-R1-Distill-Llama-8B
```

135
```bash
136
137
138
139
140
141
142
? User › Hello, how are you?
✔ User · Hello, how are you?
Okay, so I'm trying to figure out how to respond to the user's greeting.
They said, "Hello, how are you?" and then followed it with "Hello! I'm just a program, but thanks for asking."
Hmm, I need to come up with a suitable reply. ...
```

143

144
145
146
147
## LLM Serving

Dynamo provides a simple way to spin up a local set of inference components including:

148
149
150
151
152
153
154
155
- **OpenAI-compatible Frontend**:
   High-performance OpenAI compatible http api server written in Rust.

- **Basic and Kv Aware Router**:
   Route and load balance traffic to a set of workers.

- **Workers**:
   Set of pre-configured LLM serving engines.
156

157
To run a minimal configuration, use a pre-configured example.
158
159
160
161
162
163
164
165

### Start Dynamo Distributed Runtime Services

To start the Dynamo Distributed Runtime services the first time:

```bash
docker compose -f deploy/docker-compose.yml up -d
```
166

167
168
169
170
171
172
173
174
175
176
177
178
179
### Start Dynamo LLM Serving Components

Next, serve a minimal configuration with an http server, basic
round-robin router, and a single worker.

```bash
cd examples/llm
dynamo serve graphs.agg:Frontend -f configs/agg.yaml
```

### Send a Request

```bash
180
181
182
curl localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
183
184
185
186
187
188
189
190
191
192
193
194
    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    "messages": [
    {
        "role": "user",
        "content": "Hello, how are you?"
    }
    ],
    "stream":false,
    "max_tokens": 300
  }' | jq
```

195

196
197
## Local Development

198
199
If you use vscode or cursor, use the `.devcontainer` folder built on [Microsoft's Extension](https://code.visualstudio.com/docs/devcontainers/containers).
For instructions, see the Dynamo repository's [devcontainer README](https://github.com/ai-dynamo/dynamo/blob/main/.devcontainer/README.md).
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218

Otherwise, to develop locally, we recommend working inside of the container:

```bash
./container/build.sh
./container/run.sh -it --mount-workspace

cargo build --release
mkdir -p /workspace/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin
cp /workspace/target/release/http /workspace/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin
cp /workspace/target/release/llmctl /workspace/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin
cp /workspace/target/release/dynamo-run /workspace/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin

uv pip install -e .
export PYTHONPATH=$PYTHONPATH:/workspace/deploy/dynamo/sdk/src:/workspace/components/planner/src
```

### Conda Environment

219
Alternatively, use a Conda environment:
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239

```bash
conda activate <ENV_NAME>

pip install nixl # Or install https://github.com/ai-dynamo/nixl from source

cargo build --release

# To install ai-dynamo-runtime from source
cd lib/bindings/python
pip install .

cd ../../../
pip install .[all]

# To test
docker compose -f deploy/docker-compose.yml up -d
cd examples/llm
dynamo serve graphs.agg:Frontend -f configs/agg.yaml
```