README.md 18.5 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<!--
SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# NVIDIA Dynamo Development Environment

20
> Warning: Dev Containers (aka `devcontainers`) is an evolving feature and we are not testing in CI. Please submit any problem/feedback using the issues on GitHub.
21

22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
## Framework-Specific Devcontainers

This directory contains framework-specific devcontainer configurations generated from a Jinja2 template:

- **`vllm/`** - Development environment for vLLM framework
- **`sglang/`** - Development environment for SGLang framework
- **`trtllm/`** - Development environment for TensorRT-LLM framework

### Template System

The devcontainer configurations are generated from:
- **`devcontainer.json.j2`** - Jinja2 template with framework variables
- **`gen_devcontainer_json.py`** - Python script to generate configs

To regenerate the framework-specific configurations after making changes:
```bash
cd .devcontainer
python3 gen_devcontainer_json.py
```

**Important**: Do not edit the generated `devcontainer.json` files directly. They contain auto-generated warnings and will be overwritten. Instead, edit the `devcontainer.json.j2` template and regenerate.

#### Why We Use Templates Instead of a Single devcontainer.json

The Dev Container Extension requires that each `devcontainer.json` file follow a specific directory convention, which results in significant duplication across framework-specific configurations. See https://code.visualstudio.com/remote/advancedcontainers/connect-multiple-containers

```
📁 project-root
    📁 .git
    📁 .devcontainer
      📁 python-container
        📄 devcontainer.json
      📁 node-container
        📄 devcontainer.json
    📁 python-src
        📄 hello.py
    📁 node-src
        📄 hello.js
    📄 docker-compose.yml
```

Alternative approaches using undocumented methods (e.g., changing devcontainer.json name, custom configurations) were explored but proved unsuccessful. The template system was developed to minimize duplication while maintaining compatibility with the Dev Container Extension's directory requirements.

Until the Microsoft Dev Container Extension adds new functionalities, this remains the recommended approach for managing multiple Dev Container configurations.

67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
```mermaid
graph TB
    subgraph "Developer Laptop"
        subgraph IDE["Cursor/VS Code"]
            EXT["Dev Container Extension"]
            SSHEXT["Remote-SSH Extension"]
        end
        TERM["iTerm/Terminal"]
    end

    subgraph WS["Linux Workstation"]
        DIR["~/dynamo<br/>(host directory)"]
        HFCACHE["~/.cache/huggingface<br/>(host cache)"]
        GITCONFIG["~/.gitconfig<br/>(host git config)"]

        subgraph CONTAINER["Docker Container<br/>vsc-dynamo-SHA-uid<br/>Running as: ubuntu user"]
            MOUNT["/home/ubuntu/dynamo<br/>(mounted directory)"]
            HFMOUNT["/home/ubuntu/.cache/huggingface<br/>(mounted cache)"]
            GITCONFIGCOPY["/home/ubuntu/.gitconfig<br/>(via Dev Container setting)"]
            TOOLS["rust-analyzer<br/>cargo<br/>etc."]
        end

89
        IMAGE["Docker Image<br/>dynamo:latest-{framework}-local-dev<br/>(vllm/sglang/trtllm)"]
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116

        IMAGE -->|"docker run<br/>as ubuntu user"| CONTAINER
    end

    TERM -->|"SSH Connection"| DIR
    SSHEXT -->|"1. Remote-SSH"| DIR
    EXT -->|"2. Dev Container:<br/>Open Folder (via ssh)"| IMAGE
    EXT -->|"3. IDE/shell via<br/>ssh+docker exec"| CONTAINER
    DIR <-.->|"Volume Mount<br/>(bidirectional)"| MOUNT
    HFCACHE <-.->|"Volume Mount<br/>(HF cache)"| HFMOUNT
    GITCONFIG -->|"Copy"| GITCONFIGCOPY

    style IDE fill:#e1f5fe
    style EXT fill:#cfe2ff
    style SSHEXT fill:#cfe2ff
    style TERM fill:#e1f5fe
    style DIR fill:#fff3e0
    style HFCACHE fill:#fff3e0
    style GITCONFIG fill:#fff3e0
    style IMAGE fill:#f3e5f5
    style CONTAINER fill:#d4edda
    style MOUNT fill:#fff9c4
    style HFMOUNT fill:#fff9c4
    style GITCONFIGCOPY fill:#fff9c4
    style TOOLS fill:#ffebee
```

117
## Prerequisites
118
119
120

Before you begin, ensure you have the following installed:

121
- [Docker](https://docs.docker.com/get-started/get-docker/) installed and configured on your host system
122
123
- IDEs: Use either the VS Code or Cursor. Both have Dev Containers extensions
- Appropriate NVIDIA drivers (compatible with CUDA 12.8+)
124
- For models that require authentication, set your Hugging Face token env var `HF_TOKEN` in your local startup (.profile or .zprofile file). Many public models do not require this token.
125
126
127

### Required Files and Directories

128
You must have the following path on your host.
129

130
- **`~/.cache/huggingface`**: This directory is mounted into the container for Hugging Face model caching.
131

132

133
## Steps to Get Started
134
135
136

Follow these steps to get your NVIDIA Dynamo development environment up and running:

137
### Step 0: Build the Development Container Image
138

139
Build the appropriate framework image (e.g., `dynamo:latest-vllm-local-dev`) from scratch from the source:
140
141

```bash
142
# Single command approach (recommended)
143
144
145
146
147
148
149
export FRAMEWORK=VLLM         # Note: any of VLLM, SGLANG, TRTLLM can be used
./container/build.sh --framework $FRAMEWORK --target local-dev

# Now you've created both dynamo:latest-vllm and dynamo:latest-vllm-local-dev
```

Alternatively, you can build a development container, then build local-dev:
150

151
152
153
154
```bash
export FRAMEWORK=VLLM

./container/build.sh --framework $FRAMEWORK
155
# Now you have a development image dynamo:latest-vllm
156
157

./container/build.sh --dev-image dynamo:latest-${FRAMEWORK,,}
158
# Now you have a local-dev image dynamo:latest-vllm-local-dev
159
160
```

161
The local-dev image will give you local user permissions matching your host user and includes extra developer utilities (debugging tools, text editors, system monitors, etc.).
162

163
164
165
166
167
168
169
170
171
### Step 1: Choose Your Framework

Select the appropriate devcontainer based on your framework:
- Use `vllm/devcontainer.json` for vLLM development
- Use `sglang/devcontainer.json` for SGLang development
- Use `trtllm/devcontainer.json` for TensorRT-LLM development

When opening the devcontainer in VS Code/Cursor, navigate to the specific framework directory (e.g., `.devcontainer/vllm/`) and open that devcontainer.json.

172
### Step 2: Install Dev Containers Extension
173

174
175
176
**For Cursor:**
- Press `Cmd+Shift+X` (Mac) or `Ctrl+Shift+X` (Linux/Windows) to open Extensions
- Search for "Dev Containers" and install the one by **Anysphere** (Do not download the version from Microsoft as it is not compatible with Cursor)
177

178
179
180
181
**For VS Code:**
- Install [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) from Microsoft marketplace


182
### Step 3: Launch the Development Environment
183

184
185
186
187
1. Open `dynamo` folder in your IDE
2. Press `Cmd+Shift+P` (Mac) or `Ctrl+Shift+P` (Linux/Windows)
3. Select "Dev Containers: Open Folder in Container", select your `dynamo` folder and open.

188
### Step 4: Optional But Highly Recommended Setup
189

190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
For the best development experience, we recommend configuring the following:

#### Git Configuration
- **Personal `.gitconfig`**: If not done already, make your own `~/.gitconfig` file with your name, email, and preferred Git settings
- **Dev Container Integration**: Enable the `Copy the .gitconfig file to the devcontainer` setting by going to IDE Settings → Extensions → Dev Containers → **☑ Copy the .gitconfig file to the devcontainer**

#### SSH Authentication
To enable seamless Git operations (push/pull) via SSH:

1. **Configure ssh-agent on your host**: Search online for "how to set up ssh-agent on Ubuntu on .profile" and ensure it's configured in your `~/.profile` so the ssh-agent runs before Cursor launches

2. **Enable SSH forwarding in IDE**:
   Dev Containers set up -> check **☑ Dev Containers: Enable SSHAgent Forwarding**

3. **Verify SSH forwarding works**:
   - The post-create.sh script will check and report SSH agent status
   - You should see "SSH agent forwarding is working" when the container starts

This setup allows you to use Git commands normally within the container without additional authentication steps.
209
210
211
212
213
214
215


### Step 5: Wait for Initialization

The container will automatically:
- Mount your local code to `/home/ubuntu/dynamo`
- Run `post-create.sh` to build the project and configure the environment
216

217
218
219
220
If `post-create.sh` fails, you can try to debug or [submit](https://github.com/ai-dynamo/dynamo/issues) an issue on GitHub.

## Development Flow

221
222
### Building Rust Code

223
224
225
226
227
228
If you make changes to Rust code and want to compile, use [cargo build](https://doc.rust-lang.org/cargo/commands/cargo-build.html). This will update Rust binaries such as dynamo-run.

```bash
cd /home/ubuntu/dynamo && cargo build --locked --profile dev
```

229
230
231
232
233
234
Verify that builds are in the pre-defined `dynamo/.build/target` and not `dynamo/workspace`:
```bash
$ cargo metadata --format-version=1 | jq -r '.target_directory'
/home/ubuntu/dynamo/.build/target  <-- this is the target path
```

235
If cargo is not installed and configured properly, you will see one or more errors, such as the following:
236
237
238
239
```
error: could not find `Cargo.toml` in $HOME or any parent directory
```

240
Lastly, before pushing code to GitHub, remember to run `cargo fmt` and `cargo clippy`
241

242
243
### Updating Python Bindings

244
245
246
247
248
If you make changes to Rust code and want to propagate to Python bindings then can use [maturin](https://www.maturin.rs/#usage) (pre-installed). This will update the Python bindings with your new Rust changes.

```bash
cd /home/ubuntu/dynamo/lib/bindings/python && maturin develop
```
249
250
251
252
253

## What's Inside
Development Environment:
- Rust and Python toolchains
- GPU acceleration
254
255
256
- VS Code or Cursor extensions for Rust and Python
- Persistent build cache in `.build/` directory enables fast incremental builds (only changed files are recompiled) via `cargo build --locked --profile dev`
- Edits to files are propagated to local repo due to the volume mount
257
258
259
260
261
262
- SSH and GPG agent passthrough orchestrated by devcontainer

File Structure:
- Local dynamo repo mounts to `/home/ubuntu/dynamo`
- Python venv in `/opt/dynamo/venv`
- Build artifacts in `dynamo/.build/target`
263
- Hugging Face cache preserved between sessions (either mounting your host .cache to the container, or your `HF_HOME` to `/home/ubuntu/.cache/huggingface`)
264
- Bash memory preserved between sessions at `/home/ubuntu/.commandhistory` using docker volume `dynamo-bashhistory`
265
- Precommit preserved between sessions at `/home/ubuntu/.cache/precommit` using docker volume `dynamo-precommit-cache`
266

267
268
269
270
271
272
273
274
275
## Documentation

To look at the docs run:
```bash
cd ~/dynamo/.build/target/doc && python3 -m http.server 8000
```

VSCode will automatically port-forward and you can check them out in your browser.

276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
## FAQ

### GPG Keys for Signing Git Commits
Signing commits using GPG should work out of the box according to [VSCode docs](https://code.visualstudio.com/remote/advancedcontainers/sharing-git-credentials#_sharing-gpg-keys).

If you run into version compatibility issues you can try:

```bash
# On Host
gpg --list-secret-keys
gpg --export-secret-keys --armor YOUR_KEY_ID > /tmp/key.asc

# In container
gpg1 --import /tmp/key.asc
git config --local gpg.program gpg1
```

> Warning: Switching local gpg to gpg1 can have ramifications when you are not in the container any longer.

295
### Custom devcontainer.json Configuration
296

297
You can create a custom devcontainer configuration by copying the main configuration to another directory inside the `.devcontainer` directory. Below is an example where the custom name is `jensen_dev`, but feel free to name the directory whatever you want:
298
299

```bash
300
301
# By convention, Dev Container will look at the project's .devcontainer/<path>/devcontainer.json file.
# Example: copy the main devcontainer configuration and then edit the new json file
302
303
mkdir -p .devcontainer/jensen_dev
cp .devcontainer/devcontainer.json .devcontainer/jensen_dev/devcontainer.json
304
305
```

306
Common customizations include additional mounts, environment variables, IDE extensions, and build arguments. When you open a new Dev Container, you can pick from any of the `.devcontainer/<path>/devcontainer.json` files available.
307

308

309
310
311
312
313
314
### SSH Keys for Git Operations

If you have ssh-agent running on the host, then `git push` should just work. If not, you may need to set up ssh-agent, or have SSH keys set up inside the container (more hassle).


## Troubleshooting
315
316
317

### Environment Variables Not Set in Container?

318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
Dev containers have limited access to host environment variables for security reasons. Here's how to properly pass environment variables:

#### Method 1: Use devcontainer.json
Add environment variables to `.devcontainer/devcontainer.json`:
```json
{
  "remoteEnv": {
    "HF_TOKEN": "${localEnv:HF_TOKEN}",
    "GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
    "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}"
  }
}
```

#### Method 2: Host Shell Configuration
In order to ensure your host environment variables are available to the Dev Containers:

**For bash users:**
- Put variables in `~/.profile` (for login shells)
- Ensure `~/.bash_profile` sources `~/.bashrc` if both exist
- Some users report that variables in `~/.bashrc` aren't picked up by dev containers, so try moving them to `~/.profile` to see if that solves the problem

**For zsh users:**
- Put variables in `~/.zprofile` (for login shells)
- Some users report that variables in `~/.zshrc` aren't picked up by dev containers, so try moving them to `~/.zprofile` to see if that solves the problem
343

344
345
#### Method 3: Environment File
Create a `.env` file in your project root and reference it in devcontainer.json:
346

347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
**Example .env file:**
```bash
# API Tokens
HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Development settings
DEBUG=true
LOG_LEVEL=debug

# Custom paths
CUSTOM_MODEL_PATH=/path/to/models
```

**Reference in devcontainer.json:**
```json
{
  "remoteEnv": {
    "ENV_FILE": "${localWorkspaceFolder}/.env"
  }
}
```
369
370

See VS Code Dev Containers [documentation](https://code.visualstudio.com/docs/devcontainers/containers) for more details.
371
372
373

### Build Issues

374
If you encounter build errors or strange compilation issues, try running `cargo clean`, then rebuild from scratch.
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420

If `cargo clean` doesn't resolve the issue, it is possible that some of the files were created by root (using the `run.sh` script). You can manually remove the build target by going to your host (outside the container), and remove the target:

```bash
sudo rm -rf <your dynamo path on the host machine>/.build/target
```

### Volume Corruption Issues

If you encounter strange errors (like `postCreateCommand` failing with exit code 1), your Docker volumes may be corrupted.

**Solution: Wipe Docker Volumes**

```bash
# Remove Dynamo volumes that are specified in devcontainer.json (may be corrupted)
docker volume rm dynamo-bashhistory dynamo-precommit-cache

# Or remove all volumes (use with caution).
docker rm -f <your running container(s)>
docker volume prune -f
```

**Note:** This resets bash history and pre-commit cache.

**Volume Mounts in devcontainer.json:**
- `dynamo-bashhistory``/home/ubuntu/.commandhistory` (bash history)
- `dynamo-precommit-cache``/home/ubuntu/.cache/pre-commit` (pre-commit cache)

### Permission Issues

If you start experiencing permission problems (e.g., "Permission denied" errors), you may need to fix file ownership outside the container. This commonly happens when `container/run.sh` runs as root, creating files with root ownership:

```bash
# Replace <user> with your actual username
cd <your dynamo directory at your host machine (not docker)>
sudo chown -R <user>:<user> .
```

This fixes ownership when files are created with different user IDs between the host and container.

### Container Starts But Immediately Stops

If you see errors like "container is not running" or "An error occurred setting up the container" in the devcontainer logs, the container is starting but then crashing immediately.

**Common Causes and Solutions:**

421
1. **Missing a local-dev image:**
422
   ```bash
423
   # Check if the required local-dev image exists
424
425
   docker images | grep dynamo

426
   # If missing, build the dev image first, then build local-dev
427
428
429
430
   export FRAMEWORK=VLLM  # Replace with VLLM, SGLANG, or TRTLLM
   ./container/build.sh --framework $FRAMEWORK
   ./container/build.sh --dev-image dynamo:latest-${FRAMEWORK,,} --framework $FRAMEWORK
   # Now you have dynamo:latest-vllm-local-dev
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
   ```

2. **Container startup failure:**
   ```bash
   # Check container logs for the specific error
   docker logs <container-id>

   # Or check all recent containers
   docker ps -a --filter "label=devcontainer.local_folder=$(pwd)"
   ```

3. **Resource issues:**
   ```bash
   # Check available system resources
   free -h
   df -h

   # Restart Docker daemon if needed
   sudo systemctl restart docker
   ```

4. **Clean slate approach:**
   ```bash
   # Remove all related containers and images
   docker ps -a --filter "label=devcontainer.local_folder=$(pwd)" -q | xargs docker rm -f
   docker images | grep "^vsc-" | awk '{print $3}' | xargs docker rmi
   ```
458
  Then rebuild without cache. In your IDE:
459
460
461
462
463
464
465
466
467
  *Dev Containers: Rebuild Without Cache and Reopen in Container*

### devcontainer.json Changes Not Being Picked Up

If you've made changes to `devcontainer.json`, `post-create.sh`, or other devcontainer-related files but they're not being applied when you rebuild the container, the changes may be cached.

**Solution: Force Devcontainer Rebuild**

1. **Rebuild Container (Recommended):**
468
   In your IDE Command Palette (Ctrl+Shift+P):
469
470
471
   *Dev Containers: Rebuild Container*

2. **If that doesn't work, rebuild without cache:**
472
   In your IDE Command Palette (Ctrl+Shift+P):
473
474
475
476
477
478
479
   *Dev Containers: Rebuild Without Cache and Reopen Container*

3. **For persistent issues, manually remove the devcontainer image:**
   ```bash
   # List devcontainer images
   docker images | grep devcontainer

480
   # And remove all IDE devcontainer images (more thorough)
481
482
   docker images | grep "^vsc-" | awk '{print $3}' | xargs docker rmi

483
   # Then rebuild in your IDE
484
485
486
487
   Dev Containers: Rebuild Container
   ```

**Note:** The "Rebuild Container Without Cache and Reopen Container" option is the most thorough and will ensure all your changes are applied, but it takes longer as it rebuilds everything from scratch.