"docs/components/profiler/profiler-guide.md" did not exist on "1412e44ba92a38afef5d026ab5225189c280014b"
README.md 18.5 KB
Newer Older
1
<!--
2
SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3
4
5
6
7
SPDX-License-Identifier: Apache-2.0
-->

# NVIDIA Dynamo Development Environment

8
> Warning: Dev Containers (aka `devcontainers`) is an evolving feature and we are not testing in CI. Please submit any problem/feedback using the issues on GitHub.
9

10
11
12
13
14
15
16
17
18
19
20
21
22
23
## Known Issues

### Docker Version Compatibility

**Docker 29.x has compatibility issues with Dev Containers (by Anysphere):**
- It is known that Docker Engine version **29.0.1** (released November 14, 2025) may cause Dev Containers to hang all the time, rendering it unusable
- The Dev Containers extension (v1.0.26) and devcontainer CLI (v0.75.0) were not tested against Docker 29.x
- Symptoms: Container builds successfully but connection hangs, requiring a manual reload to connect
- This may be fixed in a later version of the Anysphere Dev Containers extension and/or Docker Engine patch

**Recommended Docker Version:**
- Use Docker Engine **28.5.2** or earlier for stable Dev Container operation
- Docker 27.x and 28.x series are confirmed working with current Dev Containers tooling

24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
## Framework-Specific Devcontainers

This directory contains framework-specific devcontainer configurations generated from a Jinja2 template:

- **`vllm/`** - Development environment for vLLM framework
- **`sglang/`** - Development environment for SGLang framework
- **`trtllm/`** - Development environment for TensorRT-LLM framework

### Template System

The devcontainer configurations are generated from:
- **`devcontainer.json.j2`** - Jinja2 template with framework variables
- **`gen_devcontainer_json.py`** - Python script to generate configs

To regenerate the framework-specific configurations after making changes:
```bash
cd .devcontainer
python3 gen_devcontainer_json.py
```

**Important**: Do not edit the generated `devcontainer.json` files directly. They contain auto-generated warnings and will be overwritten. Instead, edit the `devcontainer.json.j2` template and regenerate.

#### Why We Use Templates Instead of a Single devcontainer.json

The Dev Container Extension requires that each `devcontainer.json` file follow a specific directory convention, which results in significant duplication across framework-specific configurations. See https://code.visualstudio.com/remote/advancedcontainers/connect-multiple-containers

```
📁 project-root
    📁 .git
    📁 .devcontainer
      📁 python-container
        📄 devcontainer.json
      📁 node-container
        📄 devcontainer.json
    📁 python-src
        📄 hello.py
    📁 node-src
        📄 hello.js
    📄 docker-compose.yml
```

Alternative approaches using undocumented methods (e.g., changing devcontainer.json name, custom configurations) were explored but proved unsuccessful. The template system was developed to minimize duplication while maintaining compatibility with the Dev Container Extension's directory requirements.

Until the Microsoft Dev Container Extension adds new functionalities, this remains the recommended approach for managing multiple Dev Container configurations.

69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
```mermaid
graph TB
    subgraph "Developer Laptop"
        subgraph IDE["Cursor/VS Code"]
            EXT["Dev Container Extension"]
            SSHEXT["Remote-SSH Extension"]
        end
        TERM["iTerm/Terminal"]
    end

    subgraph WS["Linux Workstation"]
        DIR["~/dynamo<br/>(host directory)"]
        HFCACHE["~/.cache/huggingface<br/>(host cache)"]
        GITCONFIG["~/.gitconfig<br/>(host git config)"]

84
85
86
87
        subgraph CONTAINER["Docker Container<br/>vsc-dynamo-SHA-uid<br/>Running as: dynamo user"]
            MOUNT["/home/dynamo/dynamo<br/>(mounted directory)"]
            HFMOUNT["/home/dynamo/.cache/huggingface<br/>(mounted cache)"]
            GITCONFIGCOPY["/home/dynamo/.gitconfig<br/>(via Dev Container setting)"]
88
89
90
            TOOLS["rust-analyzer<br/>cargo<br/>etc."]
        end

91
        IMAGE["Docker Image<br/>dynamo:latest-{framework}-local-dev<br/>(vllm/sglang/trtllm)"]
92

93
        IMAGE -->|"docker run<br/>as dynamo user"| CONTAINER
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
    end

    TERM -->|"SSH Connection"| DIR
    SSHEXT -->|"1. Remote-SSH"| DIR
    EXT -->|"2. Dev Container:<br/>Open Folder (via ssh)"| IMAGE
    EXT -->|"3. IDE/shell via<br/>ssh+docker exec"| CONTAINER
    DIR <-.->|"Volume Mount<br/>(bidirectional)"| MOUNT
    HFCACHE <-.->|"Volume Mount<br/>(HF cache)"| HFMOUNT
    GITCONFIG -->|"Copy"| GITCONFIGCOPY

    style IDE fill:#e1f5fe
    style EXT fill:#cfe2ff
    style SSHEXT fill:#cfe2ff
    style TERM fill:#e1f5fe
    style DIR fill:#fff3e0
    style HFCACHE fill:#fff3e0
    style GITCONFIG fill:#fff3e0
    style IMAGE fill:#f3e5f5
    style CONTAINER fill:#d4edda
    style MOUNT fill:#fff9c4
    style HFMOUNT fill:#fff9c4
    style GITCONFIGCOPY fill:#fff9c4
    style TOOLS fill:#ffebee
```

119
## Prerequisites
120
121
122

Before you begin, ensure you have the following installed:

123
- [Docker](https://docs.docker.com/get-started/get-docker/) installed and configured on your host system
124
125
- IDEs: Use either the VS Code or Cursor. Both have Dev Containers extensions
- Appropriate NVIDIA drivers (compatible with CUDA 12.8+)
126
- For models that require authentication, set your Hugging Face token env var `HF_TOKEN` in your local startup (.profile or .zprofile file). Many public models do not require this token.
127
128
129

### Required Files and Directories

130
You must have the following path on your host.
131

132
- **`~/.cache/huggingface`**: This directory is mounted into the container for Hugging Face model caching.
133

134

135
## Steps to Get Started
136
137
138

Follow these steps to get your NVIDIA Dynamo development environment up and running:

139
### Step 0: Build the Development Container Image
140

141
Build the appropriate framework image (e.g., `dynamo:latest-vllm-local-dev`) from scratch from the source:
142
143

```bash
144
# Single command approach (recommended)
145
export FRAMEWORK=VLLM         # Note: any of VLLM, SGLANG, TRTLLM can be used
146
python container/render.py --framework=${FRAMEWORK} --target=local-dev --output-short-filename
147
docker build --build-arg USER_UID=$(id -u) --build-arg USER_GID=$(id -g) -f container/rendered.Dockerfile .
148
149
150
151

# Now you've created both dynamo:latest-vllm and dynamo:latest-vllm-local-dev
```

152
The local-dev image will give you local user permissions matching your host user and includes extra developer utilities (debugging tools, text editors, system monitors, etc.).
153

154
155
156
157
158
159
160
161
162
### Step 1: Choose Your Framework

Select the appropriate devcontainer based on your framework:
- Use `vllm/devcontainer.json` for vLLM development
- Use `sglang/devcontainer.json` for SGLang development
- Use `trtllm/devcontainer.json` for TensorRT-LLM development

When opening the devcontainer in VS Code/Cursor, navigate to the specific framework directory (e.g., `.devcontainer/vllm/`) and open that devcontainer.json.

163
### Step 2: Install Dev Containers Extension
164

165
166
167
**For Cursor:**
- Press `Cmd+Shift+X` (Mac) or `Ctrl+Shift+X` (Linux/Windows) to open Extensions
- Search for "Dev Containers" and install the one by **Anysphere** (Do not download the version from Microsoft as it is not compatible with Cursor)
168

169
170
171
172
**For VS Code:**
- Install [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) from Microsoft marketplace


173
### Step 3: Launch the Development Environment
174

175
176
177
178
1. Open `dynamo` folder in your IDE
2. Press `Cmd+Shift+P` (Mac) or `Ctrl+Shift+P` (Linux/Windows)
3. Select "Dev Containers: Open Folder in Container", select your `dynamo` folder and open.

179
### Step 4: Optional But Highly Recommended Setup
180

181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
For the best development experience, we recommend configuring the following:

#### Git Configuration
- **Personal `.gitconfig`**: If not done already, make your own `~/.gitconfig` file with your name, email, and preferred Git settings
- **Dev Container Integration**: Enable the `Copy the .gitconfig file to the devcontainer` setting by going to IDE Settings → Extensions → Dev Containers → **☑ Copy the .gitconfig file to the devcontainer**

#### SSH Authentication
To enable seamless Git operations (push/pull) via SSH:

1. **Configure ssh-agent on your host**: Search online for "how to set up ssh-agent on Ubuntu on .profile" and ensure it's configured in your `~/.profile` so the ssh-agent runs before Cursor launches

2. **Enable SSH forwarding in IDE**:
   Dev Containers set up -> check **☑ Dev Containers: Enable SSHAgent Forwarding**

3. **Verify SSH forwarding works**:
   - The post-create.sh script will check and report SSH agent status
   - You should see "SSH agent forwarding is working" when the container starts

This setup allows you to use Git commands normally within the container without additional authentication steps.
200
201
202
203
204


### Step 5: Wait for Initialization

The container will automatically:
205
- Mount your local code to `/workspace`
206
- Run `post-create.sh` to build the project and configure the environment
207

208
209
210
211
If `post-create.sh` fails, you can try to debug or [submit](https://github.com/ai-dynamo/dynamo/issues) an issue on GitHub.

## Development Flow

212
213
### Building Rust Code

214
If you make changes to Rust code and want to compile, use [cargo build](https://doc.rust-lang.org/cargo/commands/cargo-build.html). This will update Rust binaries.
215
216

```bash
217
cd /workspace && cargo build --locked --profile dev
218
219
```

220
Verify that builds are in the pre-defined `target` directory:
221
222
```bash
$ cargo metadata --format-version=1 | jq -r '.target_directory'
223
/workspace/target  <-- this is the target path
224
225
```

226
If cargo is not installed and configured properly, you will see one or more errors, such as the following:
227
228
229
230
```
error: could not find `Cargo.toml` in $HOME or any parent directory
```

231
Lastly, before pushing code to GitHub, remember to run `cargo fmt` and `cargo clippy`
232

233
234
### Updating Python Bindings

235
236
237
If you make changes to Rust code and want to propagate to Python bindings then can use [maturin](https://www.maturin.rs/#usage) (pre-installed). This will update the Python bindings with your new Rust changes.

```bash
238
cd /workspace/lib/bindings/python && maturin develop
239
```
240
241
242
243
244

## What's Inside
Development Environment:
- Rust and Python toolchains
- GPU acceleration
245
- VS Code or Cursor extensions for Rust and Python
246
- Persistent build cache in `target/` directory enables fast incremental builds (only changed files are recompiled) via `cargo build --locked --profile dev`
247
- Edits to files are propagated to local repo due to the volume mount
248
249
250
- SSH and GPG agent passthrough orchestrated by devcontainer

File Structure:
251
- Local dynamo repo mounts to `/workspace`
252
- Python venv in `/opt/dynamo/venv`
253
- Build artifacts in `/workspace/target`
254
255
256
- Hugging Face cache preserved between sessions (either mounting your host .cache to the container, or your `HF_HOME` to `/home/dynamo/.cache/huggingface`)
- Bash memory preserved between sessions at `/home/dynamo/.commandhistory` using docker volume `dynamo-bashhistory`
- Precommit preserved between sessions at `/home/dynamo/.cache/precommit` using docker volume `dynamo-precommit-cache`
257

258
259
260
261
## Documentation

To look at the docs run:
```bash
262
cd /workspace/target/doc && python3 -m http.server 8000
263
264
265
266
```

VSCode will automatically port-forward and you can check them out in your browser.

267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
## FAQ

### GPG Keys for Signing Git Commits
Signing commits using GPG should work out of the box according to [VSCode docs](https://code.visualstudio.com/remote/advancedcontainers/sharing-git-credentials#_sharing-gpg-keys).

If you run into version compatibility issues you can try:

```bash
# On Host
gpg --list-secret-keys
gpg --export-secret-keys --armor YOUR_KEY_ID > /tmp/key.asc

# In container
gpg1 --import /tmp/key.asc
git config --local gpg.program gpg1
```

> Warning: Switching local gpg to gpg1 can have ramifications when you are not in the container any longer.

286
### Custom devcontainer.json Configuration
287

288
You can create a custom devcontainer configuration by copying the main configuration to another directory inside the `.devcontainer` directory. Below is an example where the custom name is `jensen_dev`, but feel free to name the directory whatever you want:
289
290

```bash
291
292
# By convention, Dev Container will look at the project's .devcontainer/<path>/devcontainer.json file.
# Example: copy the main devcontainer configuration and then edit the new json file
293
294
mkdir -p .devcontainer/jensen_dev
cp .devcontainer/devcontainer.json .devcontainer/jensen_dev/devcontainer.json
295
296
```

297
Common customizations include additional mounts, environment variables, IDE extensions, and build arguments. When you open a new Dev Container, you can pick from any of the `.devcontainer/<path>/devcontainer.json` files available.
298

299

300
301
302
303
304
305
### SSH Keys for Git Operations

If you have ssh-agent running on the host, then `git push` should just work. If not, you may need to set up ssh-agent, or have SSH keys set up inside the container (more hassle).


## Troubleshooting
306
307
308

### Environment Variables Not Set in Container?

309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
Dev containers have limited access to host environment variables for security reasons. Here's how to properly pass environment variables:

#### Method 1: Use devcontainer.json
Add environment variables to `.devcontainer/devcontainer.json`:
```json
{
  "remoteEnv": {
    "HF_TOKEN": "${localEnv:HF_TOKEN}",
    "GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
    "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}"
  }
}
```

#### Method 2: Host Shell Configuration
In order to ensure your host environment variables are available to the Dev Containers:

**For bash users:**
- Put variables in `~/.profile` (for login shells)
- Ensure `~/.bash_profile` sources `~/.bashrc` if both exist
- Some users report that variables in `~/.bashrc` aren't picked up by dev containers, so try moving them to `~/.profile` to see if that solves the problem

**For zsh users:**
- Put variables in `~/.zprofile` (for login shells)
- Some users report that variables in `~/.zshrc` aren't picked up by dev containers, so try moving them to `~/.zprofile` to see if that solves the problem
334

335
336
#### Method 3: Environment File
Create a `.env` file in your project root and reference it in devcontainer.json:
337

338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
**Example .env file:**
```bash
# API Tokens
HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Development settings
DEBUG=true
LOG_LEVEL=debug

# Custom paths
CUSTOM_MODEL_PATH=/path/to/models
```

**Reference in devcontainer.json:**
```json
{
  "remoteEnv": {
    "ENV_FILE": "${localWorkspaceFolder}/.env"
  }
}
```
360
361

See VS Code Dev Containers [documentation](https://code.visualstudio.com/docs/devcontainers/containers) for more details.
362
363
364

### Build Issues

365
If you encounter build errors or strange compilation issues, try running `cargo clean`, then rebuild from scratch.
366
367
368
369

If `cargo clean` doesn't resolve the issue, it is possible that some of the files were created by root (using the `run.sh` script). You can manually remove the build target by going to your host (outside the container), and remove the target:

```bash
370
sudo rm -rf <your dynamo path on the host machine>/target
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
```

### Volume Corruption Issues

If you encounter strange errors (like `postCreateCommand` failing with exit code 1), your Docker volumes may be corrupted.

**Solution: Wipe Docker Volumes**

```bash
# Remove Dynamo volumes that are specified in devcontainer.json (may be corrupted)
docker volume rm dynamo-bashhistory dynamo-precommit-cache

# Or remove all volumes (use with caution).
docker rm -f <your running container(s)>
docker volume prune -f
```

**Note:** This resets bash history and pre-commit cache.

**Volume Mounts in devcontainer.json:**
391
392
- `dynamo-bashhistory``/home/dynamo/.commandhistory` (bash history)
- `dynamo-precommit-cache``/home/dynamo/.cache/pre-commit` (pre-commit cache)
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411

### Permission Issues

If you start experiencing permission problems (e.g., "Permission denied" errors), you may need to fix file ownership outside the container. This commonly happens when `container/run.sh` runs as root, creating files with root ownership:

```bash
# Replace <user> with your actual username
cd <your dynamo directory at your host machine (not docker)>
sudo chown -R <user>:<user> .
```

This fixes ownership when files are created with different user IDs between the host and container.

### Container Starts But Immediately Stops

If you see errors like "container is not running" or "An error occurred setting up the container" in the devcontainer logs, the container is starting but then crashing immediately.

**Common Causes and Solutions:**

412
1. **Missing a local-dev image:**
413
   ```bash
414
   # Check if the required local-dev image exists
415
416
   docker images | grep dynamo

417
   # If missing, build the dev image first, then build local-dev
418
   export FRAMEWORK=VLLM  # Replace with VLLM, SGLANG, or TRTLLM
419
   python container/render.py --framework=${FRAMEWORK} --target=local-dev --output-short-filename
420
   docker build --build-arg USER_UID=$(id -u) --build-arg USER_GID=$(id -g) -f container/rendered.Dockerfile .
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
   ```

2. **Container startup failure:**
   ```bash
   # Check container logs for the specific error
   docker logs <container-id>

   # Or check all recent containers
   docker ps -a --filter "label=devcontainer.local_folder=$(pwd)"
   ```

3. **Resource issues:**
   ```bash
   # Check available system resources
   free -h
   df -h

   # Restart Docker daemon if needed
   sudo systemctl restart docker
   ```

4. **Clean slate approach:**
   ```bash
   # Remove all related containers and images
   docker ps -a --filter "label=devcontainer.local_folder=$(pwd)" -q | xargs docker rm -f
   docker images | grep "^vsc-" | awk '{print $3}' | xargs docker rmi
   ```
448
  Then rebuild without cache. In your IDE:
449
450
451
452
453
454
455
456
457
  *Dev Containers: Rebuild Without Cache and Reopen in Container*

### devcontainer.json Changes Not Being Picked Up

If you've made changes to `devcontainer.json`, `post-create.sh`, or other devcontainer-related files but they're not being applied when you rebuild the container, the changes may be cached.

**Solution: Force Devcontainer Rebuild**

1. **Rebuild Container (Recommended):**
458
   In your IDE Command Palette (Ctrl+Shift+P):
459
460
461
   *Dev Containers: Rebuild Container*

2. **If that doesn't work, rebuild without cache:**
462
   In your IDE Command Palette (Ctrl+Shift+P):
463
464
465
466
467
468
469
   *Dev Containers: Rebuild Without Cache and Reopen Container*

3. **For persistent issues, manually remove the devcontainer image:**
   ```bash
   # List devcontainer images
   docker images | grep devcontainer

470
   # And remove all IDE devcontainer images (more thorough)
471
472
   docker images | grep "^vsc-" | awk '{print $3}' | xargs docker rmi

473
   # Then rebuild in your IDE
474
475
476
477
   Dev Containers: Rebuild Container
   ```

**Note:** The "Rebuild Container Without Cache and Reopen Container" option is the most thorough and will ensure all your changes are applied, but it takes longer as it rebuilds everything from scratch.