Skip to content

GitLab

Unverified Commit 4e38d628 authored by Keiven C's avatar Keiven C Committed by GitHub
Browse files

feat: devcontainer.json to work for vLLM, SGLang, and TensorRT-LLM (#3228) 


Signed-off-by: default avatarKeiven Chang <keivenchang@users.noreply.github.com>
parent c142eacf
Show whitespace changes
Inline Side-by-side
Showing with 401 additions and 13 deletions
.devcontainer/README.md
View file @ 4e38d628
...@@ -19,6 +19,51 @@ limitations under the License. ...@@ -19,6 +19,51 @@ limitations under the License.
> Warning: Dev Containers (aka `devcontainers`) is an evolving feature and we are not testing in CI. Please submit any problem/feedback using the issues on GitHub. > Warning: Dev Containers (aka `devcontainers`) is an evolving feature and we are not testing in CI. Please submit any problem/feedback using the issues on GitHub.
## Framework-Specific Devcontainers
This directory contains framework-specific devcontainer configurations generated from a Jinja2 template:
- **`vllm/`** - Development environment for vLLM framework
- **`sglang/`** - Development environment for SGLang framework
- **`trtllm/`** - Development environment for TensorRT-LLM framework
### Template System
The devcontainer configurations are generated from:
- **`devcontainer.json.j2`** - Jinja2 template with framework variables
- **`gen_devcontainer_json.py`** - Python script to generate configs
To regenerate the framework-specific configurations after making changes:
```bash
cd .devcontainer
python3 gen_devcontainer_json.py
```
**Important**: Do not edit the generated `devcontainer.json` files directly. They contain auto-generated warnings and will be overwritten. Instead, edit the `devcontainer.json.j2` template and regenerate.
#### Why We Use Templates Instead of a Single devcontainer.json
The Dev Container Extension requires that each `devcontainer.json` file follow a specific directory convention, which results in significant duplication across framework-specific configurations. See https://code.visualstudio.com/remote/advancedcontainers/connect-multiple-containers
```
📁 project-root
📁 .git
📁 .devcontainer
📁 python-container
📄 devcontainer.json
📁 node-container
📄 devcontainer.json
📁 python-src
📄 hello.py
📁 node-src
📄 hello.js
📄 docker-compose.yml
```
Alternative approaches using undocumented methods (e.g., changing devcontainer.json name, custom configurations) were explored but proved unsuccessful. The template system was developed to minimize duplication while maintaining compatibility with the Dev Container Extension's directory requirements.
Until the Microsoft Dev Container Extension adds new functionalities, this remains the recommended approach for managing multiple Dev Container configurations.
```mermaid ```mermaid
graph TB graph TB
subgraph "Developer Laptop" subgraph "Developer Laptop"
...@@ -41,7 +86,7 @@ graph TB ...@@ -41,7 +86,7 @@ graph TB
TOOLS["rust-analyzer<br/>cargo<br/>etc."] TOOLS["rust-analyzer<br/>cargo<br/>etc."]
end end
IMAGE["Docker Image<br/>dynamo:latest-vllm-local-dev"] IMAGE["Docker Image<br/>dynamo:latest-{framework}-local-dev<br/>(vllm/sglang/trtllm)"]
IMAGE -->|"docker run<br/>as ubuntu user"| CONTAINER IMAGE -->|"docker run<br/>as ubuntu user"| CONTAINER
end end
...@@ -89,24 +134,41 @@ You must have the following path on your host. ...@@ -89,24 +134,41 @@ You must have the following path on your host.
Follow these steps to get your NVIDIA Dynamo development environment up and running: Follow these steps to get your NVIDIA Dynamo development environment up and running:
### Step 1: Build the Development Container Image ### Step 0: Build the Development Container Image
Build `dynamo:latest-vllm-local-dev` from scratch from the source: Build the appropriate framework image (e.g., `dynamo:latest-vllm-local-dev`) from scratch from the source:
```bash ```bash
# Single command approach (recommended) # Single command approach (recommended)
./container/build.sh --framework VLLM --target local-dev export FRAMEWORK=VLLM # Note: any of VLLM, SGLANG, TRTLLM can be used
# Creates both dynamo:latest-vllm and dynamo:latest-vllm-local-dev ./container/build.sh --framework $FRAMEWORK --target local-dev
# Now you've created both dynamo:latest-vllm and dynamo:latest-vllm-local-dev
```
Alternatively, you can build a development container, then build local-dev:
# Alternatively, you can build a development container then local-dev ```bash
./container/build.sh --framework VLLM export FRAMEWORK=VLLM
./container/build.sh --framework $FRAMEWORK
# Now you have a development image dynamo:latest-vllm # Now you have a development image dynamo:latest-vllm
./container/build.sh --dev-image dynamo:latest-vllm --framework VLLM
./container/build.sh --dev-image dynamo:latest-${FRAMEWORK,,}
# Now you have a local-dev image dynamo:latest-vllm-local-dev # Now you have a local-dev image dynamo:latest-vllm-local-dev
``` ```
The local-dev image will give you local user permissions matching your host user and includes extra developer utilities (debugging tools, text editors, system monitors, etc.). The local-dev image will give you local user permissions matching your host user and includes extra developer utilities (debugging tools, text editors, system monitors, etc.).
### Step 1: Choose Your Framework
Select the appropriate devcontainer based on your framework:
- Use `vllm/devcontainer.json` for vLLM development
- Use `sglang/devcontainer.json` for SGLang development
- Use `trtllm/devcontainer.json` for TensorRT-LLM development
When opening the devcontainer in VS Code/Cursor, navigate to the specific framework directory (e.g., `.devcontainer/vllm/`) and open that devcontainer.json.
### Step 2: Install Dev Containers Extension ### Step 2: Install Dev Containers Extension
**For Cursor:** **For Cursor:**
...@@ -362,9 +424,10 @@ If you see errors like "container is not running" or "An error occurred setting ...@@ -362,9 +424,10 @@ If you see errors like "container is not running" or "An error occurred setting
docker images | grep dynamo docker images | grep dynamo
# If missing, build the dev image first, then build local-dev # If missing, build the dev image first, then build local-dev
./container/build.sh --framework vllm export FRAMEWORK=VLLM # Replace with VLLM, SGLANG, or TRTLLM
./container/build.sh --dev-image dynamo:latest-vllm --framework vllm ./container/build.sh --framework $FRAMEWORK
# Output: dynamo:latest-vllm-local-dev ./container/build.sh --dev-image dynamo:latest-${FRAMEWORK,,} --framework $FRAMEWORK
# Now you have dynamo:latest-vllm-local-dev
``` ```
2. **Container startup failure:** 2. **Container startup failure:**
......
.devcontainer/devcontainer.json.j2 0 → 100644
View file @ 4e38d628
{
"$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.schema.json",
"copyright": [
"SPDX-FileCopyrightText: Copyright (c) {{ current_year }} NVIDIA CORPORATION & AFFILIATES. All rights reserved.",
"SPDX-License-Identifier: Apache-2.0"
],
"name": "Dynamo {{ framework.upper() }} Dev Container",
"remoteUser": "ubuntu", // Matches our container user
"updateRemoteUserUID": true, // Updates the UID of the remote user to match the host user, avoids permission errors
"image": "dynamo:latest-{{ framework }}-local-dev", // Use the latest {{ framework.upper() }} dev image
"runArgs": [
"--gpus=all",
"--network=host",
"--ipc=host",
"--cap-add=SYS_PTRACE",
"--shm-size=10G",
"--ulimit=memlock=-1",
"--ulimit=stack=67108864",
"--ulimit=nofile=65536:65536"
],
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"rust-lang.rust-analyzer"
],
"settings": {
// Disable automatic copying of .gitconfig to avoid errors
"dev.containers.copyGitConfig": false,
"terminal.integrated.defaultProfile.linux": "bash",
"terminal.integrated.cwd": "/home/ubuntu/dynamo",
"python.defaultInterpreterPath": "/opt/dynamo/venv/bin/python",
"python.linting.enabled": true,
"rust-analyzer.memoryLimit": 4096, // larger memory limit to reduce latency
"rust-analyzer.checkOnSave.command": "clippy",
"rust-analyzer.checkOnSave.enable": true,
"rust-analyzer.cargo.buildScripts.enable": true,
"rust-analyzer.cargo.targetDir": "/home/ubuntu/dynamo/.build/target",
"rust-analyzer.procMacro.enable": true,
"rust-analyzer.completion.autoimport.enable": true,
// Enhanced rust-analyzer configuration
"rust-analyzer.linkedProjects": [
"Cargo.toml",
"lib/runtime/Cargo.toml",
"lib/llm/Cargo.toml",
"lib/tokens/Cargo.toml",
"lib/bindings/python/Cargo.toml",
"launch/dynamo-run/Cargo.toml"
],
"files.trimTrailingWhitespace": true,
"files.insertFinalNewline": true
}
}
},
"workspaceFolder": "/home/ubuntu/dynamo",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/ubuntu/dynamo,type=bind,consistency=cached",
"userEnvProbe": "interactiveShell",
"postCreateCommand": "/bin/bash /home/ubuntu/dynamo/.devcontainer/post-create.sh", // Runs cargo build and pip installs packages
"containerEnv": {
"DYNAMO_HOME": "/home/ubuntu/dynamo",
"CARGO_HOME": "/home/ubuntu/dynamo/.build/.cargo",
"RUSTUP_HOME": "/home/ubuntu/dynamo/.build/.rustup",
"CARGO_TARGET_DIR": "/home/ubuntu/dynamo/.build/target"
},
"remoteEnv": {
// Optional convenience tokens passed from host. SSH agent is forwarded via your IDE setting, not here by default.
"GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
"HF_TOKEN": "${localEnv:HF_TOKEN}"
// "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}" // Optional: only if you also bind-mount the socket path
},
"mounts": [
// These are for convenience, so that the history and pre-commit cache are persisted between sessions
"source=dynamo-bashhistory,target=/home/ubuntu/.commandhistory,type=volume",
"source=dynamo-precommit-cache,target=/home/ubuntu/.cache/pre-commit,type=volume",
// Default mounts
"source=/tmp/,target=/tmp/,type=bind"
// Uncomment this to reuse your Hugging Face cache
//"source=${localEnv:HOME}/.cache/huggingface,target=/home/ubuntu/.cache/huggingface,type=bind"
]
}
.devcontainer/gen_devcontainer_json.py 0 → 100755
View file @ 4e38d628
#!/usr/bin/env python3
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""
Script to generate devcontainer.json files for different frameworks using Jinja2 template.
"""
from datetime import datetime
from pathlib import Path
from jinja2 import Environment, FileSystemLoader
def main():
"""Generate devcontainer.json files for different frameworks."""
# Define the frameworks to generate
frameworks = ["vllm", "sglang", "trtllm"]
# Get the current directory (where this script is located)
script_dir = Path(__file__).parent
# Setup Jinja2 environment
env = Environment(loader=FileSystemLoader(script_dir))
template = env.get_template("devcontainer.json.j2")
# Generate devcontainer.json for each framework
for framework in frameworks:
# Create the target directory
target_dir = script_dir / framework
target_dir.mkdir(exist_ok=True)
# Render the template with framework-specific values
current_year = datetime.now().year
rendered_content = template.render(
framework=framework, current_year=current_year
)
# Add auto-generated warning to the beginning of the file
warning_comment = """// AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
// This file was generated from devcontainer.json.j2
// To make changes, edit the .j2 template and run gen_devcontainer_json.py
"""
# Insert warning after the opening brace
lines = rendered_content.split("\n")
if lines[0].strip() == "{":
lines.insert(1, warning_comment.rstrip())
rendered_content = "\n".join(lines)
# Write the rendered content to the target file
target_file = target_dir / "devcontainer.json"
with open(target_file, "w") as f:
f.write(rendered_content)
print(f"Generated {target_file}")
print(
f"Successfully generated devcontainer.json files for: {', '.join(frameworks)}"
)
if __name__ == "__main__":
main()
.devcontainer/sglang/devcontainer.json 0 → 100644
View file @ 4e38d628
{
// AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
// This file was generated from devcontainer.json.j2
// To make changes, edit the .j2 template and run gen_devcontainer_json.py
"$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.schema.json",
"copyright": [
"SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.",
"SPDX-License-Identifier: Apache-2.0"
],
"name": "Dynamo SGLANG Dev Container",
"remoteUser": "ubuntu", // Matches our container user
"updateRemoteUserUID": true, // Updates the UID of the remote user to match the host user, avoids permission errors
"image": "dynamo:latest-sglang-local-dev", // Use the latest SGLANG dev image
"runArgs": [
"--gpus=all",
"--network=host",
"--ipc=host",
"--cap-add=SYS_PTRACE",
"--shm-size=10G",
"--ulimit=memlock=-1",
"--ulimit=stack=67108864",
"--ulimit=nofile=65536:65536"
],
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"rust-lang.rust-analyzer"
],
"settings": {
// Disable automatic copying of .gitconfig to avoid errors
"dev.containers.copyGitConfig": false,
"terminal.integrated.defaultProfile.linux": "bash",
"terminal.integrated.cwd": "/home/ubuntu/dynamo",
"python.defaultInterpreterPath": "/opt/dynamo/venv/bin/python",
"python.linting.enabled": true,
"rust-analyzer.memoryLimit": 4096, // larger memory limit to reduce latency
"rust-analyzer.checkOnSave.command": "clippy",
"rust-analyzer.checkOnSave.enable": true,
"rust-analyzer.cargo.buildScripts.enable": true,
"rust-analyzer.cargo.targetDir": "/home/ubuntu/dynamo/.build/target",
"rust-analyzer.procMacro.enable": true,
"rust-analyzer.completion.autoimport.enable": true,
// Enhanced rust-analyzer configuration
"rust-analyzer.linkedProjects": [
"Cargo.toml",
"lib/runtime/Cargo.toml",
"lib/llm/Cargo.toml",
"lib/tokens/Cargo.toml",
"lib/bindings/python/Cargo.toml",
"launch/dynamo-run/Cargo.toml"
],
"files.trimTrailingWhitespace": true,
"files.insertFinalNewline": true
}
}
},
"workspaceFolder": "/home/ubuntu/dynamo",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/ubuntu/dynamo,type=bind,consistency=cached",
"userEnvProbe": "interactiveShell",
"postCreateCommand": "/bin/bash /home/ubuntu/dynamo/.devcontainer/post-create.sh", // Runs cargo build and pip installs packages
"containerEnv": {
"DYNAMO_HOME": "/home/ubuntu/dynamo",
"CARGO_HOME": "/home/ubuntu/dynamo/.build/.cargo",
"RUSTUP_HOME": "/home/ubuntu/dynamo/.build/.rustup",
"CARGO_TARGET_DIR": "/home/ubuntu/dynamo/.build/target"
},
"remoteEnv": {
// Optional convenience tokens passed from host. SSH agent is forwarded via your IDE setting, not here by default.
"GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
"HF_TOKEN": "${localEnv:HF_TOKEN}"
// "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}" // Optional: only if you also bind-mount the socket path
},
"mounts": [
// These are for convenience, so that the history and pre-commit cache are persisted between sessions
"source=dynamo-bashhistory,target=/home/ubuntu/.commandhistory,type=volume",
"source=dynamo-precommit-cache,target=/home/ubuntu/.cache/pre-commit,type=volume",
// Default mounts
"source=/tmp/,target=/tmp/,type=bind"
// Uncomment this to reuse your Hugging Face cache
//"source=${localEnv:HOME}/.cache/huggingface,target=/home/ubuntu/.cache/huggingface,type=bind"
]
}
\ No newline at end of file
.devcontainer/trtllm/devcontainer.json 0 → 100644
View file @ 4e38d628
{
// AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
// This file was generated from devcontainer.json.j2
// To make changes, edit the .j2 template and run gen_devcontainer_json.py
"$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.schema.json",
"copyright": [
"SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.",
"SPDX-License-Identifier: Apache-2.0"
],
"name": "Dynamo TRTLLM Dev Container",
"remoteUser": "ubuntu", // Matches our container user
"updateRemoteUserUID": true, // Updates the UID of the remote user to match the host user, avoids permission errors
"image": "dynamo:latest-trtllm-local-dev", // Use the latest TRTLLM dev image
"runArgs": [
"--gpus=all",
"--network=host",
"--ipc=host",
"--cap-add=SYS_PTRACE",
"--shm-size=10G",
"--ulimit=memlock=-1",
"--ulimit=stack=67108864",
"--ulimit=nofile=65536:65536"
],
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"rust-lang.rust-analyzer"
],
"settings": {
// Disable automatic copying of .gitconfig to avoid errors
"dev.containers.copyGitConfig": false,
"terminal.integrated.defaultProfile.linux": "bash",
"terminal.integrated.cwd": "/home/ubuntu/dynamo",
"python.defaultInterpreterPath": "/opt/dynamo/venv/bin/python",
"python.linting.enabled": true,
"rust-analyzer.memoryLimit": 4096, // larger memory limit to reduce latency
"rust-analyzer.checkOnSave.command": "clippy",
"rust-analyzer.checkOnSave.enable": true,
"rust-analyzer.cargo.buildScripts.enable": true,
"rust-analyzer.cargo.targetDir": "/home/ubuntu/dynamo/.build/target",
"rust-analyzer.procMacro.enable": true,
"rust-analyzer.completion.autoimport.enable": true,
// Enhanced rust-analyzer configuration
"rust-analyzer.linkedProjects": [
"Cargo.toml",
"lib/runtime/Cargo.toml",
"lib/llm/Cargo.toml",
"lib/tokens/Cargo.toml",
"lib/bindings/python/Cargo.toml",
"launch/dynamo-run/Cargo.toml"
],
"files.trimTrailingWhitespace": true,
"files.insertFinalNewline": true
}
}
},
"workspaceFolder": "/home/ubuntu/dynamo",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/ubuntu/dynamo,type=bind,consistency=cached",
"userEnvProbe": "interactiveShell",
"postCreateCommand": "/bin/bash /home/ubuntu/dynamo/.devcontainer/post-create.sh", // Runs cargo build and pip installs packages
"containerEnv": {
"DYNAMO_HOME": "/home/ubuntu/dynamo",
"CARGO_HOME": "/home/ubuntu/dynamo/.build/.cargo",
"RUSTUP_HOME": "/home/ubuntu/dynamo/.build/.rustup",
"CARGO_TARGET_DIR": "/home/ubuntu/dynamo/.build/target"
},
"remoteEnv": {
// Optional convenience tokens passed from host. SSH agent is forwarded via your IDE setting, not here by default.
"GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
"HF_TOKEN": "${localEnv:HF_TOKEN}"
// "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}" // Optional: only if you also bind-mount the socket path
},
"mounts": [
// These are for convenience, so that the history and pre-commit cache are persisted between sessions
"source=dynamo-bashhistory,target=/home/ubuntu/.commandhistory,type=volume",
"source=dynamo-precommit-cache,target=/home/ubuntu/.cache/pre-commit,type=volume",
// Default mounts
"source=/tmp/,target=/tmp/,type=bind"
// Uncomment this to reuse your Hugging Face cache
//"source=${localEnv:HOME}/.cache/huggingface,target=/home/ubuntu/.cache/huggingface,type=bind"
]
}
\ No newline at end of file
.devcontainer/devcontainer.json → .devcontainer/vllm/devcontainer.json
View file @ 4e38d628
{ {
// AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY
// This file was generated from devcontainer.json.j2
// To make changes, edit the .j2 template and run gen_devcontainer_json.py
"$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.schema.json", "$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.schema.json",
"copyright": [ "copyright": [
"SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.", "SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.",
"SPDX-License-Identifier: Apache-2.0" "SPDX-License-Identifier: Apache-2.0"
], ],
"name": "NVIDIA Dynamo Dev Container Development", "name": "Dynamo VLLM Dev Container",
"remoteUser": "ubuntu", // Matches our container user "remoteUser": "ubuntu", // Matches our container user
"updateRemoteUserUID": true, // Updates the UID of the remote user to match the host user, avoids permission errors "updateRemoteUserUID": true, // Updates the UID of the remote user to match the host user, avoids permission errors
"image": "dynamo:latest-vllm-local-dev", // Use the latest VLLM dev image "image": "dynamo:latest-vllm-local-dev", // Use the latest VLLM dev image
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment