feat: add .devcontainer based off images in container/ (#497)

Co-authored-by: Ubuntu <ubuntu@crusoe-prod--inst-2secxwyxeao1mi2dsqk2pl9oaz7.eu-iceland1-a.comp>
Co-authored-by: Hannah Zhang <hannahz@nvidia.com>
Co-authored-by: hongkuanz <hongkuanz@nvidia.com>

.devcontainer/README.md

+<!--
+SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# NVIDIA Dynamo Development Environment
+
+> Warning: devcontainers is an experimental feature and we are not testing in CI. Please submit any feedback using the issues on github.
+
+## Prerequisites
+- [Docker](https://docs.docker.com/get-started/get-docker/) installed and configured on your host system
+- Visual Studio Code with the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) installed
+- Appropriate NVIDIA drivers (compatible with CUDA 12.8)
+
+## Quick Start
+1. Build the container image
+
+```bash
+./container/build.sh --target local-dev
+```
+
+The container will be built and give certain file permissions to your local uid and gid.
+
+> Note: Currently local-dev is only implemented for --framework VLLM
+
+2. Open Dynamo folder in VS Code
+- Press Ctrl + Shift + P
+- Select "Dev Containers: Open Folder in Container"
+
+If you want to mount your hugging face cache, go to `.devcontainer` and uncomment in the mounts section:
+
+```json
+// "source=${localEnv:HF_HOME},target=/home/ubuntu/.cache/huggingface,type=bind", // Uncomment to enable HF Cache Mount. Make sure to set HF_HOME env var in you .bashrc
+```
+Make sure HF_HOME is sourced in your .bashrc or .zshenv and your vscode default terminal is set properly.
+
+3. Wait for Initialization
+- The container will mount your local code
+- `post-create.sh` will build the project and configure the environment
+
+If `post-create.sh` fails, you can try to debug or [submit](https://github.com/ai-dynamo/dynamo/issues) an issue on github.
+
+## What's Inside
+Development Environment:
+- Rust and Python toolchains
+- GPU acceleration
+- VS Code extensions for Rust and Python
+- Persistent build cache in `.build/` directory enables fast incremental builds (only changed files are recompiled)
+
+`cargo build --locked --profile dev` to re-build
+
+- Edits to files are propogated to local repo due to the volume mount
+- SSH and GPG agent passthrough orchestrated by devcontainer
+
+File Structure:
+- Local dynamo repo mounts to `/home/ubuntu/dynamo`
+- Python venv in `/opt/dynamo/venv`
+- Build artifacts in `dynamo/.build/target`
+- HuggingFace cache preserved between sessions (Mounting local path `HF_HOME` at `/home/ubuntu/.cache/huggingface`)
+- Bash memory preserved between sessions at `/home/ubuntu/.commandhistory` using docker volume `dynamo-bashhistory`
+- Precommit peeserved between sessions at `/home/ubuntu/.cache/precommit` using docker volume `dynamo-precommit-cache`
+
+## Customization
+Edit `.devcontainer/devcontainer.json` to modify:
+- VS Code settings and extensions
+- Environment variables
+- Container configuration
+- Custom Mounts
+
+## FAQ
+
+### GPG Keys for Signing Git Commits
+Signing commits using GPG should work out of the box according to [VSCode docs](https://code.visualstudio.com/remote/advancedcontainers/sharing-git-credentials#_sharing-gpg-keys).
+
+If you run into version compatibility issues you can try:
+
+```bash
+# On Host
+gpg --list-secret-keys
+gpg --export-secret-keys --armor YOUR_KEY_ID > /tmp/key.asc
+
+# In container
+gpg1 --import /tmp/key.asc
+git config --local gpg.program gpg1
+```
+
+> Warning: Switching local gpg to gpg1 can have ramifications when you are not in the container any longer.
+
+### SSH Keys for Git Operations
+
+SSH keys need to be loaded in your SSH agent to work properly in the container. Can check out [VSCode docs](https://code.visualstudio.com/remote/advancedcontainers/sharing-git-credentials) for more details.
+
+```bash
+# In devcontainer, Check if your keys are loaded in the agent
+ssh-add -l
+
+# On local host, if your key isn't listed, add it
+eval "$(ssh-agent)"  # Start the agent if not running
+ssh-add ~/.ssh/id_rsa
+```
+
+Verify access by running `ssh -T git@github.com` in both host and container.
+
+### Environment Variables Not Set in Container?
+
+If your environment variables are not being set in your devcontainer (e.g., `echo $HF_TOKEN` returns empty), and these variables are defined in your `~/.bashrc`, there are two ways to ensure they are properly sourced:
+
+1. Add `source ~/.bashrc` to your `~/.bash_profile`, OR
+2. Add `source ~/.bashrc` to your `~/.profile` AND ensure `~/.bash_profile` does not exist
+
+Note: If both `~/.bash_profile` and `~/.profile` exist, bash will only read `~/.bash_profile` for login shells. Therefore, if you choose option 2, you must remove or rename `~/.bash_profile` to ensure `~/.profile` (and consequently `~/.bashrc`) is sourced.
+
+
+See VS Code Dev Containers [documentation](https://code.visualstudio.com/docs/devcontainers/containers) for more details.

.devcontainer/devcontainer.json

@@ -14,33 +14,64 @@
         "limitations under the License."
     ],
     "name": "NVIDIA Dynamo Development",
-    "build": {
-        "dockerfile": "Dockerfile",
-        "context": ".."
-    },
+    "remoteUser": "ubuntu", // Matches our container user
+    "updateRemoteUserUID": true, // Updates the UID of the remote user to match the host user, avoids permission errors
+    "image": "dynamo:latest-vllm-local-dev", // Use the latest VLLM local dev image
     "runArgs": [
-        "--gpus=all"
+        "--gpus=all",
+        "--network=host",
+        "--ipc=host",
+        "--cap-add=SYS_PTRACE",
+        "--shm-size=10G",
+        "--ulimit=memlock=-1",
+        "--ulimit=stack=67108864"
     ],
+    "service": "dynamo",
     "customizations": {
         "vscode": {
             "extensions": [
                 "ms-python.python",
-                "rust-lang.rust-analyzer",
-                "tamasfe.even-better-toml",
-                "github.copilot",
-                "ms-azuretools.vscode-docker"
+                "ms-python.pylint",
+                "ms-python.vscode-pylance",
+                "rust-lang.rust-analyzer"
             ],
             "settings": {
-                "python.defaultInterpreterPath": "/workspaces/ai-dynamo/venv/bin/python",
+                "terminal.integrated.defaultProfile.linux": "bash",
+                "terminal.integrated.cwd": "/home/ubuntu/dynamo",
+
+                "python.defaultInterpreterPath": "/opt/dynamo/venv/bin/python",
                 "python.linting.enabled": true,
                 "python.linting.pylintEnabled": true,
-                "editor.formatOnSave": true,
-                "rust-analyzer.checkOnSave.command": "clippy"
+
+                "rust-analyzer.checkOnSave.command": "clippy",
+                "rust-analyzer.checkOnSave.enable": true,
+                "rust-analyzer.semanticHighlighting.enable": true,
+
+                "editor.semanticHighlighting.enabled": true,
+
+                "files.trimTrailingWhitespace": true,
+                "files.insertFinalNewline": true
             }
         }
     },
-    "updateRemoteUserUID": true,
-    "postCreateCommand": "/bin/bash .devcontainer/post-create.sh",
-    "postStartCommand": "echo 'source /workspaces/ai-dynamo/venv/bin/activate' >> ~/.bashrc",
-    "remoteUser": "vscode"
+    "workspaceFolder": "/home/ubuntu",
+    "workspaceMount": "source=${localWorkspaceFolder},target=/home/ubuntu/dynamo,type=bind,consistency=cached",
+    "userEnvProbe": "interactiveShell",
+    "postCreateCommand": "/bin/bash /home/ubuntu/dynamo/.devcontainer/post-create.sh", // Runs cargo build and pip installs packages
+    "containerEnv": {
+        "GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
+        "HF_TOKEN": "${localEnv:HF_TOKEN}",
+        "CARGO_HOME": "/home/ubuntu/dynamo/.build/.cargo",
+        "RUSTUP_HOME": "/home/ubuntu/dynamo/.build/.rustup"
+    },
+    "mounts": [
+        // Uncomment for additional functionality
+        "source=${localEnv:HF_HOME},target=/home/ubuntu/.cache/huggingface,type=bind", // Uncomment to enable HF Cache Mount. Make sure to set HF_HOME env var in you .bashrc
+        "source=${localEnv:HOME}/.ssh,target=/home/ubuntu/.ssh,type=bind", // Uncomment to enable SSH Mount, note the .gitconfig should already by copied in
+
+        // Default mounts
+        "source=/tmp/,target=/tmp/,type=bind",
+        "source=dynamo-bashhistory,target=/home/ubuntu/.commandhistory,type=volume", // For bash history
+        "source=dynamo-precommit-cache,target=/home/ubuntu/.cache/pre-commit,type=volume"  // For pre-commit cache
+    ]
 }

.devcontainer/post-create.sh

@@ -15,19 +15,54 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-set -e
+trap 'echo "❌ ERROR: Command failed at line $LINENO: $BASH_COMMAND"; echo "⚠️ This was unexpected and setup was not completed. Can try to resolve yourself and then manually run the rest of the commands in this file or file a bug."' ERR
 
-# Create and activate Python virtual environment
-python3 -m venv venv
-. ./venv/bin/activate
+retry() {
+    # retries for connectivity issues in installs
+    local retries=3
+    local count=0
+    until "$@"; do
+        exit_code=$?
+        wait_time=$((2 ** count))
+        echo "Command failed with exit code $exit_code. Retrying in $wait_time seconds..."
+        sleep $wait_time
+        count=$((count + 1))
+        if [ $count -ge $retries ]; then
+            echo "Command failed after $retries attempts."
+            return $exit_code
+        fi
+    done
+    return 0
+}
 
-# Build Rust components first
-cargo build --release
+set -xe
 
-# Install Dynamo with all dependencies
-pip install -e .[all]
+# Changing permission to match local user since volume mounts default to root ownership
+sudo chown -R ubuntu:ubuntu ~/.cache/pre-commit
 
-# Install development tools
-pip install pytest isort mypy pylint pre-commit
+# Pre-commit hooks
+cd $HOME/dynamo && pre-commit install && retry pre-commit install-hooks
+pre-commit run --all-files || true # don't fail the build if pre-commit hooks fail
 
-echo "Development environment setup complete!"
\ No newline at end of file
+# Set build directory
+mkdir -p $HOME/dynamo/.build/target
+export CARGO_TARGET_DIR=$HOME/dynamo/.build/target
+
+# build project, it will be saved at $HOME/dynamo/.build/target
+cargo build --locked --profile dev --features mistralrs,sglang,vllm,python
+cargo doc --no-deps
+
+# create symlinks for the binaries in the deploy directory
+mkdir -p $HOME/dynamo/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin
+ln -sf $HOME/dynamo/.build/target/debug/dynamo-run $HOME/dynamo/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin/dynamo-run
+ln -sf $HOME/dynamo/.build/target/debug/http $HOME/dynamo/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin/http
+ln -sf $HOME/dynamo/.build/target/debug/llmctl $HOME/dynamo/deploy/dynamo/sdk/src/dynamo/sdk/cli/bin/llmctl
+
+# install the python bindings in editable mode
+cd $HOME/dynamo/lib/bindings/python && retry uv pip install -e .
+cd $HOME/dynamo && retry env DYNAMO_BIN_PATH=$HOME/dynamo/.build/target/debug uv pip install -e .
+
+# source the venv and set the VLLM_KV_CAPI_PATH in bashrc
+echo "source /opt/dynamo/venv/bin/activate" >> ~/.bashrc
+echo "export VLLM_KV_CAPI_PATH=$HOME/dynamo/.build/target/debug/libdynamo_llm_capi.so" >> ~/.bashrc
+echo "export GPG_TTY=$(tty)" >> ~/.bashrc

.gitignore

@@ -85,3 +85,8 @@ generated-values.yaml
 .build/
 **/.devcontainer/.env
 TensorRT-LLM
+
+# Local build artifacts for devcontainer
+.build/
+# Copied binaries to ignore
+deploy/dynamo/sdk/src/dynamo/sdk/cli/bin
\ No newline at end of file

README.md

@@ -133,9 +133,9 @@ curl localhost:8000/v1/chat/completions   -H "Content-Type: application/json"
 
 ### Local Development
 
-#### Container
+If you use vscode or cursor, we have a .devcontainer folder built on [Microsofts Extension](https://code.visualstudio.com/docs/devcontainers/containers). For instructions see the [ReadMe](.devcontainer/README.md) for more details.
 
-To develop locally, we recommend working inside of the container
+Otherwise, to develop locally, we recommend working inside of the container
 
 ```bash
 ./container/build.sh
@@ -150,19 +150,6 @@ cp /workspace/target/release/dynamo-run /workspace/deploy/dynamo/sdk/src/dynamo/
 uv pip install -e .
 ```
 
-#### Devcontainer Environment
-
-For a consistent development environment, you can use the provided devcontainer configuration. This requires:
-- [Docker](https://www.docker.com/products/docker-desktop)
-- [VS Code](https://code.visualstudio.com/) with the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
-
-To use the devcontainer:
-1. Open the project in VS Code
-2. Click on the button in the bottom-left corner
-3. Select "Reopen in Container"
-
-This will build and start a container with all the necessary dependencies for Dynamo development.
-
 
 #### Conda Environment
 

container/Dockerfile.tensorrt_llm

@@ -95,6 +95,7 @@ COPY LICENSE /workspace/
 COPY Cargo.toml /workspace/
 COPY Cargo.lock /workspace/
 COPY rust-toolchain.toml /workspace/
+COPY hatch_build.py /workspace/
 
 ARG CARGO_BUILD_JOBS
 # Set CARGO_BUILD_JOBS to 16 if not provided

container/Dockerfile.vllm

@@ -19,10 +19,10 @@ RUN echo "NIXL commit: ${NIXL_COMMIT}" > /opt/nixl/commit.txt
 COPY --from=nixl . .
 
 ##################################
-########## Build Image ###########
+########## Base Image ############
 ##################################
 
-FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG} AS build
+FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG} AS base
 
 USER root
 
@@ -159,11 +159,11 @@ RUN ls /opt/nixl
 # Install utilities
 RUN apt update -y && apt install -y git wget curl nvtop tmux vim
 # nats
-RUN wget https://github.com/nats-io/nats-server/releases/download/v2.10.24/nats-server-v2.10.24-amd64.deb && \
+RUN wget --tries=3 --waitretry=5 https://github.com/nats-io/nats-server/releases/download/v2.10.24/nats-server-v2.10.24-amd64.deb && \
     dpkg -i nats-server-v2.10.24-amd64.deb && rm nats-server-v2.10.24-amd64.deb
 # etcd
 ENV ETCD_VERSION="v3.5.18"
-RUN wget https://github.com/etcd-io/etcd/releases/download/$ETCD_VERSION/etcd-$ETCD_VERSION-linux-amd64.tar.gz -O /tmp/etcd.tar.gz && \
+RUN wget --tries=3 --waitretry=5 https://github.com/etcd-io/etcd/releases/download/$ETCD_VERSION/etcd-$ETCD_VERSION-linux-amd64.tar.gz -O /tmp/etcd.tar.gz && \
     mkdir -p /usr/local/bin/etcd && \
     tar -xvf /tmp/etcd.tar.gz -C /usr/local/bin/etcd --strip-components=1 && \
     rm /tmp/etcd.tar.gz
@@ -249,6 +249,56 @@ RUN wget --tries=3 --waitretry=5 "https://static.rust-lang.org/rustup/archive/1.
     rm rustup-init && \
     chmod -R a+w $RUSTUP_HOME $CARGO_HOME
 
+#######################################
+########## Local Development ##########
+#######################################
+
+FROM base AS local-dev
+
+# https://code.visualstudio.com/remote/advancedcontainers/add-nonroot-user
+# Will use the default ubuntu user, but give sudo access
+# Needed so files permissions aren't set to root ownership when writing from inside container
+
+# Don't want ubuntu to be editable, just change uid and gid. User ubuntu is hardcoded in .devcontainer
+ENV USERNAME=ubuntu
+ARG USER_UID=1000
+ARG USER_GID=1000
+
+RUN apt-get update && apt-get install -y sudo gnupg2 gnupg1 \
+    && echo "$USERNAME ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/$USERNAME \
+    && chmod 0440 /etc/sudoers.d/$USERNAME \
+    && mkdir -p /home/$USERNAME \
+    && chown -R $USERNAME:$USERNAME /home/$USERNAME \
+    && rm -rf /var/lib/apt/lists/* \
+    && chsh -s /bin/bash $USERNAME
+
+# This is a slow operation (~40s on my cpu)
+# Much better than chown -R $USERNAME:$USERNAME /opt/dynamo/venv (~10min on my cpu)
+COPY --from=base --chown=$USER_UID:$USER_GID /opt/dynamo/venv/ /opt/dynamo/venv/
+COPY --from=base --chown=$USERNAME:$USERNAME /usr/local/bin /usr/local/bin
+
+USER $USERNAME
+ENV HOME=/home/$USERNAME
+WORKDIR $HOME
+
+# https://code.visualstudio.com/remote/advancedcontainers/persist-bash-history
+RUN SNIPPET="export PROMPT_COMMAND='history -a' && export HISTFILE=$HOME/.commandhistory/.bash_history" \
+    && mkdir -p $HOME/.commandhistory \
+    && touch $HOME/.commandhistory/.bash_history \
+    && echo "$SNIPPET" >> "$HOME/.bashrc"
+
+RUN mkdir -p /home/$USERNAME/.cache/
+
+ENV VLLM_KV_CAPI_PATH=$HOME/dynamo/target/release/libdynamo_llm_capi.so
+
+ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]
+
+##################################
+########## Build Image ###########
+##################################
+
+FROM base AS build
+
 # Working directory
 WORKDIR /workspace
 
@@ -259,6 +309,7 @@ COPY LICENSE /workspace/
 COPY Cargo.toml /workspace/
 COPY Cargo.lock /workspace/
 COPY rust-toolchain.toml /workspace/
+COPY hatch_build.py /workspace/
 
 COPY lib/ /workspace/lib/
 COPY components /workspace/components

container/build.sh

@@ -323,6 +323,10 @@ if [[ $FRAMEWORK == "VLLM" ]]; then
     BUILD_ARGS+=" --build-arg NIXL_COMMIT=${NIXL_COMMIT} "
 fi
 
+if [[ $TARGET == "local-dev" ]]; then
+    BUILD_ARGS+=" --build-arg USER_UID=$(id -u) --build-arg USER_GID=$(id -g) "
+fi
+
 # BUILD DEV IMAGE
 
 BUILD_ARGS+=" --build-arg BASE_IMAGE=$BASE_IMAGE --build-arg BASE_IMAGE_TAG=$BASE_IMAGE_TAG --build-arg FRAMEWORK=$FRAMEWORK --build-arg ${FRAMEWORK}_FRAMEWORK=1 --build-arg VERSION=$VERSION --build-arg PYTHON_PACKAGE_VERSION=$PYTHON_PACKAGE_VERSION"

.devcontainer/Dockerfile → hatch_build.py

-# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -13,54 +13,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:24.04
+import os
 
-# Avoid prompts during package installation
-ENV DEBIAN_FRONTEND=noninteractive
+from hatchling.builders.hooks.plugin.interface import BuildHookInterface
 
-# Install dependencies
-RUN apt-get update && apt-get install -y \
-    build-essential \
-    curl \
-    git \
-    python3-dev \
-    python3-pip \
-    python3-venv \
-    libucx0 \
-    pkg-config \
-    libssl-dev \
-    jq \
-    wget \
-    sudo \
-    protobuf-compiler \
-    bash \
-    && rm -rf /var/lib/apt/lists/*
 
-# Create a non-root user
-ARG USERNAME=vscode
-ARG USER_UID=2008
-ARG USER_GID=$USER_UID
-
-RUN groupadd --gid $USER_GID $USERNAME \
-    && useradd --uid $USER_UID --gid $USER_GID -m $USERNAME \
-    && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME \
-    && chmod 0440 /etc/sudoers.d/$USERNAME
-
-# Set bash as default shell for the user
-RUN chsh -s /bin/bash $USERNAME
-
-# Set up Docker CLI
-RUN curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
-RUN usermod -aG docker $USERNAME
-
-# Set the working directory
-WORKDIR /workspaces/dynamo
-
-# Switch to non-root user
-USER $USERNAME
-
-# Install Rust for the vscode user
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain stable --profile default -y \
-    && . $HOME/.cargo/env
-
-ENV PATH="/home/$USERNAME/.cargo/bin:${PATH}"
\ No newline at end of file
+class CustomBuildHook(BuildHookInterface):
+    def initialize(self, version, build_data):
+        if self.target_name == "wheel":
+            bin_path = os.getenv("DYNAMO_BIN_PATH", "target/release")
+            build_data["force_include"] = {
+                f"{bin_path}/dynamo-run": "dynamo/sdk/cli/bin/dynamo-run",
+                f"{bin_path}/llmctl": "dynamo/sdk/cli/bin/llmctl",
+                f"{bin_path}/http": "dynamo/sdk/cli/bin/http",
+                f"{bin_path}/metrics": "dynamo/sdk/cli/bin/metrics",
+                f"{bin_path}/mock_worker": "dynamo/sdk/cli/bin/mock_worker",
+            }

pyproject.toml

@@ -78,12 +78,9 @@ packages = ["deploy/dynamo/sdk/src/dynamo"]
 
 # This section is for including the binaries in the wheel package
 # but doesn't make them executable scripts in the venv bin directory
-[tool.hatch.build.targets.wheel.force-include]
-"target/release/dynamo-run" = "dynamo/sdk/cli/bin/dynamo-run"
-"target/release/llmctl" = "dynamo/sdk/cli/bin/llmctl"
-"target/release/http" = "dynamo/sdk/cli/bin/http"
-"target/release/metrics" = "dynamo/sdk/cli/bin/metrics"
-"target/release/mock_worker" = "dynamo/sdk/cli/bin/mock_worker"
+
+[tool.hatch.build.hooks.custom]
+path = "hatch_build.py"
 
 [tool.codespell]
 # note: pre-commit passes explicit lists of files here, which this skip file list doesn't override -