feat: improve devcontainer setup and documentation (#2578)

Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>

.devcontainer/README.md

@@ -26,18 +26,16 @@ Before you begin, ensure you have the following installed:
 - [Docker](https://docs.docker.com/get-started/get-docker/) installed and configured on your host system
 - IDEs: Use either the VS Code or Cursor. Both have Dev Containers extensions
 - Appropriate NVIDIA drivers (compatible with CUDA 12.8+)
-- For models that require authentication, set your Hugging Face token env var `HF_TOKEN` in your local startup (.bashrc, .zshrc or .profile file). Many public models do not require this token.
+- For models that require authentication, set your Hugging Face token env var `HF_TOKEN` in your local startup (.profile or .zprofile file). Many public models do not require this token.
 
 ### Required Files and Directories
 
-The following files and directories are required on your host system for the devcontainer to work properly:
+You must have the following path on your host.
 
-- **`.gitconfig`**: Must exist in your home directory (`~/.gitconfig`). This file is mounted into the container for Git configuration.
-- **`~/.cache/huggingface`**: This directory is mounted into the container for Hugging Face model caching. If it doesn't exist, it will be created automatically.
+- **`~/.cache/huggingface`**: This directory is mounted into the container for Hugging Face model caching.
 
-If these files/directories are missing, you may encounter Docker mount errors when starting the devcontainer.
 
-## Quick Start Guide
+## Steps to Get Started
 
 Follow these steps to get your NVIDIA Dynamo development environment up and running:
 
@@ -54,30 +52,42 @@ The container will be built and give certain file permissions to your local uid
 
 ### Step 2: Install Dev Containers Extension
 
-**For VS Code:**
-- Install [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) from Microsoft marketplace
-
 **For Cursor:**
 - Press `Cmd+Shift+X` (Mac) or `Ctrl+Shift+X` (Linux/Windows) to open Extensions
 - Search for "Dev Containers" and install the one by **Anysphere** (Do not download the version from Microsoft as it is not compatible with Cursor)
 
+**For VS Code:**
+- Install [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) from Microsoft marketplace
+
+
 ### Step 3: Launch the Development Environment
 
 1. Open `dynamo` folder in your IDE
 2. Press `Cmd+Shift+P` (Mac) or `Ctrl+Shift+P` (Linux/Windows)
 3. Select "Dev Containers: Open Folder in Container", select your `dynamo` folder and open.
 
-### Step 4: Optional - Custom Hugging Face Cache
+### Step 4: Optional But Highly Recommended Setup
 
-If you want to mount a different Hugging Face cache directory than the default ~/.cache/huggingface, you can do something like below:
+For the best development experience, we recommend configuring the following:
+
+#### Git Configuration
+- **Personal `.gitconfig`**: If not done already, make your own `~/.gitconfig` file with your name, email, and preferred Git settings
+- **Dev Container Integration**: Enable the `Copy the .gitconfig file to the devcontainer` setting by going to IDE Settings → Extensions → Dev Containers → **☑ Copy the .gitconfig file to the devcontainer**
+
+#### SSH Authentication
+To enable seamless Git operations (push/pull) via SSH:
+
+1. **Configure ssh-agent on your host**: Search online for "how to set up ssh-agent on Ubuntu on .profile" and ensure it's configured in your `~/.profile` so the ssh-agent runs before Cursor launches
+
+2. **Enable SSH forwarding in IDE**:
+   Dev Containers set up -> check **☑ Dev Containers: Enable SSHAgent Forwarding**
+
+3. **Verify SSH forwarding works**:
+   - The post-create.sh script will check and report SSH agent status
+   - You should see "SSH agent forwarding is working" when the container starts
+
+This setup allows you to use Git commands normally within the container without additional authentication steps.
 
-1. Go to `.devcontainer` folder
-2. Add this line in the mounts section:
-   ```json
-   "source=${localEnv:HF_HOME},target=/home/ubuntu/.cache/huggingface,type=bind",
-   // Mount from your custom HF_HOME to the container.
-   ```
-3. Make sure HF_HOME is sourced in your .bashrc or .zshenv and your IDE default terminal is set properly
 
 ### Step 5: Wait for Initialization
 
@@ -103,12 +113,12 @@ $ cargo metadata --format-version=1 | jq -r '.target_directory'
 /home/ubuntu/dynamo/.build/target  <-- this is the target path
 ```
 
-If cargo is not installed and configured property, you will see an error, such as the following:
+If cargo is not installed and configured properly, you will see one or more errors, such as the following:
 ```
 error: could not find `Cargo.toml` in $HOME or any parent directory
 ```
 
-Before pushing code to GitHub, remember to run `cargo fmt` and `cargo clippy`
+Lastly, before pushing code to GitHub, remember to run `cargo fmt` and `cargo clippy`
 
 ### Updating Python Bindings
 
@@ -175,12 +185,13 @@ git config --local gpg.program gpg1
 You can create a custom devcontainer configuration by copying the main configuration to another directory inside the `.devcontainer` directory. Below is an example where the custom name is `jensen_dev`, but feel free to name the directory whatever you want:
 
 ```bash
-# Copy the main devcontainer configuration and then edit the new json file
+# By convention, Dev Container will look at the project's .devcontainer/<path>/devcontainer.json file.
+# Example: copy the main devcontainer configuration and then edit the new json file
 mkdir -p .devcontainer/jensen_dev
 cp .devcontainer/devcontainer.json .devcontainer/jensen_dev/devcontainer.json
 ```
 
-Common customizations include additional mounts, environment variables, VS Code extensions, and build arguments. When you open a new Dev Container, you can pick from any of the `.devcontainer/*/devcontainer.json` files available.
+Common customizations include additional mounts, environment variables, IDE extensions, and build arguments. When you open a new Dev Container, you can pick from any of the `.devcontainer/<path>/devcontainer.json` files available.
 
 ### SSH Keys for Git Operations
 
@@ -191,18 +202,63 @@ If you have ssh-agent running on the host, then `git push` should just work. If
 
 ### Environment Variables Not Set in Container?
 
-If your environment variables are not being set in your devcontainer (e.g., `echo $HF_TOKEN` returns empty), and these variables are defined in your `~/.bashrc`, there are two ways to ensure they are properly sourced:
+Dev containers have limited access to host environment variables for security reasons. Here's how to properly pass environment variables:
+
+#### Method 1: Use devcontainer.json
+Add environment variables to `.devcontainer/devcontainer.json`:
+```json
+{
+  "remoteEnv": {
+    "HF_TOKEN": "${localEnv:HF_TOKEN}",
+    "GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
+    "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}"
+  }
+}
+```
+
+#### Method 2: Host Shell Configuration
+In order to ensure your host environment variables are available to the Dev Containers:
+
+**For bash users:**
+- Put variables in `~/.profile` (for login shells)
+- Ensure `~/.bash_profile` sources `~/.bashrc` if both exist
+- Some users report that variables in `~/.bashrc` aren't picked up by dev containers, so try moving them to `~/.profile` to see if that solves the problem
+
+**For zsh users:**
+- Put variables in `~/.zprofile` (for login shells)
+- Some users report that variables in `~/.zshrc` aren't picked up by dev containers, so try moving them to `~/.zprofile` to see if that solves the problem
 
-1. Add `source ~/.bashrc` to your `~/.bash_profile`, OR
-2. Add `source ~/.bashrc` to your `~/.profile` AND ensure `~/.bash_profile` does not exist
+#### Method 3: Environment File
+Create a `.env` file in your project root and reference it in devcontainer.json:
 
-Note: If both `~/.bash_profile` and `~/.profile` exist, bash will only read `~/.bash_profile` for login shells. Therefore, if you choose option 2, you must remove or rename `~/.bash_profile` to ensure `~/.profile` (and consequently `~/.bashrc`) is sourced.
+**Example .env file:**
+```bash
+# API Tokens
+HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+# Development settings
+DEBUG=true
+LOG_LEVEL=debug
+
+# Custom paths
+CUSTOM_MODEL_PATH=/path/to/models
+```
+
+**Reference in devcontainer.json:**
+```json
+{
+  "remoteEnv": {
+    "ENV_FILE": "${localWorkspaceFolder}/.env"
+  }
+}
+```
 
 See VS Code Dev Containers [documentation](https://code.visualstudio.com/docs/devcontainers/containers) for more details.
 
 ### Build Issues
 
-If you encounter build errors or strange compilation issues, try running `cargo clean` to clear the build cache and rebuild from scratch.
+If you encounter build errors or strange compilation issues, try running `cargo clean`, then rebuild from scratch.
 
 If `cargo clean` doesn't resolve the issue, it is possible that some of the files were created by root (using the `run.sh` script). You can manually remove the build target by going to your host (outside the container), and remove the target:
 
@@ -283,7 +339,7 @@ If you see errors like "container is not running" or "An error occurred setting
    docker ps -a --filter "label=devcontainer.local_folder=$(pwd)" -q | xargs docker rm -f
    docker images | grep "^vsc-" | awk '{print $3}' | xargs docker rmi
    ```
-  Then rebuild without cache. In VS Code or Cursor command:
+  Then rebuild without cache. In your IDE:
   *Dev Containers: Rebuild Without Cache and Reopen in Container*
 
 ### devcontainer.json Changes Not Being Picked Up
@@ -293,11 +349,11 @@ If you've made changes to `devcontainer.json`, `post-create.sh`, or other devcon
 **Solution: Force Devcontainer Rebuild**
 
 1. **Rebuild Container (Recommended):**
-   In VS Code or Cursor Command Palette (Ctrl+Shift+P):
+   In your IDE Command Palette (Ctrl+Shift+P):
    *Dev Containers: Rebuild Container*
 
 2. **If that doesn't work, rebuild without cache:**
-   In VS Code or Cursor Command Palette (Ctrl+Shift+P):
+   In your IDE Command Palette (Ctrl+Shift+P):
    *Dev Containers: Rebuild Without Cache and Reopen Container*
 
 3. **For persistent issues, manually remove the devcontainer image:**
@@ -305,10 +361,10 @@ If you've made changes to `devcontainer.json`, `post-create.sh`, or other devcon
    # List devcontainer images
    docker images | grep devcontainer
 
-   # And remove all VS Code devcontainer images (more thorough)
+   # And remove all IDE devcontainer images (more thorough)
    docker images | grep "^vsc-" | awk '{print $3}' | xargs docker rmi
 
-   # Then rebuild in VS Code
+   # Then rebuild in your IDE
    Dev Containers: Rebuild Container
    ```
 

.devcontainer/devcontainer.json

 {
-    "$schema": "https://json-schema.org/draft-07/schema#",
+    "$schema": "https://raw.githubusercontent.com/devcontainers/spec/main/schemas/devContainer.schema.json",
     "copyright": [
         "SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.",
-        "SPDX-License-Identifier: Apache-2.0",
-        "Licensed under the Apache License, Version 2.0 (the \"License\");",
-        "you may not use this file except in compliance with the License.",
-        "You may obtain a copy of the License at",
-        "http://www.apache.org/licenses/LICENSE-2.0",
-        "Unless required by applicable law or agreed to in writing, software",
-        "distributed under the License is distributed on an \"AS IS\" BASIS,",
-        "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.",
-        "See the License for the specific language governing permissions and",
-        "limitations under the License."
+        "SPDX-License-Identifier: Apache-2.0"
     ],
     "name": "NVIDIA Dynamo Dev Container Development",
     "remoteUser": "ubuntu", // Matches our container user
@@ -27,7 +18,6 @@
         "--ulimit=stack=67108864",
         "--ulimit=nofile=65536:65536"
     ],
-    "service": "dynamo",
     "customizations": {
         "vscode": {
             "extensions": [
@@ -74,13 +64,13 @@
         "DYNAMO_HOME": "/home/ubuntu/dynamo",
         "CARGO_HOME": "/home/ubuntu/dynamo/.build/.cargo",
         "RUSTUP_HOME": "/home/ubuntu/dynamo/.build/.rustup",
-        "CARGO_TARGET_DIR": "/home/ubuntu/dynamo/.build/target",
+        "CARGO_TARGET_DIR": "/home/ubuntu/dynamo/.build/target"
     },
     "remoteEnv": {
-        // These are optional, but are convenient to have, especially the SSH_AUTH_SOCK.
-        "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}",
+        // Optional convenience tokens passed from host. SSH agent is forwarded via your IDE setting, not here by default.
         "GITHUB_TOKEN": "${localEnv:GITHUB_TOKEN}",
         "HF_TOKEN": "${localEnv:HF_TOKEN}"
+        // "SSH_AUTH_SOCK": "${env:SSH_AUTH_SOCK}" // Optional: only if you also bind-mount the socket path
     },
     "mounts": [
         // These are for convenience, so that the history and pre-commit cache are persisted between sessions

.devcontainer/post-create.sh

@@ -99,12 +99,12 @@ $HOME/dynamo/deploy/dynamo_check.py --import-check-only
 { set +x; } 2>/dev/null
 
 # Check SSH agent forwarding status
-echo "🔍 Checking SSH agent forwarding status..."
 if [ -n "$SSH_AUTH_SOCK" ]; then
     if ssh-add -l > /dev/null 2>&1; then
-        echo "SSH agent forwarding is working - found $(ssh-add -l | wc -l) key(s)"
+        echo "SSH agent forwarding is working - found $(ssh-add -l | wc -l) key(s):"
+        ssh-add -l
     else
-        echo "⚠️  SSH_AUTH_SOCK is set but ssh-add failed - agent may not be accessible"
+        echo "⚠️ SSH_AUTH_SOCK is set but ssh-add failed - agent may not be accessible"
     fi
 else
     echo "⚠️ SSH agent forwarding not configured - SSH_AUTH_SOCK is not set"