Skip to content

GitLab

Unverified Commit d2635a7e authored by mohammedabdulwahhab's avatar mohammedabdulwahhab Committed by GitHub
Browse files

docs: Fixes to dynamo deploy docs (#902) 


Signed-off-by: default avatarmohammedabdulwahhab <furkhan324@berkeley.edu>
Co-authored-by: default avatarhhzhang16 <54051230+hhzhang16@users.noreply.github.com>
parent a82f350a
Show whitespace changes
Inline Side-by-side
Showing with 72 additions and 123 deletions
docs/guides/dynamo_deploy/operator_deployment.md
View file @ d2635a7e
...@@ -6,15 +6,14 @@ This guide walks you through deploying an inference graph created with the Dynam ...@@ -6,15 +6,14 @@ This guide walks you through deploying an inference graph created with the Dynam
Before proceeding with deployment, ensure you have: Before proceeding with deployment, ensure you have:
- [Dynamo CLI](../README.md#installation) installed - [Dynamo Python package](../README.md#installation) installed
- A Kubernetes cluster with the [Dynamo cloud platform](dynamo_cloud.md) installed - A Kubernetes cluster with the [Dynamo cloud platform](dynamo_cloud.md) installed
- Ubuntu 24.04 as the base image for your services - Ubuntu 24.04 as the base image for your services
- Required dependencies: - Required dependencies:
- Helm package manager - Helm package manager
- Dynamo SDK and CLI tools
- Rust packages and toolchain - Rust packages and toolchain
You must have first followed the instructions in [deploy/dynamo/helm/README.md](../../deploy/dynamo/helm/README.md) to install Dynamo Cloud on your Kubernetes cluster. You must have first followed the instructions in [deploy/dynamo/helm/README.md](../../../deploy/dynamo/helm/README.md) to install Dynamo Cloud on your Kubernetes cluster.
**Note**: Note the `KUBE_NS` variable in the following steps must match the Kubernetes namespace where you installed Dynamo Cloud. You must also expose the `dynamo-store` service externally. This will be the endpoint the CLI uses to interface with Dynamo Cloud. **Note**: Note the `KUBE_NS` variable in the following steps must match the Kubernetes namespace where you installed Dynamo Cloud. You must also expose the `dynamo-store` service externally. This will be the endpoint the CLI uses to interface with Dynamo Cloud.
## Understanding the Deployment Process ## Understanding the Deployment Process
...@@ -39,9 +38,12 @@ The deployment process involves two main steps: ...@@ -39,9 +38,12 @@ The deployment process involves two main steps:
## Deployment Steps ## Deployment Steps
### 1. Login to Dynamo Cloud ### 1. Configure Environment Variables
First, configure your environment and login to the Dynamo cloud: First, set up your environment variables for working with Dynamo Cloud. You have two options for accessing the `dynamo-store` service:
#### Option 1: Using Port-Forward (Local Development)
This is the simplest approach for local development and testing:
```bash ```bash
# Set your project root directory # Set your project root directory
...@@ -50,21 +52,41 @@ export PROJECT_ROOT=$(pwd) ...@@ -50,21 +52,41 @@ export PROJECT_ROOT=$(pwd)
# Set your Kubernetes namespace (must match the namespace where Dynamo cloud is installed) # Set your Kubernetes namespace (must match the namespace where Dynamo cloud is installed)
export KUBE_NS=hello-world export KUBE_NS=hello-world
# Set the externally accessible endpoint to the `dynamo-store` service within your Dynamo Cloud installation # In a separate terminal, run port-forward to expose the dynamo-store service locally
export DYNAMO_CLOUD=https://${KUBE_NS}.dev.aire.nvidia.com kubectl port-forward svc/dynamo-store 8080:80 -n $KUBE_NS
# Set DYNAMO_CLOUD to use the local port-forward endpoint
export DYNAMO_CLOUD=http://localhost:8080
``` ```
The `DYNAMO_CLOUD` environment variable is required for all Dynamo deployment commands. Make sure it's set before running any deployment operations. #### Option 2: Using Ingress/VirtualService (Production)
For production environments, you should use proper ingress configuration:
### 2. Build the Dynamo Base Image ```bash
# Set your project root directory
export PROJECT_ROOT=$(pwd)
# Set your Kubernetes namespace (must match the namespace where Dynamo cloud is installed)
export KUBE_NS=hello-world
# Set DYNAMO_CLOUD to your externally accessible endpoint
# This could be your Ingress hostname or VirtualService URL
export DYNAMO_CLOUD=https://dynamo-cloud.nvidia.com # Replace with your actual endpoint
```
> [!NOTE] > [!NOTE]
> For instructions on building and pushing the Dynamo base image, see the [Building the Dynamo Base Image](../../README.md#building-the-dynamo-base-image) section in the main README. > The `DYNAMO_CLOUD` environment variable is required for all Dynamo deployment commands. Make sure it's set before running any deployment operations.
> The Dynamo base image must be pushed to the same registry as the pipelines images configured in the [Dynamo Cloud](dynamo_cloud.md) deployment.
### 2. Build the Dynamo Base Image
Before building your service, you need to ensure the base image is properly set up:
1. For detailed instructions on building and pushing the Dynamo base image, see the [Building the Dynamo Base Image](../../../README.md#building-the-dynamo-base-image) section in the main README.
2. Export the image from the previous step to your environment.
```bash ```bash
# Set your runtime image name # Export the image from the previous step to your environment
export DYNAMO_IMAGE=<dynamo_docker_image_name> export DYNAMO_IMAGE=<your-registry>/<your-image-name>:<your-tag>
# Navigate to your project directory # Navigate to your project directory
cd $PROJECT_ROOT/examples/hello_world cd $PROJECT_ROOT/examples/hello_world
......
examples/hello_world/README.md
View file @ d2635a7e
...@@ -89,103 +89,49 @@ curl -X 'POST' \ ...@@ -89,103 +89,49 @@ curl -X 'POST' \
## Deploying to and Running the Example in Kubernetes ## Deploying to and Running the Example in Kubernetes
There are two ways to deploy the hello world example: This example can be deployed to a Kubernetes cluster using [Dynamo Cloud](../../docs/guides/dynamo_deploy/dynamo_cloud.md) and the Dynamo CLI.
1. Manually using helm charts
2. Using the Dynamo cloud Kubernetes platform and the Dynamo deploy CLI.
#### Deploying with helm charts ### Prerequisites
The instructions for deploying the hello world example using helm charts can be found at [Deploying Dynamo Inference Graphs to Kubernetes using Helm](../../docs/guides/dynamo_deploy.md). The guide covers:
1. Setting up a local Kubernetes cluster with MicroK8s
2. Installing required dependencies like NATS and etcd
3. Building and containerizing the pipeline
4. Deploying using Helm charts
5. Testing the deployment
#### Deploying with the Dynamo cloud platform
This example can be deployed to a Kubernetes cluster using Dynamo cloud and the Dynamo deploy CLI.
##### Prerequisites
Before deploying, ensure you have:
- Dynamo CLI installed
- Ubuntu 24.04 as the base image
- Required dependencies:
- Helm package manager
- Dynamo SDK and CLI tools
- Rust packages and toolchain
You must have first followed the instructions in [deploy/dynamo/helm/README.md](../../deploy/dynamo/helm/README.md) to create your Dynamo cloud deployment. You must have first followed the instructions in [deploy/dynamo/helm/README.md](../../deploy/dynamo/helm/README.md) to create your Dynamo cloud deployment.
##### Understanding the Build and Deployment Process ### Deployment Steps
The deployment process involves two distinct build steps: For detailed deployment instructions, please refer to the [Operator Deployment Guide](../../docs/guides/dynamo_deploy/operator_deployment.md). The following are the specific commands for the hello world example:
1. **Local `dynamo build`**: This step creates a Dynamo service archive that contains:
- Your service code and dependencies
- Service configuration and metadata
- Runtime requirements
- The service graph definition
2. **Remote Image Build**: When you create a deployment, a `yatai-dynamonim-image-builder` pod is created in your cluster. This pod:
- Takes the Dynamo service archive created in step 1
- Containerizes it using the specified base image
- Pushes the final container image to your cluster's registry
##### Deployment Steps
1. **Login to Dynamo Cloud**
```bash ```bash
# Set your project root directory
export PROJECT_ROOT=$(pwd) export PROJECT_ROOT=$(pwd)
export KUBE_NS=hello-world # Must match your Kubernetes namespace
export DYNAMO_CLOUD=https://${KUBE_NS}.dev.aire.nvidia.com
```
The `DYNAMO_CLOUD` environment variable is required for all Dynamo deployment commands. Make sure it's set before running any deployment operations. # Configure environment variables (see operator_deployment.md for details)
export KUBE_NS=hello-world
export DYNAMO_CLOUD=http://localhost:8080 # If using port-forward
# OR
# export DYNAMO_CLOUD=https://dynamo-cloud.nvidia.com # If using Ingress/VirtualService
2. **Build the Dynamo Base Image** # Build the Dynamo base image (see operator_deployment.md for details)
export DYNAMO_IMAGE=<your-registry>/<your-image-name>:<your-tag>
> [!NOTE] # Build the service
> For instructions on building and pushing the Dynamo base image, see the [Building the Dynamo Base Image](../../README.md#building-the-dynamo-base-image) section in the main README.
```bash
# Set runtime image name
export DYNAMO_IMAGE=<dynamo_docker_image_name>
# Prepare your project for deployment.
cd $PROJECT_ROOT/examples/hello_world cd $PROJECT_ROOT/examples/hello_world
DYNAMO_TAG=$(dynamo build hello_world:Frontend | grep "Successfully built" | awk '{ print $3 }' | sed 's/\.$//') DYNAMO_TAG=$(dynamo build hello_world:Frontend | grep "Successfully built" | awk '{ print $3 }' | sed 's/\.$//')
```
3. **Deploy to Kubernetes**
```bash # Deploy to Kubernetes
echo $DYNAMO_TAG
export DEPLOYMENT_NAME=ci-hw export DEPLOYMENT_NAME=ci-hw
dynamo deployment create $DYNAMO_TAG -n $DEPLOYMENT_NAME dynamo deployment create $DYNAMO_TAG -n $DEPLOYMENT_NAME
``` ```
4. **Test the deployment** ### Testing the Deployment
Once you create the Dynamo deployment, a pod prefixed with `yatai-dynamonim-image-builder` will begin running. Once it finishes running, it will create the pods necessary. Once the pods prefixed with `$HELM_RELEASE` are up and running, you can test out your example!
Find your frontend pod using one of these methods: Once the deployment is complete, you can test it using:
```bash ```bash
# Method 1: List all pods and find the frontend pod manually # Find your frontend pod
kubectl get pods -n ${KUBE_NS} | grep frontend | cat
# Method 2: Use a label selector to find the frontend pod automatically
export FRONTEND_POD=$(kubectl get pods -n ${KUBE_NS} | grep "${DEPLOYMENT_NAME}-frontend" | sort -k1 | tail -n1 | awk '{print $1}') export FRONTEND_POD=$(kubectl get pods -n ${KUBE_NS} | grep "${DEPLOYMENT_NAME}-frontend" | sort -k1 | tail -n1 | awk '{print $1}')
# Forward the pod's port to localhost # Forward the pod's port to localhost
kubectl port-forward pod/$FRONTEND_POD 8000:8000 -n ${KUBE_NS} kubectl port-forward pod/$FRONTEND_POD 8000:8000 -n ${KUBE_NS}
# Note: We forward directly to the pod's port 8000 rather than the service port because the frontend component listens on port 8000 internally.
# Test the API endpoint # Test the API endpoint
curl -X 'POST' 'http://localhost:8000/generate' \ curl -X 'POST' 'http://localhost:8000/generate' \
-H 'accept: text/event-stream' \ -H 'accept: text/event-stream' \
...@@ -193,6 +139,8 @@ curl -X 'POST' 'http://localhost:8000/generate' \ ...@@ -193,6 +139,8 @@ curl -X 'POST' 'http://localhost:8000/generate' \
-d '{"text": "test"}' -d '{"text": "test"}'
``` ```
For more details on managing deployments, testing, and troubleshooting, please refer to the [Operator Deployment Guide](../../docs/guides/dynamo_deploy/operator_deployment.md).
## Expected Output ## Expected Output
When you send the request with "test" as input, the response will show how the text flows through each service: When you send the request with "test" as input, the response will show how the text flows through each service:
......
examples/llm/README.md
View file @ d2635a7e
...@@ -182,74 +182,51 @@ See [close deployment](../../docs/guides/dynamo_serve.md#close-deployment) secti ...@@ -182,74 +182,51 @@ See [close deployment](../../docs/guides/dynamo_serve.md#close-deployment) secti
## Deploy to Kubernetes ## Deploy to Kubernetes
These examples can be deployed to a Kubernetes cluster using Dynamo Cloud and the Dynamo deploy CLI. These examples can be deployed to a Kubernetes cluster using [Dynamo Cloud](../../docs/guides/dynamo_deploy/dynamo_cloud.md) and the Dynamo CLI.
### Prerequisites ### Prerequisites
Before deploying, ensure you have:
- Dynamo CLI installed
- Ubuntu 24.04 as the base image
- Required dependencies:
- Helm package manager
- Dynamo SDK and CLI tools
- Rust packages and toolchain
You must have first followed the instructions in [deploy/dynamo/helm/README.md](../../deploy/dynamo/helm/README.md) to install Dynamo Cloud on your Kubernetes cluster. You must have first followed the instructions in [deploy/dynamo/helm/README.md](../../deploy/dynamo/helm/README.md) to install Dynamo Cloud on your Kubernetes cluster.
**Note**: Note the `KUBE_NS` variable in the following steps must match the Kubernetes namespace where you installed Dynamo Cloud. You must also expose the `dynamo-store` service externally. This will be the endpoint the CLI uses to interface with Dynamo Cloud. **Note**: The `KUBE_NS` variable in the following steps must match the Kubernetes namespace where you installed Dynamo Cloud. You must also expose the `dynamo-store` service externally. This will be the endpoint the CLI uses to interface with Dynamo Cloud.
### Deployment Steps ### Deployment Steps
1. **Login to Dynamo Cloud** For detailed deployment instructions, please refer to the [Operator Deployment Guide](../../docs/guides/dynamo_deploy/operator_deployment.md). The following are the specific commands for the LLM examples:
```bash ```bash
# Set your project root directory
export PROJECT_ROOT=$(pwd) export PROJECT_ROOT=$(pwd)
export KUBE_NS=dynamo-cloud # Note: This must match the Kubernetes namespace where you installed Dynamo Cloud
export DYNAMO_CLOUD=https://${KUBE_NS}.dev.aire.nvidia.com # Externally accessible endpoint to the `dynamo-store` service within your Dynamo Cloud installation
```
The `DYNAMO_CLOUD` environment variable is required for all Dynamo deployment commands. Make sure it's set before running any deployment operations. # Configure environment variables (see operator_deployment.md for details)
export KUBE_NS=dynamo-cloud
export DYNAMO_CLOUD=http://localhost:8080 # If using port-forward
# OR
# export DYNAMO_CLOUD=https://dynamo-cloud.nvidia.com # If using Ingress/VirtualService
2. **Build the Dynamo Base Image** # Build the Dynamo base image (see operator_deployment.md for details)
export DYNAMO_IMAGE=<your-registry>/<your-image-name>:<your-tag>
> [!NOTE]
> For instructions on building and pushing the Dynamo base image, see the [Building the Dynamo Base Image](../../README.md#building-the-dynamo-base-image) section in the main README.
```bash # Build the service
# Set runtime image name
export DYNAMO_IMAGE=<dynamo_docker_image_name>
# Prepare your project for deployment.
cd $PROJECT_ROOT/examples/llm cd $PROJECT_ROOT/examples/llm
DYNAMO_TAG=$(dynamo build graphs.agg:Frontend | grep "Successfully built" | awk '{ print $NF }' | sed 's/\.$//') DYNAMO_TAG=$(dynamo build graphs.agg:Frontend | grep "Successfully built" | awk '{ print $NF }' | sed 's/\.$//')
```
3. **Deploy to Kubernetes** # Deploy to Kubernetes
```bash
echo $DYNAMO_TAG
export DEPLOYMENT_NAME=llm-agg export DEPLOYMENT_NAME=llm-agg
dynamo deployment create $DYNAMO_TAG -n $DEPLOYMENT_NAME -f ./configs/agg.yaml dynamo deployment create $DYNAMO_TAG -n $DEPLOYMENT_NAME -f ./configs/agg.yaml
``` ```
4. **Test the deployment** ### Testing the Deployment
Once you create the Dynamo deployment, a pod prefixed with `yatai-dynamonim-image-builder` will begin running. Once it finishes running, pods will be created using the image that was built. Once the pods prefixed with `$DEPLOYMENT_NAME` are up and running, you can test out your example!
Find your frontend pod using one of these methods: Once the deployment is complete, you can test it using:
```bash ```bash
# Method 1: List all pods and find the frontend pod manually # Find your frontend pod
kubectl get pods -n ${KUBE_NS} | grep frontend | cat
# Method 2: Use a label selector to find the frontend pod automatically
export FRONTEND_POD=$(kubectl get pods -n ${KUBE_NS} | grep "${DEPLOYMENT_NAME}-frontend" | sort -k1 | tail -n1 | awk '{print $1}') export FRONTEND_POD=$(kubectl get pods -n ${KUBE_NS} | grep "${DEPLOYMENT_NAME}-frontend" | sort -k1 | tail -n1 | awk '{print $1}')
# Forward the pod's port to localhost # Forward the pod's port to localhost
kubectl port-forward pod/$FRONTEND_POD 8000:8000 -n ${KUBE_NS} kubectl port-forward pod/$FRONTEND_POD 8000:8000 -n ${KUBE_NS}
# Note: We forward directly to the pod's port 8000 rather than the service port because the frontend component listens on port 8000 internally.
# Test the API endpoint # Test the API endpoint
curl localhost:8000/v1/chat/completions \ curl localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
...@@ -265,3 +242,5 @@ curl localhost:8000/v1/chat/completions \ ...@@ -265,3 +242,5 @@ curl localhost:8000/v1/chat/completions \
"max_tokens": 30 "max_tokens": 30
}' }'
``` ```
For more details on managing deployments, testing, and troubleshooting, please refer to the [Operator Deployment Guide](../../docs/guides/dynamo_deploy/operator_deployment.md).
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment