Unverified Commit 3057af00 authored by Thomas Montfort's avatar Thomas Montfort Committed by GitHub
Browse files

feat(dep-715): refactor helm directory and remove reference to cloud (#5042)

parent b5922693
...@@ -97,7 +97,7 @@ Client-side benchmarking runs on your local machine and connects to Kubernetes d ...@@ -97,7 +97,7 @@ Client-side benchmarking runs on your local machine and connects to Kubernetes d
Follow these steps to benchmark Dynamo deployments using client-side benchmarking: Follow these steps to benchmark Dynamo deployments using client-side benchmarking:
### Step 1: Establish Kubernetes Cluster and Install Dynamo ### Step 1: Establish Kubernetes Cluster and Install Dynamo
Set up your Kubernetes cluster with NVIDIA GPUs and install the Dynamo Cloud platform. First follow the [installation guide](/docs/kubernetes/installation_guide.md) to install Dynamo Cloud, then use [deploy/utils/README](../../deploy/utils/README.md) to set up benchmarking resources. Set up your Kubernetes cluster with NVIDIA GPUs and install the Dynamo Kubernetes Platform. First follow the [installation guide](/docs/kubernetes/installation_guide.md) to install Dynamo Kubernetes Platform, then use [deploy/utils/README](../../deploy/utils/README.md) to set up benchmarking resources.
### Step 2: Deploy DynamoGraphDeployments ### Step 2: Deploy DynamoGraphDeployments
Deploy your DynamoGraphDeployments separately using the [deployment documentation](../../examples/backends/). Each deployment should have a frontend service exposed. Deploy your DynamoGraphDeployments separately using the [deployment documentation](../../examples/backends/). Each deployment should have a frontend service exposed.
...@@ -325,7 +325,7 @@ The server-side benchmarking solution: ...@@ -325,7 +325,7 @@ The server-side benchmarking solution:
## Prerequisites ## Prerequisites
1. **Kubernetes cluster** with NVIDIA GPUs and Dynamo namespace setup (see [Dynamo Cloud/Platform docs](/docs/kubernetes/README.md)) 1. **Kubernetes cluster** with NVIDIA GPUs and Dynamo namespace setup (see [Dynamo Kubernetes Platform docs](/docs/kubernetes/README.md))
2. **Storage** PersistentVolumeClaim configured with appropriate permissions (see [deploy/utils README](../../deploy/utils/README.md)) 2. **Storage** PersistentVolumeClaim configured with appropriate permissions (see [deploy/utils README](../../deploy/utils/README.md))
3. **Docker image** containing the Dynamo benchmarking tools 3. **Docker image** containing the Dynamo benchmarking tools
......
...@@ -23,7 +23,7 @@ High-level guide to Dynamo Kubernetes deployments. Start here, then dive into sp ...@@ -23,7 +23,7 @@ High-level guide to Dynamo Kubernetes deployments. Start here, then dive into sp
**Kubernetes Namespace**: The K8s namespace where your DynamoGraphDeployment resource is created. **Kubernetes Namespace**: The K8s namespace where your DynamoGraphDeployment resource is created.
- Used for: Resource isolation, RBAC, organizing deployments - Used for: Resource isolation, RBAC, organizing deployments
- Example: `dynamo-system`, `dynamo-cloud`, `team-a-namespace` - Example: `dynamo-system`, `team-a-namespace`
**Dynamo Namespace**: The logical namespace used by Dynamo components for service discovery via etcd. **Dynamo Namespace**: The logical namespace used by Dynamo components for service discovery via etcd.
- Used for: Runtime component communication, service discovery - Used for: Runtime component communication, service discovery
...@@ -34,7 +34,7 @@ These are independent. A single Kubernetes namespace can host multiple Dynamo na ...@@ -34,7 +34,7 @@ These are independent. A single Kubernetes namespace can host multiple Dynamo na
## Pre-deployment Checks ## Pre-deployment Checks
Before deploying the platform, it is recommended to run the pre-deployment checks to ensure the cluster is ready for deployment. Please refer to the [pre-deployment checks](../../deploy/cloud/pre-deployment/README.md) for more details. Before deploying the platform, it is recommended to run the pre-deployment checks to ensure the cluster is ready for deployment. Please refer to the [pre-deployment checks](../../deploy/pre-deployment/README.md) for more details.
## 1. Install Platform First ## 1. Install Platform First
......
...@@ -1083,17 +1083,17 @@ Default container ports are configured based on component type: ...@@ -1083,17 +1083,17 @@ Default container ports are configured based on component type:
For users who want to understand the implementation details or contribute to the operator, the default values described in this document are set in the following source files: For users who want to understand the implementation details or contribute to the operator, the default values described in this document are set in the following source files:
- **Health Probes, Security Context & Pod Specifications**: [`internal/dynamo/graph.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/dynamo/graph.go) - Contains the main logic for applying default probes, security context, environment variables, shared memory, and pod configurations - **Health Probes, Security Context & Pod Specifications**: [`internal/dynamo/graph.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/dynamo/graph.go) - Contains the main logic for applying default probes, security context, environment variables, shared memory, and pod configurations
- **Component-Specific Defaults**: - **Component-Specific Defaults**:
- [`internal/dynamo/component_frontend.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/dynamo/component_frontend.go) - [`internal/dynamo/component_frontend.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/dynamo/component_frontend.go)
- [`internal/dynamo/component_worker.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/dynamo/component_worker.go) - [`internal/dynamo/component_worker.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/dynamo/component_worker.go)
- [`internal/dynamo/component_planner.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/dynamo/component_planner.go) - [`internal/dynamo/component_planner.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/dynamo/component_planner.go)
- **Image Pull Secrets**: [`internal/secrets/docker.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/secrets/docker.go) - Implements the docker secret indexer and automatic discovery - **Image Pull Secrets**: [`internal/secrets/docker.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/secrets/docker.go) - Implements the docker secret indexer and automatic discovery
- **Backend-Specific Behavior**: - **Backend-Specific Behavior**:
- [`internal/dynamo/backend_vllm.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/dynamo/backend_vllm.go) - [`internal/dynamo/backend_vllm.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/dynamo/backend_vllm.go)
- [`internal/dynamo/backend_sglang.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/dynamo/backend_sglang.go) - [`internal/dynamo/backend_sglang.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/dynamo/backend_sglang.go)
- [`internal/dynamo/backend_trtllm.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/dynamo/backend_trtllm.go) - [`internal/dynamo/backend_trtllm.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/dynamo/backend_trtllm.go)
- **Constants & Annotations**: [`internal/consts/consts.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/cloud/operator/internal/consts/consts.go) - Defines annotation keys and other constants - **Constants & Annotations**: [`internal/consts/consts.go`](https://github.com/ai-dynamo/dynamo/blob/main/deploy/operator/internal/consts/consts.go) - Defines annotation keys and other constants
## Notes ## Notes
......
...@@ -155,13 +155,13 @@ export DOCKER_SERVER=your-registry.com/ # your container registry ...@@ -155,13 +155,13 @@ export DOCKER_SERVER=your-registry.com/ # your container registry
export IMAGE_TAG=latest export IMAGE_TAG=latest
# Build operator image # Build operator image
cd deploy/cloud/operator cd deploy/operator
docker build -t $DOCKER_SERVER/dynamo-operator:$IMAGE_TAG . docker build -t $DOCKER_SERVER/dynamo-operator:$IMAGE_TAG .
docker push $DOCKER_SERVER/dynamo-operator:$IMAGE_TAG docker push $DOCKER_SERVER/dynamo-operator:$IMAGE_TAG
cd - cd -
# Install CRDs # Install CRDs
cd deploy/cloud/helm cd deploy/helm/charts
helm install dynamo-crds ./crds/ --namespace default helm install dynamo-crds ./crds/ --namespace default
# Install platform with custom operator image # Install platform with custom operator image
......
...@@ -4,7 +4,7 @@ This section describes how to use FluxCD for GitOps-based deployment of Dynamo i ...@@ -4,7 +4,7 @@ This section describes how to use FluxCD for GitOps-based deployment of Dynamo i
## Prerequisites ## Prerequisites
- A Kubernetes cluster with [Dynamo Cloud](./installation_guide.md) installed - A Kubernetes cluster with [Dynamo Kubernetes Platform](./installation_guide.md) installed
- [FluxCD](https://fluxcd.io/flux/installation/) installed in your cluster - [FluxCD](https://fluxcd.io/flux/installation/) installed in your cluster
- A Git repository to store your deployment configurations - A Git repository to store your deployment configurations
...@@ -16,9 +16,9 @@ The GitOps workflow for Dynamo deployments consists of three main steps: ...@@ -16,9 +16,9 @@ The GitOps workflow for Dynamo deployments consists of three main steps:
2. Create and commit a DynamoGraphDeployment custom resource for initial deployment 2. Create and commit a DynamoGraphDeployment custom resource for initial deployment
3. Update the graph by building a new version and updating the CR for subsequent updates 3. Update the graph by building a new version and updating the CR for subsequent updates
## Step 1: Build and Push Dynamo Cloud Operator ## Step 1: Build and Push Dynamo Operator
First, follow to [See Install Dynamo Cloud](./installation_guide.md). First, follow to [See Install Dynamo Kubernetes Platform](./installation_guide.md).
## Step 2: Create Initial Deployment ## Step 2: Create Initial Deployment
...@@ -69,7 +69,7 @@ You can monitor the deployment status using: ...@@ -69,7 +69,7 @@ You can monitor the deployment status using:
```bash ```bash
export NAMESPACE=<namespace-with-the-dynamo-cloud-operator> export NAMESPACE=<namespace-with-the-dynamo-operator>
# Check the DynamoGraphDeployment status # Check the DynamoGraphDeployment status
kubectl get dynamographdeployment llm-agg -n $NAMESPACE kubectl get dynamographdeployment llm-agg -n $NAMESPACE
......
...@@ -97,4 +97,4 @@ For practical examples of Grove-based multinode deployments in action, see the [ ...@@ -97,4 +97,4 @@ For practical examples of Grove-based multinode deployments in action, see the [
For the latest updates on Grove, refer to the [official project on GitHub](https://github.com/NVIDIA/grove). For the latest updates on Grove, refer to the [official project on GitHub](https://github.com/NVIDIA/grove).
Dynamo Cloud also allows you to install Grove and KAI Scheduler as part of the platform installation. See the [Dynamo Cloud Deployment Installation Guide](./installation_guide.md) for more details. Dynamo Kubernetes Platform also allows you to install Grove and KAI Scheduler as part of the platform installation. See the [Dynamo Kubernetes Platform Deployment Installation Guide](./installation_guide.md) for more details.
\ No newline at end of file \ No newline at end of file
...@@ -55,7 +55,7 @@ kubectl get clusterrolebinding -o json | \ ...@@ -55,7 +55,7 @@ kubectl get clusterrolebinding -o json | \
## Installation Paths ## Installation Paths
Platform is installed using Dynamo Kubernetes Platform [helm chart](../../deploy/cloud/helm/platform/README.md). Platform is installed using Dynamo Kubernetes Platform [helm chart](../../deploy/helm/charts/platform/README.md).
**Path A: Pre-built Artifacts** **Path A: Pre-built Artifacts**
- Use case: Production deployment, shared or dedicated clusters - Use case: Production deployment, shared or dedicated clusters
...@@ -198,7 +198,7 @@ export DOCKER_PASSWORD=<YOUR_NGC_CLI_API_KEY> ...@@ -198,7 +198,7 @@ export DOCKER_PASSWORD=<YOUR_NGC_CLI_API_KEY>
export IMAGE_TAG=${RELEASE_VERSION} export IMAGE_TAG=${RELEASE_VERSION}
# 2. Build operator # 2. Build operator
cd deploy/cloud/operator cd deploy/operator
# 2.1 Alternative 1 : Build and push the operator image for multiple platforms # 2.1 Alternative 1 : Build and push the operator image for multiple platforms
docker buildx create --name multiplatform --driver docker-container --bootstrap docker buildx create --name multiplatform --driver docker-container --bootstrap
...@@ -218,7 +218,7 @@ kubectl create secret docker-registry docker-imagepullsecret \ ...@@ -218,7 +218,7 @@ kubectl create secret docker-registry docker-imagepullsecret \
--docker-password=${DOCKER_PASSWORD} \ --docker-password=${DOCKER_PASSWORD} \
--namespace=${NAMESPACE} --namespace=${NAMESPACE}
cd deploy/cloud/helm cd deploy/helm/charts
# 4. Install CRDs # 4. Install CRDs
helm upgrade --install dynamo-crds ./crds/ --namespace default helm upgrade --install dynamo-crds ./crds/ --namespace default
...@@ -327,13 +327,37 @@ just add the following to the helm install command: ...@@ -327,13 +327,37 @@ just add the following to the helm install command:
``` ```
**Clean uninstall?** **Clean uninstall?**
To uninstall the platform, you can run the following command:
```
helm uninstall dynamo-platform --namespace ${NAMESPACE}
```
To uninstall the CRDs, follow these steps:
Get all of the dynamo CRDs installed in your cluster:
```bash
kubectl get crd | grep "dynamo.*nvidia.com"
```
You should see something like this:
```
dynamocomponentdeployments.nvidia.com 2025-10-21T14:49:52Z
dynamocomponents.nvidia.com 2025-10-25T05:16:10Z
dynamographdeploymentrequests.nvidia.com 2025-11-24T05:26:04Z
dynamographdeployments.nvidia.com 2025-09-04T20:56:40Z
dynamographdeploymentscalingadapters.nvidia.com 2025-12-09T21:05:59Z
dynamomodels.nvidia.com 2025-11-07T00:19:43Z
```
Delete each CRD one by one:
```bash ```bash
./uninstall.sh # Removes all CRDs and platform kubectl delete crd <crd-name>
``` ```
## Advanced Options ## Advanced Options
- [Helm Chart Configuration](../../deploy/cloud/helm/platform/README.md) - [Helm Chart Configuration](../../deploy/helm/charts/platform/README.md)
- [Create custom deployments](./deployment/create_deployment.md) - [Create custom deployments](./deployment/create_deployment.md)
- [Dynamo Operator details](./dynamo_operator.md) - [Dynamo Operator details](./dynamo_operator.md)
- [Model Express Server details](https://github.com/ai-dynamo/modelexpress) - [Model Express Server details](https://github.com/ai-dynamo/modelexpress)
...@@ -15,9 +15,9 @@ This guide demonstrates how to set up logging for Dynamo in Kubernetes using Gra ...@@ -15,9 +15,9 @@ This guide demonstrates how to set up logging for Dynamo in Kubernetes using Gra
## Prerequisites ## Prerequisites
### 1. Dynamo Cloud Kubernetes Operator ### 1. Dynamo Kubernetes Platform
This guide assumes you have installed Dynamo Cloud Kubernetes Operator. For more information, see [Dynamo Cloud Operator](../README.md). This guide assumes you have installed Dynamo Kubernetes Platform. For more information, see [Dynamo Kubernetes Platform](../README.md).
### 2. Kube-prometheus ### 2. Kube-prometheus
...@@ -29,7 +29,7 @@ While this guide does not use Prometheus, it assumes Grafana is pre-installed wi ...@@ -29,7 +29,7 @@ While this guide does not use Prometheus, it assumes Grafana is pre-installed wi
The following env variables are set: The following env variables are set:
- `MONITORING_NAMESPACE`: The namespace where Loki is installed - `MONITORING_NAMESPACE`: The namespace where Loki is installed
- `DYN_NAMESPACE`: The namespace where Dynamo Cloud Operator is installed - `DYN_NAMESPACE`: The namespace where Dynamo Kubernetes Platform is installed
```bash ```bash
export MONITORING_NAMESPACE=monitoring export MONITORING_NAMESPACE=monitoring
......
...@@ -17,7 +17,7 @@ ...@@ -17,7 +17,7 @@
**Dynamo** - NVIDIA's high-performance distributed inference framework for Large Language Models (LLMs) and generative AI models, designed for multinode environments with disaggregated serving and cache-aware routing. **Dynamo** - NVIDIA's high-performance distributed inference framework for Large Language Models (LLMs) and generative AI models, designed for multinode environments with disaggregated serving and cache-aware routing.
**Dynamo Cloud** - A Kubernetes platform providing managed deployment experience for Dynamo inference graphs. **Dynamo Kubernetes Platform** - A Kubernetes platform providing managed deployment experience for Dynamo inference graphs.
## E ## E
**Endpoint** - A specific network-accessible API within a Dynamo component, such as `generate` or `load_metrics`. **Endpoint** - A specific network-accessible API within a Dynamo component, such as `generate` or `load_metrics`.
......
...@@ -74,7 +74,7 @@ extraPodSpec: ...@@ -74,7 +74,7 @@ extraPodSpec:
Before using these templates, ensure you have: Before using these templates, ensure you have:
1. **Dynamo Cloud Platform installed** - See [Installing Dynamo Cloud](../../../../docs/kubernetes/installation_guide.md) 1. **Dynamo Kubernetes Platform installed** - See [Installing Dynamo Kubernetes Platform](../../../../docs/kubernetes/installation_guide.md)
2. **Kubernetes cluster with GPU support** 2. **Kubernetes cluster with GPU support**
3. **Container registry access** for SGLang runtime images 3. **Container registry access** for SGLang runtime images
4. **HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`) 4. **HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`)
...@@ -146,7 +146,7 @@ All templates use **DeepSeek-R1-Distill-Llama-8B** as the default model. But you ...@@ -146,7 +146,7 @@ All templates use **DeepSeek-R1-Distill-Llama-8B** as the default model. But you
- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md) - **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md)
- **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md) - **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md)
- **Platform Setup**: [Dynamo Cloud Installation](../../../../docs/kubernetes/installation_guide.md) - **Platform Setup**: [Dynamo Kubernetes Platform Installation](../../../../docs/kubernetes/installation_guide.md)
- **Examples**: [Deployment Examples](../../../../docs/examples/README.md) - **Examples**: [Deployment Examples](../../../../docs/examples/README.md)
- **Kubernetes CRDs**: [Custom Resources Documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) - **Kubernetes CRDs**: [Custom Resources Documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
......
...@@ -102,7 +102,7 @@ extraPodSpec: ...@@ -102,7 +102,7 @@ extraPodSpec:
Before using these templates, ensure you have: Before using these templates, ensure you have:
1. **Dynamo Cloud Platform installed** - See [Quickstart Guide](../../../../docs/kubernetes/README.md) 1. **Dynamo Kubernetes Platform installed** - See [Quickstart Guide](../../../../docs/kubernetes/README.md)
2. **Kubernetes cluster with GPU support** 2. **Kubernetes cluster with GPU support**
3. **Container registry access** for TensorRT-LLM runtime images 3. **Container registry access** for TensorRT-LLM runtime images
4. **HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`) 4. **HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`)
...@@ -165,7 +165,7 @@ kubectl create secret generic hf-token-secret \ ...@@ -165,7 +165,7 @@ kubectl create secret generic hf-token-secret \
Then, deploy the model using the deployment file. Then, deploy the model using the deployment file.
Export the NAMESPACE you used in your Dynamo Cloud Installation. Export the NAMESPACE you used in your Dynamo Kubernetes Platform Installation.
```bash ```bash
cd dynamo/examples/backends/trtllm/deploy cd dynamo/examples/backends/trtllm/deploy
...@@ -264,7 +264,7 @@ Configure the `model` name and `host` based on your deployment. ...@@ -264,7 +264,7 @@ Configure the `model` name and `host` based on your deployment.
- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md) - **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md)
- **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md) - **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md)
- **Platform Setup**: [Dynamo Cloud Installation](../../../../docs/kubernetes/installation_guide.md) - **Platform Setup**: [Dynamo Kubernetes Platform Installation](../../../../docs/kubernetes/installation_guide.md)
- **Examples**: [Deployment Examples](../../../../docs/examples/README.md) - **Examples**: [Deployment Examples](../../../../docs/examples/README.md)
- **Architecture Docs**: [Disaggregated Serving](../../../../docs/design_docs/disagg_serving.md), [KV-Aware Routing](../../../../docs/router/kv_cache_routing.md) - **Architecture Docs**: [Disaggregated Serving](../../../../docs/design_docs/disagg_serving.md), [KV-Aware Routing](../../../../docs/router/kv_cache_routing.md)
- **Multinode Deployment**: [Multinode Examples](../../../../docs/backends/trtllm/multinode/multinode-examples.md) - **Multinode Deployment**: [Multinode Examples](../../../../docs/backends/trtllm/multinode/multinode-examples.md)
......
...@@ -82,7 +82,7 @@ extraPodSpec: ...@@ -82,7 +82,7 @@ extraPodSpec:
Before using these templates, ensure you have: Before using these templates, ensure you have:
1. **Dynamo Cloud Platform installed** - See [Quickstart Guide](../../../../docs/kubernetes/README.md) 1. **Dynamo Kubernetes Platform installed** - See [Quickstart Guide](../../../../docs/kubernetes/README.md)
2. **Kubernetes cluster with GPU support** 2. **Kubernetes cluster with GPU support**
3. **Container registry access** for vLLM runtime images 3. **Container registry access** for vLLM runtime images
4. **HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`) 4. **HuggingFace token secret** (referenced as `envFromSecret: hf-token-secret`)
...@@ -138,7 +138,7 @@ kubectl create secret generic hf-token-secret \ ...@@ -138,7 +138,7 @@ kubectl create secret generic hf-token-secret \
Then, deploy the model using the deployment file. Then, deploy the model using the deployment file.
Export the NAMESPACE you used in your Dynamo Cloud Installation. Export the NAMESPACE you used in your Dynamo Kubernetes Platform Installation.
```bash ```bash
cd <dynamo-source-root>/examples/backends/vllm/deploy cd <dynamo-source-root>/examples/backends/vllm/deploy
...@@ -236,7 +236,7 @@ args: ...@@ -236,7 +236,7 @@ args:
- **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md) - **Deployment Guide**: [Creating Kubernetes Deployments](../../../../docs/kubernetes/deployment/create_deployment.md)
- **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md) - **Quickstart**: [Deployment Quickstart](../../../../docs/kubernetes/README.md)
- **Platform Setup**: [Dynamo Cloud Installation](../../../../docs/kubernetes/installation_guide.md) - **Platform Setup**: [Dynamo Kubernetes Platform Installation](../../../../docs/kubernetes/installation_guide.md)
- **SLA Planner**: [SLA Planner Quickstart Guide](../../../../docs/planner/sla_planner_quickstart.md) - **SLA Planner**: [SLA Planner Quickstart Guide](../../../../docs/planner/sla_planner_quickstart.md)
- **Examples**: [Deployment Examples](../../../../docs/examples/README.md) - **Examples**: [Deployment Examples](../../../../docs/examples/README.md)
- **Architecture Docs**: [Disaggregated Serving](../../../../docs/design_docs/disagg_serving.md), [KV-Aware Routing](../../../../docs/router/kv_cache_routing.md) - **Architecture Docs**: [Disaggregated Serving](../../../../docs/design_docs/disagg_serving.md), [KV-Aware Routing](../../../../docs/router/kv_cache_routing.md)
......
...@@ -11,7 +11,7 @@ This deployment pattern enables dynamic LoRA adapter loading from S3-compatible ...@@ -11,7 +11,7 @@ This deployment pattern enables dynamic LoRA adapter loading from S3-compatible
- Kubernetes cluster with GPU support - Kubernetes cluster with GPU support
- Helm 3.x installed - Helm 3.x installed
- `kubectl` configured to access your cluster - `kubectl` configured to access your cluster
- Dynamo Cloud Platform installed ([Installation Guide](../../../../../docs/kubernetes/installation_guide.md)) - Dynamo Kubernetes Platform installed ([Installation Guide](../../../../../docs/kubernetes/installation_guide.md))
- HuggingFace token for downloading Base and LoRA adapters - HuggingFace token for downloading Base and LoRA adapters
## Files in This Directory ## Files in This Directory
......
# Steps to install Dynamo Cloud from Source # Steps to install Dynamo Kubernetes Platform from Source
## 1. Build Dynamo Base Image ## 1. Build Dynamo Base Image
...@@ -12,7 +12,7 @@ aws ecr create-repository --repository-name <ECR_REPOSITORY> ...@@ -12,7 +12,7 @@ aws ecr create-repository --repository-name <ECR_REPOSITORY>
Build Image Build Image
``` ```
export NAMESPACE=dynamo-cloud export NAMESPACE=dynamo-system
export DOCKER_SERVER=<ECR_REGISTRY> export DOCKER_SERVER=<ECR_REGISTRY>
export DOCKER_USERNAME=AWS export DOCKER_USERNAME=AWS
export DOCKER_PASSWORD="$(aws ecr get-login-password --region <ECR_REGION>)" export DOCKER_PASSWORD="$(aws ecr get-login-password --region <ECR_REGION>)"
...@@ -28,12 +28,12 @@ aws ecr get-login-password | docker login --username AWS --password-stdin <ECR_R ...@@ -28,12 +28,12 @@ aws ecr get-login-password | docker login --username AWS --password-stdin <ECR_R
docker push <ECR_REGISTRY>/<ECR_REPOSITORY>:$IMAGE_TAG docker push <ECR_REGISTRY>/<ECR_REPOSITORY>:$IMAGE_TAG
``` ```
## 2. Install Dynamo Cloud ## 2. Install Dynamo Kubernetes Platform
Build and Push Operator Image Build and Push Operator Image
``` ```
cd deploy/cloud/operator cd deploy/operator
vim Earthfile # change ARG IMAGE_SUFFIX=<ECR_REPOSITORY> vim Earthfile # change ARG IMAGE_SUFFIX=<ECR_REPOSITORY>
earthly --push +docker --DOCKER_SERVER=$DOCKER_SERVER --IMAGE_TAG=$IMAGE_TAG earthly --push +docker --DOCKER_SERVER=$DOCKER_SERVER --IMAGE_TAG=$IMAGE_TAG
``` ```
...@@ -53,10 +53,10 @@ kubectl create secret generic hf-token-secret \ ...@@ -53,10 +53,10 @@ kubectl create secret generic hf-token-secret \
-n ${NAMESPACE} -n ${NAMESPACE}
``` ```
Install Dynamo Cloud Install Dynamo Kubernetes Platform
``` ```
cd dynamo/cloud/helm cd deploy/helm/charts
helm install dynamo-crds ./crds/ \ helm install dynamo-crds ./crds/ \
--namespace default \ --namespace default \
--wait \ --wait \
...@@ -88,8 +88,8 @@ Your pods should be running like below ...@@ -88,8 +88,8 @@ Your pods should be running like below
``` ```
ubuntu@ip-192-168-83-157:~/dynamo/examples/backends/vllm/deploy$ kubectl get pods -A ubuntu@ip-192-168-83-157:~/dynamo/examples/backends/vllm/deploy$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE NAMESPACE NAME READY STATUS RESTARTS AGE
dynamo-cloud dynamo-platform-dynamo-operator-controller-manager-86795c5f4j4k 2/2 Running 0 4h17m dynamo-system dynamo-platform-dynamo-operator-controller-manager-86795c5f4j4k 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-etcd-0 1/1 Running 0 4h17m dynamo-system dynamo-platform-etcd-0 1/1 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-0 2/2 Running 0 4h17m dynamo-system dynamo-platform-nats-0 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-box-5dbf45c748-bxqj7 1/1 Running 0 4h17m dynamo-system dynamo-platform-nats-box-5dbf45c748-bxqj7 1/1 Running 0 4h17m
``` ```
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
``` ```
cd dynamo/examples/backends/vllm/deploy cd dynamo/examples/backends/vllm/deploy
vim agg_router.yaml #under metadata add namespace: dynamo-cloud and change image to your built base image vim agg_router.yaml #under metadata add namespace: dynamo-system and change image to your built base image
kubectl apply -f agg_router.yaml kubectl apply -f agg_router.yaml
``` ```
...@@ -13,19 +13,19 @@ Your pods should be running like below ...@@ -13,19 +13,19 @@ Your pods should be running like below
``` ```
ubuntu@ip-192-168-83-157:~/dynamo/examples/backends/vllm/deploy$ kubectl get pods -A ubuntu@ip-192-168-83-157:~/dynamo/examples/backends/vllm/deploy$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE NAMESPACE NAME READY STATUS RESTARTS AGE
dynamo-cloud dynamo-platform-dynamo-operator-controller-manager-86795c5f4j4k 2/2 Running 0 4h17m dynamo-system dynamo-platform-dynamo-operator-controller-manager-86795c5f4j4k 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-etcd-0 1/1 Running 0 4h17m dynamo-system dynamo-platform-etcd-0 1/1 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-0 2/2 Running 0 4h17m dynamo-system dynamo-platform-nats-0 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-box-5dbf45c748-bxqj7 1/1 Running 0 4h17m dynamo-system dynamo-platform-nats-box-5dbf45c748-bxqj7 1/1 Running 0 4h17m
dynamo-cloud vllm-agg-router-frontend-79d599bb9c-fg97p 1/1 Running 0 4m9s dynamo-system vllm-agg-router-frontend-79d599bb9c-fg97p 1/1 Running 0 4m9s
dynamo-cloud vllm-agg-router-vllmdecodeworker-787d575485-hrcjp 1/1 Running 0 4m9s dynamo-system vllm-agg-router-vllmdecodeworker-787d575485-hrcjp 1/1 Running 0 4m9s
dynamo-cloud vllm-agg-router-vllmdecodeworker-787d575485-zkwdd 1/1 Running 0 4m9s dynamo-system vllm-agg-router-vllmdecodeworker-787d575485-zkwdd 1/1 Running 0 4m9s
``` ```
Test the Deployment Test the Deployment
``` ```
kubectl port-forward deployment/vllm-agg-router-frontend 8000:8000 -n dynamo-cloud kubectl port-forward deployment/vllm-agg-router-frontend 8000:8000 -n dynamo-system
curl localhost:8000/v1/chat/completions \ curl localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
......
...@@ -7,7 +7,7 @@ ...@@ -7,7 +7,7 @@
//! which stores discovery metadata for Dynamo worker pods in Kubernetes. //! which stores discovery metadata for Dynamo worker pods in Kubernetes.
//! //!
//! The CRD schema is defined in the Helm chart at: //! The CRD schema is defined in the Helm chart at:
//! `deploy/cloud/helm/crds/templates/nvidia.com_dynamoworkermetadatas.yaml` //! `deploy/helm/charts/crds/templates/nvidia.com_dynamoworkermetadatas.yaml`
use anyhow::Result; use anyhow::Result;
use k8s_openapi::apimachinery::pkg::apis::meta::v1::OwnerReference; use k8s_openapi::apimachinery::pkg::apis::meta::v1::OwnerReference;
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment