docs: add docs and example for inference gateway deployment (#1533)

deploy/inference-gateway/example/README.md

+# Installing Inference Gateway with Dynamo (Experimental)
+
+This is an experimental setup that treats each Dynamo deployment as a black box and routes traffic randomly among the deployments.
+
+This guide provides instructions for setting up the Inference Gateway with Dynamo for managing and routing inference requests.
+
+## Prerequisites
+
+- Kubernetes cluster with kubectl configured
+- NVIDIA GPU drivers installed on worker nodes
+
+## Installation Steps
+
+1. **Install Dynamo Cloud**
+
+Follow the instructions in [deploy/cloud/README.md](../../deploy/cloud/README.md) to deploy Dynamo Cloud on your Kubernetes cluster. This will set up the necessary infrastructure components for managing Dynamo inference graphs.
+
+2. **Launch 2 Dynamo Deployments**
+
+Deploy 2 Dynamo aggregated graphs following the instructions in [examples/llm/README.md](../../examples/llm/README.md):
+
+### Build Dynamo Graph
+```bash
+export DYNAMO_IMAGE=<your-registry>/<your-image-name>:<your-tag>
+
+# Build the service
+cd $PROJECT_ROOT/examples/llm
+export DYNAMO_TAG=$(dynamo build graphs.agg:Frontend | grep "Successfully built" |  awk '{ print $NF }' | sed 's/\.$//')
+```
+
+### Deploy Dynamo Graphs
+```bash
+# Deploy first graph
+export DEPLOYMENT_NAME=llm-agg1
+dynamo deployment create $DYNAMO_TAG -n $DEPLOYMENT_NAME -f ./configs/agg.yaml
+
+# Deploy second graph
+export DEPLOYMENT_NAME=llm-agg2
+dynamo deployment create $DYNAMO_TAG -n $DEPLOYMENT_NAME -f ./configs/agg.yaml
+```
+
+3. **Deploy Inference Gateway**
+
+First, deploy an inference gateway service. In this example, we'll install `kgateway` based gateway implementation.
+
+Install the Inference Extension CRDs:
+```bash
+VERSION=v0.3.0
+kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/$VERSION/manifests.yaml
+```
+
+Deploy an Inference Gateway. In this example, we'll install `Kgateway`:
+```bash
+KGTW_VERSION=v2.0.2
+
+# Install the Kgateway CRDs
+helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
+
+# Install Kgateway
+helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
+
+# Deploy the Gateway
+kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml
+kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/gateway.yaml
+```
+
+### Validate Resources
+```bash
+kubectl get gateway inference-gateway
+
+# Sample output
+# NAME                CLASS      ADDRESS   PROGRAMMED   AGE
+# inference-gateway   kgateway             True         1m
+```
+
+4. **Apply Dynamo-specific manifests**
+
+The Inference Gateway is configured through the `inference-gateway-resources.yaml` file.
+
+Deploy the Inference Gateway resources to your Kubernetes cluster:
+
+```bash
+cd deploy/inference-gateway/example
+kubectl apply -f resources
+```
+
+Key configurations include:
+- An InferenceModel resource for the DeepSeek model
+- A service for the inference gateway
+- Required RBAC roles and bindings
+- RBAC permissions
+
+5. **Verify Installation**
+
+Check that all resources are properly deployed:
+
+```bash
+kubectl get inferencepool
+kubectl get inferencemodel
+kubectl get httproute
+```
+
+Sample output:
+
+```bash
+# kubectl get inferencepool
+NAME              AGE
+dynamo-deepseek   6s
+
+# kubectl get inferencemodel
+NAME              MODEL NAME                                 INFERENCE POOL    CRITICALITY   AGE
+deep-seek-model   deepseek-ai/DeepSeek-R1-Distill-Llama-8B   dynamo-deepseek   Critical      6s
+
+# kubectl get httproute
+NAME        HOSTNAMES   AGE
+llm-route               6s
+```
+
+## Usage
+
+The Inference Gateway provides HTTP/2 endpoints for model inference. The default service is exposed on port 9002.
+
+### 1: Populate gateway URL for your k8s cluster
+```bash
+export GATEWAY_URL=<Gateway-URL>
+```
+
+To test the gateway in minikube, use the following command:
+```bash
+minikube tunnel &
+
+GATEWAY_URL=$(kubectl get svc inference-gateway -o yaml -o jsonpath='{.spec.clusterIP}')
+echo $GATEWAY_URL
+```
+
+### 2: Check models deployed to inference gateway
+
+Query models:
+```bash
+curl $GATEWAY_URL/v1/models | jq .
+```
+
+Send inference request to gateway:
+
+```bash
+curl $GATEWAY_URL/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
+    "messages": [
+    {
+        "role": "user",
+        "content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
+    }
+    ],
+    "stream":false,
+    "max_tokens": 30
+  }'
+```
\ No newline at end of file

deploy/inference-gateway/example/resources/cluster-role-binding.yaml

+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: pod-read-binding
+subjects:
+- kind: ServiceAccount
+  name: default
+  namespace: default
+roleRef:
+  kind: ClusterRole
+  name: pod-read

deploy/inference-gateway/example/resources/cluster-role.yaml

+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: pod-read
+rules:
+- apiGroups: ["inference.networking.x-k8s.io"]
+  resources: ["inferencepools"]
+  verbs: ["get", "watch", "list"]
+- apiGroups: ["inference.networking.x-k8s.io"]
+  resources: ["inferencemodels"]
+  verbs: ["get", "watch", "list"]
+- apiGroups: [""]
+  resources: ["pods"]
+  verbs: ["get", "watch", "list"]
+- apiGroups:
+  - authentication.k8s.io
+  resources:
+  - tokenreviews
+  verbs:
+  - create
+- apiGroups:
+  - authorization.k8s.io
+  resources:
+  - subjectaccessreviews
+  verbs:
+  - create
\ No newline at end of file

deploy/inference-gateway/example/resources/dynamo-epp.yaml

+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dynamo-deepseek-epp
+  namespace: default
+  labels:
+    app: dynamo-deepseek-epp
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: dynamo-deepseek-epp
+  template:
+    metadata:
+      labels:
+        app: dynamo-deepseek-epp
+    spec:
+      # Conservatively, this timeout should mirror the longest grace period of the pods within the pool
+      terminationGracePeriodSeconds: 130
+      containers:
+      - name: epp
+        image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:main
+        imagePullPolicy: Always
+        args:
+        - -poolName
+        - "dynamo-deepseek"
+        - "-poolNamespace"
+        - "default"
+        - -v
+        - "4"
+        - --zap-encoder
+        - "json"
+        - -grpcPort
+        - "9002"
+        - -grpcHealthPort
+        - "9003"
+        ports:
+        - containerPort: 9002
+        - containerPort: 9003
+        - name: metrics
+          containerPort: 9090
+        livenessProbe:
+          grpc:
+            port: 9003
+            service: inference-extension
+          initialDelaySeconds: 5
+          periodSeconds: 10
+        readinessProbe:
+          grpc:
+            port: 9003
+            service: inference-extension
+          initialDelaySeconds: 5
+          periodSeconds: 10

deploy/inference-gateway/example/resources/http-router.yaml

+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: llm-route
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: inference-gateway
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: dynamo-deepseek
+    matches:
+    - path:
+        type: PathPrefix
+        value: /
+    timeouts:
+      request: 300s
\ No newline at end of file

deploy/inference-gateway/example/resources/inference-model.yaml

+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferenceModel
+metadata:
+  name: deep-seek-model
+  namespace: default
+spec:
+  criticality: Critical
+  modelName: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
+  poolRef:
+    group: inference.networking.x-k8s.io
+    kind: InferencePool
+    name: dynamo-deepseek
\ No newline at end of file

deploy/inference-gateway/example/resources/inference-pool.yaml

+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferencePool
+metadata:
+  name: dynamo-deepseek
+  namespace: default
+spec:
+  targetPortNumber: 3000
+  selector:
+    nvidia.com/dynamo-component-type: Frontend
+  extensionRef:
+    failureMode: FailClose
+    group: ""
+    kind: Service
+    name: dynamo-deepseek-epp
\ No newline at end of file

deploy/inference-gateway/example/resources/service.yaml

+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: v1
+kind: Service
+metadata:
+  name: dynamo-deepseek-epp
+  namespace: default
+spec:
+  selector:
+    app: dynamo-deepseek-epp
+  ports:
+    - protocol: TCP
+      port: 9002
+      targetPort: 9002
+      appProtocol: http2
+  type: ClusterIP
\ No newline at end of file