feat: deprecated namespace-restricted mode (#7907)

deploy/helm/charts/platform/README.md

@@ -118,11 +118,14 @@ The chart includes built-in validation to prevent all operator conflicts:
 | dynamo-operator.etcdAddr | string | `""` | etcd server address for an external etcd instance. Only needed when using external etcd without the bundled subchart. Format: "http://hostname:port" or "https://hostname:port" |
 | dynamo-operator.nats.enabled | bool | `true` | Whether the NATS is enabled |
 | dynamo-operator.modelExpressURL | string | `""` | URL for the Model Express server if not deployed by this helm chart. This is ignored if Model Express server is installed by this helm chart (global.model-express.enabled is true). |
-| dynamo-operator.namespaceRestriction | object | `{"enabled":false,"lease":{"duration":"30s","renewInterval":"10s"},"targetNamespace":null}` | Namespace access controls for the operator |
-| dynamo-operator.namespaceRestriction.enabled | bool | `false` | Whether to restrict operator to specific namespaces. By default, the operator will run with cluster-wide permissions. Only 1 instance of the operator should be deployed in the cluster. If you want to deploy multiple operator instances, you can set this to true and specify the target namespace (by default, the target namespace is the helm release namespace). |
-| dynamo-operator.namespaceRestriction.targetNamespace | string | `nil` | Target namespace for operator deployment (leave empty for current namespace) |
-| dynamo-operator.gpuDiscovery | object | `{"enabled":true}` | GPU discovery configuration (only applies when namespaceRestriction.enabled=true) |
-| dynamo-operator.gpuDiscovery.enabled | bool | `true` | Whether to provision a ClusterRole for the namespace-scoped operator to read GPU node labels. When true (default), Helm creates a ClusterRole/ClusterRoleBinding granting node read access. Set to false if your installer lacks ClusterRole creation permissions. |
+| dynamo-operator.namespaceRestriction | object | `{"enabled":false,"lease":{"duration":"30s","renewInterval":"10s"},"targetNamespace":null}` | DEPRECATED: Namespace-restricted mode is deprecated and will be removed in a future release. Use cluster-wide mode (the default) instead. Do not enable this for new deployments. |
+| dynamo-operator.namespaceRestriction.enabled | bool | `false` | DEPRECATED: Do not enable for new deployments. Namespace-restricted mode is deprecated. |
+| dynamo-operator.namespaceRestriction.targetNamespace | string | `nil` | DEPRECATED: Only used in namespace-restricted mode, which is deprecated. |
+| dynamo-operator.namespaceRestriction.lease | object | `{"duration":"30s","renewInterval":"10s"}` | DEPRECATED: Only used in namespace-restricted mode, which is deprecated. |
+| dynamo-operator.namespaceRestriction.lease.duration | string | `"30s"` | DEPRECATED: Lease duration for namespace-restricted mode, which is deprecated. |
+| dynamo-operator.namespaceRestriction.lease.renewInterval | string | `"10s"` | DEPRECATED: Lease renew interval for namespace-restricted mode, which is deprecated. |
+| dynamo-operator.gpuDiscovery | object | `{"enabled":true}` | DEPRECATED: GPU discovery for namespace-scoped operators is deprecated along with namespace-restricted mode. |
+| dynamo-operator.gpuDiscovery.enabled | bool | `true` | DEPRECATED: Only relevant when namespaceRestriction is enabled, which is deprecated. |
 | dynamo-operator.controllerManager.tolerations | list | `[]` | Node tolerations for controller manager pods |
 | dynamo-operator.controllerManager.affinity | object | `{}` | Affinity for controller manager pods |
 | dynamo-operator.controllerManager.leaderElection.id | string | `""` | Leader election ID for cluster-wide coordination. WARNING: All cluster-wide operators must use the SAME ID to prevent split-brain. Different IDs would allow multiple leaders simultaneously. |

deploy/helm/charts/platform/components/operator/values.yaml

@@ -24,27 +24,21 @@ upgradeCRD: true
 # Environment variables to pass to operator Deployment.
 env: []
 
-# Namespace restriction configuration for the operator
-# If enabled: true and targetNamespace is empty, the operator will be restricted to the release namespace
-# If enabled: true and targetNamespace is set, the operator will be restricted to the specified namespace
-# If enabled: false, the operator will run with cluster-wide permissions
+# -- DEPRECATED: Namespace-restricted mode is deprecated and will be removed in a future release. Use cluster-wide mode (the default) instead. Do not enable this for new deployments.
 namespaceRestriction:
-  # Whether to restrict the operator to a single namespace
+  # -- DEPRECATED: Do not enable for new deployments. Namespace-restricted mode is deprecated.
   enabled: false
-  # The target namespace to restrict to. If empty, defaults to the release namespace
+  # -- DEPRECATED: Only used in namespace-restricted mode, which is deprecated.
   targetNamespace: ""
-  # Namespace scope marker lease configuration (used to prevent conflicts when running both cluster-wide and namespace-restricted operators)
+  # -- DEPRECATED: Only used in namespace-restricted mode, which is deprecated.
   lease:
-    # Duration before the namespace scope marker lease expires if not renewed (namespace-restricted mode only). When a namespace-restricted operator is running, it creates a lease in its namespace. The cluster-wide operator detects this lease and excludes that namespace from processing. If the namespace operator stops renewing the lease (e.g., crashes), the lease expires and the cluster-wide operator automatically resumes processing that namespace.
+    # -- DEPRECATED: Lease duration for namespace-restricted mode, which is deprecated.
     duration: 30s
-    # Interval for renewing the namespace scope marker lease (namespace-restricted mode only). The namespace-restricted operator renews its lease at this interval to signal it's still running.
+    # -- DEPRECATED: Lease renew interval for namespace-restricted mode, which is deprecated.
     renewInterval: 10s
-# -- GPU discovery configuration (only applies when namespaceRestriction.enabled=true)
+# -- DEPRECATED: GPU discovery for namespace-scoped operators is deprecated along with namespace-restricted mode.
 gpuDiscovery:
-  # -- Whether to provision a ClusterRole for the namespace-scoped operator to read GPU node labels.
-  # When true (default), Helm creates a ClusterRole/ClusterRoleBinding granting node read access.
-  # Set to false if your installer lacks ClusterRole creation permissions; you must then provide
-  # hardware config manually in each DynamoGraphDeploymentRequest.
+  # -- DEPRECATED: Only relevant when namespaceRestriction is enabled, which is deprecated.
   enabled: true
 controllerManager:
   tolerations: []

deploy/helm/charts/platform/templates/NOTES.txt

@@ -14,6 +14,23 @@
 # limitations under the License.
 
 {{- $operatorValues := index .Values "dynamo-operator" }}
+{{- if $operatorValues.namespaceRestriction.enabled }}
+
+================================================================================
+  DEPRECATION WARNING
+================================================================================
+  Namespace-restricted mode (namespaceRestriction.enabled=true) is DEPRECATED
+  and will be removed in a future release.
+
+  The operator is configured for namespace: {{ default .Release.Namespace $operatorValues.namespaceRestriction.targetNamespace }}
+
+  Please migrate to cluster-wide mode by removing the namespaceRestriction
+  configuration from your values.
+
+  See: https://github.com/ai-dynamo/dynamo/blob/main/docs/kubernetes/installation-guide.md
+================================================================================
+
+{{- end }}
 {{- if and $operatorValues.namespaceRestriction.enabled $operatorValues.gpuDiscovery.enabled }}
 GPU Discovery: ENABLED — the operator will automatically detect GPU hardware from cluster nodes.
 {{- else if $operatorValues.namespaceRestriction.enabled }}

deploy/helm/charts/platform/values.yaml

@@ -69,24 +69,22 @@ dynamo-operator:
 
   # -- URL for the Model Express server if not deployed by this helm chart. This is ignored if Model Express server is installed by this helm chart (global.model-express.enabled is true).
   modelExpressURL: ""
-  # -- Namespace access controls for the operator
+  # -- DEPRECATED: Namespace-restricted mode is deprecated and will be removed in a future release. Use cluster-wide mode (the default) instead. Do not enable this for new deployments.
   namespaceRestriction:
-    # -- Whether to restrict operator to specific namespaces. By default, the operator will run with cluster-wide permissions. Only 1 instance of the operator should be deployed in the cluster. If you want to deploy multiple operator instances, you can set this to true and specify the target namespace (by default, the target namespace is the helm release namespace).
+    # -- DEPRECATED: Do not enable for new deployments. Namespace-restricted mode is deprecated.
     enabled: false
-    # -- Target namespace for operator deployment (leave empty for current namespace)
+    # -- DEPRECATED: Only used in namespace-restricted mode, which is deprecated.
     targetNamespace:
-    # Namespace scope marker lease configuration (used to prevent conflicts when running both cluster-wide and namespace-restricted operators)
+    # -- DEPRECATED: Only used in namespace-restricted mode, which is deprecated.
     lease:
-      # Duration before the namespace scope marker lease expires if not renewed (namespace-restricted mode only). When a namespace-restricted operator is running, it creates a lease in its namespace. The cluster-wide operator detects this lease and excludes that namespace from processing. If the namespace operator stops renewing the lease (e.g., crashes), the lease expires and the cluster-wide operator automatically resumes processing that namespace.
+      # -- DEPRECATED: Lease duration for namespace-restricted mode, which is deprecated.
       duration: 30s
-      # Interval for renewing the namespace scope marker lease (namespace-restricted mode only). The namespace-restricted operator renews its lease at this interval to signal it's still running.
+      # -- DEPRECATED: Lease renew interval for namespace-restricted mode, which is deprecated.
       renewInterval: 10s
 
-  # -- GPU discovery configuration (only applies when namespaceRestriction.enabled=true)
+  # -- DEPRECATED: GPU discovery for namespace-scoped operators is deprecated along with namespace-restricted mode.
   gpuDiscovery:
-    # -- Whether to provision a ClusterRole for the namespace-scoped operator to read GPU node labels.
-    # When true (default), Helm creates a ClusterRole/ClusterRoleBinding granting node read access.
-    # Set to false if your installer lacks ClusterRole creation permissions.
+    # -- DEPRECATED: Only relevant when namespaceRestriction is enabled, which is deprecated.
     enabled: true
 
   # -- The Dynamo discovery backend to use. Default is "kubernetes" for Kubernetes API service discovery. Set to "etcd" to use ETCD for discovery. --

deploy/operator/.golangci.yml

@@ -35,6 +35,11 @@ issues:
       linters:
         - lll
         - nakedret
+    # Intentional internal usage of deprecated namespace-restricted mode fields/packages.
+    # The feature is soft-deprecated (still functional) so internal references are expected.
+    - linters:
+        - staticcheck
+      text: "SA1019:.*deprecated.*namespace"
 linters:
   disable-all: true
   enable:

deploy/operator/api/config/v1alpha1/types.go

@@ -145,13 +145,15 @@ type LeaderElectionConfiguration struct {
 
 // NamespaceConfiguration determines operator namespace mode.
 type NamespaceConfiguration struct {
-	// Restricted is the namespace to restrict to. Empty = cluster-wide mode.
+	// Deprecated: Namespace-restricted mode is deprecated and will be removed in a future release.
+	// Use cluster-wide mode (leave Restricted empty) instead.
 	Restricted string `json:"restricted"`
-	// Scope holds namespace scope lease settings (namespace-restricted mode only)
+	// Deprecated: Scope is only used in namespace-restricted mode, which is deprecated.
 	Scope NamespaceScopeConfiguration `json:"scope"`
 }
 
-// NamespaceScopeConfiguration holds lease settings for namespace-restricted mode.
+// Deprecated: NamespaceScopeConfiguration is used only by the deprecated namespace-restricted
+// mode and will be removed in a future release.
 type NamespaceScopeConfiguration struct {
 	// LeaseDuration is the duration of namespace scope marker lease before expiration
 	// +kubebuilder:default="30s"

deploy/operator/cmd/main.go

@@ -26,6 +26,7 @@ import (
 	"fmt"
 	"net/http"
 	"os"
+	"strings"
 	"time"
 
 	// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
@@ -262,6 +263,16 @@ func main() {
 			restrictedNamespace: {},
 		}
 		setupLog.Info("Restricted namespace configured, launching in restricted mode", "namespace", restrictedNamespace)
+
+		banner := strings.Repeat("=", 80)
+		setupLog.Error(nil, banner)
+		setupLog.Error(nil, "DEPRECATION WARNING: Namespace-restricted mode is deprecated "+
+			"and will be removed in a future release.")
+		setupLog.Error(nil, "The operator is running in namespace-restricted mode",
+			"namespace", restrictedNamespace)
+		setupLog.Error(nil, "Please migrate to cluster-wide mode "+
+			"by removing the namespaceRestriction configuration.")
+		setupLog.Error(nil, banner)
 	} else {
 		setupLog.Info("No restricted namespace configured, launching in cluster-wide mode")
 	}

deploy/operator/internal/namespace_scope/lease_manager.go

@@ -15,6 +15,8 @@
  * limitations under the License.
  */
 
+// Deprecated: Package namespace_scope implements the lease-based coordination mechanism for the
+// deprecated namespace-restricted operator mode. It will be removed in a future release.
 package namespace_scope
 
 import (
@@ -37,7 +39,8 @@ const (
 	LeaseName = "dynamo-operator-namespace-scope"
 )
 
-// LeaseManager manages the namespace scope marker lease
+// Deprecated: LeaseManager manages the namespace scope marker lease for the deprecated
+// namespace-restricted operator mode.
 type LeaseManager struct {
 	client          kubernetes.Interface
 	namespace       string

deploy/operator/internal/namespace_scope/lease_watcher.go

@@ -15,6 +15,8 @@
  * limitations under the License.
  */
 
+// Deprecated: Package namespace_scope implements the lease-based coordination mechanism for the
+// deprecated namespace-restricted operator mode. It will be removed in a future release.
 package namespace_scope
 
 import (
@@ -33,9 +35,9 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/log"
 )
 
-// LeaseWatcher watches for namespace scope marker leases and maintains
-// an exclusion list for the cluster-wide operator.
-// It implements the ExcludedNamespacesInterface from controller_common.
+// Deprecated: LeaseWatcher watches for namespace scope marker leases and maintains
+// an exclusion list for the cluster-wide operator. It is part of the deprecated
+// namespace-restricted operator mode.
 type LeaseWatcher struct {
 	excludedNamespaces sync.Map // map[string]*coordinationv1.Lease (namespace -> lease object)
 	informerFactory    informers.SharedInformerFactory

docs/kubernetes/README.md

@@ -63,10 +63,7 @@ helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz --namespace
 
 **For Shared/Multi-Tenant Clusters:**
 
-If your cluster has namespace-restricted Dynamo operators, add this flag to step 2:
-```bash
---set dynamo-operator.namespaceRestriction.enabled=true
-```
+> **DEPRECATED:** Namespace-restricted mode (`namespaceRestriction.enabled=true`) is deprecated and will be removed in a future release. Use cluster-wide mode (the default) instead.
 
 For more details or customization options (including multinode deployments), see **[Installation Guide for Dynamo Kubernetes Platform](installation-guide.md)**.
 

docs/kubernetes/api-reference.md

@@ -2014,15 +2014,16 @@ _Appears in:_
 
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
-| `restricted` _string_ | Restricted is the namespace to restrict to. Empty = cluster-wide mode. |  |  |
-| `scope` _[NamespaceScopeConfiguration](#namespacescopeconfiguration)_ | Scope holds namespace scope lease settings (namespace-restricted mode only) |  |  |
+| `restricted` _string_ | Deprecated: Namespace-restricted mode is deprecated and will be removed in a future release.<br />Use cluster-wide mode (leave Restricted empty) instead. |  |  |
+| `scope` _[NamespaceScopeConfiguration](#namespacescopeconfiguration)_ | Deprecated: Scope is only used in namespace-restricted mode, which is deprecated. |  |  |
 
 
 #### NamespaceScopeConfiguration
 
 
 
-NamespaceScopeConfiguration holds lease settings for namespace-restricted mode.
+Deprecated: NamespaceScopeConfiguration is used only by the deprecated namespace-restricted
+mode and will be removed in a future release.
 
 
 

docs/kubernetes/installation-guide.md

@@ -14,7 +14,6 @@ Determine your cluster environment:
 - CRDs already installed cluster-wide - skip CRD installation step
 - A cluster-wide Dynamo operator is likely already running
 - **Do NOT install another operator** - use the existing cluster-wide operator
-- Only install a namespace-restricted operator if you specifically need to prevent the cluster-wide operator from managing your namespace (e.g., testing operator features you're developing)
 
 **Dedicated Cluster** (full cluster admin access):
 - You install CRDs yourself
@@ -136,20 +135,7 @@ helm install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz --namespace
 
 **For Shared/Multi-Tenant Clusters:**
 
-If your cluster has namespace-restricted Dynamo operators, you MUST add namespace restriction to your installation:
-
-```bash
-# Add this flag to the helm install command above
---set dynamo-operator.namespaceRestriction.enabled=true
-```
-
-Note: Use the full path `dynamo-operator.namespaceRestriction.enabled=true` (not just `namespaceRestriction.enabled=true`).
-
-If you see this validation error, you need namespace restriction:
-```
-VALIDATION ERROR: Cannot install cluster-wide Dynamo operator.
-Found existing namespace-restricted Dynamo operators in namespaces: ...
-```
+> **DEPRECATED:** Namespace-restricted mode (`namespaceRestriction.enabled=true`) is deprecated and will be removed in a future release. New deployments should use the default cluster-wide mode. If you are currently using namespace-restricted mode, plan to migrate to cluster-wide mode.
 
 > [!TIP]
 > For multinode deployments, you need to install multinode orchestration components:
@@ -196,17 +182,13 @@ Found existing namespace-restricted Dynamo operators in namespaces: ...
 --set "dynamo-operator.modelExpressURL=http://model-express-server.model-express.svc.cluster.local:8080"
 ```
 
-> [!TIP]
-> By default, Dynamo Operator is installed cluster-wide and will monitor all namespaces.
-> If you wish to restrict the operator to monitor only a specific namespace (the helm release namespace by default), you can set the namespaceRestriction.enabled to true.
-> You can also change the restricted namespace by setting the targetNamespace property.
+> [!WARNING]
+> **DEPRECATED:** Namespace-restricted mode is deprecated and will be removed in a future release.
+> By default, Dynamo Operator is installed cluster-wide and will monitor all namespaces. This is the recommended and only supported mode going forward.
 
-```bash
---set "dynamo-operator.namespaceRestriction.enabled=true"
---set "dynamo-operator.namespaceRestriction.targetNamespace=dynamo-namespace" # optional
-```
+### GPU Discovery for DynamoGraphDeploymentRequests (Deprecated Namespace-Scoped Mode)
 
-### GPU Discovery for DynamoGraphDeploymentRequests with Namespace-Scoped Operators
+> **DEPRECATED:** This section applies only to the deprecated namespace-restricted mode. New deployments should use cluster-wide mode, which has GPU discovery by default.
 
 GPU discovery is **enabled by default** for namespace-scoped operators. The Helm chart automatically provisions a ClusterRole/ClusterRoleBinding granting the operator read-only access to node GPU labels.
 
@@ -270,15 +252,13 @@ cd deploy/helm/charts
 # 4. Install Platform (CRDs are automatically installed by the chart)
 helm dep build ./platform/
 
-# To install cluster-wide instead, set NS_RESTRICT_FLAGS="" (empty) or omit that line entirely.
+# NOTE: Namespace-restricted mode is DEPRECATED. Use cluster-wide mode (the default).
 
-NS_RESTRICT_FLAGS="--set dynamo-operator.namespaceRestriction.enabled=true"
 helm install dynamo-platform ./platform/ \
   --namespace "${NAMESPACE}" \
   --set "dynamo-operator.controllerManager.manager.image.repository=${DOCKER_SERVER}/kubernetes-operator" \
   --set "dynamo-operator.controllerManager.manager.image.tag=${IMAGE_TAG}" \
-  --set "dynamo-operator.imagePullSecrets[0].name=docker-imagepullsecret" \
-  ${NS_RESTRICT_FLAGS}
+  --set "dynamo-operator.imagePullSecrets[0].name=docker-imagepullsecret"
 
 ```
 
@@ -327,12 +307,7 @@ Found existing namespace-restricted Dynamo operators in namespaces: ...
 
 Cause: Attempting cluster-wide install on a shared cluster with existing namespace-restricted operators.
 
-Solution: Add namespace restriction to your installation:
-```bash
---set dynamo-operator.namespaceRestriction.enabled=true
-```
-
-Note: Use the full path `dynamo-operator.namespaceRestriction.enabled=true` (not just `namespaceRestriction.enabled=true`).
+Solution: Migrate the existing namespace-restricted operators to cluster-wide mode. Namespace-restricted mode is deprecated and should no longer be used.
 
 **CRDs already exist**
 

docs/kubernetes/topology-aware-scheduling.md

@@ -15,7 +15,7 @@ TAS is **opt-in**. Existing deployments without topology constraints continue to
 | **Grove** | Installed on the cluster. See the [Grove Installation Guide](https://github.com/NVIDIA/grove/blob/main/docs/installation.md). |
 | **ClusterTopology CR** | A cluster-scoped `ClusterTopology` resource configured by the cluster admin, mapping topology domain names to node labels. See [Grove documentation](https://github.com/NVIDIA/grove) for setup instructions. |
 | **KAI Scheduler** | [KAI Scheduler](https://github.com/NVIDIA/KAI-Scheduler) is required by Grove for topology-aware pod placement. |
-| **Dynamo operator** | The latest Dynamo operator Helm chart includes read-only RBAC for `clustertopologies.grove.io` via a dedicated ClusterRole. This works for both cluster-wide and namespace-restricted operator deployments — no extra configuration is needed. |
+| **Dynamo operator** | The latest Dynamo operator Helm chart includes read-only RBAC for `clustertopologies.grove.io` via a dedicated ClusterRole. No extra configuration is needed. |
 
 ## Topology Domains
 

docs/kubernetes/webhooks.md

@@ -18,7 +18,6 @@ All webhook types (validating, mutating, conversion, etc.) share the same **webh
 - ✅ **Shared certificate infrastructure** - All webhook types use the same TLS certificates
 - ✅ **Automatic certificate generation and rotation** - Built-in cert-controller, no manual management required
 - ✅ **cert-manager integration** - Optional integration for custom PKI or organizational certificate policies
-- ✅ **Multi-operator support** - Lease-based coordination for cluster-wide and namespace-restricted deployments
 - ✅ **Immutability enforcement** - Critical fields protected via CEL validation rules
 
 ### Current Webhook Types
@@ -165,7 +164,7 @@ webhook:
       values: ["disabled"]
 ```
 
-**Note:** For **namespace-restricted operators**, the namespace selector is automatically set to validate only the operator's namespace. This configuration is ignored in namespace-restricted mode.
+**Note:** For **namespace-restricted operators** (deprecated), the namespace selector is automatically set to validate only the operator's namespace. This configuration is ignored in namespace-restricted mode.
 
 ---
 
@@ -335,7 +334,9 @@ helm install dynamo-platform . -n <namespace> -f values.yaml
 
 ---
 
-## Multi-Operator Deployments
+## Multi-Operator Deployments (DEPRECATED)
+
+> **DEPRECATED:** Namespace-restricted mode and multi-operator deployments are deprecated and will be removed in a future release. Use a single cluster-wide operator instead.
 
 The operator supports running both **cluster-wide** and **namespace-restricted** instances simultaneously using a **lease-based coordination mechanism**.
 
@@ -640,9 +641,7 @@ helm upgrade <release> dynamo-platform -n <namespace>
 ### Multi-Tenant Deployments
 
 1. ✅ **Deploy one cluster-wide operator** for platform-wide validation
-2. ✅ **Deploy namespace-restricted operators** for tenant-specific namespaces
-3. ✅ **Monitor lease health** to ensure coordination works correctly
-4. ✅ **Use unique release names** per namespace to avoid naming conflicts
+2. ~~Deploy namespace-restricted operators for tenant-specific namespaces~~ (**DEPRECATED** - use cluster-wide mode instead)
 
 ---