kata-deploy: add podLabels, podAnnotations and affinity to DaemonSet

Expose podLabels, podAnnotations and affinity in values.yaml so operators can apply more granular scheduling controls on the kata-deploy DaemonSet alongside the existing nodeSelector and tolerations knobs. Defaults are empty, so existing deployments render the same manifest. When NFD is enabled, the chart merges the hard-coded virtualization nodeAffinity with any user provided affinity. Signed-off-by: Zachary Spar <zspar@coreweave.com>
2026-07-09 22:24:51 +00:00 · 2026-06-30 16:21:28 -04:00
parent 068bd806d3
commit 5e4946b21e
5 changed files with 351 additions and 67 deletions
--- a/docs/helm-configuration.md
+++ b/docs/helm-configuration.md
@@ -16,7 +16,7 @@ helm show values --version X.Y.Z oci://ghcr.io/kata-containers/kata-deploy-chart

 Kata ships with a number of pre-built artifacts and runtimes. You may selectively enable or disable specific shims. For example:

-```yaml title="values.yaml"
+```yaml
 shims:
  disableAll: true
  qemu:
@@ -332,7 +332,7 @@ $ helm install kata-deploy \

 You can also use a values file:

-```yaml title="values.yaml"
+```yaml
 nodeSelector:
  kata-containers: "enabled"
  node-type: "worker"
@@ -342,6 +342,69 @@ nodeSelector:
 $ helm install kata-deploy -f values.yaml "${CHART}" --version "${VERSION}"
 ```

+### `podLabels`
+
+You can add extra labels to the kata-deploy DaemonSet pods. These are applied
+in addition to the `name: kata-deploy` label that the chart uses internally.
+
+```sh
+$ helm install kata-deploy \
+  --set podLabels.team=platform \
+  "${CHART}" --version "${VERSION}"
+```
+
+Or via a values file:
+
+```yaml
+podLabels:
+  team: platform
+```
+
+### `affinity`
+
+Use `affinity` when you need more granular scheduling controls than
+`nodeSelector` alone. `nodeSelector` only matches exact key/value pairs on a
+node; affinity gives you `matchExpressions` (e.g. `In`, `NotIn`) and rules
+about other pods on the same node. For example, you might want kata-deploy on
+nodes reserved for your platform team but *not* on nodes that run the GPU
+operator.
+
+```sh
+# First, label the nodes where kata-deploy should run
+$ kubectl label nodes worker-node-1 node.cloud/reserved=platform-team
+$ kubectl label nodes worker-node-2 node.cloud/reserved=platform-team
+
+# Then install the chart with affinity
+$ helm install kata-deploy -f values.yaml "${CHART}" --version "${VERSION}"
+```
+
+```yaml
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: node.cloud/reserved
+              operator: In
+              values:
+                - platform-team
+  podAntiAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      - labelSelector:
+          matchExpressions:
+            - key: app
+              operator: In
+              values:
+                - gpu-operator
+        topologyKey: kubernetes.io/hostname
+```
+
+When `node-feature-discovery.enabled=true`, the chart also merges in a
+`nodeAffinity` rule that requires hardware virtualization support. If you set
+`affinity.nodeAffinity` yourself, your required `nodeSelectorTerms` are
+combined with the built-in virtualization terms. Other affinity fields
+(`podAffinity`, `podAntiAffinity`, and so on) are passed through as-is.
+
 ### Multiple Kata installations on the Same Node

 For debugging, testing and other use-case it is possible to deploy multiple
--- a/tests/functional/kata-deploy/kata-deploy-scheduling.bats
+++ b/tests/functional/kata-deploy/kata-deploy-scheduling.bats
@@ -0,0 +1,129 @@
+#!/usr/bin/env bats
+# Copyright (c) 2026 The Kata Containers Authors
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# Helm template tests for kata-deploy DaemonSet scheduling options
+# (podLabels, podAnnotations, affinity). No cluster required.
+
+load "${BATS_TEST_DIRNAME}/../../common.bash"
+
+source "${BATS_TEST_DIRNAME}/lib/helm-deploy.bash"
+
+CHART_PATH="$(get_chart_path)"
+RENDERED="/tmp/kata-deploy-scheduling-rendered.yaml"
+
+render_chart() {
+	helm template kata-deploy "${CHART_PATH}" \
+		--set image.reference=quay.io/kata-containers/kata-deploy \
+		--set image.tag=latest \
+		"$@" > "${RENDERED}"
+}
+
+# Extract the kata-deploy DaemonSet manifest (not kata-monitor or NFD subchart).
+extract_kata_deploy_ds() {
+	awk '
+		/^kind: DaemonSet$/ { buf = $0 "\n"; in_ds = 1; has_name = 0; next }
+		in_ds {
+			buf = buf $0 "\n"
+			if ($0 ~ /^  name: kata-deploy$/) { has_name = 1 }
+			if ($0 ~ /^---$/) {
+				if (has_name) { printf "%s", buf; exit }
+				in_ds = 0; buf = ""; has_name = 0
+				next
+			}
+		}
+		END { if (has_name && in_ds) { printf "%s", buf } }
+	' "${RENDERED}"
+}
+
+# =============================================================================
+# Template Rendering Tests (no cluster required)
+# =============================================================================
+
+@test "Helm template: default values keep single name pod label and no affinity" {
+	render_chart
+
+	local ds
+	ds=$(extract_kata_deploy_ds)
+
+	[[ -n "${ds}" ]]
+	echo "${ds}" | grep -q "name: kata-deploy"
+	echo "${ds}" | grep -A5 "template:" | grep -A3 "labels:" | grep -q "name: kata-deploy"
+	! echo "${ds}" | grep -A10 "template:" | grep -A5 "metadata:" | grep -q "annotations:"
+	! echo "${ds}" | grep -q "affinity:"
+}
+
+@test "Helm template: podLabels are applied to pod template" {
+	render_chart --set podLabels.team=platform
+
+	local ds
+	ds=$(extract_kata_deploy_ds)
+
+	echo "${ds}" | grep -A5 "template:" | grep -A4 "labels:" | grep -q "name: kata-deploy"
+	echo "${ds}" | grep -A5 "template:" | grep -A4 "labels:" | grep -q "team: platform"
+}
+
+@test "Helm template: user affinity is applied to pod spec" {
+	local values_file
+	values_file=$(mktemp)
+	cat > "${values_file}" <<EOF
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: node.cloud/reserved
+              operator: In
+              values:
+                - platform-team
+  podAntiAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      - labelSelector:
+          matchExpressions:
+            - key: app
+              operator: In
+              values:
+                - gpu-operator
+        topologyKey: kubernetes.io/hostname
+EOF
+
+	render_chart -f "${values_file}"
+	rm -f "${values_file}"
+
+	local ds
+	ds=$(extract_kata_deploy_ds)
+
+	echo "${ds}" | grep -q "affinity:"
+	echo "${ds}" | grep -q "node.cloud/reserved"
+	echo "${ds}" | grep -q "platform-team"
+	echo "${ds}" | grep -q "podAntiAffinity:"
+	echo "${ds}" | grep -q "gpu-operator"
+}
+
+@test "Helm template: NFD enabled merges virtualization nodeAffinity with user nodeAffinity" {
+	local values_file
+	values_file=$(mktemp)
+	cat > "${values_file}" <<EOF
+affinity:
+  nodeAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      nodeSelectorTerms:
+        - matchExpressions:
+            - key: node.cloud/reserved
+              operator: In
+              values:
+                - platform-team
+EOF
+
+	render_chart -f "${values_file}" --set node-feature-discovery.enabled=true
+	rm -f "${values_file}"
+
+	local ds
+	ds=$(extract_kata_deploy_ds)
+
+	echo "${ds}" | grep -q "node.cloud/reserved"
+	echo "${ds}" | grep -q "platform-team"
+	echo "${ds}" | grep -q "feature.node.kubernetes.io/cpu-cpuid.VMX"
+	echo "${ds}" | grep -q "feature.node.kubernetes.io/cpu-cpuid.SVM"
+}
--- a/tools/packaging/kata-deploy/helm-chart/kata-deploy/templates/_helpers.tpl
+++ b/tools/packaging/kata-deploy/helm-chart/kata-deploy/templates/_helpers.tpl
@@ -921,3 +921,111 @@ Returns "true" when at least one default shim has a non-empty dropIn value.
 {{- end -}}
 {{- if $has -}}true{{- end -}}
 {{- end -}}
+
+{{/*
+NFD virtualization nodeAffinity for the kata-deploy DaemonSet.
+Applied when node-feature-discovery is managed by this chart (enabled: true).
+Kata Containers requires hardware virtualization support to function.
+
+Note: Virtualization checks are ONLY enforced when node-feature-discovery is
+      managed by kata-deploy. If node-feature-discovery is installed
+      independently (enabled: false), no checks are applied because we cannot
+      guarantee the external node-feature-discovery configuration and labels.
+
+NOTE: For kata-remote/peer-pods support in the future, add a condition here:
+      if and (index .Values "node-feature-discovery" "enabled") (not .Values.cloud-api-adaptor.enabled)
+*/}}
+{{- define "kata-deploy.nfdVirtualizationNodeAffinity" -}}
+nodeAffinity:
+  requiredDuringSchedulingIgnoredDuringExecution:
+    nodeSelectorTerms:
+    # x86_64: Intel VT-x (VMX) support
+    - matchExpressions:
+      - key: feature.node.kubernetes.io/cpu-cpuid.VMX
+        operator: In
+        values:
+        - "true"
+      - key: kubernetes.io/arch
+        operator: In
+        values:
+        - "amd64"
+    # x86_64: AMD-V (SVM) support
+    - matchExpressions:
+      - key: feature.node.kubernetes.io/cpu-cpuid.SVM
+        operator: In
+        values:
+        - "true"
+      - key: kubernetes.io/arch
+        operator: In
+        values:
+        - "amd64"
+    # aarch64: Allow all ARM64 nodes (virtualization check not yet implemented)
+    # TODO: Implement proper virtualization detection for aarch64
+    - matchExpressions:
+      - key: kubernetes.io/arch
+        operator: In
+        values:
+        - "arm64"
+        - "aarch64"
+    # s390x: Allow all s390x nodes (virtualization check not yet implemented)
+    # TODO: Implement proper virtualization detection for s390x
+    - matchExpressions:
+      - key: kubernetes.io/arch
+        operator: In
+        values:
+        - "s390x"
+    # ppc64le: Allow all ppc64le nodes (virtualization check not yet implemented)
+    # TODO: Implement proper virtualization detection for ppc64le
+    - matchExpressions:
+      - key: kubernetes.io/arch
+        operator: In
+        values:
+        - "ppc64le"
+    # riscv64: Allow all RISC-V nodes (virtualization support not yet available)
+    # TODO: Implement virtualization detection when RISC-V virt support is available
+    - matchExpressions:
+      - key: kubernetes.io/arch
+        operator: In
+        values:
+        - "riscv64"
+{{- end -}}
+
+{{/*
+Merged affinity for the kata-deploy DaemonSet.
+When NFD is enabled, the built-in virtualization nodeAffinity is always applied.
+If the user sets affinity.nodeAffinity, their required nodeSelectorTerms are
+merged with the NFD terms (each NFD term is AND-ed with each user term).
+*/}}
+{{- define "kata-deploy.daemonsetAffinity" -}}
+{{- $affinity := .Values.affinity | default dict | deepCopy -}}
+{{- if index .Values "node-feature-discovery" "enabled" -}}
+{{- $nfd := include "kata-deploy.nfdVirtualizationNodeAffinity" . | fromYaml -}}
+{{- $nfdNodeAffinity := $nfd.nodeAffinity -}}
+{{- if not (hasKey $affinity "nodeAffinity") -}}
+{{- $affinity = merge $affinity $nfd -}}
+{{- else -}}
+{{- $userNodeAffinity := $affinity.nodeAffinity | deepCopy -}}
+{{- $nfdRequired := $nfdNodeAffinity.requiredDuringSchedulingIgnoredDuringExecution | default dict -}}
+{{- $nfdTerms := $nfdRequired.nodeSelectorTerms | default list -}}
+{{- $userRequired := $userNodeAffinity.requiredDuringSchedulingIgnoredDuringExecution | default dict -}}
+{{- $userTerms := $userRequired.nodeSelectorTerms | default list -}}
+{{- $mergedTerms := list -}}
+{{- if $userTerms -}}
+{{- range $nfdTerm := $nfdTerms -}}
+{{- range $userTerm := $userTerms -}}
+{{- $nfdExprs := $nfdTerm.matchExpressions | default list -}}
+{{- $userExprs := $userTerm.matchExpressions | default list -}}
+{{- $mergedTerms = append $mergedTerms (dict "matchExpressions" (concat $nfdExprs $userExprs)) -}}
+{{- end -}}
+{{- end -}}
+{{- else -}}
+{{- $mergedTerms = $nfdTerms -}}
+{{- end -}}
+{{- $_ := set $userNodeAffinity "requiredDuringSchedulingIgnoredDuringExecution" (dict "nodeSelectorTerms" $mergedTerms) -}}
+{{- $_ := set $affinity "nodeAffinity" $userNodeAffinity -}}
+{{- end -}}
+{{- end -}}
+{{- if $affinity -}}
+{{- $affinity | toYaml -}}
+{{- end -}}
+{{- end -}}
--- a/tools/packaging/kata-deploy/helm-chart/kata-deploy/templates/kata-deploy.yaml
+++ b/tools/packaging/kata-deploy/helm-chart/kata-deploy/templates/kata-deploy.yaml
@@ -43,6 +43,13 @@ spec:
        name: {{ .Chart.Name }}-{{ .Values.env.multiInstallSuffix }}
 {{- else }}
        name: {{ .Chart.Name }}
+{{- end }}
+{{- with .Values.podLabels }}
+{{- toYaml . | nindent 8 }}
+{{- end }}
+{{- with .Values.podAnnotations }}
+      annotations:
+{{- toYaml . | nindent 8 }}
 {{- end }}
    spec:
 {{- with .Values.imagePullSecrets }}
@@ -65,72 +72,10 @@ spec:
 {{- with .Values.priorityClassName }}
      priorityClassName: {{ . | quote }}
 {{- end }}
-{{- if index .Values "node-feature-discovery" "enabled" }}
-      # When node-feature-discovery is managed by this chart (enabled: true), enforce virtualization requirements.
-      # Kata Containers requires hardware virtualization support to function.
-      #
-      # Note: Virtualization checks are ONLY enforced when node-feature-discovery is
-      #       managed by kata-deploy
-      #
-      #       If node-feature-discovery is installed independently (enabled: false),
-      #       no checks are applied because we cannot guarantee the external
-      #       node-feature-discovery configuration and labels.
-      #
-      # NOTE: For kata-remote/peer-pods support in the future, add a condition here:
-      #       if and (index .Values "node-feature-discovery" "enabled") (not .Values.cloud-api-adaptor.enabled)
+{{- $dsAffinity := include "kata-deploy.daemonsetAffinity" . | trim -}}
+{{- if $dsAffinity }}
      affinity:
-        nodeAffinity:
-          requiredDuringSchedulingIgnoredDuringExecution:
-            nodeSelectorTerms:
-            # x86_64: Intel VT-x (VMX) support
-            - matchExpressions:
-              - key: feature.node.kubernetes.io/cpu-cpuid.VMX
-                operator: In
-                values:
-                - "true"
-              - key: kubernetes.io/arch
-                operator: In
-                values:
-                - "amd64"
-            # x86_64: AMD-V (SVM) support
-            - matchExpressions:
-              - key: feature.node.kubernetes.io/cpu-cpuid.SVM
-                operator: In
-                values:
-                - "true"
-              - key: kubernetes.io/arch
-                operator: In
-                values:
-                - "amd64"
-            # aarch64: Allow all ARM64 nodes (virtualization check not yet implemented)
-            # TODO: Implement proper virtualization detection for aarch64
-            - matchExpressions:
-              - key: kubernetes.io/arch
-                operator: In
-                values:
-                - "arm64"
-                - "aarch64"
-            # s390x: Allow all s390x nodes (virtualization check not yet implemented)
-            # TODO: Implement proper virtualization detection for s390x
-            - matchExpressions:
-              - key: kubernetes.io/arch
-                operator: In
-                values:
-                - "s390x"
-            # ppc64le: Allow all ppc64le nodes (virtualization check not yet implemented)
-            # TODO: Implement proper virtualization detection for ppc64le
-            - matchExpressions:
-              - key: kubernetes.io/arch
-                operator: In
-                values:
-                - "ppc64le"
-            # riscv64: Allow all RISC-V nodes (virtualization support not yet available)
-            # TODO: Implement virtualization detection when RISC-V virt support is available
-            - matchExpressions:
-              - key: kubernetes.io/arch
-                operator: In
-                values:
-                - "riscv64"
+{{- $dsAffinity | nindent 8 }}
 {{- end }}
      hostPID: true
      # Allow SIGTERM cleanup (uninstall) to complete before force-kill.
--- a/tools/packaging/kata-deploy/helm-chart/kata-deploy/values.yaml
+++ b/tools/packaging/kata-deploy/helm-chart/kata-deploy/values.yaml
@@ -198,6 +198,45 @@ tolerations: []
 #     - "katacontainers.io/kata-runtime:NoSchedule"
 startupTaints: []

+# Extra labels for the kata-deploy DaemonSet pods (in addition to the
+# required "name: kata-deploy" label used by the chart).
+# Examples:
+# podLabels:
+#   team: platform
+podLabels: {}
+
+# Annotations for the kata-deploy DaemonSet pods.
+# Examples:
+# podAnnotations:
+#   prometheus.io/scrape: "false"
+podAnnotations: {}
+
+# Affinity rules for the kata-deploy DaemonSet.
+# nodeSelector matches exact key/value pairs; affinity supports matchExpressions
+# and rules about other pods on the same node (e.g. podAntiAffinity).
+# When node-feature-discovery.enabled=true, the chart merges in a virtualization
+# nodeAffinity rule; user affinity.nodeAffinity required terms are combined with it.
+# Examples:
+# affinity:
+#   nodeAffinity:
+#     requiredDuringSchedulingIgnoredDuringExecution:
+#       nodeSelectorTerms:
+#         - matchExpressions:
+#             - key: node.cloud/reserved
+#               operator: In
+#               values:
+#                 - platform-team
+#   podAntiAffinity:
+#     requiredDuringSchedulingIgnoredDuringExecution:
+#       - labelSelector:
+#           matchExpressions:
+#             - key: app
+#               operator: In
+#               values:
+#                 - gpu-operator
+#         topologyKey: kubernetes.io/hostname
+affinity: {}
+
 # Priority class name for the kata-deploy DaemonSet pods.
 #
 # kata-deploy is an infrastructure DaemonSet that installs Kata runtime