kata-containers/tools/packaging/kata-deploy/helm-chart

Kata Containers Deploy – Helm Chart

A Helm chart that installs the kata-deploy DaemonSet and its helper assets, enabling Kata Containers runtimes on your Kubernetes, K3s, RKE2, or K0s cluster.

TL;DR

# Install directly from the official ghcr.io OCI regitry
# update the VERSION X.YY.Z to your needs or just use the latest

export VERSION=$(curl -sSL https://api.github.com/repos/kata-containers/kata-containers/releases/latest | jq .tag_name | tr -d '"')
export CHART="oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy"

$ helm install kata-deploy "${CHART}" --version "${VERSION}"

# See everything you can configure
$ helm show values "${CHART}" --version "${VERSION}"

Prerequisites

• Kubernetes ≥ v1.22 – v1.22 is the first release where the CRI v1 API became the default and RuntimeClass left alpha. The chart depends on those stable interfaces; earlier clusters need feature‑gates or CRI shims that are out of scope.

• Kata Release 3.12 - v3.12.0 introduced publishing the helm-chart on the release page for easier consumption, since v3.8.0 we shipped the helm-chart via source code in the kata-containers Github repository.

• CRI‑compatible runtime (containerd or CRI‑O). If one wants to use the multiInstallSuffix feature one needs at least containerd-2.0 which supports drop-in config files

• Nodes must allow loading kernel modules and installing Kata artifacts (the chart runs privileged containers to do so)

Installing Helm

If Helm is not yet on your workstation or CI runner, install Helm v3 (v3.9 or newer recommended):

# Quick one‑liner (Linux/macOS)
$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Or via your package manager
$ sudo apt-get install helm        # Debian/Ubuntu
$ brew install helm                # Homebrew on macOS / Linuxbrew

Verify the installation:

$ helm version

Installing the Chart

Before attempting installing the chart one may first consult the table below Configuration Reference for all the default values. Some default values may not fit all use-cases so update as needed. A prime example may be the k8sDistribution which per default is set to k8s.

To see which chart versions are available either use the CLI

$ helm show chart oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy

or visit kata-deploy-charts

If one wants to wait until the Helm chart has deployed every object in the chart one can use --wait --timeout 10m --atomic. If the timeout expires or anything fails, Helm rolls the release back to its previous state.

$ helm install kata-deploy \        # release name
  --namespace kube-system \         # recommended namespace
  --wait --timeout 10m --atomic \
  "${CHART}" --version  "${VERSION}"

If one does not want to wait for the object via Helm or one wants rather use kubectl use Helm like this:

$ helm install kata-deploy \                         # release name
  --namespace kube-system \                          # recommended namespace
  "${CHART}" --version  "${VERSION}"

Updating Settings

Forgot to enable an option? Re‑use the values already on the cluster and only mutate what you need:

# List existing releases
$ helm ls -A

# Upgrade in‑place, keeping everything else the same
$ helm upgrade kata-deploy -n kube-system \
  --reuse-values \
  --set env.defaultShim=qemu-runtime-rs \
  "${CHART}" --version  "${VERSION}"

Uninstalling

$ helm uninstall kata-deploy -n kube-system

During uninstall, Helm will report that some resources were kept due to the resource policy (ServiceAccount, ClusterRole, ClusterRoleBinding). This is normal. A post-delete hook Job runs after uninstall and removes those resources so no cluster-wide RBAC is left behind.

Configuration Reference

All values can be overridden with --set key=value or a custom -f myvalues.yaml.

Key	Description	Default
imagePullPolicy	Set the DaemonSet pull policy	Always
imagePullSecrets	Enable pulling from a private registry via pull secret	""
image.reference	Fully qualified image reference	quay.io/kata-containers/kata-deploy
image.tag	Tag of the image reference	""
k8sDistribution	Set the k8s distribution to use: k8s, k0s, k3s, rke2, microk8s	k8s
nodeSelector	Node labels for pod assignment. Allows restricting deployment to specific nodes	{}
runtimeClasses.enabled	Enable Helm-managed runtimeClass creation (recommended)	true
runtimeClasses.createDefault	Create a default runtimeClass alias for the default shim	false
runtimeClasses.defaultName	Name for the default runtimeClass	kata
env.debug	Enable debugging in the configuration.toml	false
env.shims	List of shims to deploy	clh cloud-hypervisor dragonball fc qemu qemu-coco-dev qemu-coco-dev-runtime-rs qemu-runtime-rs qemu-se-runtime-rs qemu-snp qemu-tdx stratovirt qemu-nvidia-gpu qemu-nvidia-gpu-snp qemu-nvidia-gpu-tdx qemu-cca
env.shims_x86_64	List of shims to deploy for x86_64 (if set, overrides shims)	""
env.shims_aarch64	List of shims to deploy for aarch64 (if set, overrides shims)	""
env.shims_s390x	List of shims to deploy for s390x (if set, overrides shims)	""
env.shims_ppc64le	List of shims to deploy for ppc64le (if set, overrides shims)	""
env.defaultShim	The default shim to use if none specified	qemu
env.defaultShim_x86_64	The default shim to use if none specified for x86_64 (if set, overrides defaultShim)	""
env.defaultShim_aarch64	The default shim to use if none specified for aarch64 (if set, overrides defaultShim)	""
env.defaultShim_s390x	The default shim to use if none specified for s390x (if set, overrides defaultShim)	""
env.defaultShim_ppc64le	The default shim to use if none specified for ppc64le (if set, overrides defaultShim)	""
env.createRuntimeClasses	DEPRECATED - Use runtimeClasses.enabled instead. Script-based runtimeClass creation	false
env.createDefaultRuntimeClass	DEPRECATED - Use runtimeClasses.createDefault instead	false
env.allowedHypervisorAnnotations	Enable the provided annotations to be enabled when launching a Container or Pod, per default the annotations are disabled	""
env.snapshotterHandlerMapping	Provide the snapshotter handler for each shim	""
env.snapshotterHandlerMapping_x86_64	Provide the snapshotter handler for each shim for x86_64 (if set, overrides snapshotterHandlerMapping)	""
env.snapshotterHandlerMapping_aarch64	Provide the snapshotter handler for each shim for aarch64 (if set, overrides snapshotterHandlerMapping)	""
env.snapshotterHandlerMapping_s390x	Provide the snapshotter handler for each shim for s390x (if set, overrides snapshotterHandlerMapping)	""
env.snapshotterHandlerMapping_ppc64le	Provide the snapshotter handler for each shim for ppc64le (if set, overrides snapshotterHandlerMapping)	""
evn.agentHttpsProxy	HTTPS_PROXY=...	""
env.agentHttpProxy	specifies a list of addresses that should bypass a configured proxy server	""
env.pullTypeMapping	Type of container image pulling, examples are guest-pull or default	""
env.pullTypeMapping_x86_64	Type of container image pulling for x86_64 (if set, overrides pullTypeMapping)	""
env.pullTypeMapping_aarch64	Type of container image pulling for aarch64 (if set, overrides pullTypeMapping)	""
env.pullTypeMapping_s390x	Type of container image pulling for s390x (if set, overrides pullTypeMapping)	""
env.pullTypeMapping_ppc64le	Type of container image pulling for ppc64le (if set, overrides pullTypeMapping)	""
env.installationPrefix	Prefix where to install the Kata artifacts	/opt/kata
env.hostOS	Provide host-OS setting, e.g. cbl-mariner to do additional configurations	""
env.multiInstallSuffix	Enable multiple Kata installation on the same node with suffix e.g. /opt/kata-PR12232	""
env._experimentalSetupSnapshotter	Deploys (nydus) and/or sets up (erofs, nydus) the snapshotter(s) specified as the value (supports multiple snapshotters, separated by commas; e.g., nydus,erofs)	""
env._experimentalForceGuestPull	Enables experimental_force_guest_pull for the shim(s) specified as the value (supports multiple shims, separated by commas; e.g., qemu-tdx,qemu-snp)	""
env._experimentalForceGuestPull_x86_64	Enables experimental_force_guest_pull for the shim(s) specified as the value for x86_64 (if set, overrides _experimentalForceGuestPull)	""
env._experimentalForceGuestPull_aarch64	Enables experimental_force_guest_pull for the shim(s) specified as the value for aarch64 (if set, overrides _experimentalForceGuestPull)	""
env._experimentalForceGuestPull_s390x	Enables experimental_force_guest_pull for the shim(s) specified as the value for s390x (if set, overrides _experimentalForceGuestPull)	""
env._experimentalForceGuestPull_ppc64le	Enables experimental_force_guest_pull for the shim(s) specified as the value for ppc64le (if set, overrides _experimentalForceGuestPull)	""

Structured Configuration

NEW: Starting with Kata Containers v3.23.0, a new structured configuration format is available for configuring shims. This provides better type safety, clearer organization, and per-shim configuration options.

Migration from Legacy Format

The legacy env.* configuration format is deprecated and will be removed in 2 releases. Users are encouraged to migrate to the new structured format.

Deprecated fields (will be removed in 2 releases):

• env.shims, env.shims_x86_64, env.shims_aarch64, env.shims_s390x, env.shims_ppc64le
• env.defaultShim, env.defaultShim_x86_64, env.defaultShim_aarch64, env.defaultShim_s390x, env.defaultShim_ppc64le
• env.allowedHypervisorAnnotations
• env.snapshotterHandlerMapping, env.snapshotterHandlerMapping_x86_64, etc.
• env.pullTypeMapping, env.pullTypeMapping_x86_64, etc.
• env.agentHttpsProxy, env.agentNoProxy
• env._experimentalSetupSnapshotter
• env._experimentalForceGuestPull, env._experimentalForceGuestPull_x86_64, etc.
• env.debug

New Structured Format

The new format uses a shims section where each shim can be configured individually:

# Enable debug mode globally
debug: false

# Configure snapshotter setup
snapshotter:
  setup: []  # ["nydus", "erofs"] or []

# Configure shims
shims:
  # Disable all shims at once (useful when enabling only specific shims or custom runtimes)
  disableAll: false

  qemu:
    enabled: true
    supportedArches:
      - amd64
      - arm64
      - s390x
      - ppc64le
    allowedHypervisorAnnotations: []
    containerd:
      snapshotter: ""
  qemu-snp:
    enabled: true
    supportedArches:
      - amd64
    allowedHypervisorAnnotations: []
    containerd:
      snapshotter: nydus
      forceGuestPull: false
    crio:
      guestPull: true
    agent:
      httpsProxy: ""
      noProxy: ""
    # Optional: set runtimeClass.nodeSelector to pin TEE to specific nodes (always applied). If unset, NFD TEE labels are auto-injected when NFD is detected.

# Default shim per architecture
defaultShim:
  amd64: qemu
  arm64: qemu
  s390x: qemu
  ppc64le: qemu

Key Benefits

1. Per-shim configuration: Each shim can have its own settings for snapshotter, guest pull, agent proxy, etc.
2. Architecture-aware: Shims declare which architectures they support
3. Type safety: Structured format reduces configuration errors
4. Easy to use: All shims are enabled by default in values.yaml, so you can use the chart directly without modification
5. Disable all at once: Use shims.disableAll: true to disable all standard shims, useful when enabling only specific shims or using custom runtimes only

Example: Enable qemu shim with new format

shims:
  qemu:
    enabled: true
    supportedArches:
      - amd64
      - arm64

defaultShim:
  amd64: qemu
  arm64: qemu

Backward Compatibility

The chart maintains full backward compatibility with the legacy env.* format. If legacy values are set, they take precedence over the new structured format. This allows for gradual migration.

Default Configuration

The default values.yaml file has all shims enabled by default, making it easy to use the chart directly without modification:

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION

This includes all available Kata Containers shims:

• Standard shims: qemu, qemu-runtime-rs, clh, cloud-hypervisor, dragonball, fc
• TEE shims: qemu-snp, qemu-snp-runtime-rs, qemu-tdx, qemu-tdx-runtime-rs, qemu-se, qemu-se-runtime-rs, qemu-cca, qemu-coco-dev, qemu-coco-dev-runtime-rs
• NVIDIA GPU shims: qemu-nvidia-gpu, qemu-nvidia-gpu-snp, qemu-nvidia-gpu-tdx
• Remote shims: remote (for peer-pods/cloud-api-adaptor, disabled by default)

To enable only specific shims, you can override the configuration:

# Custom values file - enable only qemu shim
shims:
  qemu:
    enabled: true
  clh:
    enabled: false
  cloud-hypervisor:
    enabled: false
  # ... disable other shims as needed

Example Values Files

For convenience, we also provide example values files that demonstrate specific use cases:

try-kata-tee.values.yaml - Trusted Execution Environment Shims

This file enables only the TEE (Trusted Execution Environment) shims for confidential computing:

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION \
  -f try-kata-tee.values.yaml

Includes:

• qemu-snp - AMD SEV-SNP (amd64)
• qemu-tdx - Intel TDX (amd64)
• qemu-se - IBM Secure Execution for Linux (SEL) (s390x)
• qemu-se-runtime-rs - IBM Secure Execution for Linux (SEL) Rust runtime (s390x)
• qemu-cca - Arm Confidential Compute Architecture (arm64)
• qemu-coco-dev - Confidential Containers development (amd64, s390x)
• qemu-coco-dev-runtime-rs - Confidential Containers development Rust runtime (amd64, s390x)

try-kata-nvidia-gpu.values.yaml - NVIDIA GPU Shims

This file enables only the NVIDIA GPU-enabled shims:

helm install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version VERSION \
  -f try-kata-nvidia-gpu.values.yaml

Includes:

• qemu-nvidia-gpu - Standard NVIDIA GPU support (amd64, arm64)
• qemu-nvidia-gpu-snp - NVIDIA GPU with AMD SEV-SNP (amd64)
• qemu-nvidia-gpu-tdx - NVIDIA GPU with Intel TDX (amd64)

Note: These example files are located in the chart directory. When installing from the OCI registry, you'll need to download them separately or clone the repository to access them.

RuntimeClass Node Selectors for TEE Shims

Manual configuration: Any nodeSelector you set under shims.<shim>.runtimeClass.nodeSelector is always applied to that shim's RuntimeClass, whether or not NFD is present. Use this when you want to pin TEE workloads to specific nodes (e.g. without NFD, or with custom labels).

Auto-inject when NFD is present: If you do not set a runtimeClass.nodeSelector for a TEE shim, the chart can automatically inject NFD-based labels when NFD is detected in the cluster (deployed by this chart with node-feature-discovery.enabled=true or found externally):

• AMD SEV-SNP shims: amd.feature.node.kubernetes.io/snp: "true"
• Intel TDX shims: intel.feature.node.kubernetes.io/tdx: "true"
• IBM Secure Execution for Linux (SEL) shims (s390x): feature.node.kubernetes.io/cpu-security.se.enabled: "true"

The chart uses Helm's lookup function to detect NFD (by looking for the node-feature-discovery-worker DaemonSet). Auto-inject only runs when NFD is detected and no manual runtimeClass.nodeSelector is set for that shim.

Note: NFD detection requires cluster access. During helm template (dry-run without a cluster), external NFD is not seen, so auto-injected labels are not added. Manual runtimeClass.nodeSelector values are still applied in all cases.

RuntimeClass Management

NEW: Starting with Kata Containers v3.23.0, runtimeClasses are managed by Helm by default, providing better lifecycle management and integration.

Features:

• Automatic Creation: runtimeClasses are automatically created for all configured shims
• Lifecycle Management: Helm manages creation, updates, and deletion of runtimeClasses

Configuration:

runtimeClasses:
  enabled: true                  # Enable Helm-managed `runtimeClasses` (default)
  createDefault: false           # Create a default "kata" `runtimeClass`
  defaultName: "kata"            # Name for the default `runtimeClass`

When runtimeClasses.enabled: true (default), the Helm chart creates runtimeClass resources for all enabled shims (either from the new structured shims configuration or from the legacy env.shims format).

The kata-deploy script will no longer create runtimeClasses (env.createRuntimeClasses defaults to "false").

Example: only qemu shim and debug enabled

Use shims.disableAll=true to disable all shims at once, then enable only the ones you need:

# Using --set flags (disable all, then enable qemu)
$ helm install kata-deploy \
  --set shims.disableAll=true \
  --set shims.qemu.enabled=true \
  --set debug=true \
  "${CHART}" --version  "${VERSION}"

Or use a custom values file:

# custom-values.yaml
debug: true
shims:
  disableAll: true
  qemu:
    enabled: true

$ helm install kata-deploy \
  -f custom-values.yaml \
  "${CHART}" --version  "${VERSION}"

Example: Deploy only to specific nodes using nodeSelector

# First, label the nodes where you want kata-containers to be installed
$ kubectl label nodes worker-node-1 kata-containers=enabled
$ kubectl label nodes worker-node-2 kata-containers=enabled

# Then install the chart with `nodeSelector`
$ helm install kata-deploy \
  --set nodeSelector.kata-containers="enabled" \
  "${CHART}" --version  "${VERSION}"

You can also use a values file:

# values.yaml
nodeSelector:
  kata-containers: "enabled"
  node-type: "worker"

$ helm install kata-deploy -f values.yaml "${CHART}" --version "${VERSION}"

Example: Multiple Kata installations on the same node

For debugging, testing and other use-case it is possible to deploy multiple versions of Kata on the very same node. All the needed artifacts are getting the mulitInstallSuffix appended to distinguish each installation. BEWARE that one needs at least containerd-2.0 since this version has drop-in conf support which is a prerequisite for the mulitInstallSuffix to work properly.

$ helm install kata-deploy-cicd       \
  -n kata-deploy-cicd                 \
  --set env.multiInstallSuffix=cicd   \
  --set env.debug=true                \
  "${CHART}" --version  "${VERSION}"

Note: runtimeClasses are automatically created by Helm (via runtimeClasses.enabled=true, which is the default).

Now verify the installation by examining the runtimeClasses:

$ kubectl get runtimeClasses
NAME                            HANDLER                         AGE
kata-clh-cicd                   kata-clh-cicd                   77s
kata-cloud-hypervisor-cicd      kata-cloud-hypervisor-cicd      77s
kata-dragonball-cicd            kata-dragonball-cicd            77s
kata-fc-cicd                    kata-fc-cicd                    77s
kata-qemu-cicd                  kata-qemu-cicd                  77s
kata-qemu-coco-dev-cicd         kata-qemu-coco-dev-cicd         77s
kata-qemu-nvidia-gpu-cicd       kata-qemu-nvidia-gpu-cicd       77s
kata-qemu-nvidia-gpu-snp-cicd   kata-qemu-nvidia-gpu-snp-cicd   77s
kata-qemu-nvidia-gpu-tdx-cicd   kata-qemu-nvidia-gpu-tdx-cicd   76s
kata-qemu-runtime-rs-cicd       kata-qemu-runtime-rs-cicd       77s
kata-qemu-se-runtime-rs-cicd    kata-qemu-se-runtime-rs-cicd    77s
kata-qemu-snp-cicd              kata-qemu-snp-cicd              77s
kata-qemu-tdx-cicd              kata-qemu-tdx-cicd              77s
kata-stratovirt-cicd            kata-stratovirt-cicd            77s

Customizing Configuration with Drop-in Files

When kata-deploy installs Kata Containers, the base configuration files should not be modified directly. Instead, use drop-in configuration files to customize settings. This approach ensures your customizations survive kata-deploy upgrades.

How Drop-in Files Work

The Kata runtime reads the base configuration file and then applies any .toml files found in the config.d/ directory alongside it. Files are processed in alphabetical order, with later files overriding earlier settings.

Creating Custom Drop-in Files

To add custom settings, create a .toml file in the appropriate config.d/ directory. Use a numeric prefix to control the order of application.

Reserved prefixes (used by kata-deploy):

• 10-*: Core kata-deploy settings
• 20-*: Debug settings
• 30-*: Kernel parameters

Recommended prefixes for custom settings: 50-89

Example: Adding Custom Kernel Parameters

# SSH into the node or use kubectl exec
sudo mkdir -p /opt/kata/share/defaults/kata-containers/runtimes/qemu/config.d/
sudo cat > /opt/kata/share/defaults/kata-containers/runtimes/qemu/config.d/50-custom.toml << 'EOF'
[hypervisor.qemu]
kernel_params = "my_param=value"
EOF

Example: Changing Default Memory Size

sudo cat > /opt/kata/share/defaults/kata-containers/runtimes/qemu/config.d/50-memory.toml << 'EOF'
[hypervisor.qemu]
default_memory = 4096
EOF

Custom Runtimes

For more complex customizations, you can define custom runtimes in your Helm values. Custom runtimes create isolated configuration directories with their own drop-in files:

customRuntimes:
  enabled: true
  runtimes:
    - handler: kata-custom
      baseConfig: qemu
      dropInFile: /path/to/your/config.toml

This creates a new Runtime Class kata-custom that extends the qemu configuration with your custom settings.