Commit e8bb619b1b ("runtime: do not require OVMF for qemu-nvidia-cpu
on arm64") dropped the generic OVMF/AAVMF firmware inheritance for the
Go qemu-nvidia-cpu runtime, but assumed runtime-rs already did the right
thing by default. It didn't: the runtime-rs profile still pointed at the
generic @FIRMWAREPATH@ / @FIRMWAREVOLUMEPATH@ and still pulled in the
ovmf component on aarch64.
Mirror the fix on the runtime-rs side by introducing dedicated empty
FIRMWAREPATH_NV_CPU / FIRMWAREVOLUMEPATH_NV_CPU variables, wiring them
into the qemu-nvidia-cpu-runtime-rs configuration, and dropping ovmf
from its aarch64 shim components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Make qemu-nvidia-cpu-runtime-rs the default shim on amd64 and arm64 for
the kata-deploy NVIDIA CPU profile, so installs from
try-kata-nvidia-cpu.values.yaml select the Rust runtime by default
instead of the Go qemu-nvidia-cpu shim.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
EROFS fs-verity depends on the backing filesystem supporting fs-verity,
which cannot be guaranteed on an arbitrary node. Rather than gate the
NVIDIA CPU profile on a host feature we do not control, disable fs-verity
so it works on the widest possible range of setups; we take that hit
deliberately to keep the profile generic.
Move this out of the test harness and into the kata-deploy NVIDIA CPU
profile: try-kata-nvidia-cpu.values.yaml now ships a containerd erofs
snapshotter drop-in that disables fs-verity and pins the memory-backed
default_size, and selects memory-backed rw layers with dm-verity for the
lower (image) layers. Because containerd.userDropIn is loaded after
kata-deploy's generated config, it overrides the built-in enable_fsverity
default.
The CI helm helper no longer special-cases NVIDIA CPU to inject
enable_fsverity=false. Instead it honors an erofs snapshotter drop-in
already provided by the base values file and only synthesizes the default
default_size drop-in when the profile does not provide its own.
Note that this drop-in targets containerd's built-in EROFS snapshotter,
available only on containerd >= 2.2.0 (config version 3, conf.d auto-import),
which is the same minimum kata-deploy already enforces for the EROFS
snapshotter.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Resolve kata-monitor log level from monitor.logLevel, the chart-wide
logLevel, and debug:true, matching the precedence used by kata-deploy.
Fixes#13382
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Derive the runtime endpoint from k8sDistribution and mount the matching
host socket directory so kata-monitor can reach containerd on
non-vanilla k8s deployments.
Fixes#13378
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Use zstd level 10 (via kata_tar_zstd) when creating static build
tarballs so CI artifacts stay under GitHub's 2GB limit without
paying the build-time cost of maximum compression.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Reuse the kubelet runtimeRequestTimeout validation for guest-pull
configurations.
Guest pull can also keep kubelet waiting during CreateContainer. Warn
when kata-deploy can see that runtimeRequestTimeout is below the same
suggested timeout used for EROFS layer conversion.
Detect both the regular guest-pull mapping and the experimental force
guest-pull configuration so nodes get the warning for either path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Extend the kata-deploy host-check stage with EROFS-specific
prerequisite validation.
Check that containerd supports the EROFS snapshotter, required
kernel features are available either as loaded modules or built-in
kernel configuration, and the host erofs-utils version satisfies the
minimum required by kata-deploy's configured mkfs options. Also warn
when kubelet runtimeRequestTimeout appears too low for EROFS layer
conversion, which can run during CreateContainer.
Read kubelet runtimeRequestTimeout from kubelet /configz through the
apiserver node proxy, and add the corresponding RBAC permission. Keep
fs-verity validation warning-only because user drop-ins may override
kata-deploy's default enable_fsverity setting and kata-deploy does
not yet validate the backing filesystem's fs-verity feature.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Switch the per-architecture default shim from the Go runtime ("qemu") to the
Rust runtime ("qemu-runtime-rs") wherever a runtime-rs build exists, and
align CI/test fallbacks with the new default.
Document the defaultShim values and Go runtime deprecation in the Helm and
config migration guides using mkdocs admonitions.
Fixes: kata-containers#9546
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Use dedicated empty firmware makefile variables so the NVIDIA CPU
runtime does not inherit generic OVMF/AAVMF firmware paths.
runtime-rs already sets the correct option by default, so, no changed
needed on that front.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The CI agent tarball is GNU-linked (USE_DEVMAPPER=yes) and does not run
reliably on Alpine/musl initrd.
Use Ubuntu noble for x86_64 and aarch64 vanilla initrd, matching the
other initrd variants, with kata-agent as /sbin/init.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Extend the post-delete RBAC cleanup Job to remove job-mode dispatcher
identities (ServiceAccount, Role, RoleBinding, ClusterRole, and
ClusterRoleBinding) so reinstalls are not blocked by leftovers.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The job dispatcher now GETs each in-flight Job by name when tracking
progress, so it no longer needs the list verb on batch/jobs. Trim the
dispatcher Role to match.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a try-kata-nvidia-cpu.values.yaml example for the CPU-only
NVIDIA runtime classes and make the Kubernetes CI Helm helper use it
for qemu-nvidia-cpu handlers.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
dmverity_mode=auto can preserve existing EROFS layers without
dm-verity metadata, which lets mixed-mode nodes keep running with
partially unprotected image layers.
Use dmverity_mode=on when kata-deploy enables EROFS dm-verity so
missing metadata is reported instead of silently accepted.
Also tighten the Kubernetes probe. Instead of checking for at least one
dm-verity line and one EROFS dm mount in dmesg, derive the expected
workload layer count from the root overlay lowerdirs and require matching
kata-verity devices and dm-backed EROFS mounts.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Make qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs first-class kata-deploy
citizens: add them to the shim allow-lists and per-arch defaults in the
kata-deploy binary (amd64 + arm64), expose them as shims in the Helm
values and give them a RuntimeClass with a lightweight pod overhead
(320Mi/250m) - they carry no GPU, so the GPU 10Gi budget doesn't apply.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Teach the shim-v2 build to feed the "nvidia" base root hash into the Go
runtime's KERNELVERITYPARAMS_NV_BASE - its KERNELVERITYPARAMS_NV is the
monolithic nvidia-gpu hash - so qemu-nvidia-cpu boots the base image
verity-backed just like qemu-nvidia-cpu-runtime-rs already does.
List the new qemu-nvidia-cpu / qemu-nvidia-cpu-runtime-rs classes in
shim-components.json so the build ships them: both boot the nvidia base
image with the nvidia kernel and no GPU extension, dropping OVMF on x86_64
and keeping it on aarch64 (which boots via UEFI).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The proposal told us to move it under docs/design/ once it was accepted
and implemented. The systemd base landed on main and the NVIDIA split
(nvidia base + gpu extension) is what this series wires up, so it's no
longer a proposal - promote it to docs/design/composable-vm-images.md,
flip the status accordingly and repoint the in-tree references (agent,
osbuilder and kata-deploy) at the new path.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add the rootfs-image-nvidia-{base,gpu-extension} build targets to the
local-build Makefile and the nvgpu bundle.
Build the base/extension rootfs images in the amd64/arm64 static tarball
workflows, and point the runtime-rs NVIDIA shim components at the
composable base + gpu extension (+ coco extension for the confidential
classes). The Go shim components keep the monolithic nvidia-gpu /
nvidia-gpu-confidential images.
Ship cryptsetup in the CoCo guest-components extension (and bundle
libnvat's non-glibc dependency closure next to the nvidia attester): the
chiseled nvidia base image carries only veritysetup, so the coco-extension
manifest's usr/sbin PATH entry needs a cryptsetup to resolve for NVIDIA
confidential encrypted-storage.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Flip the runtime-rs NVIDIA GPU configs (plain/tdx/snp) from the monolithic
image to the driver-agnostic nvidia base boot image plus a cold-plugged,
driver-versioned gpu extension mounted at /run/kata-extensions/gpu.
Confidential runtime-rs configs additionally cold-plug the CoCo
guest-components extension and enforce the nvidia base dm-verity hash.
The Go runtime keeps booting the monolithic nvidia-gpu / nvidia-gpu-confidential
images, mirroring the CoCo split where runtime-rs is the first-class citizen and
Go stays on the monolithic layout.
Add the IMAGE{NAME,PATH}_NV_{BASE,EXTENSION} and NVIDIAGPUEXTENSIONVERITYPARAMS
make vars (runtime-rs only), and read the nvidia base + gpu-extension root
hashes into runtime-rs's shim-v2 opts while the Go opts keep reading the
monolithic nvidia-gpu / nvidia-gpu-confidential hashes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Carve the monolithic NVIDIA GPU rootfs into a driver-agnostic
nvidia base image (NVRC init + agent + base libs + in-tree modules)
and a driver-versioned gpu extension (GPU userspace, configs, firmware,
NVIDIA modules) laid out for /run/kata-extensions/gpu. Both halves are
produced from the same chiseled tree via a partition-after step, so
the monolith build stays byte-identical and continues to ship.
The shared stage-one (driver install) is reused across the monolith,
base and extension variants; nvidia base is cached without the driver/ctk
versions so one base image can back multiple driver extensions.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
We've been carrying a full, 1200+ line busybox .config for the NVIDIA
images. That's a pain to review and to keep in sync with a busybox
bump, as basically every default that changes upstream ends up as noise
in our file.
Switch to shipping only the handful of options we actually care about in
a small fragment and let busybox's own Kconfig fill in the rest at build
time: start from allnoconfig, drop the baseline entries for whatever the
fragment overrides (Kconfig keeps the first value it reads), append the
fragment and run silentoldconfig. This reproduces the exact same
.config we had frozen, module loading included, so there's no functional
change here.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The InfiniBand and Mellanox mlx5 options used to be carried (as modules)
in the NVIDIA GPU config fragments. They're now enabled built-in by the
common mlx5.conf fragment, which every kernel build already picks up, so
keeping them here is redundant - and the =m entries here only end up
fighting the built-in =y ones from the common fragment.
Drop the HGX/DGX block from both the x86_64 and arm64 GPU fragments and
bump kata_config_version so the kernel is rebuilt.
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Replace the kustomize-based Trustee deployment with the upstream Trustee
Helm chart (deployment/helm-chart) for every platform, including TDX and
s390x, which the previous iteration still left on kustomize. The chart
deploys KBS, AS and RVPS as three separate images, so versions.yaml pins
image_kbs/image_as/image_rvps (from the upstream staged-images registry)
alongside the trustee revision, and bumps trustee to a build that carries
the IBM SE material mounting support and the SHA-512 report-data change.
guest-components is bumped in lockstep so IBM SE attestation agrees on the
SHA-512 runtime-data digest.
KBS runs with authorization_mode = "AuthenticatedAuthorization", so admin
requests from kbs-client carry the bearer token minted by the chart's
bootstrap job instead of relying on an anonymous, allow-all admin API. As
reused CI runners may still carry a kbs-client from an older Trustee that
predates the --admin-token-file flag, the client is rebuilt from the pinned
sources unless the installed one already understands it.
On the Intel TDX CI runner only the Attestation Service reaches the public
Intel DCAP collateral endpoint, so it is the sole component that needs the
corporate proxy. We set https_proxy on the AS alone: the KBS <-> AS <-> RVPS
traffic is plain HTTP/gRPC and in-cluster, and routing it through the proxy
breaks those connections. The Intel DCAP and NVIDIA verifiers are configured
by the chart out of the box, so TDX and SNP need no extra verifier knobs.
For IBM SE (s390x) the SE verifier now runs inside the gRPC Attestation
Service, so the SE materials are mounted on the AS Pod (not KBS) through the
chart's as.ibmse.* knobs, which create a node-local PV/PVC. The AS runs as a
non-root user with CAP_DAC_OVERRIDE dropped, hence fsGroup=1000 and a
group-readable hdr.bin so it can read the mounted materials.
The chart's default CPU requests reserve more of the small single-node CI
runners than the old single-Pod deployment did, starving test workloads that
ask for a couple of Guaranteed CPUs. Trustee requests are shrunk (limits kept
generous) so those Pods schedule again.
Debug logging is enabled on all components (log_level=debug) to ease
triaging attestation / resource-retrieval failures in CI, and the release is
rolled out with 'helm upgrade --install' so re-runs on long-lived CI
clusters reconcile in place.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Rust 1.95's clippy introduces new `collapsible_match` lints for `if`
blocks nested inside match arms that can be expressed as match guards.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
When 1.97 has been released, in compliance with our toolchain guidance
we should bump to rust 1.95
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Merge NFD and user nodeSelectorTerms with merge/concat.
Ignore the name key in podLabels, document podAnnotations
and extend bats coverage for annotations.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Expose podLabels, podAnnotations and affinity in values.yaml so
operators can apply more granular scheduling controls on the
kata-deploy DaemonSet alongside the existing nodeSelector and
tolerations knobs.
Defaults are empty, so existing deployments render the same manifest.
When NFD is enabled, the chart merges the hard-coded virtualization
nodeAffinity with any user provided affinity.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
The kata-deploy payload downloads the nydus-snapshotter host binaries
(containerd-nydus-grpc, nydus-overlayfs) from the upstream per-arch
release asset, which is glibc dynamically linked and fails to exec on
musl-only hosts that have no glibc dynamic loader.
nydus-snapshotter also publishes a statically linked release asset.
Honour the existing STATIC_RUNTIME=yes umbrella flag (which already
builds the kata Go host binaries static) in the nydus component build so
that, with a single flag, the whole host-binary set of the payload is
static and runs on a musl-only host. STATIC_RUNTIME=yes maps to a new
STATIC_NYDUS_SNAPSHOTTER build-arg on the nydus downloader stage of
Dockerfile.components, which selects the linux-static asset.
The static nydus asset is published for amd64 only, so the build fails
fast on other architectures when the flag is set. The default build is
unchanged.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Add an opt-in STATIC_RUNTIME=yes knob to the shim-v2 static build. When
set, the Go host binaries are built with the runtime STATIC=yes profile
(CGO_ENABLED=0, no PIE), producing fully static, libc-agnostic binaries
in the kata-deploy payload.
This lets a payload image be produced that runs on musl-only hosts which
have no glibc dynamic loader, where the default glibc-linked binaries
fail to exec. The default build is unchanged.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Generate etc/kata-extensions/components.toml inside the CoCo extension rootfs
before building the image, so the manifest becomes part of the
dm-verity-measured erofs image.
The manifest declares the extension's path-only components (ocicrypt config,
pause bundle) and the launchable processes (attestation-agent,
confidential-data-hub, api-server-rest) with paths relative to the extension
mount point (/run/kata-extensions/coco) and `${var}` placeholders resolved
by kata-agent at runtime. This is the data the agent consumes to launch
the CoCo guest components without per-bundle code changes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add install_image_coco_extension() to kata-deploy-binaries.sh which:
- Unpacks the CoCo guest components and pause image tarballs into a
temporary rootfs directory (under the repo root so Docker-in-Docker
volume mounts resolve correctly)
- Calls image_builder.sh with USE_DOCKER=1, FS_TYPE=erofs,
MEASURED_ROOTFS=yes, SKIP_DAX_HEADER=yes, and SKIP_ROOTFS_CHECK=yes
to produce kata-containers-coco-extension.img + root_hash_coco-extension.txt
Add the rootfs-image-coco-extension-tarball Makefile target with
dependencies on pause-image-tarball and coco-guest-components-tarball.
The standard confidential image keeps pause-image and
coco-guest-components for now so it remains a usable standalone
monolithic CoCo image during the transition to base + extension.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
During the transition to composable (base + extension) CoCo images, run the
two runtimes on different image layouts so CI exercises both paths via
the existing qemu-{snp,tdx,coco-dev} (Go) and -runtime-rs (Rust) shims:
- runtime-rs (Rust) CoCo templates use the standard base image plus a
CoCo extension extra_image:
image = "@IMAGEPATH@" (was @IMAGECONFIDENTIALPATH@)
[[hypervisor.qemu.guest_extension_images]]
name = "coco"
path = "@COCOIMAGEPATH@"
verity_params = "@COCOVERITYPARAMS@"
COCOIMAGENAME/COCOIMAGEPATH/COCOVERITYPARAMS are added to the
runtime-rs Makefile only.
- runtime (Go) CoCo templates keep the monolithic confidential image
(@IMAGECONFIDENTIALPATH@) and pull rootfs-image-confidential.
shim-components.json reflects this: Go CoCo shims depend on
rootfs-image-confidential while the runtime-rs CoCo shims depend on
rootfs-image + rootfs-image-coco-extension.
shim-v2/build.sh feeds each runtime its own dm-verity params: Rust gets
the measured base image hash (@KERNELVERITYPARAMS@) plus the extension hash
(@COCOVERITYPARAMS@); Go gets the monolithic confidential image hash
(@KERNELVERITYPARAMS@). This is wired per make invocation so the same
@KERNELVERITYPARAMS@ placeholder resolves correctly for each.
Once the split path is validated we can flip the Go templates too and
drop the monolithic confidential image.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Since the switch to per-component tarballs, versions.yaml was no longer
installed on host nodes by kata-deploy (DaemonSet/Helm). Previously it
was bundled into the merged kata-static.tar.zst; now it must be shipped
explicitly.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
kata-deploy installs the Kata runtime asynchronously (DaemonSet or
per-node Jobs), labelling the node katacontainers.io/kata-runtime=true
as its final step. On a freshly provisioned node, a pod that requests a
Kata RuntimeClass can be scheduled before the runtime binaries exist and
fail to start. There is currently no mechanism to gate scheduling on the
runtime actually being ready.
Add a STARTUP_TAINTS option (helm value: startupTaints) listing taints
to remove once install completes and the node is labelled. Operators can
provision nodes with a startup taint (e.g. a Karpenter NodePool
startupTaints entry) that keeps Kata workloads off the node until
kata-deploy removes it as the very last install action, strictly after
artifacts are installed, the CRI runtime is configured and restarted,
and the node is labelled kata-capable.
Each entry is a bare key (any effect) or key:effect. Removal is
best-effort and idempotent: an absent taint is skipped, and a failure to
remove leaves the taint in place (workloads stay gated) rather than
admitting them prematurely. The default is empty, so the behaviour is
opt-in and a no-op for existing deployments. The taint is wired through
the shared commonEnv helper, so both daemonset and job deployment modes
honour it. No RBAC change is needed: kata-deploy already holds
nodes [get, patch].
Fixes: #13283
Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Pankaj Walke <punkwalker@gmail.com>
The deploy will read EROFS_SNAPSHOTTER_MODE and EROFS_DMVERITY from
the environment to enable dmverity_mode and enable_dmverity in the
containerd erofs snapshotter/differ config.
Add validation for the mode value and use an explicit 300s timeout
for node-readiness checks during kata-deply in github CI.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
