Add an opt-in "devkit" path to kata-deploy that wires the devkit debug guest
extension into a dedicated per-shim runtime, so operators can debug a sandbox
with a rich shell without touching their normal RuntimeClasses.
Helm gains a `devkit` value (only effective together with `debug`, since the
debug console must be enabled to reach it). When set it emits DEVKIT=true and,
for every enabled shim, an extra `kata-<shim>-devkit` RuntimeClass reusing the
shim's overhead and node selectors. The RuntimeClass template is refactored to
compute name/handler from an optional `variant`, keeping existing behavior.
In the deploy binary, DEVKIT synthesizes one kata-<shim>-devkit custom runtime
per enabled shim. Modeling it as a custom runtime reuses the whole custom
runtime install/register/cleanup machinery; the only devkit-specific piece is a
25-devkit.toml drop-in that cold-plugs the extension image
(guest_extension_images name="devkit", with verity_params read from
root_hash_devkit-extension.txt when present) and points the agent debug console
at /run/kata-extensions/devkit/bin/devkit-sh. The generic
rootfs-image-devkit-extension tarball is extracted based on the devkit flag
rather than per-shim shim-components.json membership.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a self-contained "devkit" guest extension: a minimal Ubuntu (glibc +
busybox + apt) rootfs with common debug tools prebaked (strace, ltrace,
iproute2, procps, lsof, tcpdump, pciutils, util-linux, ...), built as a
measured erofs+dm-verity image mounted at /run/kata-extensions/devkit.
The production guest rootfs is minimal (and, for some bases, shell-less), so
the agent debug console has no rich interactive shell. Rather than rebuilding
the whole rootfs with debug tooling, this optional extension can be
cold-plugged alongside any base image. At runtime the guest helper scripts
overlay a writable tmpfs on the read-only extension and chroot in, so apt and
every tool run natively against a normal root filesystem; `apt install <pkg>`
inside the debug shell pulls anything else into the overlay on demand.
busybox-static (/usr/bin/busybox.static) bootstraps the overlay/chroot from the
shell-less base (the guest's dynamic loader is not present there yet); it is
installed at a dedicated path so it never clobbers the extension's own busybox
or /bin/sh, which are needed unclobbered inside the chroot.
The rootfs is built through osbuilder (rootfs.sh, with a ROOTFS_ONLY mode that
skips the agent/systemd setup) using mmdebstrap, the same tool the base guest
rootfs uses, so the devkit needs no bespoke chroot or docker-export logic. The
resulting tree is handed to image_builder.sh (mirroring the CoCo extension): it
ships no kata-agent, kernel or driver userspace.
No runtime wiring is added here; that follows in later commits. Nothing
consumes the image yet.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
kata-ctl provides `kata-ctl exec <sandbox-id>`, the runtime-rs entry point
into a sandbox's agent debug console. It ships in the kata-tools bundle
rather than with the shim binaries, so it is not staged into the kata-deploy
image and never lands on the node - leaving runtime-rs deployments without a
first-class way to reach the debug console.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The guest kernel build derived a CROSS_COMPILE=<arch>-linux-gnu- prefix
whenever the target arch differed from the host, the static-build wrapper
switched to a per-arch `-cross-build` image (with buildx/--platform for
s390x), and the builder image installed gcc-<arch>-linux-gnu /
binutils-<arch>-linux-gnu cross toolchains.
None of this is exercised on our side: the kernel is always built on a
native-arch runner. Remove the CROSS_COMPILE plumbing, the cross-build
image/buildx handling and the cross toolchain install.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Building QEMU for a foreign arch relied on configure-hypervisor.sh
emitting --cross-prefix, build-qemu.sh passing --cc/--cross-prefix for
s390x, and the builder image installing foreign-arch -dev packages
(via dpkg multiarch/DPKG_ARCH) plus a gcc-<arch>-linux-gnu cross toolchain.
None of this is exercised on our side: QEMU is always built on a
native-arch runner. Remove the cross-prefix logic, the s390x cross
configure branch, and the DPKG_ARCH/cross-gcc plumbing from the Dockerfile
and build-base-qemu.sh.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The shared lib.sh exported CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM so the
per-component static-build scripts could switch to `-cross-build` builder
images, run `docker buildx build --platform=...` under QEMU emulation and,
for shim-v2, swap in the messense/rust-musl-cross image with a matching
cross CC.
These cross paths are not exercised on our side (all builds run on
native-arch runners), so remove them: builder images are now built with a
plain `docker build`, and shim-v2 builds runtime-rs natively without the
musl-cross image or CC override.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The kata-deploy local-build pipeline auto-enabled a cross build whenever
the host arch differed from TARGET_ARCH, wiring up docker buildx, QEMU
binfmt emulation and an s390x cross toolchain (incl. building s390-tools
genprotimg with CROSS_COMPILE) in the builder image.
None of this is exercised on our side: every release/CI build runs on a
native-arch runner, so the cross paths are dead code. Remove the
CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM/binfmt machinery and the s390x
cross toolchain from the builder image; the tarballs are always built for
the host arch now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Commit e8bb619b1b ("runtime: do not require OVMF for qemu-nvidia-cpu
on arm64") dropped the generic OVMF/AAVMF firmware inheritance for the
Go qemu-nvidia-cpu runtime, but assumed runtime-rs already did the right
thing by default. It didn't: the runtime-rs profile still pointed at the
generic @FIRMWAREPATH@ / @FIRMWAREVOLUMEPATH@ and still pulled in the
ovmf component on aarch64.
Mirror the fix on the runtime-rs side by introducing dedicated empty
FIRMWAREPATH_NV_CPU / FIRMWAREVOLUMEPATH_NV_CPU variables, wiring them
into the qemu-nvidia-cpu-runtime-rs configuration, and dropping ovmf
from its aarch64 shim components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Make qemu-nvidia-cpu-runtime-rs the default shim on amd64 and arm64 for
the kata-deploy NVIDIA CPU profile, so installs from
try-kata-nvidia-cpu.values.yaml select the Rust runtime by default
instead of the Go qemu-nvidia-cpu shim.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
EROFS fs-verity depends on the backing filesystem supporting fs-verity,
which cannot be guaranteed on an arbitrary node. Rather than gate the
NVIDIA CPU profile on a host feature we do not control, disable fs-verity
so it works on the widest possible range of setups; we take that hit
deliberately to keep the profile generic.
Move this out of the test harness and into the kata-deploy NVIDIA CPU
profile: try-kata-nvidia-cpu.values.yaml now ships a containerd erofs
snapshotter drop-in that disables fs-verity and pins the memory-backed
default_size, and selects memory-backed rw layers with dm-verity for the
lower (image) layers. Because containerd.userDropIn is loaded after
kata-deploy's generated config, it overrides the built-in enable_fsverity
default.
The CI helm helper no longer special-cases NVIDIA CPU to inject
enable_fsverity=false. Instead it honors an erofs snapshotter drop-in
already provided by the base values file and only synthesizes the default
default_size drop-in when the profile does not provide its own.
Note that this drop-in targets containerd's built-in EROFS snapshotter,
available only on containerd >= 2.2.0 (config version 3, conf.d auto-import),
which is the same minimum kata-deploy already enforces for the EROFS
snapshotter.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Drop the unfinished ARM CCA stack (qemu-cca shim, experimental QEMU/OVMF
and kernel assets, packaging targets, and CI builds) since it is not
tested or shipped in any runtime bundle.
This will be supported by the moment we have development done for real
hardware available.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Drop uscan-only fields, duplicate qemu.tag, and unused image/initrd
URLs from versions.yaml, and teach gen_versions_txt to treat
qemu.version as either a branch or tag ref.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Resolve kata-monitor log level from monitor.logLevel, the chart-wide
logLevel, and debug:true, matching the precedence used by kata-deploy.
Fixes#13382
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Derive the runtime endpoint from k8sDistribution and mount the matching
host socket directory so kata-monitor can reach containerd on
non-vanilla k8s deployments.
Fixes#13378
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Use zstd level 10 (via kata_tar_zstd) when creating static build
tarballs so CI artifacts stay under GitHub's 2GB limit without
paying the build-time cost of maximum compression.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Reuse the kubelet runtimeRequestTimeout validation for guest-pull
configurations.
Guest pull can also keep kubelet waiting during CreateContainer. Warn
when kata-deploy can see that runtimeRequestTimeout is below the same
suggested timeout used for EROFS layer conversion.
Detect both the regular guest-pull mapping and the experimental force
guest-pull configuration so nodes get the warning for either path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Extend the kata-deploy host-check stage with EROFS-specific
prerequisite validation.
Check that containerd supports the EROFS snapshotter, required
kernel features are available either as loaded modules or built-in
kernel configuration, and the host erofs-utils version satisfies the
minimum required by kata-deploy's configured mkfs options. Also warn
when kubelet runtimeRequestTimeout appears too low for EROFS layer
conversion, which can run during CreateContainer.
Read kubelet runtimeRequestTimeout from kubelet /configz through the
apiserver node proxy, and add the corresponding RBAC permission. Keep
fs-verity validation warning-only because user drop-ins may override
kata-deploy's default enable_fsverity setting and kata-deploy does
not yet validate the backing filesystem's fs-verity feature.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Switch the per-architecture default shim from the Go runtime ("qemu") to the
Rust runtime ("qemu-runtime-rs") wherever a runtime-rs build exists, and
align CI/test fallbacks with the new default.
Document the defaultShim values and Go runtime deprecation in the Helm and
config migration guides using mkdocs admonitions.
Fixes: kata-containers#9546
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Use dedicated empty firmware makefile variables so the NVIDIA CPU
runtime does not inherit generic OVMF/AAVMF firmware paths.
runtime-rs already sets the correct option by default, so, no changed
needed on that front.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The CI agent tarball is GNU-linked (USE_DEVMAPPER=yes) and does not run
reliably on Alpine/musl initrd.
Use Ubuntu noble for x86_64 and aarch64 vanilla initrd, matching the
other initrd variants, with kata-agent as /sbin/init.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Extend the post-delete RBAC cleanup Job to remove job-mode dispatcher
identities (ServiceAccount, Role, RoleBinding, ClusterRole, and
ClusterRoleBinding) so reinstalls are not blocked by leftovers.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The job dispatcher now GETs each in-flight Job by name when tracking
progress, so it no longer needs the list verb on batch/jobs. Trim the
dispatcher Role to match.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a try-kata-nvidia-cpu.values.yaml example for the CPU-only
NVIDIA runtime classes and make the Kubernetes CI Helm helper use it
for qemu-nvidia-cpu handlers.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
dmverity_mode=auto can preserve existing EROFS layers without
dm-verity metadata, which lets mixed-mode nodes keep running with
partially unprotected image layers.
Use dmverity_mode=on when kata-deploy enables EROFS dm-verity so
missing metadata is reported instead of silently accepted.
Also tighten the Kubernetes probe. Instead of checking for at least one
dm-verity line and one EROFS dm mount in dmesg, derive the expected
workload layer count from the root overlay lowerdirs and require matching
kata-verity devices and dm-backed EROFS mounts.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Make qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs first-class kata-deploy
citizens: add them to the shim allow-lists and per-arch defaults in the
kata-deploy binary (amd64 + arm64), expose them as shims in the Helm
values and give them a RuntimeClass with a lightweight pod overhead
(320Mi/250m) - they carry no GPU, so the GPU 10Gi budget doesn't apply.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Teach the shim-v2 build to feed the "nvidia" base root hash into the Go
runtime's KERNELVERITYPARAMS_NV_BASE - its KERNELVERITYPARAMS_NV is the
monolithic nvidia-gpu hash - so qemu-nvidia-cpu boots the base image
verity-backed just like qemu-nvidia-cpu-runtime-rs already does.
List the new qemu-nvidia-cpu / qemu-nvidia-cpu-runtime-rs classes in
shim-components.json so the build ships them: both boot the nvidia base
image with the nvidia kernel and no GPU extension, dropping OVMF on x86_64
and keeping it on aarch64 (which boots via UEFI).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The proposal told us to move it under docs/design/ once it was accepted
and implemented. The systemd base landed on main and the NVIDIA split
(nvidia base + gpu extension) is what this series wires up, so it's no
longer a proposal - promote it to docs/design/composable-vm-images.md,
flip the status accordingly and repoint the in-tree references (agent,
osbuilder and kata-deploy) at the new path.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add the rootfs-image-nvidia-{base,gpu-extension} build targets to the
local-build Makefile and the nvgpu bundle.
Build the base/extension rootfs images in the amd64/arm64 static tarball
workflows, and point the runtime-rs NVIDIA shim components at the
composable base + gpu extension (+ coco extension for the confidential
classes). The Go shim components keep the monolithic nvidia-gpu /
nvidia-gpu-confidential images.
Ship cryptsetup in the CoCo guest-components extension (and bundle
libnvat's non-glibc dependency closure next to the nvidia attester): the
chiseled nvidia base image carries only veritysetup, so the coco-extension
manifest's usr/sbin PATH entry needs a cryptsetup to resolve for NVIDIA
confidential encrypted-storage.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Flip the runtime-rs NVIDIA GPU configs (plain/tdx/snp) from the monolithic
image to the driver-agnostic nvidia base boot image plus a cold-plugged,
driver-versioned gpu extension mounted at /run/kata-extensions/gpu.
Confidential runtime-rs configs additionally cold-plug the CoCo
guest-components extension and enforce the nvidia base dm-verity hash.
The Go runtime keeps booting the monolithic nvidia-gpu / nvidia-gpu-confidential
images, mirroring the CoCo split where runtime-rs is the first-class citizen and
Go stays on the monolithic layout.
Add the IMAGE{NAME,PATH}_NV_{BASE,EXTENSION} and NVIDIAGPUEXTENSIONVERITYPARAMS
make vars (runtime-rs only), and read the nvidia base + gpu-extension root
hashes into runtime-rs's shim-v2 opts while the Go opts keep reading the
monolithic nvidia-gpu / nvidia-gpu-confidential hashes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Carve the monolithic NVIDIA GPU rootfs into a driver-agnostic
nvidia base image (NVRC init + agent + base libs + in-tree modules)
and a driver-versioned gpu extension (GPU userspace, configs, firmware,
NVIDIA modules) laid out for /run/kata-extensions/gpu. Both halves are
produced from the same chiseled tree via a partition-after step, so
the monolith build stays byte-identical and continues to ship.
The shared stage-one (driver install) is reused across the monolith,
base and extension variants; nvidia base is cached without the driver/ctk
versions so one base image can back multiple driver extensions.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
We've been carrying a full, 1200+ line busybox .config for the NVIDIA
images. That's a pain to review and to keep in sync with a busybox
bump, as basically every default that changes upstream ends up as noise
in our file.
Switch to shipping only the handful of options we actually care about in
a small fragment and let busybox's own Kconfig fill in the rest at build
time: start from allnoconfig, drop the baseline entries for whatever the
fragment overrides (Kconfig keeps the first value it reads), append the
fragment and run silentoldconfig. This reproduces the exact same
.config we had frozen, module loading included, so there's no functional
change here.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The InfiniBand and Mellanox mlx5 options used to be carried (as modules)
in the NVIDIA GPU config fragments. They're now enabled built-in by the
common mlx5.conf fragment, which every kernel build already picks up, so
keeping them here is redundant - and the =m entries here only end up
fighting the built-in =y ones from the common fragment.
Drop the HGX/DGX block from both the x86_64 and arm64 GPU fragments and
bump kata_config_version so the kernel is rebuilt.
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Replace the kustomize-based Trustee deployment with the upstream Trustee
Helm chart (deployment/helm-chart) for every platform, including TDX and
s390x, which the previous iteration still left on kustomize. The chart
deploys KBS, AS and RVPS as three separate images, so versions.yaml pins
image_kbs/image_as/image_rvps (from the upstream staged-images registry)
alongside the trustee revision, and bumps trustee to a build that carries
the IBM SE material mounting support and the SHA-512 report-data change.
guest-components is bumped in lockstep so IBM SE attestation agrees on the
SHA-512 runtime-data digest.
KBS runs with authorization_mode = "AuthenticatedAuthorization", so admin
requests from kbs-client carry the bearer token minted by the chart's
bootstrap job instead of relying on an anonymous, allow-all admin API. As
reused CI runners may still carry a kbs-client from an older Trustee that
predates the --admin-token-file flag, the client is rebuilt from the pinned
sources unless the installed one already understands it.
On the Intel TDX CI runner only the Attestation Service reaches the public
Intel DCAP collateral endpoint, so it is the sole component that needs the
corporate proxy. We set https_proxy on the AS alone: the KBS <-> AS <-> RVPS
traffic is plain HTTP/gRPC and in-cluster, and routing it through the proxy
breaks those connections. The Intel DCAP and NVIDIA verifiers are configured
by the chart out of the box, so TDX and SNP need no extra verifier knobs.
For IBM SE (s390x) the SE verifier now runs inside the gRPC Attestation
Service, so the SE materials are mounted on the AS Pod (not KBS) through the
chart's as.ibmse.* knobs, which create a node-local PV/PVC. The AS runs as a
non-root user with CAP_DAC_OVERRIDE dropped, hence fsGroup=1000 and a
group-readable hdr.bin so it can read the mounted materials.
The chart's default CPU requests reserve more of the small single-node CI
runners than the old single-Pod deployment did, starving test workloads that
ask for a couple of Guaranteed CPUs. Trustee requests are shrunk (limits kept
generous) so those Pods schedule again.
Debug logging is enabled on all components (log_level=debug) to ease
triaging attestation / resource-retrieval failures in CI, and the release is
rolled out with 'helm upgrade --install' so re-runs on long-lived CI
clusters reconcile in place.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Rust 1.95's clippy introduces new `collapsible_match` lints for `if`
blocks nested inside match arms that can be expressed as match guards.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
When 1.97 has been released, in compliance with our toolchain guidance
we should bump to rust 1.95
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
