get_qemu_image_name() hashed the entire tools/packaging/scripts
directory, so any packaging script change - including ones with no
effect on the QEMU build - invalidated the builder image tag and
triggered unnecessary rebuilds in CI. Hash only
configure-hypervisor.sh, the script that actually shapes the build.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude <noreply@anthropic.com>
The shellcheck pass replaced die "Invalid option" with a bare
usage-and-exit, dropping the offending option from the error output
and making CI logs harder to debug. Restore the die with the option
name, matching the behavior on main.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude <noreply@anthropic.com>
For x86_64 on QEMU < 10.1.0 the option list carried both
--disable-avx512bw and --enable-avx512bw under identical conditions
(the enable wins by ordering, so AVX512BW was built in). Remove the
disable so the speed-tagged enable is expressed once, unambiguously.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude <noreply@anthropic.com>
The option list carried both --disable-vmdk and --enable-vmdk (the
latter wins by ordering, so VMDK was enabled). Remove the disable so
the EROFS requirement is expressed once, unambiguously.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude <noreply@anthropic.com>
The --without-default-devices flag was parked in a comment while the
per-architecture device allowlist was incomplete: the runtimes emit
devices that were missing from it, so a build with the flag active
produced a QEMU that failed at VM start rather than at build time.
Complete the allowlist with every device the Go runtime and runtime-rs
can emit:
- VIRTIO_SCSI virtio-scsi + scsi-hd (block_device_driver=virtio-scsi)
- VHOST_USER_BLK / VHOST_USER_SCSI vhost-user block backends
- NVDIMM DAX rootfs image (x86_64, aarch64, ppc64le)
- PCIE_PORT pcie-root-port: hot-plug slots and root-port topology
- XIO3130 x3130-upstream/xio3130-downstream: switch-port topology
- PCI_BRIDGE pci-bridge: bridge-port topology
- PCIE_PCI_BRIDGE pcie-pci-bridge
- PXB pxb-pcie on x86_64 (NUMA-pinned GPU root complexes);
aarch64 already carried it as a CXL dependency
- PVPANIC_ISA pvpanic guest kernel panic reporting (x86_64)
The allowlist is split into transport-independent device models
(_COMMON_DEVS, all architectures) and PCI transport + PCIe slot
topology (_PCIE_DEVS, x86_64/aarch64; ppc64le takes the conventional
PCI subset since pSeries PHBs have no PCIe ports; s390x is pure CCW).
A post-build verify_devices check runs `qemu-system-* -device help`
and fails the build if any required device is missing, so allowlist
gaps surface at build time, never at VM start. The check is skipped
when cross-compiling since the binary cannot run on the build host.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude <noreply@anthropic.com>
An earlier commit in this PR accidentally removed the shellcheck
source directives, CROSS_BUILD initializer, repo_root_dir fallback,
and nounset-safe ${1:-} defaults, causing the "Shellcheck required"
CI check to fail.
Restore the file to match the main-branch state for these lines.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
avx512bw is an x86_64 host-CPU JIT option that does not exist in
QEMU's configure on s390x, aarch64, or ppc64le. Previously the
build succeeded because the Docker builder cache hid the failure;
our script change invalidated the cache and exposed the bug.
Add [[ "${arch}" == x86_64 ]] alongside the existing version guard
so the flag is only passed where configure accepts it.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace rm -rf qemu + clone-into-relative-path with mktemp -d so the
build never touches a pre-existing qemu/ directory in the bind-mounted
host working directory. The trap ensures cleanup on any exit path.
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
--disable-avx512bw and --enable-avx512bw were removed in QEMU 10.1.0.
Passing them to qemu-snp-experimental (11.0.0) or any other post-10.1.0
build causes configure to abort with 'unknown option'.
Add the same version guard that upstream main carries.
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Use 'CONFIG_VIRTIO_MEM=' prefix match instead of '\b' word boundary in
grep -v: \b is a GNU extension not guaranteed by POSIX, making the
ppc64le VIRTIO_MEM exclusion unreliable on non-GNU grep implementations.
Add a comment on the rm -rf call to make clear this script always runs
inside a dedicated build container, so deleting a relative path is safe.
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
QEMU ships firmware for every emulated platform it was compiled for.
Ship only the firmware the target architecture actually boots:
- x86_64: edk2-x86_64-*.fd (OVMF)
- aarch64: edk2-aarch64-*.fd (AAVMF)
- s390x: s390-ccw.img (CCW BIOS)
- ppc64le: slof.bin / vof.bin (Open Firmware)
Firmware for unrelated architectures (ARM32, RISC-V, LoongArch, HPPA …)
is added to the blacklist at packaging time so it never lands in the
deployed tarball. This reduces installed size and removes executable
blobs that have no function on the target host.
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
--without-default-devices uses allnoconfig semantics: every Kconfig
entry with 'default y' is suppressed, which silently removes all virtio
devices from the binary. Write the minimal set of CONFIG_* entries we
actually need into each arch's default.mak before configure runs.
Also fix two bugs in the existing code:
- default.mak was written after ./configure, so meson never picked up
the entries; move it before the configure call
- aarch64 path used arm-softmmu (32-bit) instead of aarch64-softmmu
- make had a typo: 'nproc +--ignore' instead of 'nproc --ignore'
Common set: VIRTIO_{BLK,NET,SERIAL,RNG,BALLOON,MEM}, VHOST_USER_FS,
VIRTIO_9P, VHOST_VSOCK, VFIO_PCI, IOMMUFD. Architecture-specific:
- x86_64: Q35, VTD/VTD_ACCEL (IOMMUFD-backed), AMD_IOMMU
- aarch64: ARM_VIRT, PXB, CXL/CXL_MEM_DEVICE (required for Rubin vCXL)
- s390x: S390_CCW_VIRTIO, VIRTIO_CCW, VFIO_CCW/VFIO_AP
(VIRTIO_PCI/VFIO_PCI/IOMMUFD excluded — CCW bus, not PCI)
- ppc64le: PSERIES (VIRTIO_MEM excluded — not supported on ppc64)
Build only the system emulator binary instead of the full tool set;
ppc64le uses the 'ppc64' target name.
All four arch configs validated with minikconf --allnoconfig (exit 0).
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Add --enable-vhost-kernel and --enable-vhost-user to restore vhost
acceleration silently broken by auto_features=disabled (both features
are 'auto' in meson and become disabled under --without-default-features).
Add --enable-strip and --enable-lto: strip removes debug symbols from
installed binaries; LTO enables cross-unit dead-code elimination which
with our minimal device set typically shrinks the binary by 15-25% at
the cost of ~2-3x longer build times.
Annotate the aarch64 TCG gap (all other arches already pass
--disable-tcg) with a TODO so it is visible during review without
changing behaviour before aarch64 maintainers have been consulted.
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Additionally to the qemu script dir we need to also include the
changes made in configure-hypervisor.sh
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
We have a very long list of disable calls for QEMU that we
can replace by using the configure script to disable all default
features and default devices.
We enable selectively the features that we need.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
The guest kernel build derived a CROSS_COMPILE=<arch>-linux-gnu- prefix
whenever the target arch differed from the host, the static-build wrapper
switched to a per-arch `-cross-build` image (with buildx/--platform for
s390x), and the builder image installed gcc-<arch>-linux-gnu /
binutils-<arch>-linux-gnu cross toolchains.
None of this is exercised on our side: the kernel is always built on a
native-arch runner. Remove the CROSS_COMPILE plumbing, the cross-build
image/buildx handling and the cross toolchain install.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Building QEMU for a foreign arch relied on configure-hypervisor.sh
emitting --cross-prefix, build-qemu.sh passing --cc/--cross-prefix for
s390x, and the builder image installing foreign-arch -dev packages
(via dpkg multiarch/DPKG_ARCH) plus a gcc-<arch>-linux-gnu cross toolchain.
None of this is exercised on our side: QEMU is always built on a
native-arch runner. Remove the cross-prefix logic, the s390x cross
configure branch, and the DPKG_ARCH/cross-gcc plumbing from the Dockerfile
and build-base-qemu.sh.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The shared lib.sh exported CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM so the
per-component static-build scripts could switch to `-cross-build` builder
images, run `docker buildx build --platform=...` under QEMU emulation and,
for shim-v2, swap in the messense/rust-musl-cross image with a matching
cross CC.
These cross paths are not exercised on our side (all builds run on
native-arch runners), so remove them: builder images are now built with a
plain `docker build`, and shim-v2 builds runtime-rs natively without the
musl-cross image or CC override.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The kata-deploy local-build pipeline auto-enabled a cross build whenever
the host arch differed from TARGET_ARCH, wiring up docker buildx, QEMU
binfmt emulation and an s390x cross toolchain (incl. building s390-tools
genprotimg with CROSS_COMPILE) in the builder image.
None of this is exercised on our side: every release/CI build runs on a
native-arch runner, so the cross paths are dead code. Remove the
CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM/binfmt machinery and the s390x
cross toolchain from the builder image; the tarballs are always built for
the host arch now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Commit e8bb619b1b ("runtime: do not require OVMF for qemu-nvidia-cpu
on arm64") dropped the generic OVMF/AAVMF firmware inheritance for the
Go qemu-nvidia-cpu runtime, but assumed runtime-rs already did the right
thing by default. It didn't: the runtime-rs profile still pointed at the
generic @FIRMWAREPATH@ / @FIRMWAREVOLUMEPATH@ and still pulled in the
ovmf component on aarch64.
Mirror the fix on the runtime-rs side by introducing dedicated empty
FIRMWAREPATH_NV_CPU / FIRMWAREVOLUMEPATH_NV_CPU variables, wiring them
into the qemu-nvidia-cpu-runtime-rs configuration, and dropping ovmf
from its aarch64 shim components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Make qemu-nvidia-cpu-runtime-rs the default shim on amd64 and arm64 for
the kata-deploy NVIDIA CPU profile, so installs from
try-kata-nvidia-cpu.values.yaml select the Rust runtime by default
instead of the Go qemu-nvidia-cpu shim.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
EROFS fs-verity depends on the backing filesystem supporting fs-verity,
which cannot be guaranteed on an arbitrary node. Rather than gate the
NVIDIA CPU profile on a host feature we do not control, disable fs-verity
so it works on the widest possible range of setups; we take that hit
deliberately to keep the profile generic.
Move this out of the test harness and into the kata-deploy NVIDIA CPU
profile: try-kata-nvidia-cpu.values.yaml now ships a containerd erofs
snapshotter drop-in that disables fs-verity and pins the memory-backed
default_size, and selects memory-backed rw layers with dm-verity for the
lower (image) layers. Because containerd.userDropIn is loaded after
kata-deploy's generated config, it overrides the built-in enable_fsverity
default.
The CI helm helper no longer special-cases NVIDIA CPU to inject
enable_fsverity=false. Instead it honors an erofs snapshotter drop-in
already provided by the base values file and only synthesizes the default
default_size drop-in when the profile does not provide its own.
Note that this drop-in targets containerd's built-in EROFS snapshotter,
available only on containerd >= 2.2.0 (config version 3, conf.d auto-import),
which is the same minimum kata-deploy already enforces for the EROFS
snapshotter.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Drop the unfinished ARM CCA stack (qemu-cca shim, experimental QEMU/OVMF
and kernel assets, packaging targets, and CI builds) since it is not
tested or shipped in any runtime bundle.
This will be supported by the moment we have development done for real
hardware available.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Drop uscan-only fields, duplicate qemu.tag, and unused image/initrd
URLs from versions.yaml, and teach gen_versions_txt to treat
qemu.version as either a branch or tag ref.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Resolve kata-monitor log level from monitor.logLevel, the chart-wide
logLevel, and debug:true, matching the precedence used by kata-deploy.
Fixes#13382
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Derive the runtime endpoint from k8sDistribution and mount the matching
host socket directory so kata-monitor can reach containerd on
non-vanilla k8s deployments.
Fixes#13378
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Use zstd level 10 (via kata_tar_zstd) when creating static build
tarballs so CI artifacts stay under GitHub's 2GB limit without
paying the build-time cost of maximum compression.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Reuse the kubelet runtimeRequestTimeout validation for guest-pull
configurations.
Guest pull can also keep kubelet waiting during CreateContainer. Warn
when kata-deploy can see that runtimeRequestTimeout is below the same
suggested timeout used for EROFS layer conversion.
Detect both the regular guest-pull mapping and the experimental force
guest-pull configuration so nodes get the warning for either path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Extend the kata-deploy host-check stage with EROFS-specific
prerequisite validation.
Check that containerd supports the EROFS snapshotter, required
kernel features are available either as loaded modules or built-in
kernel configuration, and the host erofs-utils version satisfies the
minimum required by kata-deploy's configured mkfs options. Also warn
when kubelet runtimeRequestTimeout appears too low for EROFS layer
conversion, which can run during CreateContainer.
Read kubelet runtimeRequestTimeout from kubelet /configz through the
apiserver node proxy, and add the corresponding RBAC permission. Keep
fs-verity validation warning-only because user drop-ins may override
kata-deploy's default enable_fsverity setting and kata-deploy does
not yet validate the backing filesystem's fs-verity feature.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Switch the per-architecture default shim from the Go runtime ("qemu") to the
Rust runtime ("qemu-runtime-rs") wherever a runtime-rs build exists, and
align CI/test fallbacks with the new default.
Document the defaultShim values and Go runtime deprecation in the Helm and
config migration guides using mkdocs admonitions.
Fixes: kata-containers#9546
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Use dedicated empty firmware makefile variables so the NVIDIA CPU
runtime does not inherit generic OVMF/AAVMF firmware paths.
runtime-rs already sets the correct option by default, so, no changed
needed on that front.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The CI agent tarball is GNU-linked (USE_DEVMAPPER=yes) and does not run
reliably on Alpine/musl initrd.
Use Ubuntu noble for x86_64 and aarch64 vanilla initrd, matching the
other initrd variants, with kata-agent as /sbin/init.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
