Building QEMU for a foreign arch relied on configure-hypervisor.sh
emitting --cross-prefix, build-qemu.sh passing --cc/--cross-prefix for
s390x, and the builder image installing foreign-arch -dev packages
(via dpkg multiarch/DPKG_ARCH) plus a gcc-<arch>-linux-gnu cross toolchain.
None of this is exercised on our side: QEMU is always built on a
native-arch runner. Remove the cross-prefix logic, the s390x cross
configure branch, and the DPKG_ARCH/cross-gcc plumbing from the Dockerfile
and build-base-qemu.sh.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The shared lib.sh exported CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM so the
per-component static-build scripts could switch to `-cross-build` builder
images, run `docker buildx build --platform=...` under QEMU emulation and,
for shim-v2, swap in the messense/rust-musl-cross image with a matching
cross CC.
These cross paths are not exercised on our side (all builds run on
native-arch runners), so remove them: builder images are now built with a
plain `docker build`, and shim-v2 builds runtime-rs natively without the
musl-cross image or CC override.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The kata-deploy local-build pipeline auto-enabled a cross build whenever
the host arch differed from TARGET_ARCH, wiring up docker buildx, QEMU
binfmt emulation and an s390x cross toolchain (incl. building s390-tools
genprotimg with CROSS_COMPILE) in the builder image.
None of this is exercised on our side: every release/CI build runs on a
native-arch runner, so the cross paths are dead code. Remove the
CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM/binfmt machinery and the s390x
cross toolchain from the builder image; the tarballs are always built for
the host arch now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The `HOST_ARCH != ARCH` branch in utils.mk existed solely to support
cross-compiling the Rust components (overriding the linker via `CC`).
Cross-compilation is not exercised on our side: all release/CI builds run
on native-arch runners, so this path is dead code. Remove it.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Fail startup when KATA_CONF_FILE does not resolve to one of the shipped
default config files, preventing untrusted config path injection via
environment overrides.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Stop accepting io.katacontainers.config_path so config selection comes
only from shim options, KATA_CONF_FILE, and defaults.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The sandbox_bind_mounts option accepts host paths that runtime-rs
exposes to the guest shared filesystem. Runtime annotations are not
gated by the hypervisor annotation allowlist, allowing an untrusted
workload annotation to add arbitrary host paths to the sandbox bind
mount configuration.
Ignore KATA_ANNO_CFG_SANDBOX_BIND_MOUNTS when processing workload
annotations. Sandbox bind mounts remain available through the
operator-controlled TOML configuration, preserving the sandbox-level
host/guest sharing feature while removing the untrusted annotation
path.
Really appreciate it that Anthropic and Ada Logics report this issue.
Reported-by: Anthropic and Ada Logics
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Move PCI enumeration, VFIO helpers, and InfiniBand diagnostics out of
kata-sys-util into the external pcilibs-rs crate so they can be reused
independently of kata-containers.
Depend on pcilibs-rs 0.1.0 from crates.io.
Add pcilibs to the cspell dictionary for the new crate reference in
src/libs/README.md.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
open() waits for the binary logger readiness handshake with an unbounded
async read(). If the logger never writes to fd 5 (or inherits a bad fd),
container start can hang indefinitely.
Let's add a timeout around the readiness read and fail the start path if
it expires (the child will be killed on drop due to kill_on_drop(true)).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Suggested-by: CoPilot
Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Set the non-confidential NVIDIA runtime-rs emptyDir mode to
block-plain through a dedicated Makefile default.
This prepares the runtime class for configurations where filesystem
sharing is disabled and emptyDir volumes need to be backed by
guest-mounted block devices instead of shared-fs.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Support containerd binary:// stdio URIs in runtime-rs by spawning the
logger with stdout, stderr, and readiness on file descriptors 3, 4, and 5.
Preserve URI arguments and container metadata, and stream output through
nonblocking pipes with backpressure.
Processes using a binary logger fall back from passfd to the existing agent
streams so the host shim can feed the logger. After EOF, allow the logger to
drain before using SIGTERM and SIGKILL as bounded fallbacks.
Fixes: #4391
Related: #4420
Signed-off-by: cayoub-oai <276123840+cayoub-oai@users.noreply.github.com>
Assisted-By: OpenAI Codex
Commit e8bb619b1b ("runtime: do not require OVMF for qemu-nvidia-cpu
on arm64") dropped the generic OVMF/AAVMF firmware inheritance for the
Go qemu-nvidia-cpu runtime, but assumed runtime-rs already did the right
thing by default. It didn't: the runtime-rs profile still pointed at the
generic @FIRMWAREPATH@ / @FIRMWAREVOLUMEPATH@ and still pulled in the
ovmf component on aarch64.
Mirror the fix on the runtime-rs side by introducing dedicated empty
FIRMWAREPATH_NV_CPU / FIRMWAREVOLUMEPATH_NV_CPU variables, wiring them
into the qemu-nvidia-cpu-runtime-rs configuration, and dropping ovmf
from its aarch64 shim components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
The termination-log tests exercise termination-message propagation over
the agent GetDiagnosticData RPC, which only matters when the host cannot
read guest files directly, i.e. on shared_fs=none configurations. They
previously gated on is_confidential_runtime_class and therefore skipped
on the NVIDIA CPU runtime-rs handler, which now also runs with
shared_fs=none.
Add an is_shared_fs_none_runtime_class helper (confidential runtime
classes plus qemu-nvidia-cpu-runtime-rs) and use it for the setup and
teardown gating so the tests run on the NVIDIA CPU runtime-rs handler as
well. The plain Go qemu-nvidia-cpu class still uses virtio-fs and is
intentionally excluded. Keep the "blocked by default CoCo policy"
assertion confidential-only, since non-confidential shared_fs=none
classes do not ship a default policy that denies the RPC.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Make qemu-nvidia-cpu-runtime-rs the default shim on amd64 and arm64 for
the kata-deploy NVIDIA CPU profile, so installs from
try-kata-nvidia-cpu.values.yaml select the Rust runtime by default
instead of the Go qemu-nvidia-cpu shim.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
EROFS fs-verity depends on the backing filesystem supporting fs-verity,
which cannot be guaranteed on an arbitrary node. Rather than gate the
NVIDIA CPU profile on a host feature we do not control, disable fs-verity
so it works on the widest possible range of setups; we take that hit
deliberately to keep the profile generic.
Move this out of the test harness and into the kata-deploy NVIDIA CPU
profile: try-kata-nvidia-cpu.values.yaml now ships a containerd erofs
snapshotter drop-in that disables fs-verity and pins the memory-backed
default_size, and selects memory-backed rw layers with dm-verity for the
lower (image) layers. Because containerd.userDropIn is loaded after
kata-deploy's generated config, it overrides the built-in enable_fsverity
default.
The CI helm helper no longer special-cases NVIDIA CPU to inject
enable_fsverity=false. Instead it honors an erofs snapshotter drop-in
already provided by the base values file and only synthesizes the default
default_size drop-in when the profile does not provide its own.
Note that this drop-in targets containerd's built-in EROFS snapshotter,
available only on containerd >= 2.2.0 (config version 3, conf.d auto-import),
which is the same minimum kata-deploy already enforces for the EROFS
snapshotter.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Set the non-confidential NVIDIA runtime-rs emptyDir mode to
block-plain through a dedicated Makefile default.
This prepares the runtime class for configurations where filesystem
sharing is disabled and emptyDir volumes need to be backed by
guest-mounted block devices instead of shared-fs.
Also add the shared configure_nvidia_runtime_rs_shared_fs_dropin()
test helper, which keeps the NVIDIA runtime-rs Docker/nerdctl smoke
tests on virtio-fs while the shared_fs=none + EROFS snapshotter path
is exercised by Kubernetes CI.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
buildNUMATopologyForVFIOHostSet assumed a fixed 2 MiB huge page size
(memAlign = 2) when distributing memory across guest NUMA nodes. On
hosts where /dev/hugepages is backed by 1 GiB (or other) huge pages,
per-node memory was not aligned to the actual page size, so QEMU could
not back the memory with huge pages unless the operator intentionally
aligned memory requests to 1GiB.
Inspect the real huge page size via statfs on /dev/hugepages, which
reports the huge page size as the hugetlbfs filesystem block size, and
align per-node memory to that value instead of assuming 2 MiB.
Update TestBuildNUMATopologyHugePages to derive the expected per-node
memory from the detected page size so it holds on both 2 MiB and 1 GiB
hosts.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
With the plugins support added to CoCo Resource URI scheme,
users may want to use get_cdh_resource() to pull resources
from Trustee plugins.
Currently, get_cdh_resource() unconditionally uses kbs://
scheme which prevents callers to specify plugin paths as
kbs+<plugin>://.
Move kbs:// scheme out of get_cdh_resource() helper and update
call sites accordingly. Currently, only s390x path uses this
so only one caller needs to be updated.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Only enable discard for block-plain emptyDir volumes when the
hypervisor reports block device discard support.
The resource manager now queries hypervisor capabilities once and
passes block discard support through VolumeContext. Block emptyDir
volume setup uses that capability to decide whether to add the
discard mount option and related mount metadata, avoiding unsupported
discard mounts on hypervisors that cannot expose discard/unmap to the
guest.
It will correctly adjust the related codes with the new support.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Map BlockConfigModern.discard_unmap to Dragonball block device
sparse support when adding modern block devices.
This lets runtime-rs request guest-visible discard support through
the Dragonball block device configuration while leaving vhost-user
block devices disabled.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add a sparse flag to Dragonball block device configuration and use
it to expose virtio-blk discard support to the guest. Handle discard
requests by validating the requested range and punching holes in
sparse local file backends, returning an I/O error instead of
panicking on discard failures.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add the Cloud Hypervisor disk sparse option to the local DiskConfig
model and keep its default aligned with upstream. Map
BlockConfigModern.discard_unmap to DiskConfig.sparse so block devices
that request discard/unmap explicitly expose sparse discard support
through the CH API.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
To tells if the hypervisor exposes block discard/unmap support to
the guest, we introduce a new capability of discard/unmap named
`BlockDeviceDiscardSupport` in the capability set.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove the remaining legacy Block device implementation now that
BlockModern covers all block paths, finishing the migration started
by the QEMU and DeviceManager legacy-path drops.
Delete virtio_blk.rs (BlockConfig, BlockDevice and its Device impl)
and stop exporting it from driver/mod.rs. Drop the BlockCfg variant
from DeviceConfig and the Block variant from DeviceType. Remove the
DeviceConfig::BlockCfg handling and the create_block_device helper
from DeviceManager, and drop DeviceType::Block from the QEMU
hotunplug unsupported list.
No new behavior; pure removal of the dead legacy path.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove DeviceType::Block so BlockModern is the single code path,
completing the BlockConfig -> BlockConfigModern migration at the
Device Manager layer.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The preceding "Use BlockConfigModern for ..." series converted every
block-device producer in the Qemu path to emit BlockModern. The Qemu
inner module still held the matching legacy Block consumer code across
all three lifecycle ops (coldplug/hotplug/hotunplug), now unreachable.
Remove it so BlockModern is the single code path, completing the
BlockConfig -> BlockConfigModern migration at the Qemu layer.
The hotunplug arm for Block now returns an explicit "unsupported"
error instead of a stale QMP detach. The new added Block(_) is just a
place holder and it will be removed in a follow-up commit.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for firecacker.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for agent-ctl vm_ops.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the erofs rootfs path from BlockConfig to BlockConfigModern,
routing its block-device attachment through BlockCfgModern.
This covers the rw layer, single-device raw, multi-device VMDK, and
GPT-partitioned VMDK attachment points in erofs_rootfs.rs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for swap task.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for block rootfs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the host block-device (major/minor) passthrough path in
ResourceManagerInner from BlockConfig to BlockConfigModern, routing it
through DeviceConfig::BlockCfgModern. This continues the BlockModern
migration to cover block-mode devices carried in the OCI spec.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
