The Go runtime confidential and NVIDIA GPU QEMU configs booted the
monolithic rootfs images (kata-containers-confidential.img,
kata-containers-nvidia-gpu.img, kata-containers-nvidia-gpu-confidential.img)
that bake every guest component into a single rootfs.
Switch them to the composable layout already used by runtime-rs: boot the
measured base image (kata-containers.img / kata-containers-nvidia.img) and
cold-plug the purpose-specific guest extension images:
- qemu-coco-dev, qemu-tdx, qemu-snp: base + coco extension
- qemu-nvidia-gpu: nvidia base + gpu extension
- qemu-nvidia-gpu-snp/-tdx: nvidia base + gpu + coco extensions
The dm-verity params for the NVIDIA confidential configs move from
KERNELVERITYPARAMS_CONFIDENTIAL_NV to KERNELVERITYPARAMS_NV (the nvidia
base hash), and the Makefile gains the coco/gpu extension image path and
verity vars (COCOIMAGEPATH/COCOVERITYPARAMS,
IMAGEPATH_NV_EXTENSION/NVIDIAGPUEXTENSIONVERITYPARAMS). No runtime code
change is needed: the Go QEMU backend already cold-plugs
guest_extension_images.
The main motivation is size: the monolithic images duplicate content that
already ships in the base + extension images, and once the Go runtime no
longer needs them we can stop building them (see follow-up commits). That
keeps the released kata-static tarball within GitHub's release asset size
limit (2 GiB).
The monolithic images are no longer referenced by any shipped config, but
their build targets are intentionally left in place so they can still be
built manually.
Note: this composable path had only been validated for runtime-rs so far;
the Go-runtime TEE/GPU boots (TDX, SEV-SNP, NVIDIA GPU) still need
validation on real hardware.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Dragonball now identifies every network device by its iface_id and
refuses a device without one, so pass the endpoint's dev_id through
explicitly instead of leaving the field to its empty default. This
completes the change started two commits back: every network backend
hands dragonball a device name the runtime knows.
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
virtio-net, vhost-net and vhost-user-net were spread across three
device managers with three config types, three error enums and three
insert paths, even though they are all network interfaces of the guest
behind one InsertNetworkDevice request. Merge them into a single
NetworkDeviceMgr with one info_list, dispatching on the Backend enum
per device, the way the block manager keeps its backends in one list
keyed by BlockDeviceType. The unified NetworkInterfaceConfig moves
from api/v1 into the manager, as the other managers define theirs,
and the per-backend config conversions disappear.
Device identity becomes the caller-supplied iface_id for every
backend, matching how every other device class is identified, instead
of vhost-user-net borrowing its socket path as its id. The socket path
becomes a conflict-checked resource alongside the tap name and the
guest MAC, so two devices claiming one vhost-user socket are refused
(DuplicatedUdsPath) rather than silently collapsing into one entry.
Behaviour changes, all deliberate:
- A device with an empty iface_id is refused (MissingIfaceId). The id
is the interface name inside the guest, so an unnamed device would
silently take the place of another unnamed one and the VM would boot
with an interface missing. Callers that relied on inserting
vhost-user-net devices without an id must now supply one.
- Hotplugging a device whose id is already in use is refused
(DeviceIDAlreadyExist): the old config-update path would have left
the already-attached device live on the bus with no handle left to
remove it. Before boot, a repeated id still reconfigures the device
as it does for every other device class.
- Teardown no longer stops at the first device that fails to be
destroyed; the failure is logged and the remaining devices are still
removed. This path newly matters because vhost-user-net devices now
retain their device handle like the other backends do.
The unused open_tap helper (never called since the file was introduced)
and the NetworkDeviceInfo alias are dropped rather than carried over.
vhost-user-net devices inserted through runtime-rs receive their id in
the following commit; with only this commit they are refused for the
missing id.
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
The virtio-net and vhost-net paths hand dragonball an iface_id taken
from the virt interface name, but the vhost-user-net endpoint left
dev_id unassigned, so the device carried no name of its own and was
identified by its socket path inside the hypervisor. Assign dev_id from
the endpoint's interface name so every network backend is identified
the same way by a name the runtime knows.
Also construct dragonball's vhost-user config with struct update syntax
so this commit builds both before and after dragonball's config gains
an iface_id field; the follow-up commit passes dev_id through
explicitly once that field exists.
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
kata-ctl shipped a Cross.toml and a "Cross-builds" README section
describing how to cross compile the tool for other architectures with
cross-rs (installing foreign-arch libssl via dpkg multiarch pre-build
hooks).
This is not exercised on our side: kata-ctl is built on native-arch
runners. Remove Cross.toml and the cross-build documentation.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Fail startup when KATA_CONF_FILE does not resolve to one of the shipped
default config files, preventing untrusted config path injection via
environment overrides.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Stop accepting io.katacontainers.config_path so config selection comes
only from shim options, KATA_CONF_FILE, and defaults.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The sandbox_bind_mounts option accepts host paths that runtime-rs
exposes to the guest shared filesystem. Runtime annotations are not
gated by the hypervisor annotation allowlist, allowing an untrusted
workload annotation to add arbitrary host paths to the sandbox bind
mount configuration.
Ignore KATA_ANNO_CFG_SANDBOX_BIND_MOUNTS when processing workload
annotations. Sandbox bind mounts remain available through the
operator-controlled TOML configuration, preserving the sandbox-level
host/guest sharing feature while removing the untrusted annotation
path.
Really appreciate it that Anthropic and Ada Logics report this issue.
Reported-by: Anthropic and Ada Logics
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Move PCI enumeration, VFIO helpers, and InfiniBand diagnostics out of
kata-sys-util into the external pcilibs-rs crate so they can be reused
independently of kata-containers.
Depend on pcilibs-rs 0.1.0 from crates.io.
Add pcilibs to the cspell dictionary for the new crate reference in
src/libs/README.md.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
open() waits for the binary logger readiness handshake with an unbounded
async read(). If the logger never writes to fd 5 (or inherits a bad fd),
container start can hang indefinitely.
Let's add a timeout around the readiness read and fail the start path if
it expires (the child will be killed on drop due to kill_on_drop(true)).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Suggested-by: CoPilot
Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Set the non-confidential NVIDIA runtime-rs emptyDir mode to
block-plain through a dedicated Makefile default.
This prepares the runtime class for configurations where filesystem
sharing is disabled and emptyDir volumes need to be backed by
guest-mounted block devices instead of shared-fs.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Support containerd binary:// stdio URIs in runtime-rs by spawning the
logger with stdout, stderr, and readiness on file descriptors 3, 4, and 5.
Preserve URI arguments and container metadata, and stream output through
nonblocking pipes with backpressure.
Processes using a binary logger fall back from passfd to the existing agent
streams so the host shim can feed the logger. After EOF, allow the logger to
drain before using SIGTERM and SIGKILL as bounded fallbacks.
Fixes: #4391
Related: #4420
Signed-off-by: cayoub-oai <276123840+cayoub-oai@users.noreply.github.com>
Assisted-By: OpenAI Codex
Commit e8bb619b1b ("runtime: do not require OVMF for qemu-nvidia-cpu
on arm64") dropped the generic OVMF/AAVMF firmware inheritance for the
Go qemu-nvidia-cpu runtime, but assumed runtime-rs already did the right
thing by default. It didn't: the runtime-rs profile still pointed at the
generic @FIRMWAREPATH@ / @FIRMWAREVOLUMEPATH@ and still pulled in the
ovmf component on aarch64.
Mirror the fix on the runtime-rs side by introducing dedicated empty
FIRMWAREPATH_NV_CPU / FIRMWAREVOLUMEPATH_NV_CPU variables, wiring them
into the qemu-nvidia-cpu-runtime-rs configuration, and dropping ovmf
from its aarch64 shim components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Set the non-confidential NVIDIA runtime-rs emptyDir mode to
block-plain through a dedicated Makefile default.
This prepares the runtime class for configurations where filesystem
sharing is disabled and emptyDir volumes need to be backed by
guest-mounted block devices instead of shared-fs.
Also add the shared configure_nvidia_runtime_rs_shared_fs_dropin()
test helper, which keeps the NVIDIA runtime-rs Docker/nerdctl smoke
tests on virtio-fs while the shared_fs=none + EROFS snapshotter path
is exercised by Kubernetes CI.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
buildNUMATopologyForVFIOHostSet assumed a fixed 2 MiB huge page size
(memAlign = 2) when distributing memory across guest NUMA nodes. On
hosts where /dev/hugepages is backed by 1 GiB (or other) huge pages,
per-node memory was not aligned to the actual page size, so QEMU could
not back the memory with huge pages unless the operator intentionally
aligned memory requests to 1GiB.
Inspect the real huge page size via statfs on /dev/hugepages, which
reports the huge page size as the hugetlbfs filesystem block size, and
align per-node memory to that value instead of assuming 2 MiB.
Update TestBuildNUMATopologyHugePages to derive the expected per-node
memory from the detected page size so it holds on both 2 MiB and 1 GiB
hosts.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
With the plugins support added to CoCo Resource URI scheme,
users may want to use get_cdh_resource() to pull resources
from Trustee plugins.
Currently, get_cdh_resource() unconditionally uses kbs://
scheme which prevents callers to specify plugin paths as
kbs+<plugin>://.
Move kbs:// scheme out of get_cdh_resource() helper and update
call sites accordingly. Currently, only s390x path uses this
so only one caller needs to be updated.
Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com>
Only enable discard for block-plain emptyDir volumes when the
hypervisor reports block device discard support.
The resource manager now queries hypervisor capabilities once and
passes block discard support through VolumeContext. Block emptyDir
volume setup uses that capability to decide whether to add the
discard mount option and related mount metadata, avoiding unsupported
discard mounts on hypervisors that cannot expose discard/unmap to the
guest.
It will correctly adjust the related codes with the new support.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Map BlockConfigModern.discard_unmap to Dragonball block device
sparse support when adding modern block devices.
This lets runtime-rs request guest-visible discard support through
the Dragonball block device configuration while leaving vhost-user
block devices disabled.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add a sparse flag to Dragonball block device configuration and use
it to expose virtio-blk discard support to the guest. Handle discard
requests by validating the requested range and punching holes in
sparse local file backends, returning an I/O error instead of
panicking on discard failures.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add the Cloud Hypervisor disk sparse option to the local DiskConfig
model and keep its default aligned with upstream. Map
BlockConfigModern.discard_unmap to DiskConfig.sparse so block devices
that request discard/unmap explicitly expose sparse discard support
through the CH API.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
To tells if the hypervisor exposes block discard/unmap support to
the guest, we introduce a new capability of discard/unmap named
`BlockDeviceDiscardSupport` in the capability set.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove the remaining legacy Block device implementation now that
BlockModern covers all block paths, finishing the migration started
by the QEMU and DeviceManager legacy-path drops.
Delete virtio_blk.rs (BlockConfig, BlockDevice and its Device impl)
and stop exporting it from driver/mod.rs. Drop the BlockCfg variant
from DeviceConfig and the Block variant from DeviceType. Remove the
DeviceConfig::BlockCfg handling and the create_block_device helper
from DeviceManager, and drop DeviceType::Block from the QEMU
hotunplug unsupported list.
No new behavior; pure removal of the dead legacy path.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove DeviceType::Block so BlockModern is the single code path,
completing the BlockConfig -> BlockConfigModern migration at the
Device Manager layer.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The preceding "Use BlockConfigModern for ..." series converted every
block-device producer in the Qemu path to emit BlockModern. The Qemu
inner module still held the matching legacy Block consumer code across
all three lifecycle ops (coldplug/hotplug/hotunplug), now unreachable.
Remove it so BlockModern is the single code path, completing the
BlockConfig -> BlockConfigModern migration at the Qemu layer.
The hotunplug arm for Block now returns an explicit "unsupported"
error instead of a stale QMP detach. The new added Block(_) is just a
place holder and it will be removed in a follow-up commit.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for firecacker.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for agent-ctl vm_ops.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the erofs rootfs path from BlockConfig to BlockConfigModern,
routing its block-device attachment through BlockCfgModern.
This covers the rw layer, single-device raw, multi-device VMDK, and
GPT-partitioned VMDK attachment points in erofs_rootfs.rs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for swap task.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for block rootfs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the host block-device (major/minor) passthrough path in
ResourceManagerInner from BlockConfig to BlockConfigModern, routing it
through DeviceConfig::BlockCfgModern. This continues the BlockModern
migration to cover block-mode devices carried in the OCI spec.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Switch the InitData resource path from BlockConfig to BlockConfigModern
so the InitData is handled through the modern block device pipeline,
consistent with the recent BlockModern migration for block volumes.
This changes ResourceConfig::InitData to wrap BlockConfigModern,
routes its handling through DeviceConfig::BlockCfgModern, and updates
its related helper function in virt_container to build a
BlockConfigModern.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Switch the VmRootfs resource path from BlockConfig to BlockConfigModern
so the VM rootfs is handled through the modern block device pipeline,
consistent with the recent BlockModern migration for block volumes.
This changes ResourceConfig::VmRootfs to wrap BlockConfigModern, routes
its handling through DeviceConfig::BlockCfgModern, and updates
prepare_rootfs_config() in virt_container to build a BlockConfigModern.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce a new field of serial_override in the BlockModern config.
Add a DeviceType::BlockModern branch to the QEMU QMP hotplug path,
mirroring the existing Block semantics: skip block devices that
represent the initrd, and dispatch to nvdimm or blk/ccw/scsi
according to driver_option. Snapshot the config parameters under
lock and run the asynchronous hotplug without holding it to avoid
awaiting across the lock.
Pass the real block format and, when enabled, the first independent
iothread (indep_iothread_0) to hotplug_block_device instead of the
default format and None, and return DeviceType::BlockModern so the
hotplugged device is tracked.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Treat SPDK and VFIO direct volumes as a two-step mount. The agent
storage entry mounts the device inside the guest with block storage
options, while the OCI mount bind-mounts that guest path into the
container.
Filter bind flags from storage options, preserve direct-volume
mountInfo options when present, and add ro when the volume is
read-only.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Build regular block volumes with BlockConfigModern and submit them
through DeviceConfig::BlockCfgModern. Preserve the existing block
device attributes while also recording the host path required by the
modern block device model.
Update block volume helper call sites with the default option slice
so this change remains buildable with the modern helper signature.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Switch the Cloud Hypervisor device handling path from the legacy
Block device variant to BlockModern. Keep the block device behind
its shared mutex while snapshotting the configuration for the
hotplug request, and persist the PCI path assigned by Cloud
Hypervisor back into the shared device state.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce build_bind_mount_options() to construct OCI mount options for
container-side bind mount, preferring volume_options over OCI options
and ensuring a bind/rbind flag is present.
And addfilter_block_storage_options() to strip bind/rbind flags from
Storage.options, preventing ENOTDIR when the agent mounts a block
device to a filesystem inside the guest.
And then Extend handle_block_volume() with a volume_options parameter
and replace hardcoded ro/empty options with the new filter+ro logic,
narrowing the function to a BlockModern-only path by removing the
DeviceType::Block branch.
Fix generate_shared_path to always create under rw/ since ro/
is a read-only bind mount of rw/ and direct creation fails with ReadOnly
FS. Switch the OCI Mount type from fs_type to KATA_MOUNT_BIND_TYPE because
the mount source is a directory (guest_path), not a block device.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Move the block device type definitions shared between BlockDevice and
BlockDeviceModern — VIRTIO_BLOCK_{PCI,MMIO,CCW}/VIRTIO_PMEM constants,
KATA_*_DEV_TYPE aliases, and the BlockDeviceAio / BlockDeviceFormat
enums — from virtio_blk.rs into virtio_blk_modern.rs, so they live
alongside the BlockModern handler that will replace the legacy
BlockDevice.
Add a `format: BlockDeviceFormat` field to BlockConfigModern to match
the legacy BlockConfig. Update mod.rs re-exports accordingly.
No behavior change; pure relocation plus the new field.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
