Align the composable VM images proposal with the naming convention used
throughout the implementation: the generic mechanism and its data model
are now "guest extension image" (config key `guest_extension_images`,
struct `GuestExtensionImage`), and the runtime artifacts use the matching
"extension" vocabulary (`/run/kata-extensions/<name>`,
`kata.extension.<name>.verity_params`, `kata-extension-mount@.service`,
the `extension-<name>` virtio-blk serial / dm-verity target, and the
`kata-containers-coco-extension.img` image).
It also documents how template-unit instances are enabled: a systemd
generator instantiates kata-extension-mount@<name>.service for each
extension on the kernel command line, so the rootfs build stays
extension-agnostic.
This replaces the earlier mixed "extra_images"/"addon" terminology and
addresses the naming-consistency review feedback on the series.
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Generate etc/kata-extensions/components.toml inside the CoCo extension rootfs
before building the image, so the manifest becomes part of the
dm-verity-measured erofs image.
The manifest declares the extension's path-only components (ocicrypt config,
pause bundle) and the launchable processes (attestation-agent,
confidential-data-hub, api-server-rest) with paths relative to the extension
mount point (/run/kata-extensions/coco) and `${var}` placeholders resolved
by kata-agent at runtime. This is the data the agent consumes to launch
the CoCo guest components without per-bundle code changes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Make the agent's handling of CoCo guest components data-driven so that a
new extension bundle can be introduced without changing agent code.
Add a new `extension` module defining a generic component manifest contract.
When a CoCo extension image is mounted at /run/kata-extensions/coco/, the agent
reads etc/kata-extensions/components.toml from it to discover:
- path-only components (looked up by id under the `[paths]` table), used
for the ocicrypt config and the pause bundle, and
- launchable processes (`[[process]]` entries) describing the binary
path, args, optional args, config file, environment, socket to wait on
and timeout.
`${var}` tokens in process fields are substituted from a fixed,
documented context the agent builds at runtime (attestation socket/uri,
config paths, rest-api features, initdata toml path, launch timeout,
resolved ocicrypt config path). Referencing an unknown variable is a
hard error, and manifest paths are validated against traversal, so the
contract stays fail-closed.
Processes are gated by a numeric `level` mapped from
guest_components_procs (AttestationAgent < ConfidentialDataHub <
ApiServerRest), preserving the existing implication semantics.
When no extension is mounted, launch_plan() returns None and the agent uses
a built-in plan that reproduces the legacy behaviour byte-for-byte
(binaries under /usr/local/bin, /etc/ocicrypt_config.json, /pause_bundle).
Monolithic / non-split confidential images are therefore unaffected.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add rootfs-image-coco-extension to the build-asset-rootfs matrix in the
amd64, arm64 and s390x static tarball workflows so the CoCo extension image
is built as part of CI. Keep building rootfs-image-confidential as well:
the runtime defaults to base + extension, but we retain the monolithic
confidential image for now during the transition.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add install_image_coco_extension() to kata-deploy-binaries.sh which:
- Unpacks the CoCo guest components and pause image tarballs into a
temporary rootfs directory (under the repo root so Docker-in-Docker
volume mounts resolve correctly)
- Calls image_builder.sh with USE_DOCKER=1, FS_TYPE=erofs,
MEASURED_ROOTFS=yes, SKIP_DAX_HEADER=yes, and SKIP_ROOTFS_CHECK=yes
to produce kata-containers-coco-extension.img + root_hash_coco-extension.txt
Add the rootfs-image-coco-extension-tarball Makefile target with
dependencies on pause-image-tarball and coco-guest-components-tarball.
The standard confidential image keeps pause-image and
coco-guest-components for now so it remains a usable standalone
monolithic CoCo image during the transition to base + extension.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
During the transition to composable (base + extension) CoCo images, run the
two runtimes on different image layouts so CI exercises both paths via
the existing qemu-{snp,tdx,coco-dev} (Go) and -runtime-rs (Rust) shims:
- runtime-rs (Rust) CoCo templates use the standard base image plus a
CoCo extension extra_image:
image = "@IMAGEPATH@" (was @IMAGECONFIDENTIALPATH@)
[[hypervisor.qemu.guest_extension_images]]
name = "coco"
path = "@COCOIMAGEPATH@"
verity_params = "@COCOVERITYPARAMS@"
COCOIMAGENAME/COCOIMAGEPATH/COCOVERITYPARAMS are added to the
runtime-rs Makefile only.
- runtime (Go) CoCo templates keep the monolithic confidential image
(@IMAGECONFIDENTIALPATH@) and pull rootfs-image-confidential.
shim-components.json reflects this: Go CoCo shims depend on
rootfs-image-confidential while the runtime-rs CoCo shims depend on
rootfs-image + rootfs-image-coco-extension.
shim-v2/build.sh feeds each runtime its own dm-verity params: Rust gets
the measured base image hash (@KERNELVERITYPARAMS@) plus the extension hash
(@COCOVERITYPARAMS@); Go gets the monolithic confidential image hash
(@KERNELVERITYPARAMS@). This is wired per make invocation so the same
@KERNELVERITYPARAMS@ placeholder resolves correctly for each.
Once the split path is validated we can flip the Go templates too and
drop the monolithic confidential image.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Extension images (e.g. the CoCo guest components extension) are not full root
filesystems -- they contain only the binaries and configuration that
get bind-mounted into the real rootfs at boot. The existing
check_rootfs() validation requires /sbin/init and systemd, which are
not present in extension images.
Add a SKIP_ROOTFS_CHECK environment variable that, when set to "yes",
bypasses the check_rootfs() call. Forward the variable into the
container environment when using the Docker-based build path so it
works in both direct and containerised invocations.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a systemd template unit kata-extension-mount@.service and companion
helper scripts (kata-extension-mount.sh, kata-extension-umount.sh) that handle
the guest-side setup of extension block device images.
The mount script:
- Discovers the block device by matching its virtio serial (extension-<name>)
- Reads verity parameters from kata.extension.<name>.verity_params on the
kernel command line
- Creates a dm-verity device and mounts the EROFS filesystem read-only
at /run/kata-extensions/<name>/
The unit uses ConditionKernelCommandLine=kata.extension.%i.verity_params so
it only activates when that extension is configured for the VM, and is
ordered before kata-agent.service so extension contents are available when
the agent starts.
A systemd generator (kata-extension-mount-generator) instantiates the
template for every kata.extension.<name>.verity_params entry on the kernel
command line. Because the runtime emits one such entry per configured
guest_extension_images, extensions are enabled purely from the cmdline and
adding a new one needs no change to the rootfs build.
The agent Makefile is updated to install the unit, scripts and generator.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Mirror the Rust runtime's guest_extension_images support in the Go runtime:
- hypervisor.go: add GuestExtensionImage struct and GuestExtensionImages field to
HypervisorConfig.
- config.go: parse [[hypervisor.qemu.guest_extension_images]] TOML sections
via toGuestExtensionImages(), validating that every extension has a non-empty
name and valid dm-verity parameters.
- qemu.go: in buildDevices(), attach each extra image as a virtio-blk
drive with ID extension-<name>; in kernelParameters(), append
kata.extension.<name>.verity_params=... for each extra image.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a serial_override field to BlockConfig and DeviceVirtioBlk so that
extension images get a deterministic QEMU serial (extension-<name>) instead of
the auto-generated image-<id>. This lets the guest discover each extension
via /dev/disk/by-id/virtio-extension-<name>.
Wire up the full cold-plug path for guest_extension_images:
- sandbox.rs: iterate boot_info.guest_extension_images, create read-only
BlockConfig entries with the correct serial_override, and push them
as ResourceConfig::GuestExtensionImage.
- resource lib.rs / manager_inner.rs: handle the new GuestExtensionImage variant
by delegating to the existing block device path.
- cmdline_generator.rs: use serial_override when present for the QEMU
serial= parameter; append kata.extension.<name>.verity_params=... to the
kernel command line for each extra image.
- inner.rs: propagate serial_override from BlockConfig to the cmdline
generator's add_block_device().
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Introduce the GuestExtensionImage type to describe extension block-device images
(e.g. CoCo guest components) that are cold-plugged into the VM
alongside the main rootfs.
Each GuestExtensionImage carries a short name (used as the virtio-blk serial
and in kernel cmdline verity parameters), a host-side path, and
dm-verity parameters (required -- extension images must be integrity-
protected).
A new guest_extension_images Vec is added to BootInfo so that both the Rust
and Go runtimes can iterate the configured extensions when building the
VM device list and kernel command line.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The genpolicy guest-pull check now fails closed when an image carries
supplemental groups that containerd won't reproduce while pulling the
layers inside the guest, and quay.io/prometheus/busybox:latest happens
to ship gid 10 (wheel). pod-number-cpu.yaml was simply missed when the
other busybox manifests got their securityContext, so genpolicy bails
out on the expected/actual gid mismatch.
Let's declare supplementalGroups: [10] explicitly, just like we already
do for the other busybox-based pods, so the generated policy lines up
with what containerd actually applies.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Since the switch to per-component tarballs, versions.yaml was no longer
installed on host nodes by kata-deploy (DaemonSet/Helm). Previously it
was bundled into the merged kata-static.tar.zst; now it must be shipped
explicitly.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a check-agent-policy-coverage job to static-checks.yaml
that runs ci/check_agent_policy_coverage.sh on every pull request.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add ci/check_agent_policy_coverage.sh, a static check that asserts
every RPC method declared in the AgentService block of agent.proto has:
1. A policy gate (is_allowed or do_set_policy call) in the corresponding
handler in src/agent/src/rpc.rs.
2. A default rule entry in src/tools/genpolicy/rules.rego.
This prevents future handlers from silently bypassing the policy
boundary — the gap that prompted this work was MemAgentMemcgSet and
MemAgentCompactSet, which previously had no is_allowed call and no
rules.rego entry.
The script is self-contained and can be run standalone from any
directory in the repository tree.
Generated-By: IBM Bob
Assisted-by: Google AI Mode
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
ResizeVolumeRequest is declared in agent.proto and has a rules.rego
default entry, but the kata-agent had no corresponding ttrpc handler
in the AgentService impl block. This meant the CI policy-coverage
check (ci/check_agent_policy_coverage.sh) correctly flagged it as
ungated.
Add a resize_volume handler that:
- calls is_allowed(&req) so the policy boundary is enforced, and
- returns UNIMPLEMENTED, reflecting that the kata-agent has never
carried out volume resizes itself (runtime-rs exercises this RPC
via its own agent client; the kata-agent path was simply absent).
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Six AgentService RPC request types were missing default rule entries in
rules.rego, meaning genpolicy-generated policies would not include them
in their output and the entries would be absent from reference policies.
All six already have is_allowed gates in the agent RPC handlers.
Add the missing defaults and set them all to false
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add default-deny entries for MemAgentMemcgConfig and MemAgentCompactConfig
to rules.rego, and the corresponding allow entries to the two reference
OPA policies (allow-all.rego and allow-all-except-exec-process.rego).
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Bump path-absolutize in rustjail from 1.2.1 to 3.1.1
Transitive dependency path-dedot is updated from 1.2.4 to 3.1.1,
which replaces the lazy_static dependency with once_cell, and the
slash-formatter dependency is dropped entirely.
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Update the workspace thiserror dependency to 2.0.18 and
normalize direct workspace crate pins to use the workspace
version.
Fix thiserror 2.x format-string breakage in Dragonball by
replacing mixed positional/implicit arguments in #[error(...)]
attributes with explicit named arguments.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
I'm not sure why we aren't catching this elsewhere, but
when running checks I saw the unused
import warning, so thought I would fix it
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The mem_agent_memcg_set and mem_agent_compact_set handlers in rpc.rs
were the only AgentService methods that did not call is_allowed() before
executing. Because the policy engine was never consulted, no OPA/Rego
policy — including the strict default-deny policy generated by genpolicy
— could deny these requests.
Add is_allowed(&config).await? as the first statement of both handlers,
matching the pattern used by every other handler in the file.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This commit enhances the VFIO-AP testing suite by adding
support for cold-plug testing in runtime-rs and updating
configuration files to support the new cold_plug_vfio parameter.
Changes include:
1. Configuration updates:
- Added cold_plug_vfio parameter to configuration-qemu-runtime-rs.toml.in
with default value "no-port" (disabled)
- Added cold_plug_vfio parameter to configuration-qemu-se-runtime-rs.toml.in
with value "root-port" (enabled for Secure Execution)
2. Test enhancements:
- Updated setup_hotplug() to explicitly set cold_plug_vfio to "no-port"
for runtime-rs to ensure hotplug-only mode
- Implemented setup_coldplug() for runtime-rs (previously unsupported)
to enable cold-plug testing with "root-port" configuration
- Added new test case (Test 4) for runtime-rs VFIO-AP cold-plug
functionality using zcrypttest validation
This enables comprehensive testing of both hot-plug and cold-plug VFIO
device assignment scenarios for s390x CEX devices in runtime-rs.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add support for cold-plugging raw VFIO devices in standalone
container scenarios (e.g., `ctr --device /dev/vfio/0`) where
devices are specified directly in the OCI spec rather than
through K8S CDI/device plugins.
This implements a fallback path that:
- Reads linux.devices from the OCI spec in the container bundle
- Identifies character devices under /dev/vfio
- Cold-plugs them before VM boot using the configured cold_plug_vfio mode
- Handles VFIO-AP devices specially by using NoPort topology
- Mirrors the Go runtime's coldOrHotPlugVFIO() behavior
The implementation:
- Adds prepare_coldplug_raw_vfio_devices() method to VirtSandbox
- Exports is_vfio_ap_device() helper for device type detection
- Integrates with existing CDI device cold-plug flow
- Only activates when cold_plug_vfio is configured (not "no-port")
- Skips processing when bundle path is unavailable or OCI spec missing
Plus, update resource manager:
- Skipping hotplug for cold-plugged mediated AP devices
- Handling the AP devices to get exposed to the agent
This enables VFIO device passthrough for standalone containers while
maintaining compatibility with Kubernetes pod resource API workflows.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add runtime-rs support for s390x VFIO-AP mediated devices across
QEMU device creation, hotplug/coldplug flows, and agent device
translation.
Introduce a dedicated vfio-ap QEMU device generator using the sysfsdev
path and route mediated AP devices through it instead of the PCI VFIO
path. Skip PCI-specific guest path handling for these devices since
they do not use PCI root ports or guest PCI addresses.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This commit adds a fallback mechanism in `get_image_reference()` to
handle SingleContainer scenarios where no container type annotation
is present.
The issue occurs when standalone container runtimes (like nerdctl)
create containers without the Kubernetes container type annotation.
In such cases, `container_type_with_id()` returns `SingleContainer`
as the default, but `get_image_reference()` previously only checked
for image names when a container type annotation was explicitly set
to either PodSandbox or PodContainer.
This caused image reference lookups to fail for standalone containers,
even though the image name annotations (io.kubernetes.cri.image-name
or io.kubernetes.cri-o.ImageName) were present in the spec.
The fallback logic now checks for image name annotations directly
when no container type annotation is found, supporting both:
- io.kubernetes.cri.image-name (standard CRI)
- io.kubernetes.cri-o.ImageName (CRI-O specific)
This maintains backward compatibility with Kubernetes pod scenarios
while enabling support for standalone container runtimes that don't
provide container type annotations.
Test cases have been added to verify the fallback behavior for both
CRI and CRI-O image name annotations in SingleContainer scenarios.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add e2fsprogs to the Azure Linux UVM rootfs package list.
This enables using the block-plain emptyDir feature for the clh-azure
scenario.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add Kubernetes integration coverage for block-plain emptyDir storage.
The test enables block-plain emptyDirs, verifies that the guest sees an
ext4 block-backed mount, checks kubelet eviction when the emptyDir
sizeLimit is exceeded, and checks sparse image reclaim through discard.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Include the mkfs.ext4 binary and runtime libraries in NVIDIA guest
rootfs builds.
The agent formats block-plain emptyDir disk images inside the guest, so
NVIDIA rootfs variants need the same e2fsprogs pieces that CDH secure
storage already depends on.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Replace the encrypted-emptyDir boolean setting with an emptydir_type
setting that can describe shared-fs, block-encrypted, and block-plain
emptyDirs.
Add policy storage templates for block encrypted and block plain emptyDirs
with the create-filesystem driver option. Plain block emptyDirs also carry
the discard mount option. The block storage source pattern is relaxed to
match the runtime-rs values observed for block devices.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Teach the agent to create an ext4 filesystem for fresh block volumes
when the storage request carries the block create-filesystem driver
option.
The create-filesystem option is the freshness and format signal, while
storage.fstype remains the filesystem type source. Plain block emptyDirs
are formatted directly by the agent. Encrypted block emptyDirs must carry
the same create-filesystem signal together with ephemeral encryption and
continue through CDH secure_mount.
Reject ephemeral encryption without the create-filesystem signal so
existing direct block storage cannot accidentally enter the fresh emptyDir
path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Pass block-device discard support through the runtime-rs QEMU stack.
Block device configuration now reaches both QEMU startup and hotplug
paths. The backend receives discard=unmap, and virtio-blk frontend
devices advertise discard support when requested. The SCSI hotplug path
keeps its existing frontend arguments.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Pass block-device discard support through the Go QEMU stack.
Block drives can now carry a DiscardUnmap request into govmm. QEMU
command-line and QMP hotplug paths set discard=unmap on the backend and
enable discard on virtio-blk frontends, while leaving SCSI frontend
arguments unchanged.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add runtime-rs support for the block-plain emptyDir mode.
Disk-backed Kubernetes emptyDir mounts remain bind mounts so the block
emptyDir volume handler can intercept them. The handler creates a sparse
disk.img in the kubelet emptyDir directory, attaches it as a block device,
and rewrites the container mount to the agent-visible block storage path.
The same handler now covers encrypted and plain block emptyDirs. Fresh
block emptyDirs request filesystem creation through a dedicated metadata
flag. Plain emptyDirs add discard support, while encrypted emptyDirs keep
the existing ephemeral encryption metadata.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add Go runtime support for the block-plain emptyDir mode.
Disk-backed Kubernetes emptyDir mounts remain bind mounts so the block
emptyDir handling path can intercept them. The runtime creates a sparse
disk.img in the kubelet emptyDir directory and records direct-volume
metadata for the agent-visible block storage path.
Fresh block emptyDirs request filesystem creation through a dedicated
metadata flag. Plain emptyDirs also record discard support on the block
device. Encrypted emptyDirs keep the existing ephemeral encryption
metadata and carry the same filesystem-creation signal.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The IBM SEL runtime requires a larger overhead_memory budget than
other TEE runtimes (SNP, TDX) because the kernel command line baked
into the SE image sets:
swiotlb=262144 (262144 × 2 KiB slots = 512 MiB)
This buffer is pre-allocated at boot from the guest's physical RAM
before any workload runs.
With static_sandbox_resource_mgmt = true the VM gets:
vm_memory = overhead_memory + container_limit
In k8s-limit-range.bats, DEFOVERHEADMEMSZ_TEE (128 MiB) resulted in
a 256 MiB VM when a container with a 128 MiB memory limit was scheduled
— far too small to even fit the swiotlb allocation, causing boot failure.
In a similar way, the failure is also observed for k8s-oom.bats.
Introduce DEFOVERHEADMEMSZ_TEE_SE := 768 MiB, sized to cover:
- 512 MiB swiotlb bounce buffer (fixed by sealed kernel cmdline)
- ~128 MiB SE kernel + initrd + agent baseline
- ~128 MiB headroom for other stuff
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Update the two affected entries in required-tests.yaml accordingly
so the gatekeeper keeps matching them instead of blocking subsequent
PRs after this one merges.
Co-authored-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Enable devicemapper (dm-verity) support in CI by forwarding the
USE_DEVMAPPER environment variable to the agent build step across
all architectures (amd64, arm64, s390x).
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The deploy will read EROFS_SNAPSHOTTER_MODE and EROFS_DMVERITY from
the environment to enable dmverity_mode and enable_dmverity in the
containerd erofs snapshotter/differ config.
Add validation for the mode value and use an explicit 300s timeout
for node-readiness checks during kata-deply in github CI.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
