As the machine is off and I cannot connect to the BMC to turn
it back on, let's avoid PRs running those tests for now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The `hotplug_vfio_on_root_bus` configuration option is no longer needed
in the runtime-rs.
Remove the option from:
- `DeviceInfo` struct and its `hotplug_vfio_on_root_bus` field
- `PCIeTopology` struct and its construction in `topology.rs`
- `HypervisorInfo` struct and its `hotplug_vfio_on_root_bus` field
- `KATA_ANNO_CFG_HYPERVISOR_HOTPLUG_VFIO_ON_ROOT_BUS` annotation constant
and its handler in `Annotation::apply()`
- DragonBall validation check in `ConfigPlugin::adjust_config()`
- All runtime-rs TOML config templates (qemu, fc, nvidia GPU variants)
- Test fixture TOML files
Update the documentation having a reference to it.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Align the composable VM images proposal with the naming convention used
throughout the implementation: the generic mechanism and its data model
are now "guest extension image" (config key `guest_extension_images`,
struct `GuestExtensionImage`), and the runtime artifacts use the matching
"extension" vocabulary (`/run/kata-extensions/<name>`,
`kata.extension.<name>.verity_params`, `kata-extension-mount@.service`,
the `extension-<name>` virtio-blk serial / dm-verity target, and the
`kata-containers-coco-extension.img` image).
It also documents how template-unit instances are enabled: a systemd
generator instantiates kata-extension-mount@<name>.service for each
extension on the kernel command line, so the rootfs build stays
extension-agnostic.
This replaces the earlier mixed "extra_images"/"addon" terminology and
addresses the naming-consistency review feedback on the series.
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Generate etc/kata-extensions/components.toml inside the CoCo extension rootfs
before building the image, so the manifest becomes part of the
dm-verity-measured erofs image.
The manifest declares the extension's path-only components (ocicrypt config,
pause bundle) and the launchable processes (attestation-agent,
confidential-data-hub, api-server-rest) with paths relative to the extension
mount point (/run/kata-extensions/coco) and `${var}` placeholders resolved
by kata-agent at runtime. This is the data the agent consumes to launch
the CoCo guest components without per-bundle code changes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Make the agent's handling of CoCo guest components data-driven so that a
new extension bundle can be introduced without changing agent code.
Add a new `extension` module defining a generic component manifest contract.
When a CoCo extension image is mounted at /run/kata-extensions/coco/, the agent
reads etc/kata-extensions/components.toml from it to discover:
- path-only components (looked up by id under the `[paths]` table), used
for the ocicrypt config and the pause bundle, and
- launchable processes (`[[process]]` entries) describing the binary
path, args, optional args, config file, environment, socket to wait on
and timeout.
`${var}` tokens in process fields are substituted from a fixed,
documented context the agent builds at runtime (attestation socket/uri,
config paths, rest-api features, initdata toml path, launch timeout,
resolved ocicrypt config path). Referencing an unknown variable is a
hard error, and manifest paths are validated against traversal, so the
contract stays fail-closed.
Processes are gated by a numeric `level` mapped from
guest_components_procs (AttestationAgent < ConfidentialDataHub <
ApiServerRest), preserving the existing implication semantics.
When no extension is mounted, launch_plan() returns None and the agent uses
a built-in plan that reproduces the legacy behaviour byte-for-byte
(binaries under /usr/local/bin, /etc/ocicrypt_config.json, /pause_bundle).
Monolithic / non-split confidential images are therefore unaffected.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add rootfs-image-coco-extension to the build-asset-rootfs matrix in the
amd64, arm64 and s390x static tarball workflows so the CoCo extension image
is built as part of CI. Keep building rootfs-image-confidential as well:
the runtime defaults to base + extension, but we retain the monolithic
confidential image for now during the transition.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add install_image_coco_extension() to kata-deploy-binaries.sh which:
- Unpacks the CoCo guest components and pause image tarballs into a
temporary rootfs directory (under the repo root so Docker-in-Docker
volume mounts resolve correctly)
- Calls image_builder.sh with USE_DOCKER=1, FS_TYPE=erofs,
MEASURED_ROOTFS=yes, SKIP_DAX_HEADER=yes, and SKIP_ROOTFS_CHECK=yes
to produce kata-containers-coco-extension.img + root_hash_coco-extension.txt
Add the rootfs-image-coco-extension-tarball Makefile target with
dependencies on pause-image-tarball and coco-guest-components-tarball.
The standard confidential image keeps pause-image and
coco-guest-components for now so it remains a usable standalone
monolithic CoCo image during the transition to base + extension.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
During the transition to composable (base + extension) CoCo images, run the
two runtimes on different image layouts so CI exercises both paths via
the existing qemu-{snp,tdx,coco-dev} (Go) and -runtime-rs (Rust) shims:
- runtime-rs (Rust) CoCo templates use the standard base image plus a
CoCo extension extra_image:
image = "@IMAGEPATH@" (was @IMAGECONFIDENTIALPATH@)
[[hypervisor.qemu.guest_extension_images]]
name = "coco"
path = "@COCOIMAGEPATH@"
verity_params = "@COCOVERITYPARAMS@"
COCOIMAGENAME/COCOIMAGEPATH/COCOVERITYPARAMS are added to the
runtime-rs Makefile only.
- runtime (Go) CoCo templates keep the monolithic confidential image
(@IMAGECONFIDENTIALPATH@) and pull rootfs-image-confidential.
shim-components.json reflects this: Go CoCo shims depend on
rootfs-image-confidential while the runtime-rs CoCo shims depend on
rootfs-image + rootfs-image-coco-extension.
shim-v2/build.sh feeds each runtime its own dm-verity params: Rust gets
the measured base image hash (@KERNELVERITYPARAMS@) plus the extension hash
(@COCOVERITYPARAMS@); Go gets the monolithic confidential image hash
(@KERNELVERITYPARAMS@). This is wired per make invocation so the same
@KERNELVERITYPARAMS@ placeholder resolves correctly for each.
Once the split path is validated we can flip the Go templates too and
drop the monolithic confidential image.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Extension images (e.g. the CoCo guest components extension) are not full root
filesystems -- they contain only the binaries and configuration that
get bind-mounted into the real rootfs at boot. The existing
check_rootfs() validation requires /sbin/init and systemd, which are
not present in extension images.
Add a SKIP_ROOTFS_CHECK environment variable that, when set to "yes",
bypasses the check_rootfs() call. Forward the variable into the
container environment when using the Docker-based build path so it
works in both direct and containerised invocations.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a systemd template unit kata-extension-mount@.service and companion
helper scripts (kata-extension-mount.sh, kata-extension-umount.sh) that handle
the guest-side setup of extension block device images.
The mount script:
- Discovers the block device by matching its virtio serial (extension-<name>)
- Reads verity parameters from kata.extension.<name>.verity_params on the
kernel command line
- Creates a dm-verity device and mounts the EROFS filesystem read-only
at /run/kata-extensions/<name>/
The unit uses ConditionKernelCommandLine=kata.extension.%i.verity_params so
it only activates when that extension is configured for the VM, and is
ordered before kata-agent.service so extension contents are available when
the agent starts.
A systemd generator (kata-extension-mount-generator) instantiates the
template for every kata.extension.<name>.verity_params entry on the kernel
command line. Because the runtime emits one such entry per configured
guest_extension_images, extensions are enabled purely from the cmdline and
adding a new one needs no change to the rootfs build.
The agent Makefile is updated to install the unit, scripts and generator.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Mirror the Rust runtime's guest_extension_images support in the Go runtime:
- hypervisor.go: add GuestExtensionImage struct and GuestExtensionImages field to
HypervisorConfig.
- config.go: parse [[hypervisor.qemu.guest_extension_images]] TOML sections
via toGuestExtensionImages(), validating that every extension has a non-empty
name and valid dm-verity parameters.
- qemu.go: in buildDevices(), attach each extra image as a virtio-blk
drive with ID extension-<name>; in kernelParameters(), append
kata.extension.<name>.verity_params=... for each extra image.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a serial_override field to BlockConfig and DeviceVirtioBlk so that
extension images get a deterministic QEMU serial (extension-<name>) instead of
the auto-generated image-<id>. This lets the guest discover each extension
via /dev/disk/by-id/virtio-extension-<name>.
Wire up the full cold-plug path for guest_extension_images:
- sandbox.rs: iterate boot_info.guest_extension_images, create read-only
BlockConfig entries with the correct serial_override, and push them
as ResourceConfig::GuestExtensionImage.
- resource lib.rs / manager_inner.rs: handle the new GuestExtensionImage variant
by delegating to the existing block device path.
- cmdline_generator.rs: use serial_override when present for the QEMU
serial= parameter; append kata.extension.<name>.verity_params=... to the
kernel command line for each extra image.
- inner.rs: propagate serial_override from BlockConfig to the cmdline
generator's add_block_device().
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Introduce the GuestExtensionImage type to describe extension block-device images
(e.g. CoCo guest components) that are cold-plugged into the VM
alongside the main rootfs.
Each GuestExtensionImage carries a short name (used as the virtio-blk serial
and in kernel cmdline verity parameters), a host-side path, and
dm-verity parameters (required -- extension images must be integrity-
protected).
A new guest_extension_images Vec is added to BootInfo so that both the Rust
and Go runtimes can iterate the configured extensions when building the
VM device list and kernel command line.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
With #12373 and #13280 in, emptyDirs are now shared from the host with
virtio-fs when supported, and properly accounted for by Kubelet.
So we can extend the sizeLimit-based pod eviction test introduced by #13127
to virtio-fs-based emptyDirs (originally only for block-based emptyDirs).
Generated-by: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The genpolicy guest-pull check now fails closed when an image carries
supplemental groups that containerd won't reproduce while pulling the
layers inside the guest, and quay.io/prometheus/busybox:latest happens
to ship gid 10 (wheel). pod-number-cpu.yaml was simply missed when the
other busybox manifests got their securityContext, so genpolicy bails
out on the expected/actual gid mismatch.
Let's declare supplementalGroups: [10] explicitly, just like we already
do for the other busybox-based pods, so the generated policy lines up
with what containerd actually applies.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Bumps github.com/opencontainers/runc from v1.2.8 to v1.3.6 to fix
GHSA-xjvp-4fhw-gc47 (CVE, CVSS 4.8).
The libcontainer/cgroups/systemd sub-package was removed from runc in
v1.3.x and moved to the standalone github.com/opencontainers/cgroups
module. Update the import in pkg/resourcecontrol/utils_linux.go
accordingly, and promote github.com/opencontainers/cgroups to a direct
dependency.
Fixes: https://osv.dev/GHSA-xjvp-4fhw-gc47
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Since the switch to per-component tarballs, versions.yaml was no longer
installed on host nodes by kata-deploy (DaemonSet/Helm). Previously it
was bundled into the merged kata-static.tar.zst; now it must be shipped
explicitly.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a check-agent-policy-coverage job to static-checks.yaml
that runs ci/check_agent_policy_coverage.sh on every pull request.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add ci/check_agent_policy_coverage.sh, a static check that asserts
every RPC method declared in the AgentService block of agent.proto has:
1. A policy gate (is_allowed or do_set_policy call) in the corresponding
handler in src/agent/src/rpc.rs.
2. A default rule entry in src/tools/genpolicy/rules.rego.
This prevents future handlers from silently bypassing the policy
boundary — the gap that prompted this work was MemAgentMemcgSet and
MemAgentCompactSet, which previously had no is_allowed call and no
rules.rego entry.
The script is self-contained and can be run standalone from any
directory in the repository tree.
Generated-By: IBM Bob
Assisted-by: Google AI Mode
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
ResizeVolumeRequest is declared in agent.proto and has a rules.rego
default entry, but the kata-agent had no corresponding ttrpc handler
in the AgentService impl block. This meant the CI policy-coverage
check (ci/check_agent_policy_coverage.sh) correctly flagged it as
ungated.
Add a resize_volume handler that:
- calls is_allowed(&req) so the policy boundary is enforced, and
- returns UNIMPLEMENTED, reflecting that the kata-agent has never
carried out volume resizes itself (runtime-rs exercises this RPC
via its own agent client; the kata-agent path was simply absent).
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Six AgentService RPC request types were missing default rule entries in
rules.rego, meaning genpolicy-generated policies would not include them
in their output and the entries would be absent from reference policies.
All six already have is_allowed gates in the agent RPC handlers.
Add the missing defaults and set them all to false
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add default-deny entries for MemAgentMemcgConfig and MemAgentCompactConfig
to rules.rego, and the corresponding allow entries to the two reference
OPA policies (allow-all.rego and allow-all-except-exec-process.rego).
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Bump path-absolutize in rustjail from 1.2.1 to 3.1.1
Transitive dependency path-dedot is updated from 1.2.4 to 3.1.1,
which replaces the lazy_static dependency with once_cell, and the
slash-formatter dependency is dropped entirely.
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Update the workspace thiserror dependency to 2.0.18 and
normalize direct workspace crate pins to use the workspace
version.
Fix thiserror 2.x format-string breakage in Dragonball by
replacing mixed positional/implicit arguments in #[error(...)]
attributes with explicit named arguments.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
I'm not sure why we aren't catching this elsewhere, but
when running checks I saw the unused
import warning, so thought I would fix it
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The mem_agent_memcg_set and mem_agent_compact_set handlers in rpc.rs
were the only AgentService methods that did not call is_allowed() before
executing. Because the policy engine was never consulted, no OPA/Rego
policy — including the strict default-deny policy generated by genpolicy
— could deny these requests.
Add is_allowed(&config).await? as the first statement of both handlers,
matching the pattern used by every other handler in the file.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This commit enhances the VFIO-AP testing suite by adding
support for cold-plug testing in runtime-rs and updating
configuration files to support the new cold_plug_vfio parameter.
Changes include:
1. Configuration updates:
- Added cold_plug_vfio parameter to configuration-qemu-runtime-rs.toml.in
with default value "no-port" (disabled)
- Added cold_plug_vfio parameter to configuration-qemu-se-runtime-rs.toml.in
with value "root-port" (enabled for Secure Execution)
2. Test enhancements:
- Updated setup_hotplug() to explicitly set cold_plug_vfio to "no-port"
for runtime-rs to ensure hotplug-only mode
- Implemented setup_coldplug() for runtime-rs (previously unsupported)
to enable cold-plug testing with "root-port" configuration
- Added new test case (Test 4) for runtime-rs VFIO-AP cold-plug
functionality using zcrypttest validation
This enables comprehensive testing of both hot-plug and cold-plug VFIO
device assignment scenarios for s390x CEX devices in runtime-rs.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add support for cold-plugging raw VFIO devices in standalone
container scenarios (e.g., `ctr --device /dev/vfio/0`) where
devices are specified directly in the OCI spec rather than
through K8S CDI/device plugins.
This implements a fallback path that:
- Reads linux.devices from the OCI spec in the container bundle
- Identifies character devices under /dev/vfio
- Cold-plugs them before VM boot using the configured cold_plug_vfio mode
- Handles VFIO-AP devices specially by using NoPort topology
- Mirrors the Go runtime's coldOrHotPlugVFIO() behavior
The implementation:
- Adds prepare_coldplug_raw_vfio_devices() method to VirtSandbox
- Exports is_vfio_ap_device() helper for device type detection
- Integrates with existing CDI device cold-plug flow
- Only activates when cold_plug_vfio is configured (not "no-port")
- Skips processing when bundle path is unavailable or OCI spec missing
Plus, update resource manager:
- Skipping hotplug for cold-plugged mediated AP devices
- Handling the AP devices to get exposed to the agent
This enables VFIO device passthrough for standalone containers while
maintaining compatibility with Kubernetes pod resource API workflows.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add runtime-rs support for s390x VFIO-AP mediated devices across
QEMU device creation, hotplug/coldplug flows, and agent device
translation.
Introduce a dedicated vfio-ap QEMU device generator using the sysfsdev
path and route mediated AP devices through it instead of the PCI VFIO
path. Skip PCI-specific guest path handling for these devices since
they do not use PCI root ports or guest PCI addresses.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This commit adds a fallback mechanism in `get_image_reference()` to
handle SingleContainer scenarios where no container type annotation
is present.
The issue occurs when standalone container runtimes (like nerdctl)
create containers without the Kubernetes container type annotation.
In such cases, `container_type_with_id()` returns `SingleContainer`
as the default, but `get_image_reference()` previously only checked
for image names when a container type annotation was explicitly set
to either PodSandbox or PodContainer.
This caused image reference lookups to fail for standalone containers,
even though the image name annotations (io.kubernetes.cri.image-name
or io.kubernetes.cri-o.ImageName) were present in the spec.
The fallback logic now checks for image name annotations directly
when no container type annotation is found, supporting both:
- io.kubernetes.cri.image-name (standard CRI)
- io.kubernetes.cri-o.ImageName (CRI-O specific)
This maintains backward compatibility with Kubernetes pod scenarios
while enabling support for standalone container runtimes that don't
provide container type annotations.
Test cases have been added to verify the fallback behavior for both
CRI and CRI-O image name annotations in SingleContainer scenarios.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add e2fsprogs to the Azure Linux UVM rootfs package list.
This enables using the block-plain emptyDir feature for the clh-azure
scenario.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add Kubernetes integration coverage for block-plain emptyDir storage.
The test enables block-plain emptyDirs, verifies that the guest sees an
ext4 block-backed mount, checks kubelet eviction when the emptyDir
sizeLimit is exceeded, and checks sparse image reclaim through discard.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Include the mkfs.ext4 binary and runtime libraries in NVIDIA guest
rootfs builds.
The agent formats block-plain emptyDir disk images inside the guest, so
NVIDIA rootfs variants need the same e2fsprogs pieces that CDH secure
storage already depends on.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Replace the encrypted-emptyDir boolean setting with an emptydir_type
setting that can describe shared-fs, block-encrypted, and block-plain
emptyDirs.
Add policy storage templates for block encrypted and block plain emptyDirs
with the create-filesystem driver option. Plain block emptyDirs also carry
the discard mount option. The block storage source pattern is relaxed to
match the runtime-rs values observed for block devices.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Teach the agent to create an ext4 filesystem for fresh block volumes
when the storage request carries the block create-filesystem driver
option.
The create-filesystem option is the freshness and format signal, while
storage.fstype remains the filesystem type source. Plain block emptyDirs
are formatted directly by the agent. Encrypted block emptyDirs must carry
the same create-filesystem signal together with ephemeral encryption and
continue through CDH secure_mount.
Reject ephemeral encryption without the create-filesystem signal so
existing direct block storage cannot accidentally enter the fresh emptyDir
path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
