With #12373 and #13280 in, emptyDirs are now shared from the host with
virtio-fs when supported, and properly accounted for by Kubelet.
So we can extend the sizeLimit-based pod eviction test introduced by #13127
to virtio-fs-based emptyDirs (originally only for block-based emptyDirs).
Generated-by: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The genpolicy guest-pull check now fails closed when an image carries
supplemental groups that containerd won't reproduce while pulling the
layers inside the guest, and quay.io/prometheus/busybox:latest happens
to ship gid 10 (wheel). pod-number-cpu.yaml was simply missed when the
other busybox manifests got their securityContext, so genpolicy bails
out on the expected/actual gid mismatch.
Let's declare supplementalGroups: [10] explicitly, just like we already
do for the other busybox-based pods, so the generated policy lines up
with what containerd actually applies.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Since the switch to per-component tarballs, versions.yaml was no longer
installed on host nodes by kata-deploy (DaemonSet/Helm). Previously it
was bundled into the merged kata-static.tar.zst; now it must be shipped
explicitly.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a check-agent-policy-coverage job to static-checks.yaml
that runs ci/check_agent_policy_coverage.sh on every pull request.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add ci/check_agent_policy_coverage.sh, a static check that asserts
every RPC method declared in the AgentService block of agent.proto has:
1. A policy gate (is_allowed or do_set_policy call) in the corresponding
handler in src/agent/src/rpc.rs.
2. A default rule entry in src/tools/genpolicy/rules.rego.
This prevents future handlers from silently bypassing the policy
boundary — the gap that prompted this work was MemAgentMemcgSet and
MemAgentCompactSet, which previously had no is_allowed call and no
rules.rego entry.
The script is self-contained and can be run standalone from any
directory in the repository tree.
Generated-By: IBM Bob
Assisted-by: Google AI Mode
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
ResizeVolumeRequest is declared in agent.proto and has a rules.rego
default entry, but the kata-agent had no corresponding ttrpc handler
in the AgentService impl block. This meant the CI policy-coverage
check (ci/check_agent_policy_coverage.sh) correctly flagged it as
ungated.
Add a resize_volume handler that:
- calls is_allowed(&req) so the policy boundary is enforced, and
- returns UNIMPLEMENTED, reflecting that the kata-agent has never
carried out volume resizes itself (runtime-rs exercises this RPC
via its own agent client; the kata-agent path was simply absent).
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Six AgentService RPC request types were missing default rule entries in
rules.rego, meaning genpolicy-generated policies would not include them
in their output and the entries would be absent from reference policies.
All six already have is_allowed gates in the agent RPC handlers.
Add the missing defaults and set them all to false
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add default-deny entries for MemAgentMemcgConfig and MemAgentCompactConfig
to rules.rego, and the corresponding allow entries to the two reference
OPA policies (allow-all.rego and allow-all-except-exec-process.rego).
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Bump path-absolutize in rustjail from 1.2.1 to 3.1.1
Transitive dependency path-dedot is updated from 1.2.4 to 3.1.1,
which replaces the lazy_static dependency with once_cell, and the
slash-formatter dependency is dropped entirely.
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Update the workspace thiserror dependency to 2.0.18 and
normalize direct workspace crate pins to use the workspace
version.
Fix thiserror 2.x format-string breakage in Dragonball by
replacing mixed positional/implicit arguments in #[error(...)]
attributes with explicit named arguments.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
I'm not sure why we aren't catching this elsewhere, but
when running checks I saw the unused
import warning, so thought I would fix it
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The mem_agent_memcg_set and mem_agent_compact_set handlers in rpc.rs
were the only AgentService methods that did not call is_allowed() before
executing. Because the policy engine was never consulted, no OPA/Rego
policy — including the strict default-deny policy generated by genpolicy
— could deny these requests.
Add is_allowed(&config).await? as the first statement of both handlers,
matching the pattern used by every other handler in the file.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This commit enhances the VFIO-AP testing suite by adding
support for cold-plug testing in runtime-rs and updating
configuration files to support the new cold_plug_vfio parameter.
Changes include:
1. Configuration updates:
- Added cold_plug_vfio parameter to configuration-qemu-runtime-rs.toml.in
with default value "no-port" (disabled)
- Added cold_plug_vfio parameter to configuration-qemu-se-runtime-rs.toml.in
with value "root-port" (enabled for Secure Execution)
2. Test enhancements:
- Updated setup_hotplug() to explicitly set cold_plug_vfio to "no-port"
for runtime-rs to ensure hotplug-only mode
- Implemented setup_coldplug() for runtime-rs (previously unsupported)
to enable cold-plug testing with "root-port" configuration
- Added new test case (Test 4) for runtime-rs VFIO-AP cold-plug
functionality using zcrypttest validation
This enables comprehensive testing of both hot-plug and cold-plug VFIO
device assignment scenarios for s390x CEX devices in runtime-rs.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add support for cold-plugging raw VFIO devices in standalone
container scenarios (e.g., `ctr --device /dev/vfio/0`) where
devices are specified directly in the OCI spec rather than
through K8S CDI/device plugins.
This implements a fallback path that:
- Reads linux.devices from the OCI spec in the container bundle
- Identifies character devices under /dev/vfio
- Cold-plugs them before VM boot using the configured cold_plug_vfio mode
- Handles VFIO-AP devices specially by using NoPort topology
- Mirrors the Go runtime's coldOrHotPlugVFIO() behavior
The implementation:
- Adds prepare_coldplug_raw_vfio_devices() method to VirtSandbox
- Exports is_vfio_ap_device() helper for device type detection
- Integrates with existing CDI device cold-plug flow
- Only activates when cold_plug_vfio is configured (not "no-port")
- Skips processing when bundle path is unavailable or OCI spec missing
Plus, update resource manager:
- Skipping hotplug for cold-plugged mediated AP devices
- Handling the AP devices to get exposed to the agent
This enables VFIO device passthrough for standalone containers while
maintaining compatibility with Kubernetes pod resource API workflows.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add runtime-rs support for s390x VFIO-AP mediated devices across
QEMU device creation, hotplug/coldplug flows, and agent device
translation.
Introduce a dedicated vfio-ap QEMU device generator using the sysfsdev
path and route mediated AP devices through it instead of the PCI VFIO
path. Skip PCI-specific guest path handling for these devices since
they do not use PCI root ports or guest PCI addresses.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This commit adds a fallback mechanism in `get_image_reference()` to
handle SingleContainer scenarios where no container type annotation
is present.
The issue occurs when standalone container runtimes (like nerdctl)
create containers without the Kubernetes container type annotation.
In such cases, `container_type_with_id()` returns `SingleContainer`
as the default, but `get_image_reference()` previously only checked
for image names when a container type annotation was explicitly set
to either PodSandbox or PodContainer.
This caused image reference lookups to fail for standalone containers,
even though the image name annotations (io.kubernetes.cri.image-name
or io.kubernetes.cri-o.ImageName) were present in the spec.
The fallback logic now checks for image name annotations directly
when no container type annotation is found, supporting both:
- io.kubernetes.cri.image-name (standard CRI)
- io.kubernetes.cri-o.ImageName (CRI-O specific)
This maintains backward compatibility with Kubernetes pod scenarios
while enabling support for standalone container runtimes that don't
provide container type annotations.
Test cases have been added to verify the fallback behavior for both
CRI and CRI-O image name annotations in SingleContainer scenarios.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add e2fsprogs to the Azure Linux UVM rootfs package list.
This enables using the block-plain emptyDir feature for the clh-azure
scenario.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add Kubernetes integration coverage for block-plain emptyDir storage.
The test enables block-plain emptyDirs, verifies that the guest sees an
ext4 block-backed mount, checks kubelet eviction when the emptyDir
sizeLimit is exceeded, and checks sparse image reclaim through discard.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Include the mkfs.ext4 binary and runtime libraries in NVIDIA guest
rootfs builds.
The agent formats block-plain emptyDir disk images inside the guest, so
NVIDIA rootfs variants need the same e2fsprogs pieces that CDH secure
storage already depends on.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Replace the encrypted-emptyDir boolean setting with an emptydir_type
setting that can describe shared-fs, block-encrypted, and block-plain
emptyDirs.
Add policy storage templates for block encrypted and block plain emptyDirs
with the create-filesystem driver option. Plain block emptyDirs also carry
the discard mount option. The block storage source pattern is relaxed to
match the runtime-rs values observed for block devices.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Teach the agent to create an ext4 filesystem for fresh block volumes
when the storage request carries the block create-filesystem driver
option.
The create-filesystem option is the freshness and format signal, while
storage.fstype remains the filesystem type source. Plain block emptyDirs
are formatted directly by the agent. Encrypted block emptyDirs must carry
the same create-filesystem signal together with ephemeral encryption and
continue through CDH secure_mount.
Reject ephemeral encryption without the create-filesystem signal so
existing direct block storage cannot accidentally enter the fresh emptyDir
path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Pass block-device discard support through the runtime-rs QEMU stack.
Block device configuration now reaches both QEMU startup and hotplug
paths. The backend receives discard=unmap, and virtio-blk frontend
devices advertise discard support when requested. The SCSI hotplug path
keeps its existing frontend arguments.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Pass block-device discard support through the Go QEMU stack.
Block drives can now carry a DiscardUnmap request into govmm. QEMU
command-line and QMP hotplug paths set discard=unmap on the backend and
enable discard on virtio-blk frontends, while leaving SCSI frontend
arguments unchanged.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add runtime-rs support for the block-plain emptyDir mode.
Disk-backed Kubernetes emptyDir mounts remain bind mounts so the block
emptyDir volume handler can intercept them. The handler creates a sparse
disk.img in the kubelet emptyDir directory, attaches it as a block device,
and rewrites the container mount to the agent-visible block storage path.
The same handler now covers encrypted and plain block emptyDirs. Fresh
block emptyDirs request filesystem creation through a dedicated metadata
flag. Plain emptyDirs add discard support, while encrypted emptyDirs keep
the existing ephemeral encryption metadata.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add Go runtime support for the block-plain emptyDir mode.
Disk-backed Kubernetes emptyDir mounts remain bind mounts so the block
emptyDir handling path can intercept them. The runtime creates a sparse
disk.img in the kubelet emptyDir directory and records direct-volume
metadata for the agent-visible block storage path.
Fresh block emptyDirs request filesystem creation through a dedicated
metadata flag. Plain emptyDirs also record discard support on the block
device. Encrypted emptyDirs keep the existing ephemeral encryption
metadata and carry the same filesystem-creation signal.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The IBM SEL runtime requires a larger overhead_memory budget than
other TEE runtimes (SNP, TDX) because the kernel command line baked
into the SE image sets:
swiotlb=262144 (262144 × 2 KiB slots = 512 MiB)
This buffer is pre-allocated at boot from the guest's physical RAM
before any workload runs.
With static_sandbox_resource_mgmt = true the VM gets:
vm_memory = overhead_memory + container_limit
In k8s-limit-range.bats, DEFOVERHEADMEMSZ_TEE (128 MiB) resulted in
a 256 MiB VM when a container with a 128 MiB memory limit was scheduled
— far too small to even fit the swiotlb allocation, causing boot failure.
In a similar way, the failure is also observed for k8s-oom.bats.
Introduce DEFOVERHEADMEMSZ_TEE_SE := 768 MiB, sized to cover:
- 512 MiB swiotlb bounce buffer (fixed by sealed kernel cmdline)
- ~128 MiB SE kernel + initrd + agent baseline
- ~128 MiB headroom for other stuff
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Update the two affected entries in required-tests.yaml accordingly
so the gatekeeper keeps matching them instead of blocking subsequent
PRs after this one merges.
Co-authored-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Enable devicemapper (dm-verity) support in CI by forwarding the
USE_DEVMAPPER environment variable to the agent build step across
all architectures (amd64, arm64, s390x).
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The deploy will read EROFS_SNAPSHOTTER_MODE and EROFS_DMVERITY from
the environment to enable dmverity_mode and enable_dmverity in the
containerd erofs snapshotter/differ config.
Add validation for the mode value and use an explicit 300s timeout
for node-readiness checks during kata-deply in github CI.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Install libdevmapper-dev and pkg-config in the agent build container
so devicemapper-sys can link against libdevmapper. Add the GNU libc
rustup target alongside musl since USE_DEVMAPPER forces LIBC=gnu.
Forward USE_DEVMAPPER through build.sh and build-static-agent.sh.
And you can compile the device mapper in kata-agent as below:
```
$ make LIBC=gnu USE_DEVMAPPER=yes
```
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Pass USE_DEVMAPPER through the Docker environment in local build
scripts. Extract the OCI tag sanitization logic into a public helper
of sanitize_tag_component to keep push and pull paths consistent.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Expose erofsSnapshotterMode in the helm chart values and render it as
the EROFS_SNAPSHOTTER_MODE environment variable in the kata-deploy pod.
Update gha-run-k8s-common.sh to load dm-mod/dm-verity kernel modules
and configure the erofs default size when the mode is "integrity".
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add k8s-erofs-dmverity.bats integration test that verifies dm-verity
protected EROFS layers work end-to-end, and register the integrity
mode in the CoCo test matrix.
This commit introduces two new files to enable it:
- k8s-erofs-dmverity.bats
- pod-erofs-dmverity-probe.yaml
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Replace the agent's inline devicemapper implementation with the libs
kata-types::dmverity module. The agent's devicemapper Cargo feature
now forwards to kata-types/devicemapper, removing the direct
libdevmapper link dependency from the agent crate. Gate all dm-verity
imports, constants, and call sites behind libdevmapper.
Add USE_DEVMAPPER Makefile variable (default no) that appends the
devicemapper feature flag and forces LIBC=gnu when enabled.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
This commit is just a enhancement without any functionality changes.
Replace the sequential loop in handle_multi_layer_erofs_group with
join_all-based concurrent mounting. Base device paths and mount
directories are pre-resolved before spawning futures to avoid lock
contention. On partial failure, successfully mounted layers are
unmounted and dm-verity devices cleaned up before propagating the
error.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce a new `dmverity` module in kata-types that provides dm-verity
device creation, destruction and lifecycle management via devicemapper
ioctls. The module is conditionally compiled behind the `devicemapper`
feature flag, which also pulls in tokio for async device-node polling.
The workspace devicemapper dependency is pinned to a specific git
revision for reproducible builds.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
When a GPT-partitioned VMDK is split into individual partition images,
padding files may be generated between partitions to maintain correct
byte offsets. These were not tracked for cleanup, leading to stale
temporary files after container removal.
Iterate over the partition layout and check for pad-{idx}.img files
alongside the head image; add any that exist to gpt_metadata_paths
so they are removed during teardown.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Wire the dm-verity helpers into the layer mount flow so that GPT
partitions carrying verity metadata are mounted through a verified
device-mapper target instead of the raw partition.
Refactor wait_and_mount_layer to resolve partition path and verity
device as separate steps: create a dm-verity device when
X-kata.dmverity-enabled=true is set, fall back to direct partition
mount otherwise, and return the verity device path for cleanup
tracking.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
