Block-backed emptyDir volumes are sized using the capacity of the host
filesystem rather than Kubernetes sizeLimit. On large filesystems,
ext4 metadata can consume enough space to trigger pod eviction.
Document the expected overhead, its effect on usable storage, current
mitigations, and the issue tracking potential improvements.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Update OS version and containerd references, and make a few minor
adjustments and clarifications. For the clarifications, move the
section "Deploy pods with Kata agent security policies" further up
so that the installation of the genpolicy tool has been elaborated
on before attempting to run NVIDIA CI tests locally.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Enabled TestContainerMemoryUpdate support for runtime-rs on s390x and
disabled static_sandbox_resource_mgmt to dynamically update the memory.
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
Integrate virtio-mem into memory hotplug and hotunplug operations:
- Modified hotplug_memory() to automatically detect virtio-mem device
and use resize instead of pc-dimm hotplug
- Modified hotunplug_memory() to use virtio-mem resize for memory
decrease with saturating_sub to prevent underflow
- Modified hotplugged_memory_size() to account for virtio-mem devices
- Falls back to pc-dimm when virtio-mem is not available
This completes the virtio-mem integration, enabling dynamic memory
management with virtio-mem-ccw on s390x while maintaining full
backward compatibility with pc-dimm.
Assisted-by: IBM Bob
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
Call setup_virtio_mem() during VM startup when enable_virtio_mem flag
is set:
- Added setup call after QMP connection is established
- Proper error handling with context
Assisted-by: IBM Bob
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
In 7afdfc7388 we
just disabled the undocumented-permissions zizmor warning,
but now with AI we can roll this fix more easily, so remove
the disabling of the rule and add the comments
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
When VFIO GPUs span both host NUMA nodes, emit guest NUMA nodes even
if vCPUs < node count (memory-only nodes), and pin each pxb-pcie to
bus=pcie.0 so QEMU does not attach the second expander to the first.
Also fix checkVCPUsPinningNUMA to tolerate vCPUs < NUMA node count.
The memory-only NUMA topology (emitted for pxb-pcie placement when
default_vcpus=1) creates more guest NUMA nodes than there are vCPUs;
the previous hard error silently tore down every sandbox. Now the
available vCPU(s) are pinned to the first node(s) and memory-only
nodes are skipped.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
As the machine is off and I cannot connect to the BMC to turn
it back on, let's avoid PRs running those tests for now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The `hotplug_vfio_on_root_bus` configuration option is no longer needed
in the runtime-rs.
Remove the option from:
- `DeviceInfo` struct and its `hotplug_vfio_on_root_bus` field
- `PCIeTopology` struct and its construction in `topology.rs`
- `HypervisorInfo` struct and its `hotplug_vfio_on_root_bus` field
- `KATA_ANNO_CFG_HYPERVISOR_HOTPLUG_VFIO_ON_ROOT_BUS` annotation constant
and its handler in `Annotation::apply()`
- DragonBall validation check in `ConfigPlugin::adjust_config()`
- All runtime-rs TOML config templates (qemu, fc, nvidia GPU variants)
- Test fixture TOML files
Update the documentation having a reference to it.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Align the composable VM images proposal with the naming convention used
throughout the implementation: the generic mechanism and its data model
are now "guest extension image" (config key `guest_extension_images`,
struct `GuestExtensionImage`), and the runtime artifacts use the matching
"extension" vocabulary (`/run/kata-extensions/<name>`,
`kata.extension.<name>.verity_params`, `kata-extension-mount@.service`,
the `extension-<name>` virtio-blk serial / dm-verity target, and the
`kata-containers-coco-extension.img` image).
It also documents how template-unit instances are enabled: a systemd
generator instantiates kata-extension-mount@<name>.service for each
extension on the kernel command line, so the rootfs build stays
extension-agnostic.
This replaces the earlier mixed "extra_images"/"addon" terminology and
addresses the naming-consistency review feedback on the series.
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Generate etc/kata-extensions/components.toml inside the CoCo extension rootfs
before building the image, so the manifest becomes part of the
dm-verity-measured erofs image.
The manifest declares the extension's path-only components (ocicrypt config,
pause bundle) and the launchable processes (attestation-agent,
confidential-data-hub, api-server-rest) with paths relative to the extension
mount point (/run/kata-extensions/coco) and `${var}` placeholders resolved
by kata-agent at runtime. This is the data the agent consumes to launch
the CoCo guest components without per-bundle code changes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Make the agent's handling of CoCo guest components data-driven so that a
new extension bundle can be introduced without changing agent code.
Add a new `extension` module defining a generic component manifest contract.
When a CoCo extension image is mounted at /run/kata-extensions/coco/, the agent
reads etc/kata-extensions/components.toml from it to discover:
- path-only components (looked up by id under the `[paths]` table), used
for the ocicrypt config and the pause bundle, and
- launchable processes (`[[process]]` entries) describing the binary
path, args, optional args, config file, environment, socket to wait on
and timeout.
`${var}` tokens in process fields are substituted from a fixed,
documented context the agent builds at runtime (attestation socket/uri,
config paths, rest-api features, initdata toml path, launch timeout,
resolved ocicrypt config path). Referencing an unknown variable is a
hard error, and manifest paths are validated against traversal, so the
contract stays fail-closed.
Processes are gated by a numeric `level` mapped from
guest_components_procs (AttestationAgent < ConfidentialDataHub <
ApiServerRest), preserving the existing implication semantics.
When no extension is mounted, launch_plan() returns None and the agent uses
a built-in plan that reproduces the legacy behaviour byte-for-byte
(binaries under /usr/local/bin, /etc/ocicrypt_config.json, /pause_bundle).
Monolithic / non-split confidential images are therefore unaffected.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add rootfs-image-coco-extension to the build-asset-rootfs matrix in the
amd64, arm64 and s390x static tarball workflows so the CoCo extension image
is built as part of CI. Keep building rootfs-image-confidential as well:
the runtime defaults to base + extension, but we retain the monolithic
confidential image for now during the transition.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add install_image_coco_extension() to kata-deploy-binaries.sh which:
- Unpacks the CoCo guest components and pause image tarballs into a
temporary rootfs directory (under the repo root so Docker-in-Docker
volume mounts resolve correctly)
- Calls image_builder.sh with USE_DOCKER=1, FS_TYPE=erofs,
MEASURED_ROOTFS=yes, SKIP_DAX_HEADER=yes, and SKIP_ROOTFS_CHECK=yes
to produce kata-containers-coco-extension.img + root_hash_coco-extension.txt
Add the rootfs-image-coco-extension-tarball Makefile target with
dependencies on pause-image-tarball and coco-guest-components-tarball.
The standard confidential image keeps pause-image and
coco-guest-components for now so it remains a usable standalone
monolithic CoCo image during the transition to base + extension.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
During the transition to composable (base + extension) CoCo images, run the
two runtimes on different image layouts so CI exercises both paths via
the existing qemu-{snp,tdx,coco-dev} (Go) and -runtime-rs (Rust) shims:
- runtime-rs (Rust) CoCo templates use the standard base image plus a
CoCo extension extra_image:
image = "@IMAGEPATH@" (was @IMAGECONFIDENTIALPATH@)
[[hypervisor.qemu.guest_extension_images]]
name = "coco"
path = "@COCOIMAGEPATH@"
verity_params = "@COCOVERITYPARAMS@"
COCOIMAGENAME/COCOIMAGEPATH/COCOVERITYPARAMS are added to the
runtime-rs Makefile only.
- runtime (Go) CoCo templates keep the monolithic confidential image
(@IMAGECONFIDENTIALPATH@) and pull rootfs-image-confidential.
shim-components.json reflects this: Go CoCo shims depend on
rootfs-image-confidential while the runtime-rs CoCo shims depend on
rootfs-image + rootfs-image-coco-extension.
shim-v2/build.sh feeds each runtime its own dm-verity params: Rust gets
the measured base image hash (@KERNELVERITYPARAMS@) plus the extension hash
(@COCOVERITYPARAMS@); Go gets the monolithic confidential image hash
(@KERNELVERITYPARAMS@). This is wired per make invocation so the same
@KERNELVERITYPARAMS@ placeholder resolves correctly for each.
Once the split path is validated we can flip the Go templates too and
drop the monolithic confidential image.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Extension images (e.g. the CoCo guest components extension) are not full root
filesystems -- they contain only the binaries and configuration that
get bind-mounted into the real rootfs at boot. The existing
check_rootfs() validation requires /sbin/init and systemd, which are
not present in extension images.
Add a SKIP_ROOTFS_CHECK environment variable that, when set to "yes",
bypasses the check_rootfs() call. Forward the variable into the
container environment when using the Docker-based build path so it
works in both direct and containerised invocations.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a systemd template unit kata-extension-mount@.service and companion
helper scripts (kata-extension-mount.sh, kata-extension-umount.sh) that handle
the guest-side setup of extension block device images.
The mount script:
- Discovers the block device by matching its virtio serial (extension-<name>)
- Reads verity parameters from kata.extension.<name>.verity_params on the
kernel command line
- Creates a dm-verity device and mounts the EROFS filesystem read-only
at /run/kata-extensions/<name>/
The unit uses ConditionKernelCommandLine=kata.extension.%i.verity_params so
it only activates when that extension is configured for the VM, and is
ordered before kata-agent.service so extension contents are available when
the agent starts.
A systemd generator (kata-extension-mount-generator) instantiates the
template for every kata.extension.<name>.verity_params entry on the kernel
command line. Because the runtime emits one such entry per configured
guest_extension_images, extensions are enabled purely from the cmdline and
adding a new one needs no change to the rootfs build.
The agent Makefile is updated to install the unit, scripts and generator.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Mirror the Rust runtime's guest_extension_images support in the Go runtime:
- hypervisor.go: add GuestExtensionImage struct and GuestExtensionImages field to
HypervisorConfig.
- config.go: parse [[hypervisor.qemu.guest_extension_images]] TOML sections
via toGuestExtensionImages(), validating that every extension has a non-empty
name and valid dm-verity parameters.
- qemu.go: in buildDevices(), attach each extra image as a virtio-blk
drive with ID extension-<name>; in kernelParameters(), append
kata.extension.<name>.verity_params=... for each extra image.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a serial_override field to BlockConfig and DeviceVirtioBlk so that
extension images get a deterministic QEMU serial (extension-<name>) instead of
the auto-generated image-<id>. This lets the guest discover each extension
via /dev/disk/by-id/virtio-extension-<name>.
Wire up the full cold-plug path for guest_extension_images:
- sandbox.rs: iterate boot_info.guest_extension_images, create read-only
BlockConfig entries with the correct serial_override, and push them
as ResourceConfig::GuestExtensionImage.
- resource lib.rs / manager_inner.rs: handle the new GuestExtensionImage variant
by delegating to the existing block device path.
- cmdline_generator.rs: use serial_override when present for the QEMU
serial= parameter; append kata.extension.<name>.verity_params=... to the
kernel command line for each extra image.
- inner.rs: propagate serial_override from BlockConfig to the cmdline
generator's add_block_device().
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Introduce the GuestExtensionImage type to describe extension block-device images
(e.g. CoCo guest components) that are cold-plugged into the VM
alongside the main rootfs.
Each GuestExtensionImage carries a short name (used as the virtio-blk serial
and in kernel cmdline verity parameters), a host-side path, and
dm-verity parameters (required -- extension images must be integrity-
protected).
A new guest_extension_images Vec is added to BootInfo so that both the Rust
and Go runtimes can iterate the configured extensions when building the
VM device list and kernel command line.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
With #12373 and #13280 in, emptyDirs are now shared from the host with
virtio-fs when supported, and properly accounted for by Kubelet.
So we can extend the sizeLimit-based pod eviction test introduced by #13127
to virtio-fs-based emptyDirs (originally only for block-based emptyDirs).
Generated-by: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The genpolicy guest-pull check now fails closed when an image carries
supplemental groups that containerd won't reproduce while pulling the
layers inside the guest, and quay.io/prometheus/busybox:latest happens
to ship gid 10 (wheel). pod-number-cpu.yaml was simply missed when the
other busybox manifests got their securityContext, so genpolicy bails
out on the expected/actual gid mismatch.
Let's declare supplementalGroups: [10] explicitly, just like we already
do for the other busybox-based pods, so the generated policy lines up
with what containerd actually applies.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Bumps github.com/opencontainers/runc from v1.2.8 to v1.3.6 to fix
GHSA-xjvp-4fhw-gc47 (CVE, CVSS 4.8).
The libcontainer/cgroups/systemd sub-package was removed from runc in
v1.3.x and moved to the standalone github.com/opencontainers/cgroups
module. Update the import in pkg/resourcecontrol/utils_linux.go
accordingly, and promote github.com/opencontainers/cgroups to a direct
dependency.
Fixes: https://osv.dev/GHSA-xjvp-4fhw-gc47
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Since the switch to per-component tarballs, versions.yaml was no longer
installed on host nodes by kata-deploy (DaemonSet/Helm). Previously it
was bundled into the merged kata-static.tar.zst; now it must be shipped
explicitly.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a check-agent-policy-coverage job to static-checks.yaml
that runs ci/check_agent_policy_coverage.sh on every pull request.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add ci/check_agent_policy_coverage.sh, a static check that asserts
every RPC method declared in the AgentService block of agent.proto has:
1. A policy gate (is_allowed or do_set_policy call) in the corresponding
handler in src/agent/src/rpc.rs.
2. A default rule entry in src/tools/genpolicy/rules.rego.
This prevents future handlers from silently bypassing the policy
boundary — the gap that prompted this work was MemAgentMemcgSet and
MemAgentCompactSet, which previously had no is_allowed call and no
rules.rego entry.
The script is self-contained and can be run standalone from any
directory in the repository tree.
Generated-By: IBM Bob
Assisted-by: Google AI Mode
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
ResizeVolumeRequest is declared in agent.proto and has a rules.rego
default entry, but the kata-agent had no corresponding ttrpc handler
in the AgentService impl block. This meant the CI policy-coverage
check (ci/check_agent_policy_coverage.sh) correctly flagged it as
ungated.
Add a resize_volume handler that:
- calls is_allowed(&req) so the policy boundary is enforced, and
- returns UNIMPLEMENTED, reflecting that the kata-agent has never
carried out volume resizes itself (runtime-rs exercises this RPC
via its own agent client; the kata-agent path was simply absent).
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Six AgentService RPC request types were missing default rule entries in
rules.rego, meaning genpolicy-generated policies would not include them
in their output and the entries would be absent from reference policies.
All six already have is_allowed gates in the agent RPC handlers.
Add the missing defaults and set them all to false
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
