The kubelet PodResources API is only available under Kubernetes. For
single-container engines (Docker/nerdctl/podman) the CDI runtime applies
the device's containerEdits directly to the OCI spec, so the VFIO nodes
(e.g. /dev/vfio/devices/vfio0) show up in linux.devices instead.
Discover those nodes from the OCI spec in addition to the PodResources
API and feed both into the same cold-plug path, de-duplicating by host
path. This reuses the existing Kubernetes machinery so GPU passthrough
works for single containers with no changes to device handling.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
A discrete GPU usually shares its IOMMU group with an audio function
(e.g. NVIDIA 02:00.1). Cold-plugging every function in the group made
QEMU emit a vfio-pci/pcie-root-port/iommufd triple per function while
reusing the GPU's root-port and IOMMUFD object ids, so QEMU aborted with
a duplicate-id error before it was reachable over QMP.
Apply the existing IOMMU_IGNORE classification while discovering the
group so audio controllers and bridges are pruned, leaving only the
passthrough-capable device(s). Also fix filter_bridge_device to parse
the sysfs class attribute as hex; the previous decimal parse always
failed, which is why the filter never removed anything.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
Set the new pci_path field in cloud-hypervisor network test config so
unit tests compile with the updated NetworkConfig struct.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Persist and forward the hot-plugged NIC guest PCI path so the agent can
resolve sysfs location and update interfaces reliably.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Network interface hot-plug places the NIC on the pcie-pci-bridge nested
under a root port. Track the bridge slots already occupied (queried via
query-pci) and reserve bridge slot ranges correctly so repeated NIC
hot-plug operations do not collide on the same bridge slot.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Network interface hot-plug on Q35 with OVMF needs a hot-pluggable PCIe
slot to attach the NIC to after boot. Add the nested root-port plus
pcie-pci-bridge topology and the related QEMU argument ordering so a
network device can be hot-plugged into the bridge.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Use a nested root-port plus pcie-pci-bridge topology on Q35+OVMF so
firmware reserves hot-plug resources and NIC hot-plug works reliably.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The kata-deploy payload downloads the nydus-snapshotter host binaries
(containerd-nydus-grpc, nydus-overlayfs) from the upstream per-arch
release asset, which is glibc dynamically linked and fails to exec on
musl-only hosts that have no glibc dynamic loader.
nydus-snapshotter also publishes a statically linked release asset.
Honour the existing STATIC_RUNTIME=yes umbrella flag (which already
builds the kata Go host binaries static) in the nydus component build so
that, with a single flag, the whole host-binary set of the payload is
static and runs on a musl-only host. STATIC_RUNTIME=yes maps to a new
STATIC_NYDUS_SNAPSHOTTER build-arg on the nydus downloader stage of
Dockerfile.components, which selects the linux-static asset.
The static nydus asset is published for amd64 only, so the build fails
fast on other architectures when the flag is set. The default build is
unchanged.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Add an opt-in STATIC_RUNTIME=yes knob to the shim-v2 static build. When
set, the Go host binaries are built with the runtime STATIC=yes profile
(CGO_ENABLED=0, no PIE), producing fully static, libc-agnostic binaries
in the kata-deploy payload.
This lets a payload image be produced that runs on musl-only hosts which
have no glibc dynamic loader, where the default glibc-linked binaries
fail to exec. The default build is unchanged.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
The host KVM capability checks obtained the KVM_CREATE_VM and
KVM_CHECK_EXTENSION ioctl request numbers from <linux/kvm.h> via cgo
(kata-check.go for the generic check, hypervisor_linux_arm64.go for the
Arm RME check). This was the only remaining use of cgo in the runtime,
and it forced the runtime to be linked against the host libc.
Replacing these with plain Go constants lets kata-runtime,
containerd-shim-kata-v2 and kata-monitor build with CGO_ENABLED=0,
which is a prerequisite for producing fully static, libc-agnostic
binaries that run on musl-only hosts (the kata-deploy payload otherwise
fails to exec on a glibc-free distribution because its requested dynamic
loader is absent).
The ioctl numbers are stable kernel ABI. The asm-generic encoding used
by amd64, arm64, s390x and riscv64 gives _IO(KVMIO, nr) == (0xAE << 8) |
nr, while powerpc uses the legacy encoding (_IOC_NONE == 1, direction
shifted by 29). The values are split into kvm_ioctls_generic.go and
kvm_ioctls_ppc64le.go accordingly so each architecture keeps the exact
value its kernel headers would have produced.
No functional change: the runtime issues the same ioctls as before.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
A VFIO-backed network interface used to be hotplugged only as a side
effect of creating the workload container that references the VFIO
device in its OCI spec: setupNetworks() skipped VfioEndpoints when it
ran at sandbox start and applied their configuration later, on the
second invocation from createContainer(), guarded by the
hotplugNetworkConfigApplied flag.
Init containers run before that workload container, so they started
with the VFIO interface absent and thus without pod networking.
Attach the device at sandbox scope instead, from setupNetworks() at
sandbox start, before any container is created:
- Add Sandbox.hotplugVfioNetworkDevice(): resolve the VFIO device
node backing the endpoint's host BDF, hotplug it and record the
guest PCI path on the endpoint. When the workload container later
references the same VFIO group through a device plugin, the device
manager finds the device by major:minor and only bumps reference
counts, so it is neither plugged twice nor unplugged when that
container exits.
- Configure all endpoints, VfioEndpoints included, in a single
setupNetworks() pass at sandbox start. The Container parameter, the
second invocation from createContainer() and the
hotplugNetworkConfigApplied flag are no longer needed.
Add unit tests for the hotplugVfioNetworkDevice() guard paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: PiotrProkop <pprokop@nvidia.com>
Block-backed emptyDir volumes are sized using the capacity of the host
filesystem rather than Kubernetes sizeLimit. On large filesystems,
ext4 metadata can consume enough space to trigger pod eviction.
Document the expected overhead, its effect on usable storage, current
mitigations, and the issue tracking potential improvements.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Update OS version and containerd references, and make a few minor
adjustments and clarifications. For the clarifications, move the
section "Deploy pods with Kata agent security policies" further up
so that the installation of the genpolicy tool has been elaborated
on before attempting to run NVIDIA CI tests locally.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Migrate kata-types and kata-agent from toml 0.5 to 1.x:
- Consolidate kata-types onto the workspace toml version
- Replace toml::to_vec() (removed in 1.x) with to_string().into_bytes()
- Map toml::de::Error with .map_err(std::io::Error::other) where functions
return io::Result (From impl was removed in 1.x)
- Update test assertions: toml 1.x serializes strings with double quotes
instead of single quotes
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
No API changes required: Patch, PatchOperation, and patch() have
compatible signatures across the 2.x → 4.x range.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Migrate hooks.rs from the removed Popen/PopenConfig/CommunicateError API
to the subprocess 1.x Exec/Job builder API:
- Replace Popen::create() + PopenConfig with Exec builder + Job
- communicate_start().read_string() → job.communicate()?.limit_time().read_string()
- read_string() now returns io::Result<(String,String)> (no Option wrapper)
- ExitStatus enum variants replaced with ExitStatus::code() / signal() methods
- Timeout errors propagated with io::ErrorKind::TimedOut as before
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Update hyper to 1.10.1 and hyperlocal to 0.9.1 (required for hyper 1.x
compatibility). Add http-body-util 0.1, hyper-util 0.1 and bytes 1.0 as
workspace dependencies.
API changes applied across shim-interface, runtimes, hypervisor
(firecracker) and resource (nydus_client) crates:
- hyper::Body → http_body_util::Full<Bytes> for outgoing bodies
- hyper::body::Incoming for incoming server request bodies
- hyper::Client → hyper_util::client::legacy::Client
- hyper::Server + make_service_fn → TcpListener loop +
hyper_util::server::conn::auto::Builder (kata-ctl http_server)
- hyper::server::conn::Http → hyper::server::conn::http1::Builder
with hyper_util::rt::TokioIo wrapper (runtimes shim_mgmt server)
- hyper::body::to_bytes() → BodyExt::collect().await?.to_bytes()
- Client::unix() now requires explicit body type parameter Full<Bytes>
- Drop removed 'stream' feature flag from hyper dependency declarations
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Enabled TestContainerMemoryUpdate support for runtime-rs on s390x and
disabled static_sandbox_resource_mgmt to dynamically update the memory.
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
Integrate virtio-mem into memory hotplug and hotunplug operations:
- Modified hotplug_memory() to automatically detect virtio-mem device
and use resize instead of pc-dimm hotplug
- Modified hotunplug_memory() to use virtio-mem resize for memory
decrease with saturating_sub to prevent underflow
- Modified hotplugged_memory_size() to account for virtio-mem devices
- Falls back to pc-dimm when virtio-mem is not available
This completes the virtio-mem integration, enabling dynamic memory
management with virtio-mem-ccw on s390x while maintaining full
backward compatibility with pc-dimm.
Assisted-by: IBM Bob
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
Call setup_virtio_mem() during VM startup when enable_virtio_mem flag
is set:
- Added setup call after QMP connection is established
- Proper error handling with context
Assisted-by: IBM Bob
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
In 7afdfc7388 we
just disabled the undocumented-permissions zizmor warning,
but now with AI we can roll this fix more easily, so remove
the disabling of the rule and add the comments
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
When VFIO GPUs span both host NUMA nodes, emit guest NUMA nodes even
if vCPUs < node count (memory-only nodes), and pin each pxb-pcie to
bus=pcie.0 so QEMU does not attach the second expander to the first.
Also fix checkVCPUsPinningNUMA to tolerate vCPUs < NUMA node count.
The memory-only NUMA topology (emitted for pxb-pcie placement when
default_vcpus=1) creates more guest NUMA nodes than there are vCPUs;
the previous hard error silently tore down every sandbox. Now the
available vCPU(s) are pinned to the first node(s) and memory-only
nodes are skipped.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
As the machine is off and I cannot connect to the BMC to turn
it back on, let's avoid PRs running those tests for now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The `hotplug_vfio_on_root_bus` configuration option is no longer needed
in the runtime-rs.
Remove the option from:
- `DeviceInfo` struct and its `hotplug_vfio_on_root_bus` field
- `PCIeTopology` struct and its construction in `topology.rs`
- `HypervisorInfo` struct and its `hotplug_vfio_on_root_bus` field
- `KATA_ANNO_CFG_HYPERVISOR_HOTPLUG_VFIO_ON_ROOT_BUS` annotation constant
and its handler in `Annotation::apply()`
- DragonBall validation check in `ConfigPlugin::adjust_config()`
- All runtime-rs TOML config templates (qemu, fc, nvidia GPU variants)
- Test fixture TOML files
Update the documentation having a reference to it.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Align the composable VM images proposal with the naming convention used
throughout the implementation: the generic mechanism and its data model
are now "guest extension image" (config key `guest_extension_images`,
struct `GuestExtensionImage`), and the runtime artifacts use the matching
"extension" vocabulary (`/run/kata-extensions/<name>`,
`kata.extension.<name>.verity_params`, `kata-extension-mount@.service`,
the `extension-<name>` virtio-blk serial / dm-verity target, and the
`kata-containers-coco-extension.img` image).
It also documents how template-unit instances are enabled: a systemd
generator instantiates kata-extension-mount@<name>.service for each
extension on the kernel command line, so the rootfs build stays
extension-agnostic.
This replaces the earlier mixed "extra_images"/"addon" terminology and
addresses the naming-consistency review feedback on the series.
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Generate etc/kata-extensions/components.toml inside the CoCo extension rootfs
before building the image, so the manifest becomes part of the
dm-verity-measured erofs image.
The manifest declares the extension's path-only components (ocicrypt config,
pause bundle) and the launchable processes (attestation-agent,
confidential-data-hub, api-server-rest) with paths relative to the extension
mount point (/run/kata-extensions/coco) and `${var}` placeholders resolved
by kata-agent at runtime. This is the data the agent consumes to launch
the CoCo guest components without per-bundle code changes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Make the agent's handling of CoCo guest components data-driven so that a
new extension bundle can be introduced without changing agent code.
Add a new `extension` module defining a generic component manifest contract.
When a CoCo extension image is mounted at /run/kata-extensions/coco/, the agent
reads etc/kata-extensions/components.toml from it to discover:
- path-only components (looked up by id under the `[paths]` table), used
for the ocicrypt config and the pause bundle, and
- launchable processes (`[[process]]` entries) describing the binary
path, args, optional args, config file, environment, socket to wait on
and timeout.
`${var}` tokens in process fields are substituted from a fixed,
documented context the agent builds at runtime (attestation socket/uri,
config paths, rest-api features, initdata toml path, launch timeout,
resolved ocicrypt config path). Referencing an unknown variable is a
hard error, and manifest paths are validated against traversal, so the
contract stays fail-closed.
Processes are gated by a numeric `level` mapped from
guest_components_procs (AttestationAgent < ConfidentialDataHub <
ApiServerRest), preserving the existing implication semantics.
When no extension is mounted, launch_plan() returns None and the agent uses
a built-in plan that reproduces the legacy behaviour byte-for-byte
(binaries under /usr/local/bin, /etc/ocicrypt_config.json, /pause_bundle).
Monolithic / non-split confidential images are therefore unaffected.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
