Add the nvrc static-build target and make the NVIDIA rootfs targets
depend on it (the rootfs build consumes kata-static-nvrc.tar.zst), and
add the rootfs-image-nvidia-gpu-{base,extension} build targets to the
local-build Makefile and the nvgpu bundle.
Build nvrc and the base/extension rootfs images in the amd64/arm64 static
tarball workflows (nvrc is baked into the rootfs and pruned from the
release like the other guest components), and point the runtime-rs NVIDIA
shim components at the composable base + gpu extension (+ coco extension for
the confidential classes). The Go shim components keep the monolithic
nvidia-gpu / nvidia-gpu-confidential images.
Ship cryptsetup in the CoCo guest-components extension (and bundle
libnvat's non-glibc dependency closure next to the nvidia attester): the
chiseled base-nvidia image carries only veritysetup, so the coco-extension
manifest's usr/sbin PATH entry needs a cryptsetup to resolve for NVIDIA
confidential encrypted-storage.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Flip the runtime-rs NVIDIA GPU configs (plain/tdx/snp) from the monolithic
image to the driver-agnostic base-nvidia boot image plus a cold-plugged,
driver-versioned gpu extension mounted at /run/kata-extensions/gpu.
Confidential runtime-rs configs additionally cold-plug the CoCo
guest-components extension and enforce the base-nvidia dm-verity hash.
The Go runtime keeps booting the monolithic nvidia-gpu / nvidia-gpu-confidential
images, mirroring the CoCo split where runtime-rs is the first-class citizen and
Go stays on the monolithic layout.
Add the IMAGE{NAME,PATH}_NV_{BASE,EXTENSION} and NVIDIAGPUEXTENSIONVERITYPARAMS
make vars (runtime-rs only), and read the base-nvidia + gpu-extension root
hashes into runtime-rs's shim-v2 opts while the Go opts keep reading the
monolithic nvidia-gpu / nvidia-gpu-confidential hashes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Carve the monolithic NVIDIA GPU rootfs into a driver-agnostic
base-nvidia image (NVRC init + agent + base libs + in-tree modules)
and a driver-versioned gpu extension (GPU userspace, configs, firmware,
NVIDIA modules) laid out for /run/kata-extensions/gpu. Both halves are
produced from the same chiseled tree via a partition-after step, so
the monolith build stays byte-identical and continues to ship.
The shared stage-one (driver install) is reused across the monolith,
base and extension variants; base-nvidia is cached without the driver/ctk
versions so one base image can back multiple driver extensions.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a Kata static-build component for NVRC (tools/packaging/static-build/nvrc)
that clones the NVRC repository at a pinned git reference and builds a static
musl binary, mirroring the coco-guest-components build. This lets the
composable-image work track NVRC changes from source instead of depending on a
signed upstream release.
The ref points at NVIDIA/nvrc#167 (fidencio/nvrc topic/adapt-for-composable-kata-images,
20d38ea), which carries the guest-side extension mounting. This is a CI-only
crutch: once NVRC cuts a signed release with that work we go back to
downloading it from `url`, so this commit must be dropped before merge.
- versions.yaml: add repo/ref/toolchain to externals.nvrc (and bump the
version); the existing url is kept for the signed-release path.
- lib.sh: add get_nvrc_image_name for the builder image.
- kata-deploy-binaries.sh: add the nvrc build target, install_nvrc and its
tarball path; get_latest_nvidia_nvrc_version now tracks ref+toolchain so
the dependent NVIDIA images rebuild when the ref changes.
- nvidia_rootfs.sh: consume kata-static-nvrc.tar.zst instead of downloading
and cosign-verifying the release; the Sigstore .cert/.sig are now copied
only when present (i.e. the signed-release path), since a source build
emits just the binary.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
TestKilledVmmCleanup SIGKILLs the VMM and then removes the pod. After a
hard VMM kill the shim exits promptly (which the test asserts), so the
container's TaskExit event may not have reached containerd before the
shim was gone, leaving CRI's view of the container as "running" and
making a plain `crictl rmp` fail with "container is still running, to
stop first".
Retry the normal removal a few times to give the event time to arrive,
and only fall back to `crictl rmp -f` if it keeps failing -- which is how
a crashed sandbox is recovered in practice (kubelet GC / manual force
removal). The essential guarantee, that no shim process is leaked after
the VMM is killed, is unchanged.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Docker and containerd rely on the TaskExit event and a prompt Wait RPC,
not just the eventual sandbox teardown. With the original ordering the
shim tore the sandbox down (Stop/Delete the guest) *before* publishing
the container exit, so for a slow guest shutdown -- e.g. the nvidia-gpu
config with an 8GB /dev/shm memory-backend on a constrained CI runner --
containerd could SIGKILL the shim before the exit was published. The
`docker run --rm` removal then races the dead-shim cleanup and fails,
even though the container itself exited 0.
Publish the container exit (c.exitCh) and the TaskExit event *before*
the sandbox teardown, and run the teardown without holding s.mu so
concurrent Delete()/Shutdown() RPCs are not blocked.
To keep the guarantees the previous ordering provided:
* teardownWg lets Shutdown() wait for an in-flight teardown to finish,
so the sandbox run directory (watched by kata-monitor) and the CRI
state are gone before the shim exits -- without holding s.mu across
the slow guest shutdown; and
* teardownOnce serialises wait()'s teardown with watchSandbox()'s
killed-VMM teardown so the (not internally synchronized)
Sandbox.Stop/Delete never run concurrently.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
For `docker run --rm`, containerd invokes the shim `delete` binary
(cleanupAfterDeadShim) once the container task exits. Kata's Cleanup
path re-loads the sandbox and calls StopContainer/DeleteContainer/Stop,
each of which lazily connects to the guest agent over vsock.
When the sandbox was already torn down by the main shim (the common
case for a short-lived `docker run --rm`), the VM -- and its agent --
are gone, so that vsock connect blocks until containerd's delete
timeout SIGKILLs the binary. The removal then fails and `docker run
--rm` returns non-zero even though the container itself exited 0.
Detect the already-dead hypervisor (its pidfile is gone / the pid no
longer maps to a live process) at the start of CleanupContainer and
mark the agent dead. Subsequent agent RPCs then fail fast with "Dead
agent" and the force path performs only host-side cleanup, so the
delete binary returns promptly instead of hanging.
The legitimate "shim crashed but VM still alive" cleanup is unaffected:
the hypervisor is still running, so the agent is not marked dead and
the normal agent-based teardown proceeds.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
virtio-mem is enabled on s390x, where memory hotplug goes through a
virtio-mem-ccw device set up during VM initialization. That device --
and the later resize -- require a memory hotplug region (a non-zero
maxmem/slots on the QEMU command line).
The runtime-rs command line generator, however, zeroes maxmem/slots
whenever the guest uses the shared /dev/shm memory-backend with a
non-nvdimm rootfs (the s390x case: virtio-blk-ccw rootfs). With no
hotplug region reserved, two things broke on s390x:
* setup_virtio_mem failed VM start with "the configuration is not
prepared for memory devices, consider specifying the maxmem option";
* with virtio-mem unavailable, memory resize fell back to pc-dimm,
which is not a valid device model on s390x, breaking
TestContainerMemoryUpdate:
'pc-dimm' is not a valid device model name
Keep the hotplug region when virtio-mem is enabled, mirroring the Go
runtime which reserves maxmem and hotplugs via virtio-mem-ccw on s390x.
This lets setup_virtio_mem succeed at boot and makes memory resize use
virtio-mem instead of pc-dimm.
Also guard setup_virtio_mem behind QemuCmdLine::has_memory_hotplug_region()
as a defensive fallback: if some configuration still ends up without a
hotplug region, skip virtio-mem setup (like static-sizing arches such as
arm64) rather than failing VM start.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's ensure we have more debug logs to go through in case of failures.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Extend basic-ci-amd64 VMM matrix coverage to include qemu-nvidia-gpu and
qemu-nvidia-gpu-runtime-rs so this branch is validated in CI.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
NVIDIA GPU configs default pod_resource_api_sock to the kubelet Pod
Resources API path. On non-Kubernetes hosts that path is usually missing;
use CDI sandbox annotations for cold-plug instead of failing kubelet lookup.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The container create flow forced every non-pod-container to the
"pod_sandbox" type. Standalone engines (Docker/nerdctl/podman) surface
as SingleContainer, so they were mislabeled as pod sandboxes.
The agent skips CDI device injection when the container type is
"pod_sandbox", so the NVIDIA CDI edits carried in the "cdi.k8s.io/*"
annotations were never applied and the GPU userspace (e.g. nvidia-smi)
was missing in the guest. Emit the actual container type instead, which
matches the Go runtime and lets the agent inject CDI devices for the
single-container flow.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
ContainerType already renders as "single_container" via Display, but
from_str rejected it. Accept it so the value can round-trip, matching
the container type the Go runtime uses for standalone containers.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
The kubelet PodResources API is only available under Kubernetes. For
single-container engines (Docker/nerdctl/podman) the CDI runtime applies
the device's containerEdits directly to the OCI spec, so the VFIO nodes
(e.g. /dev/vfio/devices/vfio0) show up in linux.devices instead.
Discover those nodes from the OCI spec in addition to the PodResources
API and feed both into the same cold-plug path, de-duplicating by host
path. This reuses the existing Kubernetes machinery so GPU passthrough
works for single containers with no changes to device handling.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
A discrete GPU usually shares its IOMMU group with an audio function
(e.g. NVIDIA 02:00.1). Cold-plugging every function in the group made
QEMU emit a vfio-pci/pcie-root-port/iommufd triple per function while
reusing the GPU's root-port and IOMMUFD object ids, so QEMU aborted with
a duplicate-id error before it was reachable over QMP.
Apply the existing IOMMU_IGNORE classification while discovering the
group so audio controllers and bridges are pruned, leaving only the
passthrough-capable device(s). Also fix filter_bridge_device to parse
the sysfs class attribute as hex; the previous decimal parse always
failed, which is why the filter never removed anything.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
Set the new pci_path field in cloud-hypervisor network test config so
unit tests compile with the updated NetworkConfig struct.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Persist and forward the hot-plugged NIC guest PCI path so the agent can
resolve sysfs location and update interfaces reliably.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Network interface hot-plug places the NIC on the pcie-pci-bridge nested
under a root port. Track the bridge slots already occupied (queried via
query-pci) and reserve bridge slot ranges correctly so repeated NIC
hot-plug operations do not collide on the same bridge slot.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Network interface hot-plug on Q35 with OVMF needs a hot-pluggable PCIe
slot to attach the NIC to after boot. Add the nested root-port plus
pcie-pci-bridge topology and the related QEMU argument ordering so a
network device can be hot-plugged into the bridge.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Use a nested root-port plus pcie-pci-bridge topology on Q35+OVMF so
firmware reserves hot-plug resources and NIC hot-plug works reliably.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Layer tar entries may include current-directory components, such as
./etc/passwd, when images are produced by common tar commands.
Normalize layer paths before matching passwd, group, and whiteout
entries. This lets genpolicy handle equivalent relative paths while
skipping empty, unsafe, or absolute paths.
Fixes: #12692
Assisted-By: OpenAI Codex
Signed-off-by: Han Zhang <ihanzhzh@gmail.com>
Add kata-deploy-scheduling.bats to KATA_DEPLOY_TEST_UNION so the new
Helm scheduling template tests run in CI.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Merge NFD and user nodeSelectorTerms with merge/concat.
Ignore the name key in podLabels, document podAnnotations
and extend bats coverage for annotations.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Expose podLabels, podAnnotations and affinity in values.yaml so
operators can apply more granular scheduling controls on the
kata-deploy DaemonSet alongside the existing nodeSelector and
tolerations knobs.
Defaults are empty, so existing deployments render the same manifest.
When NFD is enabled, the chart merges the hard-coded virtualization
nodeAffinity with any user provided affinity.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
The kata-deploy payload downloads the nydus-snapshotter host binaries
(containerd-nydus-grpc, nydus-overlayfs) from the upstream per-arch
release asset, which is glibc dynamically linked and fails to exec on
musl-only hosts that have no glibc dynamic loader.
nydus-snapshotter also publishes a statically linked release asset.
Honour the existing STATIC_RUNTIME=yes umbrella flag (which already
builds the kata Go host binaries static) in the nydus component build so
that, with a single flag, the whole host-binary set of the payload is
static and runs on a musl-only host. STATIC_RUNTIME=yes maps to a new
STATIC_NYDUS_SNAPSHOTTER build-arg on the nydus downloader stage of
Dockerfile.components, which selects the linux-static asset.
The static nydus asset is published for amd64 only, so the build fails
fast on other architectures when the flag is set. The default build is
unchanged.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Add an opt-in STATIC_RUNTIME=yes knob to the shim-v2 static build. When
set, the Go host binaries are built with the runtime STATIC=yes profile
(CGO_ENABLED=0, no PIE), producing fully static, libc-agnostic binaries
in the kata-deploy payload.
This lets a payload image be produced that runs on musl-only hosts which
have no glibc dynamic loader, where the default glibc-linked binaries
fail to exec. The default build is unchanged.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
The host KVM capability checks obtained the KVM_CREATE_VM and
KVM_CHECK_EXTENSION ioctl request numbers from <linux/kvm.h> via cgo
(kata-check.go for the generic check, hypervisor_linux_arm64.go for the
Arm RME check). This was the only remaining use of cgo in the runtime,
and it forced the runtime to be linked against the host libc.
Replacing these with plain Go constants lets kata-runtime,
containerd-shim-kata-v2 and kata-monitor build with CGO_ENABLED=0,
which is a prerequisite for producing fully static, libc-agnostic
binaries that run on musl-only hosts (the kata-deploy payload otherwise
fails to exec on a glibc-free distribution because its requested dynamic
loader is absent).
The ioctl numbers are stable kernel ABI. The asm-generic encoding used
by amd64, arm64, s390x and riscv64 gives _IO(KVMIO, nr) == (0xAE << 8) |
nr, while powerpc uses the legacy encoding (_IOC_NONE == 1, direction
shifted by 29). The values are split into kvm_ioctls_generic.go and
kvm_ioctls_ppc64le.go accordingly so each architecture keeps the exact
value its kernel headers would have produced.
No functional change: the runtime issues the same ioctls as before.
Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
A VFIO-backed network interface used to be hotplugged only as a side
effect of creating the workload container that references the VFIO
device in its OCI spec: setupNetworks() skipped VfioEndpoints when it
ran at sandbox start and applied their configuration later, on the
second invocation from createContainer(), guarded by the
hotplugNetworkConfigApplied flag.
Init containers run before that workload container, so they started
with the VFIO interface absent and thus without pod networking.
Attach the device at sandbox scope instead, from setupNetworks() at
sandbox start, before any container is created:
- Add Sandbox.hotplugVfioNetworkDevice(): resolve the VFIO device
node backing the endpoint's host BDF, hotplug it and record the
guest PCI path on the endpoint. When the workload container later
references the same VFIO group through a device plugin, the device
manager finds the device by major:minor and only bumps reference
counts, so it is neither plugged twice nor unplugged when that
container exits.
- Configure all endpoints, VfioEndpoints included, in a single
setupNetworks() pass at sandbox start. The Container parameter, the
second invocation from createContainer() and the
hotplugNetworkConfigApplied flag are no longer needed.
Add unit tests for the hotplugVfioNetworkDevice() guard paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: PiotrProkop <pprokop@nvidia.com>
Block-backed emptyDir volumes are sized using the capacity of the host
filesystem rather than Kubernetes sizeLimit. On large filesystems,
ext4 metadata can consume enough space to trigger pod eviction.
Document the expected overhead, its effect on usable storage, current
mitigations, and the issue tracking potential improvements.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Update OS version and containerd references, and make a few minor
adjustments and clarifications. For the clarifications, move the
section "Deploy pods with Kata agent security policies" further up
so that the installation of the genpolicy tool has been elaborated
on before attempting to run NVIDIA CI tests locally.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
