Use dedicated empty firmware makefile variables so the NVIDIA CPU
runtime does not inherit generic OVMF/AAVMF firmware paths.
runtime-rs already sets the correct option by default, so, no changed
needed on that front.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The network hotplug (handle_network_device) passed `queue_num`, a
queue pair count, straight as `num_queues`, while cloud-hypervisor's
virtio-net expects a raw virtqueue count.
With the default network_queues=1 this hit "Number of queues (1) to
virtio_net should be higher than 2" on the post-start netdev add,
failing sandbox start.
Convert the pair count to the raw queue count (queue_num.max(1) * 2),
matching the coldplug path which already does network_queues_pairs * 2.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
NetworkConfig.queue_num is a queue pair count, but Dragonball's
virtio/vhost-net backends expect an even raw virtqueue count. Passing
it through without doubling caused InvalidQueueNum(1) on the default
network_queues=1, failing sandbox start.
Convert pair count to raw count (queue_num.max(1) * 2) at both
Dragonball network entry points, mirroring the Cloud Hypervisor
backend.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
As part of the effort to thread network queues from the top-level
configuration all the way down to each endpoint instead of having
low-level helpers re-read the global configuration, propagate the
per-network queue_num into the QEMU cmdline generator and hotplug
path.
add_network_device() and get_network_device() now take an explicit
num_queues argument sourced from network.config.queue_num, replacing
the direct read of network_info.network_queues, so each network
endpoint honors its own configured queue count.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
As there's no field to map the configuration's network_queues item
in the `DanConfig`, this commit introduces a network_queues to do this.
And accordingly, we also make it passed down from sandbox layer to
Dan network configurations.
To make it work well, it make it more robust for queues and queue_size
settings with checking logics. And related UT is added.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Set queue_num (derived from net_pair.queues, one TX + one RX per
queue pair) and queue_size (256) when building the network pair for
veth, macvlan, ipvlan and vlan endpoints so the virtio-net device is
created with the configured multi-queue layout.
Meanwhile, update endpoint tests accordingly.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Number of virtio queue pairs (each pair = 1 RX + 1 TX).
Derived from `network_queues` in the hypervisor TOML config.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Move runtime-rs onto containerd-shim/containerd-shim-protos 0.11.0,
which drops the legacy cgroups-rs 0.3.x dependency. Adapt to ttrpc
0.9 API changes for async connect, server listener setup, and event
forwarding.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Composer <cursor@cursor.com>
Unify agent, rustjail, and runtime-rs on cgroups-rs 0.5.1 from
crates.io instead of the v0.3.5 git fork. Add a rustjail shim for
the 0.5.x API layout where the legacy cgroupfs types moved under
cgroups::fs.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Composer <cursor@cursor.com>
cdi-rs 1.1.1 is the first release built against oci-spec 0.10, so
the Spec type its inject_devices() takes now unifies with the
workspace oci-spec and the kata-agent builds again.
kata-agent and pod-resources-rs each carried their own pin (0.1.1
and 0.1.2). Hoist the dependency to the root workspace like
oci-spec in the previous commit, so both consumers stay on the
same cdi types and future bumps apply to the whole tree at once.
The kata code compiles unchanged against the 0.1.2 -> 1.1.1 jump;
the API surface we use (annotations, cache, spec_dirs,
inject_devices) is source-compatible.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Bumping oci-spec to 0.10.0 in the root manifest broke the
kata-agent build: kata-types, protocols and kata-sys-util
declare their own oci-spec pin (0.8.1) instead of inheriting
the workspace dependency, so the bump left them behind on
0.8.4 while the agent and runtime-rs crates moved to 0.10.0.
Since 0.x lines are semver-incompatible, cargo compiles both
versions side by side, and every oci-spec type these libs
expose in their public API (e.g. oci_spec::runtime::Spec) no
longer matches the type the agent expects — a wall of
"expected Spec, found Spec" mismatches.
Switch the three crates to `workspace = true` like the other
members so the root manifest is the single source of truth
and future bumps apply to the whole tree atomically. The
`runtime` feature is inherited from the workspace declaration.
No resolution change yet: both sides currently resolve to
0.8.4; the actual bump follows in the next commit.
Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
On hosts with memory-only NUMA nodes (GPU/CXL memory, e.g. the empty
nodes 1-8 a Grace-Hopper GH200 exposes next to its single CPU node), the
auto NUMA path built a guest node for every host node and then failed in
DistributeVCPUsProportionally with "HostCPUs for NUMA node N must not be
empty". This broke sandbox creation for any qemu-nvidia-cpu pod once
enable_numa is on, not just the NUMA tests.
Filter CPU-less nodes out of the auto-derived topology in
maybeRightSizeAutoNUMA() so the guest CPU topology only spans nodes that
can host vCPUs. Keep the full set when a VFIO device is attached, since a
passed-through GPU may live on a CPU-less node that still needs a guest
NUMA node for pxb-pcie placement.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
qemu-nvidia-gpu boots the driver-agnostic NVIDIA base image and then
cold-plugs the GPU extension on top. Nothing, however, just boots that
base image on NVIDIA hosts that have no GPU (or where the GPU isn't handed
to the guest), even though the base image is perfectly usable on its own.
Add qemu-nvidia-cpu (Go) and qemu-nvidia-cpu-runtime-rs (Rust), cloned
from the qemu-nvidia-gpu configs with every GPU-passthrough bit dropped:
no guest_extension_images, cold_plug_vfio = "no-port", pcie_root_port = 0,
no VFIO pod-resource wiring and no GPU NUMA validation. Resource defaults
and timeouts are pulled back in line with qemu[-runtime-rs] (1 vCPU,
DEFMEMSZ, small overheads) rather than the GPU-tuned values, and firmware
is left to the arch default so x86_64 boots without OVMF.
Both classes boot the measured nvidia base image verity-backed. The Rust
runtime already maps KERNELVERITYPARAMS_NV to the base hash; the Go runtime
maps it to the monolith, so a dedicated KERNELVERITYPARAMS_NV_BASE var is
added for qemu-nvidia-cpu.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Under static_sandbox_resource_mgmt the VM is sized to exactly what OCI
requests (base default_vcpus plus workload CPU limits) and CPU hotplug is
off, so DefaultMaxVCPUs is set to the sized NumVCPUs().
When a pod requests default_vcpus=0 with no CPU limit, that sized count
rounds to 0. DefaultMaxVCPUs then becomes 0, which the hypervisor later
expands to the host maximum (govmm.MaxVCPUs()). With NUMA enabled the SMP
topology becomes sockets=maxcpus=host_max, e.g. maxcpus=240, which fails
to boot on hosts where KVM caps the recommended vCPU count (the sandbox
never comes up and k8s-sandbox-vcpus-allocation / k8s-cpu-ns time out on
qemu-nvidia-cpu).
A VM cannot boot with zero vCPUs (QEMU already clamps -smp to 1), so floor
DefaultMaxVCPUs at that same single boot vCPU rather than falling back to
the host maximum. This keeps the value equal to the OCI-requested count in
every non-degenerate case and never consults the host maximum under static
management.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Use an explicit memory-backend-ram object for normal,
non-file-backed QEMU memory and bind it through the machine
memory-backend option.
This aligns the runtime-rs QEMU command line more closely with the
runtime-go QEMU path when filesystem sharing is disabled. Further,
when enable_mem_prealloc is enabled, runtime-rs now emits
prealloc=on for that backend, matching the runtime-go behavior.
This supports #13363, where the non-confidential NVIDIA runtime-rs
handler disables filesystem sharing and uses the EROFS snapshotter as
its Kubernetes image-layer transport. This is expected to reduce the
likelihood of the QMP startup delays seen in #13084 for NVIDIA GPU
passthrough with filesystem sharing disabled. A separate issue on the
behavior of the QMP timeout logic is additionally tracked in #13343.
In addition to these changes, #12187 tracks a machine-centric QEMU
refactor. In particular, the phase 3 work described there plans to
lift memory backends into QEMU Objects. After this work, command
line generation will be refactored, and, full NUMA support be ported
to runtime-rs.
This commit also adds the first command-line generator unit tests for
this file to cover the new RAM backend behavior.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
When containerd wants to terminate a container, it sends a KillRequest with exec_id="", all=false, signal=SIGTERM.
Kata-shim translates that to a SignalProcessRequest with exec_id="", signal=SIGTERM
Kata agent used to interpret exec_id="" as implying all=true.
That was correct for forceful deletion of containers, as it uses a KillRequest with all=true, signal=SIGKILL.
However, for graceful termination, we want to only signal the root process of the container (all=false).
This changes makes it so that Kata agent interprets exec_id="" && signal=SIGKILL as implying all=true, and uses all=false otherwise.
It also fixes an unrelated bug, where Kata agent would signal the root process twice when all=true.
Fixes#13152 - sending SIGTERM twice to the container's init process on graceful shutdown of a container.
Signed-off-by: Bozhidar Marinov <bozhidar.marinov1@digits.schwarz>
Co-authored-by: Markus Rudy <webmaster@burgerdev.de>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The proposal told us to move it under docs/design/ once it was accepted
and implemented. The systemd base landed on main and the NVIDIA split
(nvidia base + gpu extension) is what this series wires up, so it's no
longer a proposal - promote it to docs/design/composable-vm-images.md,
flip the status accordingly and repoint the in-tree references (agent,
osbuilder and kata-deploy) at the new path.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
createContainer hooks run in the forked container child, where kata-agent's
async slog drain (its background thread) does not exist, so `error!` from a
failing hook is silently dropped and the failure leaves no trace at all.
Mirror hook failures to /dev/kmsg, the one log sink that survives the fork
and reaches the guest console, so a hook that can't be found or exits
non-zero is at least visible when debugging. Failures only.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Hooks were spawned by PATH-searching `args[0]`. After `env_clear()` a bare
argv[0] only resolves against the compiled-in default PATH (/bin:/usr/bin),
so a hook binary living elsewhere - e.g. a composable-image extension under
/run/kata-extensions/<name>/bin - failed to start with ENOENT.
Execute the hook's `path` instead (an absolute path, already validated in
`new`), passing `args` as argv with argv[0] = args[0]. This matches OCI
runtime-spec semantics and lets hooks run from any location.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Flip the runtime-rs NVIDIA GPU configs (plain/tdx/snp) from the monolithic
image to the driver-agnostic nvidia base boot image plus a cold-plugged,
driver-versioned gpu extension mounted at /run/kata-extensions/gpu.
Confidential runtime-rs configs additionally cold-plug the CoCo
guest-components extension and enforce the nvidia base dm-verity hash.
The Go runtime keeps booting the monolithic nvidia-gpu / nvidia-gpu-confidential
images, mirroring the CoCo split where runtime-rs is the first-class citizen and
Go stays on the monolithic layout.
Add the IMAGE{NAME,PATH}_NV_{BASE,EXTENSION} and NVIDIAGPUEXTENSIONVERITYPARAMS
make vars (runtime-rs only), and read the nvidia base + gpu-extension root
hashes into runtime-rs's shim-v2 opts while the Go opts keep reading the
monolithic nvidia-gpu / nvidia-gpu-confidential hashes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Fixes RUSTSEC-2026-0194 and RUSTSEC-2026-0195 reported against
quick-xml < 0.37.2. The quick-xml se::to_string API used by kata-ctl's
log-parser is unchanged between 0.28 and 0.41, so no call-site edits
are needed.
Ref: https://osv.dev/RUSTSEC-2026-0194
Ref: https://osv.dev/RUSTSEC-2026-0195
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Bump workspace rstest to 0.26.1 and migrate local pins in
kata-types and kata-ctl to workspace = true. No macro API
changes required across this range.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add awaitgroup to workspace dependencies at 0.7.0 and switch both
rustjail and virt_container from local pins to workspace = true.
No API changes required — WaitGroup::new(), .worker() and .wait()
are unchanged across this range.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Bump workspace sha2 to 0.11.0 and migrate the local pins in
src/agent/Cargo.toml and src/libs/kata-types/Cargo.toml to
workspace = true. No API changes required — the Digest trait
interface is unchanged across this range.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add sysinfo to the workspace dependencies at 0.39.5 and switch
kata-types from its local pin to workspace = true. No API changes
required — System::new_all() is stable across this range.
sysinfo 0.39.5 requires rustc >= 1.95. Update the workspace
rust-version to match the toolchain already set in rust-toolchain.toml.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Remove the local 0.13.0 pin and defer to the workspace version (0.22).
Migrate all callsites from the deprecated free-function API to the
engine-based API introduced in 0.20:
base64::encode(x) -> STANDARD.encode(x)
base64::decode(x) -> STANDARD.decode(x)
base64::encode_config(x, STANDARD) -> STANDARD.encode(x)
base64::decode_config(x, STANDARD) -> STANDARD.decode(x)
base64::encode_config(x, URL_SAFE) -> URL_SAFE.encode(x)
Add `use base64::Engine as _` to bring the trait methods into scope in
initdata.rs and mount.rs.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Some are format expectation changes in 1.95, but
I think some are just missing fmts that our CI doesn't
check.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Remove redundant `.into_iter()` call on `zip()` argument in
`multi_layer_erofs.rs`, as flagged by the new
`clippy::useless_conversion` lint in Rust 1.95.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Replace manual `if us != 0` guard before integer division with
`checked_div` as flagged by the new `clippy::manual_checked_ops` lint
in Rust 1.95.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Replace manual `if us != 0` guard before integer division with
`checked_div` as flagged by the new `clippy::manual_checked_ops` lint
in Rust 1.95.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Remove redundant `.into_iter()` call in `Vec::from_iter()` argument in
dragonball and firecracker's `inner_hypervisor.rs`, as flagged by the new
`clippy::useless_conversion` lint in Rust 1.95.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Rust 1.95's clippy introduces new `collapsible_match` lints for `if`
blocks nested inside match arms that can be expressed as match guards.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Rust 1.95's clippy introduces new `collapsible_match` lints for `if`
blocks nested inside match arms that can be expressed as match guards.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The CLH runtime-rs config template uses @CONFIG_CLH_IN@ in its
source-file comment, but CONFIG_CLH_IN was missing from the generated
template variables.
Add it to USER_VARS so the generated config records the template path
instead of keeping the literal placeholder.
Fixes: #12941
Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com>
Docker and containerd rely on the TaskExit event and a prompt Wait RPC,
not just the eventual sandbox teardown. With the original ordering the
shim tore the sandbox down (Stop/Delete the guest) *before* publishing
the container exit, so for a slow guest shutdown -- e.g. the nvidia-gpu
config with an 8GB /dev/shm memory-backend on a constrained CI runner --
containerd could SIGKILL the shim before the exit was published. The
`docker run --rm` removal then races the dead-shim cleanup and fails,
even though the container itself exited 0.
Publish the container exit (c.exitCh) and the TaskExit event *before*
the sandbox teardown, and run the teardown without holding s.mu so
concurrent Delete()/Shutdown() RPCs are not blocked.
To keep the guarantees the previous ordering provided:
* teardownWg lets Shutdown() wait for an in-flight teardown to finish,
so the sandbox run directory (watched by kata-monitor) and the CRI
state are gone before the shim exits -- without holding s.mu across
the slow guest shutdown; and
* teardownOnce serialises wait()'s teardown with watchSandbox()'s
killed-VMM teardown so the (not internally synchronized)
Sandbox.Stop/Delete never run concurrently.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
For `docker run --rm`, containerd invokes the shim `delete` binary
(cleanupAfterDeadShim) once the container task exits. Kata's Cleanup
path re-loads the sandbox and calls StopContainer/DeleteContainer/Stop,
each of which lazily connects to the guest agent over vsock.
When the sandbox was already torn down by the main shim (the common
case for a short-lived `docker run --rm`), the VM -- and its agent --
are gone, so that vsock connect blocks until containerd's delete
timeout SIGKILLs the binary. The removal then fails and `docker run
--rm` returns non-zero even though the container itself exited 0.
Detect the already-dead hypervisor (its pidfile is gone / the pid no
longer maps to a live process) at the start of CleanupContainer and
mark the agent dead. Subsequent agent RPCs then fail fast with "Dead
agent" and the force path performs only host-side cleanup, so the
delete binary returns promptly instead of hanging.
The legitimate "shim crashed but VM still alive" cleanup is unaffected:
the hypervisor is still running, so the agent is not marked dead and
the normal agent-based teardown proceeds.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
virtio-mem is enabled on s390x, where memory hotplug goes through a
virtio-mem-ccw device set up during VM initialization. That device --
and the later resize -- require a memory hotplug region (a non-zero
maxmem/slots on the QEMU command line).
The runtime-rs command line generator, however, zeroes maxmem/slots
whenever the guest uses the shared /dev/shm memory-backend with a
non-nvdimm rootfs (the s390x case: virtio-blk-ccw rootfs). With no
hotplug region reserved, two things broke on s390x:
* setup_virtio_mem failed VM start with "the configuration is not
prepared for memory devices, consider specifying the maxmem option";
* with virtio-mem unavailable, memory resize fell back to pc-dimm,
which is not a valid device model on s390x, breaking
TestContainerMemoryUpdate:
'pc-dimm' is not a valid device model name
Keep the hotplug region when virtio-mem is enabled, mirroring the Go
runtime which reserves maxmem and hotplugs via virtio-mem-ccw on s390x.
This lets setup_virtio_mem succeed at boot and makes memory resize use
virtio-mem instead of pc-dimm.
Also guard setup_virtio_mem behind QemuCmdLine::has_memory_hotplug_region()
as a defensive fallback: if some configuration still ends up without a
hotplug region, skip virtio-mem setup (like static-sizing arches such as
arm64) rather than failing VM start.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
NVIDIA GPU configs default pod_resource_api_sock to the kubelet Pod
Resources API path. On non-Kubernetes hosts that path is usually missing;
use CDI sandbox annotations for cold-plug instead of failing kubelet lookup.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The container create flow forced every non-pod-container to the
"pod_sandbox" type. Standalone engines (Docker/nerdctl/podman) surface
as SingleContainer, so they were mislabeled as pod sandboxes.
The agent skips CDI device injection when the container type is
"pod_sandbox", so the NVIDIA CDI edits carried in the "cdi.k8s.io/*"
annotations were never applied and the GPU userspace (e.g. nvidia-smi)
was missing in the guest. Emit the actual container type instead, which
matches the Go runtime and lets the agent inject CDI devices for the
single-container flow.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
ContainerType already renders as "single_container" via Display, but
from_str rejected it. Accept it so the value can round-trip, matching
the container type the Go runtime uses for standalone containers.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
