Add k8s-devkit-debug-console.bats, which launches a pod with the
kata-<shim>-devkit RuntimeClass, resolves its sandbox id from the node, and
reaches the agent debug console via `kata-ctl exec`. The console is an
interactive PTY, so it is driven with `script` (util-linux) fed through a FIFO,
and it asserts we land in the devkit shell (a working apk, plus an Alpine guest
id) rather than the base guest shell.
The test targets the two non-confidential runtime-rs classes it is validated on
(qemu-runtime-rs, qemu-nvidia-cpu-runtime-rs) and skips elsewhere or when the
kata-<shim>-devkit RuntimeClass is absent (devkit not deployed). Register it in
the small-host suite, add a HELM_DEVKIT knob to the Helm test helper, and turn
it on for those two hypervisors in the k8s CI (which already runs with debug).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add an opt-in "devkit" path to kata-deploy that wires the devkit debug guest
extension into a dedicated per-shim runtime, so operators can debug a sandbox
with a rich shell without touching their normal RuntimeClasses.
Helm gains a `devkit` value (only effective together with `debug`, since the
debug console must be enabled to reach it). When set it emits DEVKIT=true and,
for every enabled shim, an extra `kata-<shim>-devkit` RuntimeClass reusing the
shim's overhead and node selectors. The RuntimeClass template is refactored to
compute name/handler from an optional `variant`, keeping existing behavior.
In the deploy binary, DEVKIT synthesizes one kata-<shim>-devkit custom runtime
per enabled shim. Modeling it as a custom runtime reuses the whole custom
runtime install/register/cleanup machinery; the only devkit-specific piece is a
25-devkit.toml drop-in that cold-plugs the extension image
(guest_extension_images name="devkit", with verity_params read from
root_hash_devkit-extension.txt when present) and points the agent debug console
at /run/kata-extensions/devkit/bin/devkit-sh. The generic
rootfs-image-devkit-extension tarball is extracted based on the devkit flag
rather than per-shim shim-components.json membership.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a debug_console_shell field to the [agent] config and, when it is set,
inject agent.debug_console_shell=<path> onto the guest kernel cmdline next to
the existing debug_console options. This is how the runtime-rs shims tell the
guest agent which shell the debug console should exec (the agent added support
for the option and its extension-prefix guard in a previous commit).
An empty value is not emitted, so default configurations are unchanged.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add an agent.debug_console_shell option, parsed from the kernel cmdline
(mirroring agent.debug_console_vport) and overridable from the agent config
file. When set, the debug console prefers it over the built-in /bin/bash and
/bin/sh candidates, falling back to those if it is missing at runtime. This
lets an operator point the debug console at a richer shell shipped in a guest
extension (e.g. the devkit extension) without rebuilding the base rootfs.
The configured shell is only honored if it resolves - after canonicalization,
so symlink and ".." escapes are caught - to a path under /run/kata-extensions/.
Extension images are dm-verity measured and read-only, so this confines the
debug console to trusted, integrity-verified binaries: a host that can
influence the agent kernel cmdline (untrusted in the CoCo threat model) cannot
repoint the debug console at a container's rootfs to run tenant binaries or
read confidential data in the guest-root namespace. Anything outside the
prefix is ignored and the built-in candidates are used instead.
No runtime emits this option yet; that follows in later commits.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a self-contained "devkit" guest extension: a minimal Ubuntu (glibc +
busybox + apt) rootfs with common debug tools prebaked (strace, ltrace,
iproute2, procps, lsof, tcpdump, pciutils, util-linux, ...), built as a
measured erofs+dm-verity image mounted at /run/kata-extensions/devkit.
The production guest rootfs is minimal (and, for some bases, shell-less), so
the agent debug console has no rich interactive shell. Rather than rebuilding
the whole rootfs with debug tooling, this optional extension can be
cold-plugged alongside any base image. At runtime the guest helper scripts
overlay a writable tmpfs on the read-only extension and chroot in, so apt and
every tool run natively against a normal root filesystem; `apt install <pkg>`
inside the debug shell pulls anything else into the overlay on demand.
busybox-static (/usr/bin/busybox.static) bootstraps the overlay/chroot from the
shell-less base (the guest's dynamic loader is not present there yet); it is
installed at a dedicated path so it never clobbers the extension's own busybox
or /bin/sh, which are needed unclobbered inside the chroot.
The rootfs is built through osbuilder (rootfs.sh, with a ROOTFS_ONLY mode that
skips the agent/systemd setup) using mmdebstrap, the same tool the base guest
rootfs uses, so the devkit needs no bespoke chroot or docker-export logic. The
resulting tree is handed to image_builder.sh (mirroring the CoCo extension): it
ships no kata-agent, kernel or driver userspace.
No runtime wiring is added here; that follows in later commits. Nothing
consumes the image yet.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
kata-ctl provides `kata-ctl exec <sandbox-id>`, the runtime-rs entry point
into a sandbox's agent debug console. It ships in the kata-tools bundle
rather than with the shim binaries, so it is not staged into the kata-deploy
image and never lands on the node - leaving runtime-rs deployments without a
first-class way to reach the debug console.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
kata-ctl was only produced as an amd64 tools asset; on arm64 the
build-tools-asset matrix built genpolicy alone, so no arm64 kata-ctl tarball
existed for the kata-deploy image to stage. Add kata-ctl to the arm64 tools
matrix so `kata-ctl exec` is available on arm64 nodes when debug is enabled.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Dragonball now identifies every network device by its iface_id and
refuses a device without one, so pass the endpoint's dev_id through
explicitly instead of leaving the field to its empty default. This
completes the change started two commits back: every network backend
hands dragonball a device name the runtime knows.
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
virtio-net, vhost-net and vhost-user-net were spread across three
device managers with three config types, three error enums and three
insert paths, even though they are all network interfaces of the guest
behind one InsertNetworkDevice request. Merge them into a single
NetworkDeviceMgr with one info_list, dispatching on the Backend enum
per device, the way the block manager keeps its backends in one list
keyed by BlockDeviceType. The unified NetworkInterfaceConfig moves
from api/v1 into the manager, as the other managers define theirs,
and the per-backend config conversions disappear.
Device identity becomes the caller-supplied iface_id for every
backend, matching how every other device class is identified, instead
of vhost-user-net borrowing its socket path as its id. The socket path
becomes a conflict-checked resource alongside the tap name and the
guest MAC, so two devices claiming one vhost-user socket are refused
(DuplicatedUdsPath) rather than silently collapsing into one entry.
Behaviour changes, all deliberate:
- A device with an empty iface_id is refused (MissingIfaceId). The id
is the interface name inside the guest, so an unnamed device would
silently take the place of another unnamed one and the VM would boot
with an interface missing. Callers that relied on inserting
vhost-user-net devices without an id must now supply one.
- Hotplugging a device whose id is already in use is refused
(DeviceIDAlreadyExist): the old config-update path would have left
the already-attached device live on the bus with no handle left to
remove it. Before boot, a repeated id still reconfigures the device
as it does for every other device class.
- Teardown no longer stops at the first device that fails to be
destroyed; the failure is logged and the remaining devices are still
removed. This path newly matters because vhost-user-net devices now
retain their device handle like the other backends do.
The unused open_tap helper (never called since the file was introduced)
and the NetworkDeviceInfo alias are dropped rather than carried over.
vhost-user-net devices inserted through runtime-rs receive their id in
the following commit; with only this commit they are refused for the
missing id.
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
The virtio-net and vhost-net paths hand dragonball an iface_id taken
from the virt interface name, but the vhost-user-net endpoint left
dev_id unassigned, so the device carried no name of its own and was
identified by its socket path inside the hypervisor. Assign dev_id from
the endpoint's interface name so every network backend is identified
the same way by a name the runtime knows.
Also construct dragonball's vhost-user config with struct update syntax
so this commit builds both before and after dragonball's config gains
an iface_id field; the follow-up commit passes dev_id through
explicitly once that field exists.
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
The SE VMs how-to documented building the s390x boot image on an x86_64
host via CROSS_BUILD=true TARGET_ARCH=s390x ARCH=s390x.
Cross builds are not exercised on our side and the supporting code has been
removed, so drop this instruction; the SE boot image is built on a native
s390x host.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
When the requested ARCH differed from the host, install_libseccomp.sh
selected a <arch>-linux-gnu-gcc cross compiler and passed --host=<arch> to
configure to cross-build the static libseccomp.
This is not exercised on our side: libseccomp is always built for the host
arch. Build it natively without the cross CC or --host.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
kata-ctl shipped a Cross.toml and a "Cross-builds" README section
describing how to cross compile the tool for other architectures with
cross-rs (installing foreign-arch libssl via dpkg multiarch pre-build
hooks).
This is not exercised on our side: kata-ctl is built on native-arch
runners. Remove Cross.toml and the cross-build documentation.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Building a foreign-arch rootfs/image relied on CROSS_BUILD to enable docker
buildx/--platform emulation, a <arch>-linux-gnu-strip stripping tool, an
Ubuntu builder installing gcc-<arch>-linux-gnu + libc6-dev-<arch>-cross
(via the @ARCH@ template) and config.sh exporting a cross CC.
None of this is exercised on our side: rootfs and image are always built
on a native-arch runner. Remove the CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM
plumbing, the cross toolchain/@ARCH@ substitution and the cross CC export;
the guest agent is stripped with the native `strip`.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The guest kernel build derived a CROSS_COMPILE=<arch>-linux-gnu- prefix
whenever the target arch differed from the host, the static-build wrapper
switched to a per-arch `-cross-build` image (with buildx/--platform for
s390x), and the builder image installed gcc-<arch>-linux-gnu /
binutils-<arch>-linux-gnu cross toolchains.
None of this is exercised on our side: the kernel is always built on a
native-arch runner. Remove the CROSS_COMPILE plumbing, the cross-build
image/buildx handling and the cross toolchain install.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Building QEMU for a foreign arch relied on configure-hypervisor.sh
emitting --cross-prefix, build-qemu.sh passing --cc/--cross-prefix for
s390x, and the builder image installing foreign-arch -dev packages
(via dpkg multiarch/DPKG_ARCH) plus a gcc-<arch>-linux-gnu cross toolchain.
None of this is exercised on our side: QEMU is always built on a
native-arch runner. Remove the cross-prefix logic, the s390x cross
configure branch, and the DPKG_ARCH/cross-gcc plumbing from the Dockerfile
and build-base-qemu.sh.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The shared lib.sh exported CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM so the
per-component static-build scripts could switch to `-cross-build` builder
images, run `docker buildx build --platform=...` under QEMU emulation and,
for shim-v2, swap in the messense/rust-musl-cross image with a matching
cross CC.
These cross paths are not exercised on our side (all builds run on
native-arch runners), so remove them: builder images are now built with a
plain `docker build`, and shim-v2 builds runtime-rs natively without the
musl-cross image or CC override.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The kata-deploy local-build pipeline auto-enabled a cross build whenever
the host arch differed from TARGET_ARCH, wiring up docker buildx, QEMU
binfmt emulation and an s390x cross toolchain (incl. building s390-tools
genprotimg with CROSS_COMPILE) in the builder image.
None of this is exercised on our side: every release/CI build runs on a
native-arch runner, so the cross paths are dead code. Remove the
CROSS_BUILD/TARGET_ARCH/BUILDX/PLATFORM/binfmt machinery and the s390x
cross toolchain from the builder image; the tarballs are always built for
the host arch now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The `HOST_ARCH != ARCH` branch in utils.mk existed solely to support
cross-compiling the Rust components (overriding the linker via `CC`).
Cross-compilation is not exercised on our side: all release/CI builds run
on native-arch runners, so this path is dead code. Remove it.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Fail startup when KATA_CONF_FILE does not resolve to one of the shipped
default config files, preventing untrusted config path injection via
environment overrides.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Stop accepting io.katacontainers.config_path so config selection comes
only from shim options, KATA_CONF_FILE, and defaults.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The sandbox_bind_mounts option accepts host paths that runtime-rs
exposes to the guest shared filesystem. Runtime annotations are not
gated by the hypervisor annotation allowlist, allowing an untrusted
workload annotation to add arbitrary host paths to the sandbox bind
mount configuration.
Ignore KATA_ANNO_CFG_SANDBOX_BIND_MOUNTS when processing workload
annotations. Sandbox bind mounts remain available through the
operator-controlled TOML configuration, preserving the sandbox-level
host/guest sharing feature while removing the untrusted annotation
path.
Really appreciate it that Anthropic and Ada Logics report this issue.
Reported-by: Anthropic and Ada Logics
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Move PCI enumeration, VFIO helpers, and InfiniBand diagnostics out of
kata-sys-util into the external pcilibs-rs crate so they can be reused
independently of kata-containers.
Depend on pcilibs-rs 0.1.0 from crates.io.
Add pcilibs to the cspell dictionary for the new crate reference in
src/libs/README.md.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
open() waits for the binary logger readiness handshake with an unbounded
async read(). If the logger never writes to fd 5 (or inherits a bad fd),
container start can hang indefinitely.
Let's add a timeout around the readiness read and fail the start path if
it expires (the child will be killed on drop due to kill_on_drop(true)).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Suggested-by: CoPilot
Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Set the non-confidential NVIDIA runtime-rs emptyDir mode to
block-plain through a dedicated Makefile default.
This prepares the runtime class for configurations where filesystem
sharing is disabled and emptyDir volumes need to be backed by
guest-mounted block devices instead of shared-fs.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Support containerd binary:// stdio URIs in runtime-rs by spawning the
logger with stdout, stderr, and readiness on file descriptors 3, 4, and 5.
Preserve URI arguments and container metadata, and stream output through
nonblocking pipes with backpressure.
Processes using a binary logger fall back from passfd to the existing agent
streams so the host shim can feed the logger. After EOF, allow the logger to
drain before using SIGTERM and SIGKILL as bounded fallbacks.
Fixes: #4391
Related: #4420
Signed-off-by: cayoub-oai <276123840+cayoub-oai@users.noreply.github.com>
Assisted-By: OpenAI Codex
Commit e8bb619b1b ("runtime: do not require OVMF for qemu-nvidia-cpu
on arm64") dropped the generic OVMF/AAVMF firmware inheritance for the
Go qemu-nvidia-cpu runtime, but assumed runtime-rs already did the right
thing by default. It didn't: the runtime-rs profile still pointed at the
generic @FIRMWAREPATH@ / @FIRMWAREVOLUMEPATH@ and still pulled in the
ovmf component on aarch64.
Mirror the fix on the runtime-rs side by introducing dedicated empty
FIRMWAREPATH_NV_CPU / FIRMWAREVOLUMEPATH_NV_CPU variables, wiring them
into the qemu-nvidia-cpu-runtime-rs configuration, and dropping ovmf
from its aarch64 shim components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
The termination-log tests exercise termination-message propagation over
the agent GetDiagnosticData RPC, which only matters when the host cannot
read guest files directly, i.e. on shared_fs=none configurations. They
previously gated on is_confidential_runtime_class and therefore skipped
on the NVIDIA CPU runtime-rs handler, which now also runs with
shared_fs=none.
Add an is_shared_fs_none_runtime_class helper (confidential runtime
classes plus qemu-nvidia-cpu-runtime-rs) and use it for the setup and
teardown gating so the tests run on the NVIDIA CPU runtime-rs handler as
well. The plain Go qemu-nvidia-cpu class still uses virtio-fs and is
intentionally excluded. Keep the "blocked by default CoCo policy"
assertion confidential-only, since non-confidential shared_fs=none
classes do not ship a default policy that denies the RPC.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Make qemu-nvidia-cpu-runtime-rs the default shim on amd64 and arm64 for
the kata-deploy NVIDIA CPU profile, so installs from
try-kata-nvidia-cpu.values.yaml select the Rust runtime by default
instead of the Go qemu-nvidia-cpu shim.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
EROFS fs-verity depends on the backing filesystem supporting fs-verity,
which cannot be guaranteed on an arbitrary node. Rather than gate the
NVIDIA CPU profile on a host feature we do not control, disable fs-verity
so it works on the widest possible range of setups; we take that hit
deliberately to keep the profile generic.
Move this out of the test harness and into the kata-deploy NVIDIA CPU
profile: try-kata-nvidia-cpu.values.yaml now ships a containerd erofs
snapshotter drop-in that disables fs-verity and pins the memory-backed
default_size, and selects memory-backed rw layers with dm-verity for the
lower (image) layers. Because containerd.userDropIn is loaded after
kata-deploy's generated config, it overrides the built-in enable_fsverity
default.
The CI helm helper no longer special-cases NVIDIA CPU to inject
enable_fsverity=false. Instead it honors an erofs snapshotter drop-in
already provided by the base values file and only synthesizes the default
default_size drop-in when the profile does not provide its own.
Note that this drop-in targets containerd's built-in EROFS snapshotter,
available only on containerd >= 2.2.0 (config version 3, conf.d auto-import),
which is the same minimum kata-deploy already enforces for the EROFS
snapshotter.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Set the non-confidential NVIDIA runtime-rs emptyDir mode to
block-plain through a dedicated Makefile default.
This prepares the runtime class for configurations where filesystem
sharing is disabled and emptyDir volumes need to be backed by
guest-mounted block devices instead of shared-fs.
Also add the shared configure_nvidia_runtime_rs_shared_fs_dropin()
test helper, which keeps the NVIDIA runtime-rs Docker/nerdctl smoke
tests on virtio-fs while the shared_fs=none + EROFS snapshotter path
is exercised by Kubernetes CI.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
