Add a composable devkit extension (shell-enabled busybox) cold-plugged
alongside the NVIDIA base and gpu images so kata-runtime exec works without
rebuilding the shell-less production rootfs. Wire build and packaging,
runtime-rs drop-in examples, guest_extension_images append merge, kata-deploy
debug cold-plug with dm-verity params, and agent console shell lookup at
/run/kata-extensions/devkit.
Devkit bin/sh and devkit-apt wrappers use a shebang pointing at the
extension busybox (not /bin/sh): the NVIDIA base rootfs is shell-less, so
kernel shebang resolution against the guest root would fail and debug
console sessions would drop immediately.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The SE VMs how-to documented building the s390x boot image on an x86_64
host via CROSS_BUILD=true TARGET_ARCH=s390x ARCH=s390x.
Cross builds are not exercised on our side and the supporting code has been
removed, so drop this instruction; the SE boot image is built on a native
s390x host.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Fail startup when KATA_CONF_FILE does not resolve to one of the shipped
default config files, preventing untrusted config path injection via
environment overrides.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Stop accepting io.katacontainers.config_path so config selection comes
only from shim options, KATA_CONF_FILE, and defaults.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
This patch replaces the favicon on the mkdocs site with an
official Kata Containers logo made available by the OpenInfra
Foundation.
Signed-off-by: Ildiko Vancsa <ildiko.vancsa@gmail.com>
Document operational requirements for using the EROFS snapshotter
with Kata Containers.
Call out that EROFS layer conversion can happen during
CreateContainer, not only during image pull, so kubelet
runtimeRequestTimeout may need to be increased for large images.
Also document the kernel modules needed for EROFS and dm-verity,
the erofs-utils version required by the configured mkfs options,
and the distinction between host-side fs-verity and guest-visible
dm-verity.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Flesh out the quick start with the content from #13220 while keeping
mkdocs-material admonitions, definition lists, and links to installation.md.
Replace the manual Next steps nav block with in-prose cross-references, use
absolute GitHub URLs for out-of-tree source links, and update the top-level
README packaging and architecture links.
Fixes: kata-containers#12999
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Switch the per-architecture default shim from the Go runtime ("qemu") to the
Rust runtime ("qemu-runtime-rs") wherever a runtime-rs build exists, and
align CI/test fallbacks with the new default.
Document the defaultShim values and Go runtime deprecation in the Helm and
config migration guides using mkdocs admonitions.
Fixes: kata-containers#9546
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Catalogue the configuration differences between the Go runtime and
runtime-rs for the QEMU hypervisor, using mkdocs admonitions and absolute
GitHub URLs for out-of-tree source links.
Address review feedback from #13236: remove the outdated msize_9p and
Summary sections, clarify that both runtimes use enable_debug while
log_level is a runtime-rs addition, and rename the architecture 4.0 page
so mkdocs does not treat it as a section index.
Co-authored-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Remove the RuntimeClass table in favour of prose, map `uname -m` to the
release tarball architecture names (amd64, arm64, s390x, ppc64le), mark the
Go runtime as deprecated, and point README links at installation.md.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a try-kata-nvidia-cpu.values.yaml example for the CPU-only
NVIDIA runtime classes and make the Kubernetes CI Helm helper use it
for qemu-nvidia-cpu handlers.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Roll the missing pieces into the installation guide: hardware
prerequisites (KVM, vmx/svm, vhost modules, nested-virt), an
install-method comparison table, RuntimeClass selection with a test pod,
the pre-built tarball flow with /opt/kata/ paths, and a compact Docker
section. The existing Helm chart instructions and the DaemonSet-vs-Job
deployment-mode note are preserved.
Notes use mkdocs-material admonitions (note/tip/warning) instead of `>`
blockquotes, and cross-repo source links use absolute GitHub URLs so they
resolve on the rendered docs site. Adds `mshv` (Microsoft Hypervisor
device, used in the clh-azure runtimes) to the cspell dictionary.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
The proposal told us to move it under docs/design/ once it was accepted
and implemented. The systemd base landed on main and the NVIDIA split
(nvidia base + gpu extension) is what this series wires up, so it's no
longer a proposal - promote it to docs/design/composable-vm-images.md,
flip the status accordingly and repoint the in-tree references (agent,
osbuilder and kata-deploy) at the new path.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Carve the monolithic NVIDIA GPU rootfs into a driver-agnostic
nvidia base image (NVRC init + agent + base libs + in-tree modules)
and a driver-versioned gpu extension (GPU userspace, configs, firmware,
NVIDIA modules) laid out for /run/kata-extensions/gpu. Both halves are
produced from the same chiseled tree via a partition-after step, so
the monolith build stays byte-identical and continues to ship.
The shared stage-one (driver install) is reused across the monolith,
base and extension variants; nvidia base is cached without the driver/ctk
versions so one base image can back multiple driver extensions.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Merge NFD and user nodeSelectorTerms with merge/concat.
Ignore the name key in podLabels, document podAnnotations
and extend bats coverage for annotations.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Expose podLabels, podAnnotations and affinity in values.yaml so
operators can apply more granular scheduling controls on the
kata-deploy DaemonSet alongside the existing nodeSelector and
tolerations knobs.
Defaults are empty, so existing deployments render the same manifest.
When NFD is enabled, the chart merges the hard-coded virtualization
nodeAffinity with any user provided affinity.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Block-backed emptyDir volumes are sized using the capacity of the host
filesystem rather than Kubernetes sizeLimit. On large filesystems,
ext4 metadata can consume enough space to trigger pod eviction.
Document the expected overhead, its effect on usable storage, current
mitigations, and the issue tracking potential improvements.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Update OS version and containerd references, and make a few minor
adjustments and clarifications. For the clarifications, move the
section "Deploy pods with Kata agent security policies" further up
so that the installation of the genpolicy tool has been elaborated
on before attempting to run NVIDIA CI tests locally.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
The `hotplug_vfio_on_root_bus` configuration option is no longer needed
in the runtime-rs.
Remove the option from:
- `DeviceInfo` struct and its `hotplug_vfio_on_root_bus` field
- `PCIeTopology` struct and its construction in `topology.rs`
- `HypervisorInfo` struct and its `hotplug_vfio_on_root_bus` field
- `KATA_ANNO_CFG_HYPERVISOR_HOTPLUG_VFIO_ON_ROOT_BUS` annotation constant
and its handler in `Annotation::apply()`
- DragonBall validation check in `ConfigPlugin::adjust_config()`
- All runtime-rs TOML config templates (qemu, fc, nvidia GPU variants)
- Test fixture TOML files
Update the documentation having a reference to it.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Align the composable VM images proposal with the naming convention used
throughout the implementation: the generic mechanism and its data model
are now "guest extension image" (config key `guest_extension_images`,
struct `GuestExtensionImage`), and the runtime artifacts use the matching
"extension" vocabulary (`/run/kata-extensions/<name>`,
`kata.extension.<name>.verity_params`, `kata-extension-mount@.service`,
the `extension-<name>` virtio-blk serial / dm-verity target, and the
`kata-containers-coco-extension.img` image).
It also documents how template-unit instances are enabled: a systemd
generator instantiates kata-extension-mount@<name>.service for each
extension on the kernel command line, so the rootfs build stays
extension-agnostic.
This replaces the earlier mixed "extra_images"/"addon" terminology and
addresses the naming-consistency review feedback on the series.
Assisted-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a how-to describing how runtime-rs sizes static sandboxes from
overhead plus requested CPU/memory, including that fractional vCPU
results are rounded up for VMM-visible vCPU counts, and link it from the
how-to README.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Update the composable-vm-images proposal with the design decisions we only
arrived at after experimenting with the implementation:
* Replace the hardcoded agent path-resolution table with the data-driven
components.toml manifest (process levels, args/optional_args, env,
wait_socket, ${...} substitution, and select/variants), keeping the agent
generic.
* Document the attester-variant contract: NVRC exports KATA_ATTESTER_VARIANT
and the manifest selects the stock vs NVIDIA attestation-agent.
* Document the runtime dependency requirements found during bring-up: the
nvidia attester's LD_LIBRARY_PATH (libnvat closure in the coco addon +
NVML in the gpu addon) and the NVML-init failure mode, plus CDH
secure_mount tooling placement -- plain storage (mke2fs/mkfs.ext4/dd) in
the base vs encrypted storage (cryptsetup) in the coco addon, the CDH
PATH, and the base/addon ABI lockstep.
* Reflect the storage tooling and bundled libraries in the base/coco-addon
build sections, and mark the GPU addon as implemented.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
I created this over the course of testing my VISIBLE_CDI_DEVICES
changes. I think this will be useful to folks who don't understand the
right way to deploy custom artifacts.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
Switch the NVIDIA GPU example values file to install Kata via the
Job-based deployment mode (deploymentMode: job) instead of the
always-on, privileged DaemonSet, so that nothing keeps running on the
node once the install completes.
To exercise this in our CI, make the helm_helper aware of the deployment
mode coming from the (base) values file:
- In "job" mode, clear job.nodeSelectorExpressions so the dispatcher
targets every discovered node. Our CI clusters are typically
single-node, where the only node carries the control-plane label,
and the default selector excludes control-plane/master nodes.
- There is no always-on DaemonSet to wait on in "job" mode. The
dispatcher runs as a blocking post-install hook and the final
per-node stage labels the node, so wait until at least one node
carries the katacontainers.io/kata-runtime label as the
"install complete" signal (dumping Job/pod logs on timeout).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Clean the runtime configuration section by focusing first on the helm
configuration. Then, pivot into a further explanation on how the runtime
can be directly configured. Link to where these config parameters are
explained more in-depth.
Add open-in-new-tab (already downloaded in requirements.txt) in the
mkdocs plugin config so that links don't open in the same tab.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
Document the new opt-in deploymentMode: job alongside the default
DaemonSet model in the maintained docs (not just the chart README):
- helm-configuration.md: add a "Deployment Modes (DaemonSet vs Job)"
section covering the dispatcher-driven staged install/cleanup
pipelines, why a dispatcher is used instead of Helm-rendered per-node
Jobs (O(1) release, guaranteed coverage, paced rollout, explicit
privilege split), the "re-run helm upgrade to cover newly added
nodes" model (no always-on reconcile component), and the
node-selection precedence (job.nodes > job.nodeSelector +
job.nodeSelectorExpressions) that defaults to worker nodes.
- installation.md: note that the DaemonSet is the default but no longer
the only model, linking to the section above.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Pin idna to 3.15 and pymdown-extensions to 10.21.3 to address
security vulnerabilities:
- GHSA-65pc-fj4g-8rjx (idna, severity 6.9)
- GHSA-62q4-447f-wv8h (pymdown-extensions, severity 4.3)
- GHSA-r6h4-mm7h-8pmq (pymdown-extensions, severity 2.7)
These dependencies were previously transitive and vulnerable.
They are now explicitly pinned to secure versions.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Remove the Go runtime file_mem_backend and valid_file_mem_backends
config knobs, along with the corresponding sandbox annotation handling.
The runtime still enables file-backed shared memory automatically for
virtio-fs by using /dev/shm as the backing directory. This only removes
the user-selectable backend path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
While the config knob is being parsed, it is being unused in the
rust shim. This renders the config knob useless. Remove the
file_mem_backend config option as there is no current users for it.
As this option is being usable in the go shim, we leave it intact.
For the rust shim, /dev/shm is still being used in a similar way to
the go shim when filesystem sharing is enabled (virtio-fs). Future
use cases where other file_mem_backends are being utilized are
currently planning to define these backends in a similar manner:
based on the configuration/platform, determine the proper file
memory backend, but do not let end users determine the file memory
backend.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Add comprehensive documentation for using virtio-fs-nydus shared
filesystem with Kata Containers. This guide covers:
(1) Clarify configuration options for virtio-fs-nydus and nydus image
preparation and usage.
(2) Update daemon configuration and lifecycle management and introduce
standalone, inline nydus architecture.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
kata-monitor is published as a standalone container image starting
with 3.32.0; point users at it from the metrics design doc and the
Prometheus-on-Kubernetes how-to, and switch the DaemonSet manifest to
the dedicated image (keeping the runtime endpoint/listen settings and
hostPath cleanups).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Allow operators to provide per-shim drop-in TOML for built-in runtimes
and reconcile stale override files so upgrades and migrations remain
safe when drop-ins are added or removed.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Codex
Add an RFC document describing the composable image architecture that
replaces monolithic guest rootfs images with a lean base image plus
purpose-specific addon images cold-plugged as virtio-blk devices.
The proposal covers the runtime configuration (extra_images), host-side
cold-plugging, guest-side mounting via systemd and dm-verity, agent-side
dynamic path resolution, the image build pipeline, and the security
model.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Migrate trace-forwarder from the deprecated opentelemetry-jaeger
exporter to the modern opentelemetry-otlp exporter.
This change remediates GHSA-2f9f-gq7v-9h6m (CVE-2026-43868), a
medium-severity vulnerability in Apache Thrift. The opentelemetry-jaeger
crate is no longer maintained and depends on vulnerable thrift versions
(0.13.0 and 0.16.0). The opentelemetry-otlp exporter does not use thrift
and is actively maintained.
Changes:
- Replace opentelemetry-jaeger with opentelemetry-otlp in Cargo.toml
- Update tracer.rs to use OTLP exporter instead of Jaeger exporter
- Replace --jaeger-host/--jaeger-port flags with --otlp-endpoint flag
- Update server.rs to use TracerProvider instead of SpanExporter
- Update documentation to reflect OTLP migration
- Add examples for common OTLP-compatible collectors
Breaking change: Users must update their trace-forwarder invocations
to use --otlp-endpoint instead of --jaeger-host and --jaeger-port.
Default endpoint: http://localhost:4317 (OTLP gRPC)
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Co-authored-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Update the debugging guidance to explain the shared enable_debug
baseline for Go and runtime-rs. Document runtime-rs component log_level
controls and clarify that containerd debug is not required for
runtime-rs Kata logs in journald.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
As max_unmerged_layers = 1 is just for fsmerge mode, as containerd
temperally unsupport fsmerge, we just reset it with default 0.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add k8s-nvidia-numa.bats with five tests that validate NUMA behaviour
on hosts where NUMA is configured by default (qemu-nvidia-gpu,
qemu-nvidia-gpu-snp, qemu-nvidia-gpu-tdx):
1. Multi-node sandbox (large workload spanning all host NUMA nodes):
- Guest NUMA node count matches host
- Guest vCPU distribution is balanced across nodes (max-min <= 1)
- Guest memory is distributed across NUMA nodes
- Host-side vCPU pinning is balanced across NUMA nodes
2. Right-sized single-node sandbox (small workload fitting one node):
- Guest collapses to a single NUMA node
- All host vCPU threads pinned to that one NUMA node
3. GPU passthrough with VFIO, multi-node:
- Guest NUMA topology is balanced (same as test 1)
- Guest GPU's NUMA node matches the host GPU's NUMA node
(resolved via the vfio-pci,host=<BDF> from the QEMU command
line and /sys/bus/pci/devices/<BDF>/numa_node)
- QEMU command line contains pxb-pcie and policy=bind
- Host vCPU pinning is balanced
4. GPU passthrough with VFIO, right-sized single-node: small workload
plus GPU that fits in a single host NUMA node:
- Guest collapses to a single NUMA node
- The chosen node is the GPU's host NUMA node, not just any node
that fits — verified by matching host-nodes= in the memory
backend and pxb-pcie numa_node= against the GPU's host node
- Guest GPU reports the same NUMA node as the host GPU
5. Explicit numa_mapping in the runtime TOML (QEMU-only):
- Drops a config.d/ fragment that sets numa_mapping = ["1"], so the
auto-derive + right-sizing path is bypassed entirely
- Guest sees exactly 1 NUMA node
- QEMU memory backend is bound to host node 1 (host-nodes=1,
policy=bind), not host node 0
- Host-side vCPU threads land on host node 1
- Drop-in is removed on teardown so subsequent tests are unaffected
Guest-side checks use a dedicated container image
(quay.io/kata-containers/numa) that reads sysfs and prints results to
stdout — no kubectl exec or CoCo policy overrides needed.
Host-side checks (crictl, pgrep, taskset) run directly on the host
via sudo; a standalone numa-pinning-check.sh script handles the vCPU
thread affinity inspection. The config.d/ helpers used by test 5 are
runtime-agnostic (probe Go vs runtime-rs layout on disk) but the test
is gated to qemu-* shims since runtime-rs does not yet implement
NUMA.
Skips cleanly on single-NUMA hosts, unsupported hypervisors, or when
no nvidia.com/pgpu resources are available (GPU tests only).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
