Add an integration test for the runtime-rs fix that co-locates the shim,
the VMM and virtiofsd in the pod's kubepods cgroup under
sandbox_cgroup_only on cgroup v2. The regression it guards leaves the VMM
and virtiofsd in system.slice/containerd.service while only the shim
reaches kubepods.
Start a BestEffort Kata pod and, from the host, assert the VMM and
virtiofsd share the shim's cgroup and that it is under kubepods. The shim
is selected by the pod UID in its cgroup path, so a pod in another
namespace is not picked. Skips dragonball (in-process VMM).
Generated-By: GitHub Copilot (Claude)
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Move PCI enumeration, VFIO helpers, and InfiniBand diagnostics out of
kata-sys-util into the external pcilibs-rs crate so they can be reused
independently of kata-containers.
Depend on pcilibs-rs 0.1.0 from crates.io.
Add pcilibs to the cspell dictionary for the new crate reference in
src/libs/README.md.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The termination-log tests exercise termination-message propagation over
the agent GetDiagnosticData RPC, which only matters when the host cannot
read guest files directly, i.e. on shared_fs=none configurations. They
previously gated on is_confidential_runtime_class and therefore skipped
on the NVIDIA CPU runtime-rs handler, which now also runs with
shared_fs=none.
Add an is_shared_fs_none_runtime_class helper (confidential runtime
classes plus qemu-nvidia-cpu-runtime-rs) and use it for the setup and
teardown gating so the tests run on the NVIDIA CPU runtime-rs handler as
well. The plain Go qemu-nvidia-cpu class still uses virtio-fs and is
intentionally excluded. Keep the "blocked by default CoCo policy"
assertion confidential-only, since non-confidential shared_fs=none
classes do not ship a default policy that denies the RPC.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
EROFS fs-verity depends on the backing filesystem supporting fs-verity,
which cannot be guaranteed on an arbitrary node. Rather than gate the
NVIDIA CPU profile on a host feature we do not control, disable fs-verity
so it works on the widest possible range of setups; we take that hit
deliberately to keep the profile generic.
Move this out of the test harness and into the kata-deploy NVIDIA CPU
profile: try-kata-nvidia-cpu.values.yaml now ships a containerd erofs
snapshotter drop-in that disables fs-verity and pins the memory-backed
default_size, and selects memory-backed rw layers with dm-verity for the
lower (image) layers. Because containerd.userDropIn is loaded after
kata-deploy's generated config, it overrides the built-in enable_fsverity
default.
The CI helm helper no longer special-cases NVIDIA CPU to inject
enable_fsverity=false. Instead it honors an erofs snapshotter drop-in
already provided by the base values file and only synthesizes the default
default_size drop-in when the profile does not provide its own.
Note that this drop-in targets containerd's built-in EROFS snapshotter,
available only on containerd >= 2.2.0 (config version 3, conf.d auto-import),
which is the same minimum kata-deploy already enforces for the EROFS
snapshotter.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Set the non-confidential NVIDIA runtime-rs emptyDir mode to
block-plain through a dedicated Makefile default.
This prepares the runtime class for configurations where filesystem
sharing is disabled and emptyDir volumes need to be backed by
guest-mounted block devices instead of shared-fs.
Also add the shared configure_nvidia_runtime_rs_shared_fs_dropin()
test helper, which keeps the NVIDIA runtime-rs Docker/nerdctl smoke
tests on virtio-fs while the shared_fs=none + EROFS snapshotter path
is exercised by Kubernetes CI.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Drop the unfinished ARM CCA stack (qemu-cca shim, experimental QEMU/OVMF
and kernel assets, packaging targets, and CI builds) since it is not
tested or shipped in any runtime bundle.
This will be supported by the moment we have development done for real
hardware available.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Store trustee image repositories once under images.* and derive tags
from version, updating the KBS deployment test to match.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Remove versions.yaml pins that no script reads (cryptsetup, lvm2,
kubernetes, kustomize, xurls) along with containerd.tarball_url, and
delete the unused install_kustomize() helper.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Switch the per-architecture default shim from the Go runtime ("qemu") to the
Rust runtime ("qemu-runtime-rs") wherever a runtime-rs build exists, and
align CI/test fallbacks with the new default.
Document the defaultShim values and Go runtime deprecation in the Helm and
config migration guides using mkdocs admonitions.
Fixes: kata-containers#9546
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
In confidential mode, the host is not trusted and might try to create
malicious files in the guest VM. This commit changes the implementation
of the CopyFile method to use pathrs, which is designed to prevent
accidental writes to locations outside the shared directory
(/run/kata-containers/shared/containers). It also adds a detailed
description of what the method does, and test cases for intended and
malicious use cases.
This was originally considered as a mitigation candidate for
https://github.com/kata-containers/kata-containers/security/advisories/GHSA-q49m-57vm-c8cc,
but we decided to go with the simpler change to address the immediate
risk. It's still desirable to lock down CopyFile writes for defense in
depth, thus the resubmission outside the GHSA.
Signed-off-by: Markus Rudy <mr@edgeless.systems>
Update confidential_kbs.sh to use the renamed Helm chart knobs
for IBM SEL credentials: `as.ibmse.*` → `as.verifier.se.*`.
Bump coco-trustee version and all associated image tags from
80a3c55e to b6c8dc2b accordingly.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
CI doesn't yet enable bundled Postgres for the Trustee Helm chart, but
Helm still tried to pull the Bitnami subchart from Docker Hub OCI during
dependency build, hitting unauthenticated rate limits on shared runners.
Once we switch to using Postgres in the tests, we will need to drop this
workaround and revisit this issue.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a try-kata-nvidia-cpu.values.yaml example for the CPU-only
NVIDIA runtime classes and make the Kubernetes CI Helm helper use it
for qemu-nvidia-cpu handlers.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The qemu-nvidia-cpu runtime classes do not provide GPU
passthrough, so do not list them as supported by the NVIDIA
GPU test runner.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
dmverity_mode=auto can preserve existing EROFS layers without
dm-verity metadata, which lets mixed-mode nodes keep running with
partially unprotected image layers.
Use dmverity_mode=on when kata-deploy enables EROFS dm-verity so
missing metadata is reported instead of silently accepted.
Also tighten the Kubernetes probe. Instead of checking for at least one
dm-verity line and one EROFS dm mount in dmesg, derive the expected
workload layer count from the root overlay lowerdirs and require matching
kata-verity devices and dm-backed EROFS mounts.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs to the shared hypervisor
allow-lists and to the docker, nerdctl and free-runner k8s matrices, so the
CPU-only NVIDIA classes get the same smoke coverage as the other qemu
variants on x86_64.
Add run-k8s-tests-on-nvidia-cpu-arm64.yaml and wire it into ci.yaml: it
runs the k8s test suite for qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs
on real arm64 (GH200) hardware, giving the CPU-only classes aarch64
coverage without disturbing the existing GPU arm64 CI.
The NVIDIA NUMA bats and the nv-test allow-list learn about the CPU-only
classes too, so they can be run by hand on a multi-node host, but the arm64
CI deliberately runs only the standard k8s suite: the GH200 it uses exposes
a single CPU-bearing NUMA node, so the multi-NUMA tests would merely skip.
While here, add an is_verity_enabled_runtime_class helper (the CPU-only
classes boot the verity-backed nvidia base image without being
confidential) and an is_nvidia_hypervisor helper, and use them in
k8s-measured-rootfs.bats and the kata-deploy Helm arch wiring.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Roll the missing pieces into the installation guide: hardware
prerequisites (KVM, vmx/svm, vhost modules, nested-virt), an
install-method comparison table, RuntimeClass selection with a test pod,
the pre-built tarball flow with /opt/kata/ paths, and a compact Docker
section. The existing Helm chart instructions and the DaemonSet-vs-Job
deployment-mode note are preserved.
Notes use mkdocs-material admonitions (note/tip/warning) instead of `>`
blockquotes, and cross-repo source links use absolute GitHub URLs so they
resolve on the rendered docs site. Adds `mshv` (Microsoft Hypervisor
device, used in the clh-azure runtimes) to the cspell dictionary.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
When containerd wants to terminate a container, it sends a KillRequest with exec_id="", all=false, signal=SIGTERM.
Kata-shim translates that to a SignalProcessRequest with exec_id="", signal=SIGTERM
Kata agent used to interpret exec_id="" as implying all=true.
That was correct for forceful deletion of containers, as it uses a KillRequest with all=true, signal=SIGKILL.
However, for graceful termination, we want to only signal the root process of the container (all=false).
This changes makes it so that Kata agent interprets exec_id="" && signal=SIGKILL as implying all=true, and uses all=false otherwise.
It also fixes an unrelated bug, where Kata agent would signal the root process twice when all=true.
Fixes#13152 - sending SIGTERM twice to the container's init process on graceful shutdown of a container.
Signed-off-by: Bozhidar Marinov <bozhidar.marinov1@digits.schwarz>
Co-authored-by: Markus Rudy <webmaster@burgerdev.de>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
3 trusted-storage guest-pull tests require a pod that demands
more CPU than is available on the downgraded zVSI profile
(4 vCPU x 8 GB) used for IBM Cloud s390x CI, causing scheduling
to fail with:
```
1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes
are available: 1 No preemption victims found for incoming pod.
```
Skip all three tests when running on s390x with KATA_HYPERVISOR=qemu-coco-dev*
to avoid false failures on this instance profile.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
The latest version is failing to install with the below error, so pin to
the last known good version.
WARNING: No stable version of 'aks-preview' to install. Preview versions allowed.
ERROR: The checksum of the extension does not match the expected value. Use --debug for more information.
Evidence this version fixes the issue locally:
$ az extension list-versions --name aks-preview --output table | tail -2
True False True (upgrade available) aks-preview True 21.0.0b8
True False False aks-preview True 21.0.0b9 (max compatible version)
$ az extension add --name aks-preview --version 21.0.0b9
No stable version of 'aks-preview' to install. Preview versions allowed.
The checksum of the extension does not match the expected value. Use --debug for more information.
$ az extension remove --name aks-preview
The extension aks-preview is not installed. Please install the extension via `az extension add -n aks-preview`.
$ az extension add --name aks-preview --version 21.0.0b8
No stable version of 'aks-preview' to install. Preview versions allowed.
The installed extension 'aks-preview' is in preview.
$
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Replace the kustomize-based Trustee deployment with the upstream Trustee
Helm chart (deployment/helm-chart) for every platform, including TDX and
s390x, which the previous iteration still left on kustomize. The chart
deploys KBS, AS and RVPS as three separate images, so versions.yaml pins
image_kbs/image_as/image_rvps (from the upstream staged-images registry)
alongside the trustee revision, and bumps trustee to a build that carries
the IBM SE material mounting support and the SHA-512 report-data change.
guest-components is bumped in lockstep so IBM SE attestation agrees on the
SHA-512 runtime-data digest.
KBS runs with authorization_mode = "AuthenticatedAuthorization", so admin
requests from kbs-client carry the bearer token minted by the chart's
bootstrap job instead of relying on an anonymous, allow-all admin API. As
reused CI runners may still carry a kbs-client from an older Trustee that
predates the --admin-token-file flag, the client is rebuilt from the pinned
sources unless the installed one already understands it.
On the Intel TDX CI runner only the Attestation Service reaches the public
Intel DCAP collateral endpoint, so it is the sole component that needs the
corporate proxy. We set https_proxy on the AS alone: the KBS <-> AS <-> RVPS
traffic is plain HTTP/gRPC and in-cluster, and routing it through the proxy
breaks those connections. The Intel DCAP and NVIDIA verifiers are configured
by the chart out of the box, so TDX and SNP need no extra verifier knobs.
For IBM SE (s390x) the SE verifier now runs inside the gRPC Attestation
Service, so the SE materials are mounted on the AS Pod (not KBS) through the
chart's as.ibmse.* knobs, which create a node-local PV/PVC. The AS runs as a
non-root user with CAP_DAC_OVERRIDE dropped, hence fsGroup=1000 and a
group-readable hdr.bin so it can read the mounted materials.
The chart's default CPU requests reserve more of the small single-node CI
runners than the old single-Pod deployment did, starving test workloads that
ask for a couple of Guaranteed CPUs. Trustee requests are shrunk (limits kept
generous) so those Pods schedule again.
Debug logging is enabled on all components (log_level=debug) to ease
triaging attestation / resource-retrieval failures in CI, and the release is
rolled out with 'helm upgrade --install' so re-runs on long-lived CI
clusters reconcile in place.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
TestKilledVmmCleanup SIGKILLs the VMM and then removes the pod. After a
hard VMM kill the shim exits promptly (which the test asserts), so the
container's TaskExit event may not have reached containerd before the
shim was gone, leaving CRI's view of the container as "running" and
making a plain `crictl rmp` fail with "container is still running, to
stop first".
Retry the normal removal a few times to give the event time to arrive,
and only fall back to `crictl rmp -f` if it keeps failing -- which is how
a crashed sandbox is recovered in practice (kubelet GC / manual force
removal). The essential guarantee, that no shim process is leaked after
the VMM is killed, is unchanged.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's ensure we have more debug logs to go through in case of failures.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Extend basic-ci-amd64 VMM matrix coverage to include qemu-nvidia-gpu and
qemu-nvidia-gpu-runtime-rs so this branch is validated in CI.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add kata-deploy-scheduling.bats to KATA_DEPLOY_TEST_UNION so the new
Helm scheduling template tests run in CI.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Merge NFD and user nodeSelectorTerms with merge/concat.
Ignore the name key in podLabels, document podAnnotations
and extend bats coverage for annotations.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Expose podLabels, podAnnotations and affinity in values.yaml so
operators can apply more granular scheduling controls on the
kata-deploy DaemonSet alongside the existing nodeSelector and
tolerations knobs.
Defaults are empty, so existing deployments render the same manifest.
When NFD is enabled, the chart merges the hard-coded virtualization
nodeAffinity with any user provided affinity.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Enabled TestContainerMemoryUpdate support for runtime-rs on s390x and
disabled static_sandbox_resource_mgmt to dynamically update the memory.
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
With #12373 and #13280 in, emptyDirs are now shared from the host with
virtio-fs when supported, and properly accounted for by Kubelet.
So we can extend the sizeLimit-based pod eviction test introduced by #13127
to virtio-fs-based emptyDirs (originally only for block-based emptyDirs).
Generated-by: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The genpolicy guest-pull check now fails closed when an image carries
supplemental groups that containerd won't reproduce while pulling the
layers inside the guest, and quay.io/prometheus/busybox:latest happens
to ship gid 10 (wheel). pod-number-cpu.yaml was simply missed when the
other busybox manifests got their securityContext, so genpolicy bails
out on the expected/actual gid mismatch.
Let's declare supplementalGroups: [10] explicitly, just like we already
do for the other busybox-based pods, so the generated policy lines up
with what containerd actually applies.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
This commit enhances the VFIO-AP testing suite by adding
support for cold-plug testing in runtime-rs and updating
configuration files to support the new cold_plug_vfio parameter.
Changes include:
1. Configuration updates:
- Added cold_plug_vfio parameter to configuration-qemu-runtime-rs.toml.in
with default value "no-port" (disabled)
- Added cold_plug_vfio parameter to configuration-qemu-se-runtime-rs.toml.in
with value "root-port" (enabled for Secure Execution)
2. Test enhancements:
- Updated setup_hotplug() to explicitly set cold_plug_vfio to "no-port"
for runtime-rs to ensure hotplug-only mode
- Implemented setup_coldplug() for runtime-rs (previously unsupported)
to enable cold-plug testing with "root-port" configuration
- Added new test case (Test 4) for runtime-rs VFIO-AP cold-plug
functionality using zcrypttest validation
This enables comprehensive testing of both hot-plug and cold-plug VFIO
device assignment scenarios for s390x CEX devices in runtime-rs.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
