Add an integration test for the runtime-rs fix that co-locates the shim,
the VMM and virtiofsd in the pod's kubepods cgroup under
sandbox_cgroup_only on cgroup v2. The regression it guards leaves the VMM
and virtiofsd in system.slice/containerd.service while only the shim
reaches kubepods.
Start a BestEffort Kata pod and, from the host, assert the VMM and
virtiofsd share the shim's cgroup and that it is under kubepods. The shim
is selected by the pod UID in its cgroup path, so a pod in another
namespace is not picked. Skips dragonball (in-process VMM).
Generated-By: GitHub Copilot (Claude)
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The termination-log tests exercise termination-message propagation over
the agent GetDiagnosticData RPC, which only matters when the host cannot
read guest files directly, i.e. on shared_fs=none configurations. They
previously gated on is_confidential_runtime_class and therefore skipped
on the NVIDIA CPU runtime-rs handler, which now also runs with
shared_fs=none.
Add an is_shared_fs_none_runtime_class helper (confidential runtime
classes plus qemu-nvidia-cpu-runtime-rs) and use it for the setup and
teardown gating so the tests run on the NVIDIA CPU runtime-rs handler as
well. The plain Go qemu-nvidia-cpu class still uses virtio-fs and is
intentionally excluded. Keep the "blocked by default CoCo policy"
assertion confidential-only, since non-confidential shared_fs=none
classes do not ship a default policy that denies the RPC.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor with Claude Opus 4.8
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Drop the unfinished ARM CCA stack (qemu-cca shim, experimental QEMU/OVMF
and kernel assets, packaging targets, and CI builds) since it is not
tested or shipped in any runtime bundle.
This will be supported by the moment we have development done for real
hardware available.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Store trustee image repositories once under images.* and derive tags
from version, updating the KBS deployment test to match.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Switch the per-architecture default shim from the Go runtime ("qemu") to the
Rust runtime ("qemu-runtime-rs") wherever a runtime-rs build exists, and
align CI/test fallbacks with the new default.
Document the defaultShim values and Go runtime deprecation in the Helm and
config migration guides using mkdocs admonitions.
Fixes: kata-containers#9546
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Update confidential_kbs.sh to use the renamed Helm chart knobs
for IBM SEL credentials: `as.ibmse.*` → `as.verifier.se.*`.
Bump coco-trustee version and all associated image tags from
80a3c55e to b6c8dc2b accordingly.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
CI doesn't yet enable bundled Postgres for the Trustee Helm chart, but
Helm still tried to pull the Bitnami subchart from Docker Hub OCI during
dependency build, hitting unauthenticated rate limits on shared runners.
Once we switch to using Postgres in the tests, we will need to drop this
workaround and revisit this issue.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The qemu-nvidia-cpu runtime classes do not provide GPU
passthrough, so do not list them as supported by the NVIDIA
GPU test runner.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
dmverity_mode=auto can preserve existing EROFS layers without
dm-verity metadata, which lets mixed-mode nodes keep running with
partially unprotected image layers.
Use dmverity_mode=on when kata-deploy enables EROFS dm-verity so
missing metadata is reported instead of silently accepted.
Also tighten the Kubernetes probe. Instead of checking for at least one
dm-verity line and one EROFS dm mount in dmesg, derive the expected
workload layer count from the root overlay lowerdirs and require matching
kata-verity devices and dm-backed EROFS mounts.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs to the shared hypervisor
allow-lists and to the docker, nerdctl and free-runner k8s matrices, so the
CPU-only NVIDIA classes get the same smoke coverage as the other qemu
variants on x86_64.
Add run-k8s-tests-on-nvidia-cpu-arm64.yaml and wire it into ci.yaml: it
runs the k8s test suite for qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs
on real arm64 (GH200) hardware, giving the CPU-only classes aarch64
coverage without disturbing the existing GPU arm64 CI.
The NVIDIA NUMA bats and the nv-test allow-list learn about the CPU-only
classes too, so they can be run by hand on a multi-node host, but the arm64
CI deliberately runs only the standard k8s suite: the GH200 it uses exposes
a single CPU-bearing NUMA node, so the multi-NUMA tests would merely skip.
While here, add an is_verity_enabled_runtime_class helper (the CPU-only
classes boot the verity-backed nvidia base image without being
confidential) and an is_nvidia_hypervisor helper, and use them in
k8s-measured-rootfs.bats and the kata-deploy Helm arch wiring.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
When containerd wants to terminate a container, it sends a KillRequest with exec_id="", all=false, signal=SIGTERM.
Kata-shim translates that to a SignalProcessRequest with exec_id="", signal=SIGTERM
Kata agent used to interpret exec_id="" as implying all=true.
That was correct for forceful deletion of containers, as it uses a KillRequest with all=true, signal=SIGKILL.
However, for graceful termination, we want to only signal the root process of the container (all=false).
This changes makes it so that Kata agent interprets exec_id="" && signal=SIGKILL as implying all=true, and uses all=false otherwise.
It also fixes an unrelated bug, where Kata agent would signal the root process twice when all=true.
Fixes#13152 - sending SIGTERM twice to the container's init process on graceful shutdown of a container.
Signed-off-by: Bozhidar Marinov <bozhidar.marinov1@digits.schwarz>
Co-authored-by: Markus Rudy <webmaster@burgerdev.de>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
3 trusted-storage guest-pull tests require a pod that demands
more CPU than is available on the downgraded zVSI profile
(4 vCPU x 8 GB) used for IBM Cloud s390x CI, causing scheduling
to fail with:
```
1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes
are available: 1 No preemption victims found for incoming pod.
```
Skip all three tests when running on s390x with KATA_HYPERVISOR=qemu-coco-dev*
to avoid false failures on this instance profile.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Replace the kustomize-based Trustee deployment with the upstream Trustee
Helm chart (deployment/helm-chart) for every platform, including TDX and
s390x, which the previous iteration still left on kustomize. The chart
deploys KBS, AS and RVPS as three separate images, so versions.yaml pins
image_kbs/image_as/image_rvps (from the upstream staged-images registry)
alongside the trustee revision, and bumps trustee to a build that carries
the IBM SE material mounting support and the SHA-512 report-data change.
guest-components is bumped in lockstep so IBM SE attestation agrees on the
SHA-512 runtime-data digest.
KBS runs with authorization_mode = "AuthenticatedAuthorization", so admin
requests from kbs-client carry the bearer token minted by the chart's
bootstrap job instead of relying on an anonymous, allow-all admin API. As
reused CI runners may still carry a kbs-client from an older Trustee that
predates the --admin-token-file flag, the client is rebuilt from the pinned
sources unless the installed one already understands it.
On the Intel TDX CI runner only the Attestation Service reaches the public
Intel DCAP collateral endpoint, so it is the sole component that needs the
corporate proxy. We set https_proxy on the AS alone: the KBS <-> AS <-> RVPS
traffic is plain HTTP/gRPC and in-cluster, and routing it through the proxy
breaks those connections. The Intel DCAP and NVIDIA verifiers are configured
by the chart out of the box, so TDX and SNP need no extra verifier knobs.
For IBM SE (s390x) the SE verifier now runs inside the gRPC Attestation
Service, so the SE materials are mounted on the AS Pod (not KBS) through the
chart's as.ibmse.* knobs, which create a node-local PV/PVC. The AS runs as a
non-root user with CAP_DAC_OVERRIDE dropped, hence fsGroup=1000 and a
group-readable hdr.bin so it can read the mounted materials.
The chart's default CPU requests reserve more of the small single-node CI
runners than the old single-Pod deployment did, starving test workloads that
ask for a couple of Guaranteed CPUs. Trustee requests are shrunk (limits kept
generous) so those Pods schedule again.
Debug logging is enabled on all components (log_level=debug) to ease
triaging attestation / resource-retrieval failures in CI, and the release is
rolled out with 'helm upgrade --install' so re-runs on long-lived CI
clusters reconcile in place.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
TestKilledVmmCleanup SIGKILLs the VMM and then removes the pod. After a
hard VMM kill the shim exits promptly (which the test asserts), so the
container's TaskExit event may not have reached containerd before the
shim was gone, leaving CRI's view of the container as "running" and
making a plain `crictl rmp` fail with "container is still running, to
stop first".
Retry the normal removal a few times to give the event time to arrive,
and only fall back to `crictl rmp -f` if it keeps failing -- which is how
a crashed sandbox is recovered in practice (kubelet GC / manual force
removal). The essential guarantee, that no shim process is leaked after
the VMM is killed, is unchanged.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's ensure we have more debug logs to go through in case of failures.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Extend basic-ci-amd64 VMM matrix coverage to include qemu-nvidia-gpu and
qemu-nvidia-gpu-runtime-rs so this branch is validated in CI.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Enabled TestContainerMemoryUpdate support for runtime-rs on s390x and
disabled static_sandbox_resource_mgmt to dynamically update the memory.
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
With #12373 and #13280 in, emptyDirs are now shared from the host with
virtio-fs when supported, and properly accounted for by Kubelet.
So we can extend the sizeLimit-based pod eviction test introduced by #13127
to virtio-fs-based emptyDirs (originally only for block-based emptyDirs).
Generated-by: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The genpolicy guest-pull check now fails closed when an image carries
supplemental groups that containerd won't reproduce while pulling the
layers inside the guest, and quay.io/prometheus/busybox:latest happens
to ship gid 10 (wheel). pod-number-cpu.yaml was simply missed when the
other busybox manifests got their securityContext, so genpolicy bails
out on the expected/actual gid mismatch.
Let's declare supplementalGroups: [10] explicitly, just like we already
do for the other busybox-based pods, so the generated policy lines up
with what containerd actually applies.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add e2fsprogs to the Azure Linux UVM rootfs package list.
This enables using the block-plain emptyDir feature for the clh-azure
scenario.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add Kubernetes integration coverage for block-plain emptyDir storage.
The test enables block-plain emptyDirs, verifies that the guest sees an
ext4 block-backed mount, checks kubelet eviction when the emptyDir
sizeLimit is exceeded, and checks sparse image reclaim through discard.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add k8s-erofs-dmverity.bats integration test that verifies dm-verity
protected EROFS layers work end-to-end, and register the integrity
mode in the CoCo test matrix.
This commit introduces two new files to enable it:
- k8s-erofs-dmverity.bats
- pod-erofs-dmverity-probe.yaml
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add explicit securityContext values to static Kubernetes policy
workloads whose generated policy now includes image-derived identity
data.
Refresh existing guest-pull comments to point at the Limitations
documentation instead of the old issue link.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Use the new helpers when guest-pull tests generate pods under
automatic policy generation.
Populate the securityContext values that matter for these images:
supplemental groups are set explicitly, while root UID and GID
defaults are left omitted.
This keeps runtime requests aligned with the generated policy
without requiring default-zero identity fields.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Teach the Kubernetes test helpers to inject pod-level
securityContext values into generated manifests.
Pass optional UID, GID, and supplemental group values through the
CoCo pod helpers so guest-pull tests can request explicit identity
data without open-coding yq expressions.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
When static sandbox sizing is enabled, keep configured defaults when
workloads do not specify CPU or memory limits. When limits are present,
size the VM as requested resources plus overhead_vcpus/overhead_memory
values derived from runtime-rs profile defaults.
Limit-driven vCPU sizing is clamped to a minimum of one vCPU so a 0.0
result never yields an unbootable VM, and sandbox setup fails early with
a clear, actionable error when the computed memory is 0 MiB (pointing at
memory limits or non-zero default/overhead memory settings).
This keeps static VM sizing predictable across runtime-rs profiles,
including NVIDIA ones.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add k8s-vm-templating-test.bats which exercises pod create
with the factory initialized on the target node.
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
Kata sizes VM CPU and memory from OCI limits, not Kubernetes resource
requests. Requests are consumed by the Kubernetes control plane, but
they do not drive Kata VM or sandbox sizing today.
Convert the straightforward Kata workload manifests and kata-deploy
examples from resource requests to limits so the declared resources
match the values Kata uses for VM provisioning. Keep requests where the
fixture intentionally validates Kubernetes request/limit behavior.
Update fixture expectations affected by the conversion. The LimitRange
fixture is limit-only at 500m.
Raise the policy deployment limits to 500m and 800Mi. These tests boot
CoCo/runtime-rs sandboxes with policy/initdata, and the former
100m/100Mi values became real runtime limits after the conversion,
which is too constrained for the CI environments.
Leave PVC storage requests, explicit request/limit validation fixtures,
the env resourceFieldRef request, and non-Kata workload examples
unchanged where requests are handled outside the Kata shim resource
sizing path.
If Kata later grows request-aware sandbox sizing, for example through
Sandbox API based resource plumbing, these requests can be reintroduced
where they carry the intended semantics.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The Guaranteed QoS test is currently failing for SNP and TDX runtime-rs
due to a podOverhead configuration issue. The test requests 600Mi of
memory which, combined with the 2048Mi podOverhead, exceeds 2GiB and
triggers memory management issues in confidential guests.
This is a temporary skip until the podOverhead fix is merged.
Related: https://github.com/kata-containers/kata-containers/pull/13228
Signed-off-by: Harshitha Gowda <hgowda@amd.com>
Add common Kubernetes test helpers for locating the active per-shim
Kata runtime config directory and copying/removing TOML fragments
under config.d.
Update the NVIDIA NUMA test to install its temporary numa_mapping
override through those helpers. This gives follow-up tests a shared
pattern for temporary runtime config overrides.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Increase memory request/limit values used by k8s memory and QoS
integration workloads so SNP/TDX static-sized sandboxes boot reliably
under the new sizing defaults.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
This commit is to enable qemu-runtime-rs/clh-runtime-rs and make it
compatiable with qemu-runtime-rs and clh-runtime-rs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
