Switch the per-architecture default shim from the Go runtime ("qemu") to the
Rust runtime ("qemu-runtime-rs") wherever a runtime-rs build exists, and
align CI/test fallbacks with the new default.
Document the defaultShim values and Go runtime deprecation in the Helm and
config migration guides using mkdocs admonitions.
Fixes: kata-containers#9546
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
In confidential mode, the host is not trusted and might try to create
malicious files in the guest VM. This commit changes the implementation
of the CopyFile method to use pathrs, which is designed to prevent
accidental writes to locations outside the shared directory
(/run/kata-containers/shared/containers). It also adds a detailed
description of what the method does, and test cases for intended and
malicious use cases.
This was originally considered as a mitigation candidate for
https://github.com/kata-containers/kata-containers/security/advisories/GHSA-q49m-57vm-c8cc,
but we decided to go with the simpler change to address the immediate
risk. It's still desirable to lock down CopyFile writes for defense in
depth, thus the resubmission outside the GHSA.
Signed-off-by: Markus Rudy <mr@edgeless.systems>
Update confidential_kbs.sh to use the renamed Helm chart knobs
for IBM SEL credentials: `as.ibmse.*` → `as.verifier.se.*`.
Bump coco-trustee version and all associated image tags from
80a3c55e to b6c8dc2b accordingly.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
CI doesn't yet enable bundled Postgres for the Trustee Helm chart, but
Helm still tried to pull the Bitnami subchart from Docker Hub OCI during
dependency build, hitting unauthenticated rate limits on shared runners.
Once we switch to using Postgres in the tests, we will need to drop this
workaround and revisit this issue.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a try-kata-nvidia-cpu.values.yaml example for the CPU-only
NVIDIA runtime classes and make the Kubernetes CI Helm helper use it
for qemu-nvidia-cpu handlers.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
The qemu-nvidia-cpu runtime classes do not provide GPU
passthrough, so do not list them as supported by the NVIDIA
GPU test runner.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
dmverity_mode=auto can preserve existing EROFS layers without
dm-verity metadata, which lets mixed-mode nodes keep running with
partially unprotected image layers.
Use dmverity_mode=on when kata-deploy enables EROFS dm-verity so
missing metadata is reported instead of silently accepted.
Also tighten the Kubernetes probe. Instead of checking for at least one
dm-verity line and one EROFS dm mount in dmesg, derive the expected
workload layer count from the root overlay lowerdirs and require matching
kata-verity devices and dm-backed EROFS mounts.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs to the shared hypervisor
allow-lists and to the docker, nerdctl and free-runner k8s matrices, so the
CPU-only NVIDIA classes get the same smoke coverage as the other qemu
variants on x86_64.
Add run-k8s-tests-on-nvidia-cpu-arm64.yaml and wire it into ci.yaml: it
runs the k8s test suite for qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs
on real arm64 (GH200) hardware, giving the CPU-only classes aarch64
coverage without disturbing the existing GPU arm64 CI.
The NVIDIA NUMA bats and the nv-test allow-list learn about the CPU-only
classes too, so they can be run by hand on a multi-node host, but the arm64
CI deliberately runs only the standard k8s suite: the GH200 it uses exposes
a single CPU-bearing NUMA node, so the multi-NUMA tests would merely skip.
While here, add an is_verity_enabled_runtime_class helper (the CPU-only
classes boot the verity-backed nvidia base image without being
confidential) and an is_nvidia_hypervisor helper, and use them in
k8s-measured-rootfs.bats and the kata-deploy Helm arch wiring.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Roll the missing pieces into the installation guide: hardware
prerequisites (KVM, vmx/svm, vhost modules, nested-virt), an
install-method comparison table, RuntimeClass selection with a test pod,
the pre-built tarball flow with /opt/kata/ paths, and a compact Docker
section. The existing Helm chart instructions and the DaemonSet-vs-Job
deployment-mode note are preserved.
Notes use mkdocs-material admonitions (note/tip/warning) instead of `>`
blockquotes, and cross-repo source links use absolute GitHub URLs so they
resolve on the rendered docs site. Adds `mshv` (Microsoft Hypervisor
device, used in the clh-azure runtimes) to the cspell dictionary.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
When containerd wants to terminate a container, it sends a KillRequest with exec_id="", all=false, signal=SIGTERM.
Kata-shim translates that to a SignalProcessRequest with exec_id="", signal=SIGTERM
Kata agent used to interpret exec_id="" as implying all=true.
That was correct for forceful deletion of containers, as it uses a KillRequest with all=true, signal=SIGKILL.
However, for graceful termination, we want to only signal the root process of the container (all=false).
This changes makes it so that Kata agent interprets exec_id="" && signal=SIGKILL as implying all=true, and uses all=false otherwise.
It also fixes an unrelated bug, where Kata agent would signal the root process twice when all=true.
Fixes#13152 - sending SIGTERM twice to the container's init process on graceful shutdown of a container.
Signed-off-by: Bozhidar Marinov <bozhidar.marinov1@digits.schwarz>
Co-authored-by: Markus Rudy <webmaster@burgerdev.de>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
3 trusted-storage guest-pull tests require a pod that demands
more CPU than is available on the downgraded zVSI profile
(4 vCPU x 8 GB) used for IBM Cloud s390x CI, causing scheduling
to fail with:
```
1 Insufficient cpu. no new claims to deallocate, preemption: 0/1 nodes
are available: 1 No preemption victims found for incoming pod.
```
Skip all three tests when running on s390x with KATA_HYPERVISOR=qemu-coco-dev*
to avoid false failures on this instance profile.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
The latest version is failing to install with the below error, so pin to
the last known good version.
WARNING: No stable version of 'aks-preview' to install. Preview versions allowed.
ERROR: The checksum of the extension does not match the expected value. Use --debug for more information.
Evidence this version fixes the issue locally:
$ az extension list-versions --name aks-preview --output table | tail -2
True False True (upgrade available) aks-preview True 21.0.0b8
True False False aks-preview True 21.0.0b9 (max compatible version)
$ az extension add --name aks-preview --version 21.0.0b9
No stable version of 'aks-preview' to install. Preview versions allowed.
The checksum of the extension does not match the expected value. Use --debug for more information.
$ az extension remove --name aks-preview
The extension aks-preview is not installed. Please install the extension via `az extension add -n aks-preview`.
$ az extension add --name aks-preview --version 21.0.0b8
No stable version of 'aks-preview' to install. Preview versions allowed.
The installed extension 'aks-preview' is in preview.
$
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Replace the kustomize-based Trustee deployment with the upstream Trustee
Helm chart (deployment/helm-chart) for every platform, including TDX and
s390x, which the previous iteration still left on kustomize. The chart
deploys KBS, AS and RVPS as three separate images, so versions.yaml pins
image_kbs/image_as/image_rvps (from the upstream staged-images registry)
alongside the trustee revision, and bumps trustee to a build that carries
the IBM SE material mounting support and the SHA-512 report-data change.
guest-components is bumped in lockstep so IBM SE attestation agrees on the
SHA-512 runtime-data digest.
KBS runs with authorization_mode = "AuthenticatedAuthorization", so admin
requests from kbs-client carry the bearer token minted by the chart's
bootstrap job instead of relying on an anonymous, allow-all admin API. As
reused CI runners may still carry a kbs-client from an older Trustee that
predates the --admin-token-file flag, the client is rebuilt from the pinned
sources unless the installed one already understands it.
On the Intel TDX CI runner only the Attestation Service reaches the public
Intel DCAP collateral endpoint, so it is the sole component that needs the
corporate proxy. We set https_proxy on the AS alone: the KBS <-> AS <-> RVPS
traffic is plain HTTP/gRPC and in-cluster, and routing it through the proxy
breaks those connections. The Intel DCAP and NVIDIA verifiers are configured
by the chart out of the box, so TDX and SNP need no extra verifier knobs.
For IBM SE (s390x) the SE verifier now runs inside the gRPC Attestation
Service, so the SE materials are mounted on the AS Pod (not KBS) through the
chart's as.ibmse.* knobs, which create a node-local PV/PVC. The AS runs as a
non-root user with CAP_DAC_OVERRIDE dropped, hence fsGroup=1000 and a
group-readable hdr.bin so it can read the mounted materials.
The chart's default CPU requests reserve more of the small single-node CI
runners than the old single-Pod deployment did, starving test workloads that
ask for a couple of Guaranteed CPUs. Trustee requests are shrunk (limits kept
generous) so those Pods schedule again.
Debug logging is enabled on all components (log_level=debug) to ease
triaging attestation / resource-retrieval failures in CI, and the release is
rolled out with 'helm upgrade --install' so re-runs on long-lived CI
clusters reconcile in place.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
TestKilledVmmCleanup SIGKILLs the VMM and then removes the pod. After a
hard VMM kill the shim exits promptly (which the test asserts), so the
container's TaskExit event may not have reached containerd before the
shim was gone, leaving CRI's view of the container as "running" and
making a plain `crictl rmp` fail with "container is still running, to
stop first".
Retry the normal removal a few times to give the event time to arrive,
and only fall back to `crictl rmp -f` if it keeps failing -- which is how
a crashed sandbox is recovered in practice (kubelet GC / manual force
removal). The essential guarantee, that no shim process is leaked after
the VMM is killed, is unchanged.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's ensure we have more debug logs to go through in case of failures.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Extend basic-ci-amd64 VMM matrix coverage to include qemu-nvidia-gpu and
qemu-nvidia-gpu-runtime-rs so this branch is validated in CI.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add kata-deploy-scheduling.bats to KATA_DEPLOY_TEST_UNION so the new
Helm scheduling template tests run in CI.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Merge NFD and user nodeSelectorTerms with merge/concat.
Ignore the name key in podLabels, document podAnnotations
and extend bats coverage for annotations.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Expose podLabels, podAnnotations and affinity in values.yaml so
operators can apply more granular scheduling controls on the
kata-deploy DaemonSet alongside the existing nodeSelector and
tolerations knobs.
Defaults are empty, so existing deployments render the same manifest.
When NFD is enabled, the chart merges the hard-coded virtualization
nodeAffinity with any user provided affinity.
Signed-off-by: Zachary Spar <zspar@coreweave.com>
Enabled TestContainerMemoryUpdate support for runtime-rs on s390x and
disabled static_sandbox_resource_mgmt to dynamically update the memory.
Signed-off-by: Annu Sharma <annu-sharma@ibm.com>
With #12373 and #13280 in, emptyDirs are now shared from the host with
virtio-fs when supported, and properly accounted for by Kubelet.
So we can extend the sizeLimit-based pod eviction test introduced by #13127
to virtio-fs-based emptyDirs (originally only for block-based emptyDirs).
Generated-by: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The genpolicy guest-pull check now fails closed when an image carries
supplemental groups that containerd won't reproduce while pulling the
layers inside the guest, and quay.io/prometheus/busybox:latest happens
to ship gid 10 (wheel). pod-number-cpu.yaml was simply missed when the
other busybox manifests got their securityContext, so genpolicy bails
out on the expected/actual gid mismatch.
Let's declare supplementalGroups: [10] explicitly, just like we already
do for the other busybox-based pods, so the generated policy lines up
with what containerd actually applies.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
This commit enhances the VFIO-AP testing suite by adding
support for cold-plug testing in runtime-rs and updating
configuration files to support the new cold_plug_vfio parameter.
Changes include:
1. Configuration updates:
- Added cold_plug_vfio parameter to configuration-qemu-runtime-rs.toml.in
with default value "no-port" (disabled)
- Added cold_plug_vfio parameter to configuration-qemu-se-runtime-rs.toml.in
with value "root-port" (enabled for Secure Execution)
2. Test enhancements:
- Updated setup_hotplug() to explicitly set cold_plug_vfio to "no-port"
for runtime-rs to ensure hotplug-only mode
- Implemented setup_coldplug() for runtime-rs (previously unsupported)
to enable cold-plug testing with "root-port" configuration
- Added new test case (Test 4) for runtime-rs VFIO-AP cold-plug
functionality using zcrypttest validation
This enables comprehensive testing of both hot-plug and cold-plug VFIO
device assignment scenarios for s390x CEX devices in runtime-rs.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add e2fsprogs to the Azure Linux UVM rootfs package list.
This enables using the block-plain emptyDir feature for the clh-azure
scenario.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add Kubernetes integration coverage for block-plain emptyDir storage.
The test enables block-plain emptyDirs, verifies that the guest sees an
ext4 block-backed mount, checks kubelet eviction when the emptyDir
sizeLimit is exceeded, and checks sparse image reclaim through discard.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Add k8s-erofs-dmverity.bats integration test that verifies dm-verity
protected EROFS layers work end-to-end, and register the integrity
mode in the CoCo test matrix.
This commit introduces two new files to enable it:
- k8s-erofs-dmverity.bats
- pod-erofs-dmverity-probe.yaml
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add explicit securityContext values to static Kubernetes policy
workloads whose generated policy now includes image-derived identity
data.
Refresh existing guest-pull comments to point at the Limitations
documentation instead of the old issue link.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Use the new helpers when guest-pull tests generate pods under
automatic policy generation.
Populate the securityContext values that matter for these images:
supplemental groups are set explicitly, while root UID and GID
defaults are left omitted.
This keeps runtime requests aligned with the generated policy
without requiring default-zero identity fields.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Teach the Kubernetes test helpers to inject pod-level
securityContext values into generated manifests.
Pass optional UID, GID, and supplemental group values through the
CoCo pod helpers so guest-pull tests can request explicit identity
data without open-coding yq expressions.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
When static sandbox sizing is enabled, keep configured defaults when
workloads do not specify CPU or memory limits. When limits are present,
size the VM as requested resources plus overhead_vcpus/overhead_memory
values derived from runtime-rs profile defaults.
Limit-driven vCPU sizing is clamped to a minimum of one vCPU so a 0.0
result never yields an unbootable VM, and sandbox setup fails early with
a clear, actionable error when the computed memory is 0 MiB (pointing at
memory limits or non-zero default/overhead memory settings).
This keeps static VM sizing predictable across runtime-rs profiles,
including NVIDIA ones.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Update the composable-vm-images proposal with the design decisions we only
arrived at after experimenting with the implementation:
* Replace the hardcoded agent path-resolution table with the data-driven
components.toml manifest (process levels, args/optional_args, env,
wait_socket, ${...} substitution, and select/variants), keeping the agent
generic.
* Document the attester-variant contract: NVRC exports KATA_ATTESTER_VARIANT
and the manifest selects the stock vs NVIDIA attestation-agent.
* Document the runtime dependency requirements found during bring-up: the
nvidia attester's LD_LIBRARY_PATH (libnvat closure in the coco addon +
NVML in the gpu addon) and the NVML-init failure mode, plus CDH
secure_mount tooling placement -- plain storage (mke2fs/mkfs.ext4/dd) in
the base vs encrypted storage (cryptsetup) in the coco addon, the CDH
PATH, and the base/addon ABI lockstep.
* Reflect the storage tooling and bundled libraries in the base/coco-addon
build sections, and mark the GPU addon as implemented.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
