Disable filesystem sharing for the non-confidential NVIDIA runtime-rs
handler and use the EROFS snapshotter as its Kubernetes image-layer
transport. This moves the runtime class toward a guest-owned storage
model instead of relying on virtio-fs for container image layers and
writable cache volumes.
Configure kata-deploy's NVIDIA GPU values to install EROFS and select
it for the qemu-nvidia-gpu-runtime-rs Kubernetes handler. Use
memory-backed writable layers and dm-verity for lower-layer integrity.
Adjust the NVIDIA GPU Kubernetes CI matrix so the non-confidential
runtime-rs job exercises EROFS, while the Go and TEE jobs keep their
existing snapshotter choices.
Keep the Docker smoke test path on virtio-fs. The EROFS setup is
targeted at the Kubernetes runtime-rs handler, where containerd can use
the EROFS snapshotter for image layers.
Adjust the runtime-rs NIM test selection and manifests closer to the
TEE case, where filesystem sharing is already disabled and cache
storage is guest-owned and ephemeral.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Enable the EROFS snapshotter for the non-confidential NVIDIA CPU-only
runtime-rs handler and move it to a guest-owned storage model.
The CPU runtime-rs QEMU configuration now disables filesystem sharing
and uses block-plain emptyDir. The NVIDIA CPU Kubernetes workflows
(free-runner and arm64) select the EROFS snapshotter with memory-backed
writable layers and dm-verity only for qemu-nvidia-cpu-runtime-rs, while
the Go CPU runtime keeps the default snapshotter path.
nerdctl smoke tests are kept on virtio-fs through the shared
configure_nvidia_runtime_rs_shared_fs_dropin() opt-out, since they do
not exercise the Kubernetes EROFS snapshotter mapping. NVIDIA CI hosts
lack ext4 fs-verity support, so the erofs containerd drop-in disables
fsverity for the CPU handler.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Drop the unfinished ARM CCA stack (qemu-cca shim, experimental QEMU/OVMF
and kernel assets, packaging targets, and CI builds) since it is not
tested or shipped in any runtime bundle.
This will be supported by the moment we have development done for real
hardware available.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs to the shared hypervisor
allow-lists and to the docker, nerdctl and free-runner k8s matrices, so the
CPU-only NVIDIA classes get the same smoke coverage as the other qemu
variants on x86_64.
Add run-k8s-tests-on-nvidia-cpu-arm64.yaml and wire it into ci.yaml: it
runs the k8s test suite for qemu-nvidia-cpu and qemu-nvidia-cpu-runtime-rs
on real arm64 (GH200) hardware, giving the CPU-only classes aarch64
coverage without disturbing the existing GPU arm64 CI.
The NVIDIA NUMA bats and the nv-test allow-list learn about the CPU-only
classes too, so they can be run by hand on a multi-node host, but the arm64
CI deliberately runs only the standard k8s suite: the GH200 it uses exposes
a single CPU-bearing NUMA node, so the multi-NUMA tests would merely skip.
While here, add an is_verity_enabled_runtime_class helper (the CPU-only
classes boot the verity-backed nvidia base image without being
confidential) and an is_nvidia_hypervisor helper, and use them in
k8s-measured-rootfs.bats and the kata-deploy Helm arch wiring.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The `kbs-client` installation step was timing out at 10 minutes on
the downgraded zVSI profile (4 vCPU x 8 GB). Double the timeout to
20 minutes to accommodate the slower build/download speed on this
instance profile.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add the rootfs-image-nvidia-{base,gpu-extension} build targets to the
local-build Makefile and the nvgpu bundle.
Build the base/extension rootfs images in the amd64/arm64 static tarball
workflows, and point the runtime-rs NVIDIA shim components at the
composable base + gpu extension (+ coco extension for the confidential
classes). The Go shim components keep the monolithic nvidia-gpu /
nvidia-gpu-confidential images.
Ship cryptsetup in the CoCo guest-components extension (and bundle
libnvat's non-glibc dependency closure next to the nvidia attester): the
chiseled nvidia base image carries only veritysetup, so the coco-extension
manifest's usr/sbin PATH entry needs a cryptsetup to resolve for NVIDIA
confidential encrypted-storage.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Extend basic-ci-amd64 VMM matrix coverage to include qemu-nvidia-gpu and
qemu-nvidia-gpu-runtime-rs so this branch is validated in CI.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
In 7afdfc7388 we
just disabled the undocumented-permissions zizmor warning,
but now with AI we can roll this fix more easily, so remove
the disabling of the rule and add the comments
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
As the machine is off and I cannot connect to the BMC to turn
it back on, let's avoid PRs running those tests for now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add rootfs-image-coco-extension to the build-asset-rootfs matrix in the
amd64, arm64 and s390x static tarball workflows so the CoCo extension image
is built as part of CI. Keep building rootfs-image-confidential as well:
the runtime defaults to base + extension, but we retain the monolithic
confidential image for now during the transition.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add a check-agent-policy-coverage job to static-checks.yaml
that runs ci/check_agent_policy_coverage.sh on every pull request.
Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Enable devicemapper (dm-verity) support in CI by forwarding the
USE_DEVMAPPER environment variable to the agent build step across
all architectures (amd64, arm64, s390x).
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add k8s-erofs-dmverity.bats integration test that verifies dm-verity
protected EROFS layers work end-to-end, and register the integrity
mode in the CoCo test matrix.
This commit introduces two new files to enable it:
- k8s-erofs-dmverity.bats
- pod-erofs-dmverity-probe.yaml
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce a new `dmverity` module in kata-types that provides dm-verity
device creation, destruction and lifecycle management via devicemapper
ioctls. The module is conditionally compiled behind the `devicemapper`
feature flag, which also pulls in tokio for async device-node polling.
The workspace devicemapper dependency is pinned to a specific git
revision for reproducible builds.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The kata-monitor image has no job-dispatcher sidecar, so opt out of the
kata-deploy-specific dispatcher manifest derivation in the
payload-after-push workflow by setting
KATA_DEPLOY_PUBLISH_JOB_DISPATCHER=false, mirroring the same fix already
applied to the release workflows.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add k8s-vm-templating-test.bats which exercises pod create
with the factory initialized on the target node.
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
Let's use the ActionsPZ runners for the following jobs:
- publish-kata-deploy-image-s390x
- publish-kata-monitor-image-s390x
to improve CI experiences.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
The shared _publish_multiarch_manifest() helper always derived a
"-job-dispatcher" registry from the registries it was given. However, the
dispatcher is a kata-deploy-specific sidecar image, so when the helper
was reused to publish the kata-monitor multi-arch manifest it wrongly
tried to push a non-existent kata-monitor-job-dispatcher image.
Let's gate the dispatcher derivation behind
KATA_DEPLOY_PUBLISH_JOB_DISPATCHER (defaulting to true so the
kata-deploy path is unchanged) and opt out of it when publishing the
kata-monitor manifest.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Package and ship the dispatcher built in the previous commit so the
job-mode Helm chart has an image to run.
- Dockerfile.components: build kata-deploy and kata-deploy-job-dispatcher
from the same rust-builder stage (one compile), and run fmt/clippy/
test for both crates.
- job-dispatcher/Dockerfile: a minimal distroless/static image containing
only the dispatcher binary and CA certs - it is an API client, so it
needs nothing from the host.
- local-build: kata-deploy-job-dispatcher becomes its own build component
with its own static tarball
(kata-deploy-static-kata-deploy-job-dispatcher.tar.zst); the shared
rust-builder output is reused so the two components do not recompile
the workspace locally. The payload script builds and pushes a separate
"<kata-deploy registry>-job-dispatcher" image with the same tag scheme,
and release.sh publishes its multi-arch manifest symmetrically.
- CI: add kata-deploy-job-dispatcher to the build-kata-deploy-components
matrices (its tarball is picked up by the existing kata-artifacts-*
glob), and gate it in the kata-deploy rust static checks.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Run qemu-coco-dev-runtime-rs k8s test workflow on the zVSI
only during nightly builds.
Changes:
- Modified run-k8s-tests-on-zvsi.yaml to accept vmm as workflow
inputs instead of hardcoded matrix values
- run-k8s-tests-on-zvsi passes a conditional vmm value; 4 vmms
for nightly/dev builds and 3 vmms for all other PRs.
This ensures qemu-coco-dev-runtime-rs is only tested with nydus
snapshotter during nightly CI runs, reducing PR test time while
maintaining comprehensive nightly coverage.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add qemu-coco-dev-runtime-rs to the VMM matrix in the zVSI K8S
test workflow, configured to run only with the nydus snapshotter.
Changes:
- Add qemu-coco-dev-runtime-rs to the vmm matrix
- Exclude overlayfs + qemu-coco-dev-runtime-rs combination
- Exclude devmapper + qemu-coco-dev-runtime-rs combination
- Update CoCo-related conditional steps to include the new VMM:
* KBS environment variable setup
* kbs-client uninstall/install steps
* CoCo KBS deployment
This ensures qemu-coco-dev-runtime-rs is only tested with nydus
snapshotter, while maintaining existing test configurations for
other VMMs.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Switch kata-monitor workflows from the deprecated "active" key to
"latest" so CI resolves containerd versions from versions.yaml correctly
after the key rename.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
When I implemented the OSC scanner I followed the
guidance on the the action repo to use a single workflow for
both PR and main tests and rely on a re-usable workflow.
Since then I've realised some negatives of this approach:
- Unlike actions, dependabot needs custom logic to bump
workflow pins, so we are more likely to be out of date
- A lack of transparency/notification of when updates
are needed, due to bugs/ security fixes
- The dual workflow results in skipped jobs that
clutter the UI
- No ability to customise the pre-steps, or config
As such let's take the hit of managing two workflows,
in order to give us better flexibility.
Also add the `--call-analysis=none` option as we run govulncheck
separately, so don't want to have to compile and have a slow build
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Generated-By: IBM Bob
These jobs build and push the kata-deploy OCI image, so call them
publish-kata-deploy-image-* instead of *-payload-*, matching the
kata-monitor image jobs and making the workflow easier to read.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a single-job k8s test that installs the kata-deploy helm chart
with monitor.enabled=true, pointed at the per-PR kata-monitor image
built earlier in the same run, and exercises both the rollout and the
user-visible behaviour:
* the kata-monitor DaemonSet rolls out and the pod stays up without
container restarts;
* a real kata-runtime probe pod is scheduled, then /metrics and
/sandboxes are scraped through the apiserver pod-proxy to prove
kata-monitor sees the sandbox (non-zero running-shim count plus at
least one per-sandbox kata_shim_* metric);
* after the probe pod is deleted, /metrics drops back to a zero
running-shim count.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Exercise the published kata-monitor container image (the one built by
publish-kata-monitor-payload-amd64) rather than the on-disk binary, so
integration regressions like the recent glibc/musl mismatch surface at
PR time. The kata-monitor-tests.sh script keeps the binary fallback for
ad-hoc local runs.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Drop the stale CRI-O matrix entry (its cri-tools pin was several
releases behind) along with the exclude that hid the containerd job,
and pin the remaining job to containerd's "active" track (currently
v2.2) via CONTAINERD_VERSION.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Bump the containerd version used by CI from v1.7.25 to v2.3.0.
Rename the version-range fields in versions.yaml and throughout the
GitHub Actions workflows from lts/active/version/sandbox_api to
minimum/latest to make their meaning self-evident:
minimum: "v1.7" # oldest containerd branch under test
latest: "v2.3" # newest containerd branch under test
Drop the bare version field (superseded by the matrix) and the
sandbox_api alias (covered by latest). Update all containerd_version
matrix entries in the workflow files accordingly, and update
gha-run-k8s-common.sh to resolve the new key names.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <noreply@cursor.com>
Mirror the CI payload publish flow in local builds, including image and
helm chart publishing, while reusing the same chart upload helper in
payload-after-push to avoid duplicated chart packaging logic.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
As the boot-image-se builds a fake image, the secret
CI_HKD_PATH is not necessary anymore.
Remove it from the workflows.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
- Add FAKE_SE_IMAGE mode support in SE image build scripts for CI without real SE setup
- Simplify workflow by removing build-asset-boot-image-se job
- Integrate fake-boot-image-se into build matrix instead of separate job
- Skip attestation for fake-boot-image-se builds
- Update qemu-se and qemu-se-runtime-rs shim components to use:
- rootfs-initrd-confidential instead of rootfs-image-confidential
- boot-image-se component
This change streamlines the s390x SE build process and makes it easier
to test without requiring actual Secure Execution infrastructure.
This fixes deployment issues on non-TEE systems where TEE-specific artifacts
(like boot-image-se for IBM SEL) are not included in the kata-deploy image,
while ensuring TEE systems still get all required components.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
kata-deploy currently hard-codes the EROFS snapshotter
default_size to "10G", so the CoCo EROFS CI lane only
exercises the path where the snapshotter provides an rwlayer.
Use the generic containerd.userDropIn support for the EROFS
default_size and thread it through the Kubernetes CI helpers.
Keep the kata-deploy default at "10G" to preserve current
behavior, but allow the workflow to set "0" for the runtime-rs
no-rwlayer path.
Expand the existing EROFS snapshotter job to run both values.
The override is written to containerd as a TOML string so "0"
is not parsed as an integer.
Assisted-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Drop cloud-hypervisor-glibc from local and CI kata-deploy build targets
now that Azure CLH uses the standard cloud-hypervisor artifact set.
This removes obsolete build matrix entries and installer target
handling.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Switch AKS Mariner matrix entries to clh-azure handlers and remove the
temporary host-OS based helm value overrides.
Update integration test wiring and required test labels so CI tracks the
new runtime names.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Build and publish the kata-deploy binary and CoCo guest-pull nydus
snapshotter as dedicated per-arch artifacts, then consume those tarballs
when assembling the kata-deploy image.
This avoids rebuilding those components in the payload image (which
would happen in serial) path and reduces overall CI build time.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
containerd 2.3 requires Go 1.26.3, but Kata still pins Go 1.25.10.
Use Go 1.26.3 for the sandbox-api job so that make cri-integration
can build containerd from source.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The job was disabled because TestImageLoad was failing when using the
shim sandboxer with runc due to a containerd bug (config.json not
being written to the bundle directory).
Now that check_daemon_setup uses podsandbox for the runc sanity check,
the root cause of the failure is worked around on our side and the job
can be re-enabled.
Also update the runner to ubuntu-24.04.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
