The retry loop added in efd468df3f still allows the install to declare
success while inside the kubelet's post-restart re-register window.
On rke2/k3s, `systemctl restart rke2-agent` restarts both containerd
and the kubelet, but `wait_till_node_is_ready` polls `.status.conditions[Ready]`
every 2 s and returns on the first `True` observation it sees. By default
the kubelet only publishes node status every ~10 s, so that first `True`
is almost always the stale value from before the restart — the kubelet
hasn't actually finished restarting yet. `label_node_with_retry` then
applies the label, sleeps 1 s, reads back "true" (still stale, kubelet
still down), and returns Ok. Install completes, `/readyz` flips to 200,
helm releases its `--wait`, and the bats test starts — and only then
does the kubelet finish coming up, re-register the node, and clobber
the label with its cached set. The lifecycle test sees an empty
`katacontainers.io/kata-runtime` and fails:
# Node label katacontainers.io/kata-runtime:
not ok 1 Kata artifacts are present on host after install
A single-shot verification can't distinguish "still stale true" from
"truly stable true after kubelet re-register". Replace it with a
stability window: after (re)applying the label, require it to remain
at the expected value for STABILITY_CHECKS=6 consecutive observations
spaced CHECK_INTERVAL=2 s apart (≈ 12 s — comfortably more than the
kubelet's status-update period). If the value ever drifts inside the
window, re-apply and restart the stability counter. Bounded by
MAX_APPLY_ATTEMPTS=12, so worst case is ~3 min; happy path adds ~12 s
to install.
Also add a short polling loop to the test's own label assertion as
belt-and-suspenders for any leftover transient race, matching the
existing retry pattern used for the container-runtime version check.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Temporarily skip the `TestContainerMemoryUpdate` test case
for sandbox api.
This test case is currently skipped in other VMMs (e.g.,
QEMU, Cloud-Hypervisor) due to known issues and environmental
stability concerns.
To maintain consistency across the project, we are skipping it
for sandbox as well.
A follow-up PR will be dedicated to addressing these issues and
properly enabling/refining this test case for all VMMs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
containerd 2.3 requires Go 1.26.3, but Kata still pins Go 1.25.10.
Use Go 1.26.3 for the sandbox-api job so that make cri-integration
can build containerd from source.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Creating a new container in the same sandbox VM after the previous
container has exited and been removed has never been supported by
kata-containers (neither with the go-based nor the rust-based runtime).
When the last container is removed the kata VM shuts down, so any
attempt to start a new container in the same sandbox fails.
This test exercises a use-case kata does not currently support, and it
has never been part of the passing list for good reason. Mark it
explicitly excluded with a comment so it is clear this is a deliberate
omission rather than an oversight.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The check_daemon_setup function verifies that containerd + runc are
functional before the real kata tests run. Using the shim sandboxer
for this runc check hits a known containerd bug where the OCI spec
is not populated before NewBundle is called, so config.json is never
written and containerd-shim-runc-v2 fails at startup.
See containerd/containerd#11640
The sandboxer choice is irrelevant for this sanity check, so use
podsandbox which works correctly with runc.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The agent-ctl tests are failing in the CI, but there is no log reporting,
so debugging is not possible. Add some debug to help.
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Since gc and trustee were bumped (#13046), the test
"Cannot get CDH resource when affirming policy is set without reference values"
has started failing for IBM SEL.
The attestation policy for IBM SEL returns an "affirming"
result whenever the claim can be parsed successfully,
meaning the evidence verification succeeds. As a result,
the negative test above always produces a positive result.
Skip this negative test for IBM SEL environments
(e.g. qemu-se*).
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Exit early with an error message instead of starting kata-deploy if
the value of KATA_HYPERVISOR is not expected during CI.
For example: "cloud-hypervisor" was renamed recently to
"clh-runtime-rs" and user scripts depending on the old name were
getting tangled in kata-deploy instead of just rejecting the old
value quickly.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Avoid running "git remote show origin" repeatedly when common.bash
gets sourced multiple times and target_branch was not specified by
the caller.
Repeated "git remote show origin" calls inflicted the additional
overhead of authenticating and communicating with the remote git
repository.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Now that runtime-rs supports block-encrypted emptyDir volumes, remove
the no-trusted-storage workaround templates and the is_runtime_rs
branching in the NIM test. Runtime-rs now uses the same TEE templates
as the Go runtime with emptyDir + PVC at 48Gi memory, instead of the
128Gi workaround that compensated for lacking trusted storage.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Remove the runtime-rs skip from the trusted ephemeral data storage
test now that runtime-rs implements block-encrypted emptyDir volumes.
Also remove the genpolicy drop-in that disabled encrypted_emptydir
for runtime-rs and the corresponding copy logic in tests_common.sh.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Update CDH to a newer version and:
- adjust the NVIDIA root filesystem build to reflect the change from
using libcryptsetup to using the cryptsetup binary.
- adjust image-pull test cases to conduct parallel write operations
on the /dev/trusted_store backed guest image pull location since
issue #12721 has been solved on CDH side.
Fixes#12721
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
k8s-measured-rootfs only runs on confidential runtime,
so we should move it into the subset on tests that run on TEEs
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The docs URL alive check workflow has been disabled for months
and never passed since we moved to GHA., so removes the workflow
and all associated code.
Note: Although the static-checks.sh --doc code was wider scope than
URL check, it wasn't being called anywhere else, so it was removed too.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Assisted-by: IBM Bob
Replace guest-pull image allow-all placeholders with explicit
auto-generated policies for each generated pod manifest.
Generate policy after the final YAML edits so initdata and image
pull secrets are represented in the policy inputs.
Assisted-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Teach auto_generate_policy to reuse a cc_init_data annotation by
decoding it into the temporary default-initdata.toml file.
This lets tests preserve CDH initdata while genpolicy appends the
generated agent security policy for the workload.
Assisted-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Move the Docker auth setup into common.bash so tests beyond the
NVIDIA runner can provide credentials for genpolicy image pulls.
Make the registry, username, password and output directory explicit
while preserving the nvcr.io setup used by the NIM tests.
Assisted-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Bump the go version to resolve CVEs:
- GO-2026-4918
- GO-2026-4971
- GO-2026-4976
- GO-2026-4977
- GO-2026-4980
- GO-2026-4981
- GO-2026-4982
- GO-2026-4986
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Assisted-by: IBM Bob
BATS_TEST_COMPLETED is per-test and remains empty in teardown_file.
Track file-level state so successful NIM runs skip the journal dump
while setup or test failures still include node diagnostics.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Place the NIM service into our test namespace. We are still observing
various situations where for some reasons, the NIM service appears in
the default namespace in our CI.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Wait for the NIM operator pod to run before deploying NIM services.
Add a temporary debug function to print resource placement into the
different namespaces. Remove this function again when the NIM tests
are stabilized.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Add basic genpolicy support for container environment variables sourced
from metadata.labels.
In this implementation, the relevant labels must be available as input
to the policy tool. This is slightly different from the way variables
sourced from metadata.annotations are treated by the tool: when the
relevant annotation is not available as input, the generated Policy
allows any value. Depending on metadata.labels use cases that we might
encounter maybe the labels will be handled the same way as the
annotations in the future.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Switch the rootfs bundle pull implementatio from using image-rs to
use skopeo and umoci to remove the really long crate dependency
tail that image-rs brings.
Generated-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
When kubectl wait times out the pod never reached Ready, so the
existing log collection (which runs after wait succeeds) produces
"-- No entries --" with zero useful information.
Capture kubectl describe and kubectl logs (including previous
container) immediately on timeout so the next CI run shows exactly
why the pod is stuck (ImagePullBackOff, OOMKilled, probe failures,
containerd restart hang, etc.).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
In cleanup_kata_deploy, bail out early when no kata-deploy Helm release
exists so baremetal-* pre-deploy cleanup on fresh clusters does not
block on helm uninstall --wait (up to 10m).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add qemu-nvidia-gpu-runtime-rs and qemu-nvidia-gpu-snp-runtime-rs to
the NVIDIA GPU test matrix so CI covers the new runtime-rs shims.
Introduce a `coco` boolean field in each matrix entry and use it for
all CoCo-related conditionals (KBS, snapshotter, KBS deploy/cleanup
steps). This replaces fragile name-string comparisons that were already
broken for the runtime-rs variants: `nvidia-gpu (runtime-rs)` was
incorrectly getting KBS steps, and `nvidia-gpu-snp (runtime-rs)` was
not getting the right env vars.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
1. Ignore PodAffinity's preferredDuringSchedulingIgnoredDuringExecution.
2. Ignore additional PodAffinityTerm fields.
3. Add basic tests for the new fields.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
The cron-job test workload was missing `runtimeClassName: kata`, which
meant the cron job was not actually being executed under the Kata
runtime, defeating the purpose of the test.
Set it explicitly, consistent with the sibling `job.yaml` workload.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Fix two issues in kata-deploy-lifecycle.bats that caused failures on
k3s, k0s and rke2:
run_on_host():
- `kubectl run --rm -i` causes k3s/rke2 to inject session-recording
banners into stdout, polluting command output and breaking string
assertions. Replace with a create/wait/logs/delete sequence so only
the container's actual stdout is captured.
"Artifacts are fully cleaned up after uninstall":
- After a CRI restart the kubelet may briefly report "Unknown" for the
container runtime version. Retry for up to 60s before asserting.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
