We have only one machine up and running the CIs, thus no capacity to
keep it as required for now.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
We should not ship configurations that we do not actively test.
This commit drops the following from the kata-deploy helm chart:
values.yaml:
- arm64 from supportedArches for the clh shim
- arm64 from supportedArches for the cloud-hypervisor shim
- arm64 from supportedArches for the dragonball shim
- arm64 from supportedArches for the fc shim
- arm64 from supportedArches for the qemu-nvidia-gpu shim
- the entire qemu-cca shim definition
try-kata-tee.values.yaml:
- CCA from the file description comment
- qemu-cca from the TEE shims list comment
- the entire qemu-cca shim definition
- arm64: qemu-cca from the defaultShim mapping, replaced with
arm64: qemu-coco-dev-runtime-rs (which is tested)
try-kata-nvidia-gpu.values.yaml:
- arm64 from supportedArches for the qemu-nvidia-gpu shim
- arm64: qemu-nvidia-gpu from the defaultShim mapping
Once arm64 and qemu-cca support are properly tested, they can be
re-added.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
In #12776 kata-deploy's binary was moved to the main cargo workspace,
but the Cargo.lock wasn't deleted. As it shares the main Cargo.lock tidy
this up.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The script is used to change the options used to build QEMU and **must**
be taken into consideration in case something changes, otherwise the
QEMU used by the CI would be the old cached one (ignoring any flag newly
added).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The erofs snapshotter configuration is node-wide (a single containerd
drop-in) and cannot be split per runtime handler. The Go runtime does
not support fsmerged EROFS — it rejects fsmeta.erofs mount sources with
"unsupported mount source" — so erofs is only usable with runtime-rs.
Drop qemu-coco-dev (Go) from the erofs CI matrix and add a check in
kata-deploy's configure_erofs_snapshotter() that inspects the
SNAPSHOTTER_HANDLER_MAPPING: if any Go shim is explicitly mapped to
erofs, emit a prominent warning and bail out with a clear error telling
the operator to fix the mapping.
Since all shims are now guaranteed to be runtime-rs when erofs is
active, remove the conditional is_rust_shim gating and always emit the
full erofs configuration (differ options, default_size,
max_unmerged_layers=1).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add missing containerd configuration items for erofs snapshotter to
enable fsmerged erofs feature:
Add snapshotter plugin configuration:
- default_size: "10G" # can be customized
- max_unmerged_layers: 1 # Fixed with 1
These configurations align with the documentation in
docs/how-to/how-to-use-fsmerged-erofs-with-kata.md Step 2,
ensuring the CI workflow run-k8s-tests-coco-nontee-with-erofs-snapshotter
can properly configure containerd for erofs fsmerged rootfs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The multi-layer EROFS rootfs feature relies on QEMU's VMDK flat-extent
driver to merge multiple EROFS layers into a single virtual block
device. Replace --disable-vmdk with an explicit --enable-vmdk so the
Kata static QEMU build includes VMDK support.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add aarch64/arm64 to the list of supported architectures for
qemu-coco-dev and qemu-coco-dev-runtime-rs shims across kata-deploy
configuration, Helm chart values, and test helper scripts.
Note that guest-components and the related build dependencies are not
yet wired for arm64 in these configurations; those will be addressed
separately.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Build coco-guest-components, pause-image, and rootfs-image-confidential
for arm64, which are required by qemu-coco-dev-runtime-rs.
Enable MEASURED_ROOTFS on the arm64 shim-v2 build, add the aarch64 case
to install_kernel() so the default kernel is built as a unified kernel
(with confidential guest support, like x86_64), and adjust the kernel
install naming so only CCA builds get the -confidential suffix.
Also wire rootfs-image-confidential-tarball into the aarch64 local-build
Makefile.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
The arm64 build workflow was missing the tools build entirely.
Add build-tools-asset and create-kata-tools-tarball jobs mirroring
the amd64 workflow so that genpolicy and the other tools are
available for coco-dev tests that need auto-generated policy.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Add a new extensible GetDiagnosticData RPC that retrieves diagnostic
information from the guest VM. The request carries a log_type string
field to specify what kind of data is requested, and a container_id
field to identify the target container.
The first supported log_type is "termination_log", which reads the
Kubernetes termination message file from inside the guest. This is
needed for shared_fs=none configurations where the host cannot
directly access the guest filesystem.
On the Go runtime side, the container stop() path now calls
GetDiagnosticData to copy the termination message to the host
when running with NoSharedFS and the terminationMessagePolicy
annotation is set to "File". The call is best-effort: failures
are logged as warnings rather than blocking container teardown.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Silenio Quarti <silenio_quarti@ca.ibm.com>
Add two new Helm values under `containerd`:
- `configDir`: overrides the host directory where the containerd
config lives, taking precedence over the k8sDistribution-based
auto-detection.
- `configFileName`: overrides the containerd config file name,
propagated to the kata-deploy binary via the new
CONTAINERD_CONFIG_FILE_NAME environment variable.
These are useful for non-standard containerd setups that don't match
any of the built-in k8sDistribution presets (k8s, k3s, rke2, k0s,
microk8s).
The config file name override only affects the default runtime branch
in get_containerd_paths(). The k0s/microk8s/k3s/rke2 branches are
left untouched since those runtimes have mandatory file naming
conventions.
Also fixes a spurious leading space in the k3s containerdConfPath
branch.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Update the name and move it to the static checks as we don't
need to ensure it's running for none code changes.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add a global and per-shim configurable switch to enable/disable
the overhead section in generated RuntimeClasses. This allows users
to omit overhead when it's not needed or managed externally.
Priority: per-shim > global > default(true).
Signed-off-by: LizZhang315 <123134987@qq.com>
Users were confused about which configuration file to edit because
kata-deploy copied the base config into a per-shim runtime directory
(runtimes/<shim>/) for config.d support, leaving the original file
in place untouched. This made it look like the original was the
authoritative config, when in reality the runtime was loading the
copy from the per-shim directory.
Replace the original config file with a symlink pointing to the
per-shim runtime copy after the copy is made. The runtime's
ResolvePath / EvalSymlinks follows the symlink and lands in the
per-shim directory, where it naturally finds config.d/ with all
drop-in fragments. This makes it immediately obvious that the
real configuration lives in the per-shim directory and removes the
ambiguity about which file to inspect or modify.
During cleanup, the symlink at the original location is explicitly
removed before the runtime directory is deleted.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Fix all clippy warnings triggered by -D warnings:
- install.rs: remove useless .into() conversions on PathBuf values
and replace vec! with an array literal where a Vec is not needed
- utils/toml.rs: replace while-let-on-iterator with a for loop and
drop the now-unnecessary mut on the iterator binding
- main.rs: replace match-with-single-pattern with if-let in two
places dealing with experimental_setup_snapshotter
- utils/yaml.rs: extract repeated serde_yaml::Value::String key into
a local variable, removing needless borrows on temporary values
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Ensure code formatting and compilation are verified early in the
Docker build pipeline, before tests and the release build.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Snapshotters (nydus, erofs) are containerd-specific. The validation code
already warned that EXPERIMENTAL_SETUP_SNAPSHOTTER would be ignored on
CRI-O, but the actual install/configure and uninstall loops still ran
unconditionally, attempting containerd-specific operations on CRI-O
nodes.
Guard both the install and cleanup snapshotter loops with a `runtime !=
"crio"` check so the binary itself skips snapshotter work when it
detects CRI-O as the container runtime.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Joji's added the labels for the default values.yaml, but we missed
adding those to the nvidia specific values.yaml file.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
When do cargo fmt --all, some files changes as unformatted with
`cargo fmt`. This commit is just to address it.
Just use this as an example:
```
// Generate the common drop-in files (shared with standard
// runtimes)
- write_common_drop_ins(config, &runtime.base_config,
&config_d_dir, container_runtime)?;
+ write_common_drop_ins(
+ config,
+ &runtime.base_config,
+ &config_d_dir,
+ container_runtime,
+ )?;
```
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Without those, we'd end up pulling the same / old rootfs that's cached
without re-building it in case of a bump in any of those components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Resolve externals.nydus-snapshotter version and url in the Docker image build
with yq from the repo-root versions.yaml instead of Dockerfile ARG defaults.
Drop the redundant workflow that only enforced parity between those two sources.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add tools/packaging/kata-deploy/binary as a workspace member, inherit shared
dependency versions from the root manifest, and refresh Cargo.lock.
Build the kata-deploy image from the repository root: copy the workspace
layout into the rust-builder stage, run cargo test/build with -p kata-deploy,
and adjust artifact and static asset COPY paths. Update the payload build
script to invoke docker buildx with -f .../Dockerfile from the repo root.
Add a repo-root .dockerignore to keep the Docker build context smaller.
Document running unit tests with cargo test -p kata-deploy from the root.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Update NVIDIA rootfs builder to include runtime dependencies for NVAT
Rust bindings.
The nvattest package does not include the .so file, so we need to build
from source.
Signed-off-by: Tobin Feldman-Fitzthum <tfeldmanfitz@nvidia.com>
The attestation agent will soon rely on the NVAT rust bindings, which
have some built-time dependencies.
There is currently no nvattest-dev package, so we need to build from
source to get the headers and .so file.
Signed-off-by: Tobin Feldman-Fitzthum <tfeldmanfitz@nvidia.com>
When a kata-deploy DaemonSet pod is restarted (e.g. due to a label
change or rolling update), the SIGTERM handler runs cleanup which
unconditionally removes kata artifacts and restarts containerd. This
causes containerd to lose the kata shim binary, crashing all running
kata pods on the node.
Fix this by implementing a three-stage cleanup decision:
1. If this pod's owning DaemonSet still exists (exact name match via
DAEMONSET_NAME env var), this is a pod restart — skip all cleanup.
The replacement pod will re-run install, which is idempotent.
2. If this DaemonSet is gone but other kata-deploy DaemonSets still
exist (multi-install scenario), perform instance-specific cleanup
only (snapshotters, CRI config, artifacts) but skip shared
resources (node label removal, CRI restart) to avoid disrupting
the other instances.
3. If no kata-deploy DaemonSets remain, perform full cleanup including
node label removal and CRI restart.
The Helm chart injects a DAEMONSET_NAME environment variable with the
exact DaemonSet name (including any multi-install suffix), ensuring
instance-aware lookup rather than broadly matching any DaemonSet
containing "kata-deploy".
Fixes: #12761
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Newer k3s releases (v1.34+) no longer include "k3s" in the containerd
version string at all (e.g. "containerd://2.2.2-bd1.34" instead of
"containerd://2.1.5-k3s1"). This caused kata-deploy to fall through to
the default "containerd" runtime, configuring and restarting the system
containerd service instead of k3s's embedded containerd — leaving the
kata runtime invisible to k3s.
Fix by detecting k3s/rke2 via their systemd service names (k3s,
k3s-agent, rke2-server, rke2-agent) rather than parsing the containerd
version string. This is more robust and works regardless of how k3s
formats its containerd version.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's ensure that in case nydus-snapshotter crashes for one reason or
another, the service is restarted.
This follows containerd approach, and avoids manual intervention in the
node.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's relax our RequiredBy and use a WantedBy in the nydus systemd unit
file as, in case of a nydus crash, containerd would also be put down,
causing the node to become NotReady.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Rename all host-visible names of the nydus-snapshotter instance managed
by kata-deploy from the generic "nydus-snapshotter" to "nydus-for-kata-tee".
This covers the systemd service name, the containerd proxy plugin key,
the runtime class snapshotter field, the data directory
(/var/lib/nydus-for-kata-tee), the socket path (/run/nydus-for-kata-tee/),
and the host install subdirectory.
The rename makes it immediately clear that this nydus-snapshotter instance
is the one deployed and managed by kata-deploy specifically for Kata TEE
use cases, rather than any general-purpose nydus-snapshotter that might
be present on the host.
Because the old code operated under a completely separate set of paths
(nydus-snapshotter.*), any previously deployed installation continues
to run without interference during the transition to this new naming.
CI pipelines and operators can upgrade kata-deploy on their own schedule
without having to coordinate an atomic cutover: the old service keeps
serving its existing workloads until it is explicitly replaced, and the
new deployment lands cleanly alongside it.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
SSIA, the NIM tests are breaking due to authentication issues, and those
issues are blocking other PRs.
Let's unrequire the test for now, and mark it as required again once we
fixed the auth issues.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
