Fix all clippy warnings triggered by -D warnings:
- install.rs: remove useless .into() conversions on PathBuf values
and replace vec! with an array literal where a Vec is not needed
- utils/toml.rs: replace while-let-on-iterator with a for loop and
drop the now-unnecessary mut on the iterator binding
- main.rs: replace match-with-single-pattern with if-let in two
places dealing with experimental_setup_snapshotter
- utils/yaml.rs: extract repeated serde_yaml::Value::String key into
a local variable, removing needless borrows on temporary values
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Ensure code formatting and compilation are verified early in the
Docker build pipeline, before tests and the release build.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Snapshotters (nydus, erofs) are containerd-specific. The validation code
already warned that EXPERIMENTAL_SETUP_SNAPSHOTTER would be ignored on
CRI-O, but the actual install/configure and uninstall loops still ran
unconditionally, attempting containerd-specific operations on CRI-O
nodes.
Guard both the install and cleanup snapshotter loops with a `runtime !=
"crio"` check so the binary itself skips snapshotter work when it
detects CRI-O as the container runtime.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Joji's added the labels for the default values.yaml, but we missed
adding those to the nvidia specific values.yaml file.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
When do cargo fmt --all, some files changes as unformatted with
`cargo fmt`. This commit is just to address it.
Just use this as an example:
```
// Generate the common drop-in files (shared with standard
// runtimes)
- write_common_drop_ins(config, &runtime.base_config,
&config_d_dir, container_runtime)?;
+ write_common_drop_ins(
+ config,
+ &runtime.base_config,
+ &config_d_dir,
+ container_runtime,
+ )?;
```
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Without those, we'd end up pulling the same / old rootfs that's cached
without re-building it in case of a bump in any of those components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Resolve externals.nydus-snapshotter version and url in the Docker image build
with yq from the repo-root versions.yaml instead of Dockerfile ARG defaults.
Drop the redundant workflow that only enforced parity between those two sources.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add tools/packaging/kata-deploy/binary as a workspace member, inherit shared
dependency versions from the root manifest, and refresh Cargo.lock.
Build the kata-deploy image from the repository root: copy the workspace
layout into the rust-builder stage, run cargo test/build with -p kata-deploy,
and adjust artifact and static asset COPY paths. Update the payload build
script to invoke docker buildx with -f .../Dockerfile from the repo root.
Add a repo-root .dockerignore to keep the Docker build context smaller.
Document running unit tests with cargo test -p kata-deploy from the root.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Update NVIDIA rootfs builder to include runtime dependencies for NVAT
Rust bindings.
The nvattest package does not include the .so file, so we need to build
from source.
Signed-off-by: Tobin Feldman-Fitzthum <tfeldmanfitz@nvidia.com>
The attestation agent will soon rely on the NVAT rust bindings, which
have some built-time dependencies.
There is currently no nvattest-dev package, so we need to build from
source to get the headers and .so file.
Signed-off-by: Tobin Feldman-Fitzthum <tfeldmanfitz@nvidia.com>
When a kata-deploy DaemonSet pod is restarted (e.g. due to a label
change or rolling update), the SIGTERM handler runs cleanup which
unconditionally removes kata artifacts and restarts containerd. This
causes containerd to lose the kata shim binary, crashing all running
kata pods on the node.
Fix this by implementing a three-stage cleanup decision:
1. If this pod's owning DaemonSet still exists (exact name match via
DAEMONSET_NAME env var), this is a pod restart — skip all cleanup.
The replacement pod will re-run install, which is idempotent.
2. If this DaemonSet is gone but other kata-deploy DaemonSets still
exist (multi-install scenario), perform instance-specific cleanup
only (snapshotters, CRI config, artifacts) but skip shared
resources (node label removal, CRI restart) to avoid disrupting
the other instances.
3. If no kata-deploy DaemonSets remain, perform full cleanup including
node label removal and CRI restart.
The Helm chart injects a DAEMONSET_NAME environment variable with the
exact DaemonSet name (including any multi-install suffix), ensuring
instance-aware lookup rather than broadly matching any DaemonSet
containing "kata-deploy".
Fixes: #12761
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Newer k3s releases (v1.34+) no longer include "k3s" in the containerd
version string at all (e.g. "containerd://2.2.2-bd1.34" instead of
"containerd://2.1.5-k3s1"). This caused kata-deploy to fall through to
the default "containerd" runtime, configuring and restarting the system
containerd service instead of k3s's embedded containerd — leaving the
kata runtime invisible to k3s.
Fix by detecting k3s/rke2 via their systemd service names (k3s,
k3s-agent, rke2-server, rke2-agent) rather than parsing the containerd
version string. This is more robust and works regardless of how k3s
formats its containerd version.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's ensure that in case nydus-snapshotter crashes for one reason or
another, the service is restarted.
This follows containerd approach, and avoids manual intervention in the
node.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Let's relax our RequiredBy and use a WantedBy in the nydus systemd unit
file as, in case of a nydus crash, containerd would also be put down,
causing the node to become NotReady.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Rename all host-visible names of the nydus-snapshotter instance managed
by kata-deploy from the generic "nydus-snapshotter" to "nydus-for-kata-tee".
This covers the systemd service name, the containerd proxy plugin key,
the runtime class snapshotter field, the data directory
(/var/lib/nydus-for-kata-tee), the socket path (/run/nydus-for-kata-tee/),
and the host install subdirectory.
The rename makes it immediately clear that this nydus-snapshotter instance
is the one deployed and managed by kata-deploy specifically for Kata TEE
use cases, rather than any general-purpose nydus-snapshotter that might
be present on the host.
Because the old code operated under a completely separate set of paths
(nydus-snapshotter.*), any previously deployed installation continues
to run without interference during the transition to this new naming.
CI pipelines and operators can upgrade kata-deploy on their own schedule
without having to coordinate an atomic cutover: the old service keeps
serving its existing workloads until it is explicitly replaced, and the
new deployment lands cleanly alongside it.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
SSIA, the NIM tests are breaking due to authentication issues, and those
issues are blocking other PRs.
Let's unrequire the test for now, and mark it as required again once we
fixed the auth issues.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Removing /var/lib/nydus-snapshotter during install or uninstall creates
a split-brain state: the nydus backend starts empty while containerd's
BoltDB (meta.db) still holds snapshot records from the previous run.
Any subsequent image pull then fails with:
"unable to prepare extraction snapshot:
target snapshot \"sha256:...\": already exists"
An earlier attempt cleaned up containerd's BoltDB via `ctr snapshots rm`
before wiping the directory, but that cleanup is inherently fragile:
- It requires the nydus gRPC service to be reachable at cleanup time.
If the service is stopped, crashed, or not yet running, every `ctr`
call silently fails and the stale records remain.
- Any workload still actively using a snapshot blocks the entire
cleanup, making it impossible to guarantee a clean state.
The correct invariant is that meta.db and the nydus backend always
agree. Preserving the data directory unconditionally guarantees this:
- Fresh install: data directory does not exist, nydus starts empty.
- Reinstall: existing snapshots and nydus.db are preserved, meta.db
and backend remain in sync, new binary starts cleanly.
- After uninstall: containerd is reconfigured without the nydus
proxy_plugins entry and restarted, so the snapshot records in
meta.db are completely dormant — nothing will use them. If nydus
is reinstalled later, the data directory is still present and both
sides remain in sync, so no split-brain can occur.
Any stale snapshots from previous workloads are garbage-collected by
containerd once the images referencing them are removed.
This also removes the cleanup_containerd_nydus_snapshots,
cleanup_nydus_snapshots, and cleanup_nydus_containers helpers that
were introduced by the earlier (fragile) attempt.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
When /var/lib/nydus-snapshotter is removed, containerd's BoltDB
(meta.db at /var/lib/containerd/) still holds snapshot records for
the nydus snapshotter. On the next install these stale records cause
image pulls to fail with:
"unable to prepare extraction snapshot:
target snapshot \"sha256:...\": already exists"
The failure path in core/unpack/unpacker.go:
1. sn.Prepare() → metadata layer finds the target chainID in BoltDB
→ returns AlreadyExists without touching the nydus backend.
2. sn.Stat() → metadata layer finds the BoltDB record, then calls
s.Snapshotter.Stat(bkey) on the nydus gRPC backend → NotFound
(backend was wiped).
3. The unpacker treats NotFound as a transient key-collision race and
retries 3 times; all 3 attempts hit the same dead end, and the
pull is aborted.
The commit message of 62ad0814c ("nydus: Always start from a clean
state") assumed "containerd will re-pull/re-unpack when it finds non-
existent snapshots", but that is not what happens: the metadata layer
intercepts the Prepare call in BoltDB before the backend is ever
consulted.
Fix: call cleanup_containerd_nydus_snapshots() before stopping the
nydus service (and thus before wiping its data directory) in both
install_nydus_snapshotter and uninstall_nydus_snapshotter.
The cleanup must run while the service is still up because ctr
snapshots rm goes through the metadata layer which calls the nydus
gRPC backend to physically remove the snapshot; if the service is
already stopped the backend call fails and the BoltDB record remains.
The cleanup:
- Discovers all containerd namespaces via `ctr namespaces ls -q`
(falls back to k8s.io if that fails).
- Removes containers whose Snapshotter field matches the nydus plugin
name; these become dangling references once snapshots are gone and
can confuse container reconciliation after an aborted CI run.
- Removes snapshots round by round (leaf-first) until either the list
is empty or no progress can be made (see below).
Note: containerd's GC cannot substitute for this explicit cleanup.
The image record (a GC root) references content blobs which reference
the snapshots via gc.ref labels, keeping the entire chain alive in
the GC graph even after the nydus backend is wiped.
Snapshot removal rounds
-----------------------
Snapshot chains are linear: an image with N layers produces a chain
of N snapshots, each parented on the previous. Only the current leaf
can be removed each round, so N layers require exactly N rounds.
There is no fixed round cap — the loop terminates when either the
list reaches zero (success) or a round removes nothing at all
(all remaining snapshots are actively in use by running workloads).
Active workload safety
----------------------
If active workloads still hold nydus snapshots (e.g. during a live
upgrade), no progress is made in a round and cleanup_nydus_snapshots
returns false. Both install_nydus_snapshotter and
uninstall_nydus_snapshotter gate the fs::remove_dir_all on that
return value:
- true → proceed as before: stop service, wipe data dir.
- false → stop service, skip data dir removal, log a warning.
The new nydus instance starts on the existing backend
state; running containers are left intact.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
We've been using `experimental_force_guest_pull`, but now that we have a
containerd release that should work more reliably with the multi
snapshotter setup, we want to give it a try.
Note: We need containerd 2.2.2+.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
With debug/ebpf updates in place, let's bump the kata config version.
Signed-off-by: Agam Dua <agam_dua@apple.com>
Co-authored-by: Eric Ernst <eric_ernst@apple.com>
Add missing terms to the spell check dictionary to fix CI failures
for kernel debug documentation:
- eBPF
- dwarves: Linux package with DWARF/BTF tools (pahole) required for
CONFIG_DEBUG_INFO_BTF kernel option
Also fix the casing of "ebpf" to "eBPF" in the kernel README to match
the official naming convention.
Signed-off-by: Agam Dua <agam_dua@apple.com>
Fixed a bug with the debug kernel build where common/ was repeated
after the common path variable, resulting in the debug
confs never being picked up.
This exposed a subsequent bug where the debug conf
was included in other builds, this is also fixed by creating a
separate directory for debug confs with one file at the moment,
debug.conf that contains debug configurations and bpf specific
configs.
To enable kernel builds (specifically for bpf) the dwarves package was added
to the kernel dockerfile for the pahole package.
Signed-off-by: Agam Dua <agam_dua@apple.com>
Add the debug kernel to the kata tarball alongside the other kernels.
Also update the kernel README documentation to describe the new debug
kernel build process.
Signed-off-by: Agam Dua <agam_dua@apple.com>
Adds a BPF section in the debug.conf kernel configuration options
to enable eBPF and BTF support for debug kernel builds.
Signed-off-by: Agam Dua <agam_dua@apple.com>
This supersedes https://github.com/kata-containers/kata-containers/pull/12622.
I replaced Zensical with mkdocs-materialx. Materialx is a fork of mkdocs-material
created after mkdocs-material was put into maintenance mode. We'll use this
platform until Zensical is more feature complete.
Added a few of the existing docs into the site to make a more user-friendly flow.
Signed-off-by: LandonTClipp <11232769+LandonTClipp@users.noreply.github.com>
As the NVIDIA stack has shifted to using an image for both the
confidential and non-confidential variants, we retire the initrd
build.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
With CDH requiring libcryptsetup, mkfs.ext4, dd, and their
dependencies, we will need to update the chiseled NVIDIA rootfs
accordingly.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
With the new CDH version, the secure_mount API changes.
Further, the new CDH version no longer uses the luks-encrypt-storage
script but utilizes libcryptsetup as well as mkfs.ext4 and dd. Hence, adapt
some of the CDH and Kata components build steps
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
