Commit Graph

2469 Commits

Author SHA1 Message Date
Fabiano Fidêncio
63ddcdaf39 fixup! build: ship CoCo extension component manifest in extension image 2026-07-09 18:32:05 +02:00
Fabiano Fidêncio
1cdfc8ad42 fixup! build: split NVIDIA rootfs into base-nvidia image + gpu extension 2026-07-09 17:47:41 +02:00
Fabiano Fidêncio
7c74244d69 build: wire nvrc + nvidia-gpu base/extension into CI and packaging
Add the nvrc static-build target and make the NVIDIA rootfs targets
depend on it (the rootfs build consumes kata-static-nvrc.tar.zst), and
add the rootfs-image-nvidia-gpu-{base,extension} build targets to the
local-build Makefile and the nvgpu bundle.

Build nvrc and the base/extension rootfs images in the amd64/arm64 static
tarball workflows (nvrc is baked into the rootfs and pruned from the
release like the other guest components), and point the runtime-rs NVIDIA
shim components at the composable base + gpu extension (+ coco extension for
the confidential classes). The Go shim components keep the monolithic
nvidia-gpu / nvidia-gpu-confidential images.

Ship cryptsetup in the CoCo guest-components extension (and bundle
libnvat's non-glibc dependency closure next to the nvidia attester): the
chiseled base-nvidia image carries only veritysetup, so the coco-extension
manifest's usr/sbin PATH entry needs a cryptsetup to resolve for NVIDIA
confidential encrypted-storage.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-07-09 15:17:05 +02:00
Fabiano Fidêncio
51ff020f8b config: boot composable base-nvidia image + gpu/coco extensions
Flip the runtime-rs NVIDIA GPU configs (plain/tdx/snp) from the monolithic
image to the driver-agnostic base-nvidia boot image plus a cold-plugged,
driver-versioned gpu extension mounted at /run/kata-extensions/gpu.
Confidential runtime-rs configs additionally cold-plug the CoCo
guest-components extension and enforce the base-nvidia dm-verity hash.

The Go runtime keeps booting the monolithic nvidia-gpu / nvidia-gpu-confidential
images, mirroring the CoCo split where runtime-rs is the first-class citizen and
Go stays on the monolithic layout.

Add the IMAGE{NAME,PATH}_NV_{BASE,EXTENSION} and NVIDIAGPUEXTENSIONVERITYPARAMS
make vars (runtime-rs only), and read the base-nvidia + gpu-extension root
hashes into runtime-rs's shim-v2 opts while the Go opts keep reading the
monolithic nvidia-gpu / nvidia-gpu-confidential hashes.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-07-09 15:17:05 +02:00
Fabiano Fidêncio
522ca781d7 build: split NVIDIA rootfs into base-nvidia image + gpu extension
Carve the monolithic NVIDIA GPU rootfs into a driver-agnostic
base-nvidia image (NVRC init + agent + base libs + in-tree modules)
and a driver-versioned gpu extension (GPU userspace, configs, firmware,
NVIDIA modules) laid out for /run/kata-extensions/gpu. Both halves are
produced from the same chiseled tree via a partition-after step, so
the monolith build stays byte-identical and continues to ship.

The shared stage-one (driver install) is reused across the monolith,
base and extension variants; base-nvidia is cached without the driver/ctk
versions so one base image can back multiple driver extensions.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-07-09 15:17:05 +02:00
Fabiano Fidêncio
f16b2fff3e DO-NOT-MERGE: build NVRC from a pinned git ref
Add a Kata static-build component for NVRC (tools/packaging/static-build/nvrc)
that clones the NVRC repository at a pinned git reference and builds a static
musl binary, mirroring the coco-guest-components build.  This lets the
composable-image work track NVRC changes from source instead of depending on a
signed upstream release.

The ref points at NVIDIA/nvrc#167 (fidencio/nvrc topic/adapt-for-composable-kata-images,
20d38ea), which carries the guest-side extension mounting.  This is a CI-only
crutch: once NVRC cuts a signed release with that work we go back to
downloading it from `url`, so this commit must be dropped before merge.

  - versions.yaml: add repo/ref/toolchain to externals.nvrc (and bump the
    version); the existing url is kept for the signed-release path.
  - lib.sh: add get_nvrc_image_name for the builder image.
  - kata-deploy-binaries.sh: add the nvrc build target, install_nvrc and its
    tarball path; get_latest_nvidia_nvrc_version now tracks ref+toolchain so
    the dependent NVIDIA images rebuild when the ref changes.
  - nvidia_rootfs.sh: consume kata-static-nvrc.tar.zst instead of downloading
    and cosign-verifying the release; the Sigstore .cert/.sig are now copied
    only when present (i.e. the signed-release path), since a source build
    emits just the binary.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-07-09 15:17:05 +02:00
Zachary Spar
83497cf994 kata-deploy: harden NFD nodeAffinity merge and document semantics
Replace fragile per-term Helm merge with explicit NodeSelectorTerm
construction, document (NFD OR-group) AND (user OR-group) behavior, and
add bats coverage to prevent merge regressions.

Signed-off-by: Zachary Spar <zspar@coreweave.com>
2026-07-06 12:00:47 -04:00
Zachary Spar
f9505a4cf5 kata-deploy: harden affinity merge
Merge NFD and user nodeSelectorTerms with merge/concat.

Ignore the name key in podLabels, document podAnnotations
and extend bats coverage for annotations.

Signed-off-by: Zachary Spar <zspar@coreweave.com>
2026-07-06 11:48:43 -04:00
Zachary Spar
5e4946b21e kata-deploy: add podLabels, podAnnotations and affinity to DaemonSet
Expose podLabels, podAnnotations and affinity in values.yaml so
operators can apply more granular scheduling controls on the
kata-deploy DaemonSet alongside the existing nodeSelector and
tolerations knobs.

Defaults are empty, so existing deployments render the same manifest.
When NFD is enabled, the chart merges the hard-coded virtualization
nodeAffinity with any user provided affinity.

Signed-off-by: Zachary Spar <zspar@coreweave.com>
2026-07-06 11:48:43 -04:00
Dimitris Karakasilis
df029f4f7a packaging: optionally fetch static nydus-snapshotter for the payload
The kata-deploy payload downloads the nydus-snapshotter host binaries
(containerd-nydus-grpc, nydus-overlayfs) from the upstream per-arch
release asset, which is glibc dynamically linked and fails to exec on
musl-only hosts that have no glibc dynamic loader.

nydus-snapshotter also publishes a statically linked release asset.
Honour the existing STATIC_RUNTIME=yes umbrella flag (which already
builds the kata Go host binaries static) in the nydus component build so
that, with a single flag, the whole host-binary set of the payload is
static and runs on a musl-only host. STATIC_RUNTIME=yes maps to a new
STATIC_NYDUS_SNAPSHOTTER build-arg on the nydus downloader stage of
Dockerfile.components, which selects the linux-static asset.

The static nydus asset is published for amd64 only, so the build fails
fast on other architectures when the flag is set. The default build is
unchanged.

Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
2026-07-06 12:02:13 +02:00
Dimitris Karakasilis
2d945ad10b packaging: allow building a static cgo-free shim-v2 payload
Add an opt-in STATIC_RUNTIME=yes knob to the shim-v2 static build. When
set, the Go host binaries are built with the runtime STATIC=yes profile
(CGO_ENABLED=0, no PIE), producing fully static, libc-agnostic binaries
in the kata-deploy payload.

This lets a payload image be produced that runs on musl-only hosts which
have no glibc dynamic loader, where the default glibc-linked binaries
fail to exec. The default build is unchanged.

Signed-off-by: Dimitris Karakasilis <dimitris@spectrocloud.com>
Generated-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
2026-07-06 12:02:13 +02:00
Fabiano Fidêncio
bd0e3cb54e Merge pull request #13285 from kata-containers/topic/coco-rootfs-split-for-systemd-based-images
rootfs: Add base + coco addon for systemd based images
2026-07-01 04:14:14 +02:00
Fabiano Fidêncio
344a3597fb Merge pull request #13284 from punkwalker/feat/kata-deploy-startup-taint-removal
kata-deploy: remove configurable startup taints after install
2026-06-30 22:15:07 +02:00
Fabiano Fidêncio
e01a0b8975 build: ship CoCo extension component manifest in extension image
Generate etc/kata-extensions/components.toml inside the CoCo extension rootfs
before building the image, so the manifest becomes part of the
dm-verity-measured erofs image.

The manifest declares the extension's path-only components (ocicrypt config,
pause bundle) and the launchable processes (attestation-agent,
confidential-data-hub, api-server-rest) with paths relative to the extension
mount point (/run/kata-extensions/coco) and `${var}` placeholders resolved
by kata-agent at runtime. This is the data the agent consumes to launch
the CoCo guest components without per-bundle code changes.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-30 21:51:48 +02:00
Fabiano Fidêncio
8e25e7636f build: add CoCo extension image build
Add install_image_coco_extension() to kata-deploy-binaries.sh which:
- Unpacks the CoCo guest components and pause image tarballs into a
  temporary rootfs directory (under the repo root so Docker-in-Docker
  volume mounts resolve correctly)
- Calls image_builder.sh with USE_DOCKER=1, FS_TYPE=erofs,
  MEASURED_ROOTFS=yes, SKIP_DAX_HEADER=yes, and SKIP_ROOTFS_CHECK=yes
  to produce kata-containers-coco-extension.img + root_hash_coco-extension.txt

Add the rootfs-image-coco-extension-tarball Makefile target with
dependencies on pause-image-tarball and coco-guest-components-tarball.

The standard confidential image keeps pause-image and
coco-guest-components for now so it remains a usable standalone
monolithic CoCo image during the transition to base + extension.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-30 21:51:48 +02:00
Fabiano Fidêncio
bd93a93990 config: use split CoCo image for runtime-rs, keep Go on monolithic
During the transition to composable (base + extension) CoCo images, run the
two runtimes on different image layouts so CI exercises both paths via
the existing qemu-{snp,tdx,coco-dev} (Go) and -runtime-rs (Rust) shims:

- runtime-rs (Rust) CoCo templates use the standard base image plus a
  CoCo extension extra_image:
      image = "@IMAGEPATH@"          (was @IMAGECONFIDENTIALPATH@)
      [[hypervisor.qemu.guest_extension_images]]
      name = "coco"
      path = "@COCOIMAGEPATH@"
      verity_params = "@COCOVERITYPARAMS@"
  COCOIMAGENAME/COCOIMAGEPATH/COCOVERITYPARAMS are added to the
  runtime-rs Makefile only.

- runtime (Go) CoCo templates keep the monolithic confidential image
  (@IMAGECONFIDENTIALPATH@) and pull rootfs-image-confidential.

shim-components.json reflects this: Go CoCo shims depend on
rootfs-image-confidential while the runtime-rs CoCo shims depend on
rootfs-image + rootfs-image-coco-extension.

shim-v2/build.sh feeds each runtime its own dm-verity params: Rust gets
the measured base image hash (@KERNELVERITYPARAMS@) plus the extension hash
(@COCOVERITYPARAMS@); Go gets the monolithic confidential image hash
(@KERNELVERITYPARAMS@).  This is wired per make invocation so the same
@KERNELVERITYPARAMS@ placeholder resolves correctly for each.

Once the split path is validated we can flip the Go templates too and
drop the monolithic confidential image.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-30 21:51:48 +02:00
Fabiano Fidêncio
1906a44413 image-builder: support building extension images without /sbin/init check
Extension images (e.g. the CoCo guest components extension) are not full root
filesystems -- they contain only the binaries and configuration that
get bind-mounted into the real rootfs at boot.  The existing
check_rootfs() validation requires /sbin/init and systemd, which are
not present in extension images.

Add a SKIP_ROOTFS_CHECK environment variable that, when set to "yes",
bypasses the check_rootfs() call.  Forward the variable into the
container environment when using the Docker-based build path so it
works in both direct and containerised invocations.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-30 21:51:48 +02:00
Zvonko Kaiser
9686d62cb4 Merge pull request #13299 from fidencio/topic/versions-yaml-must-be-installed
kata-deploy: restore versions.yaml installation on host nodes
2026-06-29 14:37:15 -04:00
Fabiano Fidêncio
6f47183c28 kata-deploy: restore versions.yaml installation on host nodes
Since the switch to per-component tarballs, versions.yaml was no longer
installed on host nodes by kata-deploy (DaemonSet/Helm). Previously it
was bundled into the merged kata-static.tar.zst; now it must be shipped
explicitly.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-06-29 12:07:15 +02:00
stevenhorsman
36b674ed51 workflow: wire agent policy coverage check into static-checks
Add a check-agent-policy-coverage job to static-checks.yaml
that runs ci/check_agent_policy_coverage.sh on every pull request.

Generated-By: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
2026-06-29 10:49:26 +01:00
Manuel Huber
0bf87a3459 rootfs: mariner: include e2fsprogs package
Add e2fsprogs to the Azure Linux UVM rootfs package list.
This enables using the block-plain emptyDir feature for the clh-azure
scenario.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
2026-06-26 21:05:51 +00:00
Manuel Huber
023620f880 osbuilder: add mkfs.ext4 for NVIDIA rootfs
Include the mkfs.ext4 binary and runtime libraries in NVIDIA guest
rootfs builds.

The agent formats block-plain emptyDir disk images inside the guest, so
NVIDIA rootfs variants need the same e2fsprogs pieces that CDH secure
storage already depends on.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
2026-06-26 21:05:51 +00:00
Pankaj Walke
5321c9d122 kata-deploy: remove configurable startup taints after install
kata-deploy installs the Kata runtime asynchronously (DaemonSet or
per-node Jobs), labelling the node katacontainers.io/kata-runtime=true
as its final step. On a freshly provisioned node, a pod that requests a
Kata RuntimeClass can be scheduled before the runtime binaries exist and
fail to start. There is currently no mechanism to gate scheduling on the
runtime actually being ready.

Add a STARTUP_TAINTS option (helm value: startupTaints) listing taints
to remove once install completes and the node is labelled. Operators can
provision nodes with a startup taint (e.g. a Karpenter NodePool
startupTaints entry) that keeps Kata workloads off the node until
kata-deploy removes it as the very last install action, strictly after
artifacts are installed, the CRI runtime is configured and restarted,
and the node is labelled kata-capable.

Each entry is a bare key (any effect) or key:effect. Removal is
best-effort and idempotent: an absent taint is skipped, and a failure to
remove leaves the taint in place (workloads stay gated) rather than
admitting them prematurely. The default is empty, so the behaviour is
opt-in and a no-op for existing deployments. The taint is wired through
the shared commonEnv helper, so both daemonset and job deployment modes
honour it. No RBAC change is needed: kata-deploy already holds
nodes [get, patch].

Fixes: #13283

Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Pankaj Walke <punkwalker@gmail.com>
2026-06-26 19:02:53 +00:00
Alex Lyn
e77795f573 ci: Update libs required-test names for libdevmapper dependency
Update the two affected entries in required-tests.yaml accordingly
so the gatekeeper keeps matching them instead of blocking subsequent
PRs after this one merges.

Co-authored-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
2026-06-26 09:52:47 +08:00
Alex Lyn
adcbef0c53 kata-deploy: Configure containerd erofs for dm-verity integrity mode
The deploy will read EROFS_SNAPSHOTTER_MODE and EROFS_DMVERITY from
the environment to enable dmverity_mode and enable_dmverity in the
containerd erofs snapshotter/differ config.

Add validation for the mode value and use an explicit 300s timeout
for node-readiness checks during kata-deply in github CI.

Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
2026-06-26 09:51:05 +08:00
Alex Lyn
562c9acdb2 packaging: Add libdevmapper-dev and GNU target to agent Dockerfile
Install libdevmapper-dev and pkg-config in the agent build container
so devicemapper-sys can link against libdevmapper. Add the GNU libc
rustup target alongside musl since USE_DEVMAPPER forces LIBC=gnu.

Forward USE_DEVMAPPER through build.sh and build-static-agent.sh.
And you can compile the device mapper in kata-agent as below:

```
$ make LIBC=gnu USE_DEVMAPPER=yes
```

Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
2026-06-26 09:51:05 +08:00
Alex Lyn
b084c0df36 kata-deploy: Forward USE_DEVMAPPER in local build scripts
Pass USE_DEVMAPPER through the Docker environment in local build
scripts. Extract the OCI tag sanitization logic into a public helper
of sanitize_tag_component to keep push and pull paths consistent.

Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
2026-06-26 09:51:05 +08:00
Alex Lyn
2dd9426029 kata-deploy: Add erofsSnapshotterMode helm value and integrity mode
Expose erofsSnapshotterMode in the helm chart values and render it as
the EROFS_SNAPSHOTTER_MODE environment variable in the kata-deploy pod.

Update gha-run-k8s-common.sh to load dm-mod/dm-verity kernel modules
and configure the erofs default size when the mode is "integrity".

Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
2026-06-26 09:51:05 +08:00
Fabiano Fidêncio
a664595084 kata-deploy: bump qemu RuntimeClass overhead for the aarch64 VMM
With sandbox_cgroup_only the shim, QEMU and virtiofsd run inside the
pod's memory cgroup, whose limit is the workload limit plus the
RuntimeClass pod overhead. On aarch64 the VMM host footprint is much
larger than on x86 (QEMU's own anon RSS is ~160Mi+ before any guest
RAM, on top of the shmem-backed guest memory), so the 160Mi overhead
is too small: small-memory-limit pods get their qemu-system process
OOM-killed by the pod cgroup (CONSTRAINT_MEMCG), and the agent vsock
never comes up (ENODEV), so the sandbox fails to start.

Raise the pod overhead to 320Mi for the qemu shims that run on
aarch64 (qemu, qemu-runtime-rs, qemu-coco-dev-runtime-rs). The value
is applied on all architectures for simplicity; x86 is over-provisioned
by ~160Mi, which is acceptable. The TEE/GPU shims already carry far
larger overhead and amd64-only shims (clh*, dragonball, fc) are
unaffected.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-25 13:56:11 +02:00
Aurélien Bombo
1217dd1584 Merge pull request #12373 from kata-containers/disable-guest-empty-dir
runtime: Set `disable_guest_empty_dir = true` by default
2026-06-24 20:09:46 -05:00
Aurélien Bombo
10cf6816aa kernel: Fix FUSE crash with host emptyDir
This patch was submitted by Miklos Szeredi:
https://lore.kernel.org/fuse-devel/20260528142306.1792392-1-mszeredi@redhat.com/

It fixes a FUSE oops with the k8s-shared-volume.bats test.

Fixes: #12589

Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
2026-06-24 15:22:13 -05:00
LandonTClipp
85e828cc9b docs: Add AI agent skill for doc contributions
This skill will inform AI agents how to properly write and format
docs in the new docs system. There is nothing too fancy, just reminding
agents to use mkdocs-materialx features instead of treating the
markdown like the legacy Github-based format.

Signed-off-by: LandonTClipp <lclipp@coreweave.com>
2026-06-23 08:57:37 +01:00
Fabiano Fidêncio
9761ea2235 Merge pull request #13164 from manuelh-dev/mahuber/remove-resource-requests
tests: use limits for Kata workload resources
2026-06-22 20:01:33 +02:00
Fabiano Fidêncio
f1ebefcdfb Merge pull request #13222 from fidencio/topic/nvidia-switch-to-kata-deploy-jobs
kata-deploy: nvidia: Default to the Job-based deployment mode
2026-06-22 12:55:10 +02:00
Fabiano Fidêncio
dc70b93573 release: Bump version to 3.32.0
Bump VERSION and helm-charts versions.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-06-22 01:15:24 +02:00
Fabiano Fidêncio
374a867774 Merge pull request #13196 from microsoft/cameronbaird/upstream/runtime-go-clh-templating
runtime: Enable VM Templating Support for CLH
2026-06-21 16:31:19 +02:00
Cameron Baird
65a5f272f8 ci: Introduce tests for VM template factory
Add k8s-vm-templating-test.bats which exercises pod create
with the factory initialized on the target node.

Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
2026-06-19 18:00:02 +00:00
Manuel Huber
aafd16515c tests: use limits for Kata workload manifests
Kata sizes VM CPU and memory from OCI limits, not Kubernetes resource
requests. Requests are consumed by the Kubernetes control plane, but
they do not drive Kata VM or sandbox sizing today.

Convert the straightforward Kata workload manifests and kata-deploy
examples from resource requests to limits so the declared resources
match the values Kata uses for VM provisioning. Keep requests where the
fixture intentionally validates Kubernetes request/limit behavior.

Update fixture expectations affected by the conversion. The LimitRange
fixture is limit-only at 500m.

Raise the policy deployment limits to 500m and 800Mi. These tests boot
CoCo/runtime-rs sandboxes with policy/initdata, and the former
100m/100Mi values became real runtime limits after the conversion,
which is too constrained for the CI environments.

Leave PVC storage requests, explicit request/limit validation fixtures,
the env resourceFieldRef request, and non-Kata workload examples
unchanged where requests are handled outside the Kata shim resource
sizing path.

If Kata later grows request-aware sandbox sizing, for example through
Sandbox API based resource plumbing, these requests can be reintroduced
where they carry the intended semantics.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
2026-06-19 09:38:15 -07:00
Greg Kurz
c3d98fe323 osbuilder: Simplify version fetching
`tools/osbuilder/VERSION` points to the root `VERSION` file,
just like the code does. Use that file.

Signed-off-by: Greg Kurz <groug@kaod.org>
2026-06-17 10:08:23 +02:00
Fabiano Fidêncio
5959549645 release: do not publish a kata-monitor-job-dispatcher manifest
The shared _publish_multiarch_manifest() helper always derived a
"-job-dispatcher" registry from the registries it was given. However, the
dispatcher is a kata-deploy-specific sidecar image, so when the helper
was reused to publish the kata-monitor multi-arch manifest it wrongly
tried to push a non-existent kata-monitor-job-dispatcher image.

Let's gate the dispatcher derivation behind
KATA_DEPLOY_PUBLISH_JOB_DISPATCHER (defaulting to true so the
kata-deploy path is unchanged) and opt out of it when publishing the
kata-monitor manifest.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-15 16:29:29 +02:00
Hyounggyu Choi
59fd29fb33 Merge pull request #13225 from BbolroC/use-tar-in-exporting-kate-deploy-files
packaging: Optimize kata-deploy build export using tar output
2026-06-15 14:47:01 +02:00
Hyounggyu Choi
46b8e9f027 packaging: Optimize kata-deploy build export using tar output
Replace `type=local` with `type=tar` in kata-deploy build to reduce
export time and avoid build hangs during the export-to-client-directory
phase.

Update callers to extract binaries directly from the tar archive instead
of copying from an intermediate directory.

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
2026-06-15 11:52:33 +02:00
Steve Horsman
ea999aa033 Merge pull request #13221 from manuelh-dev/mahuber/nydus-root-export
kata-deploy: export nydus snapshotter root
2026-06-15 08:55:08 +01:00
Fabiano Fidêncio
fefc0b75ab kata-deploy: nvidia: Default to the Job-based deployment mode
Switch the NVIDIA GPU example values file to install Kata via the
Job-based deployment mode (deploymentMode: job) instead of the
always-on, privileged DaemonSet, so that nothing keeps running on the
node once the install completes.

To exercise this in our CI, make the helm_helper aware of the deployment
mode coming from the (base) values file:

  - In "job" mode, clear job.nodeSelectorExpressions so the dispatcher
    targets every discovered node.  Our CI clusters are typically
    single-node, where the only node carries the control-plane label,
    and the default selector excludes control-plane/master nodes.

  - There is no always-on DaemonSet to wait on in "job" mode.  The
    dispatcher runs as a blocking post-install hook and the final
    per-node stage labels the node, so wait until at least one node
    carries the katacontainers.io/kata-runtime label as the
    "install complete" signal (dumping Job/pod logs on timeout).

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-12 22:55:11 +02:00
Manuel Huber
639420e7f5 kata-deploy: export nydus snapshotter root
containerd uses the proxy plugin root export when reporting CRI image
filesystem paths. Without this export, the CRI plugin falls back to
/var/lib/containerd/io.containerd.snapshotter.v1.<snapshotter>.

For nydus-for-kata-tee this fallback does not match the actual
snapshotter root under /var/lib/nydus-for-kata-tee.
Kubelet/cAdvisor then fails stats collection when it tries to inspect
the nonexistent fallback path.

Export the nydus proxy snapshotter root so containerd reports the real
filesystem path for resource accounting.

When using trusted ephemeral storage or a new ephemeral storage wip
feature for providing plain disks, resource accounting would not kick
in and pods which exhausted their emptyDir sizeLimits would not get
evicted.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
2026-06-12 19:06:01 +02:00
Fabiano Fidêncio
aa27490801 kata-deploy: track distroless static base by tag, not digest
The kata-deploy main image pinned its gcr.io/distroless/static-debian13
base by sha256 digest. distroless does not publish versioned tags, so a
pinned digest just goes stale with no clear upgrade path. Track the
rolling tag instead (guarded with a hadolint DL3007 ignore plus a comment
explaining why), matching the kata-deploy-job-dispatcher image base.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-12 18:58:33 +02:00
Fabiano Fidêncio
c23fe11529 kata-deploy: make verification Job aware of job deployment mode
The verification Job assumed the DaemonSet model: it waited for the
DaemonSet to exist, for its pods, and for `rollout status daemonset/...`,
then required every node in the cluster to be labeled. None of that holds
for deploymentMode: job, where install happens via the dispatcher and the
per-node Jobs it fans out, and only the targeted (worker) nodes get
labeled.

Make the hook mode-aware:
  - Hook weight: in job mode the install dispatcher runs as a
    post-install hook at weight 5, so verification now runs at weight 10
    (after it); daemonset mode keeps weight 0 (the DaemonSet is a normal
    resource).
  - Readiness wait: in job mode, wait for the install dispatcher Job to
    complete and then for the per-node install Jobs
    (kata-deploy/stage=install) to finish (with the same CRI-restart
    retry logic) instead of a DaemonSet rollout.
  - Label check: in job mode, verify exactly the nodes the dispatcher
    targeted are labeled, rather than comparing the labeled count against
    all nodes in the cluster.
  - Grant the verification ClusterRole read access to batch/jobs (used by
    the job-mode waits; harmless in daemonset mode).

The daemonset code path is unchanged and the default render (no
verification.pod) is byte-for-byte identical.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-12 18:58:33 +02:00
Fabiano Fidêncio
3d732986d2 kata-deploy: add per-node staged cleanup for job mode
Add the uninstall counterpart to the install dispatcher for
deploymentMode: job. On `helm uninstall`, a single pre-delete hook Job
runs the kata-deploy-job-dispatcher, which enumerates the targeted nodes
live and fans out one node-pinned cleanup Job per node that runs the
install pipeline in reverse and exits:

  unlabel -> revert-cri   (initContainers, run sequentially)
  remove-artifacts        (main container)

Running as a pre-delete hook means the dispatcher ServiceAccount/RBAC and
the kata-deploy host-mutation RBAC still exist while the Jobs run, so the
unlabel stage retains node get/patch access. revert-cri and
remove-artifacts are host-only operations (privileged nsenter / host
mount) and need no extra cluster RBAC.

Ordering mirrors install in reverse: unlabel first so the scheduler stops
placing kata workloads here, then revert the CRI config + restart the
runtime, then remove the on-host artifacts. Each stage is idempotent and
skips when already undone, so partially-installed nodes and re-runs are
safe.

Uninstall node selection is deliberately SEPARATE from install (a
dedicated job.cleanup.* block) and defaults to every node carrying the
katacontainers.io/kata-runtime label (set by the install label stage)
rather than re-evaluating the install selector. Because the cleanup
dispatcher resolves nodes live when it runs, this stays robust to
install-time selector drift (relabeled nodes, etc.) while remaining fully
overridable via job.cleanup.nodes / job.cleanup.nodeSelector /
job.cleanup.nodeSelectorExpressions. The default (daemonset) mode is
unaffected.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-12 18:58:33 +02:00
Fabiano Fidêncio
54878fa373 kata-deploy: add job deployment mode driven by the job-dispatcher
Phase 2 of the DaemonSet -> staged-Job migration: add an opt-in
`deploymentMode: job` that installs Kata via short-lived, per-node
install Jobs instead of the long-running DaemonSet. The DaemonSet remains
the default and is now gated behind `deploymentMode == daemonset`.

Rather than render one Job per node into the Helm release (which grows
the release secret O(nodes) and offers no rollout pacing), job mode ships
a single tiny post-install/post-upgrade hook Job that runs the
kata-deploy-job-dispatcher. The dispatcher enumerates the selected nodes
LIVE from the API server and stamps out one node-pinned install Job per
node from a constant-size ConfigMap of Job templates, keeping at most
`job.parallelism` in flight and refilling as they finish. This guarantees
per-node coverage with a paced rollout while the Helm release stays O(1)
regardless of fleet size. New nodes are picked up by re-running
`helm upgrade`; there is no always-on component.

Each per-node Job runs the staged install pipeline as ordered
initContainers and exits:

  host-check -> artifacts -> cri   (initContainers, run sequentially)
  label                            (main container)

The privilege split is explicit: the dispatcher pod is a pure
control-plane client (lists nodes, manages Jobs in its own namespace) and
runs fully unprivileged under a dedicated, least-privilege ServiceAccount
(kata-rbac.yaml); only the per-node Jobs it creates carry the privileged
kata-deploy host-mutation rights.

Node selection (templates/_helpers.tpl: nodeLabelSelector / perNodeJob):
  - job.nodes: explicit node-name list passed to the dispatcher, and
  - job.nodeSelector (equality map) ANDed with
  - job.nodeSelectorExpressions (k8s label-selector requirements:
    In / NotIn / Exists / DoesNotExist),
compiled into a single label-selector string the dispatcher resolves
live. The default expressions target worker (non-control-plane) nodes, so
no custom node labeling is required; set the expressions to [] to target
all discovered nodes.

Reuses the commonEnv/commonVolume* helpers and adds the stageContainer,
serviceAccountName, dispatcherServiceAccountName, dispatcherImage and
perNodeJob helpers shared by the dispatcher and the staged Jobs. The
default (daemonset) render is unchanged.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-12 18:58:33 +02:00
Fabiano Fidêncio
28fce44b70 kata-deploy: extract shared pod env/volumes into helm helpers
Pull the kata-deploy container's environment block and host
volume/volumeMount definitions out of the DaemonSet template into
reusable named templates in _helpers.tpl:

  - kata-deploy.commonEnv
  - kata-deploy.commonVolumeMounts
  - kata-deploy.commonVolumes

These are derived purely from chart values and are independent of the
deployment model, so they can be shared verbatim by upcoming per-node
install/cleanup Jobs without duplicating the (large) env wiring.

Pure refactor: the rendered DaemonSet is byte-for-byte identical to
before (verified via normalized `helm template` diff across default and
multiInstallSuffix/userDropIn/customRuntimes permutations).

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
2026-06-12 18:58:33 +02:00