diff --git a/docs/design/proposals/composable-vm-images.md b/docs/design/proposals/composable-vm-images.md index 4a34eddfbf..1535ab35a8 100644 --- a/docs/design/proposals/composable-vm-images.md +++ b/docs/design/proposals/composable-vm-images.md @@ -10,18 +10,18 @@ This proposal introduces a **composable image architecture** for Kata Containers guest VMs. Instead of building a single monolithic rootfs image that contains every component a workload might need, the runtime assembles a VM from a lean -**base image** plus zero or more purpose-specific **addon images** that are -cold-plugged as additional virtio-blk devices. Each addon is an EROFS image +**base image** plus zero or more purpose-specific **guest extension images** that are +cold-plugged as additional virtio-blk devices. Each extension is an EROFS image protected by dm-verity, mounted read-only inside the guest before the kata-agent starts. The first application of this architecture splits **Confidential Containers (CoCo) guest components** out of the monolithic `kata-containers-confidential.img` -into a separate `kata-containers-coco-addon.img`. +into a separate `kata-containers-coco-extension.img`. The current proposal targets **QEMU** as the hypervisor backend. The design is intentionally hypervisor-agnostic at the configuration and guest -layers — the `extra_images` configuration, guest-side systemd units, and +layers — the `guest_extension_images` configuration, guest-side systemd units, and agent path resolution work identically regardless of the hypervisor. Only the host-side device attachment is hypervisor-specific (QEMU virtio-blk cold-plug today), and extending support to other hypervisor backends @@ -50,7 +50,7 @@ pause bundle). This creates several problems: must maintain their own image build pipeline. A composable approach addresses all four issues: the base image stays small and -generic, addon images are independently built and versioned, and the runtime +generic, guest extension images are independently built and versioned, and the runtime composes them at VM creation time. ## Design @@ -58,28 +58,30 @@ composes them at VM creation time. ### Architecture overview ``` -Host Guest VM -┌──────────────────────┐ ┌─────────────────────────────────┐ -│ Runtime config │ │ / (base rootfs, erofs) │ -│ ┌────────────────┐ │ │ kata-agent, systemd, ... │ -│ │ image = base │ │ boot │ │ -│ │ extra_images: │──────────>│ /run/kata-addons/coco/ (erofs) │ -│ │ - name: coco │ │ cold │ attestation-agent │ -│ │ path: ... │ │ plug │ confidential-data-hub │ -│ │ verity: ... │ │ │ api-server-rest │ -│ └────────────────┘ │ │ ocicrypt_config.json │ -│ │ │ pause_bundle/ │ -│ QEMU │ │ │ -│ -drive base.img │ │ dm-verity protects both the │ -│ -drive coco.img │ │ base rootfs and the addon. │ -│ serial=addon-… │ └─────────────────────────────────┘ -└──────────────────────┘ +Host Guest VM +┌───────────────────────────────┐ +│ Runtime config │ +│ ┌─────────────────────────┐ │ +│ │ image = base │ │ ┌─────────────────────────────────────┐ +│ │ guest_extension_images: │──┼─────>│ / (base rootfs, erofs) │ +│ │ - name: coco │ │ boot │ kata-agent, systemd, ... │ +│ │ path: ... │ │ cold │ │ +│ │ verity: ... │ │ plug │ /run/kata-extensions/coco/ (erofs) │ +│ └─────────────────────────┘ │ │ attestation-agent │ +│ │ │ confidential-data-hub │ +│ QEMU │ │ api-server-rest │ +│ -drive base.img │ │ ocicrypt_config.json │ +│ -drive coco.img │ │ pause_bundle/ │ +│ serial=extension-coco │ │ │ +└───────────────────────────────┘ │ dm-verity protects both the │ + │ base rootfs and the extension. │ + └─────────────────────────────────────┘ ``` ### Runtime configuration -A new `extra_images` field on the hypervisor configuration accepts an ordered -list of addon images: +A new `guest_extension_images` field on the hypervisor configuration accepts an ordered +list of guest extension images: ```toml [hypervisor.qemu] @@ -87,64 +89,78 @@ image = "/opt/kata/share/kata-containers/kata-containers.img" # ...existing keys... -[[hypervisor.qemu.extra_images]] +[[hypervisor.qemu.guest_extension_images]] name = "coco" -path = "/opt/kata/share/kata-containers/kata-containers-coco-addon.img" +path = "/opt/kata/share/kata-containers/kata-containers-coco-extension.img" verity_params = "root_hash=abc...,salt=def...,data_blocks=1234,hash_block_size=4096,data_block_size=4096" ``` Each entry maps to a Rust struct: ```rust -pub struct ExtraImage { +pub struct GuestExtensionImage { pub name: String, pub path: String, pub verity_params: String, } ``` -The `name` field is the primary identifier for an addon. It must be unique +The `name` field is the primary identifier for an extension. It must be unique within the configuration and is used to: -- Set the **virtio-blk serial** to `addon-`, enabling deterministic +- Set the **virtio-blk serial** to `extension-`, enabling deterministic device discovery in the guest via `/sys/block/*/serial`. - Namespace the **kernel command-line** verity parameters as - `kata.addon..verity_params=...`. -- Name the guest **mount point** at `/run/kata-addons//`. -- Name the **dm-verity device mapper target** as `addon-`. + `kata.extension..verity_params=...`. +- Name the guest **mount point** at `/run/kata-extensions//`. +- Name the **dm-verity device mapper target** as `extension-`. -### Host-side: cold-plugging addon devices +### Host-side: cold-plugging extension devices -At VM creation time, for each entry in `extra_images`, the runtime: +At VM creation time, for each entry in `guest_extension_images`, the runtime: -1. Creates a virtio-blk device backed by the addon image file. -2. Sets the device serial to `addon-`. -3. If `verity_params` is non-empty, appends - `kata.addon..verity_params=` to the guest kernel command line. +1. Creates a virtio-blk device backed by the guest extension image file. + Extensions are **always** attached as virtio-blk, using the architecture's + virtio-blk transport (`virtio-blk-ccw` on s390x, `virtio-blk-pci` + elsewhere). Neither the VM rootfs driver (`vm_rootfs_driver`) nor the + generic block device driver (`block_device_driver`) is reused here: those + may resolve to a non-virtio-blk transport such as `virtio-pmem` (NVDIMM) or + `virtio-scsi`, and only virtio-blk devices carry the serial the guest relies + on for discovery (step 2), so a non-virtio-blk transport would leave the + extension undiscoverable and its mount unit would fail closed. +2. Sets the device serial to `extension-`. +3. Appends `kata.extension..verity_params=` to the guest kernel + command line for *every* configured extension. The value is empty for an + unmeasured extension (`verity_params = ""`, e.g. on s390x — see "Mount + script" below); the entry is emitted unconditionally because it doubles as + the guest-side activation signal for the mount unit. With an empty value the + entry renders as a bare `kata.extension..verity_params` (no `=`), which + the unit condition and generator both handle. Both the Go runtime and the Rust runtime-rs implement this logic in their -respective QEMU hypervisor backends. The addon devices are **cold-plugged** — +respective QEMU hypervisor backends. The extension devices are **cold-plugged** — they are present on the QEMU command line at VM start, not hot-plugged later. Other hypervisor backends (Cloud Hypervisor, Firecracker) do not implement -addon cold-plugging yet, but the mechanism is straightforward to add: each -backend only needs to translate the `ExtraImage` entries into its native +extension cold-plugging yet, but the mechanism is straightforward to add: each +backend only needs to translate the `GuestExtensionImage` entries into its native block device attachment API. The runtime configuration, guest-side mounting, and agent path resolution are completely hypervisor-independent. -### Guest-side: mounting addons +### Guest-side: mounting extensions -A systemd template unit `kata-addon-mount@.service` handles addon discovery and -mounting. The unit is instantiated per addon name (e.g. -`kata-addon-mount@coco.service`). +A systemd template unit `kata-extension-mount@.service` handles extension discovery and +mounting. The unit is instantiated per extension name (e.g. +`kata-extension-mount@coco.service`); a systemd generator (described below) +creates those instances automatically from the kernel command line. The mounting is performed by a systemd unit (before the agent starts) rather than by the kata-agent itself for several reasons: -- **Chicken-and-egg problem** — in the CoCo addon case, the addon carries +- **Chicken-and-egg problem** — in the CoCo extension case, the extension carries the very guest components (attestation-agent, confidential-data-hub) that - the kata-agent needs to launch. The agent cannot mount the addon because - it needs the addon's contents to be available before it starts. + the kata-agent needs to launch. The agent cannot mount the extension because + it needs the extension's contents to be available before it starts. - **Init-system ordering guarantees** — systemd provides declarative ordering (`Before=`, `After=`), conditional activation @@ -160,7 +176,7 @@ than by the kata-agent itself for several reasons: - **Non-systemd environments** — the design does not mandate systemd. In environments using a different init system (or a dedicated orchestrator like NVRC), the equivalent mounting logic can be implemented by whatever - entity manages early boot. The key requirement is that addon images are + entity manages early boot. The key requirement is that guest extension images are mounted before the kata-agent starts — how that is achieved is an init-system concern. @@ -168,18 +184,18 @@ than by the kata-agent itself for several reasons: ```ini [Unit] -Description=Mount Kata addon image %i +Description=Mount Kata guest extension image %i DefaultDependencies=no Before=kata-agent.service After=local-fs-pre.target -ConditionKernelCommandLine=kata.addon.%i.verity_params +ConditionKernelCommandLine=kata.extension.%i.verity_params OnFailure=poweroff.target [Service] Type=oneshot RemainAfterExit=yes -ExecStart=/usr/libexec/kata-addon-mount.sh %i -ExecStop=/usr/libexec/kata-addon-umount.sh %i +ExecStart=/usr/libexec/kata-extension-mount.sh %i +ExecStop=/usr/libexec/kata-extension-umount.sh %i [Install] WantedBy=kata-containers.target @@ -188,28 +204,50 @@ WantedBy=kata-containers.target Key design decisions: - **`ConditionKernelCommandLine`** — the service only activates when the - runtime has actually configured the addon. This prevents the unit from - running (and failing) in non-addon VM configurations. + runtime has actually configured the extension. This prevents the unit from + running (and failing) in non-extension VM configurations. -- **`Before=kata-agent.service`** — guarantees the addon filesystem is mounted +- **`Before=kata-agent.service`** — guarantees the extension filesystem is mounted before the agent attempts to use any component from it. -- **`OnFailure=poweroff.target`** — if the addon cannot be mounted (e.g. +- **`OnFailure=poweroff.target`** — if the extension cannot be mounted (e.g. verity verification failure, missing device), the VM is shut down immediately. A confidential VM must not continue with an unverified or - missing addon. + missing extension. + +#### Enabling instances + +> **Implementation note.** An earlier revision enabled a fixed +> `kata-extension-mount@coco.service` symlink at rootfs build time, which would +> have required editing the image build for every new extension. Review feedback +> pushed us to the generator below so the build stays extension-agnostic. + +systemd template units must be enabled per instance, and the set of extensions +is only known at runtime (from the kernel command line). A systemd generator, +`kata-extension-mount-generator`, bridges that gap: it runs in early boot, scans +`/proc/cmdline` for every `kata.extension..verity_params` entry (whether it +carries a value or, for an unmeasured extension, is a bare key), and symlinks +`kata-extension-mount@.service` into `kata-containers.target.wants`. + +Because the runtime emits exactly one such cmdline entry per configured +`guest_extension_images`, the generator enables precisely the extensions the VM +was configured with. Adding a new extension therefore requires no change to the +rootfs build — it is wired up entirely from runtime configuration. The generator +ships in the base image (installed alongside the template unit by the agent), so +it is part of the dm-verity-measured rootfs, and the cmdline it reads is itself +covered by the guest launch measurement. #### Mount script -The mount script (`kata-addon-mount.sh`) performs the following steps: +The mount script (`kata-extension-mount.sh`) performs the following steps: 1. **Device discovery** — scans `/sys/block/*/serial` for a device whose - serial matches `addon-`. This is more reliable than waiting for udev + serial matches `extension-`. This is more reliable than waiting for udev to create `/dev/disk/by-id/` symlinks, since the minimal guest environment may not run a full udev daemon. 2. **Verity parameter extraction** — reads - `kata.addon..verity_params=...` from `/proc/cmdline` and parses the + `kata.extension..verity_params=...` from `/proc/cmdline` and parses the comma-separated key=value pairs (root_hash, salt, data_blocks, hash_block_size, data_block_size). @@ -219,24 +257,58 @@ The mount script (`kata-addon-mount.sh`) performs the following steps: creation — all verity parameters are passed explicitly rather than stored in an on-disk superblock. -4. **EROFS mount** — mounts the verified device (or the raw partition if - verity is not configured) read-only at `/run/kata-addons//`. +4. **EROFS mount** — mounts the resulting device read-only at + `/run/kata-extensions//` (the dm-verity target when verified, or the + data partition directly for an unmeasured extension). + +##### Integrity policy: measured vs. unmeasured, and failing closed + +An extension can legitimately ship *without* dm-verity — for example on s390x, +where IBM Secure Execution protects the guest through a different mechanism and +images are built with `MEASURED_ROOTFS=no`. The mount script must therefore +support a raw (unverified) mount, but it must **not** let that path become a +silent downgrade: an attacker who can edit the (host-supplied) kernel command +line could otherwise strip `verity_params` from a *measured* extension and have +it mounted unverified. + +The script separates these two cases explicitly, using the **on-disk layout as +the source of truth** rather than trusting the cmdline alone. The image build +encodes its integrity policy in the partition table (see "Image build" below): +a measured extension carries a dm-verity **hash partition** (`p2`) next to the +data partition (`p1`), while an unmeasured extension has only `p1`. The script +detects the presence of the hash device and cross-checks it against the +`verity_params` carried on the kernel command line (which, in a confidential +guest, is itself part of the measured, attested boot): + +| Hash device (`p2`) | `verity_params` on cmdline | Action | +|--------------------|----------------------------|----------------------------------------------------| +| present | present | **verify** with dm-verity, then mount (normal case)| +| present | absent / empty | **refuse** — verity was stripped/disabled (tamper) | +| absent | present | **refuse** — params but nothing to verify (mismatch)| +| absent | absent / empty | **raw mount** — genuinely unmeasured extension | + +Any "refuse" path exits non-zero; combined with `OnFailure=poweroff.target` on +the mount unit, that powers the VM off rather than continuing with an unverified +or inconsistent extension. The defence has two layers: the cmdline (and thus the +root hash, or its deliberate absence) is covered by the guest launch +measurement and remote attestation, and the in-guest layout cross-check fails +closed so a measured extension can never be silently downgraded to a raw mount. ### Agent-side: data-driven component manifest > **Implementation note.** An earlier revision of this proposal described the > agent as resolving a small, hardcoded list of component paths. While -> implementing and testing the addon we found this too rigid: the addon +> implementing and testing the extension we found this too rigid: the extension > evolved to ship multiple attestation-agent flavours, per-process environment > requirements, and ordering constraints that the agent should not have to know -> about. The design below — a **data-driven manifest** owned by the addon — is +> about. The design below — a **data-driven manifest** owned by the extension — is > what we converged on so that adding or reconfiguring a bundled component > requires **no** kata-agent code change. -Each addon ships a manifest at `etc/kata-addons/components.toml`. When the addon +Each extension ships a manifest at `etc/kata-extensions/components.toml`. When the extension is mounted, the kata-agent reads it and builds its launch plan from it. The manifest declares the processes to launch and the resources they expose; all -paths are **relative to the addon mount root** (`/run/kata-addons/`). +paths are **relative to the extension mount root** (`/run/kata-extensions/`). A process entry carries: @@ -252,7 +324,7 @@ A process entry carries: `${...}` tokens in `args`, `config`, `env` values, and variant fields are substituted by the agent from a runtime context it assembles (socket and config -paths, the addon mount root `${addon_root}`, the selected `${attester_variant}`, +paths, the extension mount root `${extension_root}`, the selected `${attester_variant}`, etc.). Introducing a brand-new variable is the only change that ever needs to touch the agent. @@ -280,14 +352,14 @@ select = "${attester_variant}" [process.variants.nvidia] path = "usr/local/bin/attestation-agent-nv" - env = { LD_LIBRARY_PATH = "${addon_root}/usr/local/lib:/run/kata-addons/gpu/usr/lib" } + env = { LD_LIBRARY_PATH = "${extension_root}/usr/local/lib:/run/kata-extensions/gpu/usr/lib" } [[process]] id = "confidential-data-hub" level = 2 path = "usr/local/bin/confidential-data-hub" config = "${cdh_config_path}" -env = { OCICRYPT_KEYPROVIDER_CONFIG = "${ocicrypt_config_path}", PATH = "${addon_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin" } +env = { OCICRYPT_KEYPROVIDER_CONFIG = "${ocicrypt_config_path}", PATH = "${extension_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin" } wait_socket = "${cdh_socket}" [[process]] @@ -297,34 +369,34 @@ path = "usr/local/bin/api-server-rest" args = ["--features", "${rest_api_features}"] ``` -When **no** addon is mounted, the agent falls back to a built-in launch plan +When **no** extension is mounted, the agent falls back to a built-in launch plan that reproduces the legacy behaviour (components launched from `/usr/local/bin/...` in the rootfs). The same dual-path principle applies to the non-process resources declared under `[paths]`: the agent resolves them -inside the addon first and falls back to the legacy location otherwise: +inside the extension first and falls back to the legacy location otherwise: -| Resource | Addon path | Legacy path | +| Resource | Extension path | Legacy path | |------------------------|---------------------------------------------------------|---------------------------------| -| attestation-agent(-nv) | `/run/kata-addons/coco/usr/local/bin/attestation-agent[-nv]` | `/usr/local/bin/attestation-agent` | -| confidential-data-hub | `/run/kata-addons/coco/usr/local/bin/confidential-data-hub` | `/usr/local/bin/confidential-data-hub` | -| api-server-rest | `/run/kata-addons/coco/usr/local/bin/api-server-rest` | `/usr/local/bin/api-server-rest` | -| ocicrypt_config.json | `/run/kata-addons/coco/etc/ocicrypt_config.json` | `/etc/ocicrypt_config.json` | -| pause_bundle | `/run/kata-addons/coco/pause_bundle` | `/pause_bundle` | +| attestation-agent(-nv) | `/run/kata-extensions/coco/usr/local/bin/attestation-agent[-nv]` | `/usr/local/bin/attestation-agent` | +| confidential-data-hub | `/run/kata-extensions/coco/usr/local/bin/confidential-data-hub` | `/usr/local/bin/confidential-data-hub` | +| api-server-rest | `/run/kata-extensions/coco/usr/local/bin/api-server-rest` | `/usr/local/bin/api-server-rest` | +| ocicrypt_config.json | `/run/kata-extensions/coco/etc/ocicrypt_config.json` | `/etc/ocicrypt_config.json` | +| pause_bundle | `/run/kata-extensions/coco/pause_bundle` | `/pause_bundle` | This approach: - Preserves **backward compatibility** with existing monolithic rootfs images where CoCo components are baked into the base image. - Requires **no special rootfs modifications** — the base image does not need - stub files or directories for the addon components. + stub files or directories for the extension components. - Works transparently on a **read-only rootfs** — no bind-mounting, no remounting, no writes to the root filesystem. -- Keeps the agent **generic** — addon-specific names, env, and ordering live +- Keeps the agent **generic** — extension-specific names, env, and ordering live in the manifest, not in agent code. ### Attester variant selection and the NVRC contract -The CoCo addon ships **two** attestation-agent builds: the stock +The CoCo extension ships **two** attestation-agent builds: the stock `attestation-agent` and an NVIDIA-attester build, `attestation-agent-nv`, that collects GPU evidence in addition to the TEE evidence. Which one runs is chosen by the manifest's `select = "${attester_variant}"` selector, and the value of @@ -333,7 +405,7 @@ by the manifest's `select = "${attester_variant}"` selector, and the value of - On a plain confidential guest the kata-agent runs init itself and the variable defaults to `default` → the stock attester launches. - On a GPU guest, **NVRC** (the NVIDIA runtime config that owns early boot) - detects the GPU addon and exports `KATA_ATTESTER_VARIANT=nvidia` before + detects the GPU extension and exports `KATA_ATTESTER_VARIANT=nvidia` before exec'ing the kata-agent. The agent forwards that into the substitution context as `${attester_variant}`, so the `nvidia` variant launches. @@ -343,28 +415,28 @@ consumed by the kata-agent, which keeps the agent free of any GPU- or NVIDIA-specific knowledge — it only knows how to forward a selector into the manifest. We arrived at this split after first trying to special-case the attester inside the agent; pushing the decision out to NVRC + the manifest kept -both the agent and the addon generic. +both the agent and the extension generic. -#### Why one addon with two builds (and not two addons) +#### Why one extension with two builds (and not two extensions) -Shipping **two** attestation-agent builds inside a **single** CoCo addon is a -deliberate choice, and it is worth being explicit about it because addons are +Shipping **two** attestation-agent builds inside a **single** CoCo extension is a +deliberate choice, and it is worth being explicit about it because extensions are otherwise meant to *eliminate* duplication. -- A single CoCo addon serves **both** plain confidential guests (TEE evidence +- A single CoCo extension serves **both** plain confidential guests (TEE evidence only) and confidential GPU guests (TEE + GPU evidence). The only thing that differs between them is which attestation-agent binary runs; everything else - in the addon (confidential-data-hub, api-server-rest, ocicrypt config, pause + in the extension (confidential-data-hub, api-server-rest, ocicrypt config, pause bundle, `cryptsetup`) is shared verbatim. Splitting the NVIDIA attester into - its own addon would duplicate that shared payload and force every confidential - GPU guest to compose two CoCo-flavoured addons instead of one. + its own extension would duplicate that shared payload and force every confidential + GPU guest to compose two CoCo-flavoured extensions instead of one. - The **cost** of keeping both builds together is precisely the manifest's `select`/`variants` machinery: the manifest has to be aware that the attestation-agent comes in two flavours and pick one at runtime. We consider that a fair trade — the complexity is confined to data (the manifest), the - agent stays generic, and the addon stays a single, coherent "CoCo" unit. -- A separate addon only pays off when its contents are **substantially** - different (e.g. the GPU addon, which carries the driver userspace), not for + agent stays generic, and the extension stays a single, coherent "CoCo" unit. +- A separate extension only pays off when its contents are **substantially** + different (e.g. the GPU extension, which carries the driver userspace), not for two near-identical builds of the same component. No further attester variants are planned today, but if one appeared it would be another `[process.variants.]` entry — not a new image. @@ -382,25 +454,25 @@ physically live. turn: - `dlopen`s `libnvidia-ml.so.1` (NVML) at runtime to gather GPU evidence. NVML - is part of the **GPU addon**, mounted at `/run/kata-addons/gpu` with its driver + is part of the **GPU extension**, mounted at `/run/kata-extensions/gpu` with its driver libraries under `usr/lib`. - pulls in a closure of non-glibc libraries (libxml2, zlib, lzma, the C++ runtime, ...) that the guest rootfs does not otherwise ship. Neither set is present in a stock guest, so: -- The CoCo addon build bundles `libnvat.so` **and every non-glibc transitive +- The CoCo extension build bundles `libnvat.so` **and every non-glibc transitive dependency** next to it under `usr/local/lib`. - The `nvidia` **manifest variant** (the `[process.variants.nvidia]` entry, not a separate image) sets an `LD_LIBRARY_PATH` that lists **both** the CoCo - addon's `usr/local/lib` (for `libnvat` and its closure) **and** the GPU - addon's `usr/lib` (for NVML). Setting only the first was the cause of an + extension's `usr/local/lib` (for `libnvat` and its closure) **and** the GPU + extension's `usr/lib` (for NVML). Setting only the first was the cause of an `NVAT Error 500: NVML Initialization Failed` we hit during bring-up; the RCAR handshake then never produced GPU evidence and attestation failed. Because the agent applies manifest `env` on top of the inherited environment (without clearing it), and because the `nvidia` variant only ever runs when the -GPU addon is present, referencing the GPU addon's well-known mount path here is +GPU extension is present, referencing the GPU extension's well-known mount path here is safe. #### CDH secure-mount tooling (encrypted vs plain storage) @@ -415,17 +487,17 @@ in two different places: unconditionally. - **Encrypted storage** — `cryptsetup` LUKS-formats and opens the volume. This is a CoCo-only capability, so it belongs with the CoCo guest components in the - **coco addon**. + **coco extension**. -This is the same `veritysetup`-vs-`cryptsetup` reasoning already applied to addon -mounting: the base must always carry `veritysetup` (it opens *every* addon as a +This is the same `veritysetup`-vs-`cryptsetup` reasoning already applied to extension +mounting: the base must always carry `veritysetup` (it opens *every* extension as a dm-verity device before mounting), and `cryptsetup` shares an identical shared-library closure, so wherever `veritysetup` lives the libraries for `cryptsetup` are already present. Because CDH runs in the **base rootfs namespace** but `cryptsetup` lives in the -addon (which is not on the default search path), the manifest sets CDH's `PATH` -to `${addon_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin` — the addon's +extension (which is not on the default search path), the manifest sets CDH's `PATH` +to `${extension_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin` — the extension's `cryptsetup` first, then the base directories that carry `mke2fs`/`mkfs.ext4`/`dd`. (The kata-agent launches components with `PATH=/bin:/sbin:/usr/bin:/usr/sbin`; since setting any `env` value replaces that variable wholesale, the base @@ -437,15 +509,15 @@ How each tool is provisioned depends on the **base flavour** (see | Tool | Bucket | Full-distro base (Ubuntu) | Distroless base (chiseled NVIDIA + NVRC) | |------------------------------|--------------------|----------------------------------------------------------|---------------------------------------------------------| | `veritysetup` | base, always | `cryptsetup-bin` (unconditional in `ubuntu/config.sh`) | copied into the base layout unconditionally | -| `cryptsetup` | coco addon | also present in the full-distro base via `cryptsetup-bin`| binary bundled in the coco addon; libs come from base | +| `cryptsetup` | coco extension | also present in the full-distro base via `cryptsetup-bin`| binary bundled in the coco extension; libs come from base | | `mke2fs`/`mkfs.ext4`/`dd` | base, for CoCo | `e2fsprogs` (on `CONFIDENTIAL_GUEST=yes`) + `coreutils` | copied into the base layout | The distroless path needs explicit copying because nothing lands in a chiseled image unless placed there deliberately, and the NVIDIA base is never built with `CONFIDENTIAL_GUEST=yes`. The full-distro base is built with `CONFIDENTIAL_GUEST=yes`, so the same tools arrive as ordinary packages. In both -cases the addon's `cryptsetup` resolves its libraries against the base, which -requires the base and the coco-addon builder to stay on the **same distro/ABI** +cases the extension's `cryptsetup` resolves its libraries against the base, which +requires the base and the coco-extension builder to stay on the **same distro/ABI** (Ubuntu 24.04 "noble" today). ### Image build pipeline @@ -454,16 +526,16 @@ requires the base and the coco-addon builder to stay on the **same distro/ABI** Kata base images come in two flavours, distinguished by **who owns early boot**. This distinction — not "Ubuntu vs NVIDIA" — is what drives the differences in -how tooling is provisioned and who mounts the addons: +how tooling is provisioned and who mounts the extensions: - **Full-distro base** — ships a complete distribution with **systemd** as init. - systemd discovers and mounts the addons (via `kata-addon-mount@.service`), and + systemd discovers and mounts the extensions (via `kata-extension-mount@.service`), and the binaries/libraries the guest needs arrive as ordinary distribution packages. The standard confidential `kata-containers.img` (Ubuntu) is today's instance. - **Distroless base** — a minimal, chiseled image with no full init system. A dedicated early-boot component takes over the responsibilities systemd would - otherwise have (addon discovery and mounting, attester selection, + otherwise have (extension discovery and mounting, attester selection, orchestration), and any tooling must be placed into the image deliberately rather than pulled in as packages. The chiseled `base-nvidia` driven by **NVRC** is today's instance. @@ -481,9 +553,9 @@ guest components (attestation-agent, confidential-data-hub, api-server-rest, ocicrypt config, pause bundle). It includes: - The kata-agent -- systemd and the `kata-addon-mount@.service` template unit +- systemd and the `kata-extension-mount@.service` template unit - `cryptsetup-bin` (provides `veritysetup`) — required unconditionally so - that the base image can mount verity-protected addons regardless of whether + that the base image can mount verity-protected extensions regardless of whether the base itself was built with `CONFIDENTIAL_GUEST=yes`. On Ubuntu this same package also provides `cryptsetup`, so the full-distro base happens to carry the encrypted-storage binary too. @@ -491,7 +563,7 @@ ocicrypt config, pause bundle). It includes: `dd`, and `/etc/mke2fs.conf`. On the standard base these come from `e2fsprogs` (added when `CONFIDENTIAL_GUEST=yes`) and `coreutils`. See "Runtime dependencies" for why the encrypted-storage `cryptsetup` lives in - the addon instead. + the extension instead. The **distroless base** (`base-nvidia`, driven by NVRC) is a chiseled, driver-agnostic image rather than a full distro, so the items above do not @@ -499,9 +571,9 @@ arrive as packages — they are copied into the base layout explicitly: `veritysetup` and its library closure unconditionally, and the `mke2fs`/`mkfs.ext4`/`dd`/`mke2fs.conf` plain-storage tooling alongside it. -#### CoCo addon image +#### CoCo guest extension image -The `kata-containers-coco-addon.img` is built by: +The `kata-containers-coco-extension.img` is built by: 1. Unpacking the CoCo guest components tarball into a temporary rootfs. Besides the agent-launched binaries (attestation-agent, attestation-agent-nv, @@ -512,14 +584,14 @@ The `kata-containers-coco-addon.img` is built by: - the NVIDIA attester libraries under `usr/local/lib` — `libnvat.so` plus its non-glibc transitive closure (libxml2, zlib, lzma, the C++ runtime). 2. Unpacking the pause image tarball into the same rootfs. -3. Writing the component manifest to `etc/kata-addons/components.toml`. +3. Writing the component manifest to `etc/kata-extensions/components.toml`. 4. Running the image builder with: - `FS_TYPE=erofs` — EROFS filesystem for compact, read-only storage. - `MEASURED_ROOTFS=yes` — creates a dm-verity hash partition. - `SKIP_DAX_HEADER=yes` — no DAX header (virtio-blk, not NVDIMM). - - `SKIP_ROOTFS_CHECK=yes` — the addon has no `/sbin/init`. - - `BUILD_VARIANT=coco-addon` — produces a correctly named root hash file - (`root_hash_coco-addon.txt`). + - `SKIP_ROOTFS_CHECK=yes` — the extension has no `/sbin/init`. + - `BUILD_VARIANT=coco-extension` — produces a correctly named root hash file + (`root_hash_coco-extension.txt`). The resulting image is a two-partition disk: @@ -531,11 +603,11 @@ into the runtime configuration templates. ### Security model -The addon architecture preserves the existing security guarantees of +The extension architecture preserves the existing security guarantees of Confidential Containers: -- **dm-verity** provides integrity protection for the addon image, identical - to the existing protection on the base rootfs. Any tampering with the addon +- **dm-verity** provides integrity protection for the guest extension image, identical + to the existing protection on the base rootfs. Any tampering with the extension contents is detected at mount time. - **Verity parameters on the kernel command line** are measured by the TEE @@ -547,7 +619,7 @@ Confidential Containers: - **Mount failure = VM shutdown** — `OnFailure=poweroff.target` ensures the VM does not proceed with missing or tampered components. -- **Read-only mounts** — both the base rootfs and addon images are mounted +- **Read-only mounts** — both the base rootfs and guest extension images are mounted read-only (EROFS), preventing runtime modification. ## Alternatives considered @@ -573,7 +645,7 @@ custom mounting approach described here: - **Block device bridging** — sysext discovers images from well-known directories (`/var/lib/extensions/`, `/run/extensions/`), not from raw block devices. A bridging step would still be needed to discover the - virtio-blk addon device, verify it with dm-verity, and place or symlink it + virtio-blk extension device, verify it with dm-verity, and place or symlink it where sysext expects to find it — eliminating much of the simplification sysext would offer. @@ -585,7 +657,7 @@ custom mounting approach described here: - **Version matching** — sysext enforces `extension-release.d` metadata matching against the host `os-release`. While this is useful for general purpose systems, it adds friction in the Kata context where the base image - and addon images are built and versioned together in a controlled pipeline. + and guest extension images are built and versioned together in a controlled pipeline. - **Minimal guest environment** — the Kata guest rootfs is a minimal environment that may not ship a full systemd with sysext/confext support @@ -616,7 +688,7 @@ proposal: at VM creation time, without modifying the base image itself. The two approaches are orthogonal. A Kata base image could be built and -managed using bootc/ostree, and addon images would still be cold-plugged +managed using bootc/ostree, and guest extension images would still be cold-plugged and mounted at VM boot using the mechanism described here. bootc does not provide a mechanism for dynamically composing additional block devices into a running or booting system — it operates at the image build and deployment @@ -624,28 +696,28 @@ layer, not at the VM assembly layer. ## Future work -### Additional addon types +### Additional extension types -The architecture is designed to support multiple addon images: +The architecture is designed to support multiple guest extension images: -- **GPU addon** — the NVIDIA GPU userspace (driver libraries, NVML, the - container-toolkit binaries, kernel modules) lives in a `gpu-addon` image - mounted at `/run/kata-addons/gpu`, carved out of the same build as the +- **GPU extension** — the NVIDIA GPU userspace (driver libraries, NVML, the + container-toolkit binaries, kernel modules) lives in a `gpu-extension` image + mounted at `/run/kata-extensions/gpu`, carved out of the same build as the driver-agnostic `base-nvidia` image. NVRC orchestrates early boot, loads the - modules from the addon, and composes the GPU addon with the CoCo addon on - confidential GPU guests. This addon is implemented; its interplay with the - CoCo addon (NVML resolution, attester selection) is covered in + modules from the extension, and composes the GPU extension with the CoCo extension on + confidential GPU guests. This extension is implemented; its interplay with the + CoCo extension (NVML resolution, attester selection) is covered in "Runtime dependencies" and "Attester variant selection" above. -- **Custom addons** — users can build their own addon images for +- **Custom extensions** — users can build their own guest extension images for workload-specific libraries, models, or configurations. -### Addon ordering +### Extension ordering -When multiple addons are configured, they are cold-plugged in the order they -appear in the `extra_images` list. The current design does not enforce -explicit ordering dependencies between addons. If future use cases require -addon-to-addon dependencies, the systemd units can be extended with +When multiple extensions are configured, they are cold-plugged in the order they +appear in the `guest_extension_images` list. The current design does not enforce +explicit ordering dependencies between extensions. If future use cases require +extension-to-extension dependencies, the systemd units can be extended with appropriate `After=`/`Requires=` relationships. ### Other hypervisor backends @@ -653,58 +725,58 @@ appropriate `After=`/`Requires=` relationships. The current proposal covers QEMU only. Extending to other backends requires implementing block device cold-plug for each: -- **Cloud Hypervisor** — add `--disk` entries with the addon image path and +- **Cloud Hypervisor** — add `--disk` entries with the guest extension image path and serial. Cloud Hypervisor natively supports virtio-blk serial numbers. - **Dragonball** — attach additional virtio-blk devices through the - Dragonball VMM's block device configuration, mapping each `ExtraImage` + Dragonball VMM's block device configuration, mapping each `GuestExtensionImage` to a drive with the corresponding serial. - **Firecracker** — add block device entries via the Firecracker API with the appropriate drive ID. Serial-based discovery may need adaptation since Firecracker exposes drive IDs differently. No changes are needed in the guest-side mounting logic or the agent — the -addon device discovery via `/sys/block/*/serial` and the systemd units work +extension device discovery via `/sys/block/*/serial` and the systemd units work the same way regardless of which hypervisor attached the block device. -### Manifest-driven addon discovery via init-data +### Manifest-driven extension discovery via init-data -The current design passes verity parameters for each addon on the kernel -command line. This works well for a small number of addons but does not -scale: each additional addon adds a long `kata.addon..verity_params=...` +The current design passes verity parameters for each extension on the kernel +command line. This works well for a small number of extensions but does not +scale: each additional extension adds a long `kata.extension..verity_params=...` entry, and the kernel command line has practical size limits. A future evolution could introduce a **two-phase bootstrap**: 1. **Phase 1 (kernel-params driven)** — the kernel command line carries - verity parameters for a single, small **manifest addon** image. This + verity parameters for a single, small **manifest extension** image. This image is mounted first using the existing mechanism. -2. **Phase 2 (manifest driven)** — the manifest addon contains a - configuration file (e.g. `addons.conf`) listing all other addons with +2. **Phase 2 (manifest driven)** — the manifest extension contains a + configuration file (e.g. `extensions.conf`) listing all other extensions with their names, verity parameters, and any additional metadata. The mount - script reads this file and mounts the remaining addons accordingly. + script reads this file and mounts the remaining extensions accordingly. This approach has several advantages: -- **Kernel command line stays fixed-size** regardless of how many addons are +- **Kernel command line stays fixed-size** regardless of how many extensions are composed. - **Attestation is simplified** — the TEE firmware measures one manifest hash on the command line; the verifier only needs to validate that single hash. The chain of trust extends from the measured kernel command line to - the verified manifest to the verified addons. + the verified manifest to the verified extensions. - **Richer metadata** — the manifest can carry structured information beyond - verity parameters: addon ordering constraints, version requirements, + verity parameters: extension ordering constraints, version requirements, policies, or init-data payloads. The design of the systemd units and mount script can support both modes: -if per-addon verity parameters are present on the kernel command line, they -are used directly (current behavior); if a manifest addon is present, it -is consulted for the remaining addons. This makes the manifest mode an +if per-extension verity parameters are present on the kernel command line, they +are used directly (current behavior); if a manifest extension is present, it +is consulted for the remaining extensions. This makes the manifest mode an additive, backward-compatible evolution. -### Addon versioning and attestation +### Extension versioning and attestation -Future work may add version metadata to addon images, enabling the +Future work may add version metadata to guest extension images, enabling the attestation flow to verify not just integrity (via dm-verity) but also that the specific expected version of each component is present.