mirror of
https://github.com/kata-containers/kata-containers.git
synced 2026-07-06 18:09:23 +00:00
docs: adopt guest-extension-image naming in composable-vm proposal
Align the composable VM images proposal with the naming convention used throughout the implementation: the generic mechanism and its data model are now "guest extension image" (config key `guest_extension_images`, struct `GuestExtensionImage`), and the runtime artifacts use the matching "extension" vocabulary (`/run/kata-extensions/<name>`, `kata.extension.<name>.verity_params`, `kata-extension-mount@.service`, the `extension-<name>` virtio-blk serial / dm-verity target, and the `kata-containers-coco-extension.img` image). It also documents how template-unit instances are enabled: a systemd generator instantiates kata-extension-mount@<name>.service for each extension on the kernel command line, so the rootfs build stays extension-agnostic. This replaces the earlier mixed "extra_images"/"addon" terminology and addresses the naming-consistency review feedback on the series. Assisted-by: Cursor <cursoragent@cursor.com> Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
This commit is contained in:
Fabiano Fidêncio
committed by
Fabiano Fidêncio
Fabiano Fidêncio
parent
e01a0b8975
commit
f4d0ae1c2f
@@ -10,18 +10,18 @@
|
||||
This proposal introduces a **composable image architecture** for Kata Containers
|
||||
guest VMs. Instead of building a single monolithic rootfs image that contains
|
||||
every component a workload might need, the runtime assembles a VM from a lean
|
||||
**base image** plus zero or more purpose-specific **addon images** that are
|
||||
cold-plugged as additional virtio-blk devices. Each addon is an EROFS image
|
||||
**base image** plus zero or more purpose-specific **guest extension images** that are
|
||||
cold-plugged as additional virtio-blk devices. Each extension is an EROFS image
|
||||
protected by dm-verity, mounted read-only inside the guest before the kata-agent
|
||||
starts.
|
||||
|
||||
The first application of this architecture splits **Confidential Containers
|
||||
(CoCo) guest components** out of the monolithic `kata-containers-confidential.img`
|
||||
into a separate `kata-containers-coco-addon.img`.
|
||||
into a separate `kata-containers-coco-extension.img`.
|
||||
|
||||
The current proposal targets **QEMU** as the hypervisor backend. The
|
||||
design is intentionally hypervisor-agnostic at the configuration and guest
|
||||
layers — the `extra_images` configuration, guest-side systemd units, and
|
||||
layers — the `guest_extension_images` configuration, guest-side systemd units, and
|
||||
agent path resolution work identically regardless of the hypervisor. Only the
|
||||
host-side device attachment is hypervisor-specific (QEMU virtio-blk
|
||||
cold-plug today), and extending support to other hypervisor backends
|
||||
@@ -50,7 +50,7 @@ pause bundle). This creates several problems:
|
||||
must maintain their own image build pipeline.
|
||||
|
||||
A composable approach addresses all four issues: the base image stays small and
|
||||
generic, addon images are independently built and versioned, and the runtime
|
||||
generic, guest extension images are independently built and versioned, and the runtime
|
||||
composes them at VM creation time.
|
||||
|
||||
## Design
|
||||
@@ -58,28 +58,30 @@ composes them at VM creation time.
|
||||
### Architecture overview
|
||||
|
||||
```
|
||||
Host Guest VM
|
||||
┌──────────────────────┐ ┌─────────────────────────────────┐
|
||||
│ Runtime config │ │ / (base rootfs, erofs) │
|
||||
│ ┌────────────────┐ │ │ kata-agent, systemd, ... │
|
||||
│ │ image = base │ │ boot │ │
|
||||
│ │ extra_images: │──────────>│ /run/kata-addons/coco/ (erofs) │
|
||||
│ │ - name: coco │ │ cold │ attestation-agent │
|
||||
│ │ path: ... │ │ plug │ confidential-data-hub │
|
||||
│ │ verity: ... │ │ │ api-server-rest │
|
||||
│ └────────────────┘ │ │ ocicrypt_config.json │
|
||||
│ │ │ pause_bundle/ │
|
||||
│ QEMU │ │ │
|
||||
│ -drive base.img │ │ dm-verity protects both the │
|
||||
│ -drive coco.img │ │ base rootfs and the addon. │
|
||||
│ serial=addon-… │ └─────────────────────────────────┘
|
||||
└──────────────────────┘
|
||||
Host Guest VM
|
||||
┌───────────────────────────────┐
|
||||
│ Runtime config │
|
||||
│ ┌─────────────────────────┐ │
|
||||
│ │ image = base │ │ ┌─────────────────────────────────────┐
|
||||
│ │ guest_extension_images: │──┼─────>│ / (base rootfs, erofs) │
|
||||
│ │ - name: coco │ │ boot │ kata-agent, systemd, ... │
|
||||
│ │ path: ... │ │ cold │ │
|
||||
│ │ verity: ... │ │ plug │ /run/kata-extensions/coco/ (erofs) │
|
||||
│ └─────────────────────────┘ │ │ attestation-agent │
|
||||
│ │ │ confidential-data-hub │
|
||||
│ QEMU │ │ api-server-rest │
|
||||
│ -drive base.img │ │ ocicrypt_config.json │
|
||||
│ -drive coco.img │ │ pause_bundle/ │
|
||||
│ serial=extension-coco │ │ │
|
||||
└───────────────────────────────┘ │ dm-verity protects both the │
|
||||
│ base rootfs and the extension. │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Runtime configuration
|
||||
|
||||
A new `extra_images` field on the hypervisor configuration accepts an ordered
|
||||
list of addon images:
|
||||
A new `guest_extension_images` field on the hypervisor configuration accepts an ordered
|
||||
list of guest extension images:
|
||||
|
||||
```toml
|
||||
[hypervisor.qemu]
|
||||
@@ -87,64 +89,78 @@ image = "/opt/kata/share/kata-containers/kata-containers.img"
|
||||
|
||||
# ...existing keys...
|
||||
|
||||
[[hypervisor.qemu.extra_images]]
|
||||
[[hypervisor.qemu.guest_extension_images]]
|
||||
name = "coco"
|
||||
path = "/opt/kata/share/kata-containers/kata-containers-coco-addon.img"
|
||||
path = "/opt/kata/share/kata-containers/kata-containers-coco-extension.img"
|
||||
verity_params = "root_hash=abc...,salt=def...,data_blocks=1234,hash_block_size=4096,data_block_size=4096"
|
||||
```
|
||||
|
||||
Each entry maps to a Rust struct:
|
||||
|
||||
```rust
|
||||
pub struct ExtraImage {
|
||||
pub struct GuestExtensionImage {
|
||||
pub name: String,
|
||||
pub path: String,
|
||||
pub verity_params: String,
|
||||
}
|
||||
```
|
||||
|
||||
The `name` field is the primary identifier for an addon. It must be unique
|
||||
The `name` field is the primary identifier for an extension. It must be unique
|
||||
within the configuration and is used to:
|
||||
|
||||
- Set the **virtio-blk serial** to `addon-<name>`, enabling deterministic
|
||||
- Set the **virtio-blk serial** to `extension-<name>`, enabling deterministic
|
||||
device discovery in the guest via `/sys/block/*/serial`.
|
||||
- Namespace the **kernel command-line** verity parameters as
|
||||
`kata.addon.<name>.verity_params=...`.
|
||||
- Name the guest **mount point** at `/run/kata-addons/<name>/`.
|
||||
- Name the **dm-verity device mapper target** as `addon-<name>`.
|
||||
`kata.extension.<name>.verity_params=...`.
|
||||
- Name the guest **mount point** at `/run/kata-extensions/<name>/`.
|
||||
- Name the **dm-verity device mapper target** as `extension-<name>`.
|
||||
|
||||
### Host-side: cold-plugging addon devices
|
||||
### Host-side: cold-plugging extension devices
|
||||
|
||||
At VM creation time, for each entry in `extra_images`, the runtime:
|
||||
At VM creation time, for each entry in `guest_extension_images`, the runtime:
|
||||
|
||||
1. Creates a virtio-blk device backed by the addon image file.
|
||||
2. Sets the device serial to `addon-<name>`.
|
||||
3. If `verity_params` is non-empty, appends
|
||||
`kata.addon.<name>.verity_params=<value>` to the guest kernel command line.
|
||||
1. Creates a virtio-blk device backed by the guest extension image file.
|
||||
Extensions are **always** attached as virtio-blk, using the architecture's
|
||||
virtio-blk transport (`virtio-blk-ccw` on s390x, `virtio-blk-pci`
|
||||
elsewhere). Neither the VM rootfs driver (`vm_rootfs_driver`) nor the
|
||||
generic block device driver (`block_device_driver`) is reused here: those
|
||||
may resolve to a non-virtio-blk transport such as `virtio-pmem` (NVDIMM) or
|
||||
`virtio-scsi`, and only virtio-blk devices carry the serial the guest relies
|
||||
on for discovery (step 2), so a non-virtio-blk transport would leave the
|
||||
extension undiscoverable and its mount unit would fail closed.
|
||||
2. Sets the device serial to `extension-<name>`.
|
||||
3. Appends `kata.extension.<name>.verity_params=<value>` to the guest kernel
|
||||
command line for *every* configured extension. The value is empty for an
|
||||
unmeasured extension (`verity_params = ""`, e.g. on s390x — see "Mount
|
||||
script" below); the entry is emitted unconditionally because it doubles as
|
||||
the guest-side activation signal for the mount unit. With an empty value the
|
||||
entry renders as a bare `kata.extension.<name>.verity_params` (no `=`), which
|
||||
the unit condition and generator both handle.
|
||||
|
||||
Both the Go runtime and the Rust runtime-rs implement this logic in their
|
||||
respective QEMU hypervisor backends. The addon devices are **cold-plugged** —
|
||||
respective QEMU hypervisor backends. The extension devices are **cold-plugged** —
|
||||
they are present on the QEMU command line at VM start, not hot-plugged later.
|
||||
|
||||
Other hypervisor backends (Cloud Hypervisor, Firecracker) do not implement
|
||||
addon cold-plugging yet, but the mechanism is straightforward to add: each
|
||||
backend only needs to translate the `ExtraImage` entries into its native
|
||||
extension cold-plugging yet, but the mechanism is straightforward to add: each
|
||||
backend only needs to translate the `GuestExtensionImage` entries into its native
|
||||
block device attachment API. The runtime configuration, guest-side mounting,
|
||||
and agent path resolution are completely hypervisor-independent.
|
||||
|
||||
### Guest-side: mounting addons
|
||||
### Guest-side: mounting extensions
|
||||
|
||||
A systemd template unit `kata-addon-mount@.service` handles addon discovery and
|
||||
mounting. The unit is instantiated per addon name (e.g.
|
||||
`kata-addon-mount@coco.service`).
|
||||
A systemd template unit `kata-extension-mount@.service` handles extension discovery and
|
||||
mounting. The unit is instantiated per extension name (e.g.
|
||||
`kata-extension-mount@coco.service`); a systemd generator (described below)
|
||||
creates those instances automatically from the kernel command line.
|
||||
|
||||
The mounting is performed by a systemd unit (before the agent starts) rather
|
||||
than by the kata-agent itself for several reasons:
|
||||
|
||||
- **Chicken-and-egg problem** — in the CoCo addon case, the addon carries
|
||||
- **Chicken-and-egg problem** — in the CoCo extension case, the extension carries
|
||||
the very guest components (attestation-agent, confidential-data-hub) that
|
||||
the kata-agent needs to launch. The agent cannot mount the addon because
|
||||
it needs the addon's contents to be available before it starts.
|
||||
the kata-agent needs to launch. The agent cannot mount the extension because
|
||||
it needs the extension's contents to be available before it starts.
|
||||
|
||||
- **Init-system ordering guarantees** — systemd provides declarative
|
||||
ordering (`Before=`, `After=`), conditional activation
|
||||
@@ -160,7 +176,7 @@ than by the kata-agent itself for several reasons:
|
||||
- **Non-systemd environments** — the design does not mandate systemd. In
|
||||
environments using a different init system (or a dedicated orchestrator
|
||||
like NVRC), the equivalent mounting logic can be implemented by whatever
|
||||
entity manages early boot. The key requirement is that addon images are
|
||||
entity manages early boot. The key requirement is that guest extension images are
|
||||
mounted before the kata-agent starts — how that is achieved is an
|
||||
init-system concern.
|
||||
|
||||
@@ -168,18 +184,18 @@ than by the kata-agent itself for several reasons:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Mount Kata addon image %i
|
||||
Description=Mount Kata guest extension image %i
|
||||
DefaultDependencies=no
|
||||
Before=kata-agent.service
|
||||
After=local-fs-pre.target
|
||||
ConditionKernelCommandLine=kata.addon.%i.verity_params
|
||||
ConditionKernelCommandLine=kata.extension.%i.verity_params
|
||||
OnFailure=poweroff.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
RemainAfterExit=yes
|
||||
ExecStart=/usr/libexec/kata-addon-mount.sh %i
|
||||
ExecStop=/usr/libexec/kata-addon-umount.sh %i
|
||||
ExecStart=/usr/libexec/kata-extension-mount.sh %i
|
||||
ExecStop=/usr/libexec/kata-extension-umount.sh %i
|
||||
|
||||
[Install]
|
||||
WantedBy=kata-containers.target
|
||||
@@ -188,28 +204,50 @@ WantedBy=kata-containers.target
|
||||
Key design decisions:
|
||||
|
||||
- **`ConditionKernelCommandLine`** — the service only activates when the
|
||||
runtime has actually configured the addon. This prevents the unit from
|
||||
running (and failing) in non-addon VM configurations.
|
||||
runtime has actually configured the extension. This prevents the unit from
|
||||
running (and failing) in non-extension VM configurations.
|
||||
|
||||
- **`Before=kata-agent.service`** — guarantees the addon filesystem is mounted
|
||||
- **`Before=kata-agent.service`** — guarantees the extension filesystem is mounted
|
||||
before the agent attempts to use any component from it.
|
||||
|
||||
- **`OnFailure=poweroff.target`** — if the addon cannot be mounted (e.g.
|
||||
- **`OnFailure=poweroff.target`** — if the extension cannot be mounted (e.g.
|
||||
verity verification failure, missing device), the VM is shut down
|
||||
immediately. A confidential VM must not continue with an unverified or
|
||||
missing addon.
|
||||
missing extension.
|
||||
|
||||
#### Enabling instances
|
||||
|
||||
> **Implementation note.** An earlier revision enabled a fixed
|
||||
> `kata-extension-mount@coco.service` symlink at rootfs build time, which would
|
||||
> have required editing the image build for every new extension. Review feedback
|
||||
> pushed us to the generator below so the build stays extension-agnostic.
|
||||
|
||||
systemd template units must be enabled per instance, and the set of extensions
|
||||
is only known at runtime (from the kernel command line). A systemd generator,
|
||||
`kata-extension-mount-generator`, bridges that gap: it runs in early boot, scans
|
||||
`/proc/cmdline` for every `kata.extension.<name>.verity_params` entry (whether it
|
||||
carries a value or, for an unmeasured extension, is a bare key), and symlinks
|
||||
`kata-extension-mount@<name>.service` into `kata-containers.target.wants`.
|
||||
|
||||
Because the runtime emits exactly one such cmdline entry per configured
|
||||
`guest_extension_images`, the generator enables precisely the extensions the VM
|
||||
was configured with. Adding a new extension therefore requires no change to the
|
||||
rootfs build — it is wired up entirely from runtime configuration. The generator
|
||||
ships in the base image (installed alongside the template unit by the agent), so
|
||||
it is part of the dm-verity-measured rootfs, and the cmdline it reads is itself
|
||||
covered by the guest launch measurement.
|
||||
|
||||
#### Mount script
|
||||
|
||||
The mount script (`kata-addon-mount.sh`) performs the following steps:
|
||||
The mount script (`kata-extension-mount.sh`) performs the following steps:
|
||||
|
||||
1. **Device discovery** — scans `/sys/block/*/serial` for a device whose
|
||||
serial matches `addon-<name>`. This is more reliable than waiting for udev
|
||||
serial matches `extension-<name>`. This is more reliable than waiting for udev
|
||||
to create `/dev/disk/by-id/` symlinks, since the minimal guest environment
|
||||
may not run a full udev daemon.
|
||||
|
||||
2. **Verity parameter extraction** — reads
|
||||
`kata.addon.<name>.verity_params=...` from `/proc/cmdline` and parses the
|
||||
`kata.extension.<name>.verity_params=...` from `/proc/cmdline` and parses the
|
||||
comma-separated key=value pairs (root_hash, salt, data_blocks,
|
||||
hash_block_size, data_block_size).
|
||||
|
||||
@@ -219,24 +257,58 @@ The mount script (`kata-addon-mount.sh`) performs the following steps:
|
||||
creation — all verity parameters are passed explicitly rather than stored
|
||||
in an on-disk superblock.
|
||||
|
||||
4. **EROFS mount** — mounts the verified device (or the raw partition if
|
||||
verity is not configured) read-only at `/run/kata-addons/<name>/`.
|
||||
4. **EROFS mount** — mounts the resulting device read-only at
|
||||
`/run/kata-extensions/<name>/` (the dm-verity target when verified, or the
|
||||
data partition directly for an unmeasured extension).
|
||||
|
||||
##### Integrity policy: measured vs. unmeasured, and failing closed
|
||||
|
||||
An extension can legitimately ship *without* dm-verity — for example on s390x,
|
||||
where IBM Secure Execution protects the guest through a different mechanism and
|
||||
images are built with `MEASURED_ROOTFS=no`. The mount script must therefore
|
||||
support a raw (unverified) mount, but it must **not** let that path become a
|
||||
silent downgrade: an attacker who can edit the (host-supplied) kernel command
|
||||
line could otherwise strip `verity_params` from a *measured* extension and have
|
||||
it mounted unverified.
|
||||
|
||||
The script separates these two cases explicitly, using the **on-disk layout as
|
||||
the source of truth** rather than trusting the cmdline alone. The image build
|
||||
encodes its integrity policy in the partition table (see "Image build" below):
|
||||
a measured extension carries a dm-verity **hash partition** (`p2`) next to the
|
||||
data partition (`p1`), while an unmeasured extension has only `p1`. The script
|
||||
detects the presence of the hash device and cross-checks it against the
|
||||
`verity_params` carried on the kernel command line (which, in a confidential
|
||||
guest, is itself part of the measured, attested boot):
|
||||
|
||||
| Hash device (`p2`) | `verity_params` on cmdline | Action |
|
||||
|--------------------|----------------------------|----------------------------------------------------|
|
||||
| present | present | **verify** with dm-verity, then mount (normal case)|
|
||||
| present | absent / empty | **refuse** — verity was stripped/disabled (tamper) |
|
||||
| absent | present | **refuse** — params but nothing to verify (mismatch)|
|
||||
| absent | absent / empty | **raw mount** — genuinely unmeasured extension |
|
||||
|
||||
Any "refuse" path exits non-zero; combined with `OnFailure=poweroff.target` on
|
||||
the mount unit, that powers the VM off rather than continuing with an unverified
|
||||
or inconsistent extension. The defence has two layers: the cmdline (and thus the
|
||||
root hash, or its deliberate absence) is covered by the guest launch
|
||||
measurement and remote attestation, and the in-guest layout cross-check fails
|
||||
closed so a measured extension can never be silently downgraded to a raw mount.
|
||||
|
||||
### Agent-side: data-driven component manifest
|
||||
|
||||
> **Implementation note.** An earlier revision of this proposal described the
|
||||
> agent as resolving a small, hardcoded list of component paths. While
|
||||
> implementing and testing the addon we found this too rigid: the addon
|
||||
> implementing and testing the extension we found this too rigid: the extension
|
||||
> evolved to ship multiple attestation-agent flavours, per-process environment
|
||||
> requirements, and ordering constraints that the agent should not have to know
|
||||
> about. The design below — a **data-driven manifest** owned by the addon — is
|
||||
> about. The design below — a **data-driven manifest** owned by the extension — is
|
||||
> what we converged on so that adding or reconfiguring a bundled component
|
||||
> requires **no** kata-agent code change.
|
||||
|
||||
Each addon ships a manifest at `etc/kata-addons/components.toml`. When the addon
|
||||
Each extension ships a manifest at `etc/kata-extensions/components.toml`. When the extension
|
||||
is mounted, the kata-agent reads it and builds its launch plan from it. The
|
||||
manifest declares the processes to launch and the resources they expose; all
|
||||
paths are **relative to the addon mount root** (`/run/kata-addons/<name>`).
|
||||
paths are **relative to the extension mount root** (`/run/kata-extensions/<name>`).
|
||||
|
||||
A process entry carries:
|
||||
|
||||
@@ -252,7 +324,7 @@ A process entry carries:
|
||||
|
||||
`${...}` tokens in `args`, `config`, `env` values, and variant fields are
|
||||
substituted by the agent from a runtime context it assembles (socket and config
|
||||
paths, the addon mount root `${addon_root}`, the selected `${attester_variant}`,
|
||||
paths, the extension mount root `${extension_root}`, the selected `${attester_variant}`,
|
||||
etc.). Introducing a brand-new variable is the only change that ever needs to
|
||||
touch the agent.
|
||||
|
||||
@@ -280,14 +352,14 @@ select = "${attester_variant}"
|
||||
|
||||
[process.variants.nvidia]
|
||||
path = "usr/local/bin/attestation-agent-nv"
|
||||
env = { LD_LIBRARY_PATH = "${addon_root}/usr/local/lib:/run/kata-addons/gpu/usr/lib" }
|
||||
env = { LD_LIBRARY_PATH = "${extension_root}/usr/local/lib:/run/kata-extensions/gpu/usr/lib" }
|
||||
|
||||
[[process]]
|
||||
id = "confidential-data-hub"
|
||||
level = 2
|
||||
path = "usr/local/bin/confidential-data-hub"
|
||||
config = "${cdh_config_path}"
|
||||
env = { OCICRYPT_KEYPROVIDER_CONFIG = "${ocicrypt_config_path}", PATH = "${addon_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin" }
|
||||
env = { OCICRYPT_KEYPROVIDER_CONFIG = "${ocicrypt_config_path}", PATH = "${extension_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin" }
|
||||
wait_socket = "${cdh_socket}"
|
||||
|
||||
[[process]]
|
||||
@@ -297,34 +369,34 @@ path = "usr/local/bin/api-server-rest"
|
||||
args = ["--features", "${rest_api_features}"]
|
||||
```
|
||||
|
||||
When **no** addon is mounted, the agent falls back to a built-in launch plan
|
||||
When **no** extension is mounted, the agent falls back to a built-in launch plan
|
||||
that reproduces the legacy behaviour (components launched from
|
||||
`/usr/local/bin/...` in the rootfs). The same dual-path principle applies to
|
||||
the non-process resources declared under `[paths]`: the agent resolves them
|
||||
inside the addon first and falls back to the legacy location otherwise:
|
||||
inside the extension first and falls back to the legacy location otherwise:
|
||||
|
||||
| Resource | Addon path | Legacy path |
|
||||
| Resource | Extension path | Legacy path |
|
||||
|------------------------|---------------------------------------------------------|---------------------------------|
|
||||
| attestation-agent(-nv) | `/run/kata-addons/coco/usr/local/bin/attestation-agent[-nv]` | `/usr/local/bin/attestation-agent` |
|
||||
| confidential-data-hub | `/run/kata-addons/coco/usr/local/bin/confidential-data-hub` | `/usr/local/bin/confidential-data-hub` |
|
||||
| api-server-rest | `/run/kata-addons/coco/usr/local/bin/api-server-rest` | `/usr/local/bin/api-server-rest` |
|
||||
| ocicrypt_config.json | `/run/kata-addons/coco/etc/ocicrypt_config.json` | `/etc/ocicrypt_config.json` |
|
||||
| pause_bundle | `/run/kata-addons/coco/pause_bundle` | `/pause_bundle` |
|
||||
| attestation-agent(-nv) | `/run/kata-extensions/coco/usr/local/bin/attestation-agent[-nv]` | `/usr/local/bin/attestation-agent` |
|
||||
| confidential-data-hub | `/run/kata-extensions/coco/usr/local/bin/confidential-data-hub` | `/usr/local/bin/confidential-data-hub` |
|
||||
| api-server-rest | `/run/kata-extensions/coco/usr/local/bin/api-server-rest` | `/usr/local/bin/api-server-rest` |
|
||||
| ocicrypt_config.json | `/run/kata-extensions/coco/etc/ocicrypt_config.json` | `/etc/ocicrypt_config.json` |
|
||||
| pause_bundle | `/run/kata-extensions/coco/pause_bundle` | `/pause_bundle` |
|
||||
|
||||
This approach:
|
||||
|
||||
- Preserves **backward compatibility** with existing monolithic rootfs images
|
||||
where CoCo components are baked into the base image.
|
||||
- Requires **no special rootfs modifications** — the base image does not need
|
||||
stub files or directories for the addon components.
|
||||
stub files or directories for the extension components.
|
||||
- Works transparently on a **read-only rootfs** — no bind-mounting, no
|
||||
remounting, no writes to the root filesystem.
|
||||
- Keeps the agent **generic** — addon-specific names, env, and ordering live
|
||||
- Keeps the agent **generic** — extension-specific names, env, and ordering live
|
||||
in the manifest, not in agent code.
|
||||
|
||||
### Attester variant selection and the NVRC contract
|
||||
|
||||
The CoCo addon ships **two** attestation-agent builds: the stock
|
||||
The CoCo extension ships **two** attestation-agent builds: the stock
|
||||
`attestation-agent` and an NVIDIA-attester build, `attestation-agent-nv`, that
|
||||
collects GPU evidence in addition to the TEE evidence. Which one runs is chosen
|
||||
by the manifest's `select = "${attester_variant}"` selector, and the value of
|
||||
@@ -333,7 +405,7 @@ by the manifest's `select = "${attester_variant}"` selector, and the value of
|
||||
- On a plain confidential guest the kata-agent runs init itself and the variable
|
||||
defaults to `default` → the stock attester launches.
|
||||
- On a GPU guest, **NVRC** (the NVIDIA runtime config that owns early boot)
|
||||
detects the GPU addon and exports `KATA_ATTESTER_VARIANT=nvidia` before
|
||||
detects the GPU extension and exports `KATA_ATTESTER_VARIANT=nvidia` before
|
||||
exec'ing the kata-agent. The agent forwards that into the substitution context
|
||||
as `${attester_variant}`, so the `nvidia` variant launches.
|
||||
|
||||
@@ -343,28 +415,28 @@ consumed by the kata-agent, which keeps the agent free of any GPU- or
|
||||
NVIDIA-specific knowledge — it only knows how to forward a selector into the
|
||||
manifest. We arrived at this split after first trying to special-case the
|
||||
attester inside the agent; pushing the decision out to NVRC + the manifest kept
|
||||
both the agent and the addon generic.
|
||||
both the agent and the extension generic.
|
||||
|
||||
#### Why one addon with two builds (and not two addons)
|
||||
#### Why one extension with two builds (and not two extensions)
|
||||
|
||||
Shipping **two** attestation-agent builds inside a **single** CoCo addon is a
|
||||
deliberate choice, and it is worth being explicit about it because addons are
|
||||
Shipping **two** attestation-agent builds inside a **single** CoCo extension is a
|
||||
deliberate choice, and it is worth being explicit about it because extensions are
|
||||
otherwise meant to *eliminate* duplication.
|
||||
|
||||
- A single CoCo addon serves **both** plain confidential guests (TEE evidence
|
||||
- A single CoCo extension serves **both** plain confidential guests (TEE evidence
|
||||
only) and confidential GPU guests (TEE + GPU evidence). The only thing that
|
||||
differs between them is which attestation-agent binary runs; everything else
|
||||
in the addon (confidential-data-hub, api-server-rest, ocicrypt config, pause
|
||||
in the extension (confidential-data-hub, api-server-rest, ocicrypt config, pause
|
||||
bundle, `cryptsetup`) is shared verbatim. Splitting the NVIDIA attester into
|
||||
its own addon would duplicate that shared payload and force every confidential
|
||||
GPU guest to compose two CoCo-flavoured addons instead of one.
|
||||
its own extension would duplicate that shared payload and force every confidential
|
||||
GPU guest to compose two CoCo-flavoured extensions instead of one.
|
||||
- The **cost** of keeping both builds together is precisely the manifest's
|
||||
`select`/`variants` machinery: the manifest has to be aware that the
|
||||
attestation-agent comes in two flavours and pick one at runtime. We consider
|
||||
that a fair trade — the complexity is confined to data (the manifest), the
|
||||
agent stays generic, and the addon stays a single, coherent "CoCo" unit.
|
||||
- A separate addon only pays off when its contents are **substantially**
|
||||
different (e.g. the GPU addon, which carries the driver userspace), not for
|
||||
agent stays generic, and the extension stays a single, coherent "CoCo" unit.
|
||||
- A separate extension only pays off when its contents are **substantially**
|
||||
different (e.g. the GPU extension, which carries the driver userspace), not for
|
||||
two near-identical builds of the same component. No further attester variants
|
||||
are planned today, but if one appeared it would be another `[process.variants.<name>]`
|
||||
entry — not a new image.
|
||||
@@ -382,25 +454,25 @@ physically live.
|
||||
turn:
|
||||
|
||||
- `dlopen`s `libnvidia-ml.so.1` (NVML) at runtime to gather GPU evidence. NVML
|
||||
is part of the **GPU addon**, mounted at `/run/kata-addons/gpu` with its driver
|
||||
is part of the **GPU extension**, mounted at `/run/kata-extensions/gpu` with its driver
|
||||
libraries under `usr/lib`.
|
||||
- pulls in a closure of non-glibc libraries (libxml2, zlib, lzma, the C++
|
||||
runtime, ...) that the guest rootfs does not otherwise ship.
|
||||
|
||||
Neither set is present in a stock guest, so:
|
||||
|
||||
- The CoCo addon build bundles `libnvat.so` **and every non-glibc transitive
|
||||
- The CoCo extension build bundles `libnvat.so` **and every non-glibc transitive
|
||||
dependency** next to it under `usr/local/lib`.
|
||||
- The `nvidia` **manifest variant** (the `[process.variants.nvidia]` entry, not
|
||||
a separate image) sets an `LD_LIBRARY_PATH` that lists **both** the CoCo
|
||||
addon's `usr/local/lib` (for `libnvat` and its closure) **and** the GPU
|
||||
addon's `usr/lib` (for NVML). Setting only the first was the cause of an
|
||||
extension's `usr/local/lib` (for `libnvat` and its closure) **and** the GPU
|
||||
extension's `usr/lib` (for NVML). Setting only the first was the cause of an
|
||||
`NVAT Error 500: NVML Initialization Failed` we hit during bring-up; the
|
||||
RCAR handshake then never produced GPU evidence and attestation failed.
|
||||
|
||||
Because the agent applies manifest `env` on top of the inherited environment
|
||||
(without clearing it), and because the `nvidia` variant only ever runs when the
|
||||
GPU addon is present, referencing the GPU addon's well-known mount path here is
|
||||
GPU extension is present, referencing the GPU extension's well-known mount path here is
|
||||
safe.
|
||||
|
||||
#### CDH secure-mount tooling (encrypted vs plain storage)
|
||||
@@ -415,17 +487,17 @@ in two different places:
|
||||
unconditionally.
|
||||
- **Encrypted storage** — `cryptsetup` LUKS-formats and opens the volume. This is
|
||||
a CoCo-only capability, so it belongs with the CoCo guest components in the
|
||||
**coco addon**.
|
||||
**coco extension**.
|
||||
|
||||
This is the same `veritysetup`-vs-`cryptsetup` reasoning already applied to addon
|
||||
mounting: the base must always carry `veritysetup` (it opens *every* addon as a
|
||||
This is the same `veritysetup`-vs-`cryptsetup` reasoning already applied to extension
|
||||
mounting: the base must always carry `veritysetup` (it opens *every* extension as a
|
||||
dm-verity device before mounting), and `cryptsetup` shares an identical
|
||||
shared-library closure, so wherever `veritysetup` lives the libraries for
|
||||
`cryptsetup` are already present.
|
||||
|
||||
Because CDH runs in the **base rootfs namespace** but `cryptsetup` lives in the
|
||||
addon (which is not on the default search path), the manifest sets CDH's `PATH`
|
||||
to `${addon_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin` — the addon's
|
||||
extension (which is not on the default search path), the manifest sets CDH's `PATH`
|
||||
to `${extension_root}/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin` — the extension's
|
||||
`cryptsetup` first, then the base directories that carry `mke2fs`/`mkfs.ext4`/`dd`.
|
||||
(The kata-agent launches components with `PATH=/bin:/sbin:/usr/bin:/usr/sbin`;
|
||||
since setting any `env` value replaces that variable wholesale, the base
|
||||
@@ -437,15 +509,15 @@ How each tool is provisioned depends on the **base flavour** (see
|
||||
| Tool | Bucket | Full-distro base (Ubuntu) | Distroless base (chiseled NVIDIA + NVRC) |
|
||||
|------------------------------|--------------------|----------------------------------------------------------|---------------------------------------------------------|
|
||||
| `veritysetup` | base, always | `cryptsetup-bin` (unconditional in `ubuntu/config.sh`) | copied into the base layout unconditionally |
|
||||
| `cryptsetup` | coco addon | also present in the full-distro base via `cryptsetup-bin`| binary bundled in the coco addon; libs come from base |
|
||||
| `cryptsetup` | coco extension | also present in the full-distro base via `cryptsetup-bin`| binary bundled in the coco extension; libs come from base |
|
||||
| `mke2fs`/`mkfs.ext4`/`dd` | base, for CoCo | `e2fsprogs` (on `CONFIDENTIAL_GUEST=yes`) + `coreutils` | copied into the base layout |
|
||||
|
||||
The distroless path needs explicit copying because nothing lands in a chiseled
|
||||
image unless placed there deliberately, and the NVIDIA base is never built with
|
||||
`CONFIDENTIAL_GUEST=yes`. The full-distro base is built with
|
||||
`CONFIDENTIAL_GUEST=yes`, so the same tools arrive as ordinary packages. In both
|
||||
cases the addon's `cryptsetup` resolves its libraries against the base, which
|
||||
requires the base and the coco-addon builder to stay on the **same distro/ABI**
|
||||
cases the extension's `cryptsetup` resolves its libraries against the base, which
|
||||
requires the base and the coco-extension builder to stay on the **same distro/ABI**
|
||||
(Ubuntu 24.04 "noble" today).
|
||||
|
||||
### Image build pipeline
|
||||
@@ -454,16 +526,16 @@ requires the base and the coco-addon builder to stay on the **same distro/ABI**
|
||||
|
||||
Kata base images come in two flavours, distinguished by **who owns early boot**.
|
||||
This distinction — not "Ubuntu vs NVIDIA" — is what drives the differences in
|
||||
how tooling is provisioned and who mounts the addons:
|
||||
how tooling is provisioned and who mounts the extensions:
|
||||
|
||||
- **Full-distro base** — ships a complete distribution with **systemd** as init.
|
||||
systemd discovers and mounts the addons (via `kata-addon-mount@.service`), and
|
||||
systemd discovers and mounts the extensions (via `kata-extension-mount@.service`), and
|
||||
the binaries/libraries the guest needs arrive as ordinary distribution
|
||||
packages. The standard confidential `kata-containers.img` (Ubuntu) is today's
|
||||
instance.
|
||||
- **Distroless base** — a minimal, chiseled image with no full init system. A
|
||||
dedicated early-boot component takes over the responsibilities systemd would
|
||||
otherwise have (addon discovery and mounting, attester selection,
|
||||
otherwise have (extension discovery and mounting, attester selection,
|
||||
orchestration), and any tooling must be placed into the image deliberately
|
||||
rather than pulled in as packages. The chiseled `base-nvidia` driven by
|
||||
**NVRC** is today's instance.
|
||||
@@ -481,9 +553,9 @@ guest components (attestation-agent, confidential-data-hub, api-server-rest,
|
||||
ocicrypt config, pause bundle). It includes:
|
||||
|
||||
- The kata-agent
|
||||
- systemd and the `kata-addon-mount@.service` template unit
|
||||
- systemd and the `kata-extension-mount@.service` template unit
|
||||
- `cryptsetup-bin` (provides `veritysetup`) — required unconditionally so
|
||||
that the base image can mount verity-protected addons regardless of whether
|
||||
that the base image can mount verity-protected extensions regardless of whether
|
||||
the base itself was built with `CONFIDENTIAL_GUEST=yes`. On Ubuntu this same
|
||||
package also provides `cryptsetup`, so the full-distro base happens to carry
|
||||
the encrypted-storage binary too.
|
||||
@@ -491,7 +563,7 @@ ocicrypt config, pause bundle). It includes:
|
||||
`dd`, and `/etc/mke2fs.conf`. On the standard base these come from
|
||||
`e2fsprogs` (added when `CONFIDENTIAL_GUEST=yes`) and `coreutils`. See
|
||||
"Runtime dependencies" for why the encrypted-storage `cryptsetup` lives in
|
||||
the addon instead.
|
||||
the extension instead.
|
||||
|
||||
The **distroless base** (`base-nvidia`, driven by NVRC) is a chiseled,
|
||||
driver-agnostic image rather than a full distro, so the items above do not
|
||||
@@ -499,9 +571,9 @@ arrive as packages — they are copied into the base layout explicitly:
|
||||
`veritysetup` and its library closure unconditionally, and the
|
||||
`mke2fs`/`mkfs.ext4`/`dd`/`mke2fs.conf` plain-storage tooling alongside it.
|
||||
|
||||
#### CoCo addon image
|
||||
#### CoCo guest extension image
|
||||
|
||||
The `kata-containers-coco-addon.img` is built by:
|
||||
The `kata-containers-coco-extension.img` is built by:
|
||||
|
||||
1. Unpacking the CoCo guest components tarball into a temporary rootfs. Besides
|
||||
the agent-launched binaries (attestation-agent, attestation-agent-nv,
|
||||
@@ -512,14 +584,14 @@ The `kata-containers-coco-addon.img` is built by:
|
||||
- the NVIDIA attester libraries under `usr/local/lib` — `libnvat.so` plus its
|
||||
non-glibc transitive closure (libxml2, zlib, lzma, the C++ runtime).
|
||||
2. Unpacking the pause image tarball into the same rootfs.
|
||||
3. Writing the component manifest to `etc/kata-addons/components.toml`.
|
||||
3. Writing the component manifest to `etc/kata-extensions/components.toml`.
|
||||
4. Running the image builder with:
|
||||
- `FS_TYPE=erofs` — EROFS filesystem for compact, read-only storage.
|
||||
- `MEASURED_ROOTFS=yes` — creates a dm-verity hash partition.
|
||||
- `SKIP_DAX_HEADER=yes` — no DAX header (virtio-blk, not NVDIMM).
|
||||
- `SKIP_ROOTFS_CHECK=yes` — the addon has no `/sbin/init`.
|
||||
- `BUILD_VARIANT=coco-addon` — produces a correctly named root hash file
|
||||
(`root_hash_coco-addon.txt`).
|
||||
- `SKIP_ROOTFS_CHECK=yes` — the extension has no `/sbin/init`.
|
||||
- `BUILD_VARIANT=coco-extension` — produces a correctly named root hash file
|
||||
(`root_hash_coco-extension.txt`).
|
||||
|
||||
The resulting image is a two-partition disk:
|
||||
|
||||
@@ -531,11 +603,11 @@ into the runtime configuration templates.
|
||||
|
||||
### Security model
|
||||
|
||||
The addon architecture preserves the existing security guarantees of
|
||||
The extension architecture preserves the existing security guarantees of
|
||||
Confidential Containers:
|
||||
|
||||
- **dm-verity** provides integrity protection for the addon image, identical
|
||||
to the existing protection on the base rootfs. Any tampering with the addon
|
||||
- **dm-verity** provides integrity protection for the guest extension image, identical
|
||||
to the existing protection on the base rootfs. Any tampering with the extension
|
||||
contents is detected at mount time.
|
||||
|
||||
- **Verity parameters on the kernel command line** are measured by the TEE
|
||||
@@ -547,7 +619,7 @@ Confidential Containers:
|
||||
- **Mount failure = VM shutdown** — `OnFailure=poweroff.target` ensures the
|
||||
VM does not proceed with missing or tampered components.
|
||||
|
||||
- **Read-only mounts** — both the base rootfs and addon images are mounted
|
||||
- **Read-only mounts** — both the base rootfs and guest extension images are mounted
|
||||
read-only (EROFS), preventing runtime modification.
|
||||
|
||||
## Alternatives considered
|
||||
@@ -573,7 +645,7 @@ custom mounting approach described here:
|
||||
- **Block device bridging** — sysext discovers images from well-known
|
||||
directories (`/var/lib/extensions/`, `/run/extensions/`), not from raw
|
||||
block devices. A bridging step would still be needed to discover the
|
||||
virtio-blk addon device, verify it with dm-verity, and place or symlink it
|
||||
virtio-blk extension device, verify it with dm-verity, and place or symlink it
|
||||
where sysext expects to find it — eliminating much of the simplification
|
||||
sysext would offer.
|
||||
|
||||
@@ -585,7 +657,7 @@ custom mounting approach described here:
|
||||
- **Version matching** — sysext enforces `extension-release.d` metadata
|
||||
matching against the host `os-release`. While this is useful for general
|
||||
purpose systems, it adds friction in the Kata context where the base image
|
||||
and addon images are built and versioned together in a controlled pipeline.
|
||||
and guest extension images are built and versioned together in a controlled pipeline.
|
||||
|
||||
- **Minimal guest environment** — the Kata guest rootfs is a minimal
|
||||
environment that may not ship a full systemd with sysext/confext support
|
||||
@@ -616,7 +688,7 @@ proposal:
|
||||
at VM creation time, without modifying the base image itself.
|
||||
|
||||
The two approaches are orthogonal. A Kata base image could be built and
|
||||
managed using bootc/ostree, and addon images would still be cold-plugged
|
||||
managed using bootc/ostree, and guest extension images would still be cold-plugged
|
||||
and mounted at VM boot using the mechanism described here. bootc does not
|
||||
provide a mechanism for dynamically composing additional block devices into
|
||||
a running or booting system — it operates at the image build and deployment
|
||||
@@ -624,28 +696,28 @@ layer, not at the VM assembly layer.
|
||||
|
||||
## Future work
|
||||
|
||||
### Additional addon types
|
||||
### Additional extension types
|
||||
|
||||
The architecture is designed to support multiple addon images:
|
||||
The architecture is designed to support multiple guest extension images:
|
||||
|
||||
- **GPU addon** — the NVIDIA GPU userspace (driver libraries, NVML, the
|
||||
container-toolkit binaries, kernel modules) lives in a `gpu-addon` image
|
||||
mounted at `/run/kata-addons/gpu`, carved out of the same build as the
|
||||
- **GPU extension** — the NVIDIA GPU userspace (driver libraries, NVML, the
|
||||
container-toolkit binaries, kernel modules) lives in a `gpu-extension` image
|
||||
mounted at `/run/kata-extensions/gpu`, carved out of the same build as the
|
||||
driver-agnostic `base-nvidia` image. NVRC orchestrates early boot, loads the
|
||||
modules from the addon, and composes the GPU addon with the CoCo addon on
|
||||
confidential GPU guests. This addon is implemented; its interplay with the
|
||||
CoCo addon (NVML resolution, attester selection) is covered in
|
||||
modules from the extension, and composes the GPU extension with the CoCo extension on
|
||||
confidential GPU guests. This extension is implemented; its interplay with the
|
||||
CoCo extension (NVML resolution, attester selection) is covered in
|
||||
"Runtime dependencies" and "Attester variant selection" above.
|
||||
|
||||
- **Custom addons** — users can build their own addon images for
|
||||
- **Custom extensions** — users can build their own guest extension images for
|
||||
workload-specific libraries, models, or configurations.
|
||||
|
||||
### Addon ordering
|
||||
### Extension ordering
|
||||
|
||||
When multiple addons are configured, they are cold-plugged in the order they
|
||||
appear in the `extra_images` list. The current design does not enforce
|
||||
explicit ordering dependencies between addons. If future use cases require
|
||||
addon-to-addon dependencies, the systemd units can be extended with
|
||||
When multiple extensions are configured, they are cold-plugged in the order they
|
||||
appear in the `guest_extension_images` list. The current design does not enforce
|
||||
explicit ordering dependencies between extensions. If future use cases require
|
||||
extension-to-extension dependencies, the systemd units can be extended with
|
||||
appropriate `After=`/`Requires=` relationships.
|
||||
|
||||
### Other hypervisor backends
|
||||
@@ -653,58 +725,58 @@ appropriate `After=`/`Requires=` relationships.
|
||||
The current proposal covers QEMU only. Extending to other backends
|
||||
requires implementing block device cold-plug for each:
|
||||
|
||||
- **Cloud Hypervisor** — add `--disk` entries with the addon image path and
|
||||
- **Cloud Hypervisor** — add `--disk` entries with the guest extension image path and
|
||||
serial. Cloud Hypervisor natively supports virtio-blk serial numbers.
|
||||
- **Dragonball** — attach additional virtio-blk devices through the
|
||||
Dragonball VMM's block device configuration, mapping each `ExtraImage`
|
||||
Dragonball VMM's block device configuration, mapping each `GuestExtensionImage`
|
||||
to a drive with the corresponding serial.
|
||||
- **Firecracker** — add block device entries via the Firecracker API with
|
||||
the appropriate drive ID. Serial-based discovery may need adaptation since
|
||||
Firecracker exposes drive IDs differently.
|
||||
|
||||
No changes are needed in the guest-side mounting logic or the agent — the
|
||||
addon device discovery via `/sys/block/*/serial` and the systemd units work
|
||||
extension device discovery via `/sys/block/*/serial` and the systemd units work
|
||||
the same way regardless of which hypervisor attached the block device.
|
||||
|
||||
### Manifest-driven addon discovery via init-data
|
||||
### Manifest-driven extension discovery via init-data
|
||||
|
||||
The current design passes verity parameters for each addon on the kernel
|
||||
command line. This works well for a small number of addons but does not
|
||||
scale: each additional addon adds a long `kata.addon.<name>.verity_params=...`
|
||||
The current design passes verity parameters for each extension on the kernel
|
||||
command line. This works well for a small number of extensions but does not
|
||||
scale: each additional extension adds a long `kata.extension.<name>.verity_params=...`
|
||||
entry, and the kernel command line has practical size limits.
|
||||
|
||||
A future evolution could introduce a **two-phase bootstrap**:
|
||||
|
||||
1. **Phase 1 (kernel-params driven)** — the kernel command line carries
|
||||
verity parameters for a single, small **manifest addon** image. This
|
||||
verity parameters for a single, small **manifest extension** image. This
|
||||
image is mounted first using the existing mechanism.
|
||||
|
||||
2. **Phase 2 (manifest driven)** — the manifest addon contains a
|
||||
configuration file (e.g. `addons.conf`) listing all other addons with
|
||||
2. **Phase 2 (manifest driven)** — the manifest extension contains a
|
||||
configuration file (e.g. `extensions.conf`) listing all other extensions with
|
||||
their names, verity parameters, and any additional metadata. The mount
|
||||
script reads this file and mounts the remaining addons accordingly.
|
||||
script reads this file and mounts the remaining extensions accordingly.
|
||||
|
||||
This approach has several advantages:
|
||||
|
||||
- **Kernel command line stays fixed-size** regardless of how many addons are
|
||||
- **Kernel command line stays fixed-size** regardless of how many extensions are
|
||||
composed.
|
||||
- **Attestation is simplified** — the TEE firmware measures one manifest
|
||||
hash on the command line; the verifier only needs to validate that single
|
||||
hash. The chain of trust extends from the measured kernel command line to
|
||||
the verified manifest to the verified addons.
|
||||
the verified manifest to the verified extensions.
|
||||
- **Richer metadata** — the manifest can carry structured information beyond
|
||||
verity parameters: addon ordering constraints, version requirements,
|
||||
verity parameters: extension ordering constraints, version requirements,
|
||||
policies, or init-data payloads.
|
||||
|
||||
The design of the systemd units and mount script can support both modes:
|
||||
if per-addon verity parameters are present on the kernel command line, they
|
||||
are used directly (current behavior); if a manifest addon is present, it
|
||||
is consulted for the remaining addons. This makes the manifest mode an
|
||||
if per-extension verity parameters are present on the kernel command line, they
|
||||
are used directly (current behavior); if a manifest extension is present, it
|
||||
is consulted for the remaining extensions. This makes the manifest mode an
|
||||
additive, backward-compatible evolution.
|
||||
|
||||
### Addon versioning and attestation
|
||||
### Extension versioning and attestation
|
||||
|
||||
Future work may add version metadata to addon images, enabling the
|
||||
Future work may add version metadata to guest extension images, enabling the
|
||||
attestation flow to verify not just integrity (via dm-verity) but also that
|
||||
the specific expected version of each component is present.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user