Enable devicemapper (dm-verity) support in CI by forwarding the
USE_DEVMAPPER environment variable to the agent build step across
all architectures (amd64, arm64, s390x).
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The deploy will read EROFS_SNAPSHOTTER_MODE from the environment and,
when set to "integrity", disable fsverity/immutable and enable
dmverity_mode and enable_dmverity in the containerd erofs
snapshotter/differ config.
Add validation for the mode value and use an explicit 300s timeout
for node-readiness checks during kata-deply in github CI.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Install libdevmapper-dev and pkg-config in the agent build container
so devicemapper-sys can link against libdevmapper. Add the GNU libc
rustup target alongside musl since USE_DEVMAPPER forces LIBC=gnu.
Forward USE_DEVMAPPER through build.sh and build-static-agent.sh.
And you can compile the device mapper in kata-agent as below:
```
$ make LIBC=gnu USE_DEVMAPPER=yes
```
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Pass USE_DEVMAPPER through the Docker environment in local build
scripts. Extract the OCI tag sanitization logic into a public helper
of sanitize_tag_component to keep push and pull paths consistent.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Expose erofsSnapshotterMode in the helm chart values and render it as
the EROFS_SNAPSHOTTER_MODE environment variable in the kata-deploy pod.
Update gha-run-k8s-common.sh to load dm-mod/dm-verity kernel modules
and configure the erofs default size when the mode is "integrity".
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add k8s-erofs-dmverity.bats integration test that verifies dm-verity
protected EROFS layers work end-to-end, and register the integrity
mode in the CoCo test matrix.
This commit introduces two new files to enable it:
- k8s-erofs-dmverity.bats
- pod-erofs-dmverity-probe.yaml
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Replace the agent's inline devicemapper implementation with the libs
kata-types::dmverity module. The agent's devicemapper Cargo feature
now forwards to kata-types/devicemapper, removing the direct
libdevmapper link dependency from the agent crate. Gate all dm-verity
imports, constants, and call sites behind libdevmapper.
Add USE_DEVMAPPER Makefile variable (default no) that appends the
devicemapper feature flag and forces LIBC=gnu when enabled.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
This commit is just a enhancement without any functionality changes.
Replace the sequential loop in handle_multi_layer_erofs_group with
join_all-based concurrent mounting. Base device paths and mount
directories are pre-resolved before spawning futures to avoid lock
contention. On partial failure, successfully mounted layers are
unmounted and dm-verity devices cleaned up before propagating the
error.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce a new `dmverity` module in kata-types that provides dm-verity
device creation, destruction and lifecycle management via devicemapper
ioctls. The module is conditionally compiled behind the `devicemapper`
feature flag, which also pulls in tokio for async device-node polling.
The workspace devicemapper dependency is pinned to a specific git
revision for reproducible builds.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
When a GPT-partitioned VMDK is split into individual partition images,
padding files may be generated between partitions to maintain correct
byte offsets. These were not tracked for cleanup, leading to stale
temporary files after container removal.
Iterate over the partition layout and check for pad-{idx}.img files
alongside the head image; add any that exist to gpt_metadata_paths
so they are removed during teardown.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Wire the dm-verity helpers into the layer mount flow so that GPT
partitions carrying verity metadata are mounted through a verified
device-mapper target instead of the raw partition.
Refactor wait_and_mount_layer to resolve partition path and verity
device as separate steps: create a dm-verity device when
X-kata.dmverity-enabled=true is set, fall back to direct partition
mount otherwise, and return the verity device path for cleanup
tracking.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add per-container verity_devices tracking in Sandbox and wire the
teardown path: destroy_partition_dmverity_device removes the
device-mapper target via deferred-remove ioctl and deletes the mknod
node, cleanup_dmverity_devices iterates all devices in reverse order.
Wire into remove_container_resources (rpc.rs) so verity devices are
torn down after unmount, and record verity device paths in
add_storages (storage/mod.rs) for tracking.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
GPT-partitioned EROFS layers can carry dm-verity hashes appended after
the filesystem data within the same partition. The host runtime passes
the root hash and parameters as X-kata.dmverity.* storage options; the
agent must set up the kernel dm-verity target before mounting so that
every read is integrity-checked against the Merkle tree.
Implement dm-verity device creation: option parsing from storage
options, device name generation, and create helper via devicemapper
ioctls with hash_start_block calculation (accounting for v1 superblock
presence).
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The kata guest VM runs without udev, so device-mapper nodes under
/dev/mapper are never created automatically. Add the foundational
helpers that subsequent dm-verity integration will rely on:
It focus on the following key points:
(1) DmOptions builders that disable all udev synchronization flags,
with read-only and deferred-remove variants.
(2) mknod-based device node creation/removal under /dev/mapper, since
devtmpfs nodes are not auto-created without udev.
Also add the devicemapper crate dependency (default-features = false).
But note that the commit depends on device mapper with no-udev support
with the PR:https://github.com/stratis-storage/devicemapper-rs/pull/1036
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Extract dm-verity metadata from containerd mount annotations and pass
them through to kata-agent as X-kata.dmverity.* storage options. This
enables the agent to create dm-verity devices for integrity-verified
EROFS partitions.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
When containerd creates dm-verity-protected EROFS layers, it stores
the root hash and parameters as OCI annotations — but the format
does not directly map to the kernel dm-verity table that the guest
agent needs to construct.
Bridge this gap with functions that parse containerd's dm-verity
annotation JSON, detect whether a v1 superblock is embedded at the
hash offset (to extract the salt automatically rather than relying
on containerd's hardcoded default), and produce the X-kata.dmverity.*
storage options the agent expects.
This keeps all dm-verity metadata translation on the host side, so
the agent can consume a flat list of options without understanding
the containerd annotation schema.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add fields to DmVerityInfo needed for dm-verity device creation:
(1) salt: Optional salt value for the hash computation
(2) hash_type: dm-verity version
(3) no_superblock: whether to skip the superblock at hash offset
Uses serde defaults for backward compatibility with existing serialized
data that lacks these fields.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Update the composable-vm-images proposal with the design decisions we only
arrived at after experimenting with the implementation:
* Replace the hardcoded agent path-resolution table with the data-driven
components.toml manifest (process levels, args/optional_args, env,
wait_socket, ${...} substitution, and select/variants), keeping the agent
generic.
* Document the attester-variant contract: NVRC exports KATA_ATTESTER_VARIANT
and the manifest selects the stock vs NVIDIA attestation-agent.
* Document the runtime dependency requirements found during bring-up: the
nvidia attester's LD_LIBRARY_PATH (libnvat closure in the coco addon +
NVML in the gpu addon) and the NVML-init failure mode, plus CDH
secure_mount tooling placement -- plain storage (mke2fs/mkfs.ext4/dd) in
the base vs encrypted storage (cryptsetup) in the coco addon, the CDH
PATH, and the base/addon ABI lockstep.
* Reflect the storage tooling and bundled libraries in the base/coco-addon
build sections, and mark the GPU addon as implemented.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The kata-monitor image has no job-dispatcher sidecar, so opt out of the
kata-deploy-specific dispatcher manifest derivation in the
payload-after-push workflow by setting
KATA_DEPLOY_PUBLISH_JOB_DISPATCHER=false, mirroring the same fix already
applied to the release workflows.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
When a container process is terminated by a signal, the agent's SIGCHLD
reaper stored the raw signal number as the process exit code. As a result
a process killed by SIGKILL(9) reported exit code 9 instead of the
conventional 137 (128+9).
Apply the standard shell convention of 128+signal_number so that
signal-terminated processes report the expected exit codes, e.g.
SIGKILL(9) -> 137, SIGTERM(15) -> 143, SIGINT(2) -> 130. This mimics
runc, which encodes wait-status exit codes the same way:
https://github.com/opencontainers/runc/blob/v1.4.3/libcontainer/utils/utils.go#L19
Both runc and this new Kata behaviour follow the conventional exit code
semantics documented at https://tldp.org/LDP/abs/html/exitcodes.html.
The conversion is factored into a small helper and covered by a unit
test. The runtime and shim already pass the exit code through unchanged,
so no further changes are needed for the corrected value to surface.
Fixes: signal-terminated containers reporting raw signal numbers
Signed-off-by: PiotrProkop <pprokop@nvidia.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the "Not yet implemented" stub in QemuInner::remove_device()
with a working implementation that calls hotunplug_device() to perform
the QMP-level device removal, then cleans up the internal devices list
via retain() to remove stale coldplug entries.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce hotunplug_device() as the device-type dispatcher that routes
hot removal requests to the appropriate QMP method. Currently supports
Block and BlockModern device types, which are forwarded to
Qmp::hotunplug_block_device(). All other device types return an
explicit "unsupported" error.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Implement QMP-level block device hot-unplug by issuing device_del to
remove the frontend device and blockdev_del to remove the backend
blockdev node. For virtio-blk-ccw on s390x, the CCW subchannel slot
is also released.
Since QMP device_del is asynchronous and only initiates the removal
request, introduce wait_for_device_deleted() to poll for the
DEVICE_DELETED event before tearing down the backend. This prevents
blockdev_del from failing with "Node is still in use".
If blockdev_del fails, the error is logged but CCW cleanup still
proceeds before the error is propagated, ensuring consistent
subchannel state.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Improve the reliability of block device hotplug by ensuring that
blockdev-add nodes are properly cleaned up when subsequent device_add
operations fail.
To address this, A new method of device_add_with_rollback is introduced
to do device_add and do properly cleaned up when it fails.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The behavior we had before was that, for a starting k8s pod,
it sees enable_template=true and therefore:
1. Tries NewFactory with fetchOnly=true
2. When that fails (because template.Fetch fails to find the artifacts,
we retry with fetchOnly=false. This creates a direct factory
which creates the template from scratch
(hence we pay a full pod sandbox boot time here)
and then restores from that. Hence the boot times
are strictly worse on this path.
Now, even when enable_template=true, we don't try to force a direct factory.
Instead we just revert to the standard sandbox boot path.
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
Add k8s-vm-templating-test.bats which exercises pod create
with the factory initialized on the target node.
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
Add support for VM Template factory on the clh path.
In order to support snapshot/restore-based VM templating,
the following changes were needed:
1. For clh.go, implement SaveVM, PauseVM, restoreVM, ResumeVM
2. Remove initrd config check for VM Templating path. The
root disk image (when using image mode) is created in memory
and therefore captured in the VM snapshot.
3. Truncate the memory file to the size of the VM at factory VM
create time. This allows CLH to use the memory file
as the backing for the template VM memory, allowing O(1)
snapshot times.
4. CLH uses memory zones as backing for its memory on the template paths
5. Update StartVM in CLH to use the restore path when template is
configured and available
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
Kata sizes VM CPU and memory from OCI limits, not Kubernetes resource
requests. Requests are consumed by the Kubernetes control plane, but
they do not drive Kata VM or sandbox sizing today.
Convert the straightforward Kata workload manifests and kata-deploy
examples from resource requests to limits so the declared resources
match the values Kata uses for VM provisioning. Keep requests where the
fixture intentionally validates Kubernetes request/limit behavior.
Update fixture expectations affected by the conversion. The LimitRange
fixture is limit-only at 500m.
Raise the policy deployment limits to 500m and 800Mi. These tests boot
CoCo/runtime-rs sandboxes with policy/initdata, and the former
100m/100Mi values became real runtime limits after the conversion,
which is too constrained for the CI environments.
Leave PVC storage requests, explicit request/limit validation fixtures,
the env resourceFieldRef request, and non-Kata workload examples
unchanged where requests are handled outside the Kata shim resource
sizing path.
If Kata later grows request-aware sandbox sizing, for example through
Sandbox API based resource plumbing, these requests can be reintroduced
where they carry the intended semantics.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI Codex <codex@openai.com>
Now that the workspace version has been updated,
switch the mem-agent to pick up the new workspace version
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Upgrade the nix crate across the workspace to version 0.30.1 to address
security vulnerabilities and adopt safer file descriptor handling patterns.
### Breaking Changes in nix 0.28.0
1. **File Descriptor Type Changes**
- Functions now return `OwnedFd` instead of `RawFd` (i32)
- Functions requiring file descriptors now expect types implementing `AsFd` trait
- This provides RAII-based automatic cleanup and prevents fd leaks
2. **API Signature Changes**
- `pipe()`, `pipe2()`, `openpty()` now return `OwnedFd` tuples
- `socket()` returns `OwnedFd` instead of `RawFd`
- `open()`, `memfd_create()` return `OwnedFd`
- `setns()`, `write()`, `fcntl()` require `AsFd` trait
- `madvise()` requires `NonNull<c_void>` instead of raw pointer
- `bind()`, `listen()`, `connect()` require `AsFd` and `Backlog` type
3. **Module Feature Flags**
- Modules now require explicit feature flags (mman, reboot, etc.)
### Additional Breaking Changes in nix 0.30.1
1. **symlinkat() API Change**
- `dirfd` parameter now requires `AsFd` trait instead of `Option<RawFd>`
- Use `BorrowedFd::borrow_raw(libc::AT_FDCWD)` for current directory
2. **Type Alias Deprecation**
- `MemFdCreateFlag` renamed to `MFdFlags` for consistency
### Changes Made
**Workspace Configuration (Cargo.toml)**
- Updated nix to 0.30.1 with features: fs, mount, sched, process, ioctl,
signal, socket, feature, user, hostname, term, event, mman, reboot
**File Descriptor Handling Patterns**
- Use `BorrowedFd::borrow_raw(raw_fd)` to wrap RawFd for AsFd requirements
- Use `.as_fd().as_raw_fd()` to extract raw fd without ownership transfer
- Use `.into_raw_fd()` only when ownership transfer is needed
- Use `NonNull::new().unwrap()` for madvise pointer conversion
**Deprecated API Replacements**
- `eventfd()` → `EventFd::from_value_and_flags()`
- `Errno::from_i32()` → `Errno::from_raw()`
- `listen(fd, backlog)` → `listen(&fd, Backlog::new(backlog).unwrap())`
- `MemFdCreateFlag` → `MFdFlags`
Generated by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
We have a note in the workspace Cargo.toml that the version
there needs to be in sync with the libs versions, so just update
them to use the workspace version rather than manually managing this.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
