Add fields to DmVerityInfo needed for dm-verity device creation:
(1) salt: Optional salt value for the hash computation
(2) hash_type: dm-verity version
(3) no_superblock: whether to skip the superblock at hash offset
Uses serde defaults for backward compatibility with existing serialized
data that lacks these fields.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Now that the workspace version has been updated,
switch the mem-agent to pick up the new workspace version
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Upgrade the nix crate across the workspace to version 0.30.1 to address
security vulnerabilities and adopt safer file descriptor handling patterns.
### Breaking Changes in nix 0.28.0
1. **File Descriptor Type Changes**
- Functions now return `OwnedFd` instead of `RawFd` (i32)
- Functions requiring file descriptors now expect types implementing `AsFd` trait
- This provides RAII-based automatic cleanup and prevents fd leaks
2. **API Signature Changes**
- `pipe()`, `pipe2()`, `openpty()` now return `OwnedFd` tuples
- `socket()` returns `OwnedFd` instead of `RawFd`
- `open()`, `memfd_create()` return `OwnedFd`
- `setns()`, `write()`, `fcntl()` require `AsFd` trait
- `madvise()` requires `NonNull<c_void>` instead of raw pointer
- `bind()`, `listen()`, `connect()` require `AsFd` and `Backlog` type
3. **Module Feature Flags**
- Modules now require explicit feature flags (mman, reboot, etc.)
### Additional Breaking Changes in nix 0.30.1
1. **symlinkat() API Change**
- `dirfd` parameter now requires `AsFd` trait instead of `Option<RawFd>`
- Use `BorrowedFd::borrow_raw(libc::AT_FDCWD)` for current directory
2. **Type Alias Deprecation**
- `MemFdCreateFlag` renamed to `MFdFlags` for consistency
### Changes Made
**Workspace Configuration (Cargo.toml)**
- Updated nix to 0.30.1 with features: fs, mount, sched, process, ioctl,
signal, socket, feature, user, hostname, term, event, mman, reboot
**File Descriptor Handling Patterns**
- Use `BorrowedFd::borrow_raw(raw_fd)` to wrap RawFd for AsFd requirements
- Use `.as_fd().as_raw_fd()` to extract raw fd without ownership transfer
- Use `.into_raw_fd()` only when ownership transfer is needed
- Use `NonNull::new().unwrap()` for madvise pointer conversion
**Deprecated API Replacements**
- `eventfd()` → `EventFd::from_value_and_flags()`
- `Errno::from_i32()` → `Errno::from_raw()`
- `listen(fd, backlog)` → `listen(&fd, Backlog::new(backlog).unwrap())`
- `MemFdCreateFlag` → `MFdFlags`
Generated by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
We have a note in the workspace Cargo.toml that the version
there needs to be in sync with the libs versions, so just update
them to use the workspace version rather than manually managing this.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The unit tests added by the previous commit exposed a malfunction of the
byte-unit crate on big-endian systems(*), causing s390x CI to fail.
Bump the dependency's version to include a fix.
Signed-off-by: Pavel Mores <pmores@redhat.com>
The annotation value is implicitly in MiB but when presented to the
byte-unit crate this is interpreted as bytes. When a common value like
2048, meant to mean 2048 MiB but interpreted as 2048 B, is then converted
to MiB the result is zero which is less than the minimal allowable memory
and the runtime fails to launch.
This is fixed by adding a detection whether the annotation value contains
units or not. If it doesn't it's first converted to MiB and the rest of
the processing then goes like before.
This way we allow for the implicit MiB units when no units are given, thus
keeping compatibility with existing go shim behaviour, while also allowing
for any legal units to be given as well.
We take the opportunity to add some unit tests as well.
Signed-off-by: Pavel Mores <pmores@redhat.com>
Declare the `visible_cdi_devices` agent option (kernel param
agent.visible_cdi_devices) in kata-types so runtime-rs can opt into
emitting it to the guest, and expose it in the three NVIDIA GPU
configuration templates (qemu, qemu-snp, qemu-tdx) at runtime-rs/config/.
The agent consumes the corresponding VISIBLE_CDI_DEVICES env var to
drive CDI device requests.
Signed-off-by: LandonTClipp <lclipp@coreweave.com>
While the config knob is being parsed, it is being unused in the
rust shim. This renders the config knob useless. Remove the
file_mem_backend config option as there is no current users for it.
As this option is being usable in the go shim, we leave it intact.
For the rust shim, /dev/shm is still being used in a similar way to
the go shim when filesystem sharing is enabled (virtio-fs). Future
use cases where other file_mem_backends are being utilized are
currently planning to define these backends in a similar manner:
based on the configuration/platform, determine the proper file
memory backend, but do not let end users determine the file memory
backend.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
When using virtio-fs with nydusd's passthrough_fs, mkdir operations may
return ENOSYS on certain filesystem configurations. This causes mount
destination creation to fail unexpectedly.
Handle ENOSYS errors gracefully alongside AlreadyExists by verifying the
directory exists after the failed mkdir attempt, allowing the mount to
proceed if the directory is already present.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add "virtio-fs-nydus" as a recognized shared filesystem type in the
hypervisor configuration. This enables the standalone nydusd mode where
nydusd runs as a separate process alongside virtiofsd.
The key changes:
(1) Add VIRTIO_FS_NYDUS constant for the new shared fs type.
(2) Register virtio-fs-nydus in adjust() and validate() paths, reusing
the same virtio-fs validation logic since both use vhost-user protocol
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
To make it more flexible when users want to set this feature, one
more way to make it valid is via annotations.
The dedicated annnotation of
"io.katacontainers.config.hypervisor.indep_iothreads" is introduced
within k8s clusters.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The 'indep_iothreads' field is introduced in Hypervisor to make it
configurable for number of independent IO threads for virtio-blk
devices. When set to a value greater than 0, creates independent
IO threads that can be attached to virtio-blk devices during hotplug.
Note that it requires 'enable_iothreads' to be true for virtio-blk
devices to use these threads.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Rust 1.94 now warns about unnecessary unsafe blocks around
x86_64::__cpuid() calls. Remove the unsafe blocks as they are
no longer needed.
This fixes the following clippy warnings:
- warning: unnecessary `unsafe` block at line 129
- warning: unnecessary `unsafe` block at line 142
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Generated-By: IBM Bob
When a VF is cold-plugged in guest-kernel mode, mlx5_core binds to the
PCI device inside the VM and mlx5_ib creates IB character devices under
/dev/infiniband/ (uverbs*, rdma_cm, umad*). The container cannot reach
these devices unless they are explicitly added to its OCI spec.
Add expose_guest_infiniband_devices(), called from create_devices() when
the container carries at least one VFIO device entry. The function:
- Walks /dev/infiniband/ inside the guest VM.
- Appends each char device to spec.linux.devices.
- Inserts matching cgroup allow rules (rwm).
- Is a no-op if /dev/infiniband/ is absent or empty (no IB driver,
or VF not yet rebound), so non-RDMA pods are unaffected.
Gate the call on container_has_vfio_device() so unrelated containers
sharing the sandbox do not get IB device access widened.
Add is_vfio_device_type() and snapshot_infiniband() to
kata-sys-util/pcilibs. is_vfio_device_type() lets the agent check
device type strings against the VFIO driver name constants without
duplication. snapshot_infiniband() summarises /sys/class/infiniband,
/sys/class/infiniband_verbs, and /dev/infiniband as a single diagnostic
string for log context; it lives in pcilibs because it has no
agent-specific dependencies (pure sysfs/devfs reads).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Introduce gpt_disk.rs to compute GPT partition layouts and generate
metadata files for multi-layer EROFS rootfs. The module creates GPT
head metadata that are combined with EROFS layer images via VMDK
descriptors, presenting a single GPT-partitioned virtual disk to the
guest VM — each EROFS layer mapped to its own partition.
The layout engine calculates LBA positions for an arbitrary number of
EROFS layers, then writes a full protective-MBR + GPT image and extracts
the head (MBR + primary GPT table) segments as standalone files for
VMDK extent assembly.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Two tests relied on the side-effect of create_dir_all (removed in
the previous commit) to pass:
(1) test_get_uds_with_sid_ok: use a directory name that actually
starts with the search prefix so prefix matching works without
creating dirs.
(2) test_get_uds_with_sid_with_zero: assert Err on zero matches
instead of Ok, matching the corrected lookup behavior.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
When running kata-ctl exec <short-id>, kata-ctl may fail with:
"more than one sandbox exists with the provided prefix "ed07",
please provide a unique prefix".
At the same time, a new subdirectory named <short-id> is incorrectly
created under /run/kata/. This is wrong behavior: a short ID should be
used only to match an existing sandbox by prefix, and must not trigger
creation of a new sandbox directory when lookup fails or is ambiguous.
Update the exec path to perform prefix matching and return an error on
no match or non-unique matches, without creating any new directories.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
sb_storage_path() is a path accessor shared by both server (shim) and
client (kata-ctl). Having it call create_dir_all(KATA_PATH) on every
invocation is incorrect: the client side should never create directories
— if /run/kata/ does not exist, no shim is running.
Move the directory creation to MgmtServer::new(), which is the server-
side component that manages the shim management socket under KATA_PATH.
Make sb_storage_path() a pure accessor returning &'static str directly.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The initdata is currently being decoded, and then re-encoded with the
to_string function. This will usually not preserve the original initdata
document, and thus the initdata hash will differ between the annotation
and the block device.
This commit changes the logic to only decode the base64, but keep the
initdata document intact. Since the error message is now nested, adjust
the tests to look for the expected error in the chain.
Fixes: #12951
Signed-off-by: Markus Rudy <mr@edgeless.systems>
When emptydir_mode is "block-encrypted", host emptyDir paths must
remain as "bind" mounts so the EncryptedEmptyDirVolume handler can
intercept them in the volume dispatch chain. Previously,
update_ephemeral_storage_type() would unconditionally convert them
to "local" type, causing them to be handled as plain local volumes
instead.
Add the emptydir_mode parameter to update_ephemeral_storage_type()
and its call chain (amend_spec in container.rs) and skip the
host-emptyDir-to-local conversion when the mode is block-encrypted.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add the core volume handler for block-encrypted emptyDir support
in runtime-rs, bringing it to parity with the Go runtime (PR #10559).
When emptydir_mode is set to "block-encrypted", host emptyDir bind
mounts are intercepted and handled as follows:
1. A sparse disk image (disk.img) is created inside the emptyDir
folder, sized to match the host filesystem capacity.
2. A mountInfo.json is written under the kata direct-volume root
with volume_type "blk", fs_type "ext4", and metadata
encryptionKey=ephemeral.
3. The disk image is plugged into the guest VM as a virtio-blk
device via the hypervisor device manager.
4. An agent::Storage is built with driver_options containing
encryption_key=ephemeral and shared=true, so the kata-agent
delegates formatting and encryption to CDH using LUKS2.
The volume is registered in the dispatch chain before the regular
block-volume check, and ephemeral disk metadata is tracked for
sandbox-level cleanup at teardown.
Also re-exports EMPTYDIR_MODE_* constants from kata-types::config
so downstream crates can reference them.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add add_volume_mount_info(), is_volume_mounted(), and
remove_volume_path() to the mount module. These mirror the Go
helpers (AddMountInfo, IsVolumeMounted, Remove) in
src/runtime/pkg/direct-volume/utils.go and are needed by the
upcoming EncryptedEmptyDirVolume to write and clean up
mountInfo.json metadata for block-encrypted emptyDir volumes.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Add the emptydir_mode field to the Runtime configuration struct,
allowing runtime-rs to read the emptyDir handling mode from the
TOML config file. This is groundwork for trusted ephemeral data
storage support in runtime-rs (parity with the Go runtime).
Two modes are supported:
- shared-fs (default): share emptyDir via virtio-fs/9p.
- block-encrypted: plug a block device encrypted in-guest via
CDH/LUKS2.
Empty values default to "shared-fs"; unknown values are rejected
during validation.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
The cdh_api_timeout_ms configuration parameter wasn't being used
anywhere, so add the logic to process it as an annotation into the runtime-rs
agent config and then use that as a kernel_param.
Assisted-by IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add the cdh_api_timeout_ms parameter to runtime-rs. Note the go runtime
has cdh_api_timeout is seconds, but the suggestions was to move everything
to ms. This parameter controls the timeout for Confidential Data
Hub API timeout in the agent.
Changes:
- Add cdh_api_timeout_ms field to Agent config struct in kata-types
- Defaults to 50,000 milliseconds internally
- Add cdh_api_timeout to confidential computing configuration files:
- configuration-qemu-coco-dev-runtime-rs.toml.in
- configuration-qemu-se-runtime-rs.toml.in
- configuration-qemu-snp-runtime-rs.toml.in
- configuration-qemu-tdx-runtime-rs.toml.in
The parameter is only added to confidential computing configurations
where the Confidential Data Hub is used.
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
So many unformatted rust codes cause uncommitted change files in
rust runtime and its libs or agent sources, which can be easily
found just by `cargo fmt --all`.
Let's reduce such noisy bad experiences
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add a new runtime-rs configuration template that combines the NVIDIA GPU
cold-plug stack with Intel TDX confidential guest support. This is the
runtime-rs counterpart of the Go runtime's configuration-qemu-nvidia-gpu-tdx
template.
The template merges the GPU NV settings (VFIO cold-plug, Pod Resources API,
NV-specific kernel/image/firmware, extended timeouts) with TDX confidential
guest settings (confidential_guest, OVMF.inteltdx.fd firmware, TDX Quote
Generation Service socket, confidential NV kernel and image).
The Makefile is updated with the new config file registration and the
FIRMWARETDVFPATH_NV variable pointing to OVMF.inteltdx.fd.
Also removes a stray tdx_quote_generation_service_socket_port setting
from the SNP GPU template where it did not belong.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Query the kubelet Pod Resources API during sandbox setup to discover
which GPU devices have been allocated to the pod. When cold_plug_vfio
is enabled, the sandbox resolves CDI device specs, extracts host PCI
addresses and IOMMU groups from sysfs, and creates VfioModernCfg
device entries that get passed to the hypervisor for cold-plug.
Add pod-resources and cdi crate dependencies to the runtimes and
virt_container workspace members.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The VFIO cold-plug path needs to resolve a PCI device's sysfs address
from its /dev/vfio/ group or iommufd cdev node. Extend the PCI helpers
in kata-sys-util to support this: add a function that walks
/sys/bus/pci/devices to find a device by its IOMMU group, and expose the
guest BDF that the QEMU command line will reference.
These helpers are consumed by the runtime-rs hypervisor crate when
building VFIO device descriptors for the QEMU command line.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
The Go runtime already exposes a [runtime] pod_resource_api_sock option
that tells the shim where to find the kubelet Pod Resources API socket.
The runtime-rs VFIO cold-plug code needs the same setting so it can
query assigned GPU devices before the VM starts.
Add the field to RuntimeConfig and wire it through deserialization so
that configuration-*.toml files can set it.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Add a gRPC client crate that speaks the kubelet PodResourcesLister
service (v1). The runtime-rs VFIO cold-plug path needs this to discover
which GPU devices the kubelet has assigned to a pod so they can be
passed through to the guest before the VM boots.
The crate is intentionally kept minimal: it wraps the upstream
pod_resources.proto, exposes a Unix-domain-socket client, and
re-exports the generated types.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
`make vendor` isn't required anymore. People who need vendored code should
use the `tools/packaging/release/generate_vendor.sh` script instead.
Assisted-by: Claude AI
Signed-off-by: Greg Kurz <groug@kaod.org>
Docker 26+ with `runtimeType` shims may not include a network
namespace in the OCI spec's `linux.namespaces` and instead uses
`libnetwork-setkey` hooks to communicate the sandbox ID. Add helpers
to detect Docker containers and resolve the netns path from hook
arguments, matching the Go runtime's `DockerNetnsPath` and
`IsDockerContainer` utilities.
Fixes: #9340
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Remove NixPath's is_empty() to avoid ambiguity with the future
std::path::Path::is_empty() method and switch to
path.as_os_str().is_empty as recommended by @burgerdev.
This addresses unstable_name_collisions warnings in Rust 1.93.
Fixes: #12835
Co-authored-by: Markus Rudy <mr@edgeless.systems>
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Port the Go runtime's enable_vcpus_pinning feature to runtime-rs.
The Go runtime already lets users pin each vCPU thread to a specific
host CPU when the vCPU count matches the sandbox cpuset size, using
sched_setaffinity. This is useful for latency-sensitive workloads that
benefit from eliminating cross-CPU migration of vCPU threads.
The approach mirrors the Go implementation:
After VM start and on every container add/update/delete, we fetch the
vCPU thread IDs (via QMP query-cpus-fast for QEMU), compute the union of
all containers' OCI cpusets, and if the two counts match, pin vCPU i to
cpuset[i]. If they diverge (hotplug, container removal, etc.) we reset
all threads back to the full cpuset so nothing gets stuck on a single
core.
The pinning check lives in CgroupsResourceInner::update_sandbox_cgroups,
which already runs at exactly the right points in the lifecycle. The
enable_vcpus_pinning flag flows from the TOML config through
CgroupConfig into the cgroup resource layer, and can also be overridden
per-pod via the io.katacontainers.config.runtime.enable_vcpus_pinning
annotation.
The QEMU config templates default to false. The NV GPU configs will get
their own default (true) in a follow-up once those templates are added.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Update all versions of rand that are controlled by us to remediate
GHSA-cq8v-f236-94qc.
Note: There are still some usages of rand 0.8.5 it that are from
transitive dependencies which we can't currently update:
- fail
- phf_generator
- opentelemetry
due to them being archived, or our usage being 17 versions out of date
Also update the rand API breakages e.g. :
- rand::thread_rng() → rand::rng() (function renamed)
- rand::distributions::Alphanumeric → rand::distr::Alphanumeric (module renamed)
- rng.gen_range() → rng.random_range() (function renamed)
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add a new extensible GetDiagnosticData RPC that retrieves diagnostic
information from the guest VM. The request carries a log_type string
field to specify what kind of data is requested, and a container_id
field to identify the target container.
The first supported log_type is "termination_log", which reads the
Kubernetes termination message file from inside the guest. This is
needed for shared_fs=none configurations where the host cannot
directly access the guest filesystem.
On the Go runtime side, the container stop() path now calls
GetDiagnosticData to copy the termination message to the host
when running with NoSharedFS and the terminationMessagePolicy
annotation is set to "File". The call is best-effort: failures
are logged as warnings rather than blocking container teardown.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Silenio Quarti <silenio_quarti@ca.ibm.com>
The hardcoded DEFAULT_LAUNCH_PROCESS_TIMEOUT of 6 seconds in the kata
agent is insufficient for environments with NVIDIA GPUs and NVSwitches,
where the attestation-agent needs significantly more time to collect
evidence during initialization (e.g. ~2 seconds per NVSwitch).
When the timeout expires, the agent (PID 1) exits with an error, causing
the guest kernel to perform an orderly shutdown before the
attestation-agent has finished starting.
Make this timeout configurable via the kernel parameter
agent.launch_process_timeout (in seconds), preserving the 6-second
default for backward compatibility. The Go runtime is wired up to pass
this value from the TOML config's [agent.kata] section through to the
kernel command line.
The NVIDIA GPU configs set the new default to 15 seconds.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
