The multi-layer EROFS rootfs feature relies on QEMU's VMDK flat-extent
driver to merge multiple EROFS layers into a single virtual block
device. Replace --disable-vmdk with an explicit --enable-vmdk so the
Kata static QEMU build includes VMDK support.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Document the end-to-end workflow for using the containerd EROFS
snapshotter with Kata Containers runtime-rs, covering containerd
configuration, Kata QEMU settings, and pod deployment examples
via crictl/ctr/Kubernetes.
Include prerequisites (containerd >= 2.2, runtime-rs main branch),
QEMU VMDK format verification command, architecture diagram,
VMDK descriptor format reference, and troubleshooting guide.
Note that Cloud Hypervisor, Firecracker, and Dragonball do not
support VMDK block devices and are currently unsupported for
fsmerged EROFS rootfs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Refactor the multi-layer EROFS storage handling to improve code
maintainability and reduce duplication.
Key changes:
(1) Extract update_storage_device() to unify device state management
for both multi-layer and standard storages
(2) Simplify handle_multi_layer_storage() to focus on device creation,
returning MultiLayerProcessResult struct instead of managing state
(3) Unify the processing flow in add_storages() with clear separation:
(4) Support multiple EROFS lower layers with dynamic lower-N mount paths
(5) Improve mkdir directive handling with deferred {{ mount 1 }}
resolution
This reduces code duplication, improves readability, and makes the
storage handling logic more consistent across different storage types.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce MultiLayerErofsHandler and method of handle_multi_layer_storage
for multi-layer storage:
(1) Register MultiLayerErofsHandler to STORAGE_HANDLERS to handle
multi-layer EROFS storage with driver type 'multi-layer-erofs'.
(2) Add handle_multi_layer_erofs function to process multiple EROFS
storages with X-kata.multi-layer marker together in guest.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add multi_layer_erofs.rs implementing guest-side processing logics
of multi-layer EROFS rootfs with overlay mount support.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add handling for multi-layer EROFS rootfs in RootFsResource
handler_rootfs method. It will correctly handle the multi-layers
erofs rootfs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add erofs_rootfs.rs implementing ErofsMultiLayerRootfs for
multi-layer EROFS rootfs with VMDK descriptor generation.
It's the core implementation of Erofs rootfs within runtime.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Change Rootfs::get_storage to return Option<Vec<Storage>>
to support multi-layer rootfs with multiple storages.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add format argument to hotplug_block_device for flexibly specifying
different block formats.
With this, we can support kinds of formats, currently raw and vmdk are
supported, and some other formats will be supported in future.
Aside the formats, the corresponding handling logics are also required
to properly handle its options needed in QMP blockdev-add.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
In practice, we need more kinds of block formats, not limited to `Raw`.
This commit aims to add BlockDeviceFormat enum for kinds of block device
formats support, like RAW, VMDK, etc. And it will do some following actions
to make this changes work well, including format field in BlockConfig.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add RUNTIME_ALLOW_MOUNTS annotation to RuntimeInfo to specify
custom mount types allowed by the runtime.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Users were confused about which configuration file to edit because
kata-deploy copied the base config into a per-shim runtime directory
(runtimes/<shim>/) for config.d support, leaving the original file
in place untouched. This made it look like the original was the
authoritative config, when in reality the runtime was loading the
copy from the per-shim directory.
Replace the original config file with a symlink pointing to the
per-shim runtime copy after the copy is made. The runtime's
ResolvePath / EvalSymlinks follows the symlink and lands in the
per-shim directory, where it naturally finds config.d/ with all
drop-in fragments. This makes it immediately obvious that the
real configuration lives in the per-shim directory and removes the
ambiguity about which file to inspect or modify.
During cleanup, the symlink at the original location is explicitly
removed before the runtime directory is deleted.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Now, we include the nvrc.smi.srs=1 flag in the default kernel cmdline.
Thus, we can remove the guidance for people to add it themselves when
not using attestation. In fact, users don't really need to know about
this flag at all.
Signed-off-by: Tobin Feldman-Fitzthum <tfeldmanfitz@nvidia.com>
Fix all clippy warnings triggered by -D warnings:
- install.rs: remove useless .into() conversions on PathBuf values
and replace vec! with an array literal where a Vec is not needed
- utils/toml.rs: replace while-let-on-iterator with a for loop and
drop the now-unnecessary mut on the iterator binding
- main.rs: replace match-with-single-pattern with if-let in two
places dealing with experimental_setup_snapshotter
- utils/yaml.rs: extract repeated serde_yaml::Value::String key into
a local variable, removing needless borrows on temporary values
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Ensure code formatting and compilation are verified early in the
Docker build pipeline, before tests and the release build.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Follow-on to kata-containers/kata-containers#12396
Switch SNP config from initrd-based to image-based rootfs with
dm-verity. The runtime assembles the dm-mod.create kernel cmdline
from kernel_verity_params, and with kernel-hashes=on the root hash
is included in the SNP launch measurement.
Also add qemu-snp to the measured rootfs integration test.
Signed-off-by: Amanda Liem <aliem@amd.com>
With static_sandbox_resource_mgmt calculation fixed for runtime-rs, the
VM is correctly pre-sized at creation time. The vCPU allocation test no
longer depends on CPU hotplug, so the qemu-coco-dev* skip is no longer
needed.
Fixes: #10928
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
With static_sandbox_resource_mgmt now enabled for ARM on runtime-rs,
the VM is correctly pre-sized at creation time. The vCPU allocation
test no longer depends on CPU hotplug, so the aarch64 skip (issue
#10928) is no longer needed.
Fixes: #10928
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
runtime-rs memory hotplug hard-codes the `pc-dimm` device driver, which
is an x86-only QEMU device model. On s390x, the `s390-ccw-virtio`
machine type does not support `pc-dimm` at all — the Go runtime handles
this by using `virtio-mem-ccw` instead (controlled by the
`enable_virtio_mem` config knob, defaulting to true on s390x).
runtime-rs has no virtio-mem support, so any attempt to dynamically
hotplug memory on s390x fails with:
'pc-dimm' is not a valid device model name
This is a pre-existing limitation on main — it has never worked. It is
now visible because commit 45dfb6ff25 ("runtime-rs: Fix initial vCPU /
memory with static_sandbox_resource_mgmt") expanded runtime-rs test
coverage, causing k8s-memory.bats and k8s-oom.bats to actually exercise
this code path on s390x.
Let's enforce using static_sandbox_resources_mgmt also for s390x so the
VM is sized upfront at creation time, bypassing the broken dynamic
hotplug path entirely.
If someone decides to implement hotplug support for s390x, the work
would basically be an implemntation of virtio-mem-ccw support in the
runtime-rs QEMU backend (boot-time device creation, qom-set based
resize, and virtio-mem aware memory accounting), mirroring what the Go
runtime already does, but I'm not game for this (sorry).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
runtime-rs lacks several features needed for CPU hotplug on ARM:
pflash/UEFI firmware passthrough, SMP topology in -smp, nr_cpus
kernel parameter, and QMP vCPU add handling for the virt machine
type (which requires core-id only placement with socket/thread/die
set to -1).
Without static sandbox resource management, these gaps cause
failures in tests like k8s-memory.bats where the VM is not correctly
sized for the workload.
Enable static_sandbox_resource_mgmt for aarch64 in the QEMU
runtime-rs configuration so the VM is pre-sized at creation time,
sidestepping the need for hotplug entirely.
Together with this we're aligning the go runtime to the very same
behaviour.
Fixes: #10928
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
InitialSizeManager::setup_config() is responsible for applying the
sandbox workload sizing (computed from containerd/CRI-O sandbox
annotations) to the hypervisor configuration before VM creation.
Previously, the workload vCPU count was only logged but never actually
added to default_vcpus, so the VM was always created with only the base
vCPUs from the configuration/annotations. This caused the
k8s-sandbox-vcpus-allocation test to fail with qemu-snp-runtime-rs:
a pod with default_vcpus=0.75 and a container CPU limit of 1.2 should
see ceil(0.75 + 1.2) = 2 vCPUs, but only got 1.
Additionally, the workload memory was being added to default_memory
unconditionally, diverging from the Go runtime which only applies both
CPU and memory additions when static_sandbox_resource_mgmt is enabled.
In the non-static path, adding workload resources here would cause
double-counting: once from setup_config() at sandbox creation, and
again from update_cpu_resources()/update_mem_resources() when
individual containers are added.
Guard both additions behind static_sandbox_resource_mgmt, matching the
Go runtime's behavior in src/runtime/pkg/oci/utils.go:
if sandboxConfig.StaticResourceMgmt {
sandboxConfig.HypervisorConfig.NumVCPUsF += sandboxConfig.SandboxResources.WorkloadCPUs
sandboxConfig.HypervisorConfig.MemorySize += sandboxConfig.SandboxResources.WorkloadMemMB
}
Fixes: k8s-sandbox-vcpus-allocation test failure on qemu-snp-runtime-rs
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Snapshotters (nydus, erofs) are containerd-specific. The validation code
already warned that EXPERIMENTAL_SETUP_SNAPSHOTTER would be ignored on
CRI-O, but the actual install/configure and uninstall loops still ran
unconditionally, attempting containerd-specific operations on CRI-O
nodes.
Guard both the install and cleanup snapshotter loops with a `runtime !=
"crio"` check so the binary itself skips snapshotter work when it
detects CRI-O as the container runtime.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Joji's added the labels for the default values.yaml, but we missed
adding those to the nvidia specific values.yaml file.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
When do cargo fmt --all, some files changes as unformatted with
`cargo fmt`. This commit is just to address it.
Just use this as an example:
```
// Generate the common drop-in files (shared with standard
// runtimes)
- write_common_drop_ins(config, &runtime.base_config,
&config_d_dir, container_runtime)?;
+ write_common_drop_ins(
+ config,
+ &runtime.base_config,
+ &config_d_dir,
+ container_runtime,
+ )?;
```
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Without those, we'd end up pulling the same / old rootfs that's cached
without re-building it in case of a bump in any of those components.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Remove the msize_9p field from HypervisorInfo struct and
get_hypervisor_info() function in kata-ctl tool.
This aligns with the removal of 9p filesystem support from
the configuration and agent.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove the Virtio9pHandler implementation and its registration
from the storage handler manager:
(1) Remove Virtio9pHandler struct and StorageHandler implementation.
(2) Remove DRIVER_9P_TYPE and Virtio9pHandler from STORAGE_HANDLERS
registration.
(3) Update watcher.rs comments to remove 9p references.
This completes the removal of virtio-9p support in the agent.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove all virtio-9p related code and configurations:
(1) Remove DRIVER_9P_TYPE and VIRTIO_9P.
(2) Remove 9p validation and adjustment logic from SharedFsInfo.
(3) Remove KATA_ANNO_CFG_HYPERVISOR_MSIZE_9P annotation handling.
(4) Update test configurations to remove msize_9p settings.
(5) Update documentation and proto comments.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
(1) Update annotations documentation to reflect new shared filesystem
options (virtio-fs, inline-virtio-fs, virtio-fs-nydus, none).
(2) Replace virtio-9p references with inline-virtio-fs in config doc.
(3) Update drop-in configuration tests to use 'none' instead of 'virtio-9p'
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
As virtio-9p is never supported in runtime-rs, we have more choices to
replace it with blockfile snapshotter or erofs snapshotter(in future).
It's time to remove its documents and reduce misleading guidance.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
- Updated image-rs from rev 026694d4 to tag v0.18.0
- This update brings rsa 0.9.10 which fixes CVE-2026-21895
- Resolves vulnerability in indirect dependencies
Signed-off-by: pavithiran34 <pavithiran.p@ibm.com>
