Explains the root cause (40-bit GPA without host-phys-bits=on), the
short-term fix in the Makefile, and the post-refactor path (Q35 Platform
builder should emit host-phys-bits=on unconditionally for x86_64/KVM).
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Four items promised in replies to Copilot review:
virt.rs: add "bytes." to highmem_mmio_size doc comment so the unit is
unambiguous when emitting QEMU highmem-mmio-size=.
platform.rs: clarify BaseMachine::memory_backend doc — None is correct
for multi-socket vEGM where each socket supplies its own memory-backend
via -numa node,memdev= rather than a single machine-wide backend.
ARCHITECTURE.md: update module layout diagram to reflect what Phase 0
actually introduces (platform.rs, topology.rs, tests.rs all present;
probe.rs holds HostTopology only — PlatformProbe is Phase 1). Annotate
runtime: RuntimeFeatures fields in both Machine code samples as Phase 3+
so the doc is self-consistent with the current stubs.
tests.rs: run cargo fmt (long assignment line on check() helper).
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds fixture and ignored test (Phase 3) for Q35 CoCo (SEV-SNP) with one
GPU passed through, based on a production invocation (AMD EPYC host,
2026-07-13, H100 80GB at 0000:e1:00.0).
Key differences from Grace/virt and vanilla Q35 documented in fixture
and ARCHITECTURE.md Planned Fixtures section:
- sev-snp-guest object emitted before the -machine line (QEMU requires
the protection object first); kernel_irqchip=split required for SNP
- Memory model: memory-backend-ram with host-nodes=1,policy=bind (not
file-backed; CoCo uses RAM backend for NUMA pinning on x86)
- GPU passthrough via pxb-pcie + pcie-root-port + vfio-pci (not
vfio-pci-nohotplug); no arm-smmuv3 — x86 uses global AMD/Intel IOMMU
- iommufd is per-device (id=iommufdvfio-<uuid>), not the shared
iommufd0 used on Grace
- x-pci-vendor-id/x-pci-device-id overrides required so the guest sees
correct device IDs for measured boot attestation
- AMDSEV.fd firmware (AMD-specific OVMF, not generic OVMF.fd)
- pxb-pcie bus_nr=32 (not the Grace 1-indexed cumulative formula)
Phase 3 items enumerated: Objects::protection, Q35::kernel_irqchip and
confidential_guest_support typed fields, MemoryBackend::Ram host_nodes
and policy, per-device iommufd, VfioDevice vendor/device id overrides.
TDX capture still needed.
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Q35 vanilla kata fixture (q35_vanilla_kata_x86.args) from a production
DGX x86 invocation (2026-07-07): 2-socket NUMA, 65 vCPUs, 36864M per
socket pinned to /dev/shm, 8 cold-plug pcie-root-ports on pcie.0.
Test added as #[ignore = "Phase 3"]; it drives the Phase 3 API design.
Key differences from the Grace/virt topology that inform Phase 3 scope:
- No kernel-irqchip on vanilla Q35 (only required for CoCo)
- NUMA SHM memory model: per-socket file-backed backends with host-nodes
and policy=bind rather than a single backend on the -machine line
- Q35 GPU passthrough uses cold-plug onto pre-provisioned pcie-root-ports
(cold_plug_vfio=root-port, pcie_root_port=8, hot_plug_vfio=no-port);
no pxb-pcie, no arm-smmuv3
Platform Parity section: documents legacy Machine fields without typed
homes (machine_accelerators, confidential_guest_support), baseline
-machine output for each supported machine type, and Platform fields
required before the legacy struct can be deleted.
Planned Fixture Configurations section: enumerates the four fixture sets
needed before Phase 6 closes (vanilla virt, Q35 vanilla, CoCo+GPU,
8-GPU+4-NVSwitch), notes what data is captured vs. still needed, and
lists the new types required for each.
Known Issues and Follow-up Items section: QMP startup timeout (#13343),
seccomp_sandbox not yet in Platform, machine_accelerators /
confidential_guest_support cross-references.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Seven fixture files encode the expected QEMU argument list for each
Grace passthrough topology derived from tested production deployments:
grace_1_single_gpu -- 1 GPU, 1 SMMU, 9 NUMA nodes
grace_2_four_gpus_1_per_smmu -- 4 GPUs x 1-per-SMMU, 33 NUMA nodes
grace_3_four_gpus_2_per_smmu -- 4 GPUs x 2-per-SMMU, 33 NUMA nodes
grace_4_gpu_and_nic -- GPU + NIC, NIC emits no generic-initiators
grace_5_vcmdq -- hugepages backing + cmdqv=on on arm-smmuv3
grace_6_vegm_1_per_socket -- vEGM, 4 sockets, 1 GPU each
grace_7_vegm_2_per_socket -- vEGM, 2 sockets, 2 GPUs each
Each test is #[ignore] until the corresponding phase lands. The harness
reads fixtures as one Vec<String> element per line; blank lines and
lines starting with '#' are skipped.
The fixtures are the contract: any Platform implementation must reproduce
them exactly before a phase PR can be merged.
Refs: #12187, #12125
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
The existing cmdline_generator.rs (~3700 lines) encodes bus-assignment
logic inside each device and uses a single stringly-typed Machine struct
for all machine types. This makes adding Grace/vCMDQ/vEGM support
impractical without touching every device.
Introduce src/qemu/machine/ with the target types:
Platform -- top-level wiring point (machine + PCIe topology + objects)
Machine -- per-machine-type structs (Q35, Virt, Pseries, S390xCcwVirtio)
PciTopology -- PCIe expander buses with per-RC numa_node and BusIommu
BusIommu -- bus-attached IOMMU (SmmuV3); IntelIommu lives on Q35 directly
Objects -- typed -object registry (MemoryBackend, AcpiPciNodeLink, ...)
HostTopology -- probe result consumed by apply_host_defaults
All methods are todo!() stubs. No behaviour change; cmdline_generator.rs
is untouched. Implementations land phase by phase starting at Phase 2.
Refs: #12187, #12125
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
The current cmdline_generator.rs (~3700 lines) has three structural
problems: bus-assignment logic duplicated across every device via
#[cfg(target_arch)], a single stringly-typed Machine struct shared by
all machine types, and no NUMA-aware memory backend support.
Add ARCHITECTURE.md describing the target design before any code
changes land. The document covers:
current pain points
target module layout and core types (Platform, Machine, PciTopology,
BusIommu, Objects, HostTopology)
NUMA layout rules (ACPI SRAT ordering, 8 nodes per GPU for MIG,
highmem-mmio-size sizing, hotplug placeholder)
7 Grace platform configurations as golden test fixtures
6-phase migration plan (Phase 0 test harness through Phase 6 cleanup)
design principles
The document is a living reference updated with each phase PR.
It becomes a clean stable reference once Phase 6 completes.
Refs: #12187, #12125
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Only enable discard for block-plain emptyDir volumes when the
hypervisor reports block device discard support.
The resource manager now queries hypervisor capabilities once and
passes block discard support through VolumeContext. Block emptyDir
volume setup uses that capability to decide whether to add the
discard mount option and related mount metadata, avoiding unsupported
discard mounts on hypervisors that cannot expose discard/unmap to the
guest.
It will correctly adjust the related codes with the new support.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Map BlockConfigModern.discard_unmap to Dragonball block device
sparse support when adding modern block devices.
This lets runtime-rs request guest-visible discard support through
the Dragonball block device configuration while leaving vhost-user
block devices disabled.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add a sparse flag to Dragonball block device configuration and use
it to expose virtio-blk discard support to the guest. Handle discard
requests by validating the requested range and punching holes in
sparse local file backends, returning an I/O error instead of
panicking on discard failures.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add the Cloud Hypervisor disk sparse option to the local DiskConfig
model and keep its default aligned with upstream. Map
BlockConfigModern.discard_unmap to DiskConfig.sparse so block devices
that request discard/unmap explicitly expose sparse discard support
through the CH API.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
To tells if the hypervisor exposes block discard/unmap support to
the guest, we introduce a new capability of discard/unmap named
`BlockDeviceDiscardSupport` in the capability set.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove the remaining legacy Block device implementation now that
BlockModern covers all block paths, finishing the migration started
by the QEMU and DeviceManager legacy-path drops.
Delete virtio_blk.rs (BlockConfig, BlockDevice and its Device impl)
and stop exporting it from driver/mod.rs. Drop the BlockCfg variant
from DeviceConfig and the Block variant from DeviceType. Remove the
DeviceConfig::BlockCfg handling and the create_block_device helper
from DeviceManager, and drop DeviceType::Block from the QEMU
hotunplug unsupported list.
No new behavior; pure removal of the dead legacy path.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove DeviceType::Block so BlockModern is the single code path,
completing the BlockConfig -> BlockConfigModern migration at the
Device Manager layer.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The preceding "Use BlockConfigModern for ..." series converted every
block-device producer in the Qemu path to emit BlockModern. The Qemu
inner module still held the matching legacy Block consumer code across
all three lifecycle ops (coldplug/hotplug/hotunplug), now unreachable.
Remove it so BlockModern is the single code path, completing the
BlockConfig -> BlockConfigModern migration at the Qemu layer.
The hotunplug arm for Block now returns an explicit "unsupported"
error instead of a stale QMP detach. The new added Block(_) is just a
place holder and it will be removed in a follow-up commit.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for firecacker.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for agent-ctl vm_ops.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the erofs rootfs path from BlockConfig to BlockConfigModern,
routing its block-device attachment through BlockCfgModern.
This covers the rw layer, single-device raw, multi-device VMDK, and
GPT-partitioned VMDK attachment points in erofs_rootfs.rs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for swap task.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the block config from BlockConfig to BlockConfigModern,
routing it through DeviceConfig::BlockCfgModern for block rootfs.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Migrate the host block-device (major/minor) passthrough path in
ResourceManagerInner from BlockConfig to BlockConfigModern, routing it
through DeviceConfig::BlockCfgModern. This continues the BlockModern
migration to cover block-mode devices carried in the OCI spec.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Switch the InitData resource path from BlockConfig to BlockConfigModern
so the InitData is handled through the modern block device pipeline,
consistent with the recent BlockModern migration for block volumes.
This changes ResourceConfig::InitData to wrap BlockConfigModern,
routes its handling through DeviceConfig::BlockCfgModern, and updates
its related helper function in virt_container to build a
BlockConfigModern.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Switch the VmRootfs resource path from BlockConfig to BlockConfigModern
so the VM rootfs is handled through the modern block device pipeline,
consistent with the recent BlockModern migration for block volumes.
This changes ResourceConfig::VmRootfs to wrap BlockConfigModern, routes
its handling through DeviceConfig::BlockCfgModern, and updates
prepare_rootfs_config() in virt_container to build a BlockConfigModern.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce a new field of serial_override in the BlockModern config.
Add a DeviceType::BlockModern branch to the QEMU QMP hotplug path,
mirroring the existing Block semantics: skip block devices that
represent the initrd, and dispatch to nvdimm or blk/ccw/scsi
according to driver_option. Snapshot the config parameters under
lock and run the asynchronous hotplug without holding it to avoid
awaiting across the lock.
Pass the real block format and, when enabled, the first independent
iothread (indep_iothread_0) to hotplug_block_device instead of the
default format and None, and return DeviceType::BlockModern so the
hotplugged device is tracked.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Treat SPDK and VFIO direct volumes as a two-step mount. The agent
storage entry mounts the device inside the guest with block storage
options, while the OCI mount bind-mounts that guest path into the
container.
Filter bind flags from storage options, preserve direct-volume
mountInfo options when present, and add ro when the volume is
read-only.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Build regular block volumes with BlockConfigModern and submit them
through DeviceConfig::BlockCfgModern. Preserve the existing block
device attributes while also recording the host path required by the
modern block device model.
Update block volume helper call sites with the default option slice
so this change remains buildable with the modern helper signature.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Switch the Cloud Hypervisor device handling path from the legacy
Block device variant to BlockModern. Keep the block device behind
its shared mutex while snapshotting the configuration for the
hotplug request, and persist the PCI path assigned by Cloud
Hypervisor back into the shared device state.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce build_bind_mount_options() to construct OCI mount options for
container-side bind mount, preferring volume_options over OCI options
and ensuring a bind/rbind flag is present.
And addfilter_block_storage_options() to strip bind/rbind flags from
Storage.options, preventing ENOTDIR when the agent mounts a block
device to a filesystem inside the guest.
And then Extend handle_block_volume() with a volume_options parameter
and replace hardcoded ro/empty options with the new filter+ro logic,
narrowing the function to a BlockModern-only path by removing the
DeviceType::Block branch.
Fix generate_shared_path to always create under rw/ since ro/
is a read-only bind mount of rw/ and direct creation fails with ReadOnly
FS. Switch the OCI Mount type from fs_type to KATA_MOUNT_BIND_TYPE because
the mount source is a directory (guest_path), not a block device.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Move the block device type definitions shared between BlockDevice and
BlockDeviceModern — VIRTIO_BLOCK_{PCI,MMIO,CCW}/VIRTIO_PMEM constants,
KATA_*_DEV_TYPE aliases, and the BlockDeviceAio / BlockDeviceFormat
enums — from virtio_blk.rs into virtio_blk_modern.rs, so they live
alongside the BlockModern handler that will replace the legacy
BlockDevice.
Add a `format: BlockDeviceFormat` field to BlockConfigModern to match
the legacy BlockConfig. Update mod.rs re-exports accordingly.
No behavior change; pure relocation plus the new field.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Extend add_block_device with num_queues and queue_size parameters
to support multi-queue virtio-blk configuration, and add BlockModern
hotplug/hotunplug handler that extracts config from
Arc<Mutex<BlockDeviceModern>>, passes the new parameters to
add_block_device, and updates pci_path on successful hotplug.
And we will use BlockDeviceModern to handle block device instead of
Legacy BlockDevice.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Add BlockModern handling in DeviceManager so that device index is
correctly released both in get_device_info and in the error path.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
This patch replaces the favicon on the mkdocs site with an
official Kata Containers logo made available by the OpenInfra
Foundation.
Signed-off-by: Ildiko Vancsa <ildiko.vancsa@gmail.com>
Resolve kata-monitor log level from monitor.logLevel, the chart-wide
logLevel, and debug:true, matching the precedence used by kata-deploy.
Fixes#13382
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Derive the runtime endpoint from k8sDistribution and mount the matching
host socket directory so kata-monitor can reach containerd on
non-vanilla k8s deployments.
Fixes#13378
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
We have bumped Trustee recently for the Helm chart work, but there are a
few new features that would be good to land for Kata 4.0 and CoCo
v0.22.0.
Before too long, we will start bumping Trustee to a particular version
rather than to a commit hash.
Signed-off-by: Tobin Feldman-Fitzthum <tfeldmanfitz@nvidia.com>
Use zstd level 10 (via kata_tar_zstd) when creating static build
tarballs so CI artifacts stay under GitHub's 2GB limit without
paying the build-time cost of maximum compression.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Reuse the kubelet runtimeRequestTimeout validation for guest-pull
configurations.
Guest pull can also keep kubelet waiting during CreateContainer. Warn
when kata-deploy can see that runtimeRequestTimeout is below the same
suggested timeout used for EROFS layer conversion.
Detect both the regular guest-pull mapping and the experimental force
guest-pull configuration so nodes get the warning for either path.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Extend the kata-deploy host-check stage with EROFS-specific
prerequisite validation.
Check that containerd supports the EROFS snapshotter, required
kernel features are available either as loaded modules or built-in
kernel configuration, and the host erofs-utils version satisfies the
minimum required by kata-deploy's configured mkfs options. Also warn
when kubelet runtimeRequestTimeout appears too low for EROFS layer
conversion, which can run during CreateContainer.
Read kubelet runtimeRequestTimeout from kubelet /configz through the
apiserver node proxy, and add the corresponding RBAC permission. Keep
fs-verity validation warning-only because user drop-ins may override
kata-deploy's default enable_fsverity setting and kata-deploy does
not yet validate the backing filesystem's fs-verity feature.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
Document operational requirements for using the EROFS snapshotter
with Kata Containers.
Call out that EROFS layer conversion can happen during
CreateContainer, not only during image pull, so kubelet
runtimeRequestTimeout may need to be increased for large images.
Also document the kernel modules needed for EROFS and dm-verity,
the erofs-utils version required by the configured mkfs options,
and the distinction between host-side fs-verity and guest-visible
dm-verity.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Assisted-by: OpenAI <support@openai.com>
