From 43ac4f83c9bc2caf77845928f7a3e5cb89450a6f Mon Sep 17 00:00:00 2001 From: Zvonko Kaiser Date: Mon, 13 Jul 2026 22:20:37 +0000 Subject: [PATCH] hypervisor/qemu: add Q35 fixture, platform parity, and planned fixtures Q35 vanilla kata fixture (q35_vanilla_kata_x86.args) from a production DGX x86 invocation (2026-07-07): 2-socket NUMA, 65 vCPUs, 36864M per socket pinned to /dev/shm, 8 cold-plug pcie-root-ports on pcie.0. Test added as #[ignore = "Phase 3"]; it drives the Phase 3 API design. Key differences from the Grace/virt topology that inform Phase 3 scope: - No kernel-irqchip on vanilla Q35 (only required for CoCo) - NUMA SHM memory model: per-socket file-backed backends with host-nodes and policy=bind rather than a single backend on the -machine line - Q35 GPU passthrough uses cold-plug onto pre-provisioned pcie-root-ports (cold_plug_vfio=root-port, pcie_root_port=8, hot_plug_vfio=no-port); no pxb-pcie, no arm-smmuv3 Platform Parity section: documents legacy Machine fields without typed homes (machine_accelerators, confidential_guest_support), baseline -machine output for each supported machine type, and Platform fields required before the legacy struct can be deleted. Planned Fixture Configurations section: enumerates the four fixture sets needed before Phase 6 closes (vanilla virt, Q35 vanilla, CoCo+GPU, 8-GPU+4-NVSwitch), notes what data is captured vs. still needed, and lists the new types required for each. Known Issues and Follow-up Items section: QMP startup timeout (#13343), seccomp_sandbox not yet in Platform, machine_accelerators / confidential_guest_support cross-references. Signed-off-by: Zvonko Kaiser Assisted-by: Claude Sonnet 4.6 --- .../hypervisor/src/qemu/ARCHITECTURE.md | 186 ++++++++++++++++++ .../hypervisor/src/qemu/machine/tests.rs | 23 +++ .../fixtures/qemu/q35_vanilla_kata_x86.args | 54 +++++ 3 files changed, 263 insertions(+) create mode 100644 src/runtime-rs/crates/hypervisor/tests/fixtures/qemu/q35_vanilla_kata_x86.args diff --git a/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md b/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md index ad8ec2fdc5..a68c9cf08f 100644 --- a/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md +++ b/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md @@ -606,6 +606,192 @@ This is the phase that enables hugepages for `runtime-rs` (issue #12125). --- +## Platform Parity + +The `Platform` type must generate correct QEMU command lines for every supported +machine type, not just Grace/virt. The legacy `Machine` struct carried two raw +string fields that have no typed home yet: + +| Field | Legacy value | Machine types | +|------------------------------|--------------------------------------|-------------------| +| `accel` | `"kvm"`, `"tcg"`, `"kvm:tcg"` | all | +| `options` | raw KVM accelerator options | x86/arm | +| `kernel_irqchip` | `"on"`, `"split"`, `"off"` | Q35 only | +| `confidential_guest_support` | `"sev-snp0"`, `"tdx0"`, `""` | Q35 / virt | +| `memory_backend` | e.g. `"m0"` | virt (NUMA/EGM) | + +These are preserved as raw strings in `BaseMachine` for now. Phase 3 will +introduce typed representations. + +### Baseline `-machine` output per type + +The examples below show the minimum expected output for vanilla (no GPU) configs +derived from tested deployments. They anchor the per-machine fixture set. + +**virt (aarch64 — vanilla kata)** +```text +-machine virt,accel=kvm,gic-version=3,ras=on +``` + +**Q35 (x86_64 — vanilla kata)** +```text +-machine q35,accel=kvm +``` +(`kernel-irqchip` is absent on vanilla Q35; it is only required for CoCo.) + +**Q35 + TDX (x86_64 — CoCo)** +```text +-object tdx-guest,id=tdx,... +-machine q35,accel=kvm,kernel-irqchip=split,confidential-guest-support=tdx +``` + +**Q35 + SEV-SNP (x86_64 — CoCo)** +```text +-object sev-snp-guest,id=sev-snp,... +-machine q35,accel=kvm,kernel-irqchip=split,confidential-guest-support=sev-snp +``` + +**s390-ccw-virtio (s390x)** +```text +-machine s390-ccw-virtio,accel=kvm +``` + +Both `machine_accelerators` (the raw KVM option string) and +`confidential_guest_support` need typed representations before the legacy +`Machine` struct can be deleted. Tracked in Phase 3. + +--- + +## Planned Fixture Configurations + +The 7 Grace fixtures cover Grace GPU passthrough thoroughly. The configurations +below must also be captured as golden fixtures before Phase 6 closes. Each +entry notes the data source required: fixture content must come from actual +production QEMU invocations, not from documentation. + +### Vanilla kata — virt (aarch64) + +Basic `virt` machine with no GPU passthrough, no NUMA, no hugepages. +Represents the common ARM64 kata use-case. + +**Data needed:** capture `qemu-system-aarch64` invocation from a running +non-GPU kata pod on an ARM64 host. + +### Vanilla kata — Q35 (x86_64) + +**Production data captured** (DGX x86 host, 2026-07-07). +Fixture: `q35_vanilla_kata_x86.args`. Test: `q35_vanilla_kata_x86` (ignored, Phase 3). + +Key observations from the production invocation: + +- `-machine q35,accel=kvm` — no `kernel-irqchip` on vanilla; only required for CoCo +- NUMA memory model differs from Grace: total memory via `-m 73728M,slots=10,maxmem=127052M`; + NUMA pinning via separate `memory-backend-file` objects with `host-nodes=N,policy=bind` + backed by `/dev/shm` (not `/dev/hugepages` or `/dev/egm*`) +- Two NUMA nodes: socket 0 cpus 0-32 / 36864M, socket 1 cpus 33-65 / 36864M; + distance 20 between them +- 8 `pcie-root-port` pre-provisioned on `pcie.0` (slots 0-7) for GPU cold-plug + (`cold_plug_vfio=root-port`, `pcie_root_port=8`, `hot_plug_vfio=no-port` in + `configuration-qemu-nvidia-gpu.toml.in`); GPU VFIO devices are added to the + static QEMU command line before the VM boots, not via QMP after boot +- No `pxb-pcie`, no `arm-smmuv3` — Q35 GPU passthrough uses cold-plug onto `pcie.0` + root ports, not the static vfio-pci-nohotplug topology used on Grace + +New Platform fields required (Phase 3): +- `MemoryBackend::File { host_nodes: Option, policy: Option }` for NUMA SHM +- `Objects::numa_distances: Vec<(u32, u32, u32)>` for `-numa dist` entries +- `HostTopology` additions for NUMA SHM paths and distances + +### CoCo + GPU passthrough (SEV-SNP or TDX) + +Confidential compute requires: +- A protection object (`sev-snp-guest` or `tdx-guest`) before the machine line +- `confidential-guest-support=` on the `-machine` line +- `kernel-irqchip=split` instead of `on` (TDX/SNP requirement) +- OVMF firmware path instead of the standard BIOS (`-bios/-pflash`) +- Memory hotplug disabled (confidential memory cannot be hot-added) + +CoCo + GPU passthrough combines these with the VFIO topology from the Grace +configs. The `ProtectionDeviceConfig::SevSnp` and `::Tdx` paths exist in +`cmdline_generator.rs` (`add_sev_snp_protection_device` / `add_tdx_protection_device`); +Platform needs a typed `Objects::protection` field to carry these through. + +**Data needed:** capture from a CoCo-enabled kata pod with at least one GPU +passed through on both an AMD (SEV-SNP) and Intel (TDX) host. + +### 8 GPUs + 4 NVSwitches (DGX/HGX topology) + +NVSwitch passthrough adds a new device kind and a multi-level PCIe hierarchy +not present in the Grace configs: + +- `VfioDeviceKind::NvSwitch` is not yet defined in `topology.rs` (only `Gpu` + and `Nic` exist). +- NVSwitches currently use `VfioDeviceConfig` (not `VfioDeviceGroup`) in the + legacy path (`add_gpu_nvswitch_setup` at cmdline_generator.rs:3373). +- PCIe hierarchy: root port → `x3130-upstream` → `xio3130-downstream` → + device (three levels vs. the two levels used for GPU direct attachment). +- `add_pcie_switch_ports` (cmdline_generator.rs:3508) emits this hierarchy; + `PciTopology` has no equivalent typed representation yet. + +New types needed before a fixture can be written: +- `VfioDeviceKind::NvSwitch` +- `PciSwitchPort { upstream: PcieUpstreamPort, downstream: Vec }` on `PciRootComplex` +- `HostTopology::nvswitch_addrs` or equivalent probe field + +**Data needed:** capture from a DGX/HGX or GB200 NVL system with 8 GPUs and +4 NVSwitches passed through. Exact bus_nr arithmetic and PCIe address +assignments must come from a live invocation, not from inference. + +--- + +## Known Issues and Follow-up Items + +Misconfiguration and known defects in the current runtime-rs/QEMU path that +are related to this refactor. Items marked **post-refactor** require the new +`Platform` emission path to be wired end-to-end before they can be addressed +cleanly. + +### QMP startup timeout is hard-coded and ignores the caller's deadline (#13343) + +`QemuInner::start_vm(_timeout)` accepts the timeout argument from the +`Hypervisor` trait but silently ignores it. QMP initialisation instead uses +two independent hard-coded values: + +- 5 s per-read socket timeout (`QMP_SOCKET_TIMEOUT`) +- 50 s overall connect/init deadline (`QMP_INIT_TIMEOUT`) + +Under large-memory / VFIO / GPU passthrough conditions QEMU can take longer +than 50 s before QMP is fully responsive, causing a spurious timeout failure +even though the higher-level `start_vm` caller's intended budget was never +applied. + +The `Hypervisor::start_vm(timeout: i32)` contract is also inconsistently +interpreted across backends: QEMU and Firecracker ignore it, Dragonball +treats it as milliseconds (despite comments saying seconds), Cloud Hypervisor +treats it as seconds, and Remote treats it as seconds for the RPC deadline. + +**Minimum fix for GPU passthrough (pre-refactor):** make `QMP_INIT_TIMEOUT` +configurable via `HypervisorConfig` (e.g. `qmp_init_timeout_secs`) so +operators can raise it without patching. + +**Post-refactor:** once `Platform::to_qemu_args` drives QEMU startup, wire +the QMP connect/init step to consume the caller's remaining `start_vm` budget +rather than maintaining an independent fixed deadline. + +### `seccomp_sandbox` (-sandbox) not plumbed through Platform + +Pre-refactor the option works via the legacy `cmdline_generator.rs` path. +Post-refactor it needs a typed `Objects::seccomp_sandbox` field so the legacy +generator can be removed. Tracked in Phase 3. + +### `machine_accelerators` and `confidential_guest_support` have no typed home + +Tracked in the Platform Parity section. Both are passed as raw strings +through `BaseMachine` today; they need typed representations before the +legacy `Machine` struct can be deleted. + +--- + ## Design Principles 1. **Machine decisions stay in `Machine` and `PciTopology`.** diff --git a/src/runtime-rs/crates/hypervisor/src/qemu/machine/tests.rs b/src/runtime-rs/crates/hypervisor/src/qemu/machine/tests.rs index 6cd1c8ced3..c7dfbdf5a8 100644 --- a/src/runtime-rs/crates/hypervisor/src/qemu/machine/tests.rs +++ b/src/runtime-rs/crates/hypervisor/src/qemu/machine/tests.rs @@ -194,3 +194,26 @@ fn grace_7_vegm_2_per_socket() { "grace_7_vegm_2_per_socket.args", ); } + +// ---- Q35 x86_64: vanilla kata, 2-socket NUMA, 8 cold-plug root ports ---- +// +// Production capture: DGX x86 host, 2026-07-07. 65 vCPUs, 73728M total, +// 36864M per socket pinned to host NUMA node via /dev/shm. 8 pcie-root-ports +// pre-provisioned on pcie.0 for GPU cold-plug (hot_plug_vfio=no-port). +// +// Blocked on Phase 3: +// - Q35 machine in Platform::to_qemu_args (no gic-version, no highmem-mmio-size) +// - MemoryBackend::File { host_nodes, policy } fields for NUMA SHM pinning +// - Objects::numa_distances for -numa dist entries +// - HostTopology fields for NUMA node + SHM path per socket + +#[test] +#[ignore = "Phase 3: Q35 machine + NUMA SHM memory model not yet implemented"] +fn q35_vanilla_kata_x86() { + // HostTopology shape TBD in Phase 3. + // 2 sockets: socket 0 cpus 0-32 (host-node 0), socket 1 cpus 33-65 (host-node 1). + // No gpu_smmu_groups, no egm_sockets. + // 8 cold-plug root ports on pcie.0 (cold_plug_vfio=root-port, pcie_root_port=8); + // NUMA distance 20 between the two nodes. + todo!("Phase 3") +} diff --git a/src/runtime-rs/crates/hypervisor/tests/fixtures/qemu/q35_vanilla_kata_x86.args b/src/runtime-rs/crates/hypervisor/tests/fixtures/qemu/q35_vanilla_kata_x86.args new file mode 100644 index 0000000000..7d34c5606c --- /dev/null +++ b/src/runtime-rs/crates/hypervisor/tests/fixtures/qemu/q35_vanilla_kata_x86.args @@ -0,0 +1,54 @@ +# Q35 vanilla kata — x86_64, 2-socket NUMA, 65 vCPUs, no GPU assigned +# Source: production kata-containers log, July 2026 (NVIDIA DGX x86 host) +# +# Platform scope: machine line + NUMA SHM memory backends + NUMA topology + +# pre-provisioned pcie-root-ports for GPU cold-plug. +# NOT included here (legacy generator): -name, -uuid, -cpu, -m, -smp, all +# virtio devices, -netdev, -kernel, -append, -bios, -rtc, -pidfile, etc. +# +# HostTopology: 2 sockets +# socket 0: cpus 0-32 (33 vCPUs), host-node 0 +# socket 1: cpus 33-65 (33 vCPUs), host-node 1 +# Memory: 73728M total — 36864M per socket, pinned to host NUMA node via /dev/shm +# PCI: 8 pcie-root-ports on pcie.0 — cold-plug slots (cold_plug_vfio=root-port, +# pcie_root_port=8 in configuration-qemu-nvidia-gpu.toml.in); +# hot_plug_vfio=no-port, hotplug is disabled +# NUMA distance: 20 between the two nodes +# +# When a GPU is assigned to a container, -device vfio-pci,host=,bus=rpN +# is added to the static QEMU command line (before VM boot) by the legacy +# generator. This fixture captures the base VM without any GPU assigned. +# +# New Platform fields required (Phase 3): +# MemoryBackend::File { host_nodes: Some(N), policy: "bind" } +# Objects::numa_distances: Vec<(u32, u32, u32)> (src, dst, val) +-machine +q35,accel=kvm +-object +memory-backend-file,id=numa-mem0,size=36864M,mem-path=/dev/shm,host-nodes=0,policy=bind,share=on +-numa +node,nodeid=0,memdev=numa-mem0,cpus=0-32 +-object +memory-backend-file,id=numa-mem1,size=36864M,mem-path=/dev/shm,host-nodes=1,policy=bind,share=on +-numa +node,nodeid=1,memdev=numa-mem1,cpus=33-65 +-numa +dist,src=0,dst=1,val=20 +-numa +dist,src=1,dst=0,val=20 +-device +pcie-root-port,id=rp0,bus=pcie.0,chassis=0,slot=0,multifunction=off +-device +pcie-root-port,id=rp1,bus=pcie.0,chassis=0,slot=1,multifunction=off +-device +pcie-root-port,id=rp2,bus=pcie.0,chassis=0,slot=2,multifunction=off +-device +pcie-root-port,id=rp3,bus=pcie.0,chassis=0,slot=3,multifunction=off +-device +pcie-root-port,id=rp4,bus=pcie.0,chassis=0,slot=4,multifunction=off +-device +pcie-root-port,id=rp5,bus=pcie.0,chassis=0,slot=5,multifunction=off +-device +pcie-root-port,id=rp6,bus=pcie.0,chassis=0,slot=6,multifunction=off +-device +pcie-root-port,id=rp7,bus=pcie.0,chassis=0,slot=7,multifunction=off