From 9fb8c494457b94b60f0cb528b8a8f7f80a0a80cd Mon Sep 17 00:00:00 2001 From: Zvonko Kaiser Date: Wed, 15 Jul 2026 17:51:16 +0000 Subject: [PATCH] qemu/machine: ARCHITECTURE.md Phase 4+5 status + auto mode note MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mark Phase 4 and Phase 5 complete (✅). Enumerate what each phase delivered vs. the original plan: Phase 4: NIC groups, prealloc fix, fixture corrections, GB300 dry-run, all 12 tests passing without #[ignore]. Phase 5: prealloc=on for hugepages, vCMDQ/vEGM wired end-to-end. Add "auto" mode note under Phase 4: the kata config fields cold_plug_vfio and hot_plug_vfio currently require an explicit topology string. An "auto" value is the natural next step once PlatformProbe is implemented to scan /sys/bus/pci/devices/ and classify devices; apply_host_defaults already contains the logic to derive the right topology. Update Phase 6 checklist: add PlatformProbe/"auto" wiring and switch-port topology (NVSwitch/DAN fan-out) as explicit deliverables. Add nic_smmu_groups to the HostTopology struct block. Assisted-by: Claude Sonnet 4.6 Signed-off-by: Zvonko Kaiser --- .../hypervisor/src/qemu/ARCHITECTURE.md | 48 ++++++++++++++++--- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md b/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md index 2b2037a0cc..6848861036 100644 --- a/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md +++ b/src/runtime-rs/crates/hypervisor/src/qemu/ARCHITECTURE.md @@ -340,7 +340,13 @@ See [Config 7](#config-7--vegm-2-gpus-per-socket-4-gpus-2-sockets). /// Read from host sysfs and IOMMU group layout before constructing Platform. pub struct HostTopology { pub sockets: Vec, + /// GPU groups: each maps to one pxb+smmuv3; GPUs emit 8 acpi-generic-initiator nodes. pub gpu_smmu_groups: Vec, + /// NIC groups: each gets its own pxb+smmuv3 complex allocated after all GPU + /// complexes in bus_nr ordering. NICs emit no acpi-generic-initiator links + /// and no NUMA initiator nodes. Reuses GpuSmmuGroup struct for the same + /// socket-affinity semantics; differentiated by field name. + pub nic_smmu_groups: Vec, pub egm_sockets: Vec, pub numa_distances: Vec<(u32, u32, u32)>, // (src, dst, val) /// Minimum pre-provisioned empty root-port count on Q35 pcie.0. @@ -797,25 +803,53 @@ each supported machine type and that `HostTopology` round-trips through `q35_vanilla_kata_x86`, `q35_coco_snp_single_gpu`, `q35_coco_tdx_8gpu_4nvswitch`, `q35_vanilla_8gpu_4nvswitch`. -### Phase 4 — Multi-RC PCIe and NUMA layout +### Phase 4 — Multi-RC PCIe and NUMA layout ✅ - Emit `pxb-pcie` (with `numa_node=`) + per-RC `arm-smmuv3` from `PciTopology`. - Support N root ports per `PciRootComplex` (Config 3 shape: 2 GPUs per `SMMU`). -- Add `VfioPciNoHotplug` with typed `IommufdRef` and `VfioDeviceKind`. +- `VfioDeviceKind` enum drives device-type-specific emission (`vfio-pci-nohotplug` + for Grace GPUs, `vfio-pci` for NICs/NvSwitch/GpuPci). - Emit NUMA nodes in the correct order (CpuMem → GPU initiators → memory-only). -- `apply_host_defaults` wired end-to-end: Configs 1–4 golden fixtures pass. +- `HostTopology::nic_smmu_groups` added: NICs get their own pxb+`smmuv3` complex + but emit no `acpi-generic-initiator` links and no NUMA initiator nodes. +- `apply_host_defaults` wired end-to-end: all 7 Grace configs pass. +- `MemoryBackend::File { prealloc }` fixed: `prealloc=on` now emitted for hugepages. +- GB300 dry-run fixture: `gb300_nvl_2gpu.args` — Grace + 2 B200 GPUs, + `highmem-mmio-size=8T` (B200 BARs exceed the 4T window). +- All 12 fixture tests pass without `#[ignore]`: + `grace_1` through `grace_7`, `gb300_nvl_2gpu`, and 4 Q35 tests. -### Phase 5 — `vCMDQ` and vEGM +**`"auto"` mode (Phase 4+):** The kata config fields `cold_plug_vfio` and +`hot_plug_vfio` currently require an explicit topology string (`"root-port"`, +`"switch-port"`, etc.). An `"auto"` value is the natural next step: once +`PlatformProbe` is implemented to scan `/sys/bus/pci/devices/` and classify +devices, `apply_host_defaults` already contains all the logic to derive the +right topology automatically. `"auto"` will call the prober and select: +- `root-port` for GPU-only workloads +- `root-port + NIC pxb` when NICs are in `nic_smmu_groups` +- `switch-port` (Phase 6+) for NVSwitch/DAN fan-out +The prober is not implemented yet; `"auto"` is reserved and will error until +`PlatformProbe::probe()` lands. +>>>>>>> 208b1c739 (qemu/machine: ARCHITECTURE.md Phase 4+5 status + auto mode note) -- `SmmuV3 { cmdqv: true }` + `MemoryBackend::File { path: "/dev/hugepages/", … }`. -- `AcpiPciNodeLink::EgmMemory` + per-socket `MemoryBackend::File { path: "/dev/egmN", … }`. +### Phase 5 — `vCMDQ` and vEGM ✅ + +- `SmmuV3 { cmdqv: true }` + `MemoryBackend::File { path: "/dev/hugepages/", prealloc: true }`. +- `AcpiPciNodeLink::EgmMemory` + per-socket `MemoryBackend::File { path: "/dev/egmN", is_egm: true }`. - `EgmSocketInfo` probe wired into `apply_host_defaults`. - Configs 5–7 golden fixtures pass. +- `with_hugepages()` correctly sets `prealloc=on` on the emitted backend line. ### Phase 6 — Cleanup -- Delete dead code from `cmdline_generator.rs`. +- Delete dead code from `cmdline_generator.rs` (strangle pattern: new Platform + takes over one device class at a time; old code deleted when test coverage + confirms parity). - Remove feature flags / compat shims introduced during migration. +- Implement `PlatformProbe::probe()` and wire `"auto"` in `cold_plug_vfio` / + `hot_plug_vfio` to call `apply_host_defaults` without manual topology config. +- Add `switch-port` topology for NVSwitch/DAN fan-out + (`pcie-root-port → x3130-upstream → N xio3130-downstream`). - Final golden-test sweep across all 7 Grace configs plus existing machine types. - Remove all TODOs and decision stubs from this document.