qemu/machine: fix apply_q35_defaults slot/chassis/bus_nr for multi-GPU

Three bugs made apply_q35_defaults produce incorrect output for any
GpuSmmuGroup with more than one device:

- slot: Some(0) on every port — QEMU requires unique slots within a pxb;
  fixed to slot: Some(port_index_within_group).
- chassis: 10 on every pxb complex — must be unique per complex so QEMU
  can distinguish PCIe chassis during bus enumeration;
  fixed to 10 + group_idx.
- rp-numa{group}-{global_gpu_idx} naming — the global index made
  pxb-numa1's ports read as rp-numa1-4…rp-numa1-7 instead of
  rp-numa1-0…rp-numa1-3 as production logs show;
  fixed to use the per-pxb port_index.
- bus_nr advancing by 1 per pxb — production captures show 32-bus
  spacing (32, 64) to give room for subordinate bus assignment;
  fixed to 32 + group_idx × 32.

ARCHITECTURE.md gains a "Scalability (1–8 GPUs)" note under the VFIO
Device Assignment Model section, including the B200/B300 PCIe layout table
and an explanation of why NVSwitch classification is deferred to Phase 4.

Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Zvonko Kaiser
2026-07-15 17:04:26 +00:00
parent ca16df1917
commit af315cd1fa
2 changed files with 40 additions and 7 deletions

View File

@@ -533,6 +533,37 @@ devices emitted on `pcie.0` at VM creation, with no device attached. At runtime
devices are plugged into available slots via QMP `device_add`. DANs and dynamically
assigned VFIO NICs use this path. Mirrors the `pcie_root_port =` kata config field.
**Scalability (18 GPUs, Q35 root-port topology):**
Any count from 1 to 8 GPUs on a Q35/x86 host is expressed by varying the number
of entries in `gpu_smmu_groups` and `pci_bus_addrs`. The model scales linearly:
each GPU gets exactly one `pcie-root-port` on a `pxb-pcie`; GPUs on the same NUMA
socket share a pxb complex. `apply_q35_defaults` assigns:
- `bus_nr = 32 + group_idx × 32` per pxb (32-bus spacing matches production captures)
- `chassis = 10 + group_idx` per pxb (unique chassis per complex, e.g. 10, 11)
- `slot = port_index_within_pxb` (0-based, unique per pxb)
- `id = rp-numa{group}-{port}` (port-relative, not global GPU index)
Example layouts for B200/B300 PCIe (no NVSwitch, no NVLink):
| GPU count | NUMA nodes | pxb layout |
|-----------|-----------|------------|
| 1 | 1 | pxb-numa0 (bus_nr=32, chassis=10): slot 0 |
| 2 | 1 | pxb-numa0: slots 0-1 |
| 4 | 1 | pxb-numa0: slots 0-3 |
| 4 | 2 | pxb-numa0 (slots 0-1) + pxb-numa1 (bus_nr=64, chassis=11, slots 0-1) |
| 8 | 2 | pxb-numa0 (slots 0-3) + pxb-numa1 (slots 0-3) |
**NVSwitch classification is Phase 4:** `apply_q35_defaults` generates `GpuPci` for
every entry in `gpu_smmu_groups` because `GpuSmmuGroup::pci_bus_addrs` carries no
device-type information. NVSwitch passthrough (where NVSwitches share the pxb with
GPUs on slots 47) requires a Phase 4 extension to `HostTopology` — a separate
`nvswitch_addrs` field or a typed `VfioPassthroughEntry { addr, kind }` — so the
prober can classify each PCI address before `apply_q35_defaults` is called.
Until then, callers building an NVSwitch topology construct `Platform` directly
(as the `q35_coco_tdx_8gpu_4nvswitch` and `q35_vanilla_8gpu_4nvswitch` tests do).
Grace/aarch64 uses `vfio-pci-nohotplug` for cold-plug onto `pxb-pcie`-attached root
ports with a per-bus `arm-smmuv3` — a distinct topology from Q35 `root-port` cold-plug
even though both are classified as "cold-plug root-port" in kata config terms.

View File

@@ -334,22 +334,24 @@ impl Platform {
}
let mut gpu_idx = 0usize;
let mut bus_nr_running: u8 = 0;
for (group_idx, group) in topo.gpu_smmu_groups.iter().enumerate() {
let bus_nr = 32u8 + bus_nr_running;
bus_nr_running += 1;
// 32-bus spacing between pxb complexes: each pxb may have up to 31
// subordinate buses (one per root port + potential downstream buses).
// Production captures show bus_nr=32 for pxb-numa0, 64 for pxb-numa1.
let bus_nr = 32u8 + (group_idx as u8) * 32;
let cpu_mem_node = socket_numa_node(&topo.sockets, group.socket);
let pxb_id = format!("pxb-numa{group_idx}");
let mut root_ports = Vec::new();
for pci_addr in &group.pci_bus_addrs {
let rp_id = format!("rp-numa{group_idx}-{gpu_idx}");
for (port_idx, pci_addr) in group.pci_bus_addrs.iter().enumerate() {
// slot and chassis are per-pxb-relative; chassis increments per complex.
let rp_id = format!("rp-numa{group_idx}-{port_idx}");
root_ports.push(PciRootPort {
id: rp_id,
chassis: 10,
slot: Some(0),
chassis: 10 + group_idx as u8,
slot: Some(port_idx as u8),
multifunction: Some(false),
io_reserve: None,
device: Some(VfioDevice {