mirror of
https://github.com/kata-containers/kata-containers.git
synced 2026-07-25 22:49:59 +00:00
qemu/machine: fix apply_q35_defaults slot/chassis/bus_nr for multi-GPU
Three bugs made apply_q35_defaults produce incorrect output for any
GpuSmmuGroup with more than one device:
- slot: Some(0) on every port — QEMU requires unique slots within a pxb;
fixed to slot: Some(port_index_within_group).
- chassis: 10 on every pxb complex — must be unique per complex so QEMU
can distinguish PCIe chassis during bus enumeration;
fixed to 10 + group_idx.
- rp-numa{group}-{global_gpu_idx} naming — the global index made
pxb-numa1's ports read as rp-numa1-4…rp-numa1-7 instead of
rp-numa1-0…rp-numa1-3 as production logs show;
fixed to use the per-pxb port_index.
- bus_nr advancing by 1 per pxb — production captures show 32-bus
spacing (32, 64) to give room for subordinate bus assignment;
fixed to 32 + group_idx × 32.
ARCHITECTURE.md gains a "Scalability (1–8 GPUs)" note under the VFIO
Device Assignment Model section, including the B200/B300 PCIe layout table
and an explanation of why NVSwitch classification is deferred to Phase 4.
Assisted-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -533,6 +533,37 @@ devices emitted on `pcie.0` at VM creation, with no device attached. At runtime
|
||||
devices are plugged into available slots via QMP `device_add`. DANs and dynamically
|
||||
assigned VFIO NICs use this path. Mirrors the `pcie_root_port =` kata config field.
|
||||
|
||||
**Scalability (1–8 GPUs, Q35 root-port topology):**
|
||||
|
||||
Any count from 1 to 8 GPUs on a Q35/x86 host is expressed by varying the number
|
||||
of entries in `gpu_smmu_groups` and `pci_bus_addrs`. The model scales linearly:
|
||||
each GPU gets exactly one `pcie-root-port` on a `pxb-pcie`; GPUs on the same NUMA
|
||||
socket share a pxb complex. `apply_q35_defaults` assigns:
|
||||
|
||||
- `bus_nr = 32 + group_idx × 32` per pxb (32-bus spacing matches production captures)
|
||||
- `chassis = 10 + group_idx` per pxb (unique chassis per complex, e.g. 10, 11)
|
||||
- `slot = port_index_within_pxb` (0-based, unique per pxb)
|
||||
- `id = rp-numa{group}-{port}` (port-relative, not global GPU index)
|
||||
|
||||
Example layouts for B200/B300 PCIe (no NVSwitch, no NVLink):
|
||||
|
||||
| GPU count | NUMA nodes | pxb layout |
|
||||
|-----------|-----------|------------|
|
||||
| 1 | 1 | pxb-numa0 (bus_nr=32, chassis=10): slot 0 |
|
||||
| 2 | 1 | pxb-numa0: slots 0-1 |
|
||||
| 4 | 1 | pxb-numa0: slots 0-3 |
|
||||
| 4 | 2 | pxb-numa0 (slots 0-1) + pxb-numa1 (bus_nr=64, chassis=11, slots 0-1) |
|
||||
| 8 | 2 | pxb-numa0 (slots 0-3) + pxb-numa1 (slots 0-3) |
|
||||
|
||||
**NVSwitch classification is Phase 4:** `apply_q35_defaults` generates `GpuPci` for
|
||||
every entry in `gpu_smmu_groups` because `GpuSmmuGroup::pci_bus_addrs` carries no
|
||||
device-type information. NVSwitch passthrough (where NVSwitches share the pxb with
|
||||
GPUs on slots 4–7) requires a Phase 4 extension to `HostTopology` — a separate
|
||||
`nvswitch_addrs` field or a typed `VfioPassthroughEntry { addr, kind }` — so the
|
||||
prober can classify each PCI address before `apply_q35_defaults` is called.
|
||||
Until then, callers building an NVSwitch topology construct `Platform` directly
|
||||
(as the `q35_coco_tdx_8gpu_4nvswitch` and `q35_vanilla_8gpu_4nvswitch` tests do).
|
||||
|
||||
Grace/aarch64 uses `vfio-pci-nohotplug` for cold-plug onto `pxb-pcie`-attached root
|
||||
ports with a per-bus `arm-smmuv3` — a distinct topology from Q35 `root-port` cold-plug
|
||||
even though both are classified as "cold-plug root-port" in kata config terms.
|
||||
|
||||
@@ -334,22 +334,24 @@ impl Platform {
|
||||
}
|
||||
|
||||
let mut gpu_idx = 0usize;
|
||||
let mut bus_nr_running: u8 = 0;
|
||||
|
||||
for (group_idx, group) in topo.gpu_smmu_groups.iter().enumerate() {
|
||||
let bus_nr = 32u8 + bus_nr_running;
|
||||
bus_nr_running += 1;
|
||||
// 32-bus spacing between pxb complexes: each pxb may have up to 31
|
||||
// subordinate buses (one per root port + potential downstream buses).
|
||||
// Production captures show bus_nr=32 for pxb-numa0, 64 for pxb-numa1.
|
||||
let bus_nr = 32u8 + (group_idx as u8) * 32;
|
||||
|
||||
let cpu_mem_node = socket_numa_node(&topo.sockets, group.socket);
|
||||
let pxb_id = format!("pxb-numa{group_idx}");
|
||||
|
||||
let mut root_ports = Vec::new();
|
||||
for pci_addr in &group.pci_bus_addrs {
|
||||
let rp_id = format!("rp-numa{group_idx}-{gpu_idx}");
|
||||
for (port_idx, pci_addr) in group.pci_bus_addrs.iter().enumerate() {
|
||||
// slot and chassis are per-pxb-relative; chassis increments per complex.
|
||||
let rp_id = format!("rp-numa{group_idx}-{port_idx}");
|
||||
root_ports.push(PciRootPort {
|
||||
id: rp_id,
|
||||
chassis: 10,
|
||||
slot: Some(0),
|
||||
chassis: 10 + group_idx as u8,
|
||||
slot: Some(port_idx as u8),
|
||||
multifunction: Some(false),
|
||||
io_reserve: None,
|
||||
device: Some(VfioDevice {
|
||||
|
||||
Reference in New Issue
Block a user