mirror of
https://github.com/kata-containers/kata-containers.git
synced 2026-07-13 20:27:41 +00:00
f763e9cca9
Add k8s-nvidia-numa.bats with five tests that validate NUMA behaviour
on hosts where NUMA is configured by default (qemu-nvidia-gpu,
qemu-nvidia-gpu-snp, qemu-nvidia-gpu-tdx):
1. Multi-node sandbox (large workload spanning all host NUMA nodes):
- Guest NUMA node count matches host
- Guest vCPU distribution is balanced across nodes (max-min <= 1)
- Guest memory is distributed across NUMA nodes
- Host-side vCPU pinning is balanced across NUMA nodes
2. Right-sized single-node sandbox (small workload fitting one node):
- Guest collapses to a single NUMA node
- All host vCPU threads pinned to that one NUMA node
3. GPU passthrough with VFIO, multi-node:
- Guest NUMA topology is balanced (same as test 1)
- Guest GPU's NUMA node matches the host GPU's NUMA node
(resolved via the vfio-pci,host=<BDF> from the QEMU command
line and /sys/bus/pci/devices/<BDF>/numa_node)
- QEMU command line contains pxb-pcie and policy=bind
- Host vCPU pinning is balanced
4. GPU passthrough with VFIO, right-sized single-node: small workload
plus GPU that fits in a single host NUMA node:
- Guest collapses to a single NUMA node
- The chosen node is the GPU's host NUMA node, not just any node
that fits — verified by matching host-nodes= in the memory
backend and pxb-pcie numa_node= against the GPU's host node
- Guest GPU reports the same NUMA node as the host GPU
5. Explicit numa_mapping in the runtime TOML (QEMU-only):
- Drops a config.d/ fragment that sets numa_mapping = ["1"], so the
auto-derive + right-sizing path is bypassed entirely
- Guest sees exactly 1 NUMA node
- QEMU memory backend is bound to host node 1 (host-nodes=1,
policy=bind), not host node 0
- Host-side vCPU threads land on host node 1
- Drop-in is removed on teardown so subsequent tests are unaffected
Guest-side checks use a dedicated container image
(quay.io/kata-containers/numa) that reads sysfs and prints results to
stdout — no kubectl exec or CoCo policy overrides needed.
Host-side checks (crictl, pgrep, taskset) run directly on the host
via sudo; a standalone numa-pinning-check.sh script handles the vCPU
thread affinity inspection. The config.d/ helpers used by test 5 are
runtime-agnostic (probe Go vs runtime-rs layout on disk) but the test
is gated to qemu-* shims since runtime-rs does not yet implement
NUMA.
Skips cleanly on single-NUMA hosts, unsupported hypervisors, or when
no nvidia.com/pgpu resources are available (GPU tests only).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
74 lines
2.1 KiB
Bash
Executable File
74 lines
2.1 KiB
Bash
Executable File
#!/bin/sh
|
|
#
|
|
# Copyright (c) 2026 NVIDIA Corporation
|
|
#
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
#
|
|
# Reads guest NUMA topology from sysfs and prints structured output.
|
|
# Designed to run inside a kata VM as the container entrypoint.
|
|
#
|
|
# Output format (one key: value per line):
|
|
# numa_online: 0-1
|
|
# node0_cpus: 32
|
|
# node1_cpus: 32
|
|
# node0_mem_kb: 37078332
|
|
# node1_mem_kb: 37125524
|
|
# gpu_0000:41:00.0_numa: 1 (only if GPUs are present)
|
|
|
|
set -e
|
|
|
|
# Print results to stdout (readable via "kubectl logs"), then sleep to
|
|
# keep the pod alive so the host-side pinning check can inspect the
|
|
# QEMU process. The bats test deletes the pod when done.
|
|
|
|
# NUMA nodes online (e.g. "0-1" or "0")
|
|
online=$(cat /sys/devices/system/node/online)
|
|
echo "numa_online: ${online}"
|
|
|
|
# Per-node vCPU count
|
|
for cpulist in /sys/devices/system/node/node*/cpulist; do
|
|
node_name=$(basename "$(dirname "${cpulist}")")
|
|
cpus=$(cat "${cpulist}")
|
|
count=0
|
|
# Parse comma-separated ranges like "0-31,64-95"
|
|
IFS=","
|
|
for range in ${cpus}; do
|
|
case "${range}" in
|
|
*-*)
|
|
lo=${range%-*}
|
|
hi=${range#*-}
|
|
count=$((count + hi - lo + 1))
|
|
;;
|
|
*)
|
|
count=$((count + 1))
|
|
;;
|
|
esac
|
|
done
|
|
unset IFS
|
|
echo "${node_name}_cpus: ${count}"
|
|
done
|
|
|
|
# Per-node memory
|
|
for meminfo in /sys/devices/system/node/node*/meminfo; do
|
|
node_name=$(basename "$(dirname "${meminfo}")")
|
|
mem_kb=$(awk '/MemTotal/ {print $4}' "${meminfo}")
|
|
echo "${node_name}_mem_kb: ${mem_kb}"
|
|
done
|
|
|
|
# GPU NUMA affinity (if any GPUs are present via VFIO passthrough).
|
|
# PCI class 0x030200 = 3D controller (NVIDIA data center GPUs: A100, H100, etc.)
|
|
for numa_file in /sys/bus/pci/devices/*/numa_node; do
|
|
dev_dir=$(dirname "${numa_file}")
|
|
class=$(cat "${dev_dir}/class" 2>/dev/null) || continue
|
|
case "${class}" in
|
|
0x030200)
|
|
bdf=$(basename "${dev_dir}")
|
|
node=$(cat "${numa_file}")
|
|
echo "gpu_${bdf}_numa: ${node}"
|
|
;;
|
|
esac
|
|
done
|
|
|
|
# Keep the pod alive for host-side pinning verification.
|
|
exec sleep infinity
|