mirror of
https://github.com/kata-containers/kata-containers.git
synced 2026-07-02 07:02:16 +00:00
f763e9cca9
Add k8s-nvidia-numa.bats with five tests that validate NUMA behaviour
on hosts where NUMA is configured by default (qemu-nvidia-gpu,
qemu-nvidia-gpu-snp, qemu-nvidia-gpu-tdx):
1. Multi-node sandbox (large workload spanning all host NUMA nodes):
- Guest NUMA node count matches host
- Guest vCPU distribution is balanced across nodes (max-min <= 1)
- Guest memory is distributed across NUMA nodes
- Host-side vCPU pinning is balanced across NUMA nodes
2. Right-sized single-node sandbox (small workload fitting one node):
- Guest collapses to a single NUMA node
- All host vCPU threads pinned to that one NUMA node
3. GPU passthrough with VFIO, multi-node:
- Guest NUMA topology is balanced (same as test 1)
- Guest GPU's NUMA node matches the host GPU's NUMA node
(resolved via the vfio-pci,host=<BDF> from the QEMU command
line and /sys/bus/pci/devices/<BDF>/numa_node)
- QEMU command line contains pxb-pcie and policy=bind
- Host vCPU pinning is balanced
4. GPU passthrough with VFIO, right-sized single-node: small workload
plus GPU that fits in a single host NUMA node:
- Guest collapses to a single NUMA node
- The chosen node is the GPU's host NUMA node, not just any node
that fits — verified by matching host-nodes= in the memory
backend and pxb-pcie numa_node= against the GPU's host node
- Guest GPU reports the same NUMA node as the host GPU
5. Explicit numa_mapping in the runtime TOML (QEMU-only):
- Drops a config.d/ fragment that sets numa_mapping = ["1"], so the
auto-derive + right-sizing path is bypassed entirely
- Guest sees exactly 1 NUMA node
- QEMU memory backend is bound to host node 1 (host-nodes=1,
policy=bind), not host node 0
- Host-side vCPU threads land on host node 1
- Drop-in is removed on teardown so subsequent tests are unaffected
Guest-side checks use a dedicated container image
(quay.io/kata-containers/numa) that reads sysfs and prints results to
stdout — no kubectl exec or CoCo policy overrides needed.
Host-side checks (crictl, pgrep, taskset) run directly on the host
via sudo; a standalone numa-pinning-check.sh script handles the vCPU
thread affinity inspection. The config.d/ helpers used by test 5 are
runtime-agnostic (probe Go vs runtime-rs layout on disk) but the test
is gated to qemu-* shims since runtime-rs does not yet implement
NUMA.
Skips cleanly on single-NUMA hosts, unsupported hypervisors, or when
no nvidia.com/pgpu resources are available (GPU tests only).
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Assisted-by: Cursor <cursoragent@cursor.com>
Howto Guides
Kubernetes Integration
- Run Kata containers with
crictl - How to use Kata Containers and Containerd
- How to use Kata Containers and containerd with Kubernetes
- How to use Kata Containers and CRI-O with Kubernetes
- Kata Containers and service mesh for Kubernetes
- How to import Kata Containers logs into Fluentd
Hypervisors Integration
Currently supported hypervisors with Kata Containers include:
-
qemu -
cloud-hypervisor -
firecrackerIn the case of
firecrackerthe use of a block devicesnapshotteris needed for the VM rootfs. Refer to the following guide for additional configuration steps:
Confidential Containers Policy
Advanced Topics
- How to use Kata Containers with virtio-fs
- Setting Sysctls with Kata
- What Is VMCache and How To Enable It
- What Is VM Templating and How To Enable It
- How to Use Template in runtime-rs
- Privileged Kata Containers
- How to load kernel modules in Kata Containers
- How to use Kata Containers with
virtio-mem - How to set sandbox Kata Containers configurations with pod annotations
- How to monitor Kata Containers in K8s
- How to use hotplug memory on arm64 in Kata Containers
- How to setup swap devices in guest kernel
- How to run rootless vmm
- How to run Docker with Kata Containers
- How to run Kata Containers with
nydus - How to run Kata Containers with AMD SEV-SNP
- How to run Kata Containers with IBM Secure Execution
- How to use EROFS to build rootfs in Kata Containers
- How to run Kata Containers with kinds of Block Volumes
- How to use the Kata Agent Policy
- How to pull images in the guest
- How to use mem-agent to decrease the memory usage of Kata container
- How to use seccomp with runtime-rs
- How to use passthroughfd-IO with runtime-rs and Dragonball
- How to use EROFS snapshotter with Kata Containers
- How to use NUMA with Kata Containers