From bbef8e1705b8dd3d9c88e54560bace17f25cd00f Mon Sep 17 00:00:00 2001 From: Manuel Huber Date: Tue, 30 Jun 2026 18:41:47 +0000 Subject: [PATCH] docs: minor updates to NVIDIA guide Update OS version and containerd references, and make a few minor adjustments and clarifications. For the clarifications, move the section "Deploy pods with Kata agent security policies" further up so that the installation of the genpolicy tool has been elaborated on before attempting to run NVIDIA CI tests locally. Signed-off-by: Manuel Huber --- .../NVIDIA-GPU-passthrough-and-Kata-QEMU.md | 103 +++++++++++------- 1 file changed, 64 insertions(+), 39 deletions(-) diff --git a/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata-QEMU.md b/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata-QEMU.md index 118cf16919..a6ea8f3dba 100644 --- a/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata-QEMU.md +++ b/docs/use-cases/NVIDIA-GPU-passthrough-and-Kata-QEMU.md @@ -263,7 +263,7 @@ with their documentation in the confidential-containers documentation. The requirements for the TEE scenario are: -- Ubuntu 25.10 as host OS +- Ubuntu 26.04 as host OS - CPU with AMD SEV-SNP or Intel TDX support with proper BIOS/UEFI version and settings - CC-capable Hopper/Blackwell GPU with proper VBIOS version. @@ -281,7 +281,7 @@ selecting proper hardware and on properly configuring its firmware and OS. #### Containerd and Kubernetes First, set up your Kubernetes cluster. For instance, in Kata CI, our NVIDIA -jobs use a single-node vanilla Kubernetes cluster with containerd v2.2 +jobs use a single-node vanilla Kubernetes cluster with containerd v2.3 and Kata's current supported Kubernetes version. This cluster is being set up using the `deploy_k8s` function from the script file `tests/integration/kubernetes/gha-run.sh`. If you intend to run this script, @@ -293,7 +293,7 @@ You can execute the function as follows: $ export GH_TOKEN="" $ export KUBERNETES="vanilla" $ export CONTAINER_ENGINE="containerd" -$ export CONTAINER_ENGINE_VERSION="v2.2" +$ export CONTAINER_ENGINE_VERSION="v2.3" $ source tests/gha-run-k8s-common.sh $ deploy_k8s ``` @@ -434,9 +434,9 @@ mode. After deployment, you can transition your node(s) to the desired CC state, using either the `on`, `ppcie`, or `off` value, depending on your scenario. For the non-CC scenario, transition to the `off` state via: -`kubectl label nodes --all nvidia.com/cc.mode=off` and wait until all pods -are back running. When an actual change is exercised, various GPU Operator -operands will be restarted. +`kubectl label nodes --all nvidia.com/cc.mode=off --overwrite` and wait +until all pods are back running. When an actual change is exercised, various +GPU Operator operands will be restarted. Ensure all pods are running: @@ -476,8 +476,8 @@ spec: EOF ``` -Depending on your scenario and on the CC state, export your desired runtime -class name define the environment variable: +Depending on your scenario and on the CC state, define the environment +variable with your desired runtime class name: ```bash $ export GPU_RUNTIME_CLASS_NAME="kata-qemu-nvidia-gpu-snp" @@ -487,7 +487,7 @@ Then, deploy the sample Kubernetes pod manifest and observe the pod logs: ```bash $ envsubst < ./cuda-vectoradd-kata.yaml.in | kubectl apply -f - -$ kubectl wait --for=condition=Ready pod/cuda-vectoradd-kata --timeout=60s +$ kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/cuda-vectoradd-kata --timeout=60s $ kubectl logs -n default cuda-vectoradd-kata ``` @@ -536,17 +536,65 @@ the CC and non-CC mode. In case of the non-CC mode, you can use the variable in the above CUDA vectorAdd sample. The `kata-qemu-nvidia-gpu-snp` runtime class will **NOT** work in this mode - and vice versa. +#### Deploy pods with Kata agent security policies + +With GPU passthrough being supported by the +[genpolicy tool](https://github.com/kata-containers/kata-containers/tree/main/src/tools/genpolicy), +you can use the tool to create a Kata agent security policy. Our CI deploys +all sample pod manifests with a Kata agent security policy. + +The `genpolicy` binary is shipped in the separate Kata tools archive and is +not installed by the Kata Containers Helm chart. For evaluation on the +single-node cluster described in this guide, install the Kata tools archive +matching the Kata release selected by `VERSION`. For example, run the +following commands from the root of the Kata Containers source tree: + +```bash +$ mkdir -p kata-tools-artifacts +$ curl -fsSL \ + "https://github.com/kata-containers/kata-containers/releases/download/${VERSION}/kata-tools-static-${VERSION}-amd64.tar.zst" \ + --output kata-tools-artifacts/kata-tools-static.tar.zst +$ bash tests/integration/kubernetes/gha-run.sh install-kata-tools +$ test -x /opt/kata/bin/genpolicy +$ rm -rf kata-tools-artifacts +``` + +These commands install `genpolicy` and its settings under `/opt/kata` on the +same host that runs this single-node Kubernetes instance. This is intended +for evaluation purposes and for reproducing Kata CI locally. + +Note that, in Kata CI, we use snippets such as the following to modify the +genpolicy default settings: +```bash +[ + { + "op": "replace", + "path": "/kata_config/oci_version", + "value": "1.3.0" + } +] +``` +This modification is applied via the genpolicy drop-in configuration file +`src/tools/genpolicy/drop-in-examples/20-oci-1.3.0-drop-in.json`. +When using a newer (or older) containerd version, the OCI version field +may need to be adjusted accordingly. + #### Run Kata CI tests locally -Upstream Kata CI runs the CUDA vectorAdd test, a composite attestation test, -and a basic NIM/RAG deployment. Running CI tests for the TEE GPU scenario -requires KBS to be deployed (except for the CUDA vectorAdd test). The best -place to get started running these tests locally is to look into our +Upstream Kata CI runs the CUDA vectorAdd, NUMA, composite attestation, +NIM/RAG, and NIMService tests. For TEE GPU scenarios, KBS is required for the +attestation and NIM tests, but not for CUDA or NUMA. The best place to get +started running these tests locally is to look into our [NVIDIA CI workflow manifest](https://github.com/kata-containers/kata-containers/blob/main/.github/workflows/run-k8s-tests-on-nvidia-gpu.yaml) and into the underlying [run_kubernetes_nv_tests.sh](https://github.com/kata-containers/kata-containers/blob/main/tests/integration/kubernetes/run_kubernetes_nv_tests.sh) script. For example, to run the CUDA vectorAdd scenario against the TEE GPU -runtime class use the following commands: +runtime class use the following commands. These tests generate Kata agent +security policies, so first install `genpolicy` as described in +[Deploy pods with Kata agent security policies](#deploy-pods-with-kata-agent-security-policies). +Run the tests from a Kata Containers source tree checked out at the tag +matching `VERSION`. This keeps the deployed runtime, `genpolicy` binary and +settings, and test scripts at the same release version. ```bash # create the kata runtime class the test framework uses @@ -561,8 +609,8 @@ $ K8S_TEST_NV="k8s-nvidia-cuda.bats" ./gha-run.sh run-nv-tests > **Note:** > -> The other scenarios require an NGC API key to run, i.e., to export the -> `NGC_API_KEY` variable with a valid NGC API key. +> The NIM and NIMService tests require `NGC_API_KEY` to contain a valid NGC +> API key. #### Deploy pods using attestation @@ -591,29 +639,6 @@ attestation) regardless of which image is being pulled. To deploy your own pods using authenticated container images, or secure key release for attestation, follow steps similar to our mentioned CI samples. -#### Deploy pods with Kata agent security policies - -With GPU passthrough being supported by the -[genpolicy tool](https://github.com/kata-containers/kata-containers/tree/main/src/tools/genpolicy), -you can use the tool to create a Kata agent security policy. Our CI deploys -all sample pod manifests with a Kata agent security policy. - -Note that, in Kata CI, we use snippets such as the following to modify the -genpolicy default settings: -```bash -[ - { - "op": "replace", - "path": "/kata_config/oci_version", - "value": "1.3.0" - } -] -``` -This modification is applied via the genpolicy drop-in configuration file -`src/tools/genpolicy/drop-in-examples/20-oci-1.3.0-drop-in.json`. -When using a newer (or older) containerd version, the OCI version field -may need to be adjusted accordingly. - #### Deploy pods using your own containers and manifests You can author pod manifests leveraging your own containers, for instance,