From d9e6eb11ae50e5a811a10b1396e2c600692534a8 Mon Sep 17 00:00:00 2001 From: Joana Pecholt Date: Thu, 4 Aug 2022 18:04:05 +0200 Subject: [PATCH] docs: Guide to use SNP-VMs with Kata-Containers The guide describes how to set Kata-Containers up so that AMD SEV-SNP encrypted VMs are used when deploying confidential containers. Signed-off-by: Joana Pecholt --- docs/how-to/README.md | 3 +- ...how-to-run-kata-containers-with-SNP-VMs.md | 159 ++++++++++++++++++ docs/how-to/images/SNP-stack.svg | 4 + 3 files changed, 165 insertions(+), 1 deletion(-) create mode 100644 docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md create mode 100644 docs/how-to/images/SNP-stack.svg diff --git a/docs/how-to/README.md b/docs/how-to/README.md index aa09b49c73..1ad4860481 100644 --- a/docs/how-to/README.md +++ b/docs/how-to/README.md @@ -42,4 +42,5 @@ - [How to setup swap devices in guest kernel](how-to-setup-swap-devices-in-guest-kernel.md) - [How to run rootless vmm](how-to-run-rootless-vmm.md) - [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md) -- [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md) \ No newline at end of file +- [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md) +- [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md) diff --git a/docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md b/docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md new file mode 100644 index 0000000000..bc4e14a10d --- /dev/null +++ b/docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md @@ -0,0 +1,159 @@ +# Kata Containers with AMD SEV-SNP VMs + +## Disclaimer + +This guide is designed for developers and is - same as the Developer Guide - not intended for production systems or end users. It is advisable to only follow this guide on non-critical development systems. + +## Prerequisites + +To run Kata Containers in SNP-VMs, the following software stack is used. + +![Kubernetes integration with shimv2](./images/SNP-stack.svg) + +The host BIOS and kernel must be capable of supporting AMD SEV-SNP and configured accordingly. For Kata Containers, the host kernel with branch [`sev-snp-iommu-avic_5.19-rc6_v3`](https://github.com/AMDESE/linux/tree/sev-snp-iommu-avic_5.19-rc6_v3) and commit [`3a88547`](https://github.com/AMDESE/linux/commit/3a885471cf89156ea555341f3b737ad2a8d9d3d0) is known to work in conjunction with SEV Firmware version 1.51.3 (0xh\_1.33.03) available on AMD's [SEV developer website](https://developer.amd.com/sev/). See [AMD's guide](https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel) to configure the host accordingly. Verify that you are able to run SEV-SNP encrypted VMs first. The guest components required for Kata Containers are built as described below. + +**Tip**: It is easiest to first have Kata Containers running on your system and then modify it to run containers in SNP-VMs. Follow the [Developer guide](../Developer-Guide.md#warning) and then follow the below steps. Nonetheless, you can just follow this guide from the start. + +## How to build + +Follow all of the below steps to install Kata Containers with SNP-support from scratch. These steps mostly follow the developer guide with modifications to support SNP + +__Steps from the Developer Guide:__ +- Get all the [required components](../Developer-Guide.md#requirements-to-build-individual-components) for building the kata-runtime +- [Build the and install kata-runtime](../Developer-Guide.md#build-and-install-the-kata-containers-runtime) +- [Build a custom agent](../Developer-Guide.md#build-a-custom-kata-agent---optional) +- [Create an initrd image](../Developer-Guide.md#create-an-initrd-image---optional) by first building a rootfs, then building the initrd based on the rootfs, use a custom agent and install. `ubuntu` works as the distribution of choice. +- Get the [required components](../../tools/packaging/kernel/README.md#requirements) to build a custom kernel + +__SNP-specific steps:__ +- Build the SNP-specific kernel as shown below (see this [guide](../../tools/packaging/kernel/README.md#build-kata-containers-kernel) for more information) +```bash +$ pushd kata-containers/tools/packaging/kernel/ +$ ./build-kernel.sh -a x86_64 -x snp setup +$ ./build-kernel.sh -a x86_64 -x snp build +$ sudo -E PATH="${PATH}" ./build-kernel.sh -x snp install +$ popd +``` +- Build a current OVMF capable of SEV-SNP: +```bash +$ pushd kata-containers/tools/packaging/static-build/ovmf +$ ./build.sh +$ tar -xvf edk2-x86_64.tar.gz +$ popd +``` +- Build a custom QEMU +```bash +$ source kata-containers/tools/packaging/scripts/lib.sh +$ qemu_url="$(get_from_kata_deps "assets.hypervisor.qemu.snp.url")" +$ qemu_branch="$(get_from_kata_deps "assets.hypervisor.qemu.snp.branch")" +$ qemu_commit="$(get_from_kata_deps "assets.hypervisor.qemu.snp.commit")" +$ git clone -b "${qemu_branch}" "${qemu_url}" +$ pushd qemu +$ git checkout "${qemu_commit}" +$ ./configure --target-list=x86_64-softmmu --enable-debug +$ make -j "$(nproc)" +$ popd +``` + +### Kata Containers Configuration for SNP + +The configuration file located at `/etc/kata-containers/configuration.toml` must be adapted as follows to support SNP-VMs: +- Use the SNP-specific kernel for the guest VM (change path) +```toml +kernel = "/usr/share/kata-containers/vmlinuz-snp.container" +``` +- Enable the use of an initrd (uncomment) +```toml +initrd = "/usr/share/kata-containers/kata-containers-initrd.img" +``` +- Disable the use of a rootfs (comment out) +```toml +# image = "/usr/share/kata-containers/kata-containers.img" +``` +- Use the custom QEMU capable of SNP (change path) +```toml +path = "/path/to/qemu/build/qemu-system-x86_64" +``` +- Use `virtio-9p` device since `virtio-fs` is unsupported due to bugs / shortcomings in QEMU version [`snp-v3`](https://github.com/AMDESE/qemu/tree/snp-v3) for SEV and SEV-SNP (change value) +```toml +shared_fs = "virtio-9p" +``` +- Disable `virtiofsd` since it is no longer required (comment out) +```toml +# virtio_fs_daemon = "/usr/libexec/virtiofsd" +``` +- Disable NVDIMM (uncomment) +```toml +disable_image_nvdimm = true +``` +- Disable shared memory (uncomment) +```toml +file_mem_backend = "" +``` +- Enable confidential guests (uncomment) +```toml +confidential_guest = true +``` +- Enable SNP-VMs (uncomment) +```toml +sev_snp_guest = true +``` + - Configure an OVMF (add path) +```toml +firmware = "/path/to/kata-containers/tools/packaging/static-build/ovmf/opt/kata/share/ovmf/OVMF.fd" +``` + +## Test Kata Containers with Containerd + +With Kata Containers configured to support SNP-VMs, we use containerd to test and deploy containers in these VMs. + +### Install Containerd +If not already present, follow [this guide](./containerd-kata.md#install) to install containerd and its related components including `CNI` and the `cri-tools` (skip Kata Containers since we already installed it) + +### Containerd Configuration + +Follow [this guide](./containerd-kata.md#configuration) to configure containerd to use Kata Containers + +## Run Kata Containers in SNP-VMs + +Run the below commands to start a container. See [this guide](./containerd-kata.md#run) for more information +```bash +$ sudo ctr image pull docker.io/library/busybox:latest +$ sudo ctr run --cni --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh +``` + +### Check for active SNP: + +Inside the running container, run the following commands to check if SNP is active. It should look something like this: +``` +/ # dmesg | grep -i sev +[ 0.299242] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP +[ 0.472286] SEV: Using SNP CPUID table, 31 entries present. +[ 0.514574] SEV: SNP guest platform device initialized. +[ 0.885425] sev-guest sev-guest: Initialized SEV guest driver (using vmpck_id 0) +``` + +### Obtain an SNP Attestation Report + +To obtain an attestation report inside the container, the `/dev/sev-guest` must first be configured. As of now, the VM does not perform this step, however it can be performed inside the container, either in the terminal or in code. + +Example for shell: +``` +/ # SNP_MAJOR=$(cat /sys/devices/virtual/misc/sev-guest/dev | awk -F: '{print $1}') +/ # SNP_MINOR=$(cat /sys/devices/virtual/misc/sev-guest/dev | awk -F: '{print $2}') +/ # mknod -m 600 /dev/sev-guest c "${SNP_MAJOR}" "${SNP_MINOR}" +``` + +## Known Issues + +- Support for cgroups v2 is still [work in progress](https://github.com/kata-containers/kata-containers/issues/927). If issues occur due to cgroups v2 becoming the default in newer systems, one possible solution is to downgrade cgroups to v1: +```bash +sudo sed -i 's/^\(GRUB_CMDLINE_LINUX=".*\)"/\1 systemd.unified_cgroup_hierarchy=0"/' /etc/default/grub +sudo update-grub +sudo reboot +``` +- If both SEV and SEV-SNP are supported by the host, Kata Containers uses SEV-SNP by default. You can verify what features are enabled by checking `/sys/module/kvm_amd/parameters/sev` and `sev_snp`. This means that Kata Containers can not run both SEV-SNP-VMs and SEV-VMs at the same time. If SEV is to be used by Kata Containers instead, reload the `kvm_amd` kernel module without SNP-support, this will disable SNP-support for the entire platform. +```bash +sudo rmmod kvm_amd && sudo modprobe kvm_amd sev_snp=0 +``` + diff --git a/docs/how-to/images/SNP-stack.svg b/docs/how-to/images/SNP-stack.svg new file mode 100644 index 0000000000..9d32557f20 --- /dev/null +++ b/docs/how-to/images/SNP-stack.svg @@ -0,0 +1,4 @@ + + + +
Linux Kernel / KVM
Linux Kernel / KVM
QEMU
QEMU
BIOS
BIOS
OVMF
OVMF
Linux Kernel
Linux Kernel
Host
Host
VM
VMText is not SVG - cannot display \ No newline at end of file