diff --git a/docs/Developer-Guide.md b/docs/Developer-Guide.md index ed9bec4903..a57bd2d72b 100644 --- a/docs/Developer-Guide.md +++ b/docs/Developer-Guide.md @@ -33,51 +33,41 @@ You need to install the following to build Kata Containers components: - `make`. - `gcc` (required for building the shim and runtime). -# Build and install the Kata Containers runtime +# Build and install Kata Containers +## Build and install the Kata Containers runtime -``` -$ go get -d -u github.com/kata-containers/kata-containers -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/runtime -$ make && sudo -E PATH=$PATH make install +```bash +$ git clone https://github.com/kata-containers/kata-containers.git +$ pushd kata-containers/src/runtime +$ make && sudo -E "PATH=$PATH" make install +$ sudo mkdir -p /etc/kata-containers/ +$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers +$ popd ``` The build will create the following: - runtime binary: `/usr/local/bin/kata-runtime` and `/usr/local/bin/containerd-shim-kata-v2` -- configuration file: `/usr/share/defaults/kata-containers/configuration.toml` - -# Check hardware requirements - -You can check if your system is capable of creating a Kata Container by running the following: - -``` -$ sudo kata-runtime check -``` - -If your system is *not* able to run Kata Containers, the previous command will error out and explain why. +- configuration file: `/usr/share/defaults/kata-containers/configuration.toml` and `/etc/kata-containers/configuration.toml` ## Configure to use initrd or rootfs image Kata containers can run with either an initrd image or a rootfs image. -If you want to test with `initrd`, make sure you have `initrd = /usr/share/kata-containers/kata-containers-initrd.img` -in your configuration file, commenting out the `image` line: +If you want to test with `initrd`, make sure you have uncommented `initrd = /usr/share/kata-containers/kata-containers-initrd.img` +in your configuration file, commenting out the `image` line in +`/etc/kata-containers/configuration.toml`. For example: -`/usr/share/defaults/kata-containers/configuration.toml` and comment out the `image` line with the following. For example: - -``` -$ sudo mkdir -p /etc/kata-containers/ -$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers +```bash $ sudo sed -i 's/^\(image =.*\)/# \1/g' /etc/kata-containers/configuration.toml +$ sudo sed -i 's/^# \(initrd =.*\)/\1/g' /etc/kata-containers/configuration.toml ``` You can create the initrd image as shown in the [create an initrd image](#create-an-initrd-image---optional) section. -If you want to test with a rootfs `image`, make sure you have `image = /usr/share/kata-containers/kata-containers.img` +If you want to test with a rootfs `image`, make sure you have uncommented `image = /usr/share/kata-containers/kata-containers.img` in your configuration file, commenting out the `initrd` line. For example: -``` -$ sudo mkdir -p /etc/kata-containers/ -$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers +```bash $ sudo sed -i 's/^\(initrd =.*\)/# \1/g' /etc/kata-containers/configuration.toml ``` The rootfs image is created as shown in the [create a rootfs image](#create-a-rootfs-image) section. @@ -90,7 +80,7 @@ rootfs `image`(100MB+). Enable seccomp as follows: -``` +```bash $ sudo sed -i '/^disable_guest_seccomp/ s/true/false/' /etc/kata-containers/configuration.toml ``` @@ -100,9 +90,7 @@ This will pass container seccomp profiles to the kata agent. Enable full debug as follows: -``` -$ sudo mkdir -p /etc/kata-containers/ -$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers +```bash $ sudo sed -i -e 's/^# *\(enable_debug\).*=.*$/\1 = true/g' /etc/kata-containers/configuration.toml $ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.log=debug initcall_debug"/g' /etc/kata-containers/configuration.toml ``` @@ -175,7 +163,7 @@ and offers possible workarounds and fixes. it stores. When messages are suppressed, it is noted in the logs. This can be checked for by looking for those notifications, such as: -```sh +```bash $ sudo journalctl --since today | fgrep Suppressed Jun 29 14:51:17 mymachine systemd-journald[346]: Suppressed 4150 messages from /system.slice/docker.service ``` @@ -200,7 +188,7 @@ RateLimitBurst=0 Restart `systemd-journald` for the changes to take effect: -```sh +```bash $ sudo systemctl restart systemd-journald ``` @@ -214,25 +202,24 @@ $ sudo systemctl restart systemd-journald The agent is built with a statically linked `musl.` The default `libc` used is `musl`, but on `ppc64le` and `s390x`, `gnu` should be used. To configure this: -``` -$ export ARCH=$(uname -m) +```bash +$ export ARCH="$(uname -m)" $ if [ "$ARCH" = "ppc64le" -o "$ARCH" = "s390x" ]; then export LIBC=gnu; else export LIBC=musl; fi -$ [ ${ARCH} == "ppc64le" ] && export ARCH=powerpc64le -$ rustup target add ${ARCH}-unknown-linux-${LIBC} +$ [ "${ARCH}" == "ppc64le" ] && export ARCH=powerpc64le +$ rustup target add "${ARCH}-unknown-linux-${LIBC}" ``` To build the agent: -``` -$ go get -d -u github.com/kata-containers/kata-containers -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/agent && make +```bash +$ make -C kata-containers/src/agent ``` The agent is built with seccomp capability by default. If you want to build the agent without the seccomp capability, you need to run `make` with `SECCOMP=no` as follows. -``` -$ make -C $GOPATH/src/github.com/kata-containers/kata-containers/src/agent SECCOMP=no +```bash +$ make -C kata-containers/src/agent SECCOMP=no ``` > **Note:** @@ -240,13 +227,6 @@ $ make -C $GOPATH/src/github.com/kata-containers/kata-containers/src/agent SECCO > - If you enable seccomp in the main configuration file but build the agent without seccomp capability, > the runtime exits conservatively with an error message. -## Get the osbuilder - -``` -$ go get -d -u github.com/kata-containers/kata-containers -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder -``` - ## Create a rootfs image ### Create a local rootfs @@ -254,24 +234,26 @@ As a prerequisite, you need to install Docker. Otherwise, you will not be able to run the `rootfs.sh` script with `USE_DOCKER=true` as expected in the following example. -``` -$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs -$ sudo rm -rf ${ROOTFS_DIR} -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder -$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true ./rootfs.sh ${distro}' +```bash +$ export distro="ubuntu" # example +$ export ROOTFS_DIR="$(realpath kata-containers/tools/osbuilder/rootfs-builder/rootfs)" +$ sudo rm -rf "${ROOTFS_DIR}" +$ pushd kata-containers/tools/osbuilder/rootfs-builder +$ script -fec 'sudo -E USE_DOCKER=true ./rootfs.sh "${distro}"' +$ popd ``` You MUST choose a distribution (e.g., `ubuntu`) for `${distro}`. You can get a supported distributions list in the Kata Containers by running the following. -``` -$ ./rootfs.sh -l +```bash +$ ./kata-containers/tools/osbuilder/rootfs-builder/rootfs.sh -l ``` If you want to build the agent without seccomp capability, you need to run the `rootfs.sh` script with `SECCOMP=no` as follows. -``` -$ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}' +```bash +$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh "${distro}"' ``` > **Note:** @@ -287,17 +269,18 @@ $ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no > > - You should only do this step if you are testing with the latest version of the agent. -``` -$ sudo install -o root -g root -m 0550 -t ${ROOTFS_DIR}/usr/bin ../../../src/agent/target/x86_64-unknown-linux-musl/release/kata-agent -$ sudo install -o root -g root -m 0440 ../../../src/agent/kata-agent.service ${ROOTFS_DIR}/usr/lib/systemd/system/ -$ sudo install -o root -g root -m 0440 ../../../src/agent/kata-containers.target ${ROOTFS_DIR}/usr/lib/systemd/system/ +```bash +$ sudo install -o root -g root -m 0550 -t "${ROOTFS_DIR}/usr/bin" "${ROOTFS_DIR}/../../../../src/agent/target/x86_64-unknown-linux-musl/release/kata-agent" +$ sudo install -o root -g root -m 0440 "${ROOTFS_DIR}/../../../../src/agent/kata-agent.service" "${ROOTFS_DIR}/usr/lib/systemd/system/" +$ sudo install -o root -g root -m 0440 "${ROOTFS_DIR}/../../../../src/agent/kata-containers.target" "${ROOTFS_DIR}/usr/lib/systemd/system/" ``` ### Build a rootfs image -``` -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/image-builder -$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}' +```bash +$ pushd kata-containers/tools/osbuilder/image-builder +$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh "${ROOTFS_DIR}"' +$ popd ``` > **Notes:** @@ -313,21 +296,26 @@ $ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}' ### Install the rootfs image -``` -$ commit=$(git log --format=%h -1 HEAD) -$ date=$(date +%Y-%m-%d-%T.%N%z) +```bash +$ pushd kata-containers/tools/osbuilder/image-builder +$ commit="$(git log --format=%h -1 HEAD)" +$ date="$(date +%Y-%m-%d-%T.%N%z)" $ image="kata-containers-${date}-${commit}" $ sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}" $ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img) +$ popd ``` ## Create an initrd image - OPTIONAL ### Create a local rootfs for initrd image -``` -$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs" -$ sudo rm -rf ${ROOTFS_DIR} -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder -$ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh ${distro}' + +```bash +$ export distro="ubuntu" # example +$ export ROOTFS_DIR="$(realpath kata-containers/tools/osbuilder/rootfs-builder/rootfs)" +$ sudo rm -rf "${ROOTFS_DIR}" +$ pushd kata-containers/tools/osbuilder/rootfs-builder/ +$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh "${distro}"' +$ popd ``` `AGENT_INIT` controls if the guest image uses the Kata agent as the guest `init` process. When you create an initrd image, always set `AGENT_INIT` to `yes`. @@ -335,14 +323,14 @@ always set `AGENT_INIT` to `yes`. You MUST choose a distribution (e.g., `ubuntu`) for `${distro}`. You can get a supported distributions list in the Kata Containers by running the following. -``` -$ ./rootfs.sh -l +```bash +$ ./kata-containers/tools/osbuilder/rootfs-builder/rootfs.sh -l ``` If you want to build the agent without seccomp capability, you need to run the `rootfs.sh` script with `SECCOMP=no` as follows. -``` -$ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}' +```bash +$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh "${distro}"' ``` > **Note:** @@ -351,28 +339,31 @@ $ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no Optionally, add your custom agent binary to the rootfs with the following commands. The default `$LIBC` used is `musl`, but on ppc64le and s390x, `gnu` should be used. Also, Rust refers to ppc64le as `powerpc64le`: -``` -$ export ARCH=$(uname -m) -$ [ ${ARCH} == "ppc64le" ] || [ ${ARCH} == "s390x" ] && export LIBC=gnu || export LIBC=musl -$ [ ${ARCH} == "ppc64le" ] && export ARCH=powerpc64le -$ sudo install -o root -g root -m 0550 -T ../../../src/agent/target/${ARCH}-unknown-linux-${LIBC}/release/kata-agent ${ROOTFS_DIR}/sbin/init +```bash +$ export ARCH="$(uname -m)" +$ [ "${ARCH}" == "ppc64le" ] || [ "${ARCH}" == "s390x" ] && export LIBC=gnu || export LIBC=musl +$ [ "${ARCH}" == "ppc64le" ] && export ARCH=powerpc64le +$ sudo install -o root -g root -m 0550 -T "${ROOTFS_DIR}/../../../../src/agent/target/${ARCH}-unknown-linux-${LIBC}/release/kata-agent" "${ROOTFS_DIR}/sbin/init" ``` ### Build an initrd image -``` -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder -$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}' +```bash +$ pushd kata-containers/tools/osbuilder/initrd-builder +$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh "${ROOTFS_DIR}"' +$ popd ``` ### Install the initrd image -``` -$ commit=$(git log --format=%h -1 HEAD) -$ date=$(date +%Y-%m-%d-%T.%N%z) +```bash +$ pushd kata-containers/tools/osbuilder/initrd-builder +$ commit="$(git log --format=%h -1 HEAD)" +$ date="$(date +%Y-%m-%d-%T.%N%z)" $ image="kata-containers-initrd-${date}-${commit}" $ sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}" $ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img) +$ popd ``` # Install guest kernel images @@ -391,44 +382,44 @@ Kata Containers makes use of upstream QEMU branch. The exact version and repository utilized can be found by looking at the [versions file](../versions.yaml). Find the correct version of QEMU from the versions file: -``` -$ source ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/scripts/lib.sh -$ qemu_version=$(get_from_kata_deps "assets.hypervisor.qemu.version") -$ echo ${qemu_version} +```bash +$ source kata-containers/tools/packaging/scripts/lib.sh +$ qemu_version="$(get_from_kata_deps "assets.hypervisor.qemu.version")" +$ echo "${qemu_version}" ``` Get source from the matching branch of QEMU: -``` -$ go get -d github.com/qemu/qemu -$ cd ${GOPATH}/src/github.com/qemu/qemu -$ git checkout ${qemu_version} -$ your_qemu_directory=${GOPATH}/src/github.com/qemu/qemu +```bash +$ git clone -b "${qemu_version}" https://github.com/qemu/qemu.git +$ your_qemu_directory="$(realpath qemu)" ``` There are scripts to manage the build and packaging of QEMU. For the examples below, set your environment as: -``` -$ go get -d github.com/kata-containers/kata-containers -$ packaging_dir="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging" +```bash +$ packaging_dir="$(realpath kata-containers/tools/packaging)" ``` Kata often utilizes patches for not-yet-upstream and/or backported fixes for components, including QEMU. These can be found in the [packaging/QEMU directory](../tools/packaging/qemu/patches), and it's *recommended* that you apply them. For example, suppose that you are going to build QEMU version 5.2.0, do: -``` -$ cd $your_qemu_directory -$ $packaging_dir/scripts/apply_patches.sh $packaging_dir/qemu/patches/5.2.x/ +```bash +$ "$packaging_dir/scripts/apply_patches.sh" "$packaging_dir/qemu/patches/5.2.x/" ``` To build utilizing the same options as Kata, you should make use of the `configure-hypervisor.sh` script. For example: -``` -$ cd $your_qemu_directory -$ $packaging_dir/scripts/configure-hypervisor.sh kata-qemu > kata.cfg +```bash +$ pushd "$your_qemu_directory" +$ "$packaging_dir/scripts/configure-hypervisor.sh" kata-qemu > kata.cfg $ eval ./configure "$(cat kata.cfg)" $ make -j $(nproc --ignore=1) +# Optional $ sudo -E make install +$ popd ``` +If you do not want to install the respective QEMU version, the configuration file can be modified to point to the correct binary. In `/etc/kata-containers/configuration.toml`, change `path = "/path/to/qemu/build/qemu-system-x86_64"` to point to the correct QEMU binary. + See the [static-build script for QEMU](../tools/packaging/static-build/qemu/build-static-qemu.sh) for a reference on how to get, setup, configure and build QEMU for Kata. ### Build a custom QEMU for aarch64/arm64 - REQUIRED @@ -439,11 +430,33 @@ See the [static-build script for QEMU](../tools/packaging/static-build/qemu/buil > under upstream review for supporting NVDIMM on aarch64. > You could build the custom `qemu-system-aarch64` as required with the following command: +```bash +$ git clone https://github.com/kata-containers/tests.git +$ script -fec 'sudo -E tests/.ci/install_qemu.sh' ``` -$ go get -d github.com/kata-containers/tests -$ script -fec 'sudo -E ${GOPATH}/src/github.com/kata-containers/tests/.ci/install_qemu.sh' + +## Build `virtiofsd` + +When using the file system type virtio-fs (default), `virtiofsd` is required + +```bash +$ pushd kata-containers/tools/packaging/static-build/virtiofsd +$ ./build-static-virtiofsd.sh +$ popd ``` +Modify `/etc/kata-containers/configuration.toml` and update value `virtio_fs_daemon = "/path/to/kata-containers/tools/packaging/static-build/virtiofsd/virtiofsd/virtiofsd"` to point to the binary. + +# Check hardware requirements + +You can check if your system is capable of creating a Kata Container by running the following: + +```bash +$ sudo kata-runtime check +``` + +If your system is *not* able to run Kata Containers, the previous command will error out and explain why. + # Run Kata Containers with Containerd Refer to the [How to use Kata Containers and Containerd](how-to/containerd-kata.md) how-to guide. @@ -474,7 +487,7 @@ See [Set up a debug console](#set-up-a-debug-console). ## Checking Docker default runtime -``` +```bash $ sudo docker info 2>/dev/null | grep -i "default runtime" | cut -d: -f2- | grep -q runc && echo "SUCCESS" || echo "ERROR: Incorrect default Docker runtime" ``` ## Set up a debug console @@ -491,7 +504,7 @@ contain either `/bin/sh` or `/bin/bash`. Enable debug_console_enabled in the `configuration.toml` configuration file: -``` +```toml [agent.kata] debug_console_enabled = true ``` @@ -502,7 +515,7 @@ This will pass `agent.debug_console agent.debug_console_vport=1026` to agent as For Kata Containers `2.0.x` releases, the `kata-runtime exec` command depends on the`kata-monitor` running, in order to get the sandbox's `vsock` address to connect to. Thus, first start the `kata-monitor` process. -``` +```bash $ sudo kata-monitor ``` @@ -564,10 +577,10 @@ an additional `coreutils` package. For example using CentOS: -``` -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder -$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs -$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos' +```bash +$ pushd kata-containers/tools/osbuilder/rootfs-builder +$ export ROOTFS_DIR="$(realpath ./rootfs)" +$ script -fec 'sudo -E USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos' ``` #### Build the debug image @@ -582,9 +595,10 @@ Install the image: >**Note**: When using an initrd image, replace the below rootfs image name `kata-containers.img` >with the initrd image name `kata-containers-initrd.img`. -``` +```bash $ name="kata-containers-centos-with-debug-console.img" $ sudo install -o root -g root -m 0640 kata-containers.img "/usr/share/kata-containers/${name}" +$ popd ``` Next, modify the `image=` values in the `[hypervisor.qemu]` section of the @@ -593,7 +607,7 @@ to specify the full path to the image name specified in the previous code section. Alternatively, recreate the symbolic link so it points to the new debug image: -``` +```bash $ (cd /usr/share/kata-containers && sudo ln -sf "$name" kata-containers.img) ``` @@ -604,7 +618,7 @@ to avoid all subsequently created containers from using the debug image. Create a container as normal. For example using `crictl`: -``` +```bash $ sudo crictl run -r kata container.yaml pod.yaml ``` @@ -617,7 +631,7 @@ those for firecracker / cloud-hypervisor. Add `agent.debug_console` to the guest kernel command line to allow the agent process to start a debug console. -``` +```bash $ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.debug_console"/g' "${kata_configuration_file}" ``` @@ -638,7 +652,7 @@ between the host and the guest. The kernel command line option `agent.debug_cons Add the parameter `agent.debug_console_vport=1026` to the kernel command line as shown below: -``` +```bash sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.debug_console_vport=1026"/g' "${kata_configuration_file}" ``` @@ -651,7 +665,7 @@ Next, connect to the debug console. The VSOCKS paths vary slightly between each VMM solution. In case of cloud-hypervisor, connect to the `vsock` as shown: -``` +```bash $ sudo su -c 'cd /var/run/vc/vm/${sandbox_id}/root/ && socat stdin unix-connect:clh.sock' CONNECT 1026 ``` @@ -659,7 +673,7 @@ CONNECT 1026 **Note**: You need to type `CONNECT 1026` and press `RETURN` key after entering the `socat` command. For firecracker, connect to the `hvsock` as shown: -``` +```bash $ sudo su -c 'cd /var/run/vc/firecracker/${sandbox_id}/root/ && socat stdin unix-connect:kata.hvsock' CONNECT 1026 ``` @@ -668,7 +682,7 @@ CONNECT 1026 For QEMU, connect to the `vsock` as shown: -``` +```bash $ sudo su -c 'cd /var/run/vc/vm/${sandbox_id} && socat "stdin,raw,echo=0,escape=0x11" "unix-connect:console.sock"' ``` @@ -681,7 +695,7 @@ If the image is created using [osbuilder](../tools/osbuilder), the following YAML file exists and contains details of the image and how it was created: -``` +```bash $ cat /var/lib/osbuilder/osbuilder.yaml ``` diff --git a/docs/how-to/README.md b/docs/how-to/README.md index aa09b49c73..1ad4860481 100644 --- a/docs/how-to/README.md +++ b/docs/how-to/README.md @@ -42,4 +42,5 @@ - [How to setup swap devices in guest kernel](how-to-setup-swap-devices-in-guest-kernel.md) - [How to run rootless vmm](how-to-run-rootless-vmm.md) - [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md) -- [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md) \ No newline at end of file +- [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md) +- [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md) diff --git a/docs/how-to/containerd-kata.md b/docs/how-to/containerd-kata.md index 1ec271416d..ce61b122cc 100644 --- a/docs/how-to/containerd-kata.md +++ b/docs/how-to/containerd-kata.md @@ -77,8 +77,8 @@ $ command -v containerd You can manually install CNI plugins as follows: ```bash -$ go get github.com/containernetworking/plugins -$ pushd $GOPATH/src/github.com/containernetworking/plugins +$ git clone https://github.com/containernetworking/plugins.git +$ pushd plugins $ ./build_linux.sh $ sudo mkdir /opt/cni $ sudo cp -r bin /opt/cni/ @@ -93,8 +93,8 @@ $ popd You can install the `cri-tools` from source code: ```bash -$ go get github.com/kubernetes-sigs/cri-tools -$ pushd $GOPATH/src/github.com/kubernetes-sigs/cri-tools +$ git clone https://github.com/kubernetes-sigs/cri-tools.git +$ pushd cri-tools $ make $ sudo -E make install $ popd diff --git a/docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md b/docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md new file mode 100644 index 0000000000..bc4e14a10d --- /dev/null +++ b/docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md @@ -0,0 +1,159 @@ +# Kata Containers with AMD SEV-SNP VMs + +## Disclaimer + +This guide is designed for developers and is - same as the Developer Guide - not intended for production systems or end users. It is advisable to only follow this guide on non-critical development systems. + +## Prerequisites + +To run Kata Containers in SNP-VMs, the following software stack is used. + +![Kubernetes integration with shimv2](./images/SNP-stack.svg) + +The host BIOS and kernel must be capable of supporting AMD SEV-SNP and configured accordingly. For Kata Containers, the host kernel with branch [`sev-snp-iommu-avic_5.19-rc6_v3`](https://github.com/AMDESE/linux/tree/sev-snp-iommu-avic_5.19-rc6_v3) and commit [`3a88547`](https://github.com/AMDESE/linux/commit/3a885471cf89156ea555341f3b737ad2a8d9d3d0) is known to work in conjunction with SEV Firmware version 1.51.3 (0xh\_1.33.03) available on AMD's [SEV developer website](https://developer.amd.com/sev/). See [AMD's guide](https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel) to configure the host accordingly. Verify that you are able to run SEV-SNP encrypted VMs first. The guest components required for Kata Containers are built as described below. + +**Tip**: It is easiest to first have Kata Containers running on your system and then modify it to run containers in SNP-VMs. Follow the [Developer guide](../Developer-Guide.md#warning) and then follow the below steps. Nonetheless, you can just follow this guide from the start. + +## How to build + +Follow all of the below steps to install Kata Containers with SNP-support from scratch. These steps mostly follow the developer guide with modifications to support SNP + +__Steps from the Developer Guide:__ +- Get all the [required components](../Developer-Guide.md#requirements-to-build-individual-components) for building the kata-runtime +- [Build the and install kata-runtime](../Developer-Guide.md#build-and-install-the-kata-containers-runtime) +- [Build a custom agent](../Developer-Guide.md#build-a-custom-kata-agent---optional) +- [Create an initrd image](../Developer-Guide.md#create-an-initrd-image---optional) by first building a rootfs, then building the initrd based on the rootfs, use a custom agent and install. `ubuntu` works as the distribution of choice. +- Get the [required components](../../tools/packaging/kernel/README.md#requirements) to build a custom kernel + +__SNP-specific steps:__ +- Build the SNP-specific kernel as shown below (see this [guide](../../tools/packaging/kernel/README.md#build-kata-containers-kernel) for more information) +```bash +$ pushd kata-containers/tools/packaging/kernel/ +$ ./build-kernel.sh -a x86_64 -x snp setup +$ ./build-kernel.sh -a x86_64 -x snp build +$ sudo -E PATH="${PATH}" ./build-kernel.sh -x snp install +$ popd +``` +- Build a current OVMF capable of SEV-SNP: +```bash +$ pushd kata-containers/tools/packaging/static-build/ovmf +$ ./build.sh +$ tar -xvf edk2-x86_64.tar.gz +$ popd +``` +- Build a custom QEMU +```bash +$ source kata-containers/tools/packaging/scripts/lib.sh +$ qemu_url="$(get_from_kata_deps "assets.hypervisor.qemu.snp.url")" +$ qemu_branch="$(get_from_kata_deps "assets.hypervisor.qemu.snp.branch")" +$ qemu_commit="$(get_from_kata_deps "assets.hypervisor.qemu.snp.commit")" +$ git clone -b "${qemu_branch}" "${qemu_url}" +$ pushd qemu +$ git checkout "${qemu_commit}" +$ ./configure --target-list=x86_64-softmmu --enable-debug +$ make -j "$(nproc)" +$ popd +``` + +### Kata Containers Configuration for SNP + +The configuration file located at `/etc/kata-containers/configuration.toml` must be adapted as follows to support SNP-VMs: +- Use the SNP-specific kernel for the guest VM (change path) +```toml +kernel = "/usr/share/kata-containers/vmlinuz-snp.container" +``` +- Enable the use of an initrd (uncomment) +```toml +initrd = "/usr/share/kata-containers/kata-containers-initrd.img" +``` +- Disable the use of a rootfs (comment out) +```toml +# image = "/usr/share/kata-containers/kata-containers.img" +``` +- Use the custom QEMU capable of SNP (change path) +```toml +path = "/path/to/qemu/build/qemu-system-x86_64" +``` +- Use `virtio-9p` device since `virtio-fs` is unsupported due to bugs / shortcomings in QEMU version [`snp-v3`](https://github.com/AMDESE/qemu/tree/snp-v3) for SEV and SEV-SNP (change value) +```toml +shared_fs = "virtio-9p" +``` +- Disable `virtiofsd` since it is no longer required (comment out) +```toml +# virtio_fs_daemon = "/usr/libexec/virtiofsd" +``` +- Disable NVDIMM (uncomment) +```toml +disable_image_nvdimm = true +``` +- Disable shared memory (uncomment) +```toml +file_mem_backend = "" +``` +- Enable confidential guests (uncomment) +```toml +confidential_guest = true +``` +- Enable SNP-VMs (uncomment) +```toml +sev_snp_guest = true +``` + - Configure an OVMF (add path) +```toml +firmware = "/path/to/kata-containers/tools/packaging/static-build/ovmf/opt/kata/share/ovmf/OVMF.fd" +``` + +## Test Kata Containers with Containerd + +With Kata Containers configured to support SNP-VMs, we use containerd to test and deploy containers in these VMs. + +### Install Containerd +If not already present, follow [this guide](./containerd-kata.md#install) to install containerd and its related components including `CNI` and the `cri-tools` (skip Kata Containers since we already installed it) + +### Containerd Configuration + +Follow [this guide](./containerd-kata.md#configuration) to configure containerd to use Kata Containers + +## Run Kata Containers in SNP-VMs + +Run the below commands to start a container. See [this guide](./containerd-kata.md#run) for more information +```bash +$ sudo ctr image pull docker.io/library/busybox:latest +$ sudo ctr run --cni --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh +``` + +### Check for active SNP: + +Inside the running container, run the following commands to check if SNP is active. It should look something like this: +``` +/ # dmesg | grep -i sev +[ 0.299242] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP +[ 0.472286] SEV: Using SNP CPUID table, 31 entries present. +[ 0.514574] SEV: SNP guest platform device initialized. +[ 0.885425] sev-guest sev-guest: Initialized SEV guest driver (using vmpck_id 0) +``` + +### Obtain an SNP Attestation Report + +To obtain an attestation report inside the container, the `/dev/sev-guest` must first be configured. As of now, the VM does not perform this step, however it can be performed inside the container, either in the terminal or in code. + +Example for shell: +``` +/ # SNP_MAJOR=$(cat /sys/devices/virtual/misc/sev-guest/dev | awk -F: '{print $1}') +/ # SNP_MINOR=$(cat /sys/devices/virtual/misc/sev-guest/dev | awk -F: '{print $2}') +/ # mknod -m 600 /dev/sev-guest c "${SNP_MAJOR}" "${SNP_MINOR}" +``` + +## Known Issues + +- Support for cgroups v2 is still [work in progress](https://github.com/kata-containers/kata-containers/issues/927). If issues occur due to cgroups v2 becoming the default in newer systems, one possible solution is to downgrade cgroups to v1: +```bash +sudo sed -i 's/^\(GRUB_CMDLINE_LINUX=".*\)"/\1 systemd.unified_cgroup_hierarchy=0"/' /etc/default/grub +sudo update-grub +sudo reboot +``` +- If both SEV and SEV-SNP are supported by the host, Kata Containers uses SEV-SNP by default. You can verify what features are enabled by checking `/sys/module/kvm_amd/parameters/sev` and `sev_snp`. This means that Kata Containers can not run both SEV-SNP-VMs and SEV-VMs at the same time. If SEV is to be used by Kata Containers instead, reload the `kvm_amd` kernel module without SNP-support, this will disable SNP-support for the entire platform. +```bash +sudo rmmod kvm_amd && sudo modprobe kvm_amd sev_snp=0 +``` + diff --git a/docs/how-to/images/SNP-stack.svg b/docs/how-to/images/SNP-stack.svg new file mode 100644 index 0000000000..9d32557f20 --- /dev/null +++ b/docs/how-to/images/SNP-stack.svg @@ -0,0 +1,4 @@ + + + +
Linux Kernel / KVM
Linux Kernel / KVM
QEMU
QEMU
BIOS
BIOS
OVMF
OVMF
Linux Kernel
Linux Kernel
Host
Host
VM
VMText is not SVG - cannot display \ No newline at end of file diff --git a/src/runtime/Makefile b/src/runtime/Makefile index b05106318b..7d97a679aa 100644 --- a/src/runtime/Makefile +++ b/src/runtime/Makefile @@ -56,6 +56,7 @@ BINLIBEXECLIST := BIN_PREFIX = $(PROJECT_TYPE) PROJECT_DIR = $(PROJECT_TAG) IMAGENAME = $(PROJECT_TAG).img +INITRDNAME = $(PROJECT_TAG)-initrd.img TARGET = $(BIN_PREFIX)-runtime RUNTIME_OUTPUT = $(CURDIR)/$(TARGET) @@ -110,6 +111,7 @@ PKGLIBEXECDIR := $(LIBEXECDIR)/$(PROJECT_DIR) KERNELDIR := $(PKGDATADIR) IMAGEPATH := $(PKGDATADIR)/$(IMAGENAME) +INITRDPATH := $(PKGDATADIR)/$(INITRDNAME) FIRMWAREPATH := FIRMWAREVOLUMEPATH := @@ -401,6 +403,8 @@ USER_VARS += FCVALIDJAILERPATHS USER_VARS += SYSCONFIG USER_VARS += IMAGENAME USER_VARS += IMAGEPATH +USER_VARS += INITRDNAME +USER_VARS += INITRDPATH USER_VARS += MACHINETYPE USER_VARS += KERNELDIR USER_VARS += KERNELTYPE diff --git a/src/runtime/config/configuration-qemu.toml.in b/src/runtime/config/configuration-qemu.toml.in index d0a711dcfe..7b9166adc1 100644 --- a/src/runtime/config/configuration-qemu.toml.in +++ b/src/runtime/config/configuration-qemu.toml.in @@ -15,6 +15,7 @@ path = "@QEMUPATH@" kernel = "@KERNELPATH@" image = "@IMAGEPATH@" +# initrd = "@INITRDPATH@" machine_type = "@MACHINETYPE@" # Enable confidential guest support. @@ -33,6 +34,12 @@ machine_type = "@MACHINETYPE@" # Default false # confidential_guest = true +# Choose AMD SEV-SNP confidential guests +# In case of using confidential guests on AMD hardware that supports both SEV +# and SEV-SNP, the following enables SEV-SNP guests. SEV guests are default. +# Default false +# sev_snp_guest = true + # Enable running QEMU VMM as a non-root user. # By default QEMU VMM run as root. When this is set to true, QEMU VMM process runs as # a non-root random user. See documentation for the limitations of this mode. diff --git a/src/runtime/pkg/govmm/qemu/qemu.go b/src/runtime/pkg/govmm/qemu/qemu.go index 4b36df2ea3..5b04a01c1d 100644 --- a/src/runtime/pkg/govmm/qemu/qemu.go +++ b/src/runtime/pkg/govmm/qemu/qemu.go @@ -231,6 +231,9 @@ const ( // SEVGuest represents an SEV guest object SEVGuest ObjectType = "sev-guest" + // SNPGuest represents an SNP guest object + SNPGuest ObjectType = "sev-snp-guest" + // SecExecGuest represents an s390x Secure Execution (Protected Virtualization in QEMU) object SecExecGuest ObjectType = "s390-pv-guest" // PEFGuest represent ppc64le PEF(Protected Execution Facility) object. @@ -295,6 +298,8 @@ func (object Object) Valid() bool { case TDXGuest: return object.ID != "" && object.File != "" && object.DeviceID != "" case SEVGuest: + fallthrough + case SNPGuest: return object.ID != "" && object.File != "" && object.CBitPos != 0 && object.ReducedPhysBits != 0 case SecExecGuest: return object.ID != "" @@ -349,6 +354,8 @@ func (object Object) QemuParams(config *Config) []string { deviceParams = append(deviceParams, fmt.Sprintf("config-firmware-volume=%s", object.FirmwareVolume)) } case SEVGuest: + fallthrough + case SNPGuest: objectParams = append(objectParams, string(object.Type)) objectParams = append(objectParams, fmt.Sprintf("id=%s", object.ID)) objectParams = append(objectParams, fmt.Sprintf("cbitpos=%d", object.CBitPos)) diff --git a/src/runtime/pkg/katautils/config-settings.go.in b/src/runtime/pkg/katautils/config-settings.go.in index 37dbfee45a..601d95612c 100644 --- a/src/runtime/pkg/katautils/config-settings.go.in +++ b/src/runtime/pkg/katautils/config-settings.go.in @@ -86,6 +86,7 @@ const defaultVhostUserStorePath string = "/var/run/kata-containers/vhost-user/" const defaultRxRateLimiterMaxRate = uint64(0) const defaultTxRateLimiterMaxRate = uint64(0) const defaultConfidentialGuest = false +const defaultSevSnpGuest = false const defaultGuestSwap = false const defaultRootlessHypervisor = false const defaultDisableSeccomp = false diff --git a/src/runtime/pkg/katautils/config.go b/src/runtime/pkg/katautils/config.go index f3bc06bdf5..dd996b4032 100644 --- a/src/runtime/pkg/katautils/config.go +++ b/src/runtime/pkg/katautils/config.go @@ -149,6 +149,7 @@ type hypervisor struct { DisableVhostNet bool `toml:"disable_vhost_net"` GuestMemoryDumpPaging bool `toml:"guest_memory_dump_paging"` ConfidentialGuest bool `toml:"confidential_guest"` + SevSnpGuest bool `toml:"sev_snp_guest"` GuestSwap bool `toml:"enable_guest_swap"` Rootless bool `toml:"rootless"` DisableSeccomp bool `toml:"disable_seccomp"` @@ -827,6 +828,7 @@ func newQemuHypervisorConfig(h hypervisor) (vc.HypervisorConfig, error) { GuestMemoryDumpPath: h.GuestMemoryDumpPath, GuestMemoryDumpPaging: h.GuestMemoryDumpPaging, ConfidentialGuest: h.ConfidentialGuest, + SevSnpGuest: h.SevSnpGuest, GuestSwap: h.GuestSwap, Rootless: h.Rootless, LegacySerial: h.LegacySerial, @@ -1221,6 +1223,7 @@ func GetDefaultHypervisorConfig() vc.HypervisorConfig { TxRateLimiterMaxRate: defaultTxRateLimiterMaxRate, SGXEPCSize: defaultSGXEPCSize, ConfidentialGuest: defaultConfidentialGuest, + SevSnpGuest: defaultSevSnpGuest, GuestSwap: defaultGuestSwap, Rootless: defaultRootlessHypervisor, DisableSeccomp: defaultDisableSeccomp, diff --git a/src/runtime/virtcontainers/clh.go b/src/runtime/virtcontainers/clh.go index d93ceed675..7595eaaad2 100644 --- a/src/runtime/virtcontainers/clh.go +++ b/src/runtime/virtcontainers/clh.go @@ -422,6 +422,8 @@ func (clh *cloudHypervisor) enableProtection() error { case sevProtection: return errors.New("SEV protection is not supported by Cloud Hypervisor") + case snpProtection: + return errors.New("SEV-SNP protection is not supported by Cloud Hypervisor") default: return errors.New("This system doesn't support Confidentian Computing (Guest Protection)") diff --git a/src/runtime/virtcontainers/documentation/api/1.0/api.md b/src/runtime/virtcontainers/documentation/api/1.0/api.md index 6455e19d0b..c7215cc97c 100644 --- a/src/runtime/virtcontainers/documentation/api/1.0/api.md +++ b/src/runtime/virtcontainers/documentation/api/1.0/api.md @@ -348,6 +348,10 @@ type HypervisorConfig struct { // Enable or disable different hardware features, ranging // from memory encryption to both memory and CPU-state encryption and integrity. ConfidentialGuest bool + + // Enables SEV-SNP guests in case both AMD SEV and SNP are supported. + // SEV is default. + SevSnpGuest bool } ``` diff --git a/src/runtime/virtcontainers/hypervisor.go b/src/runtime/virtcontainers/hypervisor.go index 01604fe1bf..51cf678b8d 100644 --- a/src/runtime/virtcontainers/hypervisor.go +++ b/src/runtime/virtcontainers/hypervisor.go @@ -530,6 +530,9 @@ type HypervisorConfig struct { // from memory encryption to both memory and CPU-state encryption and integrity. ConfidentialGuest bool + // Enable SEV-SNP guests on AMD machines capable of both + SevSnpGuest bool + // BootToBeTemplate used to indicate if the VM is created to be a template VM BootToBeTemplate bool @@ -873,6 +876,11 @@ const ( // Exclude from lint checking for it won't be used on arm64 code sevProtection + // AMD Secure Encrypted Virtualization - Secure Nested Paging (SEV-SNP) + // https://developer.amd.com/sev/ + // Exclude from lint checking for it won't be used on arm64 code + snpProtection + // IBM POWER 9 Protected Execution Facility // https://www.kernel.org/doc/html/latest/powerpc/ultravisor.html // Exclude from lint checking for it won't be used on arm64 code @@ -889,6 +897,7 @@ var guestProtectionStr = [...]string{ pefProtection: "pef", seProtection: "se", sevProtection: "sev", + snpProtection: "snp", tdxProtection: "tdx", } diff --git a/src/runtime/virtcontainers/hypervisor_linux_amd64.go b/src/runtime/virtcontainers/hypervisor_linux_amd64.go index 38313d05b1..8cfc9aca9a 100644 --- a/src/runtime/virtcontainers/hypervisor_linux_amd64.go +++ b/src/runtime/virtcontainers/hypervisor_linux_amd64.go @@ -13,6 +13,8 @@ const ( tdxCPUFlag = "tdx" sevKvmParameterPath = "/sys/module/kvm_amd/parameters/sev" + + snpKvmParameterPath = "/sys/module/kvm_amd/parameters/sev_snp" ) // Implementation of this function is architecture specific @@ -26,6 +28,13 @@ func availableGuestProtection() (guestProtection, error) { if d, err := os.Stat(tdxSysFirmwareDir); (err == nil && d.IsDir()) || flags[tdxCPUFlag] { return tdxProtection, nil } + // SEV-SNP is supported and enabled when the kvm module `sev_snp` parameter is set to `Y` + // SEV-SNP support infers SEV (-ES) support + if _, err := os.Stat(snpKvmParameterPath); err == nil { + if c, err := os.ReadFile(snpKvmParameterPath); err == nil && len(c) > 0 && (c[0] == 'Y') { + return snpProtection, nil + } + } // SEV is supported and enabled when the kvm module `sev` parameter is set to `1` (or `Y` for linux >= 5.12) if _, err := os.Stat(sevKvmParameterPath); err == nil { if c, err := os.ReadFile(sevKvmParameterPath); err == nil && len(c) > 0 && (c[0] == '1' || c[0] == 'Y') { diff --git a/src/runtime/virtcontainers/qemu_amd64.go b/src/runtime/virtcontainers/qemu_amd64.go index b7680a3180..370a7cd9d8 100644 --- a/src/runtime/virtcontainers/qemu_amd64.go +++ b/src/runtime/virtcontainers/qemu_amd64.go @@ -24,6 +24,8 @@ type qemuAmd64 struct { // inherit from qemuArchBase, overwrite methods if needed qemuArchBase + snpGuest bool + vmFactory bool devLoadersCount uint32 @@ -122,6 +124,7 @@ func newQemuArch(config HypervisorConfig) (qemuArch, error) { legacySerial: config.LegacySerial, }, vmFactory: factory, + snpGuest: config.SevSnpGuest, } if config.ConfidentialGuest { @@ -169,6 +172,21 @@ func (q *qemuAmd64) bridges(number uint32) { q.Bridges = genericBridges(number, q.qemuMachine.Type) } +func (q *qemuAmd64) cpuModel() string { + var err error + cpuModel := defaultCPUModel + + // Temporary until QEMU cpu model 'host' supports AMD SEV-SNP + protection, err := availableGuestProtection() + if err == nil { + if protection == snpProtection && q.snpGuest { + cpuModel = "EPYC-v4" + } + } + + return cpuModel +} + func (q *qemuAmd64) memoryTopology(memoryMb, hostMemoryMb uint64, slots uint8) govmmQemu.Memory { return genericMemoryTopology(memoryMb, hostMemoryMb, slots, q.memoryOffset) } @@ -197,6 +215,11 @@ func (q *qemuAmd64) enableProtection() error { if err != nil { return err } + // Configure SNP only if specified in config + if q.protection == snpProtection && !q.snpGuest { + q.protection = sevProtection + } + logger := hvLogger.WithFields(logrus.Fields{ "subsystem": "qemuAmd64", "machine": q.qemuMachine, @@ -219,6 +242,13 @@ func (q *qemuAmd64) enableProtection() error { q.qemuMachine.Options += "confidential-guest-support=sev" logger.Info("Enabling SEV guest protection") return nil + case snpProtection: + if q.qemuMachine.Options != "" { + q.qemuMachine.Options += "," + } + q.qemuMachine.Options += "confidential-guest-support=snp" + logger.Info("Enabling SNP guest protection") + return nil // TODO: Add support for other x86_64 technologies @@ -263,6 +293,16 @@ func (q *qemuAmd64) appendProtectionDevice(devices []govmmQemu.Device, firmware, CBitPos: cpuid.AMDMemEncrypt.CBitPosition, ReducedPhysBits: cpuid.AMDMemEncrypt.PhysAddrReduction, }), "", nil + case snpProtection: + return append(devices, + govmmQemu.Object{ + Type: govmmQemu.SNPGuest, + ID: "snp", + Debug: false, + File: firmware, + CBitPos: cpuid.AMDMemEncrypt.CBitPosition, + ReducedPhysBits: 1, + }), "", nil case noneProtection: return devices, firmware, nil diff --git a/src/runtime/virtcontainers/qemu_amd64_test.go b/src/runtime/virtcontainers/qemu_amd64_test.go index 740cb6460b..73641ab455 100644 --- a/src/runtime/virtcontainers/qemu_amd64_test.go +++ b/src/runtime/virtcontainers/qemu_amd64_test.go @@ -293,6 +293,26 @@ func TestQemuAmd64AppendProtectionDevice(t *testing.T) { assert.Equal(expectedOut, devices) + // snp protection + amd64.(*qemuAmd64).protection = snpProtection + + devices, bios, err = amd64.appendProtectionDevice(devices, firmware, "") + assert.NoError(err) + assert.Empty(bios) + + expectedOut = append(expectedOut, + govmmQemu.Object{ + Type: govmmQemu.SNPGuest, + ID: "snp", + Debug: false, + File: firmware, + CBitPos: cpuid.AMDMemEncrypt.CBitPosition, + ReducedPhysBits: 1, + }, + ) + + assert.Equal(expectedOut, devices) + // tdxProtection amd64.(*qemuAmd64).protection = tdxProtection diff --git a/src/runtime/virtcontainers/qemu_arm64_test.go b/src/runtime/virtcontainers/qemu_arm64_test.go index d78f4474b3..15ae382223 100644 --- a/src/runtime/virtcontainers/qemu_arm64_test.go +++ b/src/runtime/virtcontainers/qemu_arm64_test.go @@ -209,6 +209,13 @@ func TestQemuArm64AppendProtectionDevice(t *testing.T) { assert.Empty(bios) assert.NoError(err) + // SNP protection + arm64.(*qemuArm64).protection = snpProtection + devices, bios, err = arm64.appendProtectionDevice(devices, firmware, "") + assert.Empty(devices) + assert.Empty(bios) + assert.NoError(err) + // TDX protection arm64.(*qemuArm64).protection = tdxProtection devices, bios, err = arm64.appendProtectionDevice(devices, firmware, "") diff --git a/src/runtime/virtcontainers/qemu_ppc64le_test.go b/src/runtime/virtcontainers/qemu_ppc64le_test.go index 15e75a91b2..fd59f51b3a 100644 --- a/src/runtime/virtcontainers/qemu_ppc64le_test.go +++ b/src/runtime/virtcontainers/qemu_ppc64le_test.go @@ -79,6 +79,12 @@ func TestQemuPPC64leAppendProtectionDevice(t *testing.T) { assert.Error(err) assert.Empty(bios) + //SNP protection + ppc64le.(*qemuPPC64le).protection = snpProtection + devices, bios, err = ppc64le.appendProtectionDevice(devices, firmware, "") + assert.Error(err) + assert.Empty(bios) + //TDX protection ppc64le.(*qemuPPC64le).protection = tdxProtection devices, bios, err = ppc64le.appendProtectionDevice(devices, firmware, "") diff --git a/src/runtime/virtcontainers/qemu_s390x_test.go b/src/runtime/virtcontainers/qemu_s390x_test.go index ada3cefc6e..909e1b87ea 100644 --- a/src/runtime/virtcontainers/qemu_s390x_test.go +++ b/src/runtime/virtcontainers/qemu_s390x_test.go @@ -136,6 +136,12 @@ func TestQemuS390xAppendProtectionDevice(t *testing.T) { assert.Error(err) assert.Empty(bios) + // SNP protection + s390x.(*qemuS390x).protection = snpProtection + devices, bios, err = s390x.appendProtectionDevice(devices, firmware, "") + assert.Error(err) + assert.Empty(bios) + // Secure Execution protection s390x.(*qemuS390x).protection = seProtection diff --git a/tools/packaging/kernel/README.md b/tools/packaging/kernel/README.md index cb4840260e..dbf9915403 100644 --- a/tools/packaging/kernel/README.md +++ b/tools/packaging/kernel/README.md @@ -9,7 +9,8 @@ automates the process to build a kernel for Kata Containers. The `build-kernel.sh` script requires an installed Golang version matching the [component build requirements](../../../docs/Developer-Guide.md#requirements-to-build-individual-components). It also requires [yq](https://github.com/mikefarah/yq) version 3.4.1 -> **Hint**: `snap install yq --channel=v3/stable` +> **Hint**: `snap install yq --channel=v3/stable` \ +> **or** `go install github.com/mikefarah/yq/v3@latest` The Linux kernel scripts further require a few packages (flex, bison, and libelf-dev) @@ -53,7 +54,7 @@ Options: ``` Example: -``` +```bash $ ./build-kernel.sh -v 5.10.25 -g nvidia -f -d setup ``` > **Note** @@ -68,8 +69,8 @@ $ ./build-kernel.sh -v 5.10.25 -g nvidia -f -d setup ## Setup kernel source code ```bash -$ go get -d -u github.com/kata-containers/kata-containers -$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel +$ git clone github.com/kata-containers/kata-containers +$ cd kata-containers/tools/packaging/kernel $ ./build-kernel.sh setup ``` diff --git a/tools/packaging/kernel/build-kernel.sh b/tools/packaging/kernel/build-kernel.sh index aa849a7961..5b8e880d2d 100755 --- a/tools/packaging/kernel/build-kernel.sh +++ b/tools/packaging/kernel/build-kernel.sh @@ -101,7 +101,7 @@ Options: -t : Hypervisor_target. -u : Kernel URL to be used to download the kernel tarball. -v : Kernel version to use if kernel path not provided. - -x : Confidential guest protection type, such as sev and tdx + -x : Confidential guest protection type, such as sev, snp and tdx EOF exit "$exit_code" } @@ -525,7 +525,7 @@ main() { x) conf_guest="${OPTARG}" case "$conf_guest" in - sev|tdx) ;; + sev|snp|tdx) ;; *) die "Confidential guest type '$conf_guest' not supported" ;; esac ;; diff --git a/tools/packaging/kernel/configs/fragments/x86_64/snp/snp.conf b/tools/packaging/kernel/configs/fragments/x86_64/snp/snp.conf new file mode 100644 index 0000000000..d629e31c93 --- /dev/null +++ b/tools/packaging/kernel/configs/fragments/x86_64/snp/snp.conf @@ -0,0 +1,10 @@ +# !s390x !ppc64le !arm64 +# enable sev-snp support +CONFIG_AMD_MEM_ENCRYPT=y +CONFIG_SEV_GUEST=y +CONFIG_VIRT_DRIVERS=y + +# Prepare kernel for direct boot using OVMF +CONFIG_EFI=y +CONFIG_EFI_STUB=y + diff --git a/versions.yaml b/versions.yaml index 20c040ead9..264b630ff6 100644 --- a/versions.yaml +++ b/versions.yaml @@ -102,6 +102,11 @@ assets: description: "VMM that uses KVM and supports TDX" url: "https://github.com/intel/qemu-dcp" tag: "SPR-BKC-QEMU-v2.5" + snp: + description: "VMM that uses KVM and supports AMD SEV-SNP" + url: "https://github.com/AMDESE/qemu" + branch: "snp-v3" + commit: "ffa95097ee" qemu-experimental: description: "QEMU with virtiofs support" @@ -162,6 +167,10 @@ assets: description: "Linux kernel that supports SEV" url: "https://cdn.kernel.org/pub/linux/kernel/v5.x/" version: "v5.19.2" + snp: + description: "Linux kernel that supports AMD SEV-SNP for VMs" + url: "https://cdn.kernel.org/pub/linux/kernel/v5.x/" + version: "v5.19.2" kernel-experimental: description: "Linux kernel with virtio-fs support"