github/kata-containers
1
0
Fork 1
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-08-02 00:02:01 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #4931 from jpecholt/snp-support

Browse Source 
Added SNP-Support for Kata-Containers
This commit is contained in:
Peng Tao 2022-09-27 14:17:54 +08:00 committed by GitHub
commit 8a2df6b31c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
23 changed files with 458 additions and 135 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

262
docs/Developer-Guide.md
View File

@ -33,51 +33,41 @@ You need to install the following to build Kata Containers components:
- `make`. - `make`.
- `gcc` (required for building the shim and runtime). - `gcc` (required for building the shim and runtime).
# Build and install the Kata Containers runtime # Build and install Kata Containers
## Build and install the Kata Containers runtime
``` ```bash
$ go get -d -u github.com/kata-containers/kata-containers $ git clone https://github.com/kata-containers/kata-containers.git
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/runtime $ pushd kata-containers/src/runtime
$ make && sudo -E PATH=$PATH make install $ make && sudo -E "PATH=$PATH" make install
$ sudo mkdir -p /etc/kata-containers/
$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
$ popd
``` ```
The build will create the following: The build will create the following:
- runtime binary: `/usr/local/bin/kata-runtime` and `/usr/local/bin/containerd-shim-kata-v2` - runtime binary: `/usr/local/bin/kata-runtime` and `/usr/local/bin/containerd-shim-kata-v2`
- configuration file: `/usr/share/defaults/kata-containers/configuration.toml` - configuration file: `/usr/share/defaults/kata-containers/configuration.toml` and `/etc/kata-containers/configuration.toml`
# Check hardware requirements
You can check if your system is capable of creating a Kata Container by running the following:
```
$ sudo kata-runtime check
```
If your system is *not* able to run Kata Containers, the previous command will error out and explain why.
## Configure to use initrd or rootfs image ## Configure to use initrd or rootfs image
Kata containers can run with either an initrd image or a rootfs image. Kata containers can run with either an initrd image or a rootfs image.
If you want to test with `initrd`, make sure you have `initrd = /usr/share/kata-containers/kata-containers-initrd.img` If you want to test with `initrd`, make sure you have uncommented `initrd = /usr/share/kata-containers/kata-containers-initrd.img`
in your configuration file, commenting out the `image` line: in your configuration file, commenting out the `image` line in
`/etc/kata-containers/configuration.toml`. For example:
`/usr/share/defaults/kata-containers/configuration.toml` and comment out the `image` line with the following. For example: ```bash
```
$ sudo mkdir -p /etc/kata-containers/
$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
$ sudo sed -i 's/^\(image =.*\)/# \1/g' /etc/kata-containers/configuration.toml $ sudo sed -i 's/^\(image =.*\)/# \1/g' /etc/kata-containers/configuration.toml
$ sudo sed -i 's/^# \(initrd =.*\)/\1/g' /etc/kata-containers/configuration.toml
``` ```
You can create the initrd image as shown in the [create an initrd image](#create-an-initrd-image---optional) section. You can create the initrd image as shown in the [create an initrd image](#create-an-initrd-image---optional) section.
If you want to test with a rootfs `image`, make sure you have `image = /usr/share/kata-containers/kata-containers.img` If you want to test with a rootfs `image`, make sure you have uncommented `image = /usr/share/kata-containers/kata-containers.img`
in your configuration file, commenting out the `initrd` line. For example: in your configuration file, commenting out the `initrd` line. For example:
``` ```bash
$ sudo mkdir -p /etc/kata-containers/
$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
$ sudo sed -i 's/^\(initrd =.*\)/# \1/g' /etc/kata-containers/configuration.toml $ sudo sed -i 's/^\(initrd =.*\)/# \1/g' /etc/kata-containers/configuration.toml
``` ```
The rootfs image is created as shown in the [create a rootfs image](#create-a-rootfs-image) section. The rootfs image is created as shown in the [create a rootfs image](#create-a-rootfs-image) section.
@ -90,7 +80,7 @@ rootfs `image`(100MB+).
Enable seccomp as follows: Enable seccomp as follows:
``` ```bash
$ sudo sed -i '/^disable_guest_seccomp/ s/true/false/' /etc/kata-containers/configuration.toml $ sudo sed -i '/^disable_guest_seccomp/ s/true/false/' /etc/kata-containers/configuration.toml
``` ```
@ -100,9 +90,7 @@ This will pass container seccomp profiles to the kata agent.
Enable full debug as follows: Enable full debug as follows:
``` ```bash
$ sudo mkdir -p /etc/kata-containers/
$ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
$ sudo sed -i -e 's/^# *\(enable_debug\).*=.*$/\1 = true/g' /etc/kata-containers/configuration.toml $ sudo sed -i -e 's/^# *\(enable_debug\).*=.*$/\1 = true/g' /etc/kata-containers/configuration.toml
$ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.log=debug initcall_debug"/g' /etc/kata-containers/configuration.toml $ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.log=debug initcall_debug"/g' /etc/kata-containers/configuration.toml
``` ```
@ -175,7 +163,7 @@ and offers possible workarounds and fixes.
it stores. When messages are suppressed, it is noted in the logs. This can be checked it stores. When messages are suppressed, it is noted in the logs. This can be checked
for by looking for those notifications, such as: for by looking for those notifications, such as:
```sh ```bash
$ sudo journalctl --since today | fgrep Suppressed $ sudo journalctl --since today | fgrep Suppressed
Jun 29 14:51:17 mymachine systemd-journald[346]: Suppressed 4150 messages from /system.slice/docker.service Jun 29 14:51:17 mymachine systemd-journald[346]: Suppressed 4150 messages from /system.slice/docker.service
``` ```
@ -200,7 +188,7 @@ RateLimitBurst=0
Restart `systemd-journald` for the changes to take effect: Restart `systemd-journald` for the changes to take effect:
```sh ```bash
$ sudo systemctl restart systemd-journald $ sudo systemctl restart systemd-journald
``` ```
@ -214,25 +202,24 @@ $ sudo systemctl restart systemd-journald
The agent is built with a statically linked `musl.` The default `libc` used is `musl`, but on `ppc64le` and `s390x`, `gnu` should be used. To configure this: The agent is built with a statically linked `musl.` The default `libc` used is `musl`, but on `ppc64le` and `s390x`, `gnu` should be used. To configure this:
``` ```bash
$ export ARCH=$(uname -m) $ export ARCH="$(uname -m)"
$ if [ "$ARCH" = "ppc64le" -o "$ARCH" = "s390x" ]; then export LIBC=gnu; else export LIBC=musl; fi $ if [ "$ARCH" = "ppc64le" -o "$ARCH" = "s390x" ]; then export LIBC=gnu; else export LIBC=musl; fi
$ [ ${ARCH} == "ppc64le" ] && export ARCH=powerpc64le $ [ "${ARCH}" == "ppc64le" ] && export ARCH=powerpc64le
$ rustup target add ${ARCH}-unknown-linux-${LIBC} $ rustup target add "${ARCH}-unknown-linux-${LIBC}"
``` ```
To build the agent: To build the agent:
``` ```bash
$ go get -d -u github.com/kata-containers/kata-containers $ make -C kata-containers/src/agent
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/src/agent && make
``` ```
The agent is built with seccomp capability by default. The agent is built with seccomp capability by default.
If you want to build the agent without the seccomp capability, you need to run `make` with `SECCOMP=no` as follows. If you want to build the agent without the seccomp capability, you need to run `make` with `SECCOMP=no` as follows.
``` ```bash
$ make -C $GOPATH/src/github.com/kata-containers/kata-containers/src/agent SECCOMP=no $ make -C kata-containers/src/agent SECCOMP=no
``` ```
> **Note:** > **Note:**
@ -240,13 +227,6 @@ $ make -C $GOPATH/src/github.com/kata-containers/kata-containers/src/agent SECCO
> - If you enable seccomp in the main configuration file but build the agent without seccomp capability, > - If you enable seccomp in the main configuration file but build the agent without seccomp capability,
> the runtime exits conservatively with an error message. > the runtime exits conservatively with an error message.
## Get the osbuilder
```
$ go get -d -u github.com/kata-containers/kata-containers
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder
```
## Create a rootfs image ## Create a rootfs image
### Create a local rootfs ### Create a local rootfs
@ -254,24 +234,26 @@ As a prerequisite, you need to install Docker. Otherwise, you will not be
able to run the `rootfs.sh` script with `USE_DOCKER=true` as expected in able to run the `rootfs.sh` script with `USE_DOCKER=true` as expected in
the following example. the following example.
``` ```bash
$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs $ export distro="ubuntu" # example
$ sudo rm -rf ${ROOTFS_DIR} $ export ROOTFS_DIR="$(realpath kata-containers/tools/osbuilder/rootfs-builder/rootfs)"
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder $ sudo rm -rf "${ROOTFS_DIR}"
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true ./rootfs.sh ${distro}' $ pushd kata-containers/tools/osbuilder/rootfs-builder
$ script -fec 'sudo -E USE_DOCKER=true ./rootfs.sh "${distro}"'
$ popd
``` ```
You MUST choose a distribution (e.g., `ubuntu`) for `${distro}`. You MUST choose a distribution (e.g., `ubuntu`) for `${distro}`.
You can get a supported distributions list in the Kata Containers by running the following. You can get a supported distributions list in the Kata Containers by running the following.
``` ```bash
$ ./rootfs.sh -l $ ./kata-containers/tools/osbuilder/rootfs-builder/rootfs.sh -l
``` ```
If you want to build the agent without seccomp capability, you need to run the `rootfs.sh` script with `SECCOMP=no` as follows. If you want to build the agent without seccomp capability, you need to run the `rootfs.sh` script with `SECCOMP=no` as follows.
``` ```bash
$ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}' $ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh "${distro}"'
``` ```
> **Note:** > **Note:**
@ -287,17 +269,18 @@ $ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no
> >
> - You should only do this step if you are testing with the latest version of the agent. > - You should only do this step if you are testing with the latest version of the agent.
``` ```bash
$ sudo install -o root -g root -m 0550 -t ${ROOTFS_DIR}/usr/bin ../../../src/agent/target/x86_64-unknown-linux-musl/release/kata-agent $ sudo install -o root -g root -m 0550 -t "${ROOTFS_DIR}/usr/bin" "${ROOTFS_DIR}/../../../../src/agent/target/x86_64-unknown-linux-musl/release/kata-agent"
$ sudo install -o root -g root -m 0440 ../../../src/agent/kata-agent.service ${ROOTFS_DIR}/usr/lib/systemd/system/ $ sudo install -o root -g root -m 0440 "${ROOTFS_DIR}/../../../../src/agent/kata-agent.service" "${ROOTFS_DIR}/usr/lib/systemd/system/"
$ sudo install -o root -g root -m 0440 ../../../src/agent/kata-containers.target ${ROOTFS_DIR}/usr/lib/systemd/system/ $ sudo install -o root -g root -m 0440 "${ROOTFS_DIR}/../../../../src/agent/kata-containers.target" "${ROOTFS_DIR}/usr/lib/systemd/system/"
``` ```
### Build a rootfs image ### Build a rootfs image
``` ```bash
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/image-builder $ pushd kata-containers/tools/osbuilder/image-builder
$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}' $ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh "${ROOTFS_DIR}"'
$ popd
``` ```
> **Notes:** > **Notes:**
@ -313,21 +296,26 @@ $ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'
### Install the rootfs image ### Install the rootfs image
``` ```bash
$ commit=$(git log --format=%h -1 HEAD) $ pushd kata-containers/tools/osbuilder/image-builder
$ date=$(date +%Y-%m-%d-%T.%N%z) $ commit="$(git log --format=%h -1 HEAD)"
$ date="$(date +%Y-%m-%d-%T.%N%z)"
$ image="kata-containers-${date}-${commit}" $ image="kata-containers-${date}-${commit}"
$ sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}" $ sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}"
$ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img) $ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img)
$ popd
``` ```
## Create an initrd image - OPTIONAL ## Create an initrd image - OPTIONAL
### Create a local rootfs for initrd image ### Create a local rootfs for initrd image
```
$ export ROOTFS_DIR="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs" ```bash
$ sudo rm -rf ${ROOTFS_DIR} $ export distro="ubuntu" # example
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder $ export ROOTFS_DIR="$(realpath kata-containers/tools/osbuilder/rootfs-builder/rootfs)"
$ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh ${distro}' $ sudo rm -rf "${ROOTFS_DIR}"
$ pushd kata-containers/tools/osbuilder/rootfs-builder/
$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./rootfs.sh "${distro}"'
$ popd
``` ```
`AGENT_INIT` controls if the guest image uses the Kata agent as the guest `init` process. When you create an initrd image, `AGENT_INIT` controls if the guest image uses the Kata agent as the guest `init` process. When you create an initrd image,
always set `AGENT_INIT` to `yes`. always set `AGENT_INIT` to `yes`.
@ -335,14 +323,14 @@ always set `AGENT_INIT` to `yes`.
You MUST choose a distribution (e.g., `ubuntu`) for `${distro}`. You MUST choose a distribution (e.g., `ubuntu`) for `${distro}`.
You can get a supported distributions list in the Kata Containers by running the following. You can get a supported distributions list in the Kata Containers by running the following.
``` ```bash
$ ./rootfs.sh -l $ ./kata-containers/tools/osbuilder/rootfs-builder/rootfs.sh -l
``` ```
If you want to build the agent without seccomp capability, you need to run the `rootfs.sh` script with `SECCOMP=no` as follows. If you want to build the agent without seccomp capability, you need to run the `rootfs.sh` script with `SECCOMP=no` as follows.
``` ```bash
$ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh ${distro}' $ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true SECCOMP=no ./rootfs.sh "${distro}"'
``` ```
> **Note:** > **Note:**
@ -351,28 +339,31 @@ $ script -fec 'sudo -E GOPATH=$GOPATH AGENT_INIT=yes USE_DOCKER=true SECCOMP=no
Optionally, add your custom agent binary to the rootfs with the following commands. The default `$LIBC` used Optionally, add your custom agent binary to the rootfs with the following commands. The default `$LIBC` used
is `musl`, but on ppc64le and s390x, `gnu` should be used. Also, Rust refers to ppc64le as `powerpc64le`: is `musl`, but on ppc64le and s390x, `gnu` should be used. Also, Rust refers to ppc64le as `powerpc64le`:
``` ```bash
$ export ARCH=$(uname -m) $ export ARCH="$(uname -m)"
$ [ ${ARCH} == "ppc64le" ] || [ ${ARCH} == "s390x" ] && export LIBC=gnu || export LIBC=musl $ [ "${ARCH}" == "ppc64le" ] || [ "${ARCH}" == "s390x" ] && export LIBC=gnu || export LIBC=musl
$ [ ${ARCH} == "ppc64le" ] && export ARCH=powerpc64le $ [ "${ARCH}" == "ppc64le" ] && export ARCH=powerpc64le
$ sudo install -o root -g root -m 0550 -T ../../../src/agent/target/${ARCH}-unknown-linux-${LIBC}/release/kata-agent ${ROOTFS_DIR}/sbin/init $ sudo install -o root -g root -m 0550 -T "${ROOTFS_DIR}/../../../../src/agent/target/${ARCH}-unknown-linux-${LIBC}/release/kata-agent" "${ROOTFS_DIR}/sbin/init"
``` ```
### Build an initrd image ### Build an initrd image
``` ```bash
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/initrd-builder $ pushd kata-containers/tools/osbuilder/initrd-builder
$ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh ${ROOTFS_DIR}' $ script -fec 'sudo -E AGENT_INIT=yes USE_DOCKER=true ./initrd_builder.sh "${ROOTFS_DIR}"'
$ popd
``` ```
### Install the initrd image ### Install the initrd image
``` ```bash
$ commit=$(git log --format=%h -1 HEAD) $ pushd kata-containers/tools/osbuilder/initrd-builder
$ date=$(date +%Y-%m-%d-%T.%N%z) $ commit="$(git log --format=%h -1 HEAD)"
$ date="$(date +%Y-%m-%d-%T.%N%z)"
$ image="kata-containers-initrd-${date}-${commit}" $ image="kata-containers-initrd-${date}-${commit}"
$ sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}" $ sudo install -o root -g root -m 0640 -D kata-containers-initrd.img "/usr/share/kata-containers/${image}"
$ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img) $ (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers-initrd.img)
$ popd
``` ```
# Install guest kernel images # Install guest kernel images
@ -391,44 +382,44 @@ Kata Containers makes use of upstream QEMU branch. The exact version
and repository utilized can be found by looking at the [versions file](../versions.yaml). and repository utilized can be found by looking at the [versions file](../versions.yaml).
Find the correct version of QEMU from the versions file: Find the correct version of QEMU from the versions file:
``` ```bash
$ source ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/scripts/lib.sh $ source kata-containers/tools/packaging/scripts/lib.sh
$ qemu_version=$(get_from_kata_deps "assets.hypervisor.qemu.version") $ qemu_version="$(get_from_kata_deps "assets.hypervisor.qemu.version")"
$ echo ${qemu_version} $ echo "${qemu_version}"
``` ```
Get source from the matching branch of QEMU: Get source from the matching branch of QEMU:
``` ```bash
$ go get -d github.com/qemu/qemu $ git clone -b "${qemu_version}" https://github.com/qemu/qemu.git
$ cd ${GOPATH}/src/github.com/qemu/qemu $ your_qemu_directory="$(realpath qemu)"
$ git checkout ${qemu_version}
$ your_qemu_directory=${GOPATH}/src/github.com/qemu/qemu
``` ```
There are scripts to manage the build and packaging of QEMU. For the examples below, set your There are scripts to manage the build and packaging of QEMU. For the examples below, set your
environment as: environment as:
``` ```bash
$ go get -d github.com/kata-containers/kata-containers $ packaging_dir="$(realpath kata-containers/tools/packaging)"
$ packaging_dir="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging"
``` ```
Kata often utilizes patches for not-yet-upstream and/or backported fixes for components, Kata often utilizes patches for not-yet-upstream and/or backported fixes for components,
including QEMU. These can be found in the [packaging/QEMU directory](../tools/packaging/qemu/patches), including QEMU. These can be found in the [packaging/QEMU directory](../tools/packaging/qemu/patches),
and it's *recommended* that you apply them. For example, suppose that you are going to build QEMU and it's *recommended* that you apply them. For example, suppose that you are going to build QEMU
version 5.2.0, do: version 5.2.0, do:
``` ```bash
$ cd $your_qemu_directory $ "$packaging_dir/scripts/apply_patches.sh" "$packaging_dir/qemu/patches/5.2.x/"
$ $packaging_dir/scripts/apply_patches.sh $packaging_dir/qemu/patches/5.2.x/
``` ```
To build utilizing the same options as Kata, you should make use of the `configure-hypervisor.sh` script. For example: To build utilizing the same options as Kata, you should make use of the `configure-hypervisor.sh` script. For example:
``` ```bash
$ cd $your_qemu_directory $ pushd "$your_qemu_directory"
$ $packaging_dir/scripts/configure-hypervisor.sh kata-qemu > kata.cfg $ "$packaging_dir/scripts/configure-hypervisor.sh" kata-qemu > kata.cfg
$ eval ./configure "$(cat kata.cfg)" $ eval ./configure "$(cat kata.cfg)"
$ make -j $(nproc --ignore=1) $ make -j $(nproc --ignore=1)
# Optional
$ sudo -E make install $ sudo -E make install
$ popd
``` ```
If you do not want to install the respective QEMU version, the configuration file can be modified to point to the correct binary. In `/etc/kata-containers/configuration.toml`, change `path = "/path/to/qemu/build/qemu-system-x86_64"` to point to the correct QEMU binary.
See the [static-build script for QEMU](../tools/packaging/static-build/qemu/build-static-qemu.sh) for a reference on how to get, setup, configure and build QEMU for Kata. See the [static-build script for QEMU](../tools/packaging/static-build/qemu/build-static-qemu.sh) for a reference on how to get, setup, configure and build QEMU for Kata.
### Build a custom QEMU for aarch64/arm64 - REQUIRED ### Build a custom QEMU for aarch64/arm64 - REQUIRED
@ -439,11 +430,33 @@ See the [static-build script for QEMU](../tools/packaging/static-build/qemu/buil
> under upstream review for supporting NVDIMM on aarch64. > under upstream review for supporting NVDIMM on aarch64.
> >
You could build the custom `qemu-system-aarch64` as required with the following command: You could build the custom `qemu-system-aarch64` as required with the following command:
```bash
$ git clone https://github.com/kata-containers/tests.git
$ script -fec 'sudo -E tests/.ci/install_qemu.sh'
``` ```
$ go get -d github.com/kata-containers/tests
$ script -fec 'sudo -E ${GOPATH}/src/github.com/kata-containers/tests/.ci/install_qemu.sh' ## Build `virtiofsd`
When using the file system type virtio-fs (default), `virtiofsd` is required
```bash
$ pushd kata-containers/tools/packaging/static-build/virtiofsd
$ ./build-static-virtiofsd.sh
$ popd
``` ```
Modify `/etc/kata-containers/configuration.toml` and update value `virtio_fs_daemon = "/path/to/kata-containers/tools/packaging/static-build/virtiofsd/virtiofsd/virtiofsd"` to point to the binary.
# Check hardware requirements
You can check if your system is capable of creating a Kata Container by running the following:
```bash
$ sudo kata-runtime check
```
If your system is *not* able to run Kata Containers, the previous command will error out and explain why.
# Run Kata Containers with Containerd # Run Kata Containers with Containerd
Refer to the [How to use Kata Containers and Containerd](how-to/containerd-kata.md) how-to guide. Refer to the [How to use Kata Containers and Containerd](how-to/containerd-kata.md) how-to guide.
@ -474,7 +487,7 @@ See [Set up a debug console](#set-up-a-debug-console).
## Checking Docker default runtime ## Checking Docker default runtime
``` ```bash
$ sudo docker info 2>/dev/null | grep -i "default runtime" | cut -d: -f2- | grep -q runc && echo "SUCCESS" || echo "ERROR: Incorrect default Docker runtime" $ sudo docker info 2>/dev/null | grep -i "default runtime" | cut -d: -f2- | grep -q runc && echo "SUCCESS" || echo "ERROR: Incorrect default Docker runtime"
``` ```
## Set up a debug console ## Set up a debug console
@ -491,7 +504,7 @@ contain either `/bin/sh` or `/bin/bash`.
Enable debug_console_enabled in the `configuration.toml` configuration file: Enable debug_console_enabled in the `configuration.toml` configuration file:
``` ```toml
[agent.kata] [agent.kata]
debug_console_enabled = true debug_console_enabled = true
``` ```
@ -502,7 +515,7 @@ This will pass `agent.debug_console agent.debug_console_vport=1026` to agent as
For Kata Containers `2.0.x` releases, the `kata-runtime exec` command depends on the`kata-monitor` running, in order to get the sandbox's `vsock` address to connect to. Thus, first start the `kata-monitor` process. For Kata Containers `2.0.x` releases, the `kata-runtime exec` command depends on the`kata-monitor` running, in order to get the sandbox's `vsock` address to connect to. Thus, first start the `kata-monitor` process.
``` ```bash
$ sudo kata-monitor $ sudo kata-monitor
``` ```
@ -564,10 +577,10 @@ an additional `coreutils` package.
For example using CentOS: For example using CentOS:
``` ```bash
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder $ pushd kata-containers/tools/osbuilder/rootfs-builder
$ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs $ export ROOTFS_DIR="$(realpath ./rootfs)"
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos' $ script -fec 'sudo -E USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos'
``` ```
#### Build the debug image #### Build the debug image
@ -582,9 +595,10 @@ Install the image:
>**Note**: When using an initrd image, replace the below rootfs image name `kata-containers.img` >**Note**: When using an initrd image, replace the below rootfs image name `kata-containers.img`
>with the initrd image name `kata-containers-initrd.img`. >with the initrd image name `kata-containers-initrd.img`.
``` ```bash
$ name="kata-containers-centos-with-debug-console.img" $ name="kata-containers-centos-with-debug-console.img"
$ sudo install -o root -g root -m 0640 kata-containers.img "/usr/share/kata-containers/${name}" $ sudo install -o root -g root -m 0640 kata-containers.img "/usr/share/kata-containers/${name}"
$ popd
``` ```
Next, modify the `image=` values in the `[hypervisor.qemu]` section of the Next, modify the `image=` values in the `[hypervisor.qemu]` section of the
@ -593,7 +607,7 @@ to specify the full path to the image name specified in the previous code
section. Alternatively, recreate the symbolic link so it points to section. Alternatively, recreate the symbolic link so it points to
the new debug image: the new debug image:
``` ```bash
$ (cd /usr/share/kata-containers && sudo ln -sf "$name" kata-containers.img) $ (cd /usr/share/kata-containers && sudo ln -sf "$name" kata-containers.img)
``` ```
@ -604,7 +618,7 @@ to avoid all subsequently created containers from using the debug image.
Create a container as normal. For example using `crictl`: Create a container as normal. For example using `crictl`:
``` ```bash
$ sudo crictl run -r kata container.yaml pod.yaml $ sudo crictl run -r kata container.yaml pod.yaml
``` ```
@ -617,7 +631,7 @@ those for firecracker / cloud-hypervisor.
Add `agent.debug_console` to the guest kernel command line to allow the agent process to start a debug console. Add `agent.debug_console` to the guest kernel command line to allow the agent process to start a debug console.
``` ```bash
$ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.debug_console"/g' "${kata_configuration_file}" $ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.debug_console"/g' "${kata_configuration_file}"
``` ```
@ -638,7 +652,7 @@ between the host and the guest. The kernel command line option `agent.debug_cons
Add the parameter `agent.debug_console_vport=1026` to the kernel command line Add the parameter `agent.debug_console_vport=1026` to the kernel command line
as shown below: as shown below:
``` ```bash
sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.debug_console_vport=1026"/g' "${kata_configuration_file}" sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.debug_console_vport=1026"/g' "${kata_configuration_file}"
``` ```
@ -651,7 +665,7 @@ Next, connect to the debug console. The VSOCKS paths vary slightly between each
VMM solution. VMM solution.
In case of cloud-hypervisor, connect to the `vsock` as shown: In case of cloud-hypervisor, connect to the `vsock` as shown:
``` ```bash
$ sudo su -c 'cd /var/run/vc/vm/${sandbox_id}/root/ && socat stdin unix-connect:clh.sock' $ sudo su -c 'cd /var/run/vc/vm/${sandbox_id}/root/ && socat stdin unix-connect:clh.sock'
CONNECT 1026 CONNECT 1026
``` ```
@ -659,7 +673,7 @@ CONNECT 1026
**Note**: You need to type `CONNECT 1026` and press `RETURN` key after entering the `socat` command. **Note**: You need to type `CONNECT 1026` and press `RETURN` key after entering the `socat` command.
For firecracker, connect to the `hvsock` as shown: For firecracker, connect to the `hvsock` as shown:
``` ```bash
$ sudo su -c 'cd /var/run/vc/firecracker/${sandbox_id}/root/ && socat stdin unix-connect:kata.hvsock' $ sudo su -c 'cd /var/run/vc/firecracker/${sandbox_id}/root/ && socat stdin unix-connect:kata.hvsock'
CONNECT 1026 CONNECT 1026
``` ```
@ -668,7 +682,7 @@ CONNECT 1026
For QEMU, connect to the `vsock` as shown: For QEMU, connect to the `vsock` as shown:
``` ```bash
$ sudo su -c 'cd /var/run/vc/vm/${sandbox_id} && socat "stdin,raw,echo=0,escape=0x11" "unix-connect:console.sock"' $ sudo su -c 'cd /var/run/vc/vm/${sandbox_id} && socat "stdin,raw,echo=0,escape=0x11" "unix-connect:console.sock"'
``` ```
@ -681,7 +695,7 @@ If the image is created using
[osbuilder](../tools/osbuilder), the following YAML [osbuilder](../tools/osbuilder), the following YAML
file exists and contains details of the image and how it was created: file exists and contains details of the image and how it was created:
``` ```bash
$ cat /var/lib/osbuilder/osbuilder.yaml $ cat /var/lib/osbuilder/osbuilder.yaml
``` ```

3
docs/how-to/README.md
View File

@ -42,4 +42,5 @@
- [How to setup swap devices in guest kernel](how-to-setup-swap-devices-in-guest-kernel.md) - [How to setup swap devices in guest kernel](how-to-setup-swap-devices-in-guest-kernel.md)
- [How to run rootless vmm](how-to-run-rootless-vmm.md) - [How to run rootless vmm](how-to-run-rootless-vmm.md)
- [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md) - [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md)
- [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md) - [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md)
- [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md)

8
docs/how-to/containerd-kata.md
View File

@ -77,8 +77,8 @@ $ command -v containerd
You can manually install CNI plugins as follows: You can manually install CNI plugins as follows:
```bash ```bash
$ go get github.com/containernetworking/plugins $ git clone https://github.com/containernetworking/plugins.git
$ pushd $GOPATH/src/github.com/containernetworking/plugins $ pushd plugins
$ ./build_linux.sh $ ./build_linux.sh
$ sudo mkdir /opt/cni $ sudo mkdir /opt/cni
$ sudo cp -r bin /opt/cni/ $ sudo cp -r bin /opt/cni/
@ -93,8 +93,8 @@ $ popd
You can install the `cri-tools` from source code: You can install the `cri-tools` from source code:
```bash ```bash
$ go get github.com/kubernetes-sigs/cri-tools $ git clone https://github.com/kubernetes-sigs/cri-tools.git
$ pushd $GOPATH/src/github.com/kubernetes-sigs/cri-tools $ pushd cri-tools
$ make $ make
$ sudo -E make install $ sudo -E make install
$ popd $ popd

159
docs/how-to/how-to-run-kata-containers-with-SNP-VMs.md Normal file
View File

@ -0,0 +1,159 @@
# Kata Containers with AMD SEV-SNP VMs
## Disclaimer
This guide is designed for developers and is - same as the Developer Guide - not intended for production systems or end users. It is advisable to only follow this guide on non-critical development systems.
## Prerequisites
To run Kata Containers in SNP-VMs, the following software stack is used.
![Kubernetes integration with shimv2](./images/SNP-stack.svg)
The host BIOS and kernel must be capable of supporting AMD SEV-SNP and configured accordingly. For Kata Containers, the host kernel with branch [`sev-snp-iommu-avic_5.19-rc6_v3`](https://github.com/AMDESE/linux/tree/sev-snp-iommu-avic_5.19-rc6_v3) and commit [`3a88547`](https://github.com/AMDESE/linux/commit/3a885471cf89156ea555341f3b737ad2a8d9d3d0) is known to work in conjunction with SEV Firmware version 1.51.3 (0xh\_1.33.03) available on AMD's [SEV developer website](https://developer.amd.com/sev/). See [AMD's guide](https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel) to configure the host accordingly. Verify that you are able to run SEV-SNP encrypted VMs first. The guest components required for Kata Containers are built as described below.
**Tip**: It is easiest to first have Kata Containers running on your system and then modify it to run containers in SNP-VMs. Follow the [Developer guide](../Developer-Guide.md#warning) and then follow the below steps. Nonetheless, you can just follow this guide from the start.
## How to build
Follow all of the below steps to install Kata Containers with SNP-support from scratch. These steps mostly follow the developer guide with modifications to support SNP
__Steps from the Developer Guide:__
- Get all the [required components](../Developer-Guide.md#requirements-to-build-individual-components) for building the kata-runtime
- [Build the and install kata-runtime](../Developer-Guide.md#build-and-install-the-kata-containers-runtime)
- [Build a custom agent](../Developer-Guide.md#build-a-custom-kata-agent---optional)
- [Create an initrd image](../Developer-Guide.md#create-an-initrd-image---optional) by first building a rootfs, then building the initrd based on the rootfs, use a custom agent and install. `ubuntu` works as the distribution of choice.
- Get the [required components](../../tools/packaging/kernel/README.md#requirements) to build a custom kernel
__SNP-specific steps:__
- Build the SNP-specific kernel as shown below (see this [guide](../../tools/packaging/kernel/README.md#build-kata-containers-kernel) for more information)
```bash
$ pushd kata-containers/tools/packaging/kernel/
$ ./build-kernel.sh -a x86_64 -x snp setup
$ ./build-kernel.sh -a x86_64 -x snp build
$ sudo -E PATH="${PATH}" ./build-kernel.sh -x snp install
$ popd
```
- Build a current OVMF capable of SEV-SNP:
```bash
$ pushd kata-containers/tools/packaging/static-build/ovmf
$ ./build.sh
$ tar -xvf edk2-x86_64.tar.gz
$ popd
```
- Build a custom QEMU
```bash
$ source kata-containers/tools/packaging/scripts/lib.sh
$ qemu_url="$(get_from_kata_deps "assets.hypervisor.qemu.snp.url")"
$ qemu_branch="$(get_from_kata_deps "assets.hypervisor.qemu.snp.branch")"
$ qemu_commit="$(get_from_kata_deps "assets.hypervisor.qemu.snp.commit")"
$ git clone -b "${qemu_branch}" "${qemu_url}"
$ pushd qemu
$ git checkout "${qemu_commit}"
$ ./configure --target-list=x86_64-softmmu --enable-debug
$ make -j "$(nproc)"
$ popd
```
### Kata Containers Configuration for SNP
The configuration file located at `/etc/kata-containers/configuration.toml` must be adapted as follows to support SNP-VMs:
- Use the SNP-specific kernel for the guest VM (change path)
```toml
kernel = "/usr/share/kata-containers/vmlinuz-snp.container"
```
- Enable the use of an initrd (uncomment)
```toml
initrd = "/usr/share/kata-containers/kata-containers-initrd.img"
```
- Disable the use of a rootfs (comment out)
```toml
# image = "/usr/share/kata-containers/kata-containers.img"
```
- Use the custom QEMU capable of SNP (change path)
```toml
path = "/path/to/qemu/build/qemu-system-x86_64"
```
- Use `virtio-9p` device since `virtio-fs` is unsupported due to bugs / shortcomings in QEMU version [`snp-v3`](https://github.com/AMDESE/qemu/tree/snp-v3) for SEV and SEV-SNP (change value)
```toml
shared_fs = "virtio-9p"
```
- Disable `virtiofsd` since it is no longer required (comment out)
```toml
# virtio_fs_daemon = "/usr/libexec/virtiofsd"
```
- Disable NVDIMM (uncomment)
```toml
disable_image_nvdimm = true
```
- Disable shared memory (uncomment)
```toml
file_mem_backend = ""
```
- Enable confidential guests (uncomment)
```toml
confidential_guest = true
```
- Enable SNP-VMs (uncomment)
```toml
sev_snp_guest = true
```
- Configure an OVMF (add path)
```toml
firmware = "/path/to/kata-containers/tools/packaging/static-build/ovmf/opt/kata/share/ovmf/OVMF.fd"
```
## Test Kata Containers with Containerd
With Kata Containers configured to support SNP-VMs, we use containerd to test and deploy containers in these VMs.
### Install Containerd
If not already present, follow [this guide](./containerd-kata.md#install) to install containerd and its related components including `CNI` and the `cri-tools` (skip Kata Containers since we already installed it)
### Containerd Configuration
Follow [this guide](./containerd-kata.md#configuration) to configure containerd to use Kata Containers
## Run Kata Containers in SNP-VMs
Run the below commands to start a container. See [this guide](./containerd-kata.md#run) for more information
```bash
$ sudo ctr image pull docker.io/library/busybox:latest
$ sudo ctr run --cni --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh
```
### Check for active SNP:
Inside the running container, run the following commands to check if SNP is active. It should look something like this:
```
/ # dmesg | grep -i sev
[ 0.299242] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
[ 0.472286] SEV: Using SNP CPUID table, 31 entries present.
[ 0.514574] SEV: SNP guest platform device initialized.
[ 0.885425] sev-guest sev-guest: Initialized SEV guest driver (using vmpck_id 0)
```
### Obtain an SNP Attestation Report
To obtain an attestation report inside the container, the `/dev/sev-guest` must first be configured. As of now, the VM does not perform this step, however it can be performed inside the container, either in the terminal or in code.
Example for shell:
```
/ # SNP_MAJOR=$(cat /sys/devices/virtual/misc/sev-guest/dev | awk -F: '{print $1}')
/ # SNP_MINOR=$(cat /sys/devices/virtual/misc/sev-guest/dev | awk -F: '{print $2}')
/ # mknod -m 600 /dev/sev-guest c "${SNP_MAJOR}" "${SNP_MINOR}"
```
## Known Issues
- Support for cgroups v2 is still [work in progress](https://github.com/kata-containers/kata-containers/issues/927). If issues occur due to cgroups v2 becoming the default in newer systems, one possible solution is to downgrade cgroups to v1:
```bash
sudo sed -i 's/^\(GRUB_CMDLINE_LINUX=".*\)"/\1 systemd.unified_cgroup_hierarchy=0"/' /etc/default/grub
sudo update-grub
sudo reboot
```
- If both SEV and SEV-SNP are supported by the host, Kata Containers uses SEV-SNP by default. You can verify what features are enabled by checking `/sys/module/kvm_amd/parameters/sev` and `sev_snp`. This means that Kata Containers can not run both SEV-SNP-VMs and SEV-VMs at the same time. If SEV is to be used by Kata Containers instead, reload the `kvm_amd` kernel module without SNP-support, this will disable SNP-support for the entire platform.
```bash
sudo rmmod kvm_amd && sudo modprobe kvm_amd sev_snp=0
```

4
docs/how-to/images/SNP-stack.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 9.0 KiB

4
src/runtime/Makefile
View File

@ -56,6 +56,7 @@ BINLIBEXECLIST :=
BIN_PREFIX = $(PROJECT_TYPE) BIN_PREFIX = $(PROJECT_TYPE)
PROJECT_DIR = $(PROJECT_TAG) PROJECT_DIR = $(PROJECT_TAG)
IMAGENAME = $(PROJECT_TAG).img IMAGENAME = $(PROJECT_TAG).img
INITRDNAME = $(PROJECT_TAG)-initrd.img
TARGET = $(BIN_PREFIX)-runtime TARGET = $(BIN_PREFIX)-runtime
RUNTIME_OUTPUT = $(CURDIR)/$(TARGET) RUNTIME_OUTPUT = $(CURDIR)/$(TARGET)
@ -110,6 +111,7 @@ PKGLIBEXECDIR := $(LIBEXECDIR)/$(PROJECT_DIR)
KERNELDIR := $(PKGDATADIR) KERNELDIR := $(PKGDATADIR)
IMAGEPATH := $(PKGDATADIR)/$(IMAGENAME) IMAGEPATH := $(PKGDATADIR)/$(IMAGENAME)
INITRDPATH := $(PKGDATADIR)/$(INITRDNAME)
FIRMWAREPATH := FIRMWAREPATH :=
FIRMWAREVOLUMEPATH := FIRMWAREVOLUMEPATH :=
@ -401,6 +403,8 @@ USER_VARS += FCVALIDJAILERPATHS
USER_VARS += SYSCONFIG USER_VARS += SYSCONFIG
USER_VARS += IMAGENAME USER_VARS += IMAGENAME
USER_VARS += IMAGEPATH USER_VARS += IMAGEPATH
USER_VARS += INITRDNAME
USER_VARS += INITRDPATH
USER_VARS += MACHINETYPE USER_VARS += MACHINETYPE
USER_VARS += KERNELDIR USER_VARS += KERNELDIR
USER_VARS += KERNELTYPE USER_VARS += KERNELTYPE

7
src/runtime/config/configuration-qemu.toml.in
View File

@ -15,6 +15,7 @@
path = "@QEMUPATH@" path = "@QEMUPATH@"
kernel = "@KERNELPATH@" kernel = "@KERNELPATH@"
image = "@IMAGEPATH@" image = "@IMAGEPATH@"
# initrd = "@INITRDPATH@"
machine_type = "@MACHINETYPE@" machine_type = "@MACHINETYPE@"
# Enable confidential guest support. # Enable confidential guest support.
@ -33,6 +34,12 @@ machine_type = "@MACHINETYPE@"
# Default false # Default false
# confidential_guest = true # confidential_guest = true
# Choose AMD SEV-SNP confidential guests
# In case of using confidential guests on AMD hardware that supports both SEV
# and SEV-SNP, the following enables SEV-SNP guests. SEV guests are default.
# Default false
# sev_snp_guest = true
# Enable running QEMU VMM as a non-root user. # Enable running QEMU VMM as a non-root user.
# By default QEMU VMM run as root. When this is set to true, QEMU VMM process runs as # By default QEMU VMM run as root. When this is set to true, QEMU VMM process runs as
# a non-root random user. See documentation for the limitations of this mode. # a non-root random user. See documentation for the limitations of this mode.

7
src/runtime/pkg/govmm/qemu/qemu.go
View File

@ -231,6 +231,9 @@ const (
// SEVGuest represents an SEV guest object // SEVGuest represents an SEV guest object
SEVGuest ObjectType = "sev-guest" SEVGuest ObjectType = "sev-guest"
// SNPGuest represents an SNP guest object
SNPGuest ObjectType = "sev-snp-guest"
// SecExecGuest represents an s390x Secure Execution (Protected Virtualization in QEMU) object // SecExecGuest represents an s390x Secure Execution (Protected Virtualization in QEMU) object
SecExecGuest ObjectType = "s390-pv-guest" SecExecGuest ObjectType = "s390-pv-guest"
// PEFGuest represent ppc64le PEF(Protected Execution Facility) object. // PEFGuest represent ppc64le PEF(Protected Execution Facility) object.
@ -295,6 +298,8 @@ func (object Object) Valid() bool {
case TDXGuest: case TDXGuest:
return object.ID != "" && object.File != "" && object.DeviceID != "" return object.ID != "" && object.File != "" && object.DeviceID != ""
case SEVGuest: case SEVGuest:
fallthrough
case SNPGuest:
return object.ID != "" && object.File != "" && object.CBitPos != 0 && object.ReducedPhysBits != 0 return object.ID != "" && object.File != "" && object.CBitPos != 0 && object.ReducedPhysBits != 0
case SecExecGuest: case SecExecGuest:
return object.ID != "" return object.ID != ""
@ -349,6 +354,8 @@ func (object Object) QemuParams(config *Config) []string {
deviceParams = append(deviceParams, fmt.Sprintf("config-firmware-volume=%s", object.FirmwareVolume)) deviceParams = append(deviceParams, fmt.Sprintf("config-firmware-volume=%s", object.FirmwareVolume))
} }
case SEVGuest: case SEVGuest:
fallthrough
case SNPGuest:
objectParams = append(objectParams, string(object.Type)) objectParams = append(objectParams, string(object.Type))
objectParams = append(objectParams, fmt.Sprintf("id=%s", object.ID)) objectParams = append(objectParams, fmt.Sprintf("id=%s", object.ID))
objectParams = append(objectParams, fmt.Sprintf("cbitpos=%d", object.CBitPos)) objectParams = append(objectParams, fmt.Sprintf("cbitpos=%d", object.CBitPos))

1
src/runtime/pkg/katautils/config-settings.go.in
View File

@ -86,6 +86,7 @@ const defaultVhostUserStorePath string = "/var/run/kata-containers/vhost-user/"
const defaultRxRateLimiterMaxRate = uint64(0) const defaultRxRateLimiterMaxRate = uint64(0)
const defaultTxRateLimiterMaxRate = uint64(0) const defaultTxRateLimiterMaxRate = uint64(0)
const defaultConfidentialGuest = false const defaultConfidentialGuest = false
const defaultSevSnpGuest = false
const defaultGuestSwap = false const defaultGuestSwap = false
const defaultRootlessHypervisor = false const defaultRootlessHypervisor = false
const defaultDisableSeccomp = false const defaultDisableSeccomp = false

3
src/runtime/pkg/katautils/config.go
View File

@ -149,6 +149,7 @@ type hypervisor struct {
DisableVhostNet bool `toml:"disable_vhost_net"` DisableVhostNet bool `toml:"disable_vhost_net"`
GuestMemoryDumpPaging bool `toml:"guest_memory_dump_paging"` GuestMemoryDumpPaging bool `toml:"guest_memory_dump_paging"`
ConfidentialGuest bool `toml:"confidential_guest"` ConfidentialGuest bool `toml:"confidential_guest"`
SevSnpGuest bool `toml:"sev_snp_guest"`
GuestSwap bool `toml:"enable_guest_swap"` GuestSwap bool `toml:"enable_guest_swap"`
Rootless bool `toml:"rootless"` Rootless bool `toml:"rootless"`
DisableSeccomp bool `toml:"disable_seccomp"` DisableSeccomp bool `toml:"disable_seccomp"`
@ -827,6 +828,7 @@ func newQemuHypervisorConfig(h hypervisor) (vc.HypervisorConfig, error) {
GuestMemoryDumpPath: h.GuestMemoryDumpPath, GuestMemoryDumpPath: h.GuestMemoryDumpPath,
GuestMemoryDumpPaging: h.GuestMemoryDumpPaging, GuestMemoryDumpPaging: h.GuestMemoryDumpPaging,
ConfidentialGuest: h.ConfidentialGuest, ConfidentialGuest: h.ConfidentialGuest,
SevSnpGuest: h.SevSnpGuest,
GuestSwap: h.GuestSwap, GuestSwap: h.GuestSwap,
Rootless: h.Rootless, Rootless: h.Rootless,
LegacySerial: h.LegacySerial, LegacySerial: h.LegacySerial,
@ -1221,6 +1223,7 @@ func GetDefaultHypervisorConfig() vc.HypervisorConfig {
TxRateLimiterMaxRate: defaultTxRateLimiterMaxRate, TxRateLimiterMaxRate: defaultTxRateLimiterMaxRate,
SGXEPCSize: defaultSGXEPCSize, SGXEPCSize: defaultSGXEPCSize,
ConfidentialGuest: defaultConfidentialGuest, ConfidentialGuest: defaultConfidentialGuest,
SevSnpGuest: defaultSevSnpGuest,
GuestSwap: defaultGuestSwap, GuestSwap: defaultGuestSwap,
Rootless: defaultRootlessHypervisor, Rootless: defaultRootlessHypervisor,
DisableSeccomp: defaultDisableSeccomp, DisableSeccomp: defaultDisableSeccomp,

2
src/runtime/virtcontainers/clh.go
View File

@ -422,6 +422,8 @@ func (clh *cloudHypervisor) enableProtection() error {
case sevProtection: case sevProtection:
return errors.New("SEV protection is not supported by Cloud Hypervisor") return errors.New("SEV protection is not supported by Cloud Hypervisor")
case snpProtection:
return errors.New("SEV-SNP protection is not supported by Cloud Hypervisor")
default: default:
return errors.New("This system doesn't support Confidentian Computing (Guest Protection)") return errors.New("This system doesn't support Confidentian Computing (Guest Protection)")

4
src/runtime/virtcontainers/documentation/api/1.0/api.md
View File

@ -348,6 +348,10 @@ type HypervisorConfig struct {
// Enable or disable different hardware features, ranging // Enable or disable different hardware features, ranging
// from memory encryption to both memory and CPU-state encryption and integrity. // from memory encryption to both memory and CPU-state encryption and integrity.
ConfidentialGuest bool ConfidentialGuest bool
// Enables SEV-SNP guests in case both AMD SEV and SNP are supported.
// SEV is default.
SevSnpGuest bool
} }
``` ```

9
src/runtime/virtcontainers/hypervisor.go
View File

@ -530,6 +530,9 @@ type HypervisorConfig struct {
// from memory encryption to both memory and CPU-state encryption and integrity. // from memory encryption to both memory and CPU-state encryption and integrity.
ConfidentialGuest bool ConfidentialGuest bool
// Enable SEV-SNP guests on AMD machines capable of both
SevSnpGuest bool
// BootToBeTemplate used to indicate if the VM is created to be a template VM // BootToBeTemplate used to indicate if the VM is created to be a template VM
BootToBeTemplate bool BootToBeTemplate bool
@ -873,6 +876,11 @@ const (
// Exclude from lint checking for it won't be used on arm64 code // Exclude from lint checking for it won't be used on arm64 code
sevProtection sevProtection
// AMD Secure Encrypted Virtualization - Secure Nested Paging (SEV-SNP)
// https://developer.amd.com/sev/
// Exclude from lint checking for it won't be used on arm64 code
snpProtection
// IBM POWER 9 Protected Execution Facility // IBM POWER 9 Protected Execution Facility
// https://www.kernel.org/doc/html/latest/powerpc/ultravisor.html // https://www.kernel.org/doc/html/latest/powerpc/ultravisor.html
// Exclude from lint checking for it won't be used on arm64 code // Exclude from lint checking for it won't be used on arm64 code
@ -889,6 +897,7 @@ var guestProtectionStr = [...]string{
pefProtection: "pef", pefProtection: "pef",
seProtection: "se", seProtection: "se",
sevProtection: "sev", sevProtection: "sev",
snpProtection: "snp",
tdxProtection: "tdx", tdxProtection: "tdx",
} }

9
src/runtime/virtcontainers/hypervisor_linux_amd64.go
View File

@ -13,6 +13,8 @@ const (
tdxCPUFlag = "tdx" tdxCPUFlag = "tdx"
sevKvmParameterPath = "/sys/module/kvm_amd/parameters/sev" sevKvmParameterPath = "/sys/module/kvm_amd/parameters/sev"
snpKvmParameterPath = "/sys/module/kvm_amd/parameters/sev_snp"
) )
// Implementation of this function is architecture specific // Implementation of this function is architecture specific
@ -26,6 +28,13 @@ func availableGuestProtection() (guestProtection, error) {
if d, err := os.Stat(tdxSysFirmwareDir); (err == nil && d.IsDir()) || flags[tdxCPUFlag] { if d, err := os.Stat(tdxSysFirmwareDir); (err == nil && d.IsDir()) || flags[tdxCPUFlag] {
return tdxProtection, nil return tdxProtection, nil
} }
// SEV-SNP is supported and enabled when the kvm module `sev_snp` parameter is set to `Y`
// SEV-SNP support infers SEV (-ES) support
if _, err := os.Stat(snpKvmParameterPath); err == nil {
if c, err := os.ReadFile(snpKvmParameterPath); err == nil && len(c) > 0 && (c[0] == 'Y') {
return snpProtection, nil
}
}
// SEV is supported and enabled when the kvm module `sev` parameter is set to `1` (or `Y` for linux >= 5.12) // SEV is supported and enabled when the kvm module `sev` parameter is set to `1` (or `Y` for linux >= 5.12)
if _, err := os.Stat(sevKvmParameterPath); err == nil { if _, err := os.Stat(sevKvmParameterPath); err == nil {
if c, err := os.ReadFile(sevKvmParameterPath); err == nil && len(c) > 0 && (c[0] == '1' || c[0] == 'Y') { if c, err := os.ReadFile(sevKvmParameterPath); err == nil && len(c) > 0 && (c[0] == '1' || c[0] == 'Y') {

40
src/runtime/virtcontainers/qemu_amd64.go
View File

@ -24,6 +24,8 @@ type qemuAmd64 struct {
// inherit from qemuArchBase, overwrite methods if needed // inherit from qemuArchBase, overwrite methods if needed
qemuArchBase qemuArchBase
snpGuest bool
vmFactory bool vmFactory bool
devLoadersCount uint32 devLoadersCount uint32
@ -122,6 +124,7 @@ func newQemuArch(config HypervisorConfig) (qemuArch, error) {
legacySerial: config.LegacySerial, legacySerial: config.LegacySerial,
}, },
vmFactory: factory, vmFactory: factory,
snpGuest: config.SevSnpGuest,
} }
if config.ConfidentialGuest { if config.ConfidentialGuest {
@ -169,6 +172,21 @@ func (q *qemuAmd64) bridges(number uint32) {
q.Bridges = genericBridges(number, q.qemuMachine.Type) q.Bridges = genericBridges(number, q.qemuMachine.Type)
} }
func (q *qemuAmd64) cpuModel() string {
var err error
cpuModel := defaultCPUModel
// Temporary until QEMU cpu model 'host' supports AMD SEV-SNP
protection, err := availableGuestProtection()
if err == nil {
if protection == snpProtection && q.snpGuest {
cpuModel = "EPYC-v4"
}
}
return cpuModel
}
func (q *qemuAmd64) memoryTopology(memoryMb, hostMemoryMb uint64, slots uint8) govmmQemu.Memory { func (q *qemuAmd64) memoryTopology(memoryMb, hostMemoryMb uint64, slots uint8) govmmQemu.Memory {
return genericMemoryTopology(memoryMb, hostMemoryMb, slots, q.memoryOffset) return genericMemoryTopology(memoryMb, hostMemoryMb, slots, q.memoryOffset)
} }
@ -197,6 +215,11 @@ func (q *qemuAmd64) enableProtection() error {
if err != nil { if err != nil {
return err return err
} }
// Configure SNP only if specified in config
if q.protection == snpProtection && !q.snpGuest {
q.protection = sevProtection
}
logger := hvLogger.WithFields(logrus.Fields{ logger := hvLogger.WithFields(logrus.Fields{
"subsystem": "qemuAmd64", "subsystem": "qemuAmd64",
"machine": q.qemuMachine, "machine": q.qemuMachine,
@ -219,6 +242,13 @@ func (q *qemuAmd64) enableProtection() error {
q.qemuMachine.Options += "confidential-guest-support=sev" q.qemuMachine.Options += "confidential-guest-support=sev"
logger.Info("Enabling SEV guest protection") logger.Info("Enabling SEV guest protection")
return nil return nil
case snpProtection:
if q.qemuMachine.Options != "" {
q.qemuMachine.Options += ","
}
q.qemuMachine.Options += "confidential-guest-support=snp"
logger.Info("Enabling SNP guest protection")
return nil
// TODO: Add support for other x86_64 technologies // TODO: Add support for other x86_64 technologies
@ -263,6 +293,16 @@ func (q *qemuAmd64) appendProtectionDevice(devices []govmmQemu.Device, firmware,
CBitPos: cpuid.AMDMemEncrypt.CBitPosition, CBitPos: cpuid.AMDMemEncrypt.CBitPosition,
ReducedPhysBits: cpuid.AMDMemEncrypt.PhysAddrReduction, ReducedPhysBits: cpuid.AMDMemEncrypt.PhysAddrReduction,
}), "", nil }), "", nil
case snpProtection:
return append(devices,
govmmQemu.Object{
Type: govmmQemu.SNPGuest,
ID: "snp",
Debug: false,
File: firmware,
CBitPos: cpuid.AMDMemEncrypt.CBitPosition,
ReducedPhysBits: 1,
}), "", nil
case noneProtection: case noneProtection:
return devices, firmware, nil return devices, firmware, nil

20
src/runtime/virtcontainers/qemu_amd64_test.go
View File

@ -293,6 +293,26 @@ func TestQemuAmd64AppendProtectionDevice(t *testing.T) {
assert.Equal(expectedOut, devices) assert.Equal(expectedOut, devices)
// snp protection
amd64.(*qemuAmd64).protection = snpProtection
devices, bios, err = amd64.appendProtectionDevice(devices, firmware, "")
assert.NoError(err)
assert.Empty(bios)
expectedOut = append(expectedOut,
govmmQemu.Object{
Type: govmmQemu.SNPGuest,
ID: "snp",
Debug: false,
File: firmware,
CBitPos: cpuid.AMDMemEncrypt.CBitPosition,
ReducedPhysBits: 1,
},
)
assert.Equal(expectedOut, devices)
// tdxProtection // tdxProtection
amd64.(*qemuAmd64).protection = tdxProtection amd64.(*qemuAmd64).protection = tdxProtection

7
src/runtime/virtcontainers/qemu_arm64_test.go
View File

@ -209,6 +209,13 @@ func TestQemuArm64AppendProtectionDevice(t *testing.T) {
assert.Empty(bios) assert.Empty(bios)
assert.NoError(err) assert.NoError(err)
// SNP protection
arm64.(*qemuArm64).protection = snpProtection
devices, bios, err = arm64.appendProtectionDevice(devices, firmware, "")
assert.Empty(devices)
assert.Empty(bios)
assert.NoError(err)
// TDX protection // TDX protection
arm64.(*qemuArm64).protection = tdxProtection arm64.(*qemuArm64).protection = tdxProtection
devices, bios, err = arm64.appendProtectionDevice(devices, firmware, "") devices, bios, err = arm64.appendProtectionDevice(devices, firmware, "")

6
src/runtime/virtcontainers/qemu_ppc64le_test.go
View File

@ -79,6 +79,12 @@ func TestQemuPPC64leAppendProtectionDevice(t *testing.T) {
assert.Error(err) assert.Error(err)
assert.Empty(bios) assert.Empty(bios)
//SNP protection
ppc64le.(*qemuPPC64le).protection = snpProtection
devices, bios, err = ppc64le.appendProtectionDevice(devices, firmware, "")
assert.Error(err)
assert.Empty(bios)
//TDX protection //TDX protection
ppc64le.(*qemuPPC64le).protection = tdxProtection ppc64le.(*qemuPPC64le).protection = tdxProtection
devices, bios, err = ppc64le.appendProtectionDevice(devices, firmware, "") devices, bios, err = ppc64le.appendProtectionDevice(devices, firmware, "")

6
src/runtime/virtcontainers/qemu_s390x_test.go
View File

@ -136,6 +136,12 @@ func TestQemuS390xAppendProtectionDevice(t *testing.T) {
assert.Error(err) assert.Error(err)
assert.Empty(bios) assert.Empty(bios)
// SNP protection
s390x.(*qemuS390x).protection = snpProtection
devices, bios, err = s390x.appendProtectionDevice(devices, firmware, "")
assert.Error(err)
assert.Empty(bios)
// Secure Execution protection // Secure Execution protection
s390x.(*qemuS390x).protection = seProtection s390x.(*qemuS390x).protection = seProtection

9
tools/packaging/kernel/README.md
View File

@ -9,7 +9,8 @@ automates the process to build a kernel for Kata Containers.
The `build-kernel.sh` script requires an installed Golang version matching the The `build-kernel.sh` script requires an installed Golang version matching the
[component build requirements](../../../docs/Developer-Guide.md#requirements-to-build-individual-components). [component build requirements](../../../docs/Developer-Guide.md#requirements-to-build-individual-components).
It also requires [yq](https://github.com/mikefarah/yq) version 3.4.1 It also requires [yq](https://github.com/mikefarah/yq) version 3.4.1
> **Hint**: `snap install yq --channel=v3/stable` > **Hint**: `snap install yq --channel=v3/stable` \
> **or** `go install github.com/mikefarah/yq/v3@latest`
The Linux kernel scripts further require a few packages (flex, bison, and libelf-dev) The Linux kernel scripts further require a few packages (flex, bison, and libelf-dev)
@ -53,7 +54,7 @@ Options:
``` ```
Example: Example:
``` ```bash
$ ./build-kernel.sh -v 5.10.25 -g nvidia -f -d setup $ ./build-kernel.sh -v 5.10.25 -g nvidia -f -d setup
``` ```
> **Note** > **Note**
@ -68,8 +69,8 @@ $ ./build-kernel.sh -v 5.10.25 -g nvidia -f -d setup
## Setup kernel source code ## Setup kernel source code
```bash ```bash
$ go get -d -u github.com/kata-containers/kata-containers $ git clone github.com/kata-containers/kata-containers
$ cd $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel $ cd kata-containers/tools/packaging/kernel
$ ./build-kernel.sh setup $ ./build-kernel.sh setup
``` ```

4
tools/packaging/kernel/build-kernel.sh
View File

@ -101,7 +101,7 @@ Options:
-t <hypervisor> : Hypervisor_target. -t <hypervisor> : Hypervisor_target.
-u <url> : Kernel URL to be used to download the kernel tarball. -u <url> : Kernel URL to be used to download the kernel tarball.
-v <version> : Kernel version to use if kernel path not provided. -v <version> : Kernel version to use if kernel path not provided.
-x <type> : Confidential guest protection type, such as sev and tdx -x <type> : Confidential guest protection type, such as sev, snp and tdx
EOF EOF
exit "$exit_code" exit "$exit_code"
} }
@ -525,7 +525,7 @@ main() {
x) x)
conf_guest="${OPTARG}" conf_guest="${OPTARG}"
case "$conf_guest" in case "$conf_guest" in
sev|tdx) ;; sev|snp|tdx) ;;
*) die "Confidential guest type '$conf_guest' not supported" ;; *) die "Confidential guest type '$conf_guest' not supported" ;;
esac esac
;; ;;

10
tools/packaging/kernel/configs/fragments/x86_64/snp/snp.conf Normal file
View File

@ -0,0 +1,10 @@
# !s390x !ppc64le !arm64
# enable sev-snp support
CONFIG_AMD_MEM_ENCRYPT=y
CONFIG_SEV_GUEST=y
CONFIG_VIRT_DRIVERS=y
# Prepare kernel for direct boot using OVMF
CONFIG_EFI=y
CONFIG_EFI_STUB=y

9
versions.yaml
View File

@ -102,6 +102,11 @@ assets:
description: "VMM that uses KVM and supports TDX" description: "VMM that uses KVM and supports TDX"
url: "https://github.com/intel/qemu-dcp" url: "https://github.com/intel/qemu-dcp"
tag: "SPR-BKC-QEMU-v2.5" tag: "SPR-BKC-QEMU-v2.5"
snp:
description: "VMM that uses KVM and supports AMD SEV-SNP"
url: "https://github.com/AMDESE/qemu"
branch: "snp-v3"
commit: "ffa95097ee"
qemu-experimental: qemu-experimental:
description: "QEMU with virtiofs support" description: "QEMU with virtiofs support"
@ -162,6 +167,10 @@ assets:
description: "Linux kernel that supports SEV" description: "Linux kernel that supports SEV"
url: "https://cdn.kernel.org/pub/linux/kernel/v5.x/" url: "https://cdn.kernel.org/pub/linux/kernel/v5.x/"
version: "v5.19.2" version: "v5.19.2"
snp:
description: "Linux kernel that supports AMD SEV-SNP for VMs"
url: "https://cdn.kernel.org/pub/linux/kernel/v5.x/"
version: "v5.19.2"
kernel-experimental: kernel-experimental:
description: "Linux kernel with virtio-fs support" description: "Linux kernel with virtio-fs support"