mirror of
https://github.com/kata-containers/kata-containers.git
synced 2025-07-31 07:19:06 +00:00
Merge pull request #1363 from eadamsintel/qat-docs
docs: Update QAT instructions to work with Kata 2.0 repos
This commit is contained in:
Chelsea Mafrica
GitHub
commit
7da3bdc68e
@ -1,56 +1,62 @@
|
||||
# Table of Contents
|
||||
|
||||
* [Table of Contents](#table-of-contents)
|
||||
* [Introduction](#introduction)
|
||||
* [Helpful Links before starting](#helpful-links-before-starting)
|
||||
* [Steps to enable Intel QAT in Kata Containers](#steps-to-enable-intel-qat-in-kata-containers)
|
||||
* [Script variables](#script-variables)
|
||||
* [Set environment variables (Every Reboot)](#set-environment-variables-every-reboot)
|
||||
* [Prepare the Clear Linux Host](#prepare-the-clear-linux-host)
|
||||
* [Identify which PCI Bus the Intel QAT card is on](#identify-which-pci-bus-the-intel-qat-card-is-on)
|
||||
* [Install necessary bundles for Clear Linux](#install-necessary-bundles-for-clear-linux)
|
||||
* [Download Intel QAT drivers](#download-intel-qat-drivers)
|
||||
* [Copy Intel QAT configuration files and enable Virtual Functions](#copy-intel-qat-configuration-files-and-enable-virtual-functions)
|
||||
* [Expose and Bind Intel QAT virtual functions to VFIO-PCI (Every reboot)](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot)
|
||||
* [Check Intel QAT virtual functions are enabled](#check-intel-qat-virtual-functions-are-enabled)
|
||||
* [Prepare Kata Containers](#prepare-kata-containers)
|
||||
* [Download Kata kernel Source](#download-kata-kernel-source)
|
||||
* [Build Kata kernel](#build-kata-kernel)
|
||||
* [Copy Kata kernel](#copy-kata-kernel)
|
||||
* [Prepare Kata root filesystem](#prepare-kata-root-filesystem)
|
||||
* [Compile Intel QAT drivers for Kata Containers kernel and add to Kata Containers rootfs](#compile-intel-qat-drivers-for-kata-containers-kernel-and-add-to-kata-containers-rootfs)
|
||||
* [Copy Kata rootfs](#copy-kata-rootfs)
|
||||
* [Update Kata configuration to point to custom kernel and rootfs](#update-kata-configuration-to-point-to-custom-kernel-and-rootfs)
|
||||
* [Verify Intel QAT works in a Docker Kata Containers container](#verify-intel-qat-works-in-a-docker-kata-containers-container)
|
||||
* [Build OpenSSL Intel QAT engine container](#build-openssl-intel-qat-engine-container)
|
||||
* [Test Intel QAT in Docker](#test-intel-qat-in-docker)
|
||||
* [Troubleshooting](#troubleshooting)
|
||||
* [Optional Scripts](#optional-scripts)
|
||||
* [Verify Intel QAT card counters are incremented](#verify-intel-qat-card-counters-are-incremented)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Introduction](#introduction)
|
||||
- [Helpful Links before starting](#helpful-links-before-starting)
|
||||
- [Steps to enable Intel® QAT in Kata Containers](#steps-to-enable-intel-qat-in-kata-containers)
|
||||
- [Script variables](#script-variables)
|
||||
- [Set environment variables (Every Reboot)](#set-environment-variables-every-reboot)
|
||||
- [Prepare the Ubuntu Host](#prepare-the-ubuntu-host)
|
||||
- [Identify which PCI Bus the Intel® QAT card is on](#identify-which-pci-bus-the-intel-qat-card-is-on)
|
||||
- [Install necessary packages for Ubuntu](#install-necessary-packages-for-ubuntu)
|
||||
- [Download Intel® QAT drivers](#download-intel-qat-drivers)
|
||||
- [Copy Intel® QAT configuration files and enable virtual functions](#copy-intel-qat-configuration-files-and-enable-virtual-functions)
|
||||
- [Expose and Bind Intel® QAT virtual functions to VFIO-PCI (Every reboot)](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot)
|
||||
- [Check Intel® QAT virtual functions are enabled](#check-intel-qat-virtual-functions-are-enabled)
|
||||
- [Prepare Kata Containers](#prepare-kata-containers)
|
||||
- [Download Kata kernel Source](#download-kata-kernel-source)
|
||||
- [Build Kata kernel](#build-kata-kernel)
|
||||
- [Copy Kata kernel](#copy-kata-kernel)
|
||||
- [Prepare Kata root filesystem](#prepare-kata-root-filesystem)
|
||||
- [Compile Intel® QAT drivers for Kata Containers kernel and add to Kata Containers rootfs](#compile-intel-qat-drivers-for-kata-containers-kernel-and-add-to-kata-containers-rootfs)
|
||||
- [Copy Kata rootfs](#copy-kata-rootfs)
|
||||
- [Verify Intel® QAT works in a container](#verify-intel-qat-works-in-a-container)
|
||||
- [Build OpenSSL Intel® QAT engine container](#build-openssl-intel-qat-engine-container)
|
||||
- [Test Intel® QAT with the ctr tool](#test-intel-qat-with-the-ctr-tool)
|
||||
- [Test Intel® QAT in Kubernetes](#test-intel-qat-in-kubernetes)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Optional Scripts](#optional-scripts)
|
||||
- [Verify Intel® QAT card counters are incremented](#verify-intel-qat-card-counters-are-incremented)
|
||||
|
||||
# Introduction
|
||||
|
||||
Intel QuickAssist Technology (Intel QAT) provides hardware acceleration
|
||||
Intel® QuickAssist Technology (QAT) provides hardware acceleration
|
||||
for security (cryptography) and compression. These instructions cover the
|
||||
steps for [Clear Linux](https://clearlinux.org) but can be adapted to any
|
||||
Linux distribution. Your distribution may already have the Intel QAT
|
||||
drivers, but it is likely they do not contain the necessary user space
|
||||
components. These instructions guide the user on how to download the kernel
|
||||
sources, compile kernel driver modules against those sources, and load them
|
||||
onto the host as well as preparing a specially built Kata Containers kernel
|
||||
and custom Kata Containers rootfs.
|
||||
steps for the latest [Ubuntu LTS release](https://ubuntu.com/download/desktop)
|
||||
which already include the QAT host driver. These instructions can be adapted to
|
||||
any Linux distribution. These instructions guide the user on how to download
|
||||
the kernel sources, compile kernel driver modules against those sources, and
|
||||
load them onto the host as well as preparing a specially built Kata Containers
|
||||
kernel and custom Kata Containers rootfs.
|
||||
|
||||
* Download kernel sources
|
||||
* Compile Kata kernel
|
||||
* Compile kernel driver modules against those sources
|
||||
* Download rootfs
|
||||
* Add driver modules to rootfs
|
||||
* Build rootfs image
|
||||
|
||||
## Helpful Links before starting
|
||||
|
||||
[Intel QAT Engine](https://github.com/intel/QAT_Engine)
|
||||
[Intel® QuickAssist Technology at `01.org`](https://01.org/intel-quickassist-technology)
|
||||
|
||||
[Intel QuickAssist Technology at `01.org`](https://01.org/intel-quickassist-technology)
|
||||
[Intel® QuickAssist Technology Engine for OpenSSL](https://github.com/intel/QAT_Engine)
|
||||
|
||||
[Intel Device Plugin for Kubernetes](https://github.com/intel/intel-device-plugins-for-kubernetes)
|
||||
|
||||
[Intel QuickAssist Crypto Poll Mode Driver](https://dpdk-docs.readthedocs.io/en/latest/cryptodevs/qat.html)
|
||||
[Intel® QuickAssist Technology for Crypto Poll Mode Driver](https://dpdk-docs.readthedocs.io/en/latest/cryptodevs/qat.html)
|
||||
|
||||
## Steps to enable Intel QAT in Kata Containers
|
||||
## Steps to enable Intel® QAT in Kata Containers
|
||||
|
||||
There are some steps to complete only once, some steps to complete with every
|
||||
reboot, and some steps to complete when the host kernel changes.
|
||||
@ -67,91 +73,95 @@ needed to point to updated drivers or different install locations.
|
||||
Make sure to check [`01.org`](https://01.org/intel-quickassist-technology) for
|
||||
the latest driver.
|
||||
|
||||
```sh
|
||||
$ export QAT_DRIVER_VER=qat1.7.l.4.8.0-00005.tar.gz
|
||||
$ export QAT_DRIVER_URL=https://01.org/sites/default/files/downloads/${QAT_DRIVER_VER}
|
||||
```bash
|
||||
$ export QAT_DRIVER_VER=qat1.7.l.4.12.0-00011.tar.gz
|
||||
$ export QAT_DRIVER_URL=https://downloadmirror.intel.com/30178/eng/${QAT_DRIVER_VER}
|
||||
$ export QAT_CONF_LOCATION=~/QAT_conf
|
||||
$ export QAT_DOCKERFILE=https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/master/demo/openssl-qat-engine/Dockerfile
|
||||
$ export QAT_SRC=~/src/QAT
|
||||
$ export GOPATH=~/src/go
|
||||
$ export OSBUILDER=~/src/osbuilder
|
||||
$ export KATA_KERNEL_LOCATION=~/kata
|
||||
$ export KATA_ROOTFS_LOCATION=~/kata
|
||||
```
|
||||
|
||||
## Prepare the Clear Linux Host
|
||||
## Prepare the Ubuntu Host
|
||||
|
||||
The host could be a bare metal instance or a virtual machine. If using a
|
||||
virtual machine, make sure that KVM nesting is enabled. The following
|
||||
instructions reference an Intel QAT. Some of the instructions must be
|
||||
modified if using a different Intel QAT device. You can identify the Intel QAT
|
||||
chipset by executing the following.
|
||||
instructions reference an Intel® C62X chipset. Some of the instructions must be
|
||||
modified if using a different Intel® QAT device. The Intel® QAT chipset can be
|
||||
identified by executing the following.
|
||||
|
||||
### Identify which PCI Bus the Intel QAT card is on
|
||||
### Identify which PCI Bus the Intel® QAT card is on
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done
|
||||
```
|
||||
|
||||
### Install necessary bundles for Clear Linux
|
||||
### Install necessary packages for Ubuntu
|
||||
|
||||
Clear Linux version 30780 (Released August 13, 2019) includes a
|
||||
`linux-firmware-qat` bundle that has the necessary QAT firmware along with a
|
||||
functional QAT host driver that works with Kata Containers.
|
||||
These packages are necessary to compile the Kata kernel, Intel® QAT driver, and to
|
||||
prepare the rootfs for Kata. [Docker](https://docs.docker.com/engine/install/ubuntu/)
|
||||
also needs to be installed to be able to build the rootfs. To test that
|
||||
everything works a Kubernetes pod is started requesting Intel® QAT resources. For the
|
||||
pass through of the virtual functions the kernel boot parameter needs to have
|
||||
`INTEL_IOMMU=on`.
|
||||
|
||||
```sh
|
||||
$ sudo swupd bundle-add network-basic linux-firmware-qat make c-basic go-basic containers-virt dev-utils devpkg-elfutils devpkg-systemd devpkg-ssl
|
||||
$ sudo clr-boot-manager update
|
||||
$ sudo systemctl enable --now docker
|
||||
```bash
|
||||
$ sudo apt update
|
||||
$ sudo apt install -y golang-go build-essential python pkg-config zlib1g-dev libudev-dev bison libelf-dev flex libtool automake autotools-dev autoconf bc libpixman-1-dev coreutils libssl-dev
|
||||
$ sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=""/GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"/' /etc/default/grub
|
||||
$ sudo update-grub
|
||||
$ sudo reboot
|
||||
```
|
||||
|
||||
### Download Intel QAT drivers
|
||||
### Download Intel® QAT drivers
|
||||
|
||||
This will download the Intel QAT drivers from [`01.org`](https://01.org/intel-quickassist-technology).
|
||||
This will download the [Intel® QAT drivers](https://01.org/intel-quickassist-technology).
|
||||
Make sure to check the website for the latest version.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $QAT_SRC
|
||||
$ cd $QAT_SRC
|
||||
$ curl -L $QAT_DRIVER_URL | tar zx
|
||||
```
|
||||
|
||||
### Copy Intel QAT configuration files and enable Virtual Functions
|
||||
### Copy Intel® QAT configuration files and enable virtual functions
|
||||
|
||||
Modify the instructions below as necessary if using a different QAT hardware
|
||||
Modify the instructions below as necessary if using a different Intel® QAT hardware
|
||||
platform. You can learn more about customizing configuration files at the
|
||||
[Intel QAT Engine repository](https://github.com/intel/QAT_Engine/#copy-the-correct-intel-quickassist-technology-driver-config-files)
|
||||
[Intel® QAT Engine repository](https://github.com/intel/QAT_Engine/#copy-the-correct-intel-quickassist-technology-driver-config-files)
|
||||
This section starts from a base config file and changes the `SSL` section to
|
||||
`SHIM` to support the OpenSSL engine. There are more tweaks that you can make
|
||||
depending on the use case and how many Intel QAT engines should be run. You
|
||||
depending on the use case and how many Intel® QAT engines should be run. You
|
||||
can find more information about how to customize in the
|
||||
[Intel® QuickAssist Technology Software for Linux* - Programmer's Guide.](https://01.org/sites/default/files/downloads/336210qatswprogrammersguiderev006.pdf)
|
||||
|
||||
> **Note: This section assumes that a QAT `c6xx` platform is used.**
|
||||
> **Note: This section assumes that a Intel® QAT `c6xx` platform is used.**
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $QAT_CONF_LOCATION
|
||||
$ cp $QAT_SRC/quickassist/utilities/adf_ctl/conf_files/c6xxvf_dev0.conf.vm $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
$ sed -i 's/\[SSL\]/\[SHIM\]/g' $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
```
|
||||
|
||||
### Expose and Bind Intel QAT virtual functions to VFIO-PCI (Every reboot)
|
||||
### Expose and Bind Intel® QAT virtual functions to VFIO-PCI (Every reboot)
|
||||
|
||||
To enable virtual functions, the host OS should have IOMMU groups enabled. In
|
||||
the UEFI Firmware Intel Virtualization Technology for Directed I/O
|
||||
(Intel VT-d) must be enabled. Also, the kernel boot parameter should be
|
||||
`intel_iommu=on` or `intel_iommu=ifgx_off`. The default in Clear Linux currently
|
||||
is `intel_iommu=igfx_off` which should work with the Intel QAT device. The
|
||||
following commands assume you installed an Intel QAT card, IOMMU is on, and
|
||||
the UEFI Firmware Intel® Virtualization Technology for Directed I/O
|
||||
(Intel® VT-d) must be enabled. Also, the kernel boot parameter should be
|
||||
`intel_iommu=on` or `intel_iommu=ifgx_off`. This should have been set from
|
||||
the instructions above. Check the output of `/proc/cmdline` to confirm. The
|
||||
following commands assume you installed an Intel® QAT card, IOMMU is on, and
|
||||
VT-d is enabled. The vendor and device ID add to the `VFIO-PCI` driver so that
|
||||
each exposed virtual function can be bound to the `VFIO-PCI` driver. Once
|
||||
complete, each virtual function passes into a Kata Containers container using
|
||||
the PCIe device passthrough feature. For Kubernetes, the Intel device plugin
|
||||
for Kubernetes handles the binding of the driver but the VF’s still must be
|
||||
the PCIe device passthrough feature. For Kubernetes, the
|
||||
[Intel device plugin](https://github.com/intel/intel-device-plugins-for-kubernetes)
|
||||
for Kubernetes handles the binding of the driver, but the VF’s still must be
|
||||
enabled.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ sudo modprobe vfio-pci
|
||||
$ QAT_PCI_BUS_PF_NUMBERS=$((lspci -d :435 && lspci -d :37c8 && lspci -d :19e2 && lspci -d :6f54) | cut -d ' ' -f 1)
|
||||
$ QAT_PCI_BUS_PF_1=$(echo $QAT_PCI_BUS_PF_NUMBERS | cut -d ' ' -f 1)
|
||||
@ -160,8 +170,10 @@ $ QAT_PCI_ID_VF=$(cat /sys/bus/pci/devices/0000:${QAT_PCI_BUS_PF_1}/virtfn0/ueve
|
||||
$ QAT_VENDOR_AND_ID_VF=$(echo ${QAT_PCI_ID_VF/PCI_ID=} | sed 's/:/ /')
|
||||
$ echo $QAT_VENDOR_AND_ID_VF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/new_id
|
||||
```
|
||||
|
||||
Loop through all the virtual functions and bind to the VFIO driver
|
||||
```sh
|
||||
|
||||
```bash
|
||||
$ for f in /sys/bus/pci/devices/0000:$QAT_PCI_BUS_PF_1/virtfn*
|
||||
do QAT_PCI_BUS_VF=$(basename $(readlink $f))
|
||||
echo $QAT_PCI_BUS_VF | sudo tee --append /sys/bus/pci/drivers/c6xxvf/unbind
|
||||
@ -169,22 +181,23 @@ $ for f in /sys/bus/pci/devices/0000:$QAT_PCI_BUS_PF_1/virtfn*
|
||||
done
|
||||
```
|
||||
|
||||
### Check Intel QAT virtual functions are enabled
|
||||
### Check Intel® QAT virtual functions are enabled
|
||||
|
||||
If the following command returns empty, then the virtual functions are not
|
||||
properly enabled. This command checks the enumerated device IDs for just the
|
||||
virtual functions. Using the Intel QAT as an example, the physical device ID
|
||||
virtual functions. Using the Intel® QAT as an example, the physical device ID
|
||||
is `37c8` and virtual function device ID is `37c9`. The following command checks
|
||||
if VF's are enabled for any of the currently known Intel QAT device ID's. The
|
||||
if VF's are enabled for any of the currently known Intel® QAT device ID's. The
|
||||
following `ls` command should show the 16 VF's bound to `VFIO-PCI`.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ for i in 0442 0443 37c9 19e3; do lspci -d 8086:$i; done
|
||||
```
|
||||
|
||||
Another way to check is to see what PCI devices that `VFIO-PCI` is mapped to.
|
||||
It should match the device ID's of the VF's.
|
||||
```sh
|
||||
|
||||
```bash
|
||||
$ ls -la /sys/bus/pci/drivers/vfio-pci
|
||||
```
|
||||
|
||||
@ -201,16 +214,16 @@ There are some patches that must be installed as well, which the
|
||||
`build-kernel.sh` script should automatically apply. If you are using a
|
||||
different kernel version, then you might need to manually apply them. Since
|
||||
the Kata Containers kernel has a minimal set of kernel flags set, you must
|
||||
create a QAT kernel fragment with the necessary `CONFIG_CRYPTO_*` options set.
|
||||
create a Intel® QAT kernel fragment with the necessary `CONFIG_CRYPTO_*` options set.
|
||||
Update the config to set some of the `CRYPTO` flags to enabled. This might
|
||||
change with different kernel versions. We tested the following instructions
|
||||
with kernel `v4.19.28-41`.
|
||||
change with different kernel versions. The following instructions were tested
|
||||
with kernel `v5.4.0-64-generic`.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $GOPATH
|
||||
$ cd $GOPATH
|
||||
$ go get -v github.com/kata-containers/packaging
|
||||
$ cat << EOF > $GOPATH/src/github.com/kata-containers/packaging/kernel/configs/fragments/common/qat.conf
|
||||
$ go get -v github.com/kata-containers/kata-containers
|
||||
$ cat << EOF > $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/configs/fragments/common/qat.conf
|
||||
CONFIG_PCIEAER=y
|
||||
CONFIG_UIO=y
|
||||
CONFIG_CRYPTO_HW=y
|
||||
@ -221,61 +234,70 @@ CONFIG_MODULE_SIG=y
|
||||
CONFIG_CRYPTO_AUTHENC=y
|
||||
CONFIG_CRYPTO_DH=y
|
||||
EOF
|
||||
$ $GOPATH/src/github.com/kata-containers/packaging/kernel/build-kernel.sh setup
|
||||
$ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh setup
|
||||
```
|
||||
|
||||
### Build Kata kernel
|
||||
|
||||
```sh
|
||||
$ export LINUX_VER=$(ls -d kata*)
|
||||
```bash
|
||||
$ cd $GOPATH
|
||||
$ export LINUX_VER=$(ls -d kata-linux-*)
|
||||
$ sed -i 's/EXTRAVERSION =/EXTRAVERSION = .qat.container/' $LINUX_VER/Makefile
|
||||
$ $GOPATH/src/github.com/kata-containers/packaging/kernel/build-kernel.sh build
|
||||
$ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh build
|
||||
```
|
||||
|
||||
|
||||
### Copy Kata kernel
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ export KATA_KERNEL_NAME=vmlinux-${LINUX_VER}_qat
|
||||
$ mkdir -p $KATA_KERNEL_LOCATION
|
||||
$ cp $LINUX_VER/arch/x86/boot/bzImage $KATA_KERNEL_LOCATION/vmlinuz-${LINUX_VER}_qat
|
||||
$ cp ${GOPATH}/${LINUX_VER}/vmlinux ${KATA_KERNEL_LOCATION}/${KATA_KERNEL_NAME}
|
||||
```
|
||||
|
||||
### Prepare Kata root filesystem
|
||||
|
||||
These instructions build upon the OS builder instructions located in the
|
||||
[Developer Guide](../Developer-Guide.md). The following instructions use Clear
|
||||
Linux (Kata Containers default) as the root filesystem with systemd as the
|
||||
init and will add in the `kmod` binary, which is not a standard binary in a
|
||||
Kata rootfs image. The `kmod` binary is necessary to load the QAT kernel
|
||||
modules when the virtual machine rootfs boots. You should install Docker on
|
||||
your system before running the following commands. If you need to use a custom
|
||||
`kata-agent`, then refer to the previous link on how to add it in.
|
||||
[Developer Guide](../Developer-Guide.md). At this point it is recommended that
|
||||
[Docker](https://docs.docker.com/engine/install/ubuntu/) is installed first, and
|
||||
then [Kata-deploy](https://github.com/kata-containers/kata-containers/tree/main/tools/packaging/kata-deploy)
|
||||
is use to install Kata. This will make sure that the correct `agent` version
|
||||
is installed into the rootfs in the steps below.
|
||||
|
||||
```sh
|
||||
$ mkdir -p $OSBUILDER
|
||||
$ cd $OSBUILDER
|
||||
$ git clone https://github.com/kata-containers/osbuilder.git
|
||||
$ export ROOTFS_DIR=${OSBUILDER}/osbuilder/rootfs-builder/rootfs
|
||||
The following instructions use Debian as the root filesystem with systemd as
|
||||
the init and will add in the `kmod` binary, which is not a standard binary in
|
||||
a Kata rootfs image. The `kmod` binary is necessary to load the Intel® QAT
|
||||
kernel modules when the virtual machine rootfs boots.
|
||||
|
||||
```bash
|
||||
$ export OSBUILDER=$GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder
|
||||
$ export ROOTFS_DIR=${OSBUILDER}/rootfs-builder/rootfs
|
||||
$ export EXTRA_PKGS='kmod'
|
||||
```
|
||||
|
||||
Make sure that the `kata-agent` version matches the installed `kata-runtime`
|
||||
version.
|
||||
```sh
|
||||
version. Also make sure the `kata-runtime` install location is in your `PATH`
|
||||
variable. The following `AGENT_VERSION` can be set manually to match
|
||||
the `kata-runtime` version if the following commands don't work.
|
||||
|
||||
```bash
|
||||
$ export PATH=$PATH:/opt/kata/bin
|
||||
$ cd $GOPATH
|
||||
$ export AGENT_VERSION=$(kata-runtime version | head -n 1 | grep -o "[0-9.]\+")
|
||||
$ cd ${OSBUILDER}/osbuilder/rootfs-builder
|
||||
$ cd ${OSBUILDER}/rootfs-builder
|
||||
$ sudo rm -rf ${ROOTFS_DIR}
|
||||
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh clearlinux'
|
||||
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh debian'
|
||||
```
|
||||
|
||||
### Compile Intel QAT drivers for Kata Containers kernel and add to Kata Containers rootfs
|
||||
### Compile Intel® QAT drivers for Kata Containers kernel and add to Kata Containers rootfs
|
||||
|
||||
After the Kata Containers kernel builds with the proper configuration flags,
|
||||
you must build the Intel QAT drivers against that Kata Containers kernel
|
||||
you must build the Intel® QAT drivers against that Kata Containers kernel
|
||||
version in a similar way they were previously built for the host OS. You must
|
||||
set the `KERNEL_SOURCE_ROOT` variable to the Kata Containers kernel source
|
||||
directory and build the Intel QAT drivers again.
|
||||
directory and build the Intel® QAT drivers again. The `make` command will
|
||||
install the Intel® QAT modules into the Kata rootfs.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ cd $GOPATH
|
||||
$ export LINUX_VER=$(ls -d kata*)
|
||||
$ export KERNEL_MAJOR_VERSION=$(awk '/^VERSION =/{print $NF}' $GOPATH/$LINUX_VER/Makefile)
|
||||
@ -284,16 +306,18 @@ $ export KERNEL_SUBLEVEL=$(awk '/^SUBLEVEL =/{print $NF}' $GOPATH/$LINUX_VER/Mak
|
||||
$ export KERNEL_EXTRAVERSION=$(awk '/^EXTRAVERSION =/{print $NF}' $GOPATH/$LINUX_VER/Makefile)
|
||||
$ export KERNEL_ROOTFS_DIR=${KERNEL_MAJOR_VERSION}.${KERNEL_PATHLEVEL}.${KERNEL_SUBLEVEL}${KERNEL_EXTRAVERSION}
|
||||
$ cd $QAT_SRC
|
||||
$ KERNEL_SOURCE_ROOT=$GOPATH/$LINUX_VER ./configure --disable-qat-lkcf --enable-icp-sriov=guest
|
||||
$ KERNEL_SOURCE_ROOT=$GOPATH/$LINUX_VER ./configure --enable-icp-sriov=guest
|
||||
$ sudo -E make all -j$(nproc)
|
||||
$ sudo -E make INSTALL_MOD_PATH=$ROOTFS_DIR qat-driver-install -j$(nproc)
|
||||
```
|
||||
|
||||
The `usdm_drv` module also needs to be copied into the rootfs modules path and
|
||||
`depmod` should be run.
|
||||
```sh
|
||||
$ sudo cp $QAT_SRC/build/usdm_drv.ko $ROOTFS_DIR/usr/lib/modules/${KERNEL_ROOTFS_DIR}/updates/drivers
|
||||
|
||||
```bash
|
||||
$ sudo cp $QAT_SRC/build/usdm_drv.ko $ROOTFS_DIR/lib/modules/${KERNEL_ROOTFS_DIR}/updates/drivers
|
||||
$ sudo depmod -a -b ${ROOTFS_DIR} ${KERNEL_ROOTFS_DIR}
|
||||
$ cd ${OSBUILDER}/osbuilder/image-builder
|
||||
$ cd ${OSBUILDER}/image-builder
|
||||
$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'
|
||||
```
|
||||
|
||||
@ -302,84 +326,225 @@ $ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'
|
||||
|
||||
### Copy Kata rootfs
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $KATA_ROOTFS_LOCATION
|
||||
$ cp ${OSBUILDER}/osbuilder/image-builder/kata-containers.img $KATA_ROOTFS_LOCATION
|
||||
$ cp ${OSBUILDER}/image-builder/kata-containers.img $KATA_ROOTFS_LOCATION
|
||||
```
|
||||
|
||||
### Update Kata configuration to point to custom kernel and rootfs
|
||||
## Verify Intel® QAT works in a container
|
||||
|
||||
You must update the `configuration.toml` for Kata Containers to point to the
|
||||
custom kernel, custom rootfs, and to specify which modules to load when the
|
||||
virtual machine is booted when a container is run. The following example
|
||||
assumes you installed an Intel QAT, and you need to load those modules.
|
||||
|
||||
```sh
|
||||
$ sudo mkdir -p /etc/kata-containers
|
||||
$ sudo cp /usr/share/defaults/kata-containers/configuration-qemu.toml /etc/kata-containers/configuration.toml
|
||||
$ sudo sed -i "s|kernel_params = \"\"|kernel_params = \"modules-load=usdm_drv,qat_c62xvf\"|g" /etc/kata-containers/configuration.toml
|
||||
$ sudo sed -i "s|\/usr\/share\/kata-containers\/kata-containers.img|${KATA_KERNEL_LOCATION}\/kata-containers.img|g" /etc/kata-containers/configuration.toml
|
||||
$ sudo sed -i "s|\/usr\/share\/kata-containers\/vmlinuz.container|${KATA_ROOTFS_LOCATION}\/vmlinuz-${LINUX_VER}_qat|g" /etc/kata-containers/configuration.toml
|
||||
```
|
||||
|
||||
## Verify Intel QAT works in a Docker Kata Containers container
|
||||
|
||||
The following instructions leverage an OpenSSL Dockerfile that builds the
|
||||
Intel QAT engine to allow OpenSSL to offload crypto functions. It is a
|
||||
convenient way to test that VFIO device passthrough for the Intel QAT VF’s are
|
||||
The following instructions uses a OpenSSL Dockerfile that builds the
|
||||
Intel® QAT engine to allow OpenSSL to offload crypto functions. It is a
|
||||
convenient way to test that VFIO device passthrough for the Intel® QAT VF’s are
|
||||
working properly with the Kata Containers VM.
|
||||
|
||||
## Build OpenSSL Intel QAT engine container
|
||||
### Build OpenSSL Intel® QAT engine container
|
||||
|
||||
Use the OpenSSL Intel QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/master/demo/openssl-qat-engine)
|
||||
Use the OpenSSL Intel® QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/master/demo/openssl-qat-engine)
|
||||
to build a container image with an optimized OpenSSL engine for
|
||||
Intel QAT. Using `docker build` with the Kata Containers runtime can sometimes
|
||||
have issues. Therefore, we recommended you change the default runtime to
|
||||
`runc` before doing a build. Instructions for this are below.
|
||||
Intel® QAT. Using `docker build` with the Kata Containers runtime can sometimes
|
||||
have issues. Therefore, make sure that `runc` is the default Docker container
|
||||
runtime.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ cd $QAT_SRC
|
||||
$ curl -O $QAT_DOCKERFILE
|
||||
$ sudo sed -i 's/kata-runtime/runc/g' /etc/systemd/system/docker.service.d/50-runtime.conf
|
||||
$ sudo systemctl daemon-reload && sudo systemctl restart docker
|
||||
$ sudo docker build -t openssl-qat-engine .
|
||||
```
|
||||
|
||||
> **Note: The Intel QAT driver version in this container might not match the
|
||||
> Intel QAT driver compiled and loaded on the host when compiling.**
|
||||
> **Note: The Intel® QAT driver version in this container might not match the
|
||||
> Intel® QAT driver compiled and loaded on the host when compiling.**
|
||||
|
||||
### Test Intel QAT in Docker
|
||||
### Test Intel® QAT with the ctr tool
|
||||
|
||||
The host should already be setup with 16 virtual functions of the Intel QAT
|
||||
card bound to `VFIO-PCI`. Verify this by looking in `/dev/vfio` for a listing
|
||||
of devices. Replace the number 90 with one of the VF’s exposed in `/dev/vfio`.
|
||||
It might require you to add an `IPC_LOCK` capability to your Docker runtime
|
||||
depending on which rootfs you use.
|
||||
The `ctr` tool can be used to interact with the containerd daemon. It may be
|
||||
more convenient to use this tool to verify the kernel and image instead of
|
||||
setting up a Kubernetes cluster. The correct Kata runtimes need to be added
|
||||
to the containerd `config.toml`. Below is a sample snippet that can be added
|
||||
to allow QEMU and Cloud Hypervisor (CLH) to work with `ctr`.
|
||||
|
||||
```sh
|
||||
$ sudo docker run -it --runtime=kata-runtime --cap-add=IPC_LOCK --cap-add=SYS_ADMIN --device=/dev/vfio/90 -v /dev:/dev -v ${QAT_CONF_LOCATION}:/etc openssl-qat-engine bash
|
||||
```
|
||||
[plugins.cri.containerd.runtimes.kata-qemu]
|
||||
runtime_type = "io.containerd.kata-qemu.v2"
|
||||
privileged_without_host_devices = true
|
||||
pod_annotations = ["io.katacontainers.*"]
|
||||
[plugins.cri.containerd.runtimes.kata-qemu.options]
|
||||
ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
|
||||
[plugins.cri.containerd.runtimes.kata-clh]
|
||||
runtime_type = "io.containerd.kata-clh.v2"
|
||||
privileged_without_host_devices = true
|
||||
pod_annotations = ["io.katacontainers.*"]
|
||||
[plugins.cri.containerd.runtimes.kata-clh.options]
|
||||
ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-clh.toml"
|
||||
```
|
||||
|
||||
Below are some commands to run in the container image to verify Intel QAT is
|
||||
In addition, containerd expects the binary to be in `/usr/local/bin` so add
|
||||
this small script so that it redirects to be able to use either QEMU or
|
||||
Cloud Hypervisor with Kata.
|
||||
|
||||
```bash
|
||||
$ echo '#!/bin/bash' | sudo tee /usr/local/bin/containerd-shim-kata-qemu-v2
|
||||
$ echo 'KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-qemu.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-qemu-v2
|
||||
$ sudo chmod +x /usr/local/bin/containerd-shim-kata-qemu-v2
|
||||
$ echo '#!/bin/bash' | sudo tee /usr/local/bin/containerd-shim-kata-clh-v2
|
||||
$ echo 'KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-clh.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-clh-v2
|
||||
$ sudo chmod +x /usr/local/bin/containerd-shim-kata-clh-v2
|
||||
```
|
||||
|
||||
After the OpenSSL image is built and imported into containerd, a Intel® QAT
|
||||
virtual function exposed in the step above can be added to the `ctr` command.
|
||||
Make sure to change the `/dev/vfio` number to one that actually exists on the
|
||||
host system. When using the `ctr` tool, the`configuration.toml` for Kata needs
|
||||
to point to the custom Kata kernel and rootfs built above and the Intel® QAT
|
||||
modules in the Kata rootfs need to load at boot. The following steps assume that
|
||||
`kata-deploy` was used to install Kata and QEMU is being tested. If using a
|
||||
different hypervisor, different install method for Kata, or a different
|
||||
Intel® QAT chipset then the command will need to be modified.
|
||||
|
||||
> **Note: The following was tested with
|
||||
[containerd v1.3.9](https://github.com/containerd/containerd/releases/tag/v1.3.9).**
|
||||
|
||||
```bash
|
||||
$ config_file="/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
|
||||
$ sudo sed -i "/kernel =/c kernel = "\"${KATA_ROOTFS_LOCATION}/${KATA_KERNEL_NAME}\""" $config_file
|
||||
$ sudo sed -i "/image =/c image = "\"${KATA_KERNEL_LOCATION}/kata-containers.img\""" $config_file
|
||||
$ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 modules-load=usdm_drv,qat_c62xvf"/g' $config_file
|
||||
$ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest
|
||||
$ sudo ctr images import openssl-qat-engine.tar
|
||||
$ sudo ctr run --runtime io.containerd.run.kata-qemu.v2 --privileged -t --rm --device=/dev/vfio/180 --mount type=bind,src=/dev,dst=/dev,options=rbind:rw --mount type=bind,src=${QAT_CONF_LOCATION}/c6xxvf_dev0.conf,dst=/etc/c6xxvf_dev0.conf,options=rbind:rw docker.io/library/openssl-qat-engine:latest bash
|
||||
```
|
||||
|
||||
Below are some commands to run in the container image to verify Intel® QAT is
|
||||
working
|
||||
|
||||
```sh
|
||||
bash-5.0# cat /proc/modules
|
||||
bash-5.0# adf_ctl restart
|
||||
bash-5.0# adf_ctl status
|
||||
bash-5.0# openssl engine -c -t qat
|
||||
root@67561dc2757a/ # cat /proc/modules
|
||||
qat_c62xvf 16384 - - Live 0xffffffffc00d9000 (OE)
|
||||
usdm_drv 86016 - - Live 0xffffffffc00e8000 (OE)
|
||||
intel_qat 249856 - - Live 0xffffffffc009b000 (OE)
|
||||
|
||||
root@67561dc2757a/ # adf_ctl restart
|
||||
Restarting all devices.
|
||||
Processing /etc/c6xxvf_dev0.conf
|
||||
|
||||
root@67561dc2757a/ # adf_ctl status
|
||||
Checking status of all devices.
|
||||
There is 1 QAT acceleration device(s) in the system:
|
||||
qat_dev0 - type: c6xxvf, inst_id: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up
|
||||
|
||||
root@67561dc2757a/ # openssl engine -c -t qat-hw
|
||||
(qat-hw) Reference implementation of QAT crypto engine v0.6.1
|
||||
[RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448]
|
||||
[ available ]
|
||||
```
|
||||
|
||||
Test with Intel QAT card acceleration
|
||||
### Test Intel® QAT in Kubernetes
|
||||
|
||||
```sh
|
||||
bash-5.0# openssl speed -engine qat -elapsed -async_jobs 72 rsa2048
|
||||
Start a Kubernetes cluster with containerd as the CRI. The host should
|
||||
already be setup with 16 virtual functions of the Intel® QAT card bound to
|
||||
`VFIO-PCI`. Verify this by looking in `/dev/vfio` for a listing of devices.
|
||||
You might need to disable Docker before initializing Kubernetes. Be aware
|
||||
that the OpenSSL container image built above will need to be exported from
|
||||
Docker and imported into containerd.
|
||||
|
||||
If Kata is installed through [`kata-deploy`](https://github.com/kata-containers/kata-containers/blob/stable-2.0/tools/packaging/kata-deploy/README.md)
|
||||
there will be multiple `configuration.toml` files associated with different
|
||||
hypervisors. Rather than add in the custom Kata kernel, Kata rootfs, and
|
||||
kernel modules to each `configuration.toml` as the default, instead use
|
||||
[annotations](https://github.com/kata-containers/kata-containers/blob/stable-2.0/docs/how-to/how-to-load-kernel-modules-with-kata.md)
|
||||
in the Kubernetes YAML file to tell Kata which kernel and rootfs to use. The
|
||||
easy way to do this is to use `kata-deploy` which will install the Kata binaries
|
||||
to `/opt` and properly configure the `/etc/containerd/config.toml` with annotation
|
||||
support. However, the `configuration.toml` needs to enable support for
|
||||
annotations as well. The following configures both QEMU and Cloud Hypervisor
|
||||
`configuration.toml` files that are currently available with Kata Container
|
||||
versions 2.0 and higher.
|
||||
|
||||
```bash
|
||||
$ sudo sed -i 's/enable_annotations\s=\s\[\]/enable_annotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-qemu.toml
|
||||
$ sudo sed -i 's/enable_annotations\s=\s\[\]/enable_annotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-clh.toml
|
||||
```
|
||||
|
||||
Test with CPU acceleration
|
||||
Export the OpenSSL image from Docker and import into containerd.
|
||||
|
||||
```bash
|
||||
$ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest
|
||||
$ sudo ctr -n=k8s.io images import openssl-qat-engine.tar
|
||||
```
|
||||
|
||||
The [Intel® QAT Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/cmd/qat_plugin/README.md)
|
||||
needs to be started so that the virtual functions can be discovered and
|
||||
used by Kubernetes.
|
||||
|
||||
The following YAML file can be used to start a Kata container with Intel® QAT
|
||||
support. If Kata is installed with `kata-deploy`, then the containerd
|
||||
`configuration.toml` should have all of the Kata runtime classes already
|
||||
populated and annotations supported. To use a Intel® QAT virtual function, the
|
||||
Intel® QAT plugin needs to be started after the VF's are bound to `VFIO-PCI` as
|
||||
described [above](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot).
|
||||
Edit the following to point to the correct Kata kernel and rootfs location
|
||||
built with Intel® QAT support.
|
||||
|
||||
```bash
|
||||
$ cat << EOF > kata-openssl-qat.yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: kata-openssl-qat
|
||||
labels:
|
||||
app: kata-openssl-qat
|
||||
annotations:
|
||||
io.katacontainers.config.hypervisor.kernel: "$KATA_KERNEL_LOCATION/$KATA_KERNEL_NAME"
|
||||
io.katacontainers.config.hypervisor.image: "$KATA_ROOTFS_LOCATION/kata-containers.img"
|
||||
io.katacontainers.config.hypervisor.kernel_params: "modules-load=usdm_drv,qat_c62xvf"
|
||||
spec:
|
||||
runtimeClassName: kata-qemu
|
||||
containers:
|
||||
- name: kata-openssl-qat
|
||||
image: docker.io/library/openssl-qat-engine:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
resources:
|
||||
limits:
|
||||
qat.intel.com/generic: 1
|
||||
cpu: 1
|
||||
securityContext:
|
||||
capabilities:
|
||||
add: ["IPC_LOCK", "SYS_ADMIN"]
|
||||
volumeMounts:
|
||||
- mountPath: /etc/c6xxvf_dev0.conf
|
||||
name: etc-mount
|
||||
- mountPath: /dev
|
||||
name: dev-mount
|
||||
volumes:
|
||||
- name: dev-mount
|
||||
hostPath:
|
||||
path: /dev
|
||||
- name: etc-mount
|
||||
hostPath:
|
||||
path: $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
EOF
|
||||
```
|
||||
|
||||
Use `kubectl` to start the pod. Verify that Intel® QAT card acceleration is
|
||||
working with the Intel® QAT engine.
|
||||
```bash
|
||||
$ kubectl apply -f kata-openssl-qat.yaml
|
||||
```
|
||||
|
||||
```sh
|
||||
bash-5.0# openssl speed -elapsed rsa2048
|
||||
$ kubectl exec -it kata-openssl-qat -- adf_ctl restart
|
||||
Restarting all devices.
|
||||
Processing /etc/c6xxvf_dev0.conf
|
||||
|
||||
$ kubectl exec -it kata-openssl-qat -- adf_ctl status
|
||||
Checking status of all devices.
|
||||
There is 1 QAT acceleration device(s) in the system:
|
||||
qat_dev0 - type: c6xxvf, inst_id: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up
|
||||
|
||||
$ kubectl exec -it kata-openssl-qat -- openssl engine -c -t qat-hw
|
||||
(qat-hw) Reference implementation of QAT crypto engine v0.6.1
|
||||
[RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448]
|
||||
[ available ]
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
@ -412,9 +577,9 @@ c6xxvf_dev10.conf c6xxvf_dev13.conf c6xxvf_dev2.conf c6xxvf_dev5.conf c6xxvf
|
||||
```
|
||||
|
||||
* Check `dmesg` inside the container to see if there are any issues with the
|
||||
Intel QAT driver.
|
||||
Intel® QAT driver.
|
||||
|
||||
* If there are issues building the OpenSSL Intel QAT container image, then
|
||||
* If there are issues building the OpenSSL Intel® QAT container image, then
|
||||
check to make sure that runc is the default runtime for building container.
|
||||
|
||||
```sh
|
||||
@ -425,17 +590,18 @@ Environment="DOCKER_DEFAULT_RUNTIME=--default-runtime runc"
|
||||
|
||||
## Optional Scripts
|
||||
|
||||
### Verify Intel QAT card counters are incremented
|
||||
### Verify Intel® QAT card counters are incremented
|
||||
|
||||
Use the `lspci` command to figure out which PCI bus the Intel QAT accelerators
|
||||
are on. The counters will increase when the accelerator is actively being
|
||||
used. To verify QAT is actively accelerating the containerized application,
|
||||
use the following instructions to check if any of the counters are
|
||||
incrementing. You will have to change the PCI device ID to match your system.
|
||||
To check the built in firmware counters, the Intel® QAT driver has to be compiled
|
||||
and installed to the host and can't rely on the built in host driver. The
|
||||
counters will increase when the accelerator is actively being used. To verify
|
||||
Intel® QAT is actively accelerating the containerized application, use the
|
||||
following instructions to check if any of the counters increment. Make
|
||||
sure to change the PCI Device ID to match whats in the system.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done
|
||||
$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b1\:00.0/fw_counters
|
||||
$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b3\:00.0/fw_counters
|
||||
$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b5\:00.0/fw_counters
|
||||
```
|
||||
```
|
||||
Loading…
Reference in New Issue
Block a user