diff --git a/docs/use-cases/using-Intel-QAT-and-kata.md b/docs/use-cases/using-Intel-QAT-and-kata.md index ffe98d508a..5ba93482d9 100644 --- a/docs/use-cases/using-Intel-QAT-and-kata.md +++ b/docs/use-cases/using-Intel-QAT-and-kata.md @@ -1,24 +1,20 @@ # Table of Contents +**Note:**: This guide used to contain an end-to-end flow to build a +custom Kata containers root filesystem with QAT out-of-tree SR-IOV virtual +function driver and run QAT enabled containers. The former is no longer necessary +so the instructions are dropped. If the use-case is still of interest, please file +an issue in either of the QAT Kubernetes specific repos linked below. + # Introduction Intel® QuickAssist Technology (QAT) provides hardware acceleration -for security (cryptography) and compression. These instructions cover the -steps for the latest [Ubuntu LTS release](https://ubuntu.com/download/desktop) -which already include the QAT host driver. These instructions can be adapted to -any Linux distribution. These instructions guide the user on how to download -the kernel sources, compile kernel driver modules against those sources, and -load them onto the host as well as preparing a specially built Kata Containers -kernel and custom Kata Containers rootfs. +for security (cryptography) and compression. Kata Containers can enable +these acceleration functions for containers using QAT SR-IOV with the +support from [Intel QAT Device Plugin for Kubernetes](https://github.com/intel/intel-device-plugins-for-kubernetes) +or [Intel QAT DRA Resource Driver for Kubernetes](https://github.com/intel/intel-resource-drivers-for-kubernetes). -* Download kernel sources -* Compile Kata kernel -* Compile kernel driver modules against those sources -* Download rootfs -* Add driver modules to rootfs -* Build rootfs image - -## Helpful Links before starting +## More Information [Intel® QuickAssist Technology at `01.org`](https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html) @@ -26,554 +22,6 @@ kernel and custom Kata Containers rootfs. [Intel Device Plugin for Kubernetes](https://github.com/intel/intel-device-plugins-for-kubernetes) +[Intel DRA Resource Driver for Kubernetes](https://github.com/intel/intel-resource-drivers-for-kubernetes) + [Intel® QuickAssist Technology for Crypto Poll Mode Driver](https://dpdk-docs.readthedocs.io/en/latest/cryptodevs/qat.html) - -## Steps to enable Intel® QAT in Kata Containers - -There are some steps to complete only once, some steps to complete with every -reboot, and some steps to complete when the host kernel changes. - -## Script variables - -The following list of variables must be set before running through the -scripts. These variables refer to locations to store modules and configuration -files on the host and links to the drivers to use. Modify these as -needed to point to updated drivers or different install locations. - -### Set environment variables (Every Reboot) - -Make sure to check [`01.org`](https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html) for -the latest driver. - -```bash -$ export QAT_DRIVER_VER=qat1.7.l.4.14.0-00031.tar.gz -$ export QAT_DRIVER_URL=https://downloadmirror.intel.com/30178/eng/${QAT_DRIVER_VER} -$ export QAT_CONF_LOCATION=~/QAT_conf -$ export QAT_DOCKERFILE=https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/main/demo/openssl-qat-engine/Dockerfile -$ export QAT_SRC=~/src/QAT -$ export GOPATH=~/src/go -$ export KATA_KERNEL_LOCATION=~/kata -$ export KATA_ROOTFS_LOCATION=~/kata -``` - -## Prepare the Ubuntu Host - -The host could be a bare metal instance or a virtual machine. If using a -virtual machine, make sure that KVM nesting is enabled. The following -instructions reference an Intel® C62X chipset. Some of the instructions must be -modified if using a different Intel® QAT device. The Intel® QAT chipset can be -identified by executing the following. - -### Identify which PCI Bus the Intel® QAT card is on - -```bash -$ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done -``` - -### Install necessary packages for Ubuntu - -These packages are necessary to compile the Kata kernel, Intel® QAT driver, and to -prepare the rootfs for Kata. [Docker](https://docs.docker.com/engine/install/ubuntu/) -also needs to be installed to be able to build the rootfs. To test that -everything works a Kubernetes pod is started requesting Intel® QAT resources. For the -pass through of the virtual functions the kernel boot parameter needs to have -`INTEL_IOMMU=on`. - -```bash -$ sudo apt update -$ sudo apt install -y golang-go build-essential python pkg-config zlib1g-dev libudev-dev bison libelf-dev flex libtool automake autotools-dev autoconf bc libpixman-1-dev coreutils libssl-dev -$ sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=""/GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"/' /etc/default/grub -$ sudo update-grub -$ sudo reboot -``` - -### Download Intel® QAT drivers - -This will download the [Intel® QAT drivers](https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html). -Make sure to check the website for the latest version. - -```bash -$ mkdir -p $QAT_SRC -$ cd $QAT_SRC -$ curl -L $QAT_DRIVER_URL | tar zx -``` - -### Copy Intel® QAT configuration files and enable virtual functions - -Modify the instructions below as necessary if using a different Intel® QAT hardware -platform. You can learn more about customizing configuration files at the -[Intel® QAT Engine repository](https://github.com/intel/QAT_Engine/#copy-the-correct-intel-quickassist-technology-driver-config-files) -This section starts from a base config file and changes the `SSL` section to -`SHIM` to support the OpenSSL engine. There are more tweaks that you can make -depending on the use case and how many Intel® QAT engines should be run. You -can find more information about how to customize in the -[Intel® QuickAssist Technology Software for Linux* - Programmer's Guide.](https://www.intel.com/content/www/us/en/content-details/709196/intel-quickassist-technology-api-programmer-s-guide.html) - -> **Note: This section assumes that a Intel® QAT `c6xx` platform is used.** - -```bash -$ mkdir -p $QAT_CONF_LOCATION -$ cp $QAT_SRC/quickassist/utilities/adf_ctl/conf_files/c6xxvf_dev0.conf.vm $QAT_CONF_LOCATION/c6xxvf_dev0.conf -$ sed -i 's/\[SSL\]/\[SHIM\]/g' $QAT_CONF_LOCATION/c6xxvf_dev0.conf -``` - -### Expose and Bind Intel® QAT virtual functions to VFIO-PCI (Every reboot) - -To enable virtual functions, the host OS should have IOMMU groups enabled. In -the UEFI Firmware Intel® Virtualization Technology for Directed I/O -(Intel® VT-d) must be enabled. Also, the kernel boot parameter should be -`intel_iommu=on` or `intel_iommu=ifgx_off`. This should have been set from -the instructions above. Check the output of `/proc/cmdline` to confirm. The -following commands assume you installed an Intel® QAT card, IOMMU is on, and -VT-d is enabled. The vendor and device ID add to the `VFIO-PCI` driver so that -each exposed virtual function can be bound to the `VFIO-PCI` driver. Once -complete, each virtual function passes into a Kata Containers container using -the PCIe device passthrough feature. For Kubernetes, the -[Intel device plugin](https://github.com/intel/intel-device-plugins-for-kubernetes) -for Kubernetes handles the binding of the driver, but the VF’s still must be -enabled. - -```bash -$ sudo modprobe vfio-pci -$ QAT_PCI_BUS_PF_NUMBERS=$((lspci -d :435 && lspci -d :37c8 && lspci -d :19e2 && lspci -d :6f54) | cut -d ' ' -f 1) -$ QAT_PCI_BUS_PF_1=$(echo $QAT_PCI_BUS_PF_NUMBERS | cut -d ' ' -f 1) -$ echo 16 | sudo tee /sys/bus/pci/devices/0000:$QAT_PCI_BUS_PF_1/sriov_numvfs -$ QAT_PCI_ID_VF=$(cat /sys/bus/pci/devices/0000:${QAT_PCI_BUS_PF_1}/virtfn0/uevent | grep PCI_ID) -$ QAT_VENDOR_AND_ID_VF=$(echo ${QAT_PCI_ID_VF/PCI_ID=} | sed 's/:/ /') -$ echo $QAT_VENDOR_AND_ID_VF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/new_id -``` - -Loop through all the virtual functions and bind to the VFIO driver - -```bash -$ for f in /sys/bus/pci/devices/0000:$QAT_PCI_BUS_PF_1/virtfn* - do QAT_PCI_BUS_VF=$(basename $(readlink $f)) - echo $QAT_PCI_BUS_VF | sudo tee --append /sys/bus/pci/drivers/c6xxvf/unbind - echo $QAT_PCI_BUS_VF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/bind - done -``` - -### Check Intel® QAT virtual functions are enabled - -If the following command returns empty, then the virtual functions are not -properly enabled. This command checks the enumerated device IDs for just the -virtual functions. Using the Intel® QAT as an example, the physical device ID -is `37c8` and virtual function device ID is `37c9`. The following command checks -if VF's are enabled for any of the currently known Intel® QAT device ID's. The -following `ls` command should show the 16 VF's bound to `VFIO-PCI`. - -```bash -$ for i in 0442 0443 37c9 19e3; do lspci -d 8086:$i; done -``` - -Another way to check is to see what PCI devices that `VFIO-PCI` is mapped to. -It should match the device ID's of the VF's. - -```bash -$ ls -la /sys/bus/pci/drivers/vfio-pci -``` - -## Prepare Kata Containers - -### Download Kata kernel Source - -This example automatically uses the latest Kata kernel supported by Kata. It -follows the instructions from the -[packaging kernel repository](../../tools/packaging/kernel) -and uses the latest Kata kernel -[config](../../tools/packaging/kernel/configs). -There are some patches that must be installed as well, which the -`build-kernel.sh` script should automatically apply. If you are using a -different kernel version, then you might need to manually apply them. Since -the Kata Containers kernel has a minimal set of kernel flags set, you must -create a Intel® QAT kernel fragment with the necessary `CONFIG_CRYPTO_*` options set. -Update the config to set some of the `CRYPTO` flags to enabled. This might -change with different kernel versions. The following instructions were tested -with kernel `v5.4.0-64-generic`. - -```bash -$ mkdir -p $GOPATH -$ cd $GOPATH -$ go get -v github.com/kata-containers/kata-containers -$ cat << EOF > $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/configs/fragments/common/qat.conf -CONFIG_PCIEAER=y -CONFIG_UIO=y -CONFIG_CRYPTO_HW=y -CONFIG_CRYPTO_DEV_QAT_C62XVF=m -CONFIG_CRYPTO_CBC=y -CONFIG_MODULES=y -CONFIG_MODULE_SIG=y -CONFIG_CRYPTO_AUTHENC=y -CONFIG_CRYPTO_DH=y -EOF -$ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh setup -``` - -### Build Kata kernel - -```bash -$ cd $GOPATH -$ export LINUX_VER=$(ls -d kata-linux-*) -$ sed -i 's/EXTRAVERSION =/EXTRAVERSION = .qat.container/' $LINUX_VER/Makefile -$ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh build -``` - -### Copy Kata kernel - -```bash -$ export KATA_KERNEL_NAME=vmlinux-${LINUX_VER}_qat -$ mkdir -p $KATA_KERNEL_LOCATION -$ cp ${GOPATH}/${LINUX_VER}/vmlinux ${KATA_KERNEL_LOCATION}/${KATA_KERNEL_NAME} -``` - -### Prepare Kata root filesystem - -These instructions build upon the OS builder instructions located in the -[Developer Guide](../Developer-Guide.md). At this point it is recommended that -[Docker](https://docs.docker.com/engine/install/ubuntu/) is installed first, and -then [Kata-deploy](../../tools/packaging/kata-deploy) -is use to install Kata. This will make sure that the correct `agent` version -is installed into the rootfs in the steps below. - -The following instructions use Ubuntu as the root filesystem with systemd as -the init and will add in the `kmod` binary, which is not a standard binary in -a Kata rootfs image. The `kmod` binary is necessary to load the Intel® QAT -kernel modules when the virtual machine rootfs boots. - -```bash -$ export OSBUILDER=$GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder -$ export ROOTFS_DIR=${OSBUILDER}/rootfs-builder/rootfs -$ export EXTRA_PKGS='kmod' -``` - -Make sure that the `kata-agent` version matches the installed `kata-runtime` -version. Also make sure the `kata-runtime` install location is in your `PATH` -variable. The following `AGENT_VERSION` can be set manually to match -the `kata-runtime` version if the following commands don't work. - -```bash -$ export PATH=$PATH:/opt/kata/bin -$ cd $GOPATH -$ export AGENT_VERSION=$(kata-runtime version | head -n 1 | grep -o "[0-9.]\+") -$ cd ${OSBUILDER}/rootfs-builder -$ sudo rm -rf ${ROOTFS_DIR} -$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh ubuntu' -``` - -### Compile Intel® QAT drivers for Kata Containers kernel and add to Kata Containers rootfs - -After the Kata Containers kernel builds with the proper configuration flags, -you must build the Intel® QAT drivers against that Kata Containers kernel -version in a similar way they were previously built for the host OS. You must -set the `KERNEL_SOURCE_ROOT` variable to the Kata Containers kernel source -directory and build the Intel® QAT drivers again. The `make` command will -install the Intel® QAT modules into the Kata rootfs. - -```bash -$ cd $GOPATH -$ export LINUX_VER=$(ls -d kata*) -$ export KERNEL_MAJOR_VERSION=$(awk '/^VERSION =/{print $NF}' $GOPATH/$LINUX_VER/Makefile) -$ export KERNEL_PATHLEVEL=$(awk '/^PATCHLEVEL =/{print $NF}' $GOPATH/$LINUX_VER/Makefile) -$ export KERNEL_SUBLEVEL=$(awk '/^SUBLEVEL =/{print $NF}' $GOPATH/$LINUX_VER/Makefile) -$ export KERNEL_EXTRAVERSION=$(awk '/^EXTRAVERSION =/{print $NF}' $GOPATH/$LINUX_VER/Makefile) -$ export KERNEL_ROOTFS_DIR=${KERNEL_MAJOR_VERSION}.${KERNEL_PATHLEVEL}.${KERNEL_SUBLEVEL}${KERNEL_EXTRAVERSION} -$ cd $QAT_SRC -$ KERNEL_SOURCE_ROOT=$GOPATH/$LINUX_VER ./configure --enable-icp-sriov=guest -$ sudo -E make all -j $(nproc) -$ sudo -E make INSTALL_MOD_PATH=$ROOTFS_DIR qat-driver-install -j $(nproc) -``` - -The `usdm_drv` module also needs to be copied into the rootfs modules path and -`depmod` should be run. - -```bash -$ sudo cp $QAT_SRC/build/usdm_drv.ko $ROOTFS_DIR/lib/modules/${KERNEL_ROOTFS_DIR}/updates/drivers -$ sudo depmod -a -b ${ROOTFS_DIR} ${KERNEL_ROOTFS_DIR} -$ cd ${OSBUILDER}/image-builder -$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}' -``` - -> **Note: Ignore any errors on modules.builtin and modules.order when running -> `depmod`.** - -### Copy Kata rootfs - -```bash -$ mkdir -p $KATA_ROOTFS_LOCATION -$ cp ${OSBUILDER}/image-builder/kata-containers.img $KATA_ROOTFS_LOCATION -``` - -## Verify Intel® QAT works in a container - -The following instructions uses a OpenSSL Dockerfile that builds the -Intel® QAT engine to allow OpenSSL to offload crypto functions. It is a -convenient way to test that VFIO device passthrough for the Intel® QAT VF’s are -working properly with the Kata Containers VM. - -### Build OpenSSL Intel® QAT engine container - -Use the OpenSSL Intel® QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/main/demo/openssl-qat-engine) -to build a container image with an optimized OpenSSL engine for -Intel® QAT. Using `docker build` with the Kata Containers runtime can sometimes -have issues. Therefore, make sure that `runc` is the default Docker container -runtime. - -```bash -$ cd $QAT_SRC -$ curl -O $QAT_DOCKERFILE -$ sudo docker build -t openssl-qat-engine . -``` - -> **Note: The Intel® QAT driver version in this container might not match the -> Intel® QAT driver compiled and loaded on the host when compiling.** - -### Test Intel® QAT with the ctr tool - -The `ctr` tool can be used to interact with the containerd daemon. It may be -more convenient to use this tool to verify the kernel and image instead of -setting up a Kubernetes cluster. The correct Kata runtimes need to be added -to the containerd `config.toml`. Below is a sample snippet that can be added -to allow QEMU and Cloud Hypervisor (CLH) to work with `ctr`. - -``` -[plugins.cri.containerd.runtimes.kata-qemu] - runtime_type = "io.containerd.kata-qemu.v2" - privileged_without_host_devices = true - pod_annotations = ["io.katacontainers.*"] - [plugins.cri.containerd.runtimes.kata-qemu.options] - ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml" -[plugins.cri.containerd.runtimes.kata-clh] - runtime_type = "io.containerd.kata-clh.v2" - privileged_without_host_devices = true - pod_annotations = ["io.katacontainers.*"] - [plugins.cri.containerd.runtimes.kata-clh.options] - ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-clh.toml" -``` - -In addition, containerd expects the binary to be in `/usr/local/bin` so add -this small script so that it redirects to be able to use either QEMU or -Cloud Hypervisor with Kata. - -```bash -$ echo '#!/usr/bin/env bash' | sudo tee /usr/local/bin/containerd-shim-kata-qemu-v2 -$ echo 'KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-qemu.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-qemu-v2 -$ sudo chmod +x /usr/local/bin/containerd-shim-kata-qemu-v2 -$ echo '#!/usr/bin/env bash' | sudo tee /usr/local/bin/containerd-shim-kata-clh-v2 -$ echo 'KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-clh.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-clh-v2 -$ sudo chmod +x /usr/local/bin/containerd-shim-kata-clh-v2 -``` - -After the OpenSSL image is built and imported into containerd, a Intel® QAT -virtual function exposed in the step above can be added to the `ctr` command. -Make sure to change the `/dev/vfio` number to one that actually exists on the -host system. When using the `ctr` tool, the`configuration.toml` for Kata needs -to point to the custom Kata kernel and rootfs built above and the Intel® QAT -modules in the Kata rootfs need to load at boot. The following steps assume that -`kata-deploy` was used to install Kata and QEMU is being tested. If using a -different hypervisor, different install method for Kata, or a different -Intel® QAT chipset then the command will need to be modified. - -> **Note: The following was tested with -[containerd v1.4.6](https://github.com/containerd/containerd/releases/tag/v1.4.6).** - -```bash -$ config_file="/opt/kata/share/defaults/kata-containers/configuration-qemu.toml" -$ sudo sed -i "/kernel =/c kernel = "\"${KATA_ROOTFS_LOCATION}/${KATA_KERNEL_NAME}\""" $config_file -$ sudo sed -i "/image =/c image = "\"${KATA_KERNEL_LOCATION}/kata-containers.img\""" $config_file -$ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 modules-load=usdm_drv,qat_c62xvf"/g' $config_file -$ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest -$ sudo ctr images import openssl-qat-engine.tar -$ sudo ctr run --runtime io.containerd.run.kata-qemu.v2 --privileged -t --rm --device=/dev/vfio/180 --mount type=bind,src=/dev,dst=/dev,options=rbind:rw --mount type=bind,src=${QAT_CONF_LOCATION}/c6xxvf_dev0.conf,dst=/etc/c6xxvf_dev0.conf,options=rbind:rw docker.io/library/openssl-qat-engine:latest bash -``` - -Below are some commands to run in the container image to verify Intel® QAT is -working - -```sh -root@67561dc2757a/ # cat /proc/modules -qat_c62xvf 16384 - - Live 0xffffffffc00d9000 (OE) -usdm_drv 86016 - - Live 0xffffffffc00e8000 (OE) -intel_qat 249856 - - Live 0xffffffffc009b000 (OE) - -root@67561dc2757a/ # adf_ctl restart -Restarting all devices. -Processing /etc/c6xxvf_dev0.conf - -root@67561dc2757a/ # adf_ctl status -Checking status of all devices. -There is 1 QAT acceleration device(s) in the system: - qat_dev0 - type: c6xxvf, inst_id: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up - -root@67561dc2757a/ # openssl engine -c -t qat-hw -(qat-hw) Reference implementation of QAT crypto engine v0.6.1 - [RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448] - [ available ] -``` - -### Test Intel® QAT in Kubernetes - -Start a Kubernetes cluster with containerd as the CRI. The host should -already be setup with 16 virtual functions of the Intel® QAT card bound to -`VFIO-PCI`. Verify this by looking in `/dev/vfio` for a listing of devices. -You might need to disable Docker before initializing Kubernetes. Be aware -that the OpenSSL container image built above will need to be exported from -Docker and imported into containerd. - -If Kata is installed through [`kata-deploy`](../../tools/packaging/kata-deploy/helm-chart/README.md) -there will be multiple `configuration.toml` files associated with different -hypervisors. Rather than add in the custom Kata kernel, Kata rootfs, and -kernel modules to each `configuration.toml` as the default, instead use -[annotations](../how-to/how-to-load-kernel-modules-with-kata.md) -in the Kubernetes YAML file to tell Kata which kernel and rootfs to use. The -easy way to do this is to use `kata-deploy` which will install the Kata binaries -to `/opt` and properly configure the `/etc/containerd/config.toml` with annotation -support. However, the `configuration.toml` needs to enable support for -annotations as well. The following configures both QEMU and Cloud Hypervisor -`configuration.toml` files that are currently available with Kata Container -versions 2.0 and higher. - -```bash -$ sudo sed -i 's/enable_annotations\s=\s\[\]/enable_annotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-qemu.toml -$ sudo sed -i 's/enable_annotations\s=\s\[\]/enable_annotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-clh.toml -``` - -Export the OpenSSL image from Docker and import into containerd. - -```bash -$ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest -$ sudo ctr -n=k8s.io images import openssl-qat-engine.tar -``` - -The [Intel® QAT Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/qat_plugin/README.md) -needs to be started so that the virtual functions can be discovered and -used by Kubernetes. - -The following YAML file can be used to start a Kata container with Intel® QAT -support. If Kata is installed with `kata-deploy`, then the containerd -`configuration.toml` should have all of the Kata runtime classes already -populated and annotations supported. To use a Intel® QAT virtual function, the -Intel® QAT plugin needs to be started after the VF's are bound to `VFIO-PCI` as -described [above](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot). -Edit the following to point to the correct Kata kernel and rootfs location -built with Intel® QAT support. - -```bash -$ cat << EOF > kata-openssl-qat.yaml -apiVersion: v1 -kind: Pod -metadata: - name: kata-openssl-qat - labels: - app: kata-openssl-qat - annotations: - io.katacontainers.config.hypervisor.kernel: "$KATA_KERNEL_LOCATION/$KATA_KERNEL_NAME" - io.katacontainers.config.hypervisor.image: "$KATA_ROOTFS_LOCATION/kata-containers.img" - io.katacontainers.config.hypervisor.kernel_params: "modules-load=usdm_drv,qat_c62xvf" -spec: - runtimeClassName: kata-qemu - containers: - - name: kata-openssl-qat - image: docker.io/library/openssl-qat-engine:latest - imagePullPolicy: IfNotPresent - resources: - limits: - qat.intel.com/generic: 1 - cpu: 1 - securityContext: - capabilities: - add: ["IPC_LOCK", "SYS_ADMIN"] - volumeMounts: - - mountPath: /etc/c6xxvf_dev0.conf - name: etc-mount - - mountPath: /dev - name: dev-mount - volumes: - - name: dev-mount - hostPath: - path: /dev - - name: etc-mount - hostPath: - path: $QAT_CONF_LOCATION/c6xxvf_dev0.conf -EOF -``` - -Use `kubectl` to start the pod. Verify that Intel® QAT card acceleration is -working with the Intel® QAT engine. -```bash -$ kubectl apply -f kata-openssl-qat.yaml -``` - -```sh -$ kubectl exec -it kata-openssl-qat -- adf_ctl restart -Restarting all devices. -Processing /etc/c6xxvf_dev0.conf - -$ kubectl exec -it kata-openssl-qat -- adf_ctl status -Checking status of all devices. -There is 1 QAT acceleration device(s) in the system: - qat_dev0 - type: c6xxvf, inst_id: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up - -$ kubectl exec -it kata-openssl-qat -- openssl engine -c -t qat-hw -(qat-hw) Reference implementation of QAT crypto engine v0.6.1 - [RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448] - [ available ] -``` - -### Troubleshooting - -* Check that `/dev/vfio` has VF’s enabled. - -```sh -$ ls /dev/vfio -57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 vfio -``` - -* Check that the modules load when inside the Kata Container. - -```sh -bash-5.0# grep -E "qat|usdm_drv" /proc/modules -qat_c62xvf 16384 - - Live 0x0000000000000000 (O) -usdm_drv 86016 - - Live 0x0000000000000000 (O) -intel_qat 184320 - - Live 0x0000000000000000 (O) -``` - -* Verify that at least the first `c6xxvf_dev0.conf` file mounts inside the -container image in `/etc`. You will need one configuration file for each VF -passed into the container. - -```sh -bash-5.0# ls /etc -c6xxvf_dev0.conf c6xxvf_dev11.conf c6xxvf_dev14.conf c6xxvf_dev3.conf c6xxvf_dev6.conf c6xxvf_dev9.conf resolv.conf -c6xxvf_dev1.conf c6xxvf_dev12.conf c6xxvf_dev15.conf c6xxvf_dev4.conf c6xxvf_dev7.conf hostname -c6xxvf_dev10.conf c6xxvf_dev13.conf c6xxvf_dev2.conf c6xxvf_dev5.conf c6xxvf_dev8.conf hosts -``` - -* Check `dmesg` inside the container to see if there are any issues with the -Intel® QAT driver. - -* If there are issues building the OpenSSL Intel® QAT container image, then -check to make sure that runc is the default runtime for building container. - -```sh -$ cat /etc/systemd/system/docker.service.d/50-runtime.conf -[Service] -Environment="DOCKER_DEFAULT_RUNTIME=--default-runtime runc" -``` - -## Optional Scripts - -### Verify Intel® QAT card counters are incremented - -To check the built in firmware counters, the Intel® QAT driver has to be compiled -and installed to the host and can't rely on the built in host driver. The -counters will increase when the accelerator is actively being used. To verify -Intel® QAT is actively accelerating the containerized application, use the -following instructions to check if any of the counters increment. Make -sure to change the PCI Device ID to match whats in the system. - -```bash -$ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done -$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b1\:00.0/fw_counters -$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b3\:00.0/fw_counters -$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b5\:00.0/fw_counters -``` diff --git a/tools/osbuilder/README.md b/tools/osbuilder/README.md index d86ecc144e..5e04d4a2c7 100644 --- a/tools/osbuilder/README.md +++ b/tools/osbuilder/README.md @@ -182,17 +182,6 @@ The Kata Containers kernel and rootfs images are by design "minimal". If advance site specific, or customized features are required, then building a customized kernel and/or rootfs may be required. -The below are some examples which may help or be useful for generating a -customized system. - -#### Intel® QuickAssist Technology (QAT) customized kernel and rootfs - -As documented in the -[Intel® QAT Kata use-case documentation](../../docs/use-cases/using-Intel-QAT-and-kata.md), -enabling this hardware requires a customized kernel and rootfs to work with Kata. -To ease building of the kernel and rootfs, a [Dockerfile](./dockerfiles/QAT) is -supplied, that when run, generates the required kernel and rootfs binaries. - ## Testing ``` diff --git a/tools/osbuilder/dockerfiles/QAT/Dockerfile b/tools/osbuilder/dockerfiles/QAT/Dockerfile deleted file mode 100644 index d13d2c6569..0000000000 --- a/tools/osbuilder/dockerfiles/QAT/Dockerfile +++ /dev/null @@ -1,66 +0,0 @@ -# Copyright (c) 2020 Intel Corporation -# -# SPDX-License-Identifier: Apache-2.0 - -# Kata osbuilder 'works best' on Fedora -FROM fedora:38 - -# Version of the Dockerfile - update if you change this file to avoid 'stale' -# images being pulled from the registry. -# Set AGENT_VERSION as an env variable to specify a specific version of Kata Agent to install - -LABEL DOCKERFILE_VERSION="2.3" - -ENV QAT_DRIVER_VER "QAT.L.4.15.0-00011.tar.gz" -ENV QAT_DRIVER_URL "https://downloadmirror.intel.com/649693/${QAT_DRIVER_VER}" -ENV QAT_CONFIGURE_OPTIONS "--enable-icp-sriov=guest" -ENV KATA_REPO_VERSION "main" -ENV AGENT_VERSION "" -ENV ROOTFS_OS "centos" -ENV OUTPUT_DIR "/output" - -RUN dnf install -y \ - bc \ - bison \ - debootstrap \ - diffutils \ - e2fsprogs \ - elfutils-libelf-devel \ - findutils \ - flex \ - gcc \ - gcc-c++ \ - git \ - kiwi \ - kmod \ - openssl \ - openssl-devel \ - make \ - parted \ - patch \ - qemu-img \ - systemd-devel \ - sudo \ - xz \ - yasm && \ - dnf clean all - -# Add in non-privileged user -RUN useradd qatbuilder -p "" && \ - echo "qatbuilder ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers - -# Pull in our local files -COPY ./run.sh /input/ -COPY ./qat.conf /input/ - -# Change to a less privileged user before running the commands -USER qatbuilder - -# Output is placed in the /output directory. -# We could make this a VOLUME to force it to be attached to the host, but let's -# just leave it as a container dir that can then be over-ridden from a host commandline -# volume setup. -# VOLUME /output - -# By default build everything -CMD ["/input/run.sh"] diff --git a/tools/osbuilder/dockerfiles/QAT/README.md b/tools/osbuilder/dockerfiles/QAT/README.md deleted file mode 100644 index 87730ebd78..0000000000 --- a/tools/osbuilder/dockerfiles/QAT/README.md +++ /dev/null @@ -1,81 +0,0 @@ -## Introduction - -The files in this directory can be used to build a modified Kata Containers rootfs -and kernel with modifications to support Intel® QuickAssist Technology (QAT) -hardware. It is designed to work with Kata Container versions 2.0 and higher. - -To properly load the driver modules, systemd init must be used. It is not adequate -to use the agent as the init. Because of this, alpine is not a valid base OS image -to use. The following rootfs OS's have been tested with this Dockerfile. - -* Clear Linux -* Debian -* Ubuntu - -The generated files will need to be copied and configured into your Kata Containers -setup. - -Please see the -[Using Intel® QuickAssist Technology and Kata](../../../../docs/use-cases/using-Intel-QAT-and-kata.md) -documentation for more specific details on how to configure a host system and -enable acceleration of workloads. - -## Building - -The image build and run are executed using Docker, from within this `QAT` folder. -It is required to use **all** the files in this directory to build the Docker -image: - -```sh -$ docker build --label kataqat --tag kataqat:latest . -$ mkdir ./output -$ docker run -ti --rm --privileged -v /dev:/dev -v $(pwd)/output:/output kataqat -``` - -> **Note:** The use of the `--privileged` and `-v /dev:/dev` arguments to the `docker run` are -> necessary, to enable the scripts within the container to generate a roofs file system. - -When complete, the generated files will be placed into the output directory. -Sample config files that have been modified with a `[SHIM`] section are also -placed into the `config` subdirectory as a reference that can be used with -Kata Containers. - -```sh -# ls -lR output -output: -total 136656 -drwxr-xr-x 2 root root 4096 Feb 11 23:59 configs --rw-r--r-- 1 root root 134217728 Feb 11 23:59 kata-containers.img --rw-r--r-- 1 root root 5710336 Feb 11 23:59 vmlinuz-kata-linux-5.4.71-84_qat - -output/configs: -total 20 --rw-r--r-- 1 root root 4082 Feb 11 23:59 200xxvf_dev0.conf --rw-r--r-- 1 root root 4082 Feb 11 23:59 c3xxxvf_dev0.conf --rw-r--r-- 1 root root 4082 Feb 11 23:59 c6xxvf_dev0.conf --rw-r--r-- 1 root root 4082 Feb 11 23:59 d15xxvf_dev0.conf --rw-r--r-- 1 root root 4082 Feb 11 23:59 dh895xccvf_dev0.conf -``` - -## Options - -A number of parameters to the scripts are configured in the `Dockerfile`, and thus can be modified -on the commandline. The `AGENT_VERSION` is not set and by default will use the -latest stable version of Kata Containers. - - -| Variable | Definition | Default value | -| -------- | ---------- | ------------- | -| `AGENT_VERSION` | Kata agent that is installed into the rootfs | | -| `KATA_REPO_VERSION` | Kata Branch or Tag to build from | `main` | -| `OUTPUT_DIR` | Directory inside container where results are stored | `/output` | -| `QAT_CONFIGURE_OPTIONS` | `configure` options for QAT driver | `--enable-icp-sriov=guest` | -| `QAT_DRIVER_URL` | URL to curl QAT driver from | `https://01.org/sites/default/files/downloads/${QAT_DRIVER_VER}` | -| `QAT_DRIVER_VER` | QAT driver version to use | `qat1.7.l.4.9.0-00008.tar.gz` | -| `ROOTFS_OS` | Operating system to use for the rootfs | `ubuntu` | - -Variables can be set on the `docker run` commandline, for example: - -```sh -$ docker run -ti --rm --privileged -e "AGENT_VERSION=2.0.0" -v /dev:/dev -v ${PWD}/output:/output kataqat -``` diff --git a/tools/osbuilder/dockerfiles/QAT/qat.conf b/tools/osbuilder/dockerfiles/QAT/qat.conf deleted file mode 100644 index dd9c9e9187..0000000000 --- a/tools/osbuilder/dockerfiles/QAT/qat.conf +++ /dev/null @@ -1,17 +0,0 @@ -# -# Copyright (c) 2021 Intel Corporation -# -# SPDX-License-Identifier: Apache-2.0 -# -CONFIG_PCIEAER=y -CONFIG_UIO=y -CONFIG_CRYPTO_HW=y -# This is a hack. By setting this QAT device as a module, we get the kernel -# to configure/build all the other parts required for QAT - and then later we -# build and load the out-of-tree QAT kernel modules instead of this one. -CONFIG_CRYPTO_DEV_QAT_C62XVF=m -CONFIG_CRYPTO_CBC=y -CONFIG_MODULES=y -CONFIG_MODULE_SIG=y -CONFIG_CRYPTO_AUTHENC=y -CONFIG_CRYPTO_DH=y diff --git a/tools/osbuilder/dockerfiles/QAT/run.sh b/tools/osbuilder/dockerfiles/QAT/run.sh deleted file mode 100755 index 539997e680..0000000000 --- a/tools/osbuilder/dockerfiles/QAT/run.sh +++ /dev/null @@ -1,174 +0,0 @@ -#!/usr/bin/env bash -# -# Copyright (c) 2021 Intel Corporation -# -# SPDX-License-Identifier: Apache-2.0 - -set -u - -# NOTE: Some env variables are set in the Dockerfile - those that are -# intended to be over-rideable. -export QAT_SRC=~/src/QAT -export ROOTFS_DIR=~/src/rootfs -export GOPATH=~/src/go -export PATH=${PATH}:/usr/local/go/bin:${GOPATH}/bin - -kata_repo=github.com/kata-containers/kata-containers -kata_repo_path=${GOPATH}/src/${kata_repo} - -grab_qat_drivers() -{ - /bin/echo -e "\n\e[1;42mDownload and extract the drivers\e[0m" - mkdir -p $QAT_SRC - cd $QAT_SRC - wget $QAT_DRIVER_URL - if [ ! -f ${QAT_SRC}/${QAT_DRIVER_VER} ];then - /bin/echo -e "\e[1;41mQAT Driver ${QAT_DRIVER_VER} doesn't exist\e[0m" - echo "Check https://01.org/intel-quickassist-technology to find the latest" - echo "QAT driver version, update the Dockerfile, and try again." - exit 1 - fi - tar xzf ${QAT_DRIVER_VER} -} - -grab_kata_repos() -{ - # Check out all the repos we will use now, so we can try and ensure they use the specified branch - # Only check out the branch needed, and make it shallow and thus space/bandwidth efficient - # Use a green prompt with white text for easy viewing - /bin/echo -e "\n\e[1;42mClone and checkout Kata repo\e[0m" - [ -d "${kata_repo_path}" ] || git clone --single-branch --branch $KATA_REPO_VERSION --depth=1 https://${kata_repo} ${kata_repo_path} -} - -configure_kernel() -{ - cp /input/qat.conf ${kata_repo_path}/tools/packaging/kernel/configs/fragments/common/qat.conf - # We need yq and go to grab kernel versions etc. - ${kata_repo_path}/ci/install_yq.sh - ${kata_repo_path}/tests/install_go.sh -p - cd ${kata_repo_path} - /bin/echo -e "\n\e[1;42mDownload and configure Kata kernel with CRYPTO support\e[0m" - ./tools/packaging/kernel/build-kernel.sh setup -} - -build_kernel() -{ - cd ${kata_repo_path} - LINUX_VER=$(ls -d kata-linux-*) - sed -i 's/EXTRAVERSION =/EXTRAVERSION = .qat.container/' $LINUX_VER/Makefile - /bin/echo -e "\n\e[1;42mBuild Kata kernel with CRYPTO support\e[0m" - ./tools/packaging/kernel/build-kernel.sh build -} - -build_rootfs() -{ - # Due to an issue with debootstrap unmounting /proc when running in a - # --privileged container, change into /proc to keep it from being umounted. - # This should only be done for Ubuntu and Debian based OS's. Other OS - # distributions had issues if building the rootfs from /proc - - if [ "${ROOTFS_OS}" == "ubuntu" ]; then - cd /proc - fi - /bin/echo -e "\n\e[1;42mDownload ${ROOTFS_OS} based rootfs\e[0m" - sudo -E SECCOMP=no EXTRA_PKGS='kmod' ${kata_repo_path}/tools/osbuilder/rootfs-builder/rootfs.sh $ROOTFS_OS -} - -build_qat_drivers() -{ - /bin/echo -e "\n\e[1;42mCompile driver modules\e[0m" - cd ${kata_repo_path} - linux_kernel_path=${kata_repo_path}/${LINUX_VER} - KERNEL_MAJOR_VERSION=$(awk '/^VERSION =/{print $NF}' ${linux_kernel_path}/Makefile) - KERNEL_PATHLEVEL=$(awk '/^PATCHLEVEL =/{print $NF}' ${linux_kernel_path}/Makefile) - KERNEL_SUBLEVEL=$(awk '/^SUBLEVEL =/{print $NF}' ${linux_kernel_path}/Makefile) - KERNEL_EXTRAVERSION=$(awk '/^EXTRAVERSION =/{print $NF}' ${linux_kernel_path}/Makefile) - KERNEL_ROOTFS_DIR=${KERNEL_MAJOR_VERSION}.${KERNEL_PATHLEVEL}.${KERNEL_SUBLEVEL}${KERNEL_EXTRAVERSION} - cd $QAT_SRC - KERNEL_SOURCE_ROOT=${linux_kernel_path} ./configure ${QAT_CONFIGURE_OPTIONS} - make all -j $(nproc) -} - -add_qat_to_rootfs() -{ - /bin/echo -e "\n\e[1;42mCopy driver modules to rootfs\e[0m" - cd $QAT_SRC - sudo -E make INSTALL_MOD_PATH=${ROOTFS_DIR} qat-driver-install -j$(nproc --ignore=1) - sudo cp $QAT_SRC/build/usdm_drv.ko ${ROOTFS_DIR}/lib/modules/${KERNEL_ROOTFS_DIR}/updates/drivers - sudo depmod -a -b ${ROOTFS_DIR} ${KERNEL_ROOTFS_DIR} - cd ${kata_repo_path}/tools/osbuilder/image-builder - /bin/echo -e "\n\e[1;42mBuild rootfs image\e[0m" - sudo -E ./image_builder.sh ${ROOTFS_DIR} -} - -copy_outputs() -{ - /bin/echo -e "\n\e[1;42mCopy kernel and rootfs to the output directory and provide sample configuration files\e[0m" - mkdir -p ${OUTPUT_DIR} || true - sudo cp ${linux_kernel_path}/arch/x86/boot/bzImage $OUTPUT_DIR/vmlinuz-${LINUX_VER}_qat - sudo cp ${linux_kernel_path}/vmlinux $OUTPUT_DIR/vmlinux-${LINUX_VER}_qat - sudo cp ${kata_repo_path}/tools/osbuilder/image-builder/kata-containers.img $OUTPUT_DIR - sudo mkdir -p ${OUTPUT_DIR}/configs || true - # Change extension from .conf.vm to just .conf and change the SSL section to - # SHIM so it works with Kata containers - for f in $QAT_SRC/quickassist/utilities/adf_ctl/conf_files/*.conf.vm; do - output_conf_file=$(basename -- "$f" .conf.vm).conf - sudo cp -- "$f" "${OUTPUT_DIR}/configs/${output_conf_file}" - sudo sed -i 's/\[SSL\]/\[SHIM\]/g' ${OUTPUT_DIR}/configs/${output_conf_file} - done - /bin/echo -e "Check the ./output directory for the kernel and rootfs\n" -} - -help() { -cat << EOF -Usage: $0 [-h] [options] - Description: - This script builds kernel and rootfs artifacts for Kata Containers, - configured and built to support QAT hardware. - Options: - -d, Enable debug mode - -h, Show this help -EOF -} - -main() -{ - local check_in_container=${OUTPUT_DIR:-} - if [ -z "${check_in_container}" ]; then - echo "Error: 'OUTPUT_DIR' not set" >&2 - echo "$0 should be run using the Dockerfile supplied." >&2 - exit 1 - fi - - local OPTIND - while getopts "dh" opt;do - case ${opt} in - d) - set -x - ;; - h) - help - exit 0; - ;; - ?) - # parse failure - help - echo "ERROR: Failed to parse arguments" - exit 1 - ;; - esac - done - shift $((OPTIND-1)) - - sudo chown -R qatbuilder:qatbuilder /home/qatbuilder - grab_qat_drivers - grab_kata_repos - configure_kernel - build_kernel - build_rootfs - build_qat_drivers - add_qat_to_rootfs - copy_outputs -} - -main "$@"