mirror of
https://github.com/kata-containers/kata-containers.git
synced 2025-07-19 18:01:01 +00:00
subsystem: docs
Updated QAT instructions This fixes #516 Signed-off-by: Eric Adams <eric.adams@intel.com>
This commit is contained in:
Eric Adams
parent
44f67f7efc
commit
b4704fec3e
@ -8,11 +8,8 @@
|
||||
* [Set environment variables (Every Reboot)](#set-environment-variables-every-reboot)
|
||||
* [Prepare the Clear Linux Host](#prepare-the-clear-linux-host)
|
||||
* [Identify which PCI Bus the Intel QAT card is on](#identify-which-pci-bus-the-intel-qat-card-is-on)
|
||||
* [Install necessary bundles for Clear Linux and disable kernel updates](#install-necessary-bundles-for-clear-linux-and-disable-kernel-updates)
|
||||
* [Allow insecure modules to load to host](#allow-insecure-modules-to-load-to-host)
|
||||
* [Replace Clear Linux kernel with LTS kernel](#replace-clear-linux-kernel-with-lts-kernel)
|
||||
* [Install necessary bundles for Clear Linux](#install-necessary-bundles-for-clear-linux)
|
||||
* [Download Intel QAT drivers](#download-intel-qat-drivers)
|
||||
* [Compile Intel QAT drivers against host kernel](#compile-intel-qat-drivers-against-host-kernel)
|
||||
* [Copy Intel QAT configuration files and enable Virtual Functions](#copy-intel-qat-configuration-files-and-enable-virtual-functions)
|
||||
* [Expose and Bind Intel QAT virtual functions to VFIO-PCI (Every reboot)](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot)
|
||||
* [Check Intel QAT virtual functions are enabled](#check-intel-qat-virtual-functions-are-enabled)
|
||||
@ -31,7 +28,6 @@
|
||||
* [Optional Scripts](#optional-scripts)
|
||||
* [Verify Intel QAT card counters are incremented](#verify-intel-qat-card-counters-are-incremented)
|
||||
|
||||
|
||||
# Introduction
|
||||
|
||||
Intel QuickAssist Technology (Intel QAT) provides hardware acceleration
|
||||
@ -97,38 +93,17 @@ chipset by executing the following.
|
||||
$ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done
|
||||
```
|
||||
|
||||
### Install necessary bundles for Clear Linux and disable kernel updates
|
||||
### Install necessary bundles for Clear Linux
|
||||
|
||||
You need the Linux kernel headers to compile the out-of-tree kernel modules to
|
||||
support Intel QAT on the host. The `os-clr-on-clr` is a large development
|
||||
bundle that you can install instead to include everything below with
|
||||
additional items. This guide installs the Linux LTS kernel version `4.19` from
|
||||
2018 because the crypto modules are set to `m` and not `y`, which allows you
|
||||
to replace them with the out-of-tree driver. Otherwise you need to build your
|
||||
own Clear Linux kernel with the QAT drivers compiled in.
|
||||
Clear Linux version 30780 (Released August 13, 2019) includes a
|
||||
`linux-firmware-qat` bundle that has the necessary QAT firmware along with a
|
||||
functional QAT host driver that works with Kata Containers.
|
||||
|
||||
```sh
|
||||
$ sudo swupd autoupdate --disable
|
||||
$ sudo swupd bundle-add network-basic kernel-lts2018 linux-lts-dev linux-dev make c-basic go-basic containers-virt dev-utils devpkg-elfutils devpkg-systemd
|
||||
$ sudo swupd bundle-add network-basic linux-firmware-qat make c-basic go-basic containers-virt dev-utils devpkg-elfutils devpkg-systemd
|
||||
$ sudo clr-boot-manager update
|
||||
$ sudo systemctl start docker
|
||||
$ sudo systemctl enable docker
|
||||
```
|
||||
|
||||
### Allow insecure modules to load to host
|
||||
|
||||
```sh
|
||||
$ sudo mkdir -p /etc/kernel/cmdline.d
|
||||
$ cat << EOF | sudo tee /etc/kernel/cmdline.d/load-modules.conf
|
||||
module.sig_unenforce
|
||||
EOF
|
||||
$ sudo clr-boot-manager update
|
||||
```
|
||||
|
||||
### Replace Clear Linux kernel with LTS kernel
|
||||
|
||||
```sh
|
||||
$ lts_kernel=$(sudo clr-boot-manager list-kernels | grep lts2018 | head -n 1)
|
||||
$ sudo clr-boot-manager set-kernel $lts_kernel
|
||||
$ sudo reboot
|
||||
```
|
||||
|
||||
@ -140,31 +115,13 @@ Make sure to check the website for the latest version.
|
||||
```sh
|
||||
$ mkdir -p $QAT_SRC
|
||||
$ cd $QAT_SRC
|
||||
$ curl -O $QAT_DRIVER_URL
|
||||
$ QAT_DRIVER_NAME=$(echo $QAT_DRIVER_URL | rev | cut -d"/" -f1 | rev)
|
||||
$ tar xzvf $QAT_DRIVER_NAME
|
||||
```
|
||||
|
||||
### Compile Intel QAT drivers against host kernel
|
||||
|
||||
Now, the Linux kernel headers bundle `linux-dev` is present. This replaces any
|
||||
existing modules provided by Clear Linux. If the Clear Linux kernel updates,
|
||||
you must rebuild and reinstall these modules against the kernel. By doing it
|
||||
this way you no longer have to rely on your distribution to include the QAT
|
||||
firmware. These modules will load at every reboot and add the `qat_service`
|
||||
helper script.
|
||||
|
||||
|
||||
```sh
|
||||
$ cd $QAT_SRC
|
||||
$ ./configure --disable-qat-lkcf
|
||||
$ sudo -E make install -j$(nproc)
|
||||
$ sudo chown -R $USER:$USER $QAT_SRC
|
||||
$ curl -L $QAT_DRIVER_URL | tar zx
|
||||
```
|
||||
|
||||
### Copy Intel QAT configuration files and enable Virtual Functions
|
||||
|
||||
Learn more about customizing configuration files at the
|
||||
Modify the instructions below as necessary if using a different QAT hardware
|
||||
platform. You can learn more about customizing configuration files at the
|
||||
[Intel QAT Engine repository](https://github.com/intel/QAT_Engine/#copy-the-correct-intel-quickassist-technology-driver-config-files)
|
||||
This section starts from a base config file and changes the `SSL` section to
|
||||
`SHIM` to support the OpenSSL engine. There are more tweaks that you can make
|
||||
@ -172,11 +129,12 @@ depending on the use case and how many Intel QAT engines should be run. You
|
||||
can find more information about how to customize in the
|
||||
[Intel® QuickAssist Technology Software for Linux* - Programmer's Guide.](https://01.org/sites/default/files/downloads/336210qatswprogrammersguiderev006.pdf)
|
||||
|
||||
> **Note: This section assumes that a QAT `c6xx` platform is used.**
|
||||
|
||||
```sh
|
||||
$ mkdir -p $QAT_CONF_LOCATION
|
||||
$ cp $QAT_SRC/build/c6xxvf_dev0.conf.vm $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
$ cp $QAT_SRC/quickassist/utilities/adf_ctl/conf_files/c6xxvf_dev0.conf.vm $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
$ sed -i 's/\[SSL\]/\[SHIM\]/g' $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
$ for i in {1..15}; do cp $QAT_CONF_LOCATION/c6xxvf_dev0.conf $QAT_CONF_LOCATION/c6xxvf_dev$i.conf; done
|
||||
```
|
||||
|
||||
### Expose and Bind Intel QAT virtual functions to VFIO-PCI (Every reboot)
|
||||
@ -196,7 +154,7 @@ enabled.
|
||||
|
||||
```sh
|
||||
$ sudo modprobe vfio-pci
|
||||
$ QAT_PCI_BUS_PF_NUMBERS=$(lspci | grep Quick | cut -d ' ' -f 1)
|
||||
$ QAT_PCI_BUS_PF_NUMBERS=$((lspci -d :435 && lspci -d :37c8 && lspci -d :19e2 && lspci -d :6f54) | cut -d ' ' -f 1)
|
||||
$ QAT_PCI_BUS_PF_1=$(echo $QAT_PCI_BUS_PF_NUMBERS | cut -d ' ' -f 1)
|
||||
$ echo 16 | sudo tee /sys/bus/pci/devices/0000:$QAT_PCI_BUS_PF_1/sriov_numvfs
|
||||
$ QAT_PCI_ID_VF=$(cat /sys/bus/pci/devices/0000:${QAT_PCI_BUS_PF_1}/virtfn0/uevent | grep PCI_ID)
|
||||
@ -244,15 +202,22 @@ There are some patches that must be installed as well, which the
|
||||
`build-kernel.sh` script should automatically apply. If you are using a
|
||||
different kernel version, then you might need to manually apply them. Since
|
||||
the Kata Containers kernel has a minimal set of kernel flags set, you must
|
||||
update the config to set some of the `CRYPTO` and `STACKPROTECTOR` flags to
|
||||
enabled. This might change with different kernel versions. We tested the
|
||||
following instructions with kernel `v4.19.28-41`.
|
||||
|
||||
create a QAT kernel fragment with the necessary `CONFIG_CRYPTO_*` options set.
|
||||
Update the config to set some of the `CRYPTO` flags to enabled. This might
|
||||
change with different kernel versions. We tested the following instructions
|
||||
with kernel `v4.19.28-41`.
|
||||
|
||||
```sh
|
||||
$ mkdir -p $GOPATH
|
||||
$ cd $GOPATH
|
||||
$ go get -v github.com/kata-containers/packaging
|
||||
$ cat << EOF > $GOPATH/src/github.com/kata-containers/packaging/kernel/configs/fragments/common/qat.conf
|
||||
CONFIG_PCIEAER=y
|
||||
CONFIG_UIO=y
|
||||
CONFIG_CRYPTO_HW=y
|
||||
CONFIG_CRYPTO_DEV_QAT_C62XVF=m
|
||||
CONFIG_CRYPTO_CBC=y
|
||||
EOF
|
||||
$ $GOPATH/src/github.com/kata-containers/packaging/kernel/build-kernel.sh setup
|
||||
```
|
||||
|
||||
@ -261,18 +226,9 @@ $ $GOPATH/src/github.com/kata-containers/packaging/kernel/build-kernel.sh setup
|
||||
```sh
|
||||
$ export LINUX_VER=$(ls -d kata*)
|
||||
$ sed -i 's/EXTRAVERSION =/EXTRAVERSION = .qat.container/' $LINUX_VER/Makefile
|
||||
```
|
||||
Change kernel config options to support QAT and compile the kernel.
|
||||
```sh
|
||||
$ sed -i 's/# CONFIG_PCI_IOV is not set/CONFIG_PCI_IOV=Y/' $LINUX_VER/.config
|
||||
$ sed -i 's/# CONFIG_CRYPTO_SHA1 is not set/CONFIG_CRYPTO_SHA1=Y/' $LINUX_VER/.config
|
||||
$ sed -i 's/# CONFIG_CRYPTO_DH is not set/CONFIG_CRYPTO_DH=Y/' $LINUX_VER/.config
|
||||
$ sed -i 's/# CONFIG_STACKPROTECTOR is not set/CONFIG_STACKPROTECTOR=Y/' $LINUX_VER/.config
|
||||
$ $GOPATH/src/github.com/kata-containers/packaging/kernel/build-kernel.sh build
|
||||
```
|
||||
|
||||
> **Note: There may be additional kernel parameters that need to be set to yes
|
||||
> when compiling.**
|
||||
|
||||
### Copy Kata kernel
|
||||
|
||||
@ -399,7 +355,7 @@ It might require you to add an `IPC_LOCK` capability to your Docker runtime
|
||||
depending on which rootfs you use.
|
||||
|
||||
```sh
|
||||
$ sudo docker run -it --runtime=kata-runtime --cap-add=IPC_LOCK --device=/dev/vfio/90 -v /dev:/dev -v ${QAT_CONF_LOCATION}:/etc openssl-qat-engine bash
|
||||
$ sudo docker run -it --runtime=kata-runtime --cap-add=IPC_LOCK --cap-add=SYS_ADMIN --device=/dev/vfio/90 -v /dev:/dev -v ${QAT_CONF_LOCATION}:/etc openssl-qat-engine bash
|
||||
```
|
||||
|
||||
Below are some commands to run in the container image to verify Intel QAT is
|
||||
|
||||
Loading…
Reference in New Issue
Block a user