mirror of
https://github.com/kata-containers/kata-containers.git
synced 2025-06-26 15:32:30 +00:00
docs: Update QAT instructions to work with Kata 2.0 repos
This fixes the guide to work with the Kata 2.0 repos and provide more details on how to verify with ctr and kubernetes. Fixes: #1362 Signed-off-by: Adams, Eric <eric.adams@intel.com>
This commit is contained in:
Adams, Eric
parent
9a02c81eb9
commit
259c179162
@ -1,56 +1,62 @@
|
||||
# Table of Contents
|
||||
|
||||
* [Table of Contents](#table-of-contents)
|
||||
* [Introduction](#introduction)
|
||||
* [Helpful Links before starting](#helpful-links-before-starting)
|
||||
* [Steps to enable Intel QAT in Kata Containers](#steps-to-enable-intel-qat-in-kata-containers)
|
||||
* [Script variables](#script-variables)
|
||||
* [Set environment variables (Every Reboot)](#set-environment-variables-every-reboot)
|
||||
* [Prepare the Clear Linux Host](#prepare-the-clear-linux-host)
|
||||
* [Identify which PCI Bus the Intel QAT card is on](#identify-which-pci-bus-the-intel-qat-card-is-on)
|
||||
* [Install necessary bundles for Clear Linux](#install-necessary-bundles-for-clear-linux)
|
||||
* [Download Intel QAT drivers](#download-intel-qat-drivers)
|
||||
* [Copy Intel QAT configuration files and enable Virtual Functions](#copy-intel-qat-configuration-files-and-enable-virtual-functions)
|
||||
* [Expose and Bind Intel QAT virtual functions to VFIO-PCI (Every reboot)](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot)
|
||||
* [Check Intel QAT virtual functions are enabled](#check-intel-qat-virtual-functions-are-enabled)
|
||||
* [Prepare Kata Containers](#prepare-kata-containers)
|
||||
* [Download Kata kernel Source](#download-kata-kernel-source)
|
||||
* [Build Kata kernel](#build-kata-kernel)
|
||||
* [Copy Kata kernel](#copy-kata-kernel)
|
||||
* [Prepare Kata root filesystem](#prepare-kata-root-filesystem)
|
||||
* [Compile Intel QAT drivers for Kata Containers kernel and add to Kata Containers rootfs](#compile-intel-qat-drivers-for-kata-containers-kernel-and-add-to-kata-containers-rootfs)
|
||||
* [Copy Kata rootfs](#copy-kata-rootfs)
|
||||
* [Update Kata configuration to point to custom kernel and rootfs](#update-kata-configuration-to-point-to-custom-kernel-and-rootfs)
|
||||
* [Verify Intel QAT works in a Docker Kata Containers container](#verify-intel-qat-works-in-a-docker-kata-containers-container)
|
||||
* [Build OpenSSL Intel QAT engine container](#build-openssl-intel-qat-engine-container)
|
||||
* [Test Intel QAT in Docker](#test-intel-qat-in-docker)
|
||||
* [Troubleshooting](#troubleshooting)
|
||||
* [Optional Scripts](#optional-scripts)
|
||||
* [Verify Intel QAT card counters are incremented](#verify-intel-qat-card-counters-are-incremented)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [Introduction](#introduction)
|
||||
- [Helpful Links before starting](#helpful-links-before-starting)
|
||||
- [Steps to enable Intel® QAT in Kata Containers](#steps-to-enable-intel-qat-in-kata-containers)
|
||||
- [Script variables](#script-variables)
|
||||
- [Set environment variables (Every Reboot)](#set-environment-variables-every-reboot)
|
||||
- [Prepare the Ubuntu Host](#prepare-the-ubuntu-host)
|
||||
- [Identify which PCI Bus the Intel® QAT card is on](#identify-which-pci-bus-the-intel-qat-card-is-on)
|
||||
- [Install necessary packages for Ubuntu](#install-necessary-packages-for-ubuntu)
|
||||
- [Download Intel® QAT drivers](#download-intel-qat-drivers)
|
||||
- [Copy Intel® QAT configuration files and enable virtual functions](#copy-intel-qat-configuration-files-and-enable-virtual-functions)
|
||||
- [Expose and Bind Intel® QAT virtual functions to VFIO-PCI (Every reboot)](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot)
|
||||
- [Check Intel® QAT virtual functions are enabled](#check-intel-qat-virtual-functions-are-enabled)
|
||||
- [Prepare Kata Containers](#prepare-kata-containers)
|
||||
- [Download Kata kernel Source](#download-kata-kernel-source)
|
||||
- [Build Kata kernel](#build-kata-kernel)
|
||||
- [Copy Kata kernel](#copy-kata-kernel)
|
||||
- [Prepare Kata root filesystem](#prepare-kata-root-filesystem)
|
||||
- [Compile Intel® QAT drivers for Kata Containers kernel and add to Kata Containers rootfs](#compile-intel-qat-drivers-for-kata-containers-kernel-and-add-to-kata-containers-rootfs)
|
||||
- [Copy Kata rootfs](#copy-kata-rootfs)
|
||||
- [Verify Intel® QAT works in a container](#verify-intel-qat-works-in-a-container)
|
||||
- [Build OpenSSL Intel® QAT engine container](#build-openssl-intel-qat-engine-container)
|
||||
- [Test Intel® QAT with the ctr tool](#test-intel-qat-with-the-ctr-tool)
|
||||
- [Test Intel® QAT in Kubernetes](#test-intel-qat-in-kubernetes)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Optional Scripts](#optional-scripts)
|
||||
- [Verify Intel® QAT card counters are incremented](#verify-intel-qat-card-counters-are-incremented)
|
||||
|
||||
# Introduction
|
||||
|
||||
Intel QuickAssist Technology (Intel QAT) provides hardware acceleration
|
||||
Intel® QuickAssist Technology (QAT) provides hardware acceleration
|
||||
for security (cryptography) and compression. These instructions cover the
|
||||
steps for [Clear Linux](https://clearlinux.org) but can be adapted to any
|
||||
Linux distribution. Your distribution may already have the Intel QAT
|
||||
drivers, but it is likely they do not contain the necessary user space
|
||||
components. These instructions guide the user on how to download the kernel
|
||||
sources, compile kernel driver modules against those sources, and load them
|
||||
onto the host as well as preparing a specially built Kata Containers kernel
|
||||
and custom Kata Containers rootfs.
|
||||
steps for the latest [Ubuntu LTS release](https://ubuntu.com/download/desktop)
|
||||
which already include the QAT host driver. These instructions can be adapted to
|
||||
any Linux distribution. These instructions guide the user on how to download
|
||||
the kernel sources, compile kernel driver modules against those sources, and
|
||||
load them onto the host as well as preparing a specially built Kata Containers
|
||||
kernel and custom Kata Containers rootfs.
|
||||
|
||||
* Download kernel sources
|
||||
* Compile Kata kernel
|
||||
* Compile kernel driver modules against those sources
|
||||
* Download rootfs
|
||||
* Add driver modules to rootfs
|
||||
* Build rootfs image
|
||||
|
||||
## Helpful Links before starting
|
||||
|
||||
[Intel QAT Engine](https://github.com/intel/QAT_Engine)
|
||||
[Intel® QuickAssist Technology at `01.org`](https://01.org/intel-quickassist-technology)
|
||||
|
||||
[Intel QuickAssist Technology at `01.org`](https://01.org/intel-quickassist-technology)
|
||||
[Intel® QuickAssist Technology Engine for OpenSSL](https://github.com/intel/QAT_Engine)
|
||||
|
||||
[Intel Device Plugin for Kubernetes](https://github.com/intel/intel-device-plugins-for-kubernetes)
|
||||
|
||||
[Intel QuickAssist Crypto Poll Mode Driver](https://dpdk-docs.readthedocs.io/en/latest/cryptodevs/qat.html)
|
||||
[Intel® QuickAssist Technology for Crypto Poll Mode Driver](https://dpdk-docs.readthedocs.io/en/latest/cryptodevs/qat.html)
|
||||
|
||||
## Steps to enable Intel QAT in Kata Containers
|
||||
## Steps to enable Intel® QAT in Kata Containers
|
||||
|
||||
There are some steps to complete only once, some steps to complete with every
|
||||
reboot, and some steps to complete when the host kernel changes.
|
||||
@ -67,91 +73,95 @@ needed to point to updated drivers or different install locations.
|
||||
Make sure to check [`01.org`](https://01.org/intel-quickassist-technology) for
|
||||
the latest driver.
|
||||
|
||||
```sh
|
||||
$ export QAT_DRIVER_VER=qat1.7.l.4.8.0-00005.tar.gz
|
||||
$ export QAT_DRIVER_URL=https://01.org/sites/default/files/downloads/${QAT_DRIVER_VER}
|
||||
```bash
|
||||
$ export QAT_DRIVER_VER=qat1.7.l.4.12.0-00011.tar.gz
|
||||
$ export QAT_DRIVER_URL=https://downloadmirror.intel.com/30178/eng/${QAT_DRIVER_VER}
|
||||
$ export QAT_CONF_LOCATION=~/QAT_conf
|
||||
$ export QAT_DOCKERFILE=https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/master/demo/openssl-qat-engine/Dockerfile
|
||||
$ export QAT_SRC=~/src/QAT
|
||||
$ export GOPATH=~/src/go
|
||||
$ export OSBUILDER=~/src/osbuilder
|
||||
$ export KATA_KERNEL_LOCATION=~/kata
|
||||
$ export KATA_ROOTFS_LOCATION=~/kata
|
||||
```
|
||||
|
||||
## Prepare the Clear Linux Host
|
||||
## Prepare the Ubuntu Host
|
||||
|
||||
The host could be a bare metal instance or a virtual machine. If using a
|
||||
virtual machine, make sure that KVM nesting is enabled. The following
|
||||
instructions reference an Intel QAT. Some of the instructions must be
|
||||
modified if using a different Intel QAT device. You can identify the Intel QAT
|
||||
chipset by executing the following.
|
||||
instructions reference an Intel® C62X chipset. Some of the instructions must be
|
||||
modified if using a different Intel® QAT device. The Intel® QAT chipset can be
|
||||
identified by executing the following.
|
||||
|
||||
### Identify which PCI Bus the Intel QAT card is on
|
||||
### Identify which PCI Bus the Intel® QAT card is on
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done
|
||||
```
|
||||
|
||||
### Install necessary bundles for Clear Linux
|
||||
### Install necessary packages for Ubuntu
|
||||
|
||||
Clear Linux version 30780 (Released August 13, 2019) includes a
|
||||
`linux-firmware-qat` bundle that has the necessary QAT firmware along with a
|
||||
functional QAT host driver that works with Kata Containers.
|
||||
These packages are necessary to compile the Kata kernel, Intel® QAT driver, and to
|
||||
prepare the rootfs for Kata. [Docker](https://docs.docker.com/engine/install/ubuntu/)
|
||||
also needs to be installed to be able to build the rootfs. To test that
|
||||
everything works a Kubernetes pod is started requesting Intel® QAT resources. For the
|
||||
pass through of the virtual functions the kernel boot parameter needs to have
|
||||
`INTEL_IOMMU=on`.
|
||||
|
||||
```sh
|
||||
$ sudo swupd bundle-add network-basic linux-firmware-qat make c-basic go-basic containers-virt dev-utils devpkg-elfutils devpkg-systemd devpkg-ssl
|
||||
$ sudo clr-boot-manager update
|
||||
$ sudo systemctl enable --now docker
|
||||
```bash
|
||||
$ sudo apt update
|
||||
$ sudo apt install -y golang-go build-essential python pkg-config zlib1g-dev libudev-dev bison libelf-dev flex libtool automake autotools-dev autoconf bc libpixman-1-dev coreutils libssl-dev
|
||||
$ sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=""/GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"/' /etc/default/grub
|
||||
$ sudo update-grub
|
||||
$ sudo reboot
|
||||
```
|
||||
|
||||
### Download Intel QAT drivers
|
||||
### Download Intel® QAT drivers
|
||||
|
||||
This will download the Intel QAT drivers from [`01.org`](https://01.org/intel-quickassist-technology).
|
||||
This will download the [Intel® QAT drivers](https://01.org/intel-quickassist-technology).
|
||||
Make sure to check the website for the latest version.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $QAT_SRC
|
||||
$ cd $QAT_SRC
|
||||
$ curl -L $QAT_DRIVER_URL | tar zx
|
||||
```
|
||||
|
||||
### Copy Intel QAT configuration files and enable Virtual Functions
|
||||
### Copy Intel® QAT configuration files and enable virtual functions
|
||||
|
||||
Modify the instructions below as necessary if using a different QAT hardware
|
||||
Modify the instructions below as necessary if using a different Intel® QAT hardware
|
||||
platform. You can learn more about customizing configuration files at the
|
||||
[Intel QAT Engine repository](https://github.com/intel/QAT_Engine/#copy-the-correct-intel-quickassist-technology-driver-config-files)
|
||||
[Intel® QAT Engine repository](https://github.com/intel/QAT_Engine/#copy-the-correct-intel-quickassist-technology-driver-config-files)
|
||||
This section starts from a base config file and changes the `SSL` section to
|
||||
`SHIM` to support the OpenSSL engine. There are more tweaks that you can make
|
||||
depending on the use case and how many Intel QAT engines should be run. You
|
||||
depending on the use case and how many Intel® QAT engines should be run. You
|
||||
can find more information about how to customize in the
|
||||
[Intel® QuickAssist Technology Software for Linux* - Programmer's Guide.](https://01.org/sites/default/files/downloads/336210qatswprogrammersguiderev006.pdf)
|
||||
|
||||
> **Note: This section assumes that a QAT `c6xx` platform is used.**
|
||||
> **Note: This section assumes that a Intel® QAT `c6xx` platform is used.**
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $QAT_CONF_LOCATION
|
||||
$ cp $QAT_SRC/quickassist/utilities/adf_ctl/conf_files/c6xxvf_dev0.conf.vm $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
$ sed -i 's/\[SSL\]/\[SHIM\]/g' $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
```
|
||||
|
||||
### Expose and Bind Intel QAT virtual functions to VFIO-PCI (Every reboot)
|
||||
### Expose and Bind Intel® QAT virtual functions to VFIO-PCI (Every reboot)
|
||||
|
||||
To enable virtual functions, the host OS should have IOMMU groups enabled. In
|
||||
the UEFI Firmware Intel Virtualization Technology for Directed I/O
|
||||
(Intel VT-d) must be enabled. Also, the kernel boot parameter should be
|
||||
`intel_iommu=on` or `intel_iommu=ifgx_off`. The default in Clear Linux currently
|
||||
is `intel_iommu=igfx_off` which should work with the Intel QAT device. The
|
||||
following commands assume you installed an Intel QAT card, IOMMU is on, and
|
||||
the UEFI Firmware Intel® Virtualization Technology for Directed I/O
|
||||
(Intel® VT-d) must be enabled. Also, the kernel boot parameter should be
|
||||
`intel_iommu=on` or `intel_iommu=ifgx_off`. This should have been set from
|
||||
the instructions above. Check the output of `/proc/cmdline` to confirm. The
|
||||
following commands assume you installed an Intel® QAT card, IOMMU is on, and
|
||||
VT-d is enabled. The vendor and device ID add to the `VFIO-PCI` driver so that
|
||||
each exposed virtual function can be bound to the `VFIO-PCI` driver. Once
|
||||
complete, each virtual function passes into a Kata Containers container using
|
||||
the PCIe device passthrough feature. For Kubernetes, the Intel device plugin
|
||||
for Kubernetes handles the binding of the driver but the VF’s still must be
|
||||
the PCIe device passthrough feature. For Kubernetes, the
|
||||
[Intel device plugin](https://github.com/intel/intel-device-plugins-for-kubernetes)
|
||||
for Kubernetes handles the binding of the driver, but the VF’s still must be
|
||||
enabled.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ sudo modprobe vfio-pci
|
||||
$ QAT_PCI_BUS_PF_NUMBERS=$((lspci -d :435 && lspci -d :37c8 && lspci -d :19e2 && lspci -d :6f54) | cut -d ' ' -f 1)
|
||||
$ QAT_PCI_BUS_PF_1=$(echo $QAT_PCI_BUS_PF_NUMBERS | cut -d ' ' -f 1)
|
||||
@ -160,8 +170,10 @@ $ QAT_PCI_ID_VF=$(cat /sys/bus/pci/devices/0000:${QAT_PCI_BUS_PF_1}/virtfn0/ueve
|
||||
$ QAT_VENDOR_AND_ID_VF=$(echo ${QAT_PCI_ID_VF/PCI_ID=} | sed 's/:/ /')
|
||||
$ echo $QAT_VENDOR_AND_ID_VF | sudo tee --append /sys/bus/pci/drivers/vfio-pci/new_id
|
||||
```
|
||||
|
||||
Loop through all the virtual functions and bind to the VFIO driver
|
||||
```sh
|
||||
|
||||
```bash
|
||||
$ for f in /sys/bus/pci/devices/0000:$QAT_PCI_BUS_PF_1/virtfn*
|
||||
do QAT_PCI_BUS_VF=$(basename $(readlink $f))
|
||||
echo $QAT_PCI_BUS_VF | sudo tee --append /sys/bus/pci/drivers/c6xxvf/unbind
|
||||
@ -169,22 +181,23 @@ $ for f in /sys/bus/pci/devices/0000:$QAT_PCI_BUS_PF_1/virtfn*
|
||||
done
|
||||
```
|
||||
|
||||
### Check Intel QAT virtual functions are enabled
|
||||
### Check Intel® QAT virtual functions are enabled
|
||||
|
||||
If the following command returns empty, then the virtual functions are not
|
||||
properly enabled. This command checks the enumerated device IDs for just the
|
||||
virtual functions. Using the Intel QAT as an example, the physical device ID
|
||||
virtual functions. Using the Intel® QAT as an example, the physical device ID
|
||||
is `37c8` and virtual function device ID is `37c9`. The following command checks
|
||||
if VF's are enabled for any of the currently known Intel QAT device ID's. The
|
||||
if VF's are enabled for any of the currently known Intel® QAT device ID's. The
|
||||
following `ls` command should show the 16 VF's bound to `VFIO-PCI`.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ for i in 0442 0443 37c9 19e3; do lspci -d 8086:$i; done
|
||||
```
|
||||
|
||||
Another way to check is to see what PCI devices that `VFIO-PCI` is mapped to.
|
||||
It should match the device ID's of the VF's.
|
||||
```sh
|
||||
|
||||
```bash
|
||||
$ ls -la /sys/bus/pci/drivers/vfio-pci
|
||||
```
|
||||
|
||||
@ -201,16 +214,16 @@ There are some patches that must be installed as well, which the
|
||||
`build-kernel.sh` script should automatically apply. If you are using a
|
||||
different kernel version, then you might need to manually apply them. Since
|
||||
the Kata Containers kernel has a minimal set of kernel flags set, you must
|
||||
create a QAT kernel fragment with the necessary `CONFIG_CRYPTO_*` options set.
|
||||
create a Intel® QAT kernel fragment with the necessary `CONFIG_CRYPTO_*` options set.
|
||||
Update the config to set some of the `CRYPTO` flags to enabled. This might
|
||||
change with different kernel versions. We tested the following instructions
|
||||
with kernel `v4.19.28-41`.
|
||||
change with different kernel versions. The following instructions were tested
|
||||
with kernel `v5.4.0-64-generic`.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $GOPATH
|
||||
$ cd $GOPATH
|
||||
$ go get -v github.com/kata-containers/packaging
|
||||
$ cat << EOF > $GOPATH/src/github.com/kata-containers/packaging/kernel/configs/fragments/common/qat.conf
|
||||
$ go get -v github.com/kata-containers/kata-containers
|
||||
$ cat << EOF > $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/configs/fragments/common/qat.conf
|
||||
CONFIG_PCIEAER=y
|
||||
CONFIG_UIO=y
|
||||
CONFIG_CRYPTO_HW=y
|
||||
@ -221,61 +234,70 @@ CONFIG_MODULE_SIG=y
|
||||
CONFIG_CRYPTO_AUTHENC=y
|
||||
CONFIG_CRYPTO_DH=y
|
||||
EOF
|
||||
$ $GOPATH/src/github.com/kata-containers/packaging/kernel/build-kernel.sh setup
|
||||
$ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh setup
|
||||
```
|
||||
|
||||
### Build Kata kernel
|
||||
|
||||
```sh
|
||||
$ export LINUX_VER=$(ls -d kata*)
|
||||
```bash
|
||||
$ cd $GOPATH
|
||||
$ export LINUX_VER=$(ls -d kata-linux-*)
|
||||
$ sed -i 's/EXTRAVERSION =/EXTRAVERSION = .qat.container/' $LINUX_VER/Makefile
|
||||
$ $GOPATH/src/github.com/kata-containers/packaging/kernel/build-kernel.sh build
|
||||
$ $GOPATH/src/github.com/kata-containers/kata-containers/tools/packaging/kernel/build-kernel.sh build
|
||||
```
|
||||
|
||||
|
||||
### Copy Kata kernel
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ export KATA_KERNEL_NAME=vmlinux-${LINUX_VER}_qat
|
||||
$ mkdir -p $KATA_KERNEL_LOCATION
|
||||
$ cp $LINUX_VER/arch/x86/boot/bzImage $KATA_KERNEL_LOCATION/vmlinuz-${LINUX_VER}_qat
|
||||
$ cp ${GOPATH}/${LINUX_VER}/vmlinux ${KATA_KERNEL_LOCATION}/${KATA_KERNEL_NAME}
|
||||
```
|
||||
|
||||
### Prepare Kata root filesystem
|
||||
|
||||
These instructions build upon the OS builder instructions located in the
|
||||
[Developer Guide](../Developer-Guide.md). The following instructions use Clear
|
||||
Linux (Kata Containers default) as the root filesystem with systemd as the
|
||||
init and will add in the `kmod` binary, which is not a standard binary in a
|
||||
Kata rootfs image. The `kmod` binary is necessary to load the QAT kernel
|
||||
modules when the virtual machine rootfs boots. You should install Docker on
|
||||
your system before running the following commands. If you need to use a custom
|
||||
`kata-agent`, then refer to the previous link on how to add it in.
|
||||
[Developer Guide](../Developer-Guide.md). At this point it is recommended that
|
||||
[Docker](https://docs.docker.com/engine/install/ubuntu/) is installed first, and
|
||||
then [Kata-deploy](https://github.com/kata-containers/kata-containers/tree/main/tools/packaging/kata-deploy)
|
||||
is use to install Kata. This will make sure that the correct `agent` version
|
||||
is installed into the rootfs in the steps below.
|
||||
|
||||
```sh
|
||||
$ mkdir -p $OSBUILDER
|
||||
$ cd $OSBUILDER
|
||||
$ git clone https://github.com/kata-containers/osbuilder.git
|
||||
$ export ROOTFS_DIR=${OSBUILDER}/osbuilder/rootfs-builder/rootfs
|
||||
The following instructions use Debian as the root filesystem with systemd as
|
||||
the init and will add in the `kmod` binary, which is not a standard binary in
|
||||
a Kata rootfs image. The `kmod` binary is necessary to load the Intel® QAT
|
||||
kernel modules when the virtual machine rootfs boots.
|
||||
|
||||
```bash
|
||||
$ export OSBUILDER=$GOPATH/src/github.com/kata-containers/kata-containers/tools/osbuilder
|
||||
$ export ROOTFS_DIR=${OSBUILDER}/rootfs-builder/rootfs
|
||||
$ export EXTRA_PKGS='kmod'
|
||||
```
|
||||
|
||||
Make sure that the `kata-agent` version matches the installed `kata-runtime`
|
||||
version.
|
||||
```sh
|
||||
version. Also make sure the `kata-runtime` install location is in your `PATH`
|
||||
variable. The following `AGENT_VERSION` can be set manually to match
|
||||
the `kata-runtime` version if the following commands don't work.
|
||||
|
||||
```bash
|
||||
$ export PATH=$PATH:/opt/kata/bin
|
||||
$ cd $GOPATH
|
||||
$ export AGENT_VERSION=$(kata-runtime version | head -n 1 | grep -o "[0-9.]\+")
|
||||
$ cd ${OSBUILDER}/osbuilder/rootfs-builder
|
||||
$ cd ${OSBUILDER}/rootfs-builder
|
||||
$ sudo rm -rf ${ROOTFS_DIR}
|
||||
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh clearlinux'
|
||||
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true SECCOMP=no ./rootfs.sh debian'
|
||||
```
|
||||
|
||||
### Compile Intel QAT drivers for Kata Containers kernel and add to Kata Containers rootfs
|
||||
### Compile Intel® QAT drivers for Kata Containers kernel and add to Kata Containers rootfs
|
||||
|
||||
After the Kata Containers kernel builds with the proper configuration flags,
|
||||
you must build the Intel QAT drivers against that Kata Containers kernel
|
||||
you must build the Intel® QAT drivers against that Kata Containers kernel
|
||||
version in a similar way they were previously built for the host OS. You must
|
||||
set the `KERNEL_SOURCE_ROOT` variable to the Kata Containers kernel source
|
||||
directory and build the Intel QAT drivers again.
|
||||
directory and build the Intel® QAT drivers again. The `make` command will
|
||||
install the Intel® QAT modules into the Kata rootfs.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ cd $GOPATH
|
||||
$ export LINUX_VER=$(ls -d kata*)
|
||||
$ export KERNEL_MAJOR_VERSION=$(awk '/^VERSION =/{print $NF}' $GOPATH/$LINUX_VER/Makefile)
|
||||
@ -284,16 +306,18 @@ $ export KERNEL_SUBLEVEL=$(awk '/^SUBLEVEL =/{print $NF}' $GOPATH/$LINUX_VER/Mak
|
||||
$ export KERNEL_EXTRAVERSION=$(awk '/^EXTRAVERSION =/{print $NF}' $GOPATH/$LINUX_VER/Makefile)
|
||||
$ export KERNEL_ROOTFS_DIR=${KERNEL_MAJOR_VERSION}.${KERNEL_PATHLEVEL}.${KERNEL_SUBLEVEL}${KERNEL_EXTRAVERSION}
|
||||
$ cd $QAT_SRC
|
||||
$ KERNEL_SOURCE_ROOT=$GOPATH/$LINUX_VER ./configure --disable-qat-lkcf --enable-icp-sriov=guest
|
||||
$ KERNEL_SOURCE_ROOT=$GOPATH/$LINUX_VER ./configure --enable-icp-sriov=guest
|
||||
$ sudo -E make all -j$(nproc)
|
||||
$ sudo -E make INSTALL_MOD_PATH=$ROOTFS_DIR qat-driver-install -j$(nproc)
|
||||
```
|
||||
|
||||
The `usdm_drv` module also needs to be copied into the rootfs modules path and
|
||||
`depmod` should be run.
|
||||
```sh
|
||||
$ sudo cp $QAT_SRC/build/usdm_drv.ko $ROOTFS_DIR/usr/lib/modules/${KERNEL_ROOTFS_DIR}/updates/drivers
|
||||
|
||||
```bash
|
||||
$ sudo cp $QAT_SRC/build/usdm_drv.ko $ROOTFS_DIR/lib/modules/${KERNEL_ROOTFS_DIR}/updates/drivers
|
||||
$ sudo depmod -a -b ${ROOTFS_DIR} ${KERNEL_ROOTFS_DIR}
|
||||
$ cd ${OSBUILDER}/osbuilder/image-builder
|
||||
$ cd ${OSBUILDER}/image-builder
|
||||
$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'
|
||||
```
|
||||
|
||||
@ -302,84 +326,225 @@ $ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'
|
||||
|
||||
### Copy Kata rootfs
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ mkdir -p $KATA_ROOTFS_LOCATION
|
||||
$ cp ${OSBUILDER}/osbuilder/image-builder/kata-containers.img $KATA_ROOTFS_LOCATION
|
||||
$ cp ${OSBUILDER}/image-builder/kata-containers.img $KATA_ROOTFS_LOCATION
|
||||
```
|
||||
|
||||
### Update Kata configuration to point to custom kernel and rootfs
|
||||
## Verify Intel® QAT works in a container
|
||||
|
||||
You must update the `configuration.toml` for Kata Containers to point to the
|
||||
custom kernel, custom rootfs, and to specify which modules to load when the
|
||||
virtual machine is booted when a container is run. The following example
|
||||
assumes you installed an Intel QAT, and you need to load those modules.
|
||||
|
||||
```sh
|
||||
$ sudo mkdir -p /etc/kata-containers
|
||||
$ sudo cp /usr/share/defaults/kata-containers/configuration-qemu.toml /etc/kata-containers/configuration.toml
|
||||
$ sudo sed -i "s|kernel_params = \"\"|kernel_params = \"modules-load=usdm_drv,qat_c62xvf\"|g" /etc/kata-containers/configuration.toml
|
||||
$ sudo sed -i "s|\/usr\/share\/kata-containers\/kata-containers.img|${KATA_KERNEL_LOCATION}\/kata-containers.img|g" /etc/kata-containers/configuration.toml
|
||||
$ sudo sed -i "s|\/usr\/share\/kata-containers\/vmlinuz.container|${KATA_ROOTFS_LOCATION}\/vmlinuz-${LINUX_VER}_qat|g" /etc/kata-containers/configuration.toml
|
||||
```
|
||||
|
||||
## Verify Intel QAT works in a Docker Kata Containers container
|
||||
|
||||
The following instructions leverage an OpenSSL Dockerfile that builds the
|
||||
Intel QAT engine to allow OpenSSL to offload crypto functions. It is a
|
||||
convenient way to test that VFIO device passthrough for the Intel QAT VF’s are
|
||||
The following instructions uses a OpenSSL Dockerfile that builds the
|
||||
Intel® QAT engine to allow OpenSSL to offload crypto functions. It is a
|
||||
convenient way to test that VFIO device passthrough for the Intel® QAT VF’s are
|
||||
working properly with the Kata Containers VM.
|
||||
|
||||
## Build OpenSSL Intel QAT engine container
|
||||
### Build OpenSSL Intel® QAT engine container
|
||||
|
||||
Use the OpenSSL Intel QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/master/demo/openssl-qat-engine)
|
||||
Use the OpenSSL Intel® QAT [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/tree/master/demo/openssl-qat-engine)
|
||||
to build a container image with an optimized OpenSSL engine for
|
||||
Intel QAT. Using `docker build` with the Kata Containers runtime can sometimes
|
||||
have issues. Therefore, we recommended you change the default runtime to
|
||||
`runc` before doing a build. Instructions for this are below.
|
||||
Intel® QAT. Using `docker build` with the Kata Containers runtime can sometimes
|
||||
have issues. Therefore, make sure that `runc` is the default Docker container
|
||||
runtime.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ cd $QAT_SRC
|
||||
$ curl -O $QAT_DOCKERFILE
|
||||
$ sudo sed -i 's/kata-runtime/runc/g' /etc/systemd/system/docker.service.d/50-runtime.conf
|
||||
$ sudo systemctl daemon-reload && sudo systemctl restart docker
|
||||
$ sudo docker build -t openssl-qat-engine .
|
||||
```
|
||||
|
||||
> **Note: The Intel QAT driver version in this container might not match the
|
||||
> Intel QAT driver compiled and loaded on the host when compiling.**
|
||||
> **Note: The Intel® QAT driver version in this container might not match the
|
||||
> Intel® QAT driver compiled and loaded on the host when compiling.**
|
||||
|
||||
### Test Intel QAT in Docker
|
||||
### Test Intel® QAT with the ctr tool
|
||||
|
||||
The host should already be setup with 16 virtual functions of the Intel QAT
|
||||
card bound to `VFIO-PCI`. Verify this by looking in `/dev/vfio` for a listing
|
||||
of devices. Replace the number 90 with one of the VF’s exposed in `/dev/vfio`.
|
||||
It might require you to add an `IPC_LOCK` capability to your Docker runtime
|
||||
depending on which rootfs you use.
|
||||
The `ctr` tool can be used to interact with the containerd daemon. It may be
|
||||
more convenient to use this tool to verify the kernel and image instead of
|
||||
setting up a Kubernetes cluster. The correct Kata runtimes need to be added
|
||||
to the containerd `config.toml`. Below is a sample snippet that can be added
|
||||
to allow QEMU and Cloud Hypervisor (CLH) to work with `ctr`.
|
||||
|
||||
```sh
|
||||
$ sudo docker run -it --runtime=kata-runtime --cap-add=IPC_LOCK --cap-add=SYS_ADMIN --device=/dev/vfio/90 -v /dev:/dev -v ${QAT_CONF_LOCATION}:/etc openssl-qat-engine bash
|
||||
```
|
||||
[plugins.cri.containerd.runtimes.kata-qemu]
|
||||
runtime_type = "io.containerd.kata-qemu.v2"
|
||||
privileged_without_host_devices = true
|
||||
pod_annotations = ["io.katacontainers.*"]
|
||||
[plugins.cri.containerd.runtimes.kata-qemu.options]
|
||||
ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
|
||||
[plugins.cri.containerd.runtimes.kata-clh]
|
||||
runtime_type = "io.containerd.kata-clh.v2"
|
||||
privileged_without_host_devices = true
|
||||
pod_annotations = ["io.katacontainers.*"]
|
||||
[plugins.cri.containerd.runtimes.kata-clh.options]
|
||||
ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-clh.toml"
|
||||
```
|
||||
|
||||
Below are some commands to run in the container image to verify Intel QAT is
|
||||
In addition, containerd expects the binary to be in `/usr/local/bin` so add
|
||||
this small script so that it redirects to be able to use either QEMU or
|
||||
Cloud Hypervisor with Kata.
|
||||
|
||||
```bash
|
||||
$ echo '#!/bin/bash' | sudo tee /usr/local/bin/containerd-shim-kata-qemu-v2
|
||||
$ echo 'KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-qemu.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-qemu-v2
|
||||
$ sudo chmod +x /usr/local/bin/containerd-shim-kata-qemu-v2
|
||||
$ echo '#!/bin/bash' | sudo tee /usr/local/bin/containerd-shim-kata-clh-v2
|
||||
$ echo 'KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-clh.toml /opt/kata/bin/containerd-shim-kata-v2 $@' | sudo tee -a /usr/local/bin/containerd-shim-kata-clh-v2
|
||||
$ sudo chmod +x /usr/local/bin/containerd-shim-kata-clh-v2
|
||||
```
|
||||
|
||||
After the OpenSSL image is built and imported into containerd, a Intel® QAT
|
||||
virtual function exposed in the step above can be added to the `ctr` command.
|
||||
Make sure to change the `/dev/vfio` number to one that actually exists on the
|
||||
host system. When using the `ctr` tool, the`configuration.toml` for Kata needs
|
||||
to point to the custom Kata kernel and rootfs built above and the Intel® QAT
|
||||
modules in the Kata rootfs need to load at boot. The following steps assume that
|
||||
`kata-deploy` was used to install Kata and QEMU is being tested. If using a
|
||||
different hypervisor, different install method for Kata, or a different
|
||||
Intel® QAT chipset then the command will need to be modified.
|
||||
|
||||
> **Note: The following was tested with
|
||||
[containerd v1.3.9](https://github.com/containerd/containerd/releases/tag/v1.3.9).**
|
||||
|
||||
```bash
|
||||
$ config_file="/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
|
||||
$ sudo sed -i "/kernel =/c kernel = "\"${KATA_ROOTFS_LOCATION}/${KATA_KERNEL_NAME}\""" $config_file
|
||||
$ sudo sed -i "/image =/c image = "\"${KATA_KERNEL_LOCATION}/kata-containers.img\""" $config_file
|
||||
$ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 modules-load=usdm_drv,qat_c62xvf"/g' $config_file
|
||||
$ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest
|
||||
$ sudo ctr images import openssl-qat-engine.tar
|
||||
$ sudo ctr run --runtime io.containerd.run.kata-qemu.v2 --privileged -t --rm --device=/dev/vfio/180 --mount type=bind,src=/dev,dst=/dev,options=rbind:rw --mount type=bind,src=${QAT_CONF_LOCATION}/c6xxvf_dev0.conf,dst=/etc/c6xxvf_dev0.conf,options=rbind:rw docker.io/library/openssl-qat-engine:latest bash
|
||||
```
|
||||
|
||||
Below are some commands to run in the container image to verify Intel® QAT is
|
||||
working
|
||||
|
||||
```sh
|
||||
bash-5.0# cat /proc/modules
|
||||
bash-5.0# adf_ctl restart
|
||||
bash-5.0# adf_ctl status
|
||||
bash-5.0# openssl engine -c -t qat
|
||||
root@67561dc2757a/ # cat /proc/modules
|
||||
qat_c62xvf 16384 - - Live 0xffffffffc00d9000 (OE)
|
||||
usdm_drv 86016 - - Live 0xffffffffc00e8000 (OE)
|
||||
intel_qat 249856 - - Live 0xffffffffc009b000 (OE)
|
||||
|
||||
root@67561dc2757a/ # adf_ctl restart
|
||||
Restarting all devices.
|
||||
Processing /etc/c6xxvf_dev0.conf
|
||||
|
||||
root@67561dc2757a/ # adf_ctl status
|
||||
Checking status of all devices.
|
||||
There is 1 QAT acceleration device(s) in the system:
|
||||
qat_dev0 - type: c6xxvf, inst_id: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up
|
||||
|
||||
root@67561dc2757a/ # openssl engine -c -t qat-hw
|
||||
(qat-hw) Reference implementation of QAT crypto engine v0.6.1
|
||||
[RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448]
|
||||
[ available ]
|
||||
```
|
||||
|
||||
Test with Intel QAT card acceleration
|
||||
### Test Intel® QAT in Kubernetes
|
||||
|
||||
```sh
|
||||
bash-5.0# openssl speed -engine qat -elapsed -async_jobs 72 rsa2048
|
||||
Start a Kubernetes cluster with containerd as the CRI. The host should
|
||||
already be setup with 16 virtual functions of the Intel® QAT card bound to
|
||||
`VFIO-PCI`. Verify this by looking in `/dev/vfio` for a listing of devices.
|
||||
You might need to disable Docker before initializing Kubernetes. Be aware
|
||||
that the OpenSSL container image built above will need to be exported from
|
||||
Docker and imported into containerd.
|
||||
|
||||
If Kata is installed through [`kata-deploy`](https://github.com/kata-containers/kata-containers/blob/stable-2.0/tools/packaging/kata-deploy/README.md)
|
||||
there will be multiple `configuration.toml` files associated with different
|
||||
hypervisors. Rather than add in the custom Kata kernel, Kata rootfs, and
|
||||
kernel modules to each `configuration.toml` as the default, instead use
|
||||
[annotations](https://github.com/kata-containers/kata-containers/blob/stable-2.0/docs/how-to/how-to-load-kernel-modules-with-kata.md)
|
||||
in the Kubernetes YAML file to tell Kata which kernel and rootfs to use. The
|
||||
easy way to do this is to use `kata-deploy` which will install the Kata binaries
|
||||
to `/opt` and properly configure the `/etc/containerd/config.toml` with annotation
|
||||
support. However, the `configuration.toml` needs to enable support for
|
||||
annotations as well. The following configures both QEMU and Cloud Hypervisor
|
||||
`configuration.toml` files that are currently available with Kata Container
|
||||
versions 2.0 and higher.
|
||||
|
||||
```bash
|
||||
$ sudo sed -i 's/enable_annotations\s=\s\[\]/enable_annotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-qemu.toml
|
||||
$ sudo sed -i 's/enable_annotations\s=\s\[\]/enable_annotations = [".*"]/' /opt/kata/share/defaults/kata-containers/configuration-clh.toml
|
||||
```
|
||||
|
||||
Test with CPU acceleration
|
||||
Export the OpenSSL image from Docker and import into containerd.
|
||||
|
||||
```bash
|
||||
$ sudo docker save -o openssl-qat-engine.tar openssl-qat-engine:latest
|
||||
$ sudo ctr -n=k8s.io images import openssl-qat-engine.tar
|
||||
```
|
||||
|
||||
The [Intel® QAT Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/cmd/qat_plugin/README.md)
|
||||
needs to be started so that the virtual functions can be discovered and
|
||||
used by Kubernetes.
|
||||
|
||||
The following YAML file can be used to start a Kata container with Intel® QAT
|
||||
support. If Kata is installed with `kata-deploy`, then the containerd
|
||||
`configuration.toml` should have all of the Kata runtime classes already
|
||||
populated and annotations supported. To use a Intel® QAT virtual function, the
|
||||
Intel® QAT plugin needs to be started after the VF's are bound to `VFIO-PCI` as
|
||||
described [above](#expose-and-bind-intel-qat-virtual-functions-to-vfio-pci-every-reboot).
|
||||
Edit the following to point to the correct Kata kernel and rootfs location
|
||||
built with Intel® QAT support.
|
||||
|
||||
```bash
|
||||
$ cat << EOF > kata-openssl-qat.yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: kata-openssl-qat
|
||||
labels:
|
||||
app: kata-openssl-qat
|
||||
annotations:
|
||||
io.katacontainers.config.hypervisor.kernel: "$KATA_KERNEL_LOCATION/$KATA_KERNEL_NAME"
|
||||
io.katacontainers.config.hypervisor.image: "$KATA_ROOTFS_LOCATION/kata-containers.img"
|
||||
io.katacontainers.config.hypervisor.kernel_params: "modules-load=usdm_drv,qat_c62xvf"
|
||||
spec:
|
||||
runtimeClassName: kata-qemu
|
||||
containers:
|
||||
- name: kata-openssl-qat
|
||||
image: docker.io/library/openssl-qat-engine:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
resources:
|
||||
limits:
|
||||
qat.intel.com/generic: 1
|
||||
cpu: 1
|
||||
securityContext:
|
||||
capabilities:
|
||||
add: ["IPC_LOCK", "SYS_ADMIN"]
|
||||
volumeMounts:
|
||||
- mountPath: /etc/c6xxvf_dev0.conf
|
||||
name: etc-mount
|
||||
- mountPath: /dev
|
||||
name: dev-mount
|
||||
volumes:
|
||||
- name: dev-mount
|
||||
hostPath:
|
||||
path: /dev
|
||||
- name: etc-mount
|
||||
hostPath:
|
||||
path: $QAT_CONF_LOCATION/c6xxvf_dev0.conf
|
||||
EOF
|
||||
```
|
||||
|
||||
Use `kubectl` to start the pod. Verify that Intel® QAT card acceleration is
|
||||
working with the Intel® QAT engine.
|
||||
```bash
|
||||
$ kubectl apply -f kata-openssl-qat.yaml
|
||||
```
|
||||
|
||||
```sh
|
||||
bash-5.0# openssl speed -elapsed rsa2048
|
||||
$ kubectl exec -it kata-openssl-qat -- adf_ctl restart
|
||||
Restarting all devices.
|
||||
Processing /etc/c6xxvf_dev0.conf
|
||||
|
||||
$ kubectl exec -it kata-openssl-qat -- adf_ctl status
|
||||
Checking status of all devices.
|
||||
There is 1 QAT acceleration device(s) in the system:
|
||||
qat_dev0 - type: c6xxvf, inst_id: 0, node_id: 0, bsf: 0000:01:01.0, #accel: 1 #engines: 1 state: up
|
||||
|
||||
$ kubectl exec -it kata-openssl-qat -- openssl engine -c -t qat-hw
|
||||
(qat-hw) Reference implementation of QAT crypto engine v0.6.1
|
||||
[RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF, HKDF, X25519, X448]
|
||||
[ available ]
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
@ -412,9 +577,9 @@ c6xxvf_dev10.conf c6xxvf_dev13.conf c6xxvf_dev2.conf c6xxvf_dev5.conf c6xxvf
|
||||
```
|
||||
|
||||
* Check `dmesg` inside the container to see if there are any issues with the
|
||||
Intel QAT driver.
|
||||
Intel® QAT driver.
|
||||
|
||||
* If there are issues building the OpenSSL Intel QAT container image, then
|
||||
* If there are issues building the OpenSSL Intel® QAT container image, then
|
||||
check to make sure that runc is the default runtime for building container.
|
||||
|
||||
```sh
|
||||
@ -425,17 +590,18 @@ Environment="DOCKER_DEFAULT_RUNTIME=--default-runtime runc"
|
||||
|
||||
## Optional Scripts
|
||||
|
||||
### Verify Intel QAT card counters are incremented
|
||||
### Verify Intel® QAT card counters are incremented
|
||||
|
||||
Use the `lspci` command to figure out which PCI bus the Intel QAT accelerators
|
||||
are on. The counters will increase when the accelerator is actively being
|
||||
used. To verify QAT is actively accelerating the containerized application,
|
||||
use the following instructions to check if any of the counters are
|
||||
incrementing. You will have to change the PCI device ID to match your system.
|
||||
To check the built in firmware counters, the Intel® QAT driver has to be compiled
|
||||
and installed to the host and can't rely on the built in host driver. The
|
||||
counters will increase when the accelerator is actively being used. To verify
|
||||
Intel® QAT is actively accelerating the containerized application, use the
|
||||
following instructions to check if any of the counters increment. Make
|
||||
sure to change the PCI Device ID to match whats in the system.
|
||||
|
||||
```sh
|
||||
```bash
|
||||
$ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done
|
||||
$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b1\:00.0/fw_counters
|
||||
$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b3\:00.0/fw_counters
|
||||
$ sudo watch cat /sys/kernel/debug/qat_c6xx_0000\:b5\:00.0/fw_counters
|
||||
```
|
||||
```
|
||||
Loading…
Reference in New Issue
Block a user