# Kata Containers with IBM Secure Execution VMs This document assumes a trusted environment with a functioning kata container, as per the [developer guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md). The term "trusted" implies that the system is authorized, authenticated and attested to use your artifacts and secrets safely. ## Tested Environment Specifications 1. Machine: IBM z16 LPAR 2. OS: Ubuntu 22.04.1 LTS 3. CPU: 16 vCPU 4. Memory: 16G ## Manual configuration ### Prerequisite 1. Host capable of Secure Execution To take advantage of the IBM Secure Execution capability, the host machine on which you intend to run workloads must be an IBM z15 (or a newer model) or an IBM LinuxONE III (or a newer model). In addition to the hardware requirement, you need to verify the CPU facility and kernel configuration, as outlined below: ``` $ # To check the protected virtualization support from kernel $ cat /sys/firmware/uv/prot_virt_host 1 $ # To check if an ultravisor reserves memory for the current boot $ sudo dmesg | grep -i ultravisor [ 0.063630] prot_virt.f9efb6: Reserving 98MB as ultravisor base storage $ # To check a facility bit for Secure Execution $ cat /proc/cpuinfo | grep 158 facilities : ... numbers ... 158 ... numbers ... ``` If any of the results are not identifiable, please reach out to the responsible cloud provider to enable the Secure Execution capability. Alternatively, if you possess administrative privileges and the facility bit is set, you can enable the Secure Execution capability by adding `prot_virt=1` to the kernel parameters and performing a system reboot like: ``` $ sudo sed -i 's/^\(parameters.*\)/\1 prot_virt=1/g' /etc/zipl.conf $ sudo zipl -V $ sudo systemctl reboot ``` Please note that the method of enabling the Secure Execution capability may vary among Linux distributions. 2. Artifacts from Kata Containers A secure image is constructed using the following artifacts - A raw kernel - An initial RAM disk The most straightforward approach to obtain these artifacts is by reusing kata-containers: ``` $ export PATH="$PATH:/opt/kata/bin" $ ls -1 $(dirname $(kata-runtime env --json | jq -r '.Kernel.Path')) config-6.1.62-121 kata-containers.img kata-containers-confidential.img kata-containers-initrd.img kata-containers-initrd-confidential.img kata-ubuntu-20.04.initrd kata-ubuntu-20.04-confidential.initrd kata-ubuntu-latest.image kata-ubuntu-latest-confidential.image vmlinux-6.1.62-121 vmlinux-6.1.62-121-confidential vmlinux.container vmlinux-confidential.container vmlinuz-6.1.62-121 vmlinuz-6.1.62-121-confidential vmlinuz.container vmlinuz-confidential.container ``` The output indicates the deployment of the kernel (`vmlinux-6.1.62-121-confidential`, though the version may vary at the time of testing), rootfs-image (`kata-ubuntu-latest-confidential.image`), and rootfs-initrd (`kata-ubuntu-20.04-confidential.initrd`). In this scenario, the available kernel and initrd can be utilized for a secure image. However, if any of these components are absent, they must be built from the [project source](https://github.com/kata-containers/kata-containers) as follows: ``` $ # Assume that the project is cloned at $GOPATH/src/github.com/kata-containers $ cd $GOPATH/src/github.com/kata-containers/kata-containers $ make rootfs-initrd-confidential-tarball $ tar -tf build/kata-static-kernel-confidential.tar.xz | grep vmlinuz ./opt/kata/share/kata-containers/vmlinuz-confidential.container ./opt/kata/share/kata-containers/vmlinuz-6.7-136-confidential $ kernel_version=6.7-136 $ tar -tf build/kata-static-rootfs-initrd-confidential.tar.xz | grep initrd ./opt/kata/share/kata-containers/kata-containers-initrd-confidential.img ./opt/kata/share/kata-containers/kata-ubuntu-20.04-confidential.initrd $ mkdir artifacts $ tar -xvf build/kata-static-kernel-confidential.tar.xz -C artifacts ./opt/kata/share/kata-containers/vmlinuz-${kernel_version}-confidential $ tar -xvf build/kata-static-rootfs-initrd-confidential.tar.xz -C artifacts ./opt/kata/share/kata-containers/kata-ubuntu-20.04-confidential.initrd $ ls artifacts/opt/kata/share/kata-containers/ kata-ubuntu-20.04-confidential.initrd vmlinuz-${kernel_version}-confidential ``` 3. Secure Image Generation Tool `genprotimg` is a utility designed to generate an IBM Secure Execution image. It can be installed either from the package manager of a distribution or from the source code. The tool is included in the `s390-tools` package. Please ensure that you have a version of the tool equal to or greater than `2.17.0`. If not, you will need to specify an additional argument, `--x-pcf '0xe0'`, when running the command. Here is an example of a native build from the source: ``` $ sudo apt-get install gcc libglib2.0-dev libssl-dev libcurl4-openssl-dev $ tool_version=v2.34.0 $ git clone -b $tool_version https://github.com/ibm-s390-linux/s390-tools.git $ pushd s390-tools/genprotimg && make && sudo make install && popd $ rm -rf s390-tools ``` 4. Host Key Document A host key document is a public key employed for encrypting a secure image, which is subsequently decrypted using a corresponding private key during the VM bootstrap process. You can obtain the host key document either through IBM's designated [Resource Link](http://www.ibm.com/servers/resourcelink)(you need to log in to access it) or by requesting it from the cloud provider responsible for the IBM Z and LinuxONE instances where your workloads are intended to run. To ensure security, it is essential to verify the authenticity and integrity of the host key document belonging to an authentic IBM machine. To achieve this, please additionally obtain the following files from the Resource Link: - IBM Z signing key certificate - IBM Z host key certificate revocation list - `DigiCert` intermediate CA certificate These files will be used for verification during secure image construction in the next section. ### Build a Secure Image Assuming you have placed a host key document at `$HOME/host-key-document`: - Host key document as `HKD-0000-0000000.crt` and two certificates and one revocation list at `$HOME/certificates`: - IBM Z signing-key certificate as `ibm-z-host-key-signing-gen2.crt` - `DigiCert` intermediate CA certificate as `DigiCertCA.crt` - IBM Z host key certificate revocation list as `ibm-z-host-key-gen2.crl` you can construct a secure image using the following procedure: ``` $ # Change a directory to the project root $ cd $GOPATH/src/github.com/kata-containers/kata-containers $ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt $ kernel_image=artifacts/opt/kata/share/kata-containers/vmlinuz-${kernel_version}-confidential $ initrd_image=artifacts/opt/kata/share/kata-containers/kata-ubuntu-20.04-confidential.initrd $ echo "panic=1 scsi_mod.scan=none swiotlb=262144 agent.log=debug" > parmfile $ genprotimg --host-key-document=${host_key_document} \ --output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \ --parmfile=parmfile --no-verify WARNING: host-key document verification is disabled. Your workload is not secured. $ file kata-containers-se.img kata-containers-se.img: data $ sudo cp kata-containers-se.img /opt/kata/share/kata-containers/ ``` It is important to note that the `--no-verify` parameter, which allows skipping the key verification process, is intended to be used solely in a development or testing environment. In production, the image construction should incorporate the verification in the following manner: ``` $ signcert=$HOME/certificates/ibm-z-host-key-signing-gen2.crt $ cacert=$HOME/certificates/DigiCertCA.crt $ crl=$HOME/certificates/ibm-z-host-key-gen2.crl $ genprotimg --host-key-document=${host_key_document} \ --output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \ --cert=${cacert} --cert=${signcert} --crl=${crl} --parmfile=parmfile ``` The steps with no verification, including the dependencies for the kernel and initrd, can be easily accomplished by issuing the following make target: ``` $ cd $GOPATH/src/github.com/kata-containers/kata-containers $ mkdir hkd_dir && cp $host_key_document hkd_dir $ HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" make boot-image-se-tarball $ ls build/kata-static-boot-image-se.tar.xz build/kata-static-boot-image-se.tar.xz ``` `SE_KERNEL_PARAMS` could be used to add any extra kernel parameters. If no additional kernel configuration is required, this can be omitted. In production, you could build an image by running the same command, but with the following environment variables for key verification: ``` $ export SIGNING_KEY_CERT_PATH=$HOME/certificates/ibm-z-host-key-signing-gen2.crt $ export INTERMEDIATE_CA_CERT_PATH=$HOME/certificates/DigiCertCA.crt $ export HOST_KEY_CRL_PATH=$HOME/certificates/ibm-z-host-key-gen2.crl ``` To build an image on the `x86_64` platform, set the following environment variables together with the variables above before `make boot-image-se-tarball`: ``` CROSS_BUILD=true TARGET_ARCH=s390x ARCH=s390x ``` ### Adjust the configuration There still remains an opportunity to fine-tune the configuration file: ``` $ export PATH=$PATH:/opt/kata/bin $ runtime_config_path=$(kata-runtime kata-env --json | jq -r '.Runtime.Config.Path') $ sudo cp ${runtime_config_path} ${runtime_config_path}.old $ # Make the following adjustment to the original config file $ diff ${runtime_config_path}.old ${runtime_config_path} 16,17c16,17 < kernel = "/opt/kata/share/kata-containers/vmlinux.container" < image = "/opt/kata/share/kata-containers/kata-containers.img" --- > kernel = "/opt/kata/share/kata-containers/kata-containers-se.img" > # image = "/opt/kata/share/kata-containers/kata-containers.img" 41c41 < # confidential_guest = true --- > confidential_guest = true 544c544 < dial_timeout = 45 --- > dial_timeout = 90 ``` ### Verification To verify the successful decryption and loading of the secure image within a test VM, please refer to the following commands: ``` $ cd $GOPATH/src/github.com/kata-containers/kata-containers $ hypervisor_command=$(kata-runtime kata-env --json | jq -r '.Hypervisor.Path') $ secure_kernel=kata-containers-se.img $ sudo $hypervisor_command -machine confidential-guest-support=pv0 \ -object s390-pv-guest,id=pv0 -accel kvm -smp 2 --m 4096 -serial mon:stdio \ --nographic --nodefaults --kernel "${secure_kernel}" [ 0.110277] Linux version 5.19.2 (root@637f067c5f7d) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP Wed May 31 09:06:49 UTC 2023 [ 0.110279] setup: Linux is running under KVM in 64-bit mode ... log skipped ... [ 1.467228] Run /init as init process {"msg":"baremount source=\"proc\", dest=\"/proc\", fs_type=\"proc\", options=\"\", flags=MS_NOSUID | MS_NODEV | MS_NOEXEC","level":"INFO","ts":"2023-06-07T10:17:23.537542429Z","pid":"1","subsystem":"baremount","name":"kata-agent","source":"agent ","version":"0.1.0"} ... log skipped ... $ # Press ctrl + a + x to exit ``` Unless the host key document is legitimate, you will encounter the following error message: ``` qemu-system-s390x: KVM PV command 2 (KVM_PV_SET_SEC_PARMS) failed: header rc 108 rrc 5 IOCTL rc: -22 Protected boot has failed: 0xa02 ``` If the hypervisor log does not indicate any errors, it provides assurance that the image has been successfully loaded, and a Virtual Machine (VM) initiated by the kata runtime will function properly. Let us proceed with the final verification by running a test container in a Kubernetes cluster. Please make user you have a running cluster like: ``` $ kubectl get node NAME STATUS ROLES AGE VERSION test-cluster Ready control-plane,master 7m28s v1.23.1 ``` Please execute the following command to run a container: ``` $ cat <