diff --git a/docs/how-to/README.md b/docs/how-to/README.md index fd5547d640..de60fc0291 100644 --- a/docs/how-to/README.md +++ b/docs/how-to/README.md @@ -44,6 +44,7 @@ - [How to run Docker with Kata Containers](how-to-run-docker-with-kata.md) - [How to run Kata Containers with `nydus`](how-to-use-virtio-fs-nydus-with-kata.md) - [How to run Kata Containers with AMD SEV-SNP](how-to-run-kata-containers-with-SNP-VMs.md) +- [How to run Kata Containers with IBM Secure Execution](how-to-run-kata-containers-with-SE-VMs.md) - [How to use EROFS to build rootfs in Kata Containers](how-to-use-erofs-build-rootfs.md) - [How to run Kata Containers with kinds of Block Volumes](how-to-run-kata-containers-with-kinds-of-Block-Volumes.md) - [How to use the Kata Agent Policy](how-to-use-the-kata-agent-policy.md) \ No newline at end of file diff --git a/docs/how-to/how-to-run-kata-containers-with-SE-VMs.md b/docs/how-to/how-to-run-kata-containers-with-SE-VMs.md new file mode 100644 index 0000000000..799668f4fa --- /dev/null +++ b/docs/how-to/how-to-run-kata-containers-with-SE-VMs.md @@ -0,0 +1,389 @@ +# Kata Containers with IBM Secure Execution VMs + +This document assumes a trusted environment with a functioning kata container, as per the +[developer guide](https://github.com/kata-containers/kata-containers/blob/main/docs/Developer-Guide.md). +The term "trusted" implies that the system is authorized, authenticated and attested to use your artifacts and secrets safely. + + +## Tested Environment Specifications + +1. Machine: IBM z16 LPAR +2. OS: Ubuntu 22.04.1 LTS +3. CPU: 16 vCPU +4. Memory: 16G + +## Manual configuration + +### Prerequisite + +1. Host capable of Secure Execution + +To take advantage of the IBM Secure Execution capability, the host machine on which +you intend to run workloads must be an IBM z15 (or a newer model) or an IBM LinuxONE III +(or a newer model). In addition to the hardware requirement, you need to verify the CPU +facility and kernel configuration, as outlined below: + +``` +$ # To check the protected virtualization support from kernel +$ cat /sys/firmware/uv/prot_virt_host +1 +$ # To check if an ultravisor reserves memory for the current boot +$ sudo dmesg | grep -i ultravisor +[ 0.063630] prot_virt.f9efb6: Reserving 98MB as ultravisor base storage +$ # To check a facility bit for Secure Execution +$ cat /proc/cpuinfo | grep 158 +facilities : ... numbers ... 158 ... numbers ... +``` + +If any of the results are not identifiable, please reach out to the responsible cloud +provider to enable the Secure Execution capability. Alternatively, if you possess +administrative privileges and the facility bit is set, you can enable the Secure Execution +capability by adding `prot_virt=1` to the kernel parameters and performing a system reboot like: + +``` +$ sudo sed -i 's/^\(parameters.*\)/\1 prot_virt=1/g' /etc/zipl.conf +$ sudo zipl -V +$ sudo systemctl reboot +``` + +Please note that the method of enabling the Secure Execution capability may vary among Linux distributions. + +2. Artifacts from Kata Containers + +A secure image is constructed using the following artifacts + +- A raw kernel +- An initial RAM disk + +The most straightforward approach to obtain these artifacts is by reusing kata-containers: + +``` +$ export PATH="$PATH:/opt/kata/bin" +$ ls -1 $(dirname $(kata-runtime env --json | jq -r '.Kernel.Path')) +config-6.1.62-121 +kata-containers.img +kata-containers-initrd.img +kata-ubuntu-20.04.initrd +kata-ubuntu-latest.image +vmlinux-6.1.62-121 +vmlinux.container +vmlinuz-6.1.62-121 +vmlinuz.container +``` + +The output indicates the deployment of the kernel (`vmlinux-6.1.62-121`, though the version +may vary at the time of testing), rootfs-image (`kata-ubuntu-latest.image`), and rootfs-initrd (`kata-ubuntu-20.04.initrd`). +In this scenario, the available kernel and initrd can be utilized for a secure image. +However, if any of these components are absent, they must be built from the +[project source](https://github.com/kata-containers/kata-containers) as follows: + +``` +$ # Assume that the project is cloned at $GOPATH/src/github.com/kata-containers +$ cd $GOPATH/src/github.com/kata-containers/kata-containers +$ sudo -E PATH=$PATH make kernel-tarball +$ sudo -E PATH=$PATH make rootfs-initrd-tarball +$ tar -tf build/kata-static-kernel.tar.xz | grep vmlinuz +./opt/kata/share/kata-containers/vmlinuz.container +./opt/kata/share/kata-containers/vmlinuz-6.1.62-121 +$ tar -tf build/kata-static-rootfs-initrd.tar.xz | grep initrd +./opt/kata/share/kata-containers/kata-containers-initrd.img +./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd +$ mkdir artifacts +$ tar -xvf build/kata-static-kernel.tar.xz -C artifacts ./opt/kata/share/kata-containers/vmlinuz-6.1.62-121 +$ tar -xvf build/kata-static-rootfs-initrd.tar.xz -C artifacts ./opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd +$ ls artifacts/opt/kata/share/kata-containers/ +kata-ubuntu-20.04.initrd vmlinuz-6.1.62-121 +``` + +3. Secure Image Generation Tool + +`genprotimg` is a utility designed to generate an IBM Secure Execution image. It can be +installed either from the package manager of a distribution or from the source code. +The tool is included in the `s390-tools` package. Please ensure that you have a version +of the tool equal to or greater than `2.17.0`. If not, you will need to specify +an additional argument, `--x-pcf '0xe0'`, when running the command. +Here is an example of a native build from the source: + +``` +$ sudo apt-get install gcc libglib2.0-dev libssl-dev libcurl4-openssl-dev +$ tool_version=v2.25.0 +$ git clone -b $tool_version https://github.com/ibm-s390-linux/s390-tools.git +$ pushd s390-tools/genprotimg && make && sudo make install && popd +$ rm -rf s390-tools +``` + +4. Host Key Document + +A host key document is a public key employed for encrypting a secure image, which is +subsequently decrypted using a corresponding private key during the VM bootstrap process. +You can obtain the host key document either through IBM's designated +[Resource Link](http://www.ibm.com/servers/resourcelink) or by requesting it from the +cloud provider responsible for the IBM Z and LinuxONE instances where your workloads are intended to run. + +To ensure security, it is essential to verify the authenticity and integrity of the host key document +belonging to an authentic IBM machine. To achieve this, please additionally obtain the following +certificates from the Resource Link: + +- IBM Z signing key certificate +- `DigiCert` intermediate CA certificate + +These files will be used for verification during secure image construction in the next section. + +### Build a Secure Image + + +Assuming you have placed a host key document at `$HOME/host-key-document`: + +- Host key document as `HKD-0000-0000000.crt` + +and two certificates at `$HOME/certificates`: + +- `DigiCert` intermediate CA certificate as `DigiCertCA.crt` +- IBM Z signing-key certificate as `ibm-z-host-key-signing.crt` + +you can construct a secure image using the following procedure: + +``` +$ # Change a directory to the project root +$ cd $GOPATH/src/github.com/kata-containers/kata-containers +$ host_key_document=$HOME/host-key-document/HKD-0000-0000000.crt +$ kernel_image=artifacts/opt/kata/share/kata-containers/vmlinuz-6.1.62-121 +$ initrd_image=artifacts/opt/kata/share/kata-containers/kata-ubuntu-20.04.initrd +$ echo "panic=1 scsi_mod.scan=none swiotlb=262144 agent.log=debug" > parmfile +$ genprotimg --host-key-document=${host_key_document} \ +--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \ +--parmfile=parmfile --no-verify +WARNING: host-key document verification is disabled. Your workload is not secured. +$ file kata-containers-se.img +kata-containers-se.img: data +$ sudo cp kata-containers-se.img /opt/kata/share/kata-containers/ +``` + +It is important to note that the `--no-verify` parameter, which allows skipping +the key verification process, is intended to be used solely in a development or +testing environment. +In production, the image construction should incorporate the verification +in the following manner: + +``` +$ cacert=$HOME/certificates/DigiCertCA.crt +$ signcert=$HOME/certificates/ibm-z-host-key-signing.crt +$ genprotimg --host-key-document=${host_key_document} \ +--output=kata-containers-se.img --image=${kernel_image} --ramdisk=${initrd_image} \ +--cert=${cacert} --cert=${signcert} --parmfile=parmfile +``` + +The steps with no verification, including the dependencies for the kernel and initrd, +can be easily accomplished by issuing the following make target: + +``` +$ cd $GOPATH/src/github.com/kata-containers/kata-containers +$ mkdir hkd_dir && cp $host_key_document hkd_dir +$ sudo -E PATH=$PATH HKD_PATH=hkd_dir SE_KERNEL_PARAMS="agent.log=debug" \ +make boot-image-se-tarball +$ ls build/kata-static-boot-image-se.tar.xz +build/kata-static-boot-image-se.tar.xz +``` + +`SE_KERNEL_PARAMS` could be used to add any extra kernel parameters. If no additional kernel configuration is required, this can be omitted. + +In production, you could build an image by running the same command, but with two +additional environment variables for key verification: + +``` +$ export SIGNING_KEY_CERT_PATH=$HOME/certificates/ibm-z-host-key-signing.crt +$ export INTERMEDIATE_CA_CERT_PATH=$HOME/certificates/DigiCertCA.crt +``` + +To build an image on the `x86_64` platform, set the following environment variables together with the variables above before `make boot-image-se-tarball`: + +``` +CROSS_BUILD=true TARGET_ARCH=s390x ARCH=s390x +``` + +### Adjust the configuration + +There still remains an opportunity to fine-tune the configuration file: + +``` +$ runtime_config_path=$(kata-runtime kata-env --json | jq -r '.Runtime.Config.Path') +$ cp ${runtime_config_path} ${runtime_config_path}.old +$ # Make the following adjustment to the original config file +$ diff ${runtime_config_path}.old ${runtime_config_path} +16,17c16,17 +< kernel = "/opt/kata/share/kata-containers/vmlinux.container" +< image = "/opt/kata/share/kata-containers/kata-containers.img" +--- +> kernel = "/opt/kata/share/kata-containers/kata-containers-se.img" +> # image = "/opt/kata/share/kata-containers/kata-containers.img" +41c41 +< # confidential_guest = true +--- +> confidential_guest = true +544c544 +< dial_timeout = 45 +--- +> dial_timeout = 90 +679c679 +< #service_offload = true +--- +> service_offload = true +``` + +### Verification + +To verify the successful decryption and loading of the secure image within a test VM, +please refer to the following commands: + +``` +$ cd $GOPATH/src/github.com/kata-containers/kata-containers +$ hypervisor_command=$(kata-runtime kata-env --json | jq -r '.Hypervisor.Path') +$ secure_kernel=kata-containers-se.img +$ sudo $hypervisor_command -machine confidential-guest-support=pv0 \ +-object s390-pv-guest,id=pv0 -accel kvm -smp 2 --m 4096 -serial mon:stdio \ +--nographic --nodefaults --kernel "${secure_kernel}" +[ 0.110277] Linux version 5.19.2 (root@637f067c5f7d) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP Wed May 31 09:06:49 UTC 2023 [ 0.110279] setup: Linux is running under KVM in 64-bit mode + +... log skipped ... + +[ 1.467228] Run /init as init process +{"msg":"baremount source=\"proc\", dest=\"/proc\", fs_type=\"proc\", options=\"\", flags=MS_NOSUID | MS_NODEV | MS_NOEXEC","level":"INFO","ts":"2023-06-07T10:17:23.537542429Z","pid":"1","subsystem":"baremount","name":"kata-agent","source":"agent +","version":"0.1.0"} + +... log skipped ... + +$ # Press ctrl + a + x to exit +``` + +If the hypervisor log does not indicate any errors, it provides assurance that the image +has been successfully loaded, and a Virtual Machine (VM) initiated by the kata runtime +will function properly. + +Let us proceed with the final verification by running a test container in a Kubernetes +cluster. Please make user you have a running cluster like: + +``` +$ kubectl get node +NAME STATUS ROLES AGE VERSION +test-cluster Ready control-plane,master 7m28s v1.23.1 +``` + +Please execute the following command to run a container: + +``` +$ cat <