kata-containers/docs/use-cases/CEX-passthrough-and-coco.md

Using IBM Crypto Express with Confidential Containers

On IBM Z (s390x), IBM Crypto Express (CEX) hardware security modules (HSM) can be passed through to virtual guests. This VFIO pass-through is domain-wise, i.e. guests can securely share one physical card. For the Accelerator and Enterprise PKCS #11 (EP11) modes of CEX, on IBM z16 and up, pass-through is also supported when using the IBM Secure Execution trusted execution environment. To maintain confidentiality when using EP11 within Secure Execution, additional steps are required. When using Secure Execution within Kata Containers, some of these steps are managed by the Kata agent, but preparation is required to make pass-through work. The Kata agent will expect required confidential information at runtime via Confidential Data Hub from Confidential Containers, and this guide assumes Confidential Containers components as a means of secret provisioning.

At the time of writing, devices for trusted execution environments are only supported via the --device option of e.g. ctr, docker, or podman, but not via Kubernetes. Refer to KEP 4113 for details.

Using a CEX card in Accelerator mode is much simpler and does not require the steps below. To do so, prepare Kata for Secure Execution, set vfio_mode = "vfio" and cold_plug_vfio = "bridge-port" in the Kata configuration.toml file and use a mediated device similar to operating without Secure Execution. The Kata agent will do the Secure Execution bind automatically.

Prerequisites

• A host kernel that supports adjunct processor (AP) pass-through with Secure Execution. Official support exists as of Ubuntu 24.04, RHEL 8.10 and 9.4, and SLES 15 SP6.
• An EP11 domain with a master key set up. In this process, you will need the master key verification pattern (MKVP) [1].
• A mediated device, created from this domain, to pass through.
• Working Kata Containers with Secure Execution.
• Working access to a key broker service (KBS) with the IBM Secure Execution verifier from a Kata container. The provided Secure Execution header must match the Kata guest image and a policy to allow the appropriate secrets for this guest must be set up.
• In Kata's configuration.toml, set vfio_mode = "vfio" and cold_plug_vfio = "bridge-port"

Prepare an association secret

An EP11 Secure Execution workload requires an association secret to be inserted in the guest and associated with the adjunct processor (AP) queue. In Kata Containers, this secret must be created and made available via Trustee, whereas the Kata agent performs the actual secret insertion and association. On a trusted system, to create an association secret using the host key document (HKD) z16.crt, a guest header hdr.bin, a CA certificate DigiCertCA.crt, an IBM signing key ibm-z-host-key-signing-gen2.crt, and let the command create a random association secret that is named my secret and save this random association secret to my_random_secret, run:

[trusted]# pvsecret create -k z16.crt --hdr hdr.bin -o my_addsecreq \
  --crt DigiCertCA.crt --crt ibm-z-host-key-signing-gen2.crt \
  association "my secret" --output-secret my_random_secret

using pvsecret from the s390-tools suite. hdr.bin must be the Secure Execution header matching the Kata guest image, i.e. the one also provided to Trustee. This command saves the add-secret request itself to my_addsecreq, and information on the secret, including the secret ID, to my_secret.yaml. This secret ID must be provided alongside the secret. Write it to my_addsecid with or without leading 0x or, using yq:

[trusted]# yq ".id" my_secret.yaml > my_addsecid

Provision the association secret with Trustee

The secret and secret ID must be provided via Trustee with respect to the MKVP. The paths where the Kata agent will expect this info are vfio_ap/${mkvp}/secret and vfio_ap/${mkvp}/secret_id, where $mkvp is the first 16 bytes (32 hex numbers) without leading 0x of the MKVP.

For example, if your MKVPs read [1] as

WK CUR: valid 0xdb3c3b3c3f097dd55ec7eb0e7fdbcb933b773619640a1a75a9161cec00000000
WK NEW: empty -

use db3c3b3c3f097dd55ec7eb0e7fdbcb93 in the provision for Trustee. With a KBS running at 127.0.0.1:8080, to store the secret and ID created above in the KBS with the authentication key kbs.key and this MKVP, run:

[trusted]# kbs-client --url http://127.0.0.1:8080 config \
  --auth-private-key kbs.key set-resource \
  --path vfio_ap/db3c3b3c3f097dd55ec7eb0e7fdbcb93/secret \
  --resource-file my_addsecreq
[trusted]# kbs-client --url http://127.0.0.1:8080 config \
  --auth-private-key kbs.key set-resource \
  --path vfio_ap/db3c3b3c3f097dd55ec7eb0e7fdbcb93/secret_id \
  --resource-file my_addsecid

Run the workload

Assuming the mediated device exists at /dev/vfio/0, run e.g.

[host]# docker run --rm --runtime io.containerd.run.kata.v2 --device /dev/vfio/0 -it ubuntu

If you have s390-tools available in the container, you can see the available CEX domains including Secure Execution info using lszcrypt -V:

[container]# lszcrypt -V
CARD.DOM TYPE  MODE        STATUS     REQUESTS  PENDING HWTYPE QDEPTH FUNCTIONS  DRIVER      SESTAT     
--------------------------------------------------------------------------------------------------------
03       CEX8P EP11-Coproc online            2        0     14     08 -----XN-F- cex4card    -          
03.0041  CEX8P EP11-Coproc online            2        0     14     08 -----XN-F- cex4queue   usable     

---

[1] If you have access to the host, the MKVP can be read at /sys/bus/ap/card${cardno}/${apqn}/mkvps, where ${cardno} is the the two-digit hexadecimal identification for the card, and ${apqn} is the APQN of the domain you want to pass, e.g. card03/03.0041 for the the domain 0x41 on card 3. This information is only readable when card and domain are not yet masked for use with VFIO. If you do not have access to the host, you should receive the MKVP from your HSM domain administrator.