mirror of
https://github.com/confidential-containers/confidential-containers.git
synced 2025-10-22 07:52:15 +00:00
Reorganize Quickstart Guide
Simplify quickstart guide to cover installation, basic usage, encryption/signing, attestation. Focus on the generic KBS. Everything else is moved to other files. Pointers to the relevant files are included where needed. Signed-off-by: Tobin Feldman-Fitzthum <tobin@ibm.com>
This commit is contained in:
Tobin Feldman-Fitzthum
committed by
Tobin Feldman-Fitzthum
Tobin Feldman-Fitzthum
parent
4b2f89ee47
commit
6a765dcbe1
97
guides/enclave-cc.md
Normal file
97
guides/enclave-cc.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Enclave-CC
|
||||
|
||||
`enclave-cc` provides process-based isolation using SGX.
|
||||
This guide assumes that you already have a Kubernetes cluster
|
||||
and have deployed the operator as described in the **Installation**
|
||||
section of the [quickstart guide](../quickstart.md).
|
||||
|
||||
## Configuring enclave-cc custom resource to use a different KBC
|
||||
|
||||
**Note** The KBC configuration changes to the enclave-cc custom resource yaml
|
||||
must be made **before** deploying it.
|
||||
|
||||
Enclave CC supports Verdictd and in order to use it, users will have to
|
||||
properly configure a decrypt_config.conf, in order to set the `KBC` (`sample_kbc`
|
||||
or `eaa_kbc`) `IP`,`PORT`, and the `SECURITY_VALIDATE` (`false` or `true`)
|
||||
```
|
||||
{
|
||||
"key_provider": "provider:attestation-agent:KBC::IP:PORT",
|
||||
"security_validate": SECURITY_VALIDATE
|
||||
}
|
||||
```
|
||||
|
||||
Once that's set according to the users needs, the user will then have to run:
|
||||
`cat decrypt_config.conf | base64 -w 0` in order to get the data encoded and
|
||||
set it accordingly [here](https://github.com/confidential-containers/operator/blob/6f241fbc056f0a5d9e1bd2c10b2cedc0782b99ff/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml#L124).
|
||||
|
||||
## Creating a sample CoCo workload using enclave-cc
|
||||
|
||||
As an example, we setup a sample *hello world*
|
||||
workload with an encrypted and cosign signed container image using the `enclave-cc` runtime class for process based TEEs.
|
||||
The deployment below assumes the hardware SGX mode build is installed by the operator. To try on a non-TEE system, please
|
||||
use simulate SGX mode build.
|
||||
|
||||
The example uses a trivial hello world C application:
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: enclave-cc-pod
|
||||
spec:
|
||||
containers:
|
||||
- image: ghcr.io/confidential-containers/test-container-enclave-cc:encrypted
|
||||
name: hello-world
|
||||
workingDir: "/run/rune/boot_instance/"
|
||||
resources:
|
||||
limits:
|
||||
sgx.intel.com/epc: 600Mi
|
||||
env:
|
||||
- name: OCCLUM_RELEASE_ENCLAVE
|
||||
value: "1"
|
||||
command:
|
||||
- /run/rune/boot_instance/build/bin/occlum-run
|
||||
- /bin/hello_world
|
||||
runtimeClassName: enclave-cc
|
||||
|
||||
```
|
||||
|
||||
**Note** When the hardware SGX mode payload is used in an SGX enabled cluster, `sgx.intel.com/epc: 600Mi`
|
||||
resource request must be added to the pod spec.
|
||||
|
||||
Again, create a pod YAML file as previously described (this time we named it `enclave-cc-pod.yaml`) .
|
||||
|
||||
Create the workload:
|
||||
```
|
||||
kubectl apply -f enclave-cc-pod.yaml
|
||||
```
|
||||
Output:
|
||||
```
|
||||
pod/enclave-cc-pod created
|
||||
```
|
||||
|
||||
Ensure the pod was created successfully (in running state):
|
||||
```
|
||||
kubectl get pods
|
||||
```
|
||||
Output:
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
enclave-cc-pod 1/1 Running 0 22s
|
||||
```
|
||||
|
||||
Check the pod is running as expected:
|
||||
```
|
||||
kubectl logs enclave-cc-pod | head -5
|
||||
```
|
||||
Output:
|
||||
```
|
||||
["init"]
|
||||
Hello world!
|
||||
|
||||
Hello world!
|
||||
|
||||
```
|
||||
|
||||
We can also verify the host does not have the image for others to use:
|
||||
```
|
||||
crictl -r unix:///run/containerd/containerd.sock image ls | grep helloworld_enc
|
||||
97
guides/ephemeral_storage.md
Normal file
97
guides/ephemeral_storage.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Trusted Ephemeral Storage for container images
|
||||
|
||||
With CoCo, container images are pulled inside the guest VM.
|
||||
By default container images are saved in guest memory which is protected by CC hardware.
|
||||
Since memory is an expensive resource, CoCo implemented [trusted ephemeral storage](https://github.com/confidential-containers/documentation/issues/39) for container image and RW layer.
|
||||
|
||||
This solution is verified with Kubernetes CSI driver [open-local](https://github.com/alibaba/open-local). Please follow this [user guide](https://github.com/alibaba/open-local/blob/main/docs/user-guide/user-guide.md) to install open-local.
|
||||
|
||||
We can use following example `trusted_store_cc.yaml` to have a try:
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: trusted-lvm-block
|
||||
spec:
|
||||
runtimeClassName: kata-qemu-tdx
|
||||
containers:
|
||||
- name: sidecar-trusted-store
|
||||
image: pause
|
||||
volumeDevices:
|
||||
- devicePath: "/dev/trusted_store"
|
||||
name: trusted-store
|
||||
- name: application
|
||||
image: busybox
|
||||
command:
|
||||
- sh
|
||||
- "-c"
|
||||
- |
|
||||
sleep 10000
|
||||
volumes:
|
||||
- name: trusted-store
|
||||
persistentVolumeClaim:
|
||||
claimName: trusted-store-block-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: trusted-store-block-pvc
|
||||
spec:
|
||||
volumeMode: Block
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
storageClassName: open-local-lvm
|
||||
```
|
||||
Before deploy the workload, we can follow this [documentation](https://github.com/kata-containers/kata-containers/blob/CCv0/docs/how-to/how-to-build-and-test-ccv0.md) and use [ccv0.sh](https://github.com/kata-containers/kata-containers/blob/CCv0/docs/how-to/ccv0.sh) to enable CoCo console debug(optional, check whether working as expected).
|
||||
|
||||
Create the workload:
|
||||
```
|
||||
kubectl apply -f trusted_store_cc.yaml
|
||||
```
|
||||
|
||||
Ensure the pod was created successfully (in running state):
|
||||
```
|
||||
kubectl get pods
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
trusted-lvm-block 2/2 Running 0 31s
|
||||
```
|
||||
|
||||
After we enable the debug option, we can login into the VM with `ccv0.sh` script:
|
||||
```
|
||||
./ccv0.sh -d open_kata_shell
|
||||
```
|
||||
|
||||
Check container image is saved in encrypted storage with following commands:
|
||||
```
|
||||
root@localhost:/# lsblk --fs
|
||||
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
|
||||
sda
|
||||
└─ephemeral_image_encrypted_disk 906M 0% /run/image
|
||||
|
||||
root@localhost:/# cryptsetup status ephemeral_image_encrypted_disk
|
||||
/dev/mapper/ephemeral_image_encrypted_disk is active and is in use.
|
||||
type: LUKS2
|
||||
cipher: aes-xts-plain64
|
||||
keysize: 512 bits
|
||||
key location: dm-crypt
|
||||
device: /dev/sda
|
||||
sector size: 4096
|
||||
offset: 32768 sectors
|
||||
size: 2064384 sectors
|
||||
mode: read/write
|
||||
|
||||
root@localhost:/# mount|grep image
|
||||
/dev/mapper/ephemeral_image_encrypted_disk on /run/image type ext4 (rw,relatime)
|
||||
|
||||
root@localhost:/# ls /run/image/
|
||||
layers lost+found overlay
|
||||
```
|
||||
|
||||
|
||||
90
guides/nontee_demo.md
Normal file
90
guides/nontee_demo.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Encrypted Container Images without Hardware Support
|
||||
|
||||
Without Confidential Computing hardware, there is no way to securely provision
|
||||
the keys for an encrypted image. Nonetheless, in this demo we describe how to
|
||||
test encrypted images suppot with the nontee `kata`/`kata-qemu` runtimeclass.
|
||||
|
||||
## Creating a CoCo workload using a pre-existing encrypted image
|
||||
|
||||
We will now proceed to download and run a sample encrypted container image using the CoCo building blocks.
|
||||
|
||||
A demo container image is provided at [docker.io/katadocker/ccv0-ssh](https://hub.docker.com/r/katadocker/ccv0-ssh).
|
||||
It is encrypted with [Attestation Agent](https://github.com/confidential-containers/attestation-agent)'s [offline file system key broker](https://github.com/confidential-containers/attestation-agent/tree/64c12fbecfe90ba974d5fe4896bf997308df298d/src/kbc_modules/offline_fs_kbc) and [`aa-offline_fs_kbc-keys.json`](https://github.com/confidential-containers/documentation/blob/main/demos/ssh-demo/aa-offline_fs_kbc-keys.json) as its key file.
|
||||
|
||||
We have prepared a sample CoCo operator custom resource that is based on the standard `ccruntime.yaml`, but in addition has the the decryption keys and configuration required to decrypt this sample container image.
|
||||
> **Note** All pods started with this sample resource will be able to decrypt the sample container and all keys shown are for demo purposes only and should not be used in production.
|
||||
|
||||
To test out creating a workload from the sample encrypted container image, we can take the following steps:
|
||||
|
||||
### Swap out the standard custom resource for our sample
|
||||
|
||||
Support for multiple custom resources in not available in the current release. Consequently, if a custom resource already exists, then you'll need to remove it first before deploying a new one. We can remove the standard custom resource with:
|
||||
```
|
||||
kubectl delete -k github.com/confidential-containers/operator/config/samples/ccruntime/<CCRUNTIME_OVERLAY>?ref=<RELEASE_VERSION>
|
||||
```
|
||||
and in it's place install the modified version with the sample container's decryption key:
|
||||
```
|
||||
kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/ssh-demo?ref=<RELEASE_VERSION>
|
||||
```
|
||||
Wait until each pod has the STATUS of Running.
|
||||
```
|
||||
kubectl get pods -n confidential-containers-system --watch
|
||||
```
|
||||
### Test creating a workload from the sample encrypted image
|
||||
|
||||
Create a new Kubernetes deployment that uses the `docker.io/katadocker/ccv0-ssh` container image with:
|
||||
```
|
||||
cat << EOF > ccv0-ssh-demo.yaml
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: ccv0-ssh
|
||||
spec:
|
||||
selector:
|
||||
app: ccv0-ssh
|
||||
ports:
|
||||
- port: 22
|
||||
---
|
||||
kind: Deployment
|
||||
apiVersion: apps/v1
|
||||
metadata:
|
||||
name: ccv0-ssh
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: ccv0-ssh
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: ccv0-ssh
|
||||
spec:
|
||||
runtimeClassName: kata
|
||||
containers:
|
||||
- name: ccv0-ssh
|
||||
image: docker.io/katadocker/ccv0-ssh
|
||||
imagePullPolicy: Always
|
||||
EOF
|
||||
```
|
||||
|
||||
Apply this with:
|
||||
```
|
||||
kubectl apply -f ccv0-ssh-demo.yaml
|
||||
```
|
||||
and waiting for the pod to start. This process should show that we are able to pull the encrypted image and using the decryption key configured in the CoCo sample guest image decrypt the container image and create a workload using it.
|
||||
|
||||
The demo image has an SSH host key embedded in it, which is protected by it's encryption, but we can download the sample private key and use this to ssh into the container and validate the host key to ensure that it hasn't been tampered with.
|
||||
|
||||
Download the SSH key with:
|
||||
```
|
||||
curl -Lo ccv0-ssh https://raw.githubusercontent.com/confidential-containers/documentation/main/demos/ssh-demo/ccv0-ssh
|
||||
```
|
||||
Ensure that the permissions are set correctly with:
|
||||
```
|
||||
chmod 600 ccv0-ssh
|
||||
```
|
||||
|
||||
We can then use the key to ssh into the container:
|
||||
```
|
||||
$ ssh -i ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}")
|
||||
```
|
||||
You will be prompted about whether the host key fingerprint is correct. This fingerprint should match the one specified in the container image: `wK7uOpqpYQczcgV00fGCh+X97sJL3f6G1Ku4rvlwtR0.`
|
||||
174
guides/troubleshooting.md
Normal file
174
guides/troubleshooting.md
Normal file
@@ -0,0 +1,174 @@
|
||||
# Troubleshooting
|
||||
|
||||
Confidential Containers integrates several components. If you run into problems,
|
||||
it can sometimes be difficult to figure out what is going on or how to move forward.
|
||||
Here are some tips.
|
||||
|
||||
If you get stuck or find a bug, please make an issue on this repository or
|
||||
the repository for the component in question, e.g.,
|
||||
[the operator](https://github.com/confidential-containers/operator/issues).
|
||||
|
||||
## Kubernetes
|
||||
|
||||
To figure out which basic area you problem is in, first make sure that your Kubernetes
|
||||
cluster can schedule non-confidential workloads on your worker node. Remove the `kata-*`
|
||||
runtime class from your pod yaml and try to run a pod. If your pod still doesn't run,
|
||||
please refer to a more general Kubernetes troubleshooting guide.
|
||||
|
||||
If your cluster is healthy but you cannot start confidential containers, you might
|
||||
be able get some helpful information from Kubernetes.
|
||||
Try `kubectl describe pod <your-pod>`
|
||||
Sometimes this will give you a useful message pointing to a failed attestation
|
||||
or some sort of missing environment setup. Most of the time you will see a
|
||||
generic message such as the following:
|
||||
|
||||
```
|
||||
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: Failed to Check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 637456061:1024: unknown
|
||||
```
|
||||
|
||||
Unfortunately this is a generic message. You'll need to go deeper to figure out
|
||||
what is going on.
|
||||
|
||||
## CoCo Debugging
|
||||
|
||||
A good next step is to figure out if things are breaking before or after the VM boots.
|
||||
You can see if there is a hypervisor process running with something like this.
|
||||
```bash
|
||||
ps -ef | grep qemu
|
||||
```
|
||||
|
||||
If you are using a different hypervisor, adjust command accordingly.
|
||||
If there are no hypervisor processes running on the worker node, the VM has
|
||||
either failed to start or was shutdown. If there is a hypervisor process,
|
||||
the problem is probably inside the guest.
|
||||
|
||||
Now is a good time to enable debug output for Kata and containerd.
|
||||
To do this, first look at the containerd config file located at
|
||||
`/etc/containerd/config.toml`. At the bottom of the file there should
|
||||
be a section for each runtime class. For example:
|
||||
|
||||
```
|
||||
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev]
|
||||
cri_handler = "cc"
|
||||
runtime_type = "io.containerd.kata-qemu-sev.v2"
|
||||
privileged_without_host_devices = true
|
||||
pod_annotations = ["io.katacontainers.*"]
|
||||
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev.options]
|
||||
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-sev.toml"
|
||||
```
|
||||
|
||||
The `ConfigPath` entry on the final line shows the path to the Kata configuration file that will be used
|
||||
for that runtime class.
|
||||
|
||||
While you are looking at the containerd config, find the `[debug]` section near the top and set `level`
|
||||
to `debug`. Make sure to restart containerd after editing the containerd config file.
|
||||
You can do this with `sudo systemctl restart containerd`.
|
||||
|
||||
Now go to the Kata config file that matches your runtime class and enable every debug option available.
|
||||
You do not need to restart any daemons when changing the Kata config file; just run another pod
|
||||
or hope that Kubernetes restarts your existing pod. Note that enabling debug options in the Kata
|
||||
config file can change the attestation evidence of a confidential guest.
|
||||
|
||||
Now you should be able to view logs from containerd with the following:
|
||||
```
|
||||
sudo journalctl -xeu containerd
|
||||
```
|
||||
|
||||
Kata writes many messages to this log. It's good to know what you're looking for. There are many
|
||||
generic messages that are not significant, often arising from a VM not shutting down cleanly
|
||||
after an unrelated issue.
|
||||
|
||||
### VM Doesn't Start
|
||||
If the VM has failed to start, you might have a problem with confidential
|
||||
computing support on your worker node. Make sure that you can start
|
||||
confidential VMs without confidential containers.
|
||||
|
||||
Check the containerd log for any obvious errors regarding VM boot.
|
||||
Try searching the log for the string `error` or for the name
|
||||
of your hypervisor i.e. `qemu` or `qemu-system-x86_64`.
|
||||
|
||||
If there are no obvious errors, try finding the hypervisor
|
||||
commandline. This should be in the containerd log if you have enabled
|
||||
debug messages correctly.
|
||||
|
||||
It might be tempting to try running the hypervisor command directly
|
||||
from the command line, but this usually isn't productive. Instead,
|
||||
try starting a standalone VM using the same kernel, initrd/disk,
|
||||
command line, firmware, and hypervisor that Kata uses.
|
||||
This might uncover some kind of system misconfiguration.
|
||||
You can also find these values in the Kata config file, but looking
|
||||
in the log is more direct.
|
||||
|
||||
Another way to print the hypervisor command is to create a bash script
|
||||
that prints any arguments it is called with to a file. Then modify the
|
||||
Kata config file so that the hypervisor path points to this scipt
|
||||
rather than to the hypervisor. This method can also be used to add
|
||||
additional parameters to the command line. Just have the bash script
|
||||
call the hypervisor with whatever arguments it received plus any that
|
||||
you want to add. This could be useful for enabling debugging or tracing
|
||||
flags in your hypervisor. For instance, if you are using QEMU and SEV
|
||||
you might want to add the argument `--trace 'kvm_sev_*'`. Make sure
|
||||
that QEMU was built with an appropriate tracing backend.
|
||||
|
||||
### VM Does Start
|
||||
|
||||
If the VM does start, search the containerd log for the string `vmconsole`.
|
||||
This will show you any guest serial output. You might see some errors
|
||||
coming from the kernel as the guest tries to boot. You might also see the
|
||||
Kata agent starting. If the Kata agent has started, you can match
|
||||
the output to the source to get some clues about what is happening.
|
||||
You might also see something more obvious, like a panic coming from
|
||||
the Kata agent.
|
||||
|
||||
#### failed to create shim task: failed to mount "/run/kata-containers/shared/containers/CONTAINER_NAME/rootfs"
|
||||
|
||||
If your CoCo Pod gets an error like showed below then it is likely the image pull policy is set to **IfNotPresent** and the image has been found in the kubelet cache. It fails because the container runtime will not delegate to the Kata agent to pull the image inside the VM and the agent in turn will try to mount the bundle rootfs that only exist in the host filesystem.
|
||||
|
||||
Therefore, you must ensure that the image pull policy is set to **Always** for any CoCo Pod. This ways the images are always handled entirely by the agent inside the VM. Worth mentioning we recognize that this behavior is suboptimal and so the community has worked on solutions to avoid constant images downloads for each and every workload.
|
||||
|
||||
```
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal Scheduled 20s default-scheduler Successfully assigned default/coco-fedora-69d9f84cd7-j597j to virtlab1012
|
||||
Normal Pulled 5s (x3 over 19s) kubelet Container image "docker.io/wainersm/coco-fedora_sshd@sha256:a7108f9f0080c429beb66e2cf0abff143c9eb9c7cf4dcde3241bc56c938d33b9" already present on machine
|
||||
Normal Created 5s (x3 over 19s) kubelet Created container coco-fedora
|
||||
Warning Failed 5s (x3 over 19s) kubelet Error: failed to create containerd task: failed to create shim task: failed to mount "/run/kata-containers/shared/containers/coco-fedora/rootfs" to "/run/kata-containers/coco-fedora/rootfs", with error: ENOENT: No such file or directory: unknown
|
||||
Warning BackOff 4s (x3 over 18s) kubelet Back-off restarting failed container
|
||||
```
|
||||
|
||||
#### Debug Console
|
||||
|
||||
One very useful deugging tool is the Kata guest debug console. You can
|
||||
enable this by editing the Kata agent configuration file and adding the lines
|
||||
``` toml
|
||||
debug_console = true
|
||||
debug_console_vport = 1026
|
||||
```
|
||||
|
||||
Enabling the debug console via the Kata Configuration file will overwrite
|
||||
any settings in the agent configuration file in the guest initrd.
|
||||
Enabling the debug console will change the launch measurement.
|
||||
|
||||
Once you've started a pod with the new configuration, get the id of the pod
|
||||
you want to access. Do this via `ps -ef | grep qemu` or equivalent.
|
||||
The id is the long id that shows up in many different arguments.
|
||||
It should look like `1a9ab65be63b8b03dfd0c75036d27f0ed09eab38abb45337fea83acd3cd7bacd`.
|
||||
Once you have the id, you can use it to access the debug console.
|
||||
```
|
||||
sudo /opt/confidential-containers/bin/kata-runtime exec <id>
|
||||
```
|
||||
You might need to symlink the appropriate Kata configuration file for your runtime
|
||||
class if the `kata-runtime` tries to look at the wrong one.
|
||||
|
||||
The debug console gives you access to the guest VM. This is a great way to
|
||||
investigate missing dependencies or incorrect configurations.
|
||||
|
||||
#### Guest Firmware Logs
|
||||
|
||||
If the VM is running but there is no guest output in the log,
|
||||
the guest might have stalled in the firmware. Firmware output will
|
||||
depend on your firmware and hypervisor. If you are using QEMU and OVMF,
|
||||
you can see the OVMF output by adding `-global isa-debugcon.iobase=0x402`
|
||||
and `-debugcon file:/tmp/ovmf.log` to the QEMU command line using the
|
||||
redirect script described above.
|
||||
585
quickstart.md
585
quickstart.md
@@ -1,28 +1,40 @@
|
||||
# Getting Started
|
||||
This document contains an overview of Confidential Containers use cases and support
|
||||
as well as a guide for installing Confidential Containers, deploying workloads,
|
||||
and troubleshooting if things go wrong.
|
||||
|
||||
## Use cases
|
||||
Confidential Containers (CoCo) provides a uniform workflow, trust model, and feature set
|
||||
across a wide array of platforms and deployment models.
|
||||
|
||||
Confidential Containers (CoCo) supports the following use cases:
|
||||
In general, using CoCo involves the following steps:
|
||||
|
||||
- Running unencrypted containers without Confidential Computing (CC) hardware
|
||||
- Running encrypted containers without CC HW (sample container images provided)
|
||||
- Running unencrypted container images or sample encrypted images with CC HW
|
||||
- Running your own encrypted container images with CC HW
|
||||
- Install CoCo using the operator
|
||||
- Add a runtimeClassName to your pod yaml
|
||||
- Deploy signed/encrypted container images (optional)
|
||||
- Setup attestation (optional)
|
||||
|
||||
The first two cases are mainly for testing and development or new users who want to
|
||||
explore the project. This guide explains all four cases below.
|
||||
|
||||
## Hardware Support and Limitations
|
||||
This guide walks through these steps and touches on some platform-specific configurations.
|
||||
For more advanced features, specific hardware setup, and troubleshooting information,
|
||||
see the [guides](./guides) directory.
|
||||
|
||||
Confidential Containers is still maturing. See [release notes](./releases) for currrent
|
||||
hardware support and limitations.
|
||||
|
||||
# Installing
|
||||
# Installation
|
||||
|
||||
You can enable Confidential Containers in an existing Kubernetes cluster using the Confidential Containers Operator.
|
||||
When installation is finished, your cluster will have new runtime classes for different hardware platforms,
|
||||
including a generic runtime for testing CoCo without confidential hardware support, a runtime using a remote hypervisor
|
||||
that allows for cloud integration, a runtime for process-based isolation using SGX, as well as runtimes for TDX and SEV.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
To run the operator you must have an existing Kubernetes cluster that meets the followng requirements.
|
||||
|
||||
- Ensure a minimum of 8GB RAM and 4 vCPU for the Kubernetes cluster node
|
||||
- Only containerd runtime based Kubernetes clusters are supported with the current CoCo release
|
||||
- The minimum Kubernetes version should be 1.24
|
||||
- Ensure at least one Kubernetes node in the cluster is having the label `node-role.kubernetes.io/worker=`
|
||||
- Ensure SELinux is disabled or not enforced (https://github.com/confidential-containers/operator/issues/115)
|
||||
|
||||
For more details on the operator, including the custom resources managed by the operator, refer to the operator [docs](https://github.com/confidential-containers/operator).
|
||||
|
||||
> **Note** If you need to quickly deploy a single-node test cluster, you can
|
||||
use the [run-local.sh
|
||||
@@ -40,19 +52,9 @@ Also, with the `enclave-cc` runtime class, the cluster must be prepared so that
|
||||
on the worker nodes is **not** on an overlayfs mount but the path is a `hostPath` mount (see
|
||||
[a sample configuration](https://github.com/confidential-containers/operator/blob/cf6a4f38114f7c5b71daec6cb666b1b40bcea140/tests/e2e/enclave-cc-kind-config.yaml#L6-L8))
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Ensure a minimum of 8GB RAM and 4 vCPU for the Kubernetes cluster node
|
||||
- Only containerd runtime based Kubernetes clusters are supported with the current CoCo release
|
||||
- The minimum Kubernetes version should be 1.24
|
||||
- Ensure at least one Kubernetes node in the cluster is having the label `node-role.kubernetes.io/worker=`
|
||||
- Ensure SELinux is disabled or not enforced (https://github.com/confidential-containers/operator/issues/115)
|
||||
|
||||
For more details on the operator, including the custom resources managed by the operator, refer to the operator [docs](https://github.com/confidential-containers/operator).
|
||||
|
||||
## Operator Installation
|
||||
|
||||
### Deploy the the operator
|
||||
### Deploy the operator
|
||||
|
||||
Deploy the operator by running the following command where `<RELEASE_VERSION>` needs to be substituted
|
||||
with the desired [release tag](https://github.com/confidential-containers/operator/tags).
|
||||
@@ -99,6 +101,25 @@ Wait until each pod has the STATUS of Running.
|
||||
kubectl get pods -n confidential-containers-system --watch
|
||||
```
|
||||
|
||||
#### Create custom resource for enclave-cc
|
||||
|
||||
**Note** For `enclave-cc` certain configuration changes, such as setting the
|
||||
URI of the KBS, must be made **before** applying the custom resource.
|
||||
Please see the [enclave-cc guide](./guides/enclave-cc.md) for more information.
|
||||
|
||||
`enclave-cc` is a form of Confidential Containers that uses process-based isolation.
|
||||
`enclave-cc` can be installed with the following custom resources.
|
||||
```
|
||||
kubectl apply -k github.com/confidential-containers/operator/config/samples/enclave-cc/sim?ref=<RELEASE_VERSION>
|
||||
```
|
||||
or
|
||||
```
|
||||
kubectl apply -k github.com/confidential-containers/operator/config/samples/enclave-cc/hw?ref=<RELEASE_VERSION>
|
||||
```
|
||||
for the **simulated** SGX mode build or **hardware** SGX mode build, respectively.
|
||||
|
||||
### Verify Installation
|
||||
|
||||
Check the `RuntimeClasses` that got created.
|
||||
|
||||
```
|
||||
@@ -124,18 +145,7 @@ Details on each of the runtime classes:
|
||||
- *kata-qemu-tdx* - using QEMU, with TDVF, and support for Intel TDX CC HW, prepared for using Verdictd and EAA KBC.
|
||||
- *kata-qemu-sev* - using QEMU, and support for AMD SEV HW
|
||||
|
||||
For the process based CoCo TEE (aka. `enclave-cc`) the operator setup steps are the same and the custom resources
|
||||
can be deployed using either
|
||||
```
|
||||
kubectl apply -k github.com/confidential-containers/operator/config/samples/enclave-cc/sim?ref=<RELEASE_VERSION>
|
||||
```
|
||||
or
|
||||
```
|
||||
kubectl apply -k github.com/confidential-containers/operator/config/samples/enclave-cc/hw?ref=<RELEASE_VERSION>
|
||||
```
|
||||
for the **simulated** SGX mode build or **hardware** SGX mode build, respectively.
|
||||
|
||||
These result in a `RuntimeClass` as follows:
|
||||
If you are using `enclave-cc` you should see the following runtime classes.
|
||||
|
||||
```
|
||||
kubectl get runtimeclass
|
||||
@@ -146,24 +156,12 @@ NAME HANDLER AGE
|
||||
enclave-cc enclave-cc 9m55s
|
||||
```
|
||||
|
||||
### Configuring enclave-cc custom resource to use a different KBC
|
||||
### Platform Setup
|
||||
|
||||
**Note** The KBC configuration changes to the enclave-cc custom resource yaml
|
||||
must be made **before** deploying it.
|
||||
|
||||
Enclave CC supports Verdictd and in order to use it, users will have to
|
||||
properly configure a decrypt_config.conf, in order to set the `KBC` (`sample_kbc`
|
||||
or `eaa_kbc`) `IP`,`PORT`, and the `SECURITY_VALIDATE` (`false` or `true`)
|
||||
```
|
||||
{
|
||||
"key_provider": "provider:attestation-agent:KBC::IP:PORT",
|
||||
"security_validate": SECURITY_VALIDATE
|
||||
}
|
||||
```
|
||||
|
||||
Once that's set according to the users needs, the user will then have to run:
|
||||
`cat decrypt_config.conf | base64 -w 0` in order to get the data encoded and
|
||||
set it accordingly [here](https://github.com/confidential-containers/operator/blob/6f241fbc056f0a5d9e1bd2c10b2cedc0782b99ff/config/samples/enclave-cc/base/ccruntime-enclave-cc.yaml#L124).
|
||||
While the operator deploys all the required binaries and artifacts and sets up runtime classes that use them,
|
||||
certain platforms may require additional configuration to enable confidential computing. For example, the host
|
||||
kernel and firmware might need to be configured.
|
||||
See the [guides](./guides) for more information.
|
||||
|
||||
# Running a workload
|
||||
|
||||
@@ -189,6 +187,10 @@ spec:
|
||||
runtimeClassName: kata
|
||||
```
|
||||
|
||||
Setting the runtimeClassName is usually the only change needed to the pod yaml, but some platforms
|
||||
support additional annotations for configuring the enclave. See the [guides](./guides) for
|
||||
more details.
|
||||
|
||||
With Confidential Containers, the workload container images are never downloaded on the host.
|
||||
For verifying that the container image doesn’t exist on the host you should log into the k8s node and ensure the following command returns an empty result:
|
||||
```
|
||||
@@ -222,184 +224,42 @@ Now go back to the k8s node and ensure that you still don’t have any bitnami/n
|
||||
root@cluster01-master-0:/home/ubuntu# crictl -r unix:///run/containerd/containerd.sock image ls | grep bitnami/nginx
|
||||
```
|
||||
|
||||
## Creating a sample CoCo workload using enclave-cc
|
||||
## Encrypted and/or signed images with attestation
|
||||
|
||||
Following the previous example that used the `kata` runtime class, we setup a sample *hello world*
|
||||
workload with an encrypted and cosign signed container image using the `enclave-cc` runtime class for process based TEEs.
|
||||
The deployment below assumes the hardware SGX mode build is installed by the operator. To try on a non-TEE system, please
|
||||
use simulate SGX mode build.
|
||||
The previous example does not involve any attestation because the workload container isn't signed or encrypted
|
||||
and the workload itself does not require any secrets.
|
||||
|
||||
The example uses a trivial hello world C application:
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: enclave-cc-pod
|
||||
spec:
|
||||
containers:
|
||||
- image: docker.io/eqmcc/helloworld_enc
|
||||
name: hello-world
|
||||
workingDir: "/run/rune/boot_instance/"
|
||||
resources:
|
||||
limits:
|
||||
sgx.intel.com/epc: 600Mi
|
||||
env:
|
||||
- name: OCCLUM_RELEASE_ENCLAVE
|
||||
value: "1"
|
||||
command:
|
||||
- /run/rune/boot_instance/build/bin/occlum-run
|
||||
- /bin/hello_world
|
||||
runtimeClassName: enclave-cc
|
||||
This is not the case for most real workloads. It is recommended to use CoCo with signed and/or encrypted images.
|
||||
The workload itself can also request secrets from the attestation agent in the guest.
|
||||
|
||||
```
|
||||
Secrets are provisioned to the guest in conjunction with an attestation, which is based on hardware evidence.
|
||||
The rest of this guide focuses on setting up more substantial encrypted/signed workloads using attestation
|
||||
and confidential hardware.
|
||||
|
||||
**Note** When the hardware SGX mode payload is used in an SGX enabled cluster, `sgx.intel.com/epc: 600Mi`
|
||||
resource request must be added to the pod spec.
|
||||
See [this guide](./guides/nontee_demo.md) if you would like to deploy an example encrypted image without
|
||||
confidential hardware.
|
||||
|
||||
Again, create a pod YAML file as previously described (this time we named it `enclave-cc-pod.yaml`) .
|
||||
CoCo has a modular attestation interface and there are a few options for attestation.
|
||||
CoCo provides a generic Key Broker Service (KBS) that the rest of this guide will be focused on.
|
||||
The SEV runtime class uses `simple-kbs`, which is described in the [SEV guide](./guides/sev.md).
|
||||
There is also `eaa_kbc`/`verdictd` which is described [here](./guides/eaa_verdictd.md).
|
||||
|
||||
Create the workload:
|
||||
```
|
||||
kubectl apply -f enclave-cc-pod.yaml
|
||||
```
|
||||
Output:
|
||||
```
|
||||
pod/enclave-cc-pod created
|
||||
```
|
||||
### Select Runtime Class
|
||||
|
||||
Ensure the pod was created successfully (in running state):
|
||||
```
|
||||
kubectl get pods
|
||||
```
|
||||
Output:
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
enclave-cc-pod 1/1 Running 0 22s
|
||||
```
|
||||
To use CoCo with confidential hardware, first switch to the appropriate runtime class.
|
||||
TDX has two runtime classes, `kata-qemu-tdx` and `kata-clh-tdx`. One uses QEMU as VMM and TDVF as firmware. The other uses Cloud Hypervisor as VMM and TD-Shim as firmware.
|
||||
|
||||
Check the pod is running as expected:
|
||||
```
|
||||
kubectl logs enclave-cc-pod | head -5
|
||||
```
|
||||
Output:
|
||||
```
|
||||
["init"]
|
||||
Hello world!
|
||||
For SEV(-ES) use the `kata-qemu-sev` runtime class and follow the [SEV guide](./guides/sev.md).
|
||||
|
||||
Hello world!
|
||||
|
||||
```
|
||||
|
||||
We can also verify the host does not have the image for others to use:
|
||||
```
|
||||
crictl -r unix:///run/containerd/containerd.sock image ls | grep helloworld_enc
|
||||
```
|
||||
|
||||
## Creating a CoCo workload using a pre-existing encrypted image
|
||||
|
||||
We will now proceed to download and run a sample encrypted container image using the CoCo building blocks.
|
||||
|
||||
A demo container image is provided at [docker.io/katadocker/ccv0-ssh](https://hub.docker.com/r/katadocker/ccv0-ssh).
|
||||
It is encrypted with [Attestation Agent](https://github.com/confidential-containers/attestation-agent)'s [offline file system key broker](https://github.com/confidential-containers/attestation-agent/tree/64c12fbecfe90ba974d5fe4896bf997308df298d/src/kbc_modules/offline_fs_kbc) and [`aa-offline_fs_kbc-keys.json`](https://github.com/confidential-containers/documentation/blob/main/demos/ssh-demo/aa-offline_fs_kbc-keys.json) as its key file.
|
||||
|
||||
We have prepared a sample CoCo operator custom resource that is based on the standard `ccruntime.yaml`, but in addition has the the decryption keys and configuration required to decrypt this sample container image.
|
||||
> **Note** All pods started with this sample resource will be able to decrypt the sample container and all keys shown are for demo purposes only and should not be used in production.
|
||||
|
||||
To test out creating a workload from the sample encrypted container image, we can take the following steps:
|
||||
|
||||
### Swap out the standard custom resource for our sample
|
||||
|
||||
Support for multiple custom resources in not available in the current release. Consequently, if a custom resource already exists, then you'll need to remove it first before deploying a new one. We can remove the standard custom resource with:
|
||||
```
|
||||
kubectl delete -k github.com/confidential-containers/operator/config/samples/ccruntime/<CCRUNTIME_OVERLAY>?ref=<RELEASE_VERSION>
|
||||
```
|
||||
and in it's place install the modified version with the sample container's decryption key:
|
||||
```
|
||||
kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/ssh-demo?ref=<RELEASE_VERSION>
|
||||
```
|
||||
Wait until each pod has the STATUS of Running.
|
||||
```
|
||||
kubectl get pods -n confidential-containers-system --watch
|
||||
```
|
||||
### Test creating a workload from the sample encrypted image
|
||||
|
||||
Create a new Kubernetes deployment that uses the `docker.io/katadocker/ccv0-ssh` container image with:
|
||||
```
|
||||
cat << EOF > ccv0-ssh-demo.yaml
|
||||
kind: Service
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: ccv0-ssh
|
||||
spec:
|
||||
selector:
|
||||
app: ccv0-ssh
|
||||
ports:
|
||||
- port: 22
|
||||
---
|
||||
kind: Deployment
|
||||
apiVersion: apps/v1
|
||||
metadata:
|
||||
name: ccv0-ssh
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: ccv0-ssh
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: ccv0-ssh
|
||||
spec:
|
||||
runtimeClassName: kata
|
||||
containers:
|
||||
- name: ccv0-ssh
|
||||
image: docker.io/katadocker/ccv0-ssh
|
||||
imagePullPolicy: Always
|
||||
EOF
|
||||
```
|
||||
|
||||
Apply this with:
|
||||
```
|
||||
kubectl apply -f ccv0-ssh-demo.yaml
|
||||
```
|
||||
and waiting for the pod to start. This process should show that we are able to pull the encrypted image and using the decryption key configured in the CoCo sample guest image decrypt the container image and create a workload using it.
|
||||
|
||||
The demo image has an SSH host key embedded in it, which is protected by it's encryption, but we can download the sample private key and use this to ssh into the container and validate the host key to ensure that it hasn't been tampered with.
|
||||
|
||||
Download the SSH key with:
|
||||
```
|
||||
curl -Lo ccv0-ssh https://raw.githubusercontent.com/confidential-containers/documentation/main/demos/ssh-demo/ccv0-ssh
|
||||
```
|
||||
Ensure that the permissions are set correctly with:
|
||||
```
|
||||
chmod 600 ccv0-ssh
|
||||
```
|
||||
|
||||
We can then use the key to ssh into the container:
|
||||
```
|
||||
$ ssh -i ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}")
|
||||
```
|
||||
You will be prompted about whether the host key fingerprint is correct. This fingerprint should match the one specified in the container image: `wK7uOpqpYQczcgV00fGCh+X97sJL3f6G1Ku4rvlwtR0.`
|
||||
|
||||
## Building an encrypted container image and deploying it as a CoCo workload on CC HW
|
||||
|
||||
For running one of the sample workloads provided in the previous step, but now taking advantage of a specific TEE vendor,
|
||||
the user will have to set the runtime class of the workload accordingly in the workload yaml file.
|
||||
|
||||
### TDX
|
||||
In case the user wants to run the workload on a TDX capable hardware, using QEMU (which uses TDVF as its firmware) the `kata-qemu-tdx` runtime class must be specified. In case the user prefers using Cloud Hypervisor (which uses TD-Shim as its firmware) then the `kata-clh-tdx` runtime class must be specified.
|
||||
|
||||
Now you can use CoCo Key Broker System cluster to create encrypted container image and deploying it, refer to: [Deploy and Configure tenant-side Key Broker System cluster](#deploy-and-configure-tenant-side-coco-key-broker-system-cluster).
|
||||
You also can use [EAA Verdictd](guides/eaa-verdictd-guide.md) to do that, refer to [Verdictd guide](guides/eaa-verdictd-guide.md).
|
||||
|
||||
### SEV
|
||||
The `kata-qemu-sev` runtime class must be specified.
|
||||
|
||||
Now you can use [simple-kbs](guides/sev-guide.md) to create encrypted container image and depolying it on SEV.
|
||||
Please refer to [sev-guide.md](guides/sev-guide.md)
|
||||
For `enclave-cc` follow the [enclave-cc guide](./guides/enclave-cc.md).
|
||||
|
||||
### Deploy and Configure tenant-side CoCo Key Broker System cluster
|
||||
|
||||
A tenant-side CoCo Key Broker System cluster include:
|
||||
The following describes how to run and provision the generic KBS.
|
||||
The KBS should be run in a trusted environment. The KBS is not just one service,
|
||||
but a combination of several.
|
||||
|
||||
A tenant-side CoCo Key Broker System cluster includes:
|
||||
- Key Broker Service (KBS): Brokering service for confidential resources.
|
||||
- Attestation Service (AS): Verifier for remote attestation.
|
||||
- Reference Value Provicer Service (RVPS): Provides reference values for AS.
|
||||
@@ -431,11 +291,10 @@ When KBS cluster is running, you can modify the policy file used by AS policy en
|
||||
|
||||
- `$KBS_DIR_PATH/data/attestation-service/opa/policy.rego`: Policy file for evidence verification of AS, refer to [AS Policy Engine](https://github.com/confidential-containers/attestation-service#policy-engine) for more infomation.
|
||||
|
||||
### Encrypt Image
|
||||
### Encrypting an Image
|
||||
|
||||
[skopeo](https://github.com/containers/skopeo) is required to encrypt the container image. Follow the instructions here to install `skopeo`:
|
||||
|
||||
[skopeo Installation](https://github.com/containers/skopeo/blob/main/install.md)
|
||||
[skopeo](https://github.com/containers/skopeo) is required to encrypt the container image.
|
||||
Follow the [instructions](https://github.com/containers/skopeo/blob/main/install.md) to install `skopeo`.
|
||||
|
||||
Use `skopeo` to encrypt an image on the same node of the KBS cluster (use busybox:latest for example):
|
||||
|
||||
@@ -452,7 +311,7 @@ tee > ocicrypt.conf <<EOF
|
||||
EOF
|
||||
|
||||
# encrypt the image
|
||||
OCICRYPT_KEYPROVIDER_CONFIG=ocicrypt.conf skopeo copy --insecure-policy --encryption-key provider:attestation-agent docker://library/busybox dir:busybox:encrypted
|
||||
OCICRYPT_KEYPROVIDER_CONFIG=ocicrypt.conf skopeo copy --insecure-policy --encryption-key provider:attestation-agent docker://library/busybox oci:busybox:encrypted
|
||||
```
|
||||
|
||||
The image will be encrypted, and things happens in the KBS cluster background include:
|
||||
@@ -463,11 +322,11 @@ The image will be encrypted, and things happens in the KBS cluster background in
|
||||
Then push the image to registry:
|
||||
|
||||
```shell
|
||||
skopeo copy dir:busybox:encrypted [SCHEME]://[REGISTRY_URL]:encrypted
|
||||
skopeo copy oci:busybox:encrypted [SCHEME]://[REGISTRY_URL]:encrypted
|
||||
```
|
||||
Be sure to replace `[SCHEME]` with registry scheme type like `docker`, replace `[REGISTRY_URL]` with the desired registry URL like `docker.io/encrypt_test/busybox`.
|
||||
|
||||
### Sign Image
|
||||
### Signing an Image
|
||||
|
||||
[cosign](https://github.com/sigstore/cosign) is required to sign the container image. Follow the instructions here to install `cosign`:
|
||||
|
||||
@@ -546,275 +405,3 @@ Deploy encrypted image as a workload:
|
||||
```shell
|
||||
kubectl apply -f encrypted-image-test-busybox.yaml
|
||||
```
|
||||
|
||||
# Trusted Ephemeral Storage for container images
|
||||
|
||||
With CoCo, container images are pulled inside the guest VM.
|
||||
By default container images are saved in guest memory which is protected by CC hardware.
|
||||
Since memory is an expensive resource, CoCo implemented [trusted ephemeral storage](https://github.com/confidential-containers/documentation/issues/39) for container image and RW layer.
|
||||
|
||||
This solution is verified with Kubernetes CSI driver [open-local](https://github.com/alibaba/open-local). Please follow this [user guide](https://github.com/alibaba/open-local/blob/main/docs/user-guide/user-guide.md) to install open-local.
|
||||
|
||||
We can use following example `trusted_store_cc.yaml` to have a try:
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: trusted-lvm-block
|
||||
spec:
|
||||
runtimeClassName: kata-qemu-tdx
|
||||
containers:
|
||||
- name: sidecar-trusted-store
|
||||
image: pause
|
||||
volumeDevices:
|
||||
- devicePath: "/dev/trusted_store"
|
||||
name: trusted-store
|
||||
- name: application
|
||||
image: busybox
|
||||
command:
|
||||
- sh
|
||||
- "-c"
|
||||
- |
|
||||
sleep 10000
|
||||
volumes:
|
||||
- name: trusted-store
|
||||
persistentVolumeClaim:
|
||||
claimName: trusted-store-block-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: trusted-store-block-pvc
|
||||
spec:
|
||||
volumeMode: Block
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
storageClassName: open-local-lvm
|
||||
```
|
||||
Before deploy the workload, we can follow this [documentation](https://github.com/kata-containers/kata-containers/blob/CCv0/docs/how-to/how-to-build-and-test-ccv0.md) and use [ccv0.sh](https://github.com/kata-containers/kata-containers/blob/CCv0/docs/how-to/ccv0.sh) to enable CoCo console debug(optional, check whether working as expected).
|
||||
|
||||
Create the workload:
|
||||
```
|
||||
kubectl apply -f trusted_store_cc.yaml
|
||||
```
|
||||
|
||||
Ensure the pod was created successfully (in running state):
|
||||
```
|
||||
kubectl get pods
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
trusted-lvm-block 2/2 Running 0 31s
|
||||
```
|
||||
|
||||
After we enable the debug option, we can login into the VM with `ccv0.sh` script:
|
||||
```
|
||||
./ccv0.sh -d open_kata_shell
|
||||
```
|
||||
|
||||
Check container image is saved in encrypted storage with following commands:
|
||||
```
|
||||
root@localhost:/# lsblk --fs
|
||||
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
|
||||
sda
|
||||
└─ephemeral_image_encrypted_disk 906M 0% /run/image
|
||||
|
||||
root@localhost:/# cryptsetup status ephemeral_image_encrypted_disk
|
||||
/dev/mapper/ephemeral_image_encrypted_disk is active and is in use.
|
||||
type: LUKS2
|
||||
cipher: aes-xts-plain64
|
||||
keysize: 512 bits
|
||||
key location: dm-crypt
|
||||
device: /dev/sda
|
||||
sector size: 4096
|
||||
offset: 32768 sectors
|
||||
size: 2064384 sectors
|
||||
mode: read/write
|
||||
|
||||
root@localhost:/# mount|grep image
|
||||
/dev/mapper/ephemeral_image_encrypted_disk on /run/image type ext4 (rw,relatime)
|
||||
|
||||
root@localhost:/# ls /run/image/
|
||||
layers lost+found overlay
|
||||
```
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
Confidential Containers integrates several components. If you run into problems,
|
||||
it can sometimes be difficult to figure out what is going on or how to move forward.
|
||||
Here are some tips.
|
||||
|
||||
If you get stuck or find a bug, please make an issue on this repository or
|
||||
the repository for the component in question, e.g.,
|
||||
[the operator](https://github.com/confidential-containers/operator/issues).
|
||||
|
||||
## Kubernetes
|
||||
|
||||
To figure out which basic area you problem is in, first make sure that your Kubernetes
|
||||
cluster can schedule non-confidential workloads on your worker node. Remove the `kata-*`
|
||||
runtime class from your pod yaml and try to run a pod. If your pod still doesn't run,
|
||||
please refer to a more general Kubernetes troubleshooting guide.
|
||||
|
||||
If your cluster is healthy but you cannot start confidential containers, you might
|
||||
be able get some helpful information from Kubernetes.
|
||||
Try `kubectl describe pod <your-pod>`
|
||||
Sometimes this will give you a useful message pointing to a failed attestation
|
||||
or some sort of missing environment setup. Most of the time you will see a
|
||||
generic message such as the following:
|
||||
|
||||
```
|
||||
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: Failed to Check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 637456061:1024: unknown
|
||||
```
|
||||
|
||||
Unfortunately this is a generic message. You'll need to go deeper to figure out
|
||||
what is going on.
|
||||
|
||||
## CoCo Debugging
|
||||
|
||||
A good next step is to figure out if things are breaking before or after the VM boots.
|
||||
You can see if there is a hypervisor process running with something like this.
|
||||
```bash
|
||||
ps -ef | grep qemu
|
||||
```
|
||||
|
||||
If you are using a different hypervisor, adjust command accordingly.
|
||||
If there are no hypervisor processes running on the worker node, the VM has
|
||||
either failed to start or was shutdown. If there is a hypervisor process,
|
||||
the problem is probably inside the guest.
|
||||
|
||||
Now is a good time to enable debug output for Kata and containerd.
|
||||
To do this, first look at the containerd config file located at
|
||||
`/etc/containerd/config.toml`. At the bottom of the file there should
|
||||
be a section for each runtime class. For example:
|
||||
|
||||
```
|
||||
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev]
|
||||
cri_handler = "cc"
|
||||
runtime_type = "io.containerd.kata-qemu-sev.v2"
|
||||
privileged_without_host_devices = true
|
||||
pod_annotations = ["io.katacontainers.*"]
|
||||
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu-sev.options]
|
||||
ConfigPath = "/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-sev.toml"
|
||||
```
|
||||
|
||||
The `ConfigPath` entry on the final line shows the path to the Kata configuration file that will be used
|
||||
for that runtime class.
|
||||
|
||||
While you are looking at the containerd config, find the `[debug]` section near the top and set `level`
|
||||
to `debug`. Make sure to restart containerd after editing the containerd config file.
|
||||
You can do this with `sudo systemctl restart containerd`.
|
||||
|
||||
Now go to the Kata config file that matches your runtime class and enable every debug option available.
|
||||
You do not need to restart any daemons when changing the Kata config file; just run another pod
|
||||
or hope that Kubernetes restarts your existing pod. Note that enabling debug options in the Kata
|
||||
config file can change the attestation evidence of a confidential guest.
|
||||
|
||||
Now you should be able to view logs from containerd with the following:
|
||||
```
|
||||
sudo journalctl -xeu containerd
|
||||
```
|
||||
|
||||
Kata writes many messages to this log. It's good to know what you're looking for. There are many
|
||||
generic messages that are not significant, often arising from a VM not shutting down cleanly
|
||||
after an unrelated issue.
|
||||
|
||||
### VM Doesn't Start
|
||||
If the VM has failed to start, you might have a problem with confidential
|
||||
computing support on your worker node. Make sure that you can start
|
||||
confidential VMs without confidential containers.
|
||||
|
||||
Check the containerd log for any obvious errors regarding VM boot.
|
||||
Try searching the log for the string `error` or for the name
|
||||
of your hypervisor i.e. `qemu` or `qemu-system-x86_64`.
|
||||
|
||||
If there are no obvious errors, try finding the hypervisor
|
||||
commandline. This should be in the containerd log if you have enabled
|
||||
debug messages correctly.
|
||||
|
||||
It might be tempting to try running the hypervisor command directly
|
||||
from the command line, but this usually isn't productive. Instead,
|
||||
try starting a standalone VM using the same kernel, initrd/disk,
|
||||
command line, firmware, and hypervisor that Kata uses.
|
||||
This might uncover some kind of system misconfiguration.
|
||||
You can also find these values in the Kata config file, but looking
|
||||
in the log is more direct.
|
||||
|
||||
Another way to print the hypervisor command is to create a bash script
|
||||
that prints any arguments it is called with to a file. Then modify the
|
||||
Kata config file so that the hypervisor path points to this scipt
|
||||
rather than to the hypervisor. This method can also be used to add
|
||||
additional parameters to the command line. Just have the bash script
|
||||
call the hypervisor with whatever arguments it received plus any that
|
||||
you want to add. This could be useful for enabling debugging or tracing
|
||||
flags in your hypervisor. For instance, if you are using QEMU and SEV
|
||||
you might want to add the argument `--trace 'kvm_sev_*'`. Make sure
|
||||
that QEMU was built with an appropriate tracing backend.
|
||||
|
||||
### VM Does Start
|
||||
|
||||
If the VM does start, search the containerd log for the string `vmconsole`.
|
||||
This will show you any guest serial output. You might see some errors
|
||||
coming from the kernel as the guest tries to boot. You might also see the
|
||||
Kata agent starting. If the Kata agent has started, you can match
|
||||
the output to the source to get some clues about what is happening.
|
||||
You might also see something more obvious, like a panic coming from
|
||||
the Kata agent.
|
||||
|
||||
#### failed to create shim task: failed to mount "/run/kata-containers/shared/containers/CONTAINER_NAME/rootfs"
|
||||
|
||||
If your CoCo Pod gets an error like showed below then it is likely the image pull policy is set to **IfNotPresent** and the image has been found in the kubelet cache. It fails because the container runtime will not delegate to the Kata agent to pull the image inside the VM and the agent in turn will try to mount the bundle rootfs that only exist in the host filesystem.
|
||||
|
||||
Therefore, you must ensure that the image pull policy is set to **Always** for any CoCo Pod. This ways the images are always handled entirely by the agent inside the VM. Worth mentioning we recognize that this behavior is suboptimal and so the community has worked on solutions to avoid constant images downloads for each and every workload.
|
||||
|
||||
```
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal Scheduled 20s default-scheduler Successfully assigned default/coco-fedora-69d9f84cd7-j597j to virtlab1012
|
||||
Normal Pulled 5s (x3 over 19s) kubelet Container image "docker.io/wainersm/coco-fedora_sshd@sha256:a7108f9f0080c429beb66e2cf0abff143c9eb9c7cf4dcde3241bc56c938d33b9" already present on machine
|
||||
Normal Created 5s (x3 over 19s) kubelet Created container coco-fedora
|
||||
Warning Failed 5s (x3 over 19s) kubelet Error: failed to create containerd task: failed to create shim task: failed to mount "/run/kata-containers/shared/containers/coco-fedora/rootfs" to "/run/kata-containers/coco-fedora/rootfs", with error: ENOENT: No such file or directory: unknown
|
||||
Warning BackOff 4s (x3 over 18s) kubelet Back-off restarting failed container
|
||||
```
|
||||
|
||||
#### Debug Console
|
||||
|
||||
One very useful deugging tool is the Kata guest debug console. You can
|
||||
enable this by editing the Kata agent configuration file and adding the lines
|
||||
``` toml
|
||||
debug_console = true
|
||||
debug_console_vport = 1026
|
||||
```
|
||||
|
||||
Enabling the debug console via the Kata Configuration file will overwrite
|
||||
any settings in the agent configuration file in the guest initrd.
|
||||
Enabling the debug console will change the launch measurement.
|
||||
|
||||
Once you've started a pod with the new configuration, get the id of the pod
|
||||
you want to access. Do this via `ps -ef | grep qemu` or equivalent.
|
||||
The id is the long id that shows up in many different arguments.
|
||||
It should look like `1a9ab65be63b8b03dfd0c75036d27f0ed09eab38abb45337fea83acd3cd7bacd`.
|
||||
Once you have the id, you can use it to access the debug console.
|
||||
```
|
||||
sudo /opt/confidential-containers/bin/kata-runtime exec <id>
|
||||
```
|
||||
You might need to symlink the appropriate Kata configuration file for your runtime
|
||||
class if the `kata-runtime` tries to look at the wrong one.
|
||||
|
||||
The debug console gives you access to the guest VM. This is a great way to
|
||||
investigate missing dependencies or incorrect configurations.
|
||||
|
||||
#### Guest Firmware Logs
|
||||
|
||||
If the VM is running but there is no guest output in the log,
|
||||
the guest might have stalled in the firmware. Firmware output will
|
||||
depend on your firmware and hypervisor. If you are using QEMU and OVMF,
|
||||
you can see the OVMF output by adding `-global isa-debugcon.iobase=0x402`
|
||||
and `-debugcon file:/tmp/ovmf.log` to the QEMU command line using the
|
||||
redirect script described above.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user