github/kata-containers
1
0
Fork 1
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-08-23 10:12:10 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #3638 from stevenhorsman/ccv0-encrypted

Browse Source 
doc: Add doc and script for ssh-demo sample
This commit is contained in:
Steve Horsman 2022-02-17 09:10:56 +00:00 committed by GitHub
commit 45e1268c70
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 173 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

147
docs/how-to/ccv0.sh
View File

@ -78,31 +78,34 @@ Overview:
Usage: Usage:
${script_name} [options] <command> ${script_name} [options] <command>
Commands: Commands:
- help: Display this help - agent_create_container: Run CreateContainer command against the agent with agent-ctl
- all: Build and install everything, test kata with containerd and capture the logs - agent_pull_image: Run PullImage command against the agent with agent-ctl
- build_and_install_all: Build and install everything - all: Build and install everything, test kata with containerd and capture the logs
- initialize: Install dependencies and check out kata-containers source - build_and_add_agent_to_rootfs: Builds the kata-agent and adds it to the rootfs
- rebuild_and_install_kata: Rebuild the kata runtime and agent and build and install the image - build_and_install_all: Build and install everything
- build_kata_runtime: Build and install the kata runtime - build_and_install_rootfs: Builds and installs the rootfs image
- configure: Configure Kata to use rootfs and enable debug - build_kata_runtime: Build and install the kata runtime
- create_rootfs: Create a local rootfs - build_qemu: Checkout, patch, build and install QEMU
- build_and_add_agent_to_rootfs:Builds the kata-agent and adds it to the rootfs - configure: Configure Kata to use rootfs and enable debug
- build_and_install_rootfs: Builds and installs the rootfs image - connect_to_ssh_demo_pod: Ssh into the ssh demo pod, showing that the decryption succeeded
- install_guest_kernel: Setup, build and install the guest kernel - create_rootfs: Create a local rootfs
- build_qemu: Checkout, patch, build and install QEMU - crictl_create_cc_container Use crictl to create a new busybox container in the kata cc pod
- init_kubernetes: initialize a Kubernetes cluster on this system - crictl_create_cc_pod Use crictl to create a new kata cc pod
- crictl_create_cc_pod Use crictl to create a new kata cc pod - crictl_delete_cc Use crictl to delete the kata cc pod sandbox and container in it
- crictl_create_cc_container Use crictl to create a new busybox container in the kata cc pod - help: Display this help
- crictl_delete_cc Use crictl to delete the kata cc pod sandbox and container in it - init_kubernetes: initialize a Kubernetes cluster on this system
- kubernetes_create_cc_pod: Create a Kata CC runtime busybox-based pod in Kubernetes - initialize: Install dependencies and check out kata-containers source
- kubernetes_delete_cc_pod: Delete the Kata CC runtime busybox-based pod in Kubernetes - install_guest_kernel: Setup, build and install the guest kernel
- open_kata_console: Stream the kata runtime's console - kubernetes_create_cc_pod: Create a Kata CC runtime busybox-based pod in Kubernetes
- open_kata_shell: Open a shell into the kata runtime - kubernetes_create_ssh_demo_pod: Create a Kata CC runtime pod based on the ssh demo
- agent_pull_image: Run PullImage command against the agent with agent-ctl - kubernetes_delete_cc_pod: Delete the Kata CC runtime busybox-based pod in Kubernetes
- shim_pull_image: Run PullImage command against the shim with ctr - kubernetes_delete_ssh_demo_pod: Delete the Kata CC runtime pod based on the ssh demo
- agent_create_container: Run CreateContainer command against the agent with agent-ctl - open_kata_console: Stream the kata runtime's console
- test: Test using kata with containerd - open_kata_shell: Open a shell into the kata runtime
- test_capture_logs: Test using kata with containerd and capture the logs in the user's home directory - rebuild_and_install_kata: Rebuild the kata runtime and agent and build and install the image
- shim_pull_image: Run PullImage command against the shim with ctr
- test_capture_logs: Test using kata with containerd and capture the logs in the user's home directory
- test: Test using kata with containerd
Options: Options:
-d: Enable debug -d: Enable debug
@ -139,7 +142,6 @@ initialize() {
# We need git to checkout and bootstrap the ci scripts and some other packages used in testing # We need git to checkout and bootstrap the ci scripts and some other packages used in testing
sudo apt-get update && sudo apt-get install -y curl git socat qemu-utils sudo apt-get update && sudo apt-get install -y curl git socat qemu-utils
PROFILE="${HOME}/.profile"
grep -qxF "export GOPATH=\${HOME}/go" "${PROFILE}" || echo "export GOPATH=\${HOME}/go" >> "${PROFILE}" grep -qxF "export GOPATH=\${HOME}/go" "${PROFILE}" || echo "export GOPATH=\${HOME}/go" >> "${PROFILE}"
grep -qxF "export GOROOT=/usr/local/go" "${PROFILE}" || echo "export GOROOT=/usr/local/go" >> "${PROFILE}" grep -qxF "export GOROOT=/usr/local/go" "${PROFILE}" || echo "export GOROOT=/usr/local/go" >> "${PROFILE}"
grep -qxF "export PATH=\${GOPATH}/bin:/usr/local/go/bin:\${PATH}" "${PROFILE}" || echo "export PATH=\${GOPATH}/bin:/usr/local/go/bin:\${PATH}" >> "${PROFILE}" grep -qxF "export PATH=\${GOPATH}/bin:/usr/local/go/bin:\${PATH}" "${PROFILE}" || echo "export PATH=\${GOPATH}/bin:/usr/local/go/bin:\${PATH}" >> "${PROFILE}"
@ -151,7 +153,7 @@ initialize() {
check_out_repos check_out_repos
pushd "${tests_repo_dir}" pushd "${tests_repo_dir}"
ci_dir_name=".ci" local ci_dir_name=".ci"
sudo -E PATH=$PATH -s "${ci_dir_name}/install_go.sh" -p -f sudo -E PATH=$PATH -s "${ci_dir_name}/install_go.sh" -p -f
sudo -E PATH=$PATH -s "${ci_dir_name}/install_rust.sh" sudo -E PATH=$PATH -s "${ci_dir_name}/install_rust.sh"
@ -202,6 +204,11 @@ configure() {
# Temp PoC verify code: Inject policy path config parameter # Temp PoC verify code: Inject policy path config parameter
sudo sed -i -e 's%^kernel_params = "\(.*\)"%kernel_params = "\1 agent.container_policy_file=/etc/containers/quay_verification/quay_policy.json"%g' /etc/kata-containers/configuration.toml sudo sed -i -e 's%^kernel_params = "\(.*\)"%kernel_params = "\1 agent.container_policy_file=/etc/containers/quay_verification/quay_policy.json"%g' /etc/kata-containers/configuration.toml
# If using AA then need to add the agent_config
if [ "${AA_KBC}" == "offline_fs_kbc" ]; then
sudo sed -i -e 's%^kernel_params = "\(.*\)"%kernel_params = "\1 agent.config_file=/etc/agent-config.toml"%g' /etc/kata-containers/configuration.toml
fi
# insert the cri_handler = "cc" into the [plugins.cri.containerd.runtimes.kata] section # insert the cri_handler = "cc" into the [plugins.cri.containerd.runtimes.kata] section
sudo sed -z -i 's/\([[:blank:]]*\)\(runtime_type = "io.containerd.kata.v2"\)/\1\2\n\1cri_handler = "cc"/' /etc/containerd/config.toml sudo sed -z -i 's/\([[:blank:]]*\)\(runtime_type = "io.containerd.kata.v2"\)/\1\2\n\1cri_handler = "cc"/' /etc/containerd/config.toml
@ -260,7 +267,7 @@ create_a_local_rootfs() {
pushd ${katacontainers_repo_dir}/tools/osbuilder/rootfs-builder pushd ${katacontainers_repo_dir}/tools/osbuilder/rootfs-builder
export distro="ubuntu" export distro="ubuntu"
[[ -z "${USE_PODMAN:-}" ]] && use_docker="${use_docker:-1}" [[ -z "${USE_PODMAN:-}" ]] && use_docker="${use_docker:-1}"
sudo -E OS_VERSION="${OS_VERSION:-}" GOPATH=$GOPATH EXTRA_PKGS="vim iputils-ping net-tools" DEBUG="${DEBUG}" USE_DOCKER="${use_docker:-}" SKOPEO=${SKOPEO:-} UMOCI=yes SECCOMP=yes ./rootfs.sh -r ${ROOTFS_DIR} ${distro} sudo -E OS_VERSION="${OS_VERSION:-}" GOPATH=$GOPATH EXTRA_PKGS="vim iputils-ping net-tools" DEBUG="${DEBUG}" USE_DOCKER="${use_docker:-}" SKOPEO=${SKOPEO:-} AA_KBC=${AA_KBC:-} UMOCI=yes SECCOMP=yes ./rootfs.sh -r ${ROOTFS_DIR} ${distro}
# Install_rust.sh during rootfs.sh switches us to the main branch of the tests repo, so switch back now # Install_rust.sh during rootfs.sh switches us to the main branch of the tests repo, so switch back now
pushd "${tests_repo_dir}" pushd "${tests_repo_dir}"
@ -268,6 +275,16 @@ create_a_local_rootfs() {
popd popd
# During the ./rootfs.sh call the kata agent is built as root, so we need to update the permissions, so we can rebuild it # During the ./rootfs.sh call the kata agent is built as root, so we need to update the permissions, so we can rebuild it
sudo chown -R ${USER}:${USER} "${katacontainers_repo_dir}/src/agent/" sudo chown -R ${USER}:${USER} "${katacontainers_repo_dir}/src/agent/"
# If offline key broker set then include ssh-demo keys and config from
# https://github.com/confidential-containers/documentation/tree/main/demos/ssh-demo
if [ "${AA_KBC}" == "offline_fs_kbc" ]; then
curl -Lo "${HOME}/aa-offline_fs_kbc-keys.json" https://raw.githubusercontent.com/confidential-containers/documentation/main/demos/ssh-demo/aa-offline_fs_kbc-keys.json
sudo mv "${HOME}/aa-offline_fs_kbc-keys.json" "${ROOTFS_DIR}/etc/aa-offline_fs_kbc-keys.json"
local rootfs_agent_config="${ROOTFS_DIR}/etc/agent-config.toml"
sudo -E AA_KBC_PARAMS="offline_fs_kbc::null" envsubst < ${katacontainers_repo_dir}/docs/how-to/data/confidential-agent-config.toml.in | sudo tee ${rootfs_agent_config}
fi
popd popd
} }
@ -300,9 +317,9 @@ build_rootfs_image() {
install_rootfs_image() { install_rootfs_image() {
pushd ${katacontainers_repo_dir}/tools/osbuilder/image-builder pushd ${katacontainers_repo_dir}/tools/osbuilder/image-builder
commit=$(git log --format=%h -1 HEAD) local commit=$(git log --format=%h -1 HEAD)
date=$(date +%Y-%m-%d-%T.%N%z) local date=$(date +%Y-%m-%d-%T.%N%z)
image="kata-containers-${date}-${commit}" local image="kata-containers-${date}-${commit}"
sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}" sudo install -o root -g root -m 0640 -D kata-containers.img "/usr/share/kata-containers/${image}"
(cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img) (cd /usr/share/kata-containers && sudo ln -sf "$image" kata-containers.img)
echo "Built Rootfs from ${ROOTFS_DIR} to /usr/share/kata-containers/${image}" echo "Built Rootfs from ${ROOTFS_DIR} to /usr/share/kata-containers/${image}"
@ -341,11 +358,11 @@ init_kubernetes() {
fi fi
# If kubernetes init has previously run we need to clean it by removing the image and resetting k8s # If kubernetes init has previously run we need to clean it by removing the image and resetting k8s
cid=$(sudo docker ps -a -q -f name=^/kata-registry$) local cid=$(sudo docker ps -a -q -f name=^/kata-registry$)
if [ -n "${cid}" ]; then if [ -n "${cid}" ]; then
sudo docker stop ${cid} && sudo docker rm ${cid} sudo docker stop ${cid} && sudo docker rm ${cid}
fi fi
k8s_nodes=$(kubectl get nodes -o name 2>/dev/null || true) local k8s_nodes=$(kubectl get nodes -o name 2>/dev/null || true)
if [ -n "${k8s_nodes}" ]; then if [ -n "${k8s_nodes}" ]; then
sudo kubeadm reset -f sudo kubeadm reset -f
fi fi
@ -367,14 +384,51 @@ EOF
} }
kubernetes_create_cc_pod() { kubernetes_create_cc_pod() {
kubectl apply -f ${k8s_pod_file} kubectl apply -f ${k8s_pod_file} && pod=$(kubectl get pods -o jsonpath='{.items..metadata.name}') && kubectl wait --for=condition=ready pods/$pod
kubectl get pods kubectl get pod $pod
} }
kubernetes_delete_cc_pod() { kubernetes_delete_cc_pod() {
kubectl delete -f ${k8s_pod_file} kubectl delete -f ${k8s_pod_file}
} }
# Check out the doc repo if required and pushd
pushd_ssh_demo() {
local doc_repo=github.com/confidential-containers/documentation
local doc_repo_dir="${GOPATH}/src/${doc_repo}"
mkdir -p $(dirname ${doc_repo_dir}) && sudo chown -R ${USER}:${USER} $(dirname ${doc_repo_dir})
if [ ! -d "${doc_repo_dir}" ]; then
git clone https://${doc_repo} "${doc_repo_dir}"
pushd "${doc_repo_dir}/demos/ssh-demo"
# Update runtimeClassName from kata-cc to kata
sudo sed -i -e 's/\([[:blank:]]*runtimeClassName: \).*/\1kata/g' "${doc_repo_dir}/demos/ssh-demo/k8s-cc-ssh.yaml"
chmod 600 ccv0-ssh
else
pushd "${doc_repo_dir}/demos/ssh-demo"
fi
}
kubernetes_create_ssh_demo_pod() {
pushd_ssh_demo
kubectl apply -f k8s-cc-ssh.yaml && pod=$(kubectl get pods -o jsonpath='{.items..metadata.name}') && kubectl wait --for=condition=ready pods/$pod
kubectl get pod $pod
popd
}
connect_to_ssh_demo_pod() {
local doc_repo=github.com/confidential-containers/documentation
local doc_repo_dir="${GOPATH}/src/${doc_repo}"
local ssh_command="ssh -i ${doc_repo_dir}/demos/ssh-demo/ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}")"
echo "Issuing command '${ssh_command}'"
${ssh_command}
}
kubernetes_delete_ssh_demo_pod() {
pushd_ssh_demo
kubectl delete -f k8s-cc-ssh.yaml
popd
}
crictl_sandbox_name=kata-cc-busybox-sandbox crictl_sandbox_name=kata-cc-busybox-sandbox
crictl_create_cc_pod() { crictl_create_cc_pod() {
# Update iptables to allow forwarding to the cni0 bridge avoiding issues caused by the docker0 bridge # Update iptables to allow forwarding to the cni0 bridge avoiding issues caused by the docker0 bridge
@ -391,7 +445,7 @@ EOF
crictl_delete_cc crictl_delete_cc
fi fi
pod_id=$(sudo crictl runp -r kata ${HOME}/pod-config.yaml) $(sudo crictl runp -r kata ${HOME}/pod-config.yaml)
sudo crictl pods sudo crictl pods
} }
@ -407,15 +461,15 @@ command:
log_path: kata-cc.0.log log_path: kata-cc.0.log
EOF EOF
pod_id=$(sudo crictl pods --name ${crictl_sandbox_name} -q) local pod_id=$(sudo crictl pods --name ${crictl_sandbox_name} -q)
container_id=$(sudo crictl create -with-pull ${pod_id} ${HOME}/container-config.yaml ${HOME}/pod-config.yaml) local container_id=$(sudo crictl create -with-pull ${pod_id} ${HOME}/container-config.yaml ${HOME}/pod-config.yaml)
sudo crictl start ${container_id} sudo crictl start ${container_id}
sudo crictl ps -a sudo crictl ps -a
} }
crictl_delete_cc() { crictl_delete_cc() {
pod_id=$(sudo crictl pods --name ${crictl_sandbox_name} -q) local pod_id=$(sudo crictl pods --name ${crictl_sandbox_name} -q)
container_id=$(sudo crictl ps --pod ${pod_id} -q) local container_id=$(sudo crictl ps --pod ${pod_id} -q)
if [ -n "${container_id}" ]; then if [ -n "${container_id}" ]; then
sudo crictl stop ${container_id} && sudo crictl rm ${container_id} sudo crictl stop ${container_id} && sudo crictl rm ${container_id}
fi fi
@ -424,7 +478,7 @@ crictl_delete_cc() {
test_kata_runtime() { test_kata_runtime() {
echo "Running ctr with the kata runtime..." echo "Running ctr with the kata runtime..."
test_image="quay.io/kata-containers/confidential-containers:signed" local test_image="quay.io/kata-containers/confidential-containers:signed"
if [ -z $(ctr images ls -q name=="${test_image}") ]; then if [ -z $(ctr images ls -q name=="${test_image}") ]; then
sudo ctr image pull "${test_image}" sudo ctr image pull "${test_image}"
fi fi
@ -503,7 +557,7 @@ agent_create_container() {
shim_pull_image() { shim_pull_image() {
get_ids get_ids
ctr_shim_command="sudo ctr --namespace k8s.io shim --id ${sandbox_id} pull-image ${PULL_IMAGE} ${CONTAINER_ID}" local ctr_shim_command="sudo ctr --namespace k8s.io shim --id ${sandbox_id} pull-image ${PULL_IMAGE} ${CONTAINER_ID}"
echo "Issuing command '${ctr_shim_command}'" echo "Issuing command '${ctr_shim_command}'"
${ctr_shim_command} ${ctr_shim_command}
} }
@ -584,6 +638,15 @@ main() {
kubernetes_delete_cc_pod) kubernetes_delete_cc_pod)
kubernetes_delete_cc_pod kubernetes_delete_cc_pod
;; ;;
kubernetes_create_ssh_demo_pod)
kubernetes_create_ssh_demo_pod
;;
connect_to_ssh_demo_pod)
connect_to_ssh_demo_pod
;;
kubernetes_delete_ssh_demo_pod)
kubernetes_delete_ssh_demo_pod
;;
test) test)
test_kata_runtime test_kata_runtime
;; ;;

69
docs/how-to/how-to-build-and-test-ccv0.md
View File

@ -107,7 +107,8 @@ or [using the Kata pod sandbox for testing with `agent-ctl` or `ctr shim`](#usin
``` ```
### Using Kubernetes for end-to-end provisioning of a Kata confidential containers pod with an unencrypted image ### Using Kubernetes for end-to-end provisioning of a Kata confidential containers pod with an unencrypted image
- Run the full build process with the Kubernetes environment variable set to `"yes"`, so the Kubernetes cluster is configured and created using the VM - Run the full build process with the Kubernetes environment variable set to `"yes"`, so the Kubernetes cluster is
configured and created using the VM
as a single node cluster: as a single node cluster:
```bash ```bash
$ export KUBERNETES="yes" $ export KUBERNETES="yes"
@ -393,6 +394,72 @@ want to protect with the attestation agent in future) fails we can run:
-rw-r--r-- 1 root root 63568 Jan 26 16:07 sha256_ebf391d3f0ba36d4b64999ebbeadc878d229faec8839254a1c2264cf47735841.mtree -rw-r--r-- 1 root root 63568 Jan 26 16:07 sha256_ebf391d3f0ba36d4b64999ebbeadc878d229faec8839254a1c2264cf47735841.mtree
``` ```
### Using Kubernetes to create a Kata confidential containers pod from the encrypted ssh demo sample image
The [ssh-demo](https://github.com/confidential-containers/documentation/tree/main/demos/ssh-demo) explains how to
demonstrate creating a Kata confidential containers pod from an encrypted image with the runtime created by the
[confidential-containers operator](https://github.com/confidential-containers/documentation/blob/main/demos/operator-demo).
To be fully confidential, this should be run on a Trusted Execution Environment, but it can be tested on generic
hardware as well.
If you wish to build the Kata confidential containers runtime to do this yourself, then you can using the following
steps:
- Run the full build process with the Kubernetes environment variable set to `"yes"`, so the Kubernetes cluster is
configured and created using the VM as a single node cluster and with `AA_KBC` set to `offline_fs_kbc`.
```bash
$ export KUBERNETES="yes"
$ export AA_KBC=offline_fs_kbc
$ ~/ccv0.sh build_and_install_all
```
- The `AA_KBC=offline_fs_kbc` mode will ensure that, when creating the rootfs of the Kata guest, the
[attestation-agent](https://github.com/confidential-containers/attestation-agent) will be added along with the
[sample offline KBC](https://github.com/confidential-containers/documentation/blob/main/demos/ssh-demo/aa-offline_fs_kbc-keys.json)
and an agent configuration file
> **Note**: Depending on how where your VMs are hosted and how IPs are shared you might get an error from docker
during matching `ERROR: toomanyrequests: Too Many Requests`. To get past
this, login into Docker Hub and pull the images used with:
> ```bash
> $ sudo docker login
> $ sudo docker pull registry:2
> $ sudo docker pull ubuntu:20.04
> ```
> then re-run the command.
- Check that your Kubernetes cluster has been correctly set-up by running :
```bash
$ kubectl get nodes
```
and checking that you see a single node e.g.
```text
NAME STATUS ROLES AGE VERSION
stevenh-ccv0-k8s1.fyre.ibm.com Ready control-plane,master 43s v1.22.0
```
- Create a sample Kata confidential containers ssh pod by running:
```bash
$ ~/ccv0.sh kubernetes_create_ssh_demo_pod
```
- As this point you should have a Kubernetes pod running the Kata confidential containers runtime that has pulled
the [sample image](https://hub.docker.com/r/katadocker/ccv0-ssh) which was encrypted by the key file that we included
in the rootfs.
During the pod deployment the image was pulled and then decrypted using the key file, on the Kata guest image, without
it ever being available to the host.
- To validate that the container is working you, can connect to the image via SSH by running:
```bash
$ ~/ccv0.sh connect_to_ssh_demo_pod
```
- During this connection the host key fingerprint is shown and should match:
`ED25519 key fingerprint is SHA256:wK7uOpqpYQczcgV00fGCh+X97sJL3f6G1Ku4rvlwtR0.`
- After you are finished connecting then run:
```bash
$ exit
```
- To delete the sample SSH demo pod run:
```bash
$ ~/ccv0.sh kubernetes_delete_ssh_demo_pod
```
## Additional script usage ## Additional script usage
As well as being able to use the script as above to build all of `kata-containers` from scratch it can be used to just As well as being able to use the script as above to build all of `kata-containers` from scratch it can be used to just