diff --git a/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md new file mode 100644 index 000000000..d53e4bd5d --- /dev/null +++ b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md @@ -0,0 +1,213 @@ +# How to use Kata Containers and CRI (containerd plugin) with Kubernetes + +This document describes how to set up a single-machine Kubernetes cluster. +The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads. + +## Requirements +- Kubernetes, kubelet, kubeadm +- cri-containerd +- Kata Containers + +For information about the supported version of these components see +Kata Containers [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml) file. + +## Install containerd(with CRI plugin enabled) + +Follow the instructions from [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md) + + + +## Install Kata Containers + +Follow the instructions to [install Kata](https://github.com/kata-containers/documentation/blob/master/install/README.md). + + + +## Install Kubernetes +Install Kubernetes in your host. See kubeadm [installation](https://kubernetes.io/docs/tasks/tools/install-kubeadm/) + + +### Configure containerd to use Kata Containers + +The CRI containerd plugin support configuration for two runtime types. + +- Default runtime: A runtime that is used by default to run workloads. +- Untrusted workload runtime: A runtime that will be used run untrusted workloads. + +#### Define the Kata runtime as `untrusted_workload_runtime` + +Configure the Kata runtime for untrusted workload with the [config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md) +`plugins.cri.containerd.untrusted_workload_runtime`. + +Unless configured otherwise, the default runtime is set to `runc`. +```bash +# Configure containerd to use Kata as untrusted_workload_runtime +$ sudo mkdir -p /etc/containerd/ +$ cat << EOT | sudo tee /etc/containerd/config.toml +[plugins] + [plugins.cri.containerd] + [plugins.cri.containerd.untrusted_workload_runtime] + runtime_type = "io.containerd.runtime.v1.linux" + runtime_engine = "/usr/bin/kata-runtime" +EOT +``` + +### Configure Kubelet to use containerd + +In order to allow kubelet use containerd (using CRI interface) configure the service to +point to containerd socket. + +```bash +# Configure k8s to use containerd +$ sudo mkdir -p /etc/systemd/system/kubelet.service.d/ +$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf +[Service] +Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" +EOF +$ sudo systemctl daemon-reload +``` + +### Optional: Configure proxy + +If you are behind a proxy this script will configure your proxy for docker +kubelet and containerd. + +```bash +# Set proxys +$ services=( +'kubelet' +'containerd' +'docker' +) + +$ for s in "${services[@]}"; do + + service_dir="/etc/systemd/system/${s}.service.d/" + sudo mkdir -p ${service_dir} + + cat << EOT | sudo tee "${service_dir}/proxy.conf" +[Service] +Environment="HTTP_PROXY=${http_proxy}" +Environment="HTTPS_PROXY=${https_proxy}" +Environment="NO_PROXY=${no_proxy}" +EOT +done +$ sudo systemctl daemon-reload +``` + +### Start Kubernetes with kubeadm + +```bash +# Mark sure containerd is up and running +$ sudo systemctl restart containerd +$ sudo systemctl status containerd + +# Prevent docker iptables rules conflict with k8s pod communication +$ sudo iptables -P FORWARD ACCEPT + +# Start cluster using kubeadm +$ sudo kubeadm init --skip-preflight-checks \ +--cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 + +$ export KUBECONFIG=/etc/kubernetes/admin.conf + +$ sudo -E kubectl get nodes +$ sudo -E kubectl get pods +``` + +### Install a pod network +Install a pod network plugin is needed to allow pods communicate with each other. + +Install flannel plugging, by following the instructions in the section *Installing a pod network* +from [Using kubeadm to Create a Cluster ](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) +guide. + + + + +```bash +# wait for pod network +$ timeout_dns=0 +$ until [ "$timeout_dns" -eq "420" ]; do + if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then + break + fi + sleep 1s + ((timeout_dns+=1)) + done + +# check pod network is running +$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false ) +``` + +### Allow run pods in master node + +By default, the cluster will not schedule pods in the master node to allow that run: + +```bash +# allow master node run pods +$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master- +``` + + +### Create a unstrusted pod using Kata Containers + +By default, all pods are created with the default runtime configured in CRI containerd plugin. +If a pod has the `io.kubernetes.cri.untrusted-workload annotation` set as +`"true"`, the CRI plugin will run the pod with the Kata Containers runtime. + +```bash +# Create untrusted pod configuration +$ cat << EOT | tee nginx-untrusted.yaml +apiVersion: v1 +kind: Pod +metadata: + name: nginx-untrusted + annotations: + io.kubernetes.cri.untrusted-workload: "true" +spec: + containers: + - name: nginx + image: nginx + +EOT + +# Create untrusted pod +$ sudo -E kubectl apply -f nginx-untrusted.yaml + +# Check pod is running +$ sudo -E kubectl get pods + +# Check qemu is running +$ ps aux | grep qemu +``` +### Delete created pod + +```bash +# Delete pod +$ sudo -E kubectl delete -f nginx-untrusted.yaml +```