kata-containers/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md

How to use Kata Containers and CRI (containerd plugin) with Kubernetes

This document describes how to set up a single-machine Kubernetes cluster. The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads.

Requirements

• Kubernetes, kubelet, kubeadm
• cri-containerd
• Kata Containers

For information about the supported version of these components see Kata Containers versions.yaml file.

Install containerd(with CRI plugin enabled)

Follow the instructions from CRI installation guide

Install Kata Containers

Follow the instructions to install Kata.

Install Kubernetes

Install Kubernetes in your host. See kubeadm installation

Configure containerd to use Kata Containers

The CRI containerd plugin support configuration for two runtime types.

• Default runtime: A runtime that is used by default to run workloads.
• Untrusted workload runtime: A runtime that will be used run untrusted workloads.

Define the Kata runtime as untrusted_workload_runtime

Configure the Kata runtime for untrusted workload with the config option plugins.cri.containerd.untrusted_workload_runtime.

Unless configured otherwise, the default runtime is set to runc.

# Configure containerd to use Kata as untrusted_workload_runtime
$ sudo mkdir -p /etc/containerd/
$ cat << EOT | sudo tee /etc/containerd/config.toml
[plugins]
    [plugins.cri.containerd]
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = "/usr/bin/kata-runtime"
EOT

Configure Kubelet to use containerd

In order to allow kubelet use containerd (using CRI interface) configure the service to point to containerd socket.

# Configure k8s to use containerd
$ sudo mkdir -p  /etc/systemd/system/kubelet.service.d/
$ cat << EOF | sudo tee  /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]                                                 
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF
$ sudo systemctl daemon-reload

Optional: Configure proxy

If you are behind a proxy this script will configure your proxy for docker kubelet and containerd.

# Set proxys
$ services=(
'kubelet'
'containerd'
'docker'
)

$ for s in "${services[@]}"; do 

	service_dir="/etc/systemd/system/${s}.service.d/"
	sudo mkdir -p ${service_dir}

	cat << EOT | sudo tee "${service_dir}/proxy.conf"
[Service]
Environment="HTTP_PROXY=${http_proxy}"
Environment="HTTPS_PROXY=${https_proxy}"
Environment="NO_PROXY=${no_proxy}"
EOT
done
$ sudo systemctl daemon-reload

Start Kubernetes with kubeadm

# Mark sure containerd is up and running
$ sudo systemctl restart containerd
$ sudo systemctl status containerd

# Prevent docker iptables rules conflict with k8s pod communication
$ sudo iptables -P FORWARD ACCEPT

# Start cluster using kubeadm
$ sudo kubeadm init --skip-preflight-checks \
--cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16

$ export KUBECONFIG=/etc/kubernetes/admin.conf

$ sudo -E kubectl get nodes
$ sudo -E kubectl get pods

Install a pod network

Install a pod network plugin is needed to allow pods communicate with each other.

Install flannel plugging, by following the instructions in the section Installing a pod network from Using kubeadm to Create a Cluster guide.

# wait for pod network
$ timeout_dns=0
$ until [ "$timeout_dns" -eq "420" ]; do
	if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
		break
	fi
	sleep 1s
	((timeout_dns+=1))
 done

# check pod network is running
$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )

Allow run pods in master node

By default, the cluster will not schedule pods in the master node to allow that run:

# allow master node run pods
$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-

Create a unstrusted pod using Kata Containers

By default, all pods are created with the default runtime configured in CRI containerd plugin. If a pod has the io.kubernetes.cri.untrusted-workload annotation set as "true", the CRI plugin will run the pod with the Kata Containers runtime.

# Create untrusted pod configuration
$ cat << EOT | tee nginx-untrusted.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx
    image: nginx
    
EOT

# Create untrusted pod
$ sudo -E kubectl apply -f nginx-untrusted.yaml

# Check pod is running
$ sudo -E kubectl get pods

# Check qemu is running
$ ps aux | grep qemu

Delete created pod

# Delete pod
$ sudo -E kubectl delete -f  nginx-untrusted.yaml