diff --git a/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md index d53e4bd5d2..e0bedea56e 100644 --- a/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md +++ b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md @@ -1,64 +1,82 @@ # How to use Kata Containers and CRI (containerd plugin) with Kubernetes -This document describes how to set up a single-machine Kubernetes cluster. -The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads. +* [Requirements](#requirements) +* [Install containerd with CRI plugin enabled](#install-containerd-with-cri-plugin-enabled) +* [Install Kata Containers](#install-kata-containers) +* [Install Kubernetes](#install-kubernetes) +* [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers) + * [Define the Kata runtime as the untrusted workload runtime](#define-the-kata-runtime-as-the-untrusted-workload-runtime) +* [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd) +* [Configure proxy - OPTIONAL](#configure-proxy---optional) +* [Start Kubernetes](#start-kubernetes) +* [Install a Pod Network](#install-a-pod-network) +* [Allow pods to run in the master node](#allow-pods-to-run-in-the-master-node) +* [Create an unstrusted pod using Kata Containers](#create-an-unstrusted-pod-using-kata-containers) +* [Delete created pod](#delete-created-pod) + +This document describes how to set up a single-machine Kubernetes (k8s) cluster. + +The Kubernetes cluster will use the +[CRI containerd plugin](https://github.com/containerd/cri) and +[Kata Containers](https://katacontainers.io) to launch untrusted workloads. + +## Requirements -## Requirements - Kubernetes, kubelet, kubeadm - cri-containerd - Kata Containers -For information about the supported version of these components see -Kata Containers [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml) file. +> **Note:** For information about the supported versions of these components, +> see the Kata Containers +> [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml) +> file. -## Install containerd(with CRI plugin enabled) +## Install containerd with CRI plugin enabled -Follow the instructions from [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md) +- Follow the instructions from the + [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md). - +- Check if `containerd` is now available + ```bash + $ command -v containerd + ``` -## Install Kata Containers +## Install Kata Containers -Follow the instructions to [install Kata](https://github.com/kata-containers/documentation/blob/master/install/README.md). - - +Follow the instructions to +[install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md). ## Install Kubernetes -Install Kubernetes in your host. See kubeadm [installation](https://kubernetes.io/docs/tasks/tools/install-kubeadm/) - - -### Configure containerd to use Kata Containers - -The CRI containerd plugin support configuration for two runtime types. - -- Default runtime: A runtime that is used by default to run workloads. -- Untrusted workload runtime: A runtime that will be used run untrusted workloads. - -#### Define the Kata runtime as `untrusted_workload_runtime` - -Configure the Kata runtime for untrusted workload with the [config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md) -`plugins.cri.containerd.untrusted_workload_runtime`. - -Unless configured otherwise, the default runtime is set to `runc`. -```bash -# Configure containerd to use Kata as untrusted_workload_runtime $ sudo mkdir -p /etc/containerd/ $ cat << EOT | sudo tee /etc/containerd/config.toml [plugins] @@ -69,145 +87,172 @@ $ cat << EOT | sudo tee /etc/containerd/config.toml EOT ``` -### Configure Kubelet to use containerd +> **Note:** Unless configured otherwise, the default runtime is set to `runc`. -In order to allow kubelet use containerd (using CRI interface) configure the service to -point to containerd socket. +## Configure Kubelet to use containerd + +In order to allow kubelet to use containerd (using the CRI interface), configure the service to point to the `containerd` socket. + +- Configure Kubernetes to use `containerd` + + ```bash + $ sudo mkdir -p /etc/systemd/system/kubelet.service.d/ + $ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf + [Service] + Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" + EOF + ``` + +- Inform systemd about the new configuration + + ```bash + $ sudo systemctl daemon-reload + ``` + +## Configure proxy - OPTIONAL + +If you are behind a proxy, use the following script to configure your proxy for docker, kubelet, and containerd: ```bash -# Configure k8s to use containerd -$ sudo mkdir -p /etc/systemd/system/kubelet.service.d/ -$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf -[Service] -Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" -EOF -$ sudo systemctl daemon-reload -``` +$ services=" +kubelet +containerd +docker +" -### Optional: Configure proxy +$ for service in ${services}; do -If you are behind a proxy this script will configure your proxy for docker -kubelet and containerd. + service_dir="/etc/systemd/system/${service}.service.d/" + sudo mkdir -p ${service_dir} -```bash -# Set proxys -$ services=( -'kubelet' -'containerd' -'docker' -) - -$ for s in "${services[@]}"; do - - service_dir="/etc/systemd/system/${s}.service.d/" - sudo mkdir -p ${service_dir} - - cat << EOT | sudo tee "${service_dir}/proxy.conf" + cat << EOT | sudo tee "${service_dir}/proxy.conf" [Service] Environment="HTTP_PROXY=${http_proxy}" Environment="HTTPS_PROXY=${https_proxy}" Environment="NO_PROXY=${no_proxy}" EOT done + $ sudo systemctl daemon-reload ``` -### Start Kubernetes with kubeadm +## Start Kubernetes + +- Make sure `containerd` is up and running + + ```bash + $ sudo systemctl restart containerd + $ sudo systemctl status containerd + ``` + +- Prevent conflicts between `docker` iptables (packet filtering) rules and k8s pod communication + + If Docker is installed on the node, it is necessary to modify the rule + below. See https://github.com/kubernetes/kubernetes/issues/40182 for further + details. + + ```bash + $ sudo iptables -P FORWARD ACCEPT + ``` + +- Start cluster using `kubeadm` + + ```bash + $ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 + $ export KUBECONFIG=/etc/kubernetes/admin.conf + $ sudo -E kubectl get nodes + $ sudo -E kubectl get pods + ``` + +## Install a Pod Network + +A pod network plugin is needed to allow pods to communicate with each other. + +- Install the `flannel` plugin by following the + [Using kubeadm to Create a Cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#instructions) + guide, starting from the **Installing a pod network** section. + +- Create a pod network using flannel + + > **Note:** There is no known way to determine programmatically the best version (commit) to use. + > See https://github.com/coreos/flannel/issues/995. + + ```bash + $ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml + ``` + +- Wait for the pod network to become available + + ```bash + # number of seconds to wait for pod network to become available + $ timeout_dns=420 + + $ while [ "$timeout_dns" -gt 0 ]; do + if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then + break + fi + + sleep 1s + ((timeout_dns--)) + done + ``` + +- Check the pod network is running + ```bash + $ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false ) + ``` + +## Allow pods to run in the master node + +By default, the cluster will not schedule pods in the master node. To enable master node scheduling: ```bash -# Mark sure containerd is up and running -$ sudo systemctl restart containerd -$ sudo systemctl status containerd - -# Prevent docker iptables rules conflict with k8s pod communication -$ sudo iptables -P FORWARD ACCEPT - -# Start cluster using kubeadm -$ sudo kubeadm init --skip-preflight-checks \ ---cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 - -$ export KUBECONFIG=/etc/kubernetes/admin.conf - -$ sudo -E kubectl get nodes -$ sudo -E kubectl get pods -``` - -### Install a pod network -Install a pod network plugin is needed to allow pods communicate with each other. - -Install flannel plugging, by following the instructions in the section *Installing a pod network* -from [Using kubeadm to Create a Cluster ](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) -guide. - - - - -```bash -# wait for pod network -$ timeout_dns=0 -$ until [ "$timeout_dns" -eq "420" ]; do - if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then - break - fi - sleep 1s - ((timeout_dns+=1)) - done - -# check pod network is running -$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false ) -``` - -### Allow run pods in master node - -By default, the cluster will not schedule pods in the master node to allow that run: - -```bash -# allow master node run pods $ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master- ``` - -### Create a unstrusted pod using Kata Containers +## Create an unstrusted pod using Kata Containers By default, all pods are created with the default runtime configured in CRI containerd plugin. -If a pod has the `io.kubernetes.cri.untrusted-workload annotation` set as -`"true"`, the CRI plugin will run the pod with the Kata Containers runtime. + +If a pod has the `io.kubernetes.cri.untrusted-workload` annotation set to `"true"`, the CRI plugin runs the pod with the +[Kata Containers runtime](https://github.com/kata-containers/runtime/blob/master/README.md). + +- Create an untrusted pod configuration + + ```bash + $ cat << EOT | tee nginx-untrusted.yaml + apiVersion: v1 + kind: Pod + metadata: + name: nginx-untrusted + annotations: + io.kubernetes.cri.untrusted-workload: "true" + spec: + containers: + - name: nginx + image: nginx + + EOT + ``` + +- Create an untrusted pod + ```bash + $ sudo -E kubectl apply -f nginx-untrusted.yaml + ``` + +- Check pod is running + + ```bash + $ sudo -E kubectl get pods + ``` + +- Check hypervisor is running + ```bash + $ ps aux | grep qemu + ``` + +## Delete created pod ```bash -# Create untrusted pod configuration -$ cat << EOT | tee nginx-untrusted.yaml -apiVersion: v1 -kind: Pod -metadata: - name: nginx-untrusted - annotations: - io.kubernetes.cri.untrusted-workload: "true" -spec: - containers: - - name: nginx - image: nginx - -EOT - -# Create untrusted pod -$ sudo -E kubectl apply -f nginx-untrusted.yaml - -# Check pod is running -$ sudo -E kubectl get pods - -# Check qemu is running -$ ps aux | grep qemu -``` -### Delete created pod - -```bash -# Delete pod -$ sudo -E kubectl delete -f nginx-untrusted.yaml +$ sudo -E kubectl delete -f nginx-untrusted.yaml ```