github/kata-containers
1
0
Fork 1
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-07-12 22:58:58 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #251 from jodh-intel/improve-k8s+cri-containerd-howto

Browse Source 
docs: Clean up k8s with cri-containerd howto
This commit is contained in:
Graham Whaley 2018-10-04 13:23:41 +01:00 committed by GitHub
commit ec9f9d428e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 208 additions and 163 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

371
how-to/how-to-use-k8s-with-cri-containerd-and-kata.md
View File

@ -1,64 +1,82 @@
# How to use Kata Containers and CRI (containerd plugin) with Kubernetes # How to use Kata Containers and CRI (containerd plugin) with Kubernetes
This document describes how to set up a single-machine Kubernetes cluster. * [Requirements](#requirements)
The Kubernetes cluster will use the CRI containerd plugin and Kata Containers to launch untrusted workloads. * [Install containerd with CRI plugin enabled](#install-containerd-with-cri-plugin-enabled)
* [Install Kata Containers](#install-kata-containers)
* [Install Kubernetes](#install-kubernetes)
* [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers)
* [Define the Kata runtime as the untrusted workload runtime](#define-the-kata-runtime-as-the-untrusted-workload-runtime)
* [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd)
* [Configure proxy - OPTIONAL](#configure-proxy---optional)
* [Start Kubernetes](#start-kubernetes)
* [Install a Pod Network](#install-a-pod-network)
* [Allow pods to run in the master node](#allow-pods-to-run-in-the-master-node)
* [Create an unstrusted pod using Kata Containers](#create-an-unstrusted-pod-using-kata-containers)
* [Delete created pod](#delete-created-pod)
This document describes how to set up a single-machine Kubernetes (k8s) cluster.
The Kubernetes cluster will use the
[CRI containerd plugin](https://github.com/containerd/cri) and
[Kata Containers](https://katacontainers.io) to launch untrusted workloads.
## Requirements
## Requirements
- Kubernetes, kubelet, kubeadm - Kubernetes, kubelet, kubeadm
- cri-containerd - cri-containerd
- Kata Containers - Kata Containers
For information about the supported version of these components see > **Note:** For information about the supported versions of these components,
Kata Containers [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml) file. > see the Kata Containers
> [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml)
> file.
## Install containerd(with CRI plugin enabled) ## Install containerd with CRI plugin enabled
Follow the instructions from [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md) - Follow the instructions from the
[CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md).
<!--- - Check if `containerd` is now available
```bash ```bash
# Check if containerd is installed $ command -v containerd
$ command -v containerd ```
```
--->
## Install Kata Containers ## Install Kata Containers
Follow the instructions to [install Kata](https://github.com/kata-containers/documentation/blob/master/install/README.md). Follow the instructions to
[install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md).
<!---
```bash
# Check if kata-runtime is installed
$ command -v kata-runtime
# Check kata is well configured
$ kata-runtime kata-env
```
--->
## Install Kubernetes ## Install Kubernetes
Install Kubernetes in your host. See kubeadm [installation](https://kubernetes.io/docs/tasks/tools/install-kubeadm/)
<!--- - Follow the instructions for
[kubeadm installation](https://kubernetes.io/docs/setup/independent/install-kubeadm/).
- Check `kubeadm` is now available
```bash
$ command -v kubeadm
```
## Configure containerd to use Kata Containers
The CRI `containerd` plugin supports configuration for two runtime types.
- **Default runtime:**
A runtime that is used by default to run workloads.
- **Untrusted workload runtime:**
A runtime that will be used to run untrusted workloads. This is appropriate
for workloads that require a higher degree of security isolation.
#### Define the Kata runtime as the untrusted workload runtime
Configure `containerd` to use the Kata runtime to run untrusted workloads by
setting the `plugins.cri.containerd.untrusted_workload_runtime`
[config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md):
```bash ```bash
# Check if kubadm is installed
$ command -v kubeadm
```
--->
### Configure containerd to use Kata Containers
The CRI containerd plugin support configuration for two runtime types.
- Default runtime: A runtime that is used by default to run workloads.
- Untrusted workload runtime: A runtime that will be used run untrusted workloads.
#### Define the Kata runtime as `untrusted_workload_runtime`
Configure the Kata runtime for untrusted workload with the [config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md)
`plugins.cri.containerd.untrusted_workload_runtime`.
Unless configured otherwise, the default runtime is set to `runc`.
```bash
# Configure containerd to use Kata as untrusted_workload_runtime
$ sudo mkdir -p /etc/containerd/ $ sudo mkdir -p /etc/containerd/
$ cat << EOT | sudo tee /etc/containerd/config.toml $ cat << EOT | sudo tee /etc/containerd/config.toml
[plugins] [plugins]
@ -69,145 +87,172 @@ $ cat << EOT | sudo tee /etc/containerd/config.toml
EOT EOT
``` ```
### Configure Kubelet to use containerd > **Note:** Unless configured otherwise, the default runtime is set to `runc`.
In order to allow kubelet use containerd (using CRI interface) configure the service to ## Configure Kubelet to use containerd
point to containerd socket.
In order to allow kubelet to use containerd (using the CRI interface), configure the service to point to the `containerd` socket.
- Configure Kubernetes to use `containerd`
```bash
$ sudo mkdir -p /etc/systemd/system/kubelet.service.d/
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF
```
- Inform systemd about the new configuration
```bash
$ sudo systemctl daemon-reload
```
## Configure proxy - OPTIONAL
If you are behind a proxy, use the following script to configure your proxy for docker, kubelet, and containerd:
```bash ```bash
# Configure k8s to use containerd $ services="
$ sudo mkdir -p /etc/systemd/system/kubelet.service.d/ kubelet
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf containerd
[Service] docker
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" "
EOF
$ sudo systemctl daemon-reload
```
### Optional: Configure proxy $ for service in ${services}; do
If you are behind a proxy this script will configure your proxy for docker service_dir="/etc/systemd/system/${service}.service.d/"
kubelet and containerd. sudo mkdir -p ${service_dir}
```bash cat << EOT | sudo tee "${service_dir}/proxy.conf"
# Set proxys
$ services=(
'kubelet'
'containerd'
'docker'
)
$ for s in "${services[@]}"; do
service_dir="/etc/systemd/system/${s}.service.d/"
sudo mkdir -p ${service_dir}
cat << EOT | sudo tee "${service_dir}/proxy.conf"
[Service] [Service]
Environment="HTTP_PROXY=${http_proxy}" Environment="HTTP_PROXY=${http_proxy}"
Environment="HTTPS_PROXY=${https_proxy}" Environment="HTTPS_PROXY=${https_proxy}"
Environment="NO_PROXY=${no_proxy}" Environment="NO_PROXY=${no_proxy}"
EOT EOT
done done
$ sudo systemctl daemon-reload $ sudo systemctl daemon-reload
``` ```
### Start Kubernetes with kubeadm ## Start Kubernetes
- Make sure `containerd` is up and running
```bash
$ sudo systemctl restart containerd
$ sudo systemctl status containerd
```
- Prevent conflicts between `docker` iptables (packet filtering) rules and k8s pod communication
If Docker is installed on the node, it is necessary to modify the rule
below. See https://github.com/kubernetes/kubernetes/issues/40182 for further
details.
```bash
$ sudo iptables -P FORWARD ACCEPT
```
- Start cluster using `kubeadm`
```bash
$ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
$ export KUBECONFIG=/etc/kubernetes/admin.conf
$ sudo -E kubectl get nodes
$ sudo -E kubectl get pods
```
## Install a Pod Network
A pod network plugin is needed to allow pods to communicate with each other.
- Install the `flannel` plugin by following the
[Using kubeadm to Create a Cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#instructions)
guide, starting from the **Installing a pod network** section.
- Create a pod network using flannel
> **Note:** There is no known way to determine programmatically the best version (commit) to use.
> See https://github.com/coreos/flannel/issues/995.
```bash
$ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
- Wait for the pod network to become available
```bash
# number of seconds to wait for pod network to become available
$ timeout_dns=420
$ while [ "$timeout_dns" -gt 0 ]; do
if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
break
fi
sleep 1s
((timeout_dns--))
done
```
- Check the pod network is running
```bash
$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )
```
## Allow pods to run in the master node
By default, the cluster will not schedule pods in the master node. To enable master node scheduling:
```bash ```bash
# Mark sure containerd is up and running
$ sudo systemctl restart containerd
$ sudo systemctl status containerd
# Prevent docker iptables rules conflict with k8s pod communication
$ sudo iptables -P FORWARD ACCEPT
# Start cluster using kubeadm
$ sudo kubeadm init --skip-preflight-checks \
--cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
$ export KUBECONFIG=/etc/kubernetes/admin.conf
$ sudo -E kubectl get nodes
$ sudo -E kubectl get pods
```
### Install a pod network
Install a pod network plugin is needed to allow pods communicate with each other.
Install flannel plugging, by following the instructions in the section *Installing a pod network*
from [Using kubeadm to Create a Cluster ](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
guide.
<!---
```bash
# Install a pod network using flannel
# There is not a programmatic way to know last what flannel commit use
# See https://github.com/coreos/flannel/issues/995
$ sudo -E kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```
--->
```bash
# wait for pod network
$ timeout_dns=0
$ until [ "$timeout_dns" -eq "420" ]; do
if sudo -E kubectl get pods --all-namespaces | grep dns | grep Running; then
break
fi
sleep 1s
((timeout_dns+=1))
done
# check pod network is running
$ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false )
```
### Allow run pods in master node
By default, the cluster will not schedule pods in the master node to allow that run:
```bash
# allow master node run pods
$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master- $ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
``` ```
## Create an unstrusted pod using Kata Containers
### Create a unstrusted pod using Kata Containers
By default, all pods are created with the default runtime configured in CRI containerd plugin. By default, all pods are created with the default runtime configured in CRI containerd plugin.
If a pod has the `io.kubernetes.cri.untrusted-workload annotation` set as
`"true"`, the CRI plugin will run the pod with the Kata Containers runtime. If a pod has the `io.kubernetes.cri.untrusted-workload` annotation set to `"true"`, the CRI plugin runs the pod with the
[Kata Containers runtime](https://github.com/kata-containers/runtime/blob/master/README.md).
- Create an untrusted pod configuration
```bash
$ cat << EOT | tee nginx-untrusted.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-untrusted
annotations:
io.kubernetes.cri.untrusted-workload: "true"
spec:
containers:
- name: nginx
image: nginx
EOT
```
- Create an untrusted pod
```bash
$ sudo -E kubectl apply -f nginx-untrusted.yaml
```
- Check pod is running
```bash
$ sudo -E kubectl get pods
```
- Check hypervisor is running
```bash
$ ps aux | grep qemu
```
## Delete created pod
```bash ```bash
# Create untrusted pod configuration $ sudo -E kubectl delete -f nginx-untrusted.yaml
$ cat << EOT | tee nginx-untrusted.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-untrusted
annotations:
io.kubernetes.cri.untrusted-workload: "true"
spec:
containers:
- name: nginx
image: nginx
EOT
# Create untrusted pod
$ sudo -E kubectl apply -f nginx-untrusted.yaml
# Check pod is running
$ sudo -E kubectl get pods
# Check qemu is running
$ ps aux | grep qemu
```
### Delete created pod
```bash
# Delete pod
$ sudo -E kubectl delete -f nginx-untrusted.yaml
``` ```