From 8a9e40451ba1aa78efe724dd9ea284716f1f9674 Mon Sep 17 00:00:00 2001 From: Xu Wang Date: Sun, 30 Dec 2018 14:42:04 +0800 Subject: [PATCH 1/6] arch-images: add a figure for shimv2 The comparison w/ or w/o shimv2 Signed-off-by: Xu Wang --- arch-images/shimv2.svg | 1 + 1 file changed, 1 insertion(+) create mode 100644 arch-images/shimv2.svg diff --git a/arch-images/shimv2.svg b/arch-images/shimv2.svg new file mode 100644 index 0000000000..c4fcbb90ff --- /dev/null +++ b/arch-images/shimv2.svg @@ -0,0 +1 @@ + \ No newline at end of file From 2bb23519f7e932f3d905c2e87d953c49b31e5b12 Mon Sep 17 00:00:00 2001 From: Xu Wang Date: Tue, 22 Jan 2019 15:39:05 +0800 Subject: [PATCH 2/6] install: mention shimv2 for the manual installation instruction - mentioned shimv2 in the configuration part of manual installation - reference the link of shim v2 api and the k8s containerd howto Signed-off-by: Xu Wang --- install/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/install/README.md b/install/README.md index 440e7fe158..10b61338ce 100644 --- a/install/README.md +++ b/install/README.md @@ -49,7 +49,9 @@ Manual installation instructions are available for [these distributions](#suppor 1. Add the Kata Containers repository to your distro package manager, and import the packages signing key. 2. Install the Kata Containers packages. 3. Install a supported container manager. -4. Configure the container manager to use `kata-runtime` as the default OCI runtime. +4. Configure the container manager to use `kata-runtime` as the default OCI runtime. Or, for Kata Containers 1.5.0 or above, configure the + `io.containerd.kata.v2` to be the runtime shim (see [containerd runtime v2 (shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) + and [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md)). > **Notes on upgrading**: > - If you are installing Kata Containers on a system that already has Clear Containers or `runv` installed, From c06e9f241f9e29a89a0542ae8504e6b46e599376 Mon Sep 17 00:00:00 2001 From: Xu Wang Date: Wed, 23 Jan 2019 14:11:48 +0800 Subject: [PATCH 3/6] developer-guide: update for shimv2 - update cri-containerd to containerd with cri plugin - suggest the shimv2 to be the preferred kubernetes integration way. Signed-off-by: Xu Wang --- Developer-Guide.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/Developer-Guide.md b/Developer-Guide.md index 0271a206b5..20933081e8 100644 --- a/Developer-Guide.md +++ b/Developer-Guide.md @@ -30,10 +30,10 @@ * [Run Kata Containers with Kubernetes](#run-kata-containers-with-kubernetes) * [Install a CRI implementation](#install-a-cri-implementation) * [CRI-O](#cri-o) - * [CRI-containerd](#cri-containerd) + * [containerd with cri plugin](#containerd-with-cri-plugin) * [Install Kubernetes](#install-kubernetes) * [Configure for CRI-O](#configure-for-cri-o) - * [Configure for CRI-containerd](#configure-for-cri-containerd) + * [Configure for containerd](#configure-for-containerd) * [Run a Kubernetes pod with Kata Containers](#run-a-kubernetes-pod-with-kata-containers) * [Troubleshoot Kata Containers](#troubleshoot-kata-containers) * [Appendices](#appendices) @@ -390,6 +390,10 @@ choose the one that you want, but you have to pick one. After choosing either CRI-O or CRI-containerd, you must make the appropriate changes to ensure it relies on the Kata Containers runtime. +As of Kata Containers 1.5, using `shimv2` with containerd 1.2.0 or above is the preferred +way to run Kata Containers with Kubernetes ([see the howto](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)). +The CRI-O will catch up soon. + ### CRI-O If you select CRI-O, follow the "CRI-O Tutorial" instructions @@ -454,13 +458,13 @@ Restart CRI-O to take changes into account $ sudo systemctl restart crio ``` -### CRI-containerd +### containerd with cri plugin -If you select CRI-containerd, follow the "Getting Started for Developers" +If you select containerd with `cri` plugin, follow the "Getting Started for Developers" instructions [here](https://github.com/containerd/cri#getting-started-for-developers) to properly install it. -To customize CRI-containerd to select Kata Containers runtime, follow our +To customize containerd to select Kata Containers runtime, follow our "Configure containerd to use Kata Containers" internal documentation [here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers). @@ -483,14 +487,14 @@ implementation you chose, and the kubelet service has to be updated accordingly. Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock" ``` -### Configure for CRI-containerd +### Configure for containerd `/etc/systemd/system/kubelet.service.d/0-cri-containerd.conf` ``` [Service] Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" ``` -For more information about CRI-containerd see the "Configure Kubelet to use containerd" +For more information about containerd see the "Configure Kubelet to use containerd" documentation [here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-kubelet-to-use-containerd). ## Run a Kubernetes pod with Kata Containers From 75501df2fa64afa352ed9ccc0cf99f23e5512de5 Mon Sep 17 00:00:00 2001 From: Xu Wang Date: Wed, 23 Jan 2019 14:12:12 +0800 Subject: [PATCH 4/6] how-to: add shimv2 as the default in the k8s and containerd howto move the containerd+kata parts to containerd-kata.md and make this file simpler Signed-off-by: Xu Wang --- ...to-use-k8s-with-cri-containerd-and-kata.md | 71 +++++-------------- 1 file changed, 16 insertions(+), 55 deletions(-) diff --git a/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md index e0bedea56e..b8d0d0638b 100644 --- a/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md +++ b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md @@ -1,13 +1,11 @@ # How to use Kata Containers and CRI (containerd plugin) with Kubernetes * [Requirements](#requirements) -* [Install containerd with CRI plugin enabled](#install-containerd-with-cri-plugin-enabled) -* [Install Kata Containers](#install-kata-containers) -* [Install Kubernetes](#install-kubernetes) -* [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers) - * [Define the Kata runtime as the untrusted workload runtime](#define-the-kata-runtime-as-the-untrusted-workload-runtime) -* [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd) -* [Configure proxy - OPTIONAL](#configure-proxy---optional) +* [Install and configure containerd](#install-and-configure-containerd) +* [Install and configure Kubernetes](#install-and-configure-kubernetes) + * [Install Kubernetes](#install-kubernetes) + * [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd) + * [Configure HTTP proxy - OPTIONAL](#configure-http-proxy---optional) * [Start Kubernetes](#start-kubernetes) * [Install a Pod Network](#install-a-pod-network) * [Allow pods to run in the master node](#allow-pods-to-run-in-the-master-node) @@ -20,10 +18,13 @@ The Kubernetes cluster will use the [CRI containerd plugin](https://github.com/containerd/cri) and [Kata Containers](https://katacontainers.io) to launch untrusted workloads. +For Kata Containers 1.5.0-rc2 and above, we will use [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](../design/shimv2.md) +to launch Kata Containers. For the previous version of Kata Containers, the Pods are launched with `kata-runtime`. + ## Requirements - Kubernetes, kubelet, kubeadm -- cri-containerd +- containerd with `cri` plug-in - Kata Containers > **Note:** For information about the supported versions of these components, @@ -31,22 +32,14 @@ The Kubernetes cluster will use the > [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml) > file. -## Install containerd with CRI plugin enabled +## Install and configure containerd -- Follow the instructions from the - [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md). +First, follow the [How to use Kata Containers and Containerd](containerd-kata.md) to install and configure containerd. +Then, make sure the containerd works with the [examples in it](containerd-kata.md#run). -- Check if `containerd` is now available - ```bash - $ command -v containerd - ``` +## Install and configure Kubernetes -## Install Kata Containers - -Follow the instructions to -[install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md). - -## Install Kubernetes +### Install Kubernetes - Follow the instructions for [kubeadm installation](https://kubernetes.io/docs/setup/independent/install-kubeadm/). @@ -57,39 +50,7 @@ Follow the instructions to $ command -v kubeadm ``` -## Configure containerd to use Kata Containers - -The CRI `containerd` plugin supports configuration for two runtime types. - -- **Default runtime:** - - A runtime that is used by default to run workloads. - -- **Untrusted workload runtime:** - - A runtime that will be used to run untrusted workloads. This is appropriate - for workloads that require a higher degree of security isolation. - -#### Define the Kata runtime as the untrusted workload runtime - -Configure `containerd` to use the Kata runtime to run untrusted workloads by -setting the `plugins.cri.containerd.untrusted_workload_runtime` -[config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md): - -```bash -$ sudo mkdir -p /etc/containerd/ -$ cat << EOT | sudo tee /etc/containerd/config.toml -[plugins] - [plugins.cri.containerd] - [plugins.cri.containerd.untrusted_workload_runtime] - runtime_type = "io.containerd.runtime.v1.linux" - runtime_engine = "/usr/bin/kata-runtime" -EOT -``` - -> **Note:** Unless configured otherwise, the default runtime is set to `runc`. - -## Configure Kubelet to use containerd +### Configure Kubelet to use containerd In order to allow kubelet to use containerd (using the CRI interface), configure the service to point to the `containerd` socket. @@ -109,7 +70,7 @@ In order to allow kubelet to use containerd (using the CRI interface), configure $ sudo systemctl daemon-reload ``` -## Configure proxy - OPTIONAL +### Configure HTTP proxy - OPTIONAL If you are behind a proxy, use the following script to configure your proxy for docker, kubelet, and containerd: From 04498bd4d2df7d5d9ee05ccc97531e31c2cbf19f Mon Sep 17 00:00:00 2001 From: Xu Wang Date: Wed, 23 Jan 2019 14:12:29 +0800 Subject: [PATCH 5/6] architecture.md: update the architecture of v1.5 - shimv2 has been introduced - vsock has been supported Fixes: #322 Signed-off-by: Xu Wang --- architecture.md | 142 +++++++++++++++++++++--------------------------- 1 file changed, 61 insertions(+), 81 deletions(-) diff --git a/architecture.md b/architecture.md index d14ca3eeea..27b1813e66 100644 --- a/architecture.md +++ b/architecture.md @@ -1,39 +1,38 @@ # Kata Containers Architecture * [Overview](#overview) - * [Hypervisor](#hypervisor) - * [Assets](#assets) - * [Guest kernel](#guest-kernel) - * [Root filesystem image](#root-filesystem-image) - * [Agent](#agent) - * [Runtime](#runtime) - * [Configuration](#configuration) - * [Significant commands](#significant-commands) - * [create](#create) - * [start](#start) - * [exec](#exec) - * [kill](#kill) - * [delete](#delete) - * [Proxy](#proxy) - * [Shim](#shim) - * [Networking](#networking) - * [Storage](#storage) - * [Kubernetes Support](#kubernetes-support) - * [Problem Statement](#problem-statement) - * [CRI-O](#cri-o) - * [OCI Annotations](#oci-annotations) - * [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes) - * [Containerd CRI Plugin (Formerly CRI-containerd)](#containerd-cri-plugin-formerly-cri-containerd) - * [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes-1) +* [Hypervisor](#hypervisor) + * [Assets](#assets) + * [Guest kernel](#guest-kernel) + * [Root filesystem image](#root-filesystem-image) +* [Agent](#agent) +* [Runtime](#runtime) + * [Configuration](#configuration) + * [Significant commands](#significant-commands) + * [create](#create) + * [start](#start) + * [exec](#exec) + * [kill](#kill) + * [delete](#delete) +* [Proxy](#proxy) +* [Shim](#shim) +* [Networking](#networking) +* [Storage](#storage) +* [Kubernetes Support](#kubernetes-support) + * [Problem Statement](#problem-statement) + * [Containerd](#containerd) + * [CRI-O](#cri-o) + * [OCI Annotations](#oci-annotations) + * [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes) * [Appendices](#appendices) * [DAX](#dax) ## Overview -This is an architectural overview of Kata Containers, based on the 1.2.0 release. +This is an architectural overview of Kata Containers, based on the 1.5.0 release. The two primary deliverables of the Kata Containers project are a container runtime -and a CRI friendly library API. +and a CRI friendly shim. There is also a CRI friendly library API behind them. The [Kata Containers runtime (kata-runtime)](https://github.com/kata-containers/runtime) is compatible with the [OCI](https://github.com/opencontainers) [runtime specification](https://github.com/opencontainers/runtime-spec) @@ -45,29 +44,39 @@ through the [CRI-O\*](https://github.com/kubernetes-incubator/cri-o) and select between the [default Docker and CRI shim runtime (runc)](https://github.com/opencontainers/runc) and `kata-runtime`. -![Docker and Kata Containers](arch-images/docker-kata.png) - `kata-runtime` creates a QEMU\*/KVM virtual machine for each container or pod, the Docker engine or Kubernetes' `kubelet` creates respectively. +![Docker and Kata Containers](arch-images/docker-kata.png) + +The [`containerd-shim-kata-v2` (shown as `shimv2` from this point onwards)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2) +is another Kata Containers entrypoint, which +implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata. +With `shimv2`, kubernetes can launch Pod and OCI compatible containers with one shim (the `shimv2`) per Pod instead +of `2N+1` shims (a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself), and no standalone +`kata-proxy` process even if no vsock is available. + +![kubernetes integration with shimv2](arch-images/shimv2.svg) + The container process is then spawned by [agent](https://github.com/kata-containers/agent), an agent process running as a daemon inside the virtual machine. kata-agent runs a gRPC server in -the guest using a virtio serial interface which QEMU exposes as a serial -device on the host. kata-runtime uses a gRPC protocol to communicate with +the guest using a virtio serial or vsock interface which QEMU exposes as a socket +file on the host. kata-runtime uses a gRPC protocol to communicate with the agent. This protocol allows the runtime to send container management -commands to the agent. The protocol is also used to pass I/O streams (stdout, -stderr, stdin) between the guest and the Docker Engine. +commands to the agent. The protocol is also used to carry the I/O streams (stdout, +stderr, stdin) between the containers and the manage engines (e.g. Docker Engine). For any given container, both the init process and all potentially executed commands within that container, together with their related I/O streams, need -to go through the virtio serial interface exported by QEMU. A [Kata Containers +to go through the virtio serial or vsock interface exported by QEMU. +In the virtio serial case, a [Kata Containers proxy (`kata-proxy`)](https://github.com/kata-containers/proxy) instance is launched for each virtual machine to handle multiplexing and demultiplexing those commands and streams. On the host, each container process's removal is handled by a reaper in the higher -layers of the container stack. In the case of Docker it is handled by `containerd-shim`. +layers of the container stack. In the case of Docker or containerd it is handled by `containerd-shim`. In the case of CRI-O it is handled by `conmon`. For clarity, for the remainder of this document the term "container process reaper" will be used to refer to either reaper. As Kata Containers processes run inside their own virtual machines, @@ -81,6 +90,10 @@ reaper. `kata-runtime` creates a `kata-shim` daemon for each container and for e OCI command received to run within an already running container (example, `docker exec`). +Since Kata Containers version 1.5, the new introduced `shimv2` has integrated the +functionalities of the reaper, the `kata-runtime`, the `kata-shim`, and the `kata-proxy`. +As a result, there will not be any of the additional processes previously listed. + The container workload, that is, the actual OCI bundle rootfs, is exported from the host to the virtual machine. In the case where a block-based graph driver is configured, virtio-scsi will be used. In all other cases a 9pfs virtio mount point @@ -562,6 +575,17 @@ a previously created virtual machine. In both cases it will get called with very arguments, so it needs the help of the Kubernetes CRI runtime to be able to distinguish a pod creation request from a container one. +### Containerd + +As of Kata Containers 1.5, using `shimv2` with containerd 1.2.0 or above is the preferred +way to run Kata Containers with Kubernetes ([see the howto](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)). +The CRI-O will catch up soon ([kubernetes-sigs/cri-o#2024](https://github.com/kubernetes-sigs/cri-o/issues/2024)). + +Refer to the following how-to guides: + +- [How to use Kata Containers and Containerd](../how-to/containerd-kata.md) +- [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](../how-to/how-to-use-k8s-with-cri-containerd-and-kata.md) + ### CRI-O #### OCI annotations @@ -607,6 +631,9 @@ with a Kubernetes pod: #### Mixing VM based and namespace based runtimes +> **Note:** Since Kubernetes 1.12, the `[Kubernetes RuntimeClass](how-to/containerd-kata.md#kubernetes-runtimeclass)` +> has been supported and the user can specify runtime without the non-standardized annotations. + One interesting evolution of the CRI-O support for `kata-runtime` is the ability to run virtual machine based pods alongside namespace ones. With CRI-O and Kata Containers, one can introduce the concept of workload trust inside a Kubernetes @@ -636,53 +663,6 @@ a pod is **not** `Privileged` the runtime selection is done as follows: | Default CRI-O trust level: `untrusted` | Kata Containers | Kata Containers | Kata Containers | -### Containerd CRI Plugin (Formerly CRI-containerd) - -The general guidelines for the Containerd CRI Plugin support is similar to the CRI-O support. You can run trusted workloads with a runtime like `runc` and then run an untrusted workload with Kata Containers. The parameters that you can modify in the containerd config to run Kata Containers along with another 'trusted' runtime are the following: - - -``` -# /etc/containerd/config.toml - -[plugins.cri] - - [plugins.cri.containerd] - - # "plugins.cri.containerd.default_runtime" is the runtime to use in containerd. - [plugins.cri.containerd.default_runtime] - # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux - runtime_type = "io.containerd.runtime.v1.linux" - - # runtime_engine is the name of the runtime engine used by containerd. - runtime_engine = "" - - # runtime_root is the directory used by containerd for runtime state. - runtime_root = "" - - # "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it. - [plugins.cri.containerd.untrusted_workload_runtime] - # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux - runtime_type = "io.containerd.runtime.v1.linux" - - # runtime_engine is the name of the runtime engine used by containerd. - runtime_engine = "/usr/bin/kata-runtime" - - # runtime_root is the directory used by containerd for runtime state. - runtime_root = "" -``` - -You can find more information on the [Containerd config documentation](https://github.com/containerd/cri/blob/master/docs/config.md) - - -#### Mixing VM based and namespace based runtimes - -The CRI Plugin supports the following annotation in a Kubernetes pod to identify as an untrusted workload - -``` -annotations: - io.kubernetes.cri.untrusted-workload: "true" -``` - # Appendices ## DAX From 9343ece91ef8d54861cb344ced2d8a791607403a Mon Sep 17 00:00:00 2001 From: Xu Wang Date: Wed, 23 Jan 2019 14:12:40 +0800 Subject: [PATCH 6/6] how-to: a standalone doc for containerd integration including shimv2 and runtime class Fixes: 322 Signed-off-by: Xu Wang --- how-to/containerd-kata.md | 359 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 359 insertions(+) create mode 100644 how-to/containerd-kata.md diff --git a/how-to/containerd-kata.md b/how-to/containerd-kata.md new file mode 100644 index 0000000000..41e179a66e --- /dev/null +++ b/how-to/containerd-kata.md @@ -0,0 +1,359 @@ +# How to use Kata Containers and Containerd + +- [Concepts](#concepts) + - [Kubernetes RuntimeClass](#kubernetes-runtimeclass) + - [Containerd Runtime V2 API (Shim V2 API)](#containerd-runtime-v2-api-(shim-v2-api)) +- [Install](#install) + - [Install Kata Containers](#install-kata-containers) + - [Install containerd with cri plugin](#install-containerd-with-cri-plugin) + - [Install CNI plugins](#install-cni-plugins) + - [Install cri-tools](#install-cri-tools) +- [Configuration](#configuration) + - [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers) + - [Configuration for cri-tools](#configuration-for-cri-tools) +- [Run](#run) + - [Launch containers with ctr command line](#launch-containers-with-ctr-command-line) + - [Launch Pods with crictl command line](#launch-pods-with-crictl-command-line) + +This document covers the installation and configuration of [containerd](https://containerd.io/) +and [Kata Containers](https://katacontainers.io). The containerd provides not only the `ctr` +command line tool, but also the [CRI](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) +interface for [Kubernetes](https://kubernetes.io) and other CRI clients. + +This document is primarily written for Kata Containers v1.5.0-rc2 or above, and containerd v1.2.0 or above. +Previous versions are addressed here, but we suggest users upgrade to the newer versions for better support. + +## Concepts + +### Kubernetes RuntimeClass + +[RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/) is a Kubernetes feature first +introduced in Kubernetes 1.12 as alpha. It is the feature for selecting the container runtime configuration to +use to run a pod’s containers. This feature is supported in `containerd` since [v1.2.0](https://github.com/containerd/containerd/releases/tag/v1.2.0). + +Before the RuntimeClass was introduced, Kubernetes was not aware of the difference of runtimes on the node. `kubelet` +creates Pod sandboxes and containers through CRI implementations, and treats all the Pods equally. However, there +are requirements to run trusted Pods (i.e. kubernetes plugin) in a native container like runC, and to run untrusted +workloads with isolated sandboxes (i.e. Kata Containers). + +As a result, the CRI implementations extended their semanitics for the requirements: + +- At the beginning, [frakti](https://github.com/kubernetes/frakti) checks the network configuration of a Pod, and + treat Pod with `host` network as trusted, while others are treated as untrusted. +- The containerd introduced an annotation for untrusted Pods since [v1.0](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md): + ```yaml + annotations: + io.kubernetes.cri.untrusted-workload: "true" + ``` +- Similarly, CRI-O introduced the annotation `io.kubernetes.cri-o.TrustedSandbox` for untrusted Pods. + +To eliminate the complexity of user configuration introduced by the non-standardized annotations and provide +extensibility, `RuntimeClass` was introduced. This gives users the ability to affect the runtime behavior +through 'RuntimeClass' without the knowledge of the CRI daemons. We suggest that users with multiple runtimes +use 'RuntimeClass' instead of the deprecated annotations. + +### Containerd Runtime V2 API (Shim V2 API) + +The [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2) +implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata. +With `shimv2`, kubernetes can launch Pod and OCI-compatible containers with one shim per Pod. Prior to `shimv2`, `2N+1` +shims (i.e. a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself) and no standalone `kata-proxy` +process were used, even with vsock not available. + +![kubernetes integration with shimv2](../arch-images/shimv2.svg) + +The shim v2 is introduced in containerd [v1.2.0](https://github.com/containerd/containerd/releases/tag/v1.2.0) and kata `shimv2` +is implemented in Kata Containers v1.5.0. + +## Install + +### Install Kata Containers + +Follow the instructions to [install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md). + +### Install containerd with cri plugin + +> **Note:** `cri` is a native plugin of containerd 1.1 and above. It is built into containerd and enabled by default. +> You do not need to install `cri` if you have containerd 1.1 or above. Just check if the `cri` plugin has been disabled +> in the containerd configuration file. + +Follow the instructions from the [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md). + +Then, check if `containerd` is now available: + +```bash +$ command -v containerd +``` + +### Install CNI plugins + +> **Note:** You do not need to install CNI plugins if you do not want to use containerd with Kubernetes. +> If you have installed Kubernetes with `kubeadm`, you might have already installed the CNI plugins. + +You can manually install CNI plugins as follows: + +```bash +$ go get github.com/containernetworking/plugins +$ pushd $GOPATH/src/github.com/containernetworking/plugins +$ ./build_linux.sh +$ sudo mkdir /opt/cni +$ sudo cp -r bin /opt/cni/ +$ popd +``` + +### Install cri-tools + +> **Note:** `cri-tools` is a set of tools for CRI used for development and testing. Users who only want +> to use containerd with kubernetes can skip the `cri-tools`. + +You can install the `cri-tools` from source code: + +```bash +$ go get github.com/kubernetes-incubator/cri-tools +$ pushd $GOPATH/src/github.com/kubernetes-incubator/cri-tools +$ make +$ sudo -E make install +$ popd +``` + +## Configuration + +### Configure containerd to use Kata Containers + +By default, the configuration of containerd is located at `/etc/containerd/config.toml`, and the +`cri` plugins are placed in the following section: + +```toml +[plugins] + [plugins.cri] + [plugins.cri.containerd] + [plugins.cri.containerd.default_runtime] + #runtime_type = "io.containerd.runtime.v1.linux" + + [plugins.cri.cni] + # conf_dir is the directory in which the admin places a CNI conf. + conf_dir = "/etc/cni/net.d" +``` + +The following sections outline how to add Kata Containers to the configurations. + +#### Kata Containers as a RuntimeClass + +For +- Kata Containers v1.5.0 or above (including 1.5.0-rc) +- Containerd v1.2.0 or above +- Kubernetes v1.12.0 or above + +The RuntimeClass is suggested. + +The following configuration includes three runtime classes: +- `plugins.cri.containerd.runtimes.runc`: the runC, and it is the default runtime. +- `plugins.cri.containerd.runtimes.kata`: The function in containerd (reference [the document here](https://github.com/containerd/containerd/tree/master/runtime/v2#binary-naming)) + where the dot-connected string `io.containerd.kata.v2` is translated to `containerd-shim-kata-v2` (i.e. the + binary name of the kata implementation of [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2)). +- `plugins.cri.containerd.runtimes.katacli`: the `containerd-shim-runc-v1` calls `kata-runtime`, which is the legacy process. + +```toml + [plugins.cri.containerd] + no_pivot = false + [plugins.cri.containerd.runtimes] + [plugins.cri.containerd.runtimes.runc] + runtime_type = "io.containerd.runc.v1" + [plugins.cri.containerd.runtimes.runc.options] + NoPivotRoot = false + NoNewKeyring = false + ShimCgroup = "" + IoUid = 0 + IoGid = 0 + BinaryName = "runc" + Root = "" + CriuPath = "" + SystemdCgroup = false + [plugins.cri.containerd.runtimes.kata] + runtime_type = "io.containerd.kata.v2" + [plugins.cri.containerd.runtimes.katacli] + runtime_type = "io.containerd.runc.v1" + [plugins.cri.containerd.runtimes.katacli.options] + NoPivotRoot = false + NoNewKeyring = false + ShimCgroup = "" + IoUid = 0 + IoGid = 0 + BinaryName = "/usr/bin/kata-runtime" + Root = "" + CriuPath = "" + SystemdCgroup = false +``` + +Based on the discussion in [containerd/containerd#2916](https://github.com/containerd/containerd/pull/2916), +the containerd configuration will have a field for runtime configuration files (likely to be`config_file`). However, +the field is not available yet, and the `shimv2` will search the configuration file in the default paths, same as +`kata-runtime` cli): `/etc/kata-containers/configuration.toml`, `/usr/share/defaults/kata-containers/configuration.toml`. +We will update this document once the `config_file` field is available. + +If you want to specify a configure file now, the following provides a workaround. The `shimv2` accepts an environment +variable, `KATA_CONF_FILE` for the config file path. Then, you can create a shell script with the following: + +```bash +#!/bin/bash +KATA_CONF_FILE=/etc/kata-containers/firecracker.toml containerd-shim-kata-v2 +``` + +Name it as `/usr/local/bin/containerd-shim-katafc-v2` and reference it in the configuration of containerd: + +``` + [plugins.cri.containerd.runtimes.kata-firecracker] + runtime_type = "io.containerd.katafc.v2" +``` + +#### Kata Containers as the runtime for untrusted workload + +For cases without RuntimeClass support, we can use the legacy annotation method to support using Kata Containers +for an untrusted workload. With the following configuration, you can run trusted workloads with a runtime such as `runc` +and then, run an untrusted workload with Kata Containers: + +```toml + [plugins.cri.containerd] + # "plugins.cri.containerd.default_runtime" is the runtime to use in containerd. + [plugins.cri.containerd.default_runtime] + # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux + runtime_type = "io.containerd.runtime.v1.linux" + + # "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it. + [plugins.cri.containerd.untrusted_workload_runtime] + # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux + runtime_type = "io.containerd.kata.v2" +``` + +For the earlier versions of Kata Containers and containerd that do not support Runtime V2 (Shim API), you can use the following alternative configuration: + +```toml + [plugins.cri.containerd] + + # "plugins.cri.containerd.default_runtime" is the runtime to use in containerd. + [plugins.cri.containerd.default_runtime] + # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux + runtime_type = "io.containerd.runtime.v1.linux" + + # "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it. + [plugins.cri.containerd.untrusted_workload_runtime] + # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux + runtime_type = "io.containerd.runtime.v1.linux" + + # runtime_engine is the name of the runtime engine used by containerd. + runtime_engine = "/usr/bin/kata-runtime" +``` + +You can find more information on the [Containerd config documentation](https://github.com/containerd/cri/blob/master/docs/config.md) + + +#### Kata Containers as the default runtime + +If you want to set Kata Containers as the only runtime in the deployment, you can simply configure as follows: + +```toml + [plugins.cri.containerd] + [plugins.cri.containerd.default_runtime] + runtime_type = "io.containerd.kata.v2" +``` + +Alternatively, for the earlier versions of Kata Containers and containerd that do not support Runtime V2 (Shim API), you can use the following alternative configuration: + +```toml + [plugins.cri.containerd] + [plugins.cri.containerd.default_runtime] + runtime_type = "io.containerd.runtime.v1.linux" + runtime_engine = "/usr/bin/kata-runtime" +``` + +### Configuration for cri-tools + +> **Note:** If you skipped the [Install cri-tools](#install-cri-tools) section, you can skip this section too. + +First, add the CNI configuration in the containerd configuration. + +The following is the configuration if you installed CNI as the *[Install CNI plugins](#install-cni-plugins)* section outlined. + +Put the CNI configuration as `/etc/cni/net.d/10-mynet.conf`: + +```json +{ + "cniVersion": "0.2.0", + "name": "mynet", + "type": "bridge", + "bridge": "cni0", + "isGateway": true, + "ipMasq": true, + "ipam": { + "type": "host-local", + "subnet": "172.19.0.0/24", + "routes": [ + { "dst": "0.0.0.0/0" } + ] + } +} +``` + +Next, reference the configuration directory through containerd `config.toml`: + +```toml +[plugins.cri.cni] + # conf_dir is the directory in which the admin places a CNI conf. + conf_dir = "/etc/cni/net.d" +``` + +The configuration file of `crictl` command line tool in `cri-tools` locates at `/etc/crictl.yaml`: + +```yaml +runtime-endpoint: unix:///var/run/containerd/containerd.sock +image-endpoint: unix:///var/run/containerd/containerd.sock +timeout: 10 +debug: true +``` + +## Run + +### Launch containers with ctr command line + +To run a container with Kata Containers through the containerd command line, you can run the following: + +```bash +$ sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh +``` + +This launchs a busybox container named `hello`, and it will be removed by `--rm` after it quits. + +### Launch Pods with crictl command line + +With the `crictl` command line of `cri-tools`, you can specify runtime class with `-r` or `--runtime` flag. + Use the following to launch Pod with `kata` runtime class with the pod in [the example](https://github.com/kubernetes-sigs/cri-tools/tree/master/docs/examples) +of `cri-tools`: + +```bash +$ sudo crictl runp -r kata podsandbox-config.yaml +36e23521e8f89fabd9044924c9aeb34890c60e85e1748e8daca7e2e673f8653e +``` + +You can add container to the launched Pod with the following: + +```bash +$ sudo crictl create 36e23521e8f89 container-config.yaml podsandbox-config.yaml +1aab7585530e62c446734f12f6899f095ce53422dafcf5a80055ba11b95f2da7 +``` + +Now, start it with the following: + +```bash +$ sudo crictl start 1aab7585530e6 +1aab7585530e6 +``` + +In Kubernetes, you need to create a RuntimeClass resource and add the RuntimeClass field in the Pod Spec +(see this [document](https://kubernetes.io/docs/concepts/containers/runtime-class/) for more information). + +If RuntimeClass is not supported, you can use the following annotation in a Kubernetes pod to identify as an untrusted workload: + +```yaml +annotations: + io.kubernetes.cri.untrusted-workload: "true" +```