github/kata-containers
1
0
Fork 1
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-06-28 08:17:37 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #332 from gnawux/shimv2

Browse Source 
document the new introduced shimv2
This commit is contained in:
Peng Tao 2019-01-24 10:43:22 +08:00 committed by GitHub
commit c60b72f5d1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 451 additions and 144 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
Developer-Guide.md
View File

@ -33,10 +33,10 @@
* [Run Kata Containers with Kubernetes](#run-kata-containers-with-kubernetes)
* [Install a CRI implementation](#install-a-cri-implementation)
* [CRI-O](#cri-o)
* [CRI-containerd](#cri-containerd)
* [containerd with cri plugin](#containerd-with-cri-plugin)
* [Install Kubernetes](#install-kubernetes)
* [Configure for CRI-O](#configure-for-cri-o)
* [Configure for CRI-containerd](#configure-for-cri-containerd)
* [Configure for containerd](#configure-for-containerd)
* [Run a Kubernetes pod with Kata Containers](#run-a-kubernetes-pod-with-kata-containers)
* [Troubleshoot Kata Containers](#troubleshoot-kata-containers)
* [Appendices](#appendices)
@ -422,6 +422,10 @@ choose the one that you want, but you have to pick one. After choosing
either CRI-O or CRI-containerd, you must make the appropriate changes
to ensure it relies on the Kata Containers runtime.
As of Kata Containers 1.5, using `shimv2` with containerd 1.2.0 or above is the preferred
way to run Kata Containers with Kubernetes ([see the howto](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)).
The CRI-O will catch up soon.
### CRI-O
If you select CRI-O, follow the "CRI-O Tutorial" instructions
@ -486,13 +490,13 @@ Restart CRI-O to take changes into account
$ sudo systemctl restart crio
```
### CRI-containerd
### containerd with cri plugin
If you select CRI-containerd, follow the "Getting Started for Developers"
If you select containerd with `cri` plugin, follow the "Getting Started for Developers"
instructions [here](https://github.com/containerd/cri#getting-started-for-developers)
to properly install it.
To customize CRI-containerd to select Kata Containers runtime, follow our
To customize containerd to select Kata Containers runtime, follow our
"Configure containerd to use Kata Containers" internal documentation
[here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers).
@ -515,14 +519,14 @@ implementation you chose, and the kubelet service has to be updated accordingly.
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock"
```
### Configure for CRI-containerd
### Configure for containerd
`/etc/systemd/system/kubelet.service.d/0-cri-containerd.conf`
```
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
```
For more information about CRI-containerd see the "Configure Kubelet to use containerd"
For more information about containerd see the "Configure Kubelet to use containerd"
documentation [here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-kubelet-to-use-containerd).
## Run a Kubernetes pod with Kata Containers

1
arch-images/shimv2.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 190 KiB

142
architecture.md
View File

@ -1,39 +1,38 @@
# Kata Containers Architecture
* [Overview](#overview)
* [Hypervisor](#hypervisor)
* [Assets](#assets)
* [Guest kernel](#guest-kernel)
* [Root filesystem image](#root-filesystem-image)
* [Agent](#agent)
* [Runtime](#runtime)
* [Configuration](#configuration)
* [Significant commands](#significant-commands)
* [create](#create)
* [start](#start)
* [exec](#exec)
* [kill](#kill)
* [delete](#delete)
* [Proxy](#proxy)
* [Shim](#shim)
* [Networking](#networking)
* [Storage](#storage)
* [Kubernetes Support](#kubernetes-support)
* [Problem Statement](#problem-statement)
* [CRI-O](#cri-o)
* [OCI Annotations](#oci-annotations)
* [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes)
* [Containerd CRI Plugin (Formerly CRI-containerd)](#containerd-cri-plugin-formerly-cri-containerd)
* [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes-1)
* [Hypervisor](#hypervisor)
* [Assets](#assets)
* [Guest kernel](#guest-kernel)
* [Root filesystem image](#root-filesystem-image)
* [Agent](#agent)
* [Runtime](#runtime)
* [Configuration](#configuration)
* [Significant commands](#significant-commands)
* [create](#create)
* [start](#start)
* [exec](#exec)
* [kill](#kill)
* [delete](#delete)
* [Proxy](#proxy)
* [Shim](#shim)
* [Networking](#networking)
* [Storage](#storage)
* [Kubernetes Support](#kubernetes-support)
* [Problem Statement](#problem-statement)
* [Containerd](#containerd)
* [CRI-O](#cri-o)
* [OCI Annotations](#oci-annotations)
* [Mixing VM based and namespace based runtimes](#mixing-vm-based-and-namespace-based-runtimes)
* [Appendices](#appendices)
* [DAX](#dax)
## Overview
This is an architectural overview of Kata Containers, based on the 1.2.0 release.
This is an architectural overview of Kata Containers, based on the 1.5.0 release.
The two primary deliverables of the Kata Containers project are a container runtime
and a CRI friendly library API.
and a CRI friendly shim. There is also a CRI friendly library API behind them.
The [Kata Containers runtime (kata-runtime)](https://github.com/kata-containers/runtime)
is compatible with the [OCI](https://github.com/opencontainers) [runtime specification](https://github.com/opencontainers/runtime-spec)
@ -45,29 +44,39 @@ through the [CRI-O\*](https://github.com/kubernetes-incubator/cri-o) and
select between the [default Docker and CRI shim runtime (runc)](https://github.com/opencontainers/runc)
and `kata-runtime`.
![Docker and Kata Containers](arch-images/docker-kata.png)
`kata-runtime` creates a QEMU\*/KVM virtual machine for each container or pod,
the Docker engine or Kubernetes' `kubelet` creates respectively.
![Docker and Kata Containers](arch-images/docker-kata.png)
The [`containerd-shim-kata-v2` (shown as `shimv2` from this point onwards)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2)
is another Kata Containers entrypoint, which
implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata.
With `shimv2`, kubernetes can launch Pod and OCI compatible containers with one shim (the `shimv2`) per Pod instead
of `2N+1` shims (a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself), and no standalone
`kata-proxy` process even if no vsock is available.
![kubernetes integration with shimv2](arch-images/shimv2.svg)
The container process is then spawned by
[agent](https://github.com/kata-containers/agent), an agent process running
as a daemon inside the virtual machine. kata-agent runs a gRPC server in
the guest using a virtio serial interface which QEMU exposes as a serial
device on the host. kata-runtime uses a gRPC protocol to communicate with
the guest using a virtio serial or vsock interface which QEMU exposes as a socket
file on the host. kata-runtime uses a gRPC protocol to communicate with
the agent. This protocol allows the runtime to send container management
commands to the agent. The protocol is also used to pass I/O streams (stdout,
stderr, stdin) between the guest and the Docker Engine.
commands to the agent. The protocol is also used to carry the I/O streams (stdout,
stderr, stdin) between the containers and the manage engines (e.g. Docker Engine).
For any given container, both the init process and all potentially executed
commands within that container, together with their related I/O streams, need
to go through the virtio serial interface exported by QEMU. A [Kata Containers
to go through the virtio serial or vsock interface exported by QEMU.
In the virtio serial case, a [Kata Containers
proxy (`kata-proxy`)](https://github.com/kata-containers/proxy) instance is
launched for each virtual machine to handle multiplexing and demultiplexing
those commands and streams.
On the host, each container process's removal is handled by a reaper in the higher
layers of the container stack. In the case of Docker it is handled by `containerd-shim`.
layers of the container stack. In the case of Docker or containerd it is handled by `containerd-shim`.
In the case of CRI-O it is handled by `conmon`. For clarity, for the remainder
of this document the term "container process reaper" will be used to refer to
either reaper. As Kata Containers processes run inside their own virtual machines,
@ -81,6 +90,10 @@ reaper. `kata-runtime` creates a `kata-shim` daemon for each container and for e
OCI command received to run within an already running container (example, `docker
exec`).
Since Kata Containers version 1.5, the new introduced `shimv2` has integrated the
functionalities of the reaper, the `kata-runtime`, the `kata-shim`, and the `kata-proxy`.
As a result, there will not be any of the additional processes previously listed.
The container workload, that is, the actual OCI bundle rootfs, is exported from the
host to the virtual machine. In the case where a block-based graph driver is
configured, virtio-scsi will be used. In all other cases a 9pfs virtio mount point
@ -562,6 +575,17 @@ a previously created virtual machine. In both cases it will get called with very
arguments, so it needs the help of the Kubernetes CRI runtime to be able to distinguish a
pod creation request from a container one.
### Containerd
As of Kata Containers 1.5, using `shimv2` with containerd 1.2.0 or above is the preferred
way to run Kata Containers with Kubernetes ([see the howto](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)).
The CRI-O will catch up soon ([kubernetes-sigs/cri-o#2024](https://github.com/kubernetes-sigs/cri-o/issues/2024)).
Refer to the following how-to guides:
- [How to use Kata Containers and Containerd](../how-to/containerd-kata.md)
- [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](../how-to/how-to-use-k8s-with-cri-containerd-and-kata.md)
### CRI-O
#### OCI annotations
@ -607,6 +631,9 @@ with a Kubernetes pod:
#### Mixing VM based and namespace based runtimes
> **Note:** Since Kubernetes 1.12, the `[Kubernetes RuntimeClass](how-to/containerd-kata.md#kubernetes-runtimeclass)`
> has been supported and the user can specify runtime without the non-standardized annotations.
One interesting evolution of the CRI-O support for `kata-runtime` is the ability
to run virtual machine based pods alongside namespace ones. With CRI-O and Kata
Containers, one can introduce the concept of workload trust inside a Kubernetes
@ -636,53 +663,6 @@ a pod is **not** `Privileged` the runtime selection is done as follows:
| Default CRI-O trust level: `untrusted` | Kata Containers | Kata Containers | Kata Containers |
### Containerd CRI Plugin (Formerly CRI-containerd)
The general guidelines for the Containerd CRI Plugin support is similar to the CRI-O support. You can run trusted workloads with a runtime like `runc` and then run an untrusted workload with Kata Containers. The parameters that you can modify in the containerd config to run Kata Containers along with another 'trusted' runtime are the following:
```
# /etc/containerd/config.toml
[plugins.cri]
[plugins.cri.containerd]
# "plugins.cri.containerd.default_runtime" is the runtime to use in containerd.
[plugins.cri.containerd.default_runtime]
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
runtime_type = "io.containerd.runtime.v1.linux"
# runtime_engine is the name of the runtime engine used by containerd.
runtime_engine = ""
# runtime_root is the directory used by containerd for runtime state.
runtime_root = ""
# "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it.
[plugins.cri.containerd.untrusted_workload_runtime]
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
runtime_type = "io.containerd.runtime.v1.linux"
# runtime_engine is the name of the runtime engine used by containerd.
runtime_engine = "/usr/bin/kata-runtime"
# runtime_root is the directory used by containerd for runtime state.
runtime_root = ""
```
You can find more information on the [Containerd config documentation](https://github.com/containerd/cri/blob/master/docs/config.md)
#### Mixing VM based and namespace based runtimes
The CRI Plugin supports the following annotation in a Kubernetes pod to identify as an untrusted workload
```
annotations:
io.kubernetes.cri.untrusted-workload: "true"
```
# Appendices
## DAX

359
how-to/containerd-kata.md Normal file
View File

@ -0,0 +1,359 @@
# How to use Kata Containers and Containerd
- [Concepts](#concepts)
- [Kubernetes RuntimeClass](#kubernetes-runtimeclass)
- [Containerd Runtime V2 API (Shim V2 API)](#containerd-runtime-v2-api-(shim-v2-api))
- [Install](#install)
- [Install Kata Containers](#install-kata-containers)
- [Install containerd with cri plugin](#install-containerd-with-cri-plugin)
- [Install CNI plugins](#install-cni-plugins)
- [Install cri-tools](#install-cri-tools)
- [Configuration](#configuration)
- [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers)
- [Configuration for cri-tools](#configuration-for-cri-tools)
- [Run](#run)
- [Launch containers with ctr command line](#launch-containers-with-ctr-command-line)
- [Launch Pods with crictl command line](#launch-pods-with-crictl-command-line)
This document covers the installation and configuration of [containerd](https://containerd.io/)
and [Kata Containers](https://katacontainers.io). The containerd provides not only the `ctr`
command line tool, but also the [CRI](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)
interface for [Kubernetes](https://kubernetes.io) and other CRI clients.
This document is primarily written for Kata Containers v1.5.0-rc2 or above, and containerd v1.2.0 or above.
Previous versions are addressed here, but we suggest users upgrade to the newer versions for better support.
## Concepts
### Kubernetes RuntimeClass
[RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/) is a Kubernetes feature first
introduced in Kubernetes 1.12 as alpha. It is the feature for selecting the container runtime configuration to
use to run a pod’s containers. This feature is supported in `containerd` since [v1.2.0](https://github.com/containerd/containerd/releases/tag/v1.2.0).
Before the RuntimeClass was introduced, Kubernetes was not aware of the difference of runtimes on the node. `kubelet`
creates Pod sandboxes and containers through CRI implementations, and treats all the Pods equally. However, there
are requirements to run trusted Pods (i.e. kubernetes plugin) in a native container like runC, and to run untrusted
workloads with isolated sandboxes (i.e. Kata Containers).
As a result, the CRI implementations extended their semanitics for the requirements:
- At the beginning, [frakti](https://github.com/kubernetes/frakti) checks the network configuration of a Pod, and
treat Pod with `host` network as trusted, while others are treated as untrusted.
- The containerd introduced an annotation for untrusted Pods since [v1.0](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md):
```yaml
annotations:
io.kubernetes.cri.untrusted-workload: "true"
```
- Similarly, CRI-O introduced the annotation `io.kubernetes.cri-o.TrustedSandbox` for untrusted Pods.
To eliminate the complexity of user configuration introduced by the non-standardized annotations and provide
extensibility, `RuntimeClass` was introduced. This gives users the ability to affect the runtime behavior
through 'RuntimeClass' without the knowledge of the CRI daemons. We suggest that users with multiple runtimes
use 'RuntimeClass' instead of the deprecated annotations.
### Containerd Runtime V2 API (Shim V2 API)
The [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2)
implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata.
With `shimv2`, kubernetes can launch Pod and OCI-compatible containers with one shim per Pod. Prior to `shimv2`, `2N+1`
shims (i.e. a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself) and no standalone `kata-proxy`
process were used, even with vsock not available.
![kubernetes integration with shimv2](../arch-images/shimv2.svg)
The shim v2 is introduced in containerd [v1.2.0](https://github.com/containerd/containerd/releases/tag/v1.2.0) and kata `shimv2`
is implemented in Kata Containers v1.5.0.
## Install
### Install Kata Containers
Follow the instructions to [install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md).
### Install containerd with cri plugin
> **Note:** `cri` is a native plugin of containerd 1.1 and above. It is built into containerd and enabled by default.
> You do not need to install `cri` if you have containerd 1.1 or above. Just check if the `cri` plugin has been disabled
> in the containerd configuration file.
Follow the instructions from the [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md).
Then, check if `containerd` is now available:
```bash
$ command -v containerd
```
### Install CNI plugins
> **Note:** You do not need to install CNI plugins if you do not want to use containerd with Kubernetes.
> If you have installed Kubernetes with `kubeadm`, you might have already installed the CNI plugins.
You can manually install CNI plugins as follows:
```bash
$ go get github.com/containernetworking/plugins
$ pushd $GOPATH/src/github.com/containernetworking/plugins
$ ./build_linux.sh
$ sudo mkdir /opt/cni
$ sudo cp -r bin /opt/cni/
$ popd
```
### Install cri-tools
> **Note:** `cri-tools` is a set of tools for CRI used for development and testing. Users who only want
> to use containerd with kubernetes can skip the `cri-tools`.
You can install the `cri-tools` from source code:
```bash
$ go get github.com/kubernetes-incubator/cri-tools
$ pushd $GOPATH/src/github.com/kubernetes-incubator/cri-tools
$ make
$ sudo -E make install
$ popd
```
## Configuration
### Configure containerd to use Kata Containers
By default, the configuration of containerd is located at `/etc/containerd/config.toml`, and the
`cri` plugins are placed in the following section:
```toml
[plugins]
[plugins.cri]
[plugins.cri.containerd]
[plugins.cri.containerd.default_runtime]
#runtime_type = "io.containerd.runtime.v1.linux"
[plugins.cri.cni]
# conf_dir is the directory in which the admin places a CNI conf.
conf_dir = "/etc/cni/net.d"
```
The following sections outline how to add Kata Containers to the configurations.
#### Kata Containers as a RuntimeClass
For
- Kata Containers v1.5.0 or above (including 1.5.0-rc)
- Containerd v1.2.0 or above
- Kubernetes v1.12.0 or above
The RuntimeClass is suggested.
The following configuration includes three runtime classes:
- `plugins.cri.containerd.runtimes.runc`: the runC, and it is the default runtime.
- `plugins.cri.containerd.runtimes.kata`: The function in containerd (reference [the document here](https://github.com/containerd/containerd/tree/master/runtime/v2#binary-naming))
where the dot-connected string `io.containerd.kata.v2` is translated to `containerd-shim-kata-v2` (i.e. the
binary name of the kata implementation of [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2)).
- `plugins.cri.containerd.runtimes.katacli`: the `containerd-shim-runc-v1` calls `kata-runtime`, which is the legacy process.
```toml
[plugins.cri.containerd]
no_pivot = false
[plugins.cri.containerd.runtimes]
[plugins.cri.containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v1"
[plugins.cri.containerd.runtimes.runc.options]
NoPivotRoot = false
NoNewKeyring = false
ShimCgroup = ""
IoUid = 0
IoGid = 0
BinaryName = "runc"
Root = ""
CriuPath = ""
SystemdCgroup = false
[plugins.cri.containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
[plugins.cri.containerd.runtimes.katacli]
runtime_type = "io.containerd.runc.v1"
[plugins.cri.containerd.runtimes.katacli.options]
NoPivotRoot = false
NoNewKeyring = false
ShimCgroup = ""
IoUid = 0
IoGid = 0
BinaryName = "/usr/bin/kata-runtime"
Root = ""
CriuPath = ""
SystemdCgroup = false
```
Based on the discussion in [containerd/containerd#2916](https://github.com/containerd/containerd/pull/2916),
the containerd configuration will have a field for runtime configuration files (likely to be`config_file`). However,
the field is not available yet, and the `shimv2` will search the configuration file in the default paths, same as
`kata-runtime` cli): `/etc/kata-containers/configuration.toml`, `/usr/share/defaults/kata-containers/configuration.toml`.
We will update this document once the `config_file` field is available.
If you want to specify a configure file now, the following provides a workaround. The `shimv2` accepts an environment
variable, `KATA_CONF_FILE` for the config file path. Then, you can create a shell script with the following:
```bash
#!/bin/bash
KATA_CONF_FILE=/etc/kata-containers/firecracker.toml containerd-shim-kata-v2
```
Name it as `/usr/local/bin/containerd-shim-katafc-v2` and reference it in the configuration of containerd:
```
[plugins.cri.containerd.runtimes.kata-firecracker]
runtime_type = "io.containerd.katafc.v2"
```
#### Kata Containers as the runtime for untrusted workload
For cases without RuntimeClass support, we can use the legacy annotation method to support using Kata Containers
for an untrusted workload. With the following configuration, you can run trusted workloads with a runtime such as `runc`
and then, run an untrusted workload with Kata Containers:
```toml
[plugins.cri.containerd]
# "plugins.cri.containerd.default_runtime" is the runtime to use in containerd.
[plugins.cri.containerd.default_runtime]
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
runtime_type = "io.containerd.runtime.v1.linux"
# "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it.
[plugins.cri.containerd.untrusted_workload_runtime]
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
runtime_type = "io.containerd.kata.v2"
```
For the earlier versions of Kata Containers and containerd that do not support Runtime V2 (Shim API), you can use the following alternative configuration:
```toml
[plugins.cri.containerd]
# "plugins.cri.containerd.default_runtime" is the runtime to use in containerd.
[plugins.cri.containerd.default_runtime]
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
runtime_type = "io.containerd.runtime.v1.linux"
# "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it.
[plugins.cri.containerd.untrusted_workload_runtime]
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
runtime_type = "io.containerd.runtime.v1.linux"
# runtime_engine is the name of the runtime engine used by containerd.
runtime_engine = "/usr/bin/kata-runtime"
```
You can find more information on the [Containerd config documentation](https://github.com/containerd/cri/blob/master/docs/config.md)
#### Kata Containers as the default runtime
If you want to set Kata Containers as the only runtime in the deployment, you can simply configure as follows:
```toml
[plugins.cri.containerd]
[plugins.cri.containerd.default_runtime]
runtime_type = "io.containerd.kata.v2"
```
Alternatively, for the earlier versions of Kata Containers and containerd that do not support Runtime V2 (Shim API), you can use the following alternative configuration:
```toml
[plugins.cri.containerd]
[plugins.cri.containerd.default_runtime]
runtime_type = "io.containerd.runtime.v1.linux"
runtime_engine = "/usr/bin/kata-runtime"
```
### Configuration for cri-tools
> **Note:** If you skipped the [Install cri-tools](#install-cri-tools) section, you can skip this section too.
First, add the CNI configuration in the containerd configuration.
The following is the configuration if you installed CNI as the *[Install CNI plugins](#install-cni-plugins)* section outlined.
Put the CNI configuration as `/etc/cni/net.d/10-mynet.conf`:
```json
{
"cniVersion": "0.2.0",
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "172.19.0.0/24",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
```
Next, reference the configuration directory through containerd `config.toml`:
```toml
[plugins.cri.cni]
# conf_dir is the directory in which the admin places a CNI conf.
conf_dir = "/etc/cni/net.d"
```
The configuration file of `crictl` command line tool in `cri-tools` locates at `/etc/crictl.yaml`:
```yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
```
## Run
### Launch containers with ctr command line
To run a container with Kata Containers through the containerd command line, you can run the following:
```bash
$ sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh
```
This launchs a busybox container named `hello`, and it will be removed by `--rm` after it quits.
### Launch Pods with crictl command line
With the `crictl` command line of `cri-tools`, you can specify runtime class with `-r` or `--runtime` flag.
Use the following to launch Pod with `kata` runtime class with the pod in [the example](https://github.com/kubernetes-sigs/cri-tools/tree/master/docs/examples)
of `cri-tools`:
```bash
$ sudo crictl runp -r kata podsandbox-config.yaml
36e23521e8f89fabd9044924c9aeb34890c60e85e1748e8daca7e2e673f8653e
```
You can add container to the launched Pod with the following:
```bash
$ sudo crictl create 36e23521e8f89 container-config.yaml podsandbox-config.yaml
1aab7585530e62c446734f12f6899f095ce53422dafcf5a80055ba11b95f2da7
```
Now, start it with the following:
```bash
$ sudo crictl start 1aab7585530e6
1aab7585530e6
```
In Kubernetes, you need to create a RuntimeClass resource and add the RuntimeClass field in the Pod Spec
(see this [document](https://kubernetes.io/docs/concepts/containers/runtime-class/) for more information).
If RuntimeClass is not supported, you can use the following annotation in a Kubernetes pod to identify as an untrusted workload:
```yaml
annotations:
io.kubernetes.cri.untrusted-workload: "true"
```

71
how-to/how-to-use-k8s-with-cri-containerd-and-kata.md
View File

@ -1,13 +1,11 @@
# How to use Kata Containers and CRI (containerd plugin) with Kubernetes
* [Requirements](#requirements)
* [Install containerd with CRI plugin enabled](#install-containerd-with-cri-plugin-enabled)
* [Install Kata Containers](#install-kata-containers)
* [Install Kubernetes](#install-kubernetes)
* [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers)
* [Define the Kata runtime as the untrusted workload runtime](#define-the-kata-runtime-as-the-untrusted-workload-runtime)
* [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd)
* [Configure proxy - OPTIONAL](#configure-proxy---optional)
* [Install and configure containerd](#install-and-configure-containerd)
* [Install and configure Kubernetes](#install-and-configure-kubernetes)
* [Install Kubernetes](#install-kubernetes)
* [Configure Kubelet to use containerd](#configure-kubelet-to-use-containerd)
* [Configure HTTP proxy - OPTIONAL](#configure-http-proxy---optional)
* [Start Kubernetes](#start-kubernetes)
* [Install a Pod Network](#install-a-pod-network)
* [Allow pods to run in the master node](#allow-pods-to-run-in-the-master-node)
@ -20,10 +18,13 @@ The Kubernetes cluster will use the
[CRI containerd plugin](https://github.com/containerd/cri) and
[Kata Containers](https://katacontainers.io) to launch untrusted workloads.
For Kata Containers 1.5.0-rc2 and above, we will use [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](../design/shimv2.md)
to launch Kata Containers. For the previous version of Kata Containers, the Pods are launched with `kata-runtime`.
## Requirements
- Kubernetes, kubelet, kubeadm
- cri-containerd
- containerd with `cri` plug-in
- Kata Containers
> **Note:** For information about the supported versions of these components,
@ -31,22 +32,14 @@ The Kubernetes cluster will use the
> [versions.yaml](https://github.com/kata-containers/runtime/blob/master/versions.yaml)
> file.
## Install containerd with CRI plugin enabled
## Install and configure containerd
- Follow the instructions from the
[CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md).
First, follow the [How to use Kata Containers and Containerd](containerd-kata.md) to install and configure containerd.
Then, make sure the containerd works with the [examples in it](containerd-kata.md#run).
- Check if `containerd` is now available
```bash
$ command -v containerd
```
## Install and configure Kubernetes
## Install Kata Containers
Follow the instructions to
[install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md).
## Install Kubernetes
### Install Kubernetes
- Follow the instructions for
[kubeadm installation](https://kubernetes.io/docs/setup/independent/install-kubeadm/).
@ -57,39 +50,7 @@ Follow the instructions to
$ command -v kubeadm
```
## Configure containerd to use Kata Containers
The CRI `containerd` plugin supports configuration for two runtime types.
- **Default runtime:**
A runtime that is used by default to run workloads.
- **Untrusted workload runtime:**
A runtime that will be used to run untrusted workloads. This is appropriate
for workloads that require a higher degree of security isolation.
#### Define the Kata runtime as the untrusted workload runtime
Configure `containerd` to use the Kata runtime to run untrusted workloads by
setting the `plugins.cri.containerd.untrusted_workload_runtime`
[config option](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md):
```bash
$ sudo mkdir -p /etc/containerd/
$ cat << EOT | sudo tee /etc/containerd/config.toml
[plugins]
[plugins.cri.containerd]
[plugins.cri.containerd.untrusted_workload_runtime]
runtime_type = "io.containerd.runtime.v1.linux"
runtime_engine = "/usr/bin/kata-runtime"
EOT
```
> **Note:** Unless configured otherwise, the default runtime is set to `runc`.
## Configure Kubelet to use containerd
### Configure Kubelet to use containerd
In order to allow kubelet to use containerd (using the CRI interface), configure the service to point to the `containerd` socket.
@ -109,7 +70,7 @@ In order to allow kubelet to use containerd (using the CRI interface), configure
$ sudo systemctl daemon-reload
```
## Configure proxy - OPTIONAL
### Configure HTTP proxy - OPTIONAL
If you are behind a proxy, use the following script to configure your proxy for docker, kubelet, and containerd:

4
install/README.md
View File

@ -49,7 +49,9 @@ Manual installation instructions are available for [these distributions](#suppor
1. Add the Kata Containers repository to your distro package manager, and import the packages signing key.
2. Install the Kata Containers packages.
3. Install a supported container manager.
4. Configure the container manager to use `kata-runtime` as the default OCI runtime.
4. Configure the container manager to use `kata-runtime` as the default OCI runtime. Or, for Kata Containers 1.5.0 or above, configure the
`io.containerd.kata.v2` to be the runtime shim (see [containerd runtime v2 (shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2)
and [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md)).
> **Notes on upgrading**:
> - If you are installing Kata Containers on a system that already has Clear Containers or `runv` installed,