mirror of
https://github.com/kata-containers/kata-containers.git
synced 2025-07-30 23:06:27 +00:00
how-to: a standalone doc for containerd integration
including shimv2 and runtime class Fixes: 322 Signed-off-by: Xu Wang <xu@hyper.sh>
This commit is contained in:
Xu Wang
parent
04498bd4d2
commit
9343ece91e
359
how-to/containerd-kata.md
Normal file
359
how-to/containerd-kata.md
Normal file
@ -0,0 +1,359 @@
|
||||
# How to use Kata Containers and Containerd
|
||||
|
||||
- [Concepts](#concepts)
|
||||
- [Kubernetes RuntimeClass](#kubernetes-runtimeclass)
|
||||
- [Containerd Runtime V2 API (Shim V2 API)](#containerd-runtime-v2-api-(shim-v2-api))
|
||||
- [Install](#install)
|
||||
- [Install Kata Containers](#install-kata-containers)
|
||||
- [Install containerd with cri plugin](#install-containerd-with-cri-plugin)
|
||||
- [Install CNI plugins](#install-cni-plugins)
|
||||
- [Install cri-tools](#install-cri-tools)
|
||||
- [Configuration](#configuration)
|
||||
- [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers)
|
||||
- [Configuration for cri-tools](#configuration-for-cri-tools)
|
||||
- [Run](#run)
|
||||
- [Launch containers with ctr command line](#launch-containers-with-ctr-command-line)
|
||||
- [Launch Pods with crictl command line](#launch-pods-with-crictl-command-line)
|
||||
|
||||
This document covers the installation and configuration of [containerd](https://containerd.io/)
|
||||
and [Kata Containers](https://katacontainers.io). The containerd provides not only the `ctr`
|
||||
command line tool, but also the [CRI](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)
|
||||
interface for [Kubernetes](https://kubernetes.io) and other CRI clients.
|
||||
|
||||
This document is primarily written for Kata Containers v1.5.0-rc2 or above, and containerd v1.2.0 or above.
|
||||
Previous versions are addressed here, but we suggest users upgrade to the newer versions for better support.
|
||||
|
||||
## Concepts
|
||||
|
||||
### Kubernetes RuntimeClass
|
||||
|
||||
[RuntimeClass](https://kubernetes.io/docs/concepts/containers/runtime-class/) is a Kubernetes feature first
|
||||
introduced in Kubernetes 1.12 as alpha. It is the feature for selecting the container runtime configuration to
|
||||
use to run a pod’s containers. This feature is supported in `containerd` since [v1.2.0](https://github.com/containerd/containerd/releases/tag/v1.2.0).
|
||||
|
||||
Before the RuntimeClass was introduced, Kubernetes was not aware of the difference of runtimes on the node. `kubelet`
|
||||
creates Pod sandboxes and containers through CRI implementations, and treats all the Pods equally. However, there
|
||||
are requirements to run trusted Pods (i.e. kubernetes plugin) in a native container like runC, and to run untrusted
|
||||
workloads with isolated sandboxes (i.e. Kata Containers).
|
||||
|
||||
As a result, the CRI implementations extended their semanitics for the requirements:
|
||||
|
||||
- At the beginning, [frakti](https://github.com/kubernetes/frakti) checks the network configuration of a Pod, and
|
||||
treat Pod with `host` network as trusted, while others are treated as untrusted.
|
||||
- The containerd introduced an annotation for untrusted Pods since [v1.0](https://github.com/containerd/cri/blob/v1.0.0-rc.0/docs/config.md):
|
||||
```yaml
|
||||
annotations:
|
||||
io.kubernetes.cri.untrusted-workload: "true"
|
||||
```
|
||||
- Similarly, CRI-O introduced the annotation `io.kubernetes.cri-o.TrustedSandbox` for untrusted Pods.
|
||||
|
||||
To eliminate the complexity of user configuration introduced by the non-standardized annotations and provide
|
||||
extensibility, `RuntimeClass` was introduced. This gives users the ability to affect the runtime behavior
|
||||
through 'RuntimeClass' without the knowledge of the CRI daemons. We suggest that users with multiple runtimes
|
||||
use 'RuntimeClass' instead of the deprecated annotations.
|
||||
|
||||
### Containerd Runtime V2 API (Shim V2 API)
|
||||
|
||||
The [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2)
|
||||
implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata.
|
||||
With `shimv2`, kubernetes can launch Pod and OCI-compatible containers with one shim per Pod. Prior to `shimv2`, `2N+1`
|
||||
shims (i.e. a `containerd-shim` and a `kata-shim` for each container and the Pod sandbox itself) and no standalone `kata-proxy`
|
||||
process were used, even with vsock not available.
|
||||
|
||||
|
||||
|
||||
The shim v2 is introduced in containerd [v1.2.0](https://github.com/containerd/containerd/releases/tag/v1.2.0) and kata `shimv2`
|
||||
is implemented in Kata Containers v1.5.0.
|
||||
|
||||
## Install
|
||||
|
||||
### Install Kata Containers
|
||||
|
||||
Follow the instructions to [install Kata Containers](https://github.com/kata-containers/documentation/blob/master/install/README.md).
|
||||
|
||||
### Install containerd with cri plugin
|
||||
|
||||
> **Note:** `cri` is a native plugin of containerd 1.1 and above. It is built into containerd and enabled by default.
|
||||
> You do not need to install `cri` if you have containerd 1.1 or above. Just check if the `cri` plugin has been disabled
|
||||
> in the containerd configuration file.
|
||||
|
||||
Follow the instructions from the [CRI installation guide](http://github.com/containerd/cri/blob/master/docs/installation.md).
|
||||
|
||||
Then, check if `containerd` is now available:
|
||||
|
||||
```bash
|
||||
$ command -v containerd
|
||||
```
|
||||
|
||||
### Install CNI plugins
|
||||
|
||||
> **Note:** You do not need to install CNI plugins if you do not want to use containerd with Kubernetes.
|
||||
> If you have installed Kubernetes with `kubeadm`, you might have already installed the CNI plugins.
|
||||
|
||||
You can manually install CNI plugins as follows:
|
||||
|
||||
```bash
|
||||
$ go get github.com/containernetworking/plugins
|
||||
$ pushd $GOPATH/src/github.com/containernetworking/plugins
|
||||
$ ./build_linux.sh
|
||||
$ sudo mkdir /opt/cni
|
||||
$ sudo cp -r bin /opt/cni/
|
||||
$ popd
|
||||
```
|
||||
|
||||
### Install cri-tools
|
||||
|
||||
> **Note:** `cri-tools` is a set of tools for CRI used for development and testing. Users who only want
|
||||
> to use containerd with kubernetes can skip the `cri-tools`.
|
||||
|
||||
You can install the `cri-tools` from source code:
|
||||
|
||||
```bash
|
||||
$ go get github.com/kubernetes-incubator/cri-tools
|
||||
$ pushd $GOPATH/src/github.com/kubernetes-incubator/cri-tools
|
||||
$ make
|
||||
$ sudo -E make install
|
||||
$ popd
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Configure containerd to use Kata Containers
|
||||
|
||||
By default, the configuration of containerd is located at `/etc/containerd/config.toml`, and the
|
||||
`cri` plugins are placed in the following section:
|
||||
|
||||
```toml
|
||||
[plugins]
|
||||
[plugins.cri]
|
||||
[plugins.cri.containerd]
|
||||
[plugins.cri.containerd.default_runtime]
|
||||
#runtime_type = "io.containerd.runtime.v1.linux"
|
||||
|
||||
[plugins.cri.cni]
|
||||
# conf_dir is the directory in which the admin places a CNI conf.
|
||||
conf_dir = "/etc/cni/net.d"
|
||||
```
|
||||
|
||||
The following sections outline how to add Kata Containers to the configurations.
|
||||
|
||||
#### Kata Containers as a RuntimeClass
|
||||
|
||||
For
|
||||
- Kata Containers v1.5.0 or above (including 1.5.0-rc)
|
||||
- Containerd v1.2.0 or above
|
||||
- Kubernetes v1.12.0 or above
|
||||
|
||||
The RuntimeClass is suggested.
|
||||
|
||||
The following configuration includes three runtime classes:
|
||||
- `plugins.cri.containerd.runtimes.runc`: the runC, and it is the default runtime.
|
||||
- `plugins.cri.containerd.runtimes.kata`: The function in containerd (reference [the document here](https://github.com/containerd/containerd/tree/master/runtime/v2#binary-naming))
|
||||
where the dot-connected string `io.containerd.kata.v2` is translated to `containerd-shim-kata-v2` (i.e. the
|
||||
binary name of the kata implementation of [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2)).
|
||||
- `plugins.cri.containerd.runtimes.katacli`: the `containerd-shim-runc-v1` calls `kata-runtime`, which is the legacy process.
|
||||
|
||||
```toml
|
||||
[plugins.cri.containerd]
|
||||
no_pivot = false
|
||||
[plugins.cri.containerd.runtimes]
|
||||
[plugins.cri.containerd.runtimes.runc]
|
||||
runtime_type = "io.containerd.runc.v1"
|
||||
[plugins.cri.containerd.runtimes.runc.options]
|
||||
NoPivotRoot = false
|
||||
NoNewKeyring = false
|
||||
ShimCgroup = ""
|
||||
IoUid = 0
|
||||
IoGid = 0
|
||||
BinaryName = "runc"
|
||||
Root = ""
|
||||
CriuPath = ""
|
||||
SystemdCgroup = false
|
||||
[plugins.cri.containerd.runtimes.kata]
|
||||
runtime_type = "io.containerd.kata.v2"
|
||||
[plugins.cri.containerd.runtimes.katacli]
|
||||
runtime_type = "io.containerd.runc.v1"
|
||||
[plugins.cri.containerd.runtimes.katacli.options]
|
||||
NoPivotRoot = false
|
||||
NoNewKeyring = false
|
||||
ShimCgroup = ""
|
||||
IoUid = 0
|
||||
IoGid = 0
|
||||
BinaryName = "/usr/bin/kata-runtime"
|
||||
Root = ""
|
||||
CriuPath = ""
|
||||
SystemdCgroup = false
|
||||
```
|
||||
|
||||
Based on the discussion in [containerd/containerd#2916](https://github.com/containerd/containerd/pull/2916),
|
||||
the containerd configuration will have a field for runtime configuration files (likely to be`config_file`). However,
|
||||
the field is not available yet, and the `shimv2` will search the configuration file in the default paths, same as
|
||||
`kata-runtime` cli): `/etc/kata-containers/configuration.toml`, `/usr/share/defaults/kata-containers/configuration.toml`.
|
||||
We will update this document once the `config_file` field is available.
|
||||
|
||||
If you want to specify a configure file now, the following provides a workaround. The `shimv2` accepts an environment
|
||||
variable, `KATA_CONF_FILE` for the config file path. Then, you can create a shell script with the following:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
KATA_CONF_FILE=/etc/kata-containers/firecracker.toml containerd-shim-kata-v2
|
||||
```
|
||||
|
||||
Name it as `/usr/local/bin/containerd-shim-katafc-v2` and reference it in the configuration of containerd:
|
||||
|
||||
```
|
||||
[plugins.cri.containerd.runtimes.kata-firecracker]
|
||||
runtime_type = "io.containerd.katafc.v2"
|
||||
```
|
||||
|
||||
#### Kata Containers as the runtime for untrusted workload
|
||||
|
||||
For cases without RuntimeClass support, we can use the legacy annotation method to support using Kata Containers
|
||||
for an untrusted workload. With the following configuration, you can run trusted workloads with a runtime such as `runc`
|
||||
and then, run an untrusted workload with Kata Containers:
|
||||
|
||||
```toml
|
||||
[plugins.cri.containerd]
|
||||
# "plugins.cri.containerd.default_runtime" is the runtime to use in containerd.
|
||||
[plugins.cri.containerd.default_runtime]
|
||||
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
|
||||
runtime_type = "io.containerd.runtime.v1.linux"
|
||||
|
||||
# "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it.
|
||||
[plugins.cri.containerd.untrusted_workload_runtime]
|
||||
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
|
||||
runtime_type = "io.containerd.kata.v2"
|
||||
```
|
||||
|
||||
For the earlier versions of Kata Containers and containerd that do not support Runtime V2 (Shim API), you can use the following alternative configuration:
|
||||
|
||||
```toml
|
||||
[plugins.cri.containerd]
|
||||
|
||||
# "plugins.cri.containerd.default_runtime" is the runtime to use in containerd.
|
||||
[plugins.cri.containerd.default_runtime]
|
||||
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
|
||||
runtime_type = "io.containerd.runtime.v1.linux"
|
||||
|
||||
# "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it.
|
||||
[plugins.cri.containerd.untrusted_workload_runtime]
|
||||
# runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
|
||||
runtime_type = "io.containerd.runtime.v1.linux"
|
||||
|
||||
# runtime_engine is the name of the runtime engine used by containerd.
|
||||
runtime_engine = "/usr/bin/kata-runtime"
|
||||
```
|
||||
|
||||
You can find more information on the [Containerd config documentation](https://github.com/containerd/cri/blob/master/docs/config.md)
|
||||
|
||||
|
||||
#### Kata Containers as the default runtime
|
||||
|
||||
If you want to set Kata Containers as the only runtime in the deployment, you can simply configure as follows:
|
||||
|
||||
```toml
|
||||
[plugins.cri.containerd]
|
||||
[plugins.cri.containerd.default_runtime]
|
||||
runtime_type = "io.containerd.kata.v2"
|
||||
```
|
||||
|
||||
Alternatively, for the earlier versions of Kata Containers and containerd that do not support Runtime V2 (Shim API), you can use the following alternative configuration:
|
||||
|
||||
```toml
|
||||
[plugins.cri.containerd]
|
||||
[plugins.cri.containerd.default_runtime]
|
||||
runtime_type = "io.containerd.runtime.v1.linux"
|
||||
runtime_engine = "/usr/bin/kata-runtime"
|
||||
```
|
||||
|
||||
### Configuration for cri-tools
|
||||
|
||||
> **Note:** If you skipped the [Install cri-tools](#install-cri-tools) section, you can skip this section too.
|
||||
|
||||
First, add the CNI configuration in the containerd configuration.
|
||||
|
||||
The following is the configuration if you installed CNI as the *[Install CNI plugins](#install-cni-plugins)* section outlined.
|
||||
|
||||
Put the CNI configuration as `/etc/cni/net.d/10-mynet.conf`:
|
||||
|
||||
```json
|
||||
{
|
||||
"cniVersion": "0.2.0",
|
||||
"name": "mynet",
|
||||
"type": "bridge",
|
||||
"bridge": "cni0",
|
||||
"isGateway": true,
|
||||
"ipMasq": true,
|
||||
"ipam": {
|
||||
"type": "host-local",
|
||||
"subnet": "172.19.0.0/24",
|
||||
"routes": [
|
||||
{ "dst": "0.0.0.0/0" }
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Next, reference the configuration directory through containerd `config.toml`:
|
||||
|
||||
```toml
|
||||
[plugins.cri.cni]
|
||||
# conf_dir is the directory in which the admin places a CNI conf.
|
||||
conf_dir = "/etc/cni/net.d"
|
||||
```
|
||||
|
||||
The configuration file of `crictl` command line tool in `cri-tools` locates at `/etc/crictl.yaml`:
|
||||
|
||||
```yaml
|
||||
runtime-endpoint: unix:///var/run/containerd/containerd.sock
|
||||
image-endpoint: unix:///var/run/containerd/containerd.sock
|
||||
timeout: 10
|
||||
debug: true
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
### Launch containers with ctr command line
|
||||
|
||||
To run a container with Kata Containers through the containerd command line, you can run the following:
|
||||
|
||||
```bash
|
||||
$ sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh
|
||||
```
|
||||
|
||||
This launchs a busybox container named `hello`, and it will be removed by `--rm` after it quits.
|
||||
|
||||
### Launch Pods with crictl command line
|
||||
|
||||
With the `crictl` command line of `cri-tools`, you can specify runtime class with `-r` or `--runtime` flag.
|
||||
Use the following to launch Pod with `kata` runtime class with the pod in [the example](https://github.com/kubernetes-sigs/cri-tools/tree/master/docs/examples)
|
||||
of `cri-tools`:
|
||||
|
||||
```bash
|
||||
$ sudo crictl runp -r kata podsandbox-config.yaml
|
||||
36e23521e8f89fabd9044924c9aeb34890c60e85e1748e8daca7e2e673f8653e
|
||||
```
|
||||
|
||||
You can add container to the launched Pod with the following:
|
||||
|
||||
```bash
|
||||
$ sudo crictl create 36e23521e8f89 container-config.yaml podsandbox-config.yaml
|
||||
1aab7585530e62c446734f12f6899f095ce53422dafcf5a80055ba11b95f2da7
|
||||
```
|
||||
|
||||
Now, start it with the following:
|
||||
|
||||
```bash
|
||||
$ sudo crictl start 1aab7585530e6
|
||||
1aab7585530e6
|
||||
```
|
||||
|
||||
In Kubernetes, you need to create a RuntimeClass resource and add the RuntimeClass field in the Pod Spec
|
||||
(see this [document](https://kubernetes.io/docs/concepts/containers/runtime-class/) for more information).
|
||||
|
||||
If RuntimeClass is not supported, you can use the following annotation in a Kubernetes pod to identify as an untrusted workload:
|
||||
|
||||
```yaml
|
||||
annotations:
|
||||
io.kubernetes.cri.untrusted-workload: "true"
|
||||
```
|
||||
Loading…
Reference in New Issue
Block a user