diff --git a/Developer-Guide.md b/Developer-Guide.md index e4fa3ce6a5..6cce20547a 100644 --- a/Developer-Guide.md +++ b/Developer-Guide.md @@ -31,13 +31,6 @@ * [Update the Docker systemd unit file](#update-the-docker-systemd-unit-file) * [Create a container using Kata](#create-a-container-using-kata) * [Run Kata Containers with Kubernetes](#run-kata-containers-with-kubernetes) - * [Install a CRI implementation](#install-a-cri-implementation) - * [CRI-O](#cri-o) - * [containerd with CRI plugin](#containerd-with-cri-plugin) - * [Install Kubernetes](#install-kubernetes) - * [Configure for CRI-O](#configure-for-cri-o) - * [Configure for containerd](#configure-for-containerd) - * [Run a Kubernetes pod with Kata Containers](#run-a-kubernetes-pod-with-kata-containers) * [Troubleshoot Kata Containers](#troubleshoot-kata-containers) * [Appendices](#appendices) * [Checking Docker default runtime](#checking-docker-default-runtime) @@ -392,166 +385,7 @@ $ sudo docker run -ti --runtime kata-runtime busybox sh ``` # Run Kata Containers with Kubernetes - -Now that Kata Containers is installed on your system, you need some -extra components to make this work with Kubernetes. - -## Install a CRI implementation - -Kata Containers runtime is an OCI compatible runtime and cannot directly -interact with the CRI API level. For this reason we rely on a CRI -implementation to translate CRI into OCI. There are two supported ways -called [CRI-O](https://github.com/kubernetes-incubator/cri-o) and -[CRI-containerd](https://github.com/containerd/cri). It is up to you to -choose the one that you want, but you have to pick one. After choosing -either CRI-O or CRI-containerd, you must make the appropriate changes -to ensure it relies on the Kata Containers runtime. - -As of Kata Containers 1.5, using `shimv2` with containerd 1.2.0 or above is the preferred -way to run Kata Containers with Kubernetes ([see the howto](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)). -The CRI-O will catch up soon. - -### CRI-O - -If you select CRI-O, follow the "CRI-O Tutorial" instructions -[here](https://github.com/kubernetes-incubator/cri-o/blob/master/tutorial.md) -to properly install it. - -Once you have installed CRI-O, you need to modify the CRI-O configuration -with information about different container runtimes. By default, we choose -`runc`, but in this case we also specify Kata Containers runtime to run -__untrusted__ workloads. In other words, this defines an alternative runtime -to be used when the workload cannot be trusted and a higher level of security -is required. An additional flag can be used to let CRI-O know if a workload -should be considered _trusted_ or _untrusted_ by default. -For further details, see the documentation -[here](https://github.com/kata-containers/documentation/blob/master/design/architecture.md#mixing-vm-based-and-namespace-based-runtimes). - -Additionally, we need CRI-O to perform the network namespace management. -Otherwise, when the VM starts the network will not be available. - -The following is an example of how to modify the `/etc/crio/crio.conf` file -in order to apply the previous explanations, and therefore get Kata Containers -runtime to invoke by CRI-O. - -```toml -# The "crio.runtime" table contains settings pertaining to the OCI -# runtime used and options for how to set up and manage the OCI runtime. -[crio.runtime] -manage_network_ns_lifecycle = true - -# runtime is the OCI compatible runtime used for trusted container workloads. -# This is a mandatory setting as this runtime will be the default one -# and will also be used for untrusted container workloads if -# runtime_untrusted_workload is not set. -runtime = "/usr/bin/runc" - -# runtime_untrusted_workload is the OCI compatible runtime used for untrusted -# container workloads. This is an optional setting, except if -# default_container_trust is set to "untrusted". -runtime_untrusted_workload = "/usr/bin/kata-runtime" - -# default_workload_trust is the default level of trust crio puts in container -# workloads. It can either be "trusted" or "untrusted", and the default -# is "trusted". -# Containers can be run through different container runtimes, depending on -# the trust hints we receive from kubelet: -# - If kubelet tags a container workload as untrusted, crio will try first to -# run it through the untrusted container workload runtime. If it is not set, -# crio will use the trusted runtime. -# - If kubelet does not provide any information about the container workload trust -# level, the selected runtime will depend on the default_container_trust setting. -# If it is set to "untrusted", then all containers except for the host privileged -# ones, will be run by the runtime_untrusted_workload runtime. Host privileged -# containers are by definition trusted and will always use the trusted container -# runtime. If default_container_trust is set to "trusted", crio will use the trusted -# container runtime for all containers. -default_workload_trust = "untrusted" - -``` - -Restart CRI-O to take changes into account -``` -$ sudo systemctl restart crio -``` - -### containerd with CRI plugin - -If you select containerd with `cri` plugin, follow the "Getting Started for Developers" -instructions [here](https://github.com/containerd/cri#getting-started-for-developers) -to properly install it. - -To customize containerd to select Kata Containers runtime, follow our -"Configure containerd to use Kata Containers" internal documentation -[here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers). - -## Install Kubernetes - -Depending on what your needs are and what you expect to do with Kubernetes, -please refer to the following -[documentation](https://kubernetes.io/docs/setup/) to install it correctly. - -Kubernetes talks with CRI implementations through a `container-runtime-endpoint`, -also called CRI socket. This socket path is different depending on which CRI -implementation you chose, and the Kubelet service has to be updated accordingly. - -### Configure for CRI-O - -`/etc/systemd/system/kubelet.service.d/0-crio.conf` -``` -[Service] -Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock" -``` - -### Configure for containerd - -`/etc/systemd/system/kubelet.service.d/0-cri-containerd.conf` -``` -[Service] -Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" -``` -For more information about containerd see the "Configure Kubelet to use containerd" -documentation [here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-kubelet-to-use-containerd). - -## Run a Kubernetes pod with Kata Containers - -After you update your Kubelet service based on the CRI implementation you -are using, reload and restart Kubelet. Then, start your cluster: -```bash -$ sudo systemctl daemon-reload -$ sudo systemctl restart kubelet - -# If using CRI-O -$ sudo kubeadm init --skip-preflight-checks --cri-socket /var/run/crio/crio.sock --pod-network-cidr=10.244.0.0/16 - -# If using CRI-containerd -$ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 - -$ export KUBECONFIG=/etc/kubernetes/admin.conf -``` - -You can force Kubelet to use Kata Containers by adding some `untrusted` -annotation to your pod configuration. In our case, this ensures Kata -Containers is the selected runtime to run the described workload. - -`nginx-untrusted.yaml` -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: nginx-untrusted - annotations: - io.kubernetes.cri.untrusted-workload: "true" -spec: - containers: - - name: nginx - image: nginx -``` - -Next, you run your pod: -``` -$ sudo -E kubectl apply -f nginx-untrusted.yaml -``` +Refer to to the [Run Kata Containers with Kubernetes](how-to/run-kata-with-k8s.md) how-to guide. # Troubleshoot Kata Containers @@ -606,7 +440,7 @@ the following steps, which assume the use of a rootfs image. > **Note:** The following debug console instructions assume a systemd-based guest > O/S image. This means you must create a rootfs for a distro that supports systemd. > Currently, all distros supported by [osbuilder](https://github.com/kata-containers/osbuilder) support systemd -> except for Alpine Linux. +> except for Alpine Linux. > > Look for `INIT_PROCESS=systemd` in the `config.sh` osbuilder rootfs config file > to verify an osbuilder distro supports systemd for the distro you want to build rootfs for. diff --git a/how-to/README.md b/how-to/README.md index 5ed416a906..72e11b4e12 100644 --- a/how-to/README.md +++ b/how-to/README.md @@ -1,11 +1,22 @@ -# Howto guides +# Howto Guides +* [Howto Guides](#howto-guides) + * [Kubernetes Integration](#kubernetes-integration) + * [Hypervisors Integration](#hypervisors-integration) + * [Advanced Topics](#advanced-topics) + +## Kubernetes Integration +- [Run Kata Containers with Kubernetes](run-kata-with-k8s.md) - [How to use Kata Containers and Containerd](containerd-kata.md) - [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](how-to-use-k8s-with-cri-containerd-and-kata.md) -- [How to use Kata Containers with virtio-fs](how-to-use-virtio-fs-with-kata.md) - [Kata Containers and service mesh for Kubernetes](service-mesh.md) + +## Hypervisors Integration - [Kata Containers with Firecracker](https://github.com/kata-containers/documentation/wiki/Initial-release-of-Kata-Containers-with-Firecracker-support) - [Kata Containers with NEMU](how-to-use-kata-containers-with-nemu.md) + +## Advanced Topics +- [How to use Kata Containers with virtio-fs](how-to-use-virtio-fs-with-kata.md) - [Setting Sysctls with Kata](how-to-use-sysctls-with-kata.md) - [What Is VMCache and How To Enable It](what-is-vm-cache-and-how-do-I-use-it.md) - [What Is VM Templating and How To Enable It](what-is-vm-templating-and-how-do-I-use-it.md) diff --git a/how-to/run-kata-with-k8s.md b/how-to/run-kata-with-k8s.md new file mode 100644 index 0000000000..d4e43cd490 --- /dev/null +++ b/how-to/run-kata-with-k8s.md @@ -0,0 +1,197 @@ +# Run Kata Containers with Kubernetes + +* [Run Kata Containers with Kubernetes](#run-kata-containers-with-kubernetes) + * [Prerequisites](#prerequisites) + * [Install a CRI implementation](#install-a-cri-implementation) + * [CRI-O](#cri-o) + * [Kubernetes Runtime Class (CRI-O v1.12 )](#kubernetes-runtime-class-cri-o-v112) + * [Untrusted annotation (until CRI-O v1.12)](#untrusted-annotation-until-cri-o-v112) + * [Network namespace management](#network-namespace-management) + * [containerd with CRI plugin](#containerd-with-cri-plugin) + * [Install Kubernetes](#install-kubernetes) + * [Configure for CRI-O](#configure-for-cri-o) + * [Configure for containerd](#configure-for-containerd) + * [Run a Kubernetes pod with Kata Containers](#run-a-kubernetes-pod-with-kata-containers) + +## Prerequisites +This guide requires Kata Containers available on your system, install-able by following [this guide](https://github.com/kata-containers/documentation/blob/master/install/README.md). + +## Install a CRI implementation + +Kubernetes CRI (Container Runtime Interface) implementations allow using any +OCI-compatible runtime with Kubernetes, such as the Kata Containers runtime. + +Kata Containers support both the [CRI-O](https://github.com/kubernetes-incubator/cri-o) and +[CRI-containerd](https://github.com/containerd/cri) CRI implementations. + +After choosing one CRI implementation, you must make the appropriate configuration +to ensure it integrates with Kata Containers. + +Kata Containers 1.5 introduced the `shimv2` for containerd 1.2.0, reducing the components +required to spawn pods and containers, and this is the preferred way to run Kata Containers with Kubernetes ([as documented here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers)). + +An equivalent shim implementation for CRI-O is planned. + +### CRI-O +For CRI-O installation instructions, refer to the [CRI-O Tutorial](https://github.com/kubernetes-incubator/cri-o/blob/master/tutorial.md) page. + +The following sections show how to set up the CRI-O configuration file (default path: `/etc/crio/crio.conf`) for Kata. + +Unless otherwise stated, all the following settings are specific to the `crio.runtime` table: +```toml +# The "crio.runtime" table contains settings pertaining to the OCI +# runtime used and options for how to set up and manage the OCI runtime. +[crio.runtime] +``` +A comprehensive documentation of the configuration file can be found [here](https://github.com/cri-o/cri-o/blob/master/docs/crio.conf.5.md). + +> **Note**: After any change to this file, the CRI-O daemon have to be restarted with: +>```` +>$ sudo systemctl restart crio +>```` + +#### Kubernetes Runtime Class (CRI-O v1.12+) +The [Kubernetes Runtime Class](https://kubernetes.io/docs/concepts/containers/runtime-class/) +is the preferred way of specifying the container runtime configuration to run a Pod's containers. +To use this feature, Kata must added as a runtime handler with: + +```toml +[crio.runtime.runtimes.kata-runtime] + runtime_path = "/usr/bin/kata-runtime" + runtime_type = "oci" +``` + +You can also add multiple entries to specify alternatives hypervisors, e.g.: +```toml +[crio.runtime.runtimes.kata-qemu] + runtime_path = "/usr/bin/kata-runtime" + runtime_type = "oci" + +[crio.runtime.runtimes.kata-fc] + runtime_path = "/usr/bin/kata-runtime" + runtime_type = "oci" +``` + +#### Untrusted annotation (until CRI-O v1.12) +The untrusted annotation is used to specify a runtime for __untrusted__ workloads, i.e. +a runtime to be used when the workload cannot be trusted and a higher level of security +is required. An additional flag can be used to let CRI-O know if a workload +should be considered _trusted_ or _untrusted_ by default. +For further details, see the documentation +[here](https://github.com/kata-containers/documentation/blob/master/design/architecture.md#mixing-vm-based-and-namespace-based-runtimes). + +```toml +# runtime is the OCI compatible runtime used for trusted container workloads. +# This is a mandatory setting as this runtime will be the default one +# and will also be used for untrusted container workloads if +# runtime_untrusted_workload is not set. +runtime = "/usr/bin/runc" + +# runtime_untrusted_workload is the OCI compatible runtime used for untrusted +# container workloads. This is an optional setting, except if +# default_container_trust is set to "untrusted". +runtime_untrusted_workload = "/usr/bin/kata-runtime" + +# default_workload_trust is the default level of trust crio puts in container +# workloads. It can either be "trusted" or "untrusted", and the default +# is "trusted". +# Containers can be run through different container runtimes, depending on +# the trust hints we receive from kubelet: +# - If kubelet tags a container workload as untrusted, crio will try first to +# run it through the untrusted container workload runtime. If it is not set, +# crio will use the trusted runtime. +# - If kubelet does not provide any information about the container workload trust +# level, the selected runtime will depend on the default_container_trust setting. +# If it is set to "untrusted", then all containers except for the host privileged +# ones, will be run by the runtime_untrusted_workload runtime. Host privileged +# containers are by definition trusted and will always use the trusted container +# runtime. If default_container_trust is set to "trusted", crio will use the trusted +# container runtime for all containers. +default_workload_trust = "untrusted" +``` + +#### Network namespace management +To enable networking for the workloads run by Kata, CRI-O needs to be configured to +manage network namespaces, by setting the following key to `true`: +```toml +manage_network_ns_lifecycle = true +``` + +### containerd with CRI plugin + +If you select containerd with `cri` plugin, follow the "Getting Started for Developers" +instructions [here](https://github.com/containerd/cri#getting-started-for-developers) +to properly install it. + +To customize containerd to select Kata Containers runtime, follow our +"Configure containerd to use Kata Containers" internal documentation +[here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-containerd-to-use-kata-containers). + +## Install Kubernetes + +Depending on what your needs are and what you expect to do with Kubernetes, +please refer to the following +[documentation](https://kubernetes.io/docs/setup/) to install it correctly. + +Kubernetes talks with CRI implementations through a `container-runtime-endpoint`, +also called CRI socket. This socket path is different depending on which CRI +implementation you chose, and the Kubelet service has to be updated accordingly. + +### Configure for CRI-O + +`/etc/systemd/system/kubelet.service.d/0-crio.conf` +``` +[Service] +Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock" +``` + +### Configure for containerd + +`/etc/systemd/system/kubelet.service.d/0-cri-containerd.conf` +``` +[Service] +Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" +``` +For more information about containerd see the "Configure Kubelet to use containerd" +documentation [here](https://github.com/kata-containers/documentation/blob/master/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md#configure-kubelet-to-use-containerd). + +## Run a Kubernetes pod with Kata Containers + +After you update your Kubelet service based on the CRI implementation you +are using, reload and restart Kubelet. Then, start your cluster: +```bash +$ sudo systemctl daemon-reload +$ sudo systemctl restart kubelet + +# If using CRI-O +$ sudo kubeadm init --skip-preflight-checks --cri-socket /var/run/crio/crio.sock --pod-network-cidr=10.244.0.0/16 + +# If using CRI-containerd +$ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 + +$ export KUBECONFIG=/etc/kubernetes/admin.conf +``` + +You can force Kubelet to use Kata Containers by adding some `untrusted` +annotation to your pod configuration. In our case, this ensures Kata +Containers is the selected runtime to run the described workload. + +`nginx-untrusted.yaml` +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: nginx-untrusted + annotations: + io.kubernetes.cri.untrusted-workload: "true" +spec: + containers: + - name: nginx + image: nginx +``` + +Next, you run your pod: +``` +$ sudo -E kubectl apply -f nginx-untrusted.yaml +``` +