diff --git a/Developer-Guide.md b/Developer-Guide.md index 1d79b0968..d8ec852cd 100644 --- a/Developer-Guide.md +++ b/Developer-Guide.md @@ -26,9 +26,9 @@ * [Install guest kernel images](#install-guest-kernel-images) * [Install a hypervisor](#install-a-hypervisor) * [Build a custom QEMU](#build-a-custom-qemu) - * [Build a custom QEMU for aarch64/arm64 - REQUIRED](#Build-a-custom-qemu-for-aarch64/arm64---required) + * [Build a custom QEMU for aarch64/arm64 - REQUIRED](#build-a-custom-qemu-for-aarch64arm64---required) * [Run Kata Containers with Docker](#run-kata-containers-with-docker) - * [Update Docker configuration](#update-docker-configuration) + * [Update the Docker systemd unit file](#update-the-docker-systemd-unit-file) * [Create a container using Kata](#create-a-container-using-kata) * [Run Kata Containers with Kubernetes](#run-kata-containers-with-kubernetes) * [Install a CRI implementation](#install-a-cri-implementation) @@ -348,8 +348,8 @@ $ curl -LOk ${kernel_url} $ tar -xf ${kernel_tar_file} $ mv .config "linux-${kernel_version}" $ pushd "linux-${kernel_version}" -$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/patches/0001-NO-UPSTREAM-9P-always-use-cached-inode-to-fill-in-v9.patch | patch -p1 -$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/patches/0002-Compile-in-evged-always.patch | patch -p1 +$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/patches/4.19.x/0001-NO-UPSTREAM-9P-always-use-cached-inode-to-fill-in-v9.patch | patch -p1 +$ curl -L https://raw.githubusercontent.com/kata-containers/packaging/master/kernel/patches/4.19.x/0002-Compile-in-evged-always.patch | patch -p1 $ make ARCH=${kernel_dir} -j$(nproc) $ kata_kernel_dir="/usr/share/kata-containers" $ kata_vmlinuz="${kata_kernel_dir}/kata-vmlinuz-${kernel_version}.container" diff --git a/Limitations.md b/Limitations.md index e98692335..46439fea4 100644 --- a/Limitations.md +++ b/Limitations.md @@ -8,7 +8,6 @@ * [events command](#events-command) * [update command](#update-command) * [Networking](#networking) - * [Adding networks dynamically](#adding-networks-dynamically) * [Docker swarm support](#docker-swarm-support) * [Resource management](#resource-management) * [docker run and shared memory](#docker-run-and-shared-memory) @@ -18,12 +17,12 @@ * [Architectural limitations](#architectural-limitations) * [Networking limitations](#networking-limitations) * [Support for joining an existing VM network](#support-for-joining-an-existing-vm-network) - * [docker --net=host](#docker---net=host) + * [docker --net=host](#docker---nethost) * [docker run --link](#docker-run---link) * [Host resource sharing](#host-resource-sharing) * [docker run --privileged](#docker-run---privileged) * [Miscellaneous](#miscellaneous) - * [Docker --security-opt partially supported](#docker---security-opt-option-partially-supported) + * [Docker --security-opt option partially supported](#docker---security-opt-option-partially-supported) * [Appendices](#appendices) * [The constraints challenge](#the-constraints-challenge) diff --git a/design/architecture.md b/design/architecture.md index f7d07b490..489bdae07 100644 --- a/design/architecture.md +++ b/design/architecture.md @@ -8,7 +8,7 @@ * [Agent](#agent) * [Runtime](#runtime) * [Configuration](#configuration) - * [Significant commands](#significant-commands) + * [Significant OCI commands](#significant-oci-commands) * [create](#create) * [start](#start) * [exec](#exec) @@ -112,8 +112,8 @@ to create virtual machines where containers will run: Depending on the host architecture, Kata Containers supports various machine types, for example `pc` and `q35` on x86 systems, `virt` on ARM systems and `pseries` on IBM Power systems. Kata Containers' -default machine type is `pc`. The default machine type and its [`Machine accelerators`](#Machine-accelerators) can -be changed by editing the runtime [`configuration`](#Configuration) file. +default machine type is `pc`. The default machine type and its [`Machine accelerators`](#machine-accelerators) can +be changed by editing the runtime [`configuration`](#configuration) file. The following QEMU features are used in Kata Containers to manage resource constraints, improve boot time and reduce memory footprint: @@ -249,9 +249,11 @@ The configuration file is also used to enable runtime [debug output](https://git Here we describe how `kata-runtime` handles the most important OCI commands. -#### [`create`](https://github.com/kata-containers/runtime/blob/master/cli/create.go) +#### `create` -When handling the OCI `create` command, `kata-runtime` goes through the following steps: +When handling the OCI +[`create`](https://github.com/kata-containers/runtime/blob/master/cli/create.go) +command, `kata-runtime` goes through the following steps: 1. Create the network namespace where we will spawn VM and shims processes. 2. Call into the pre-start hooks. One of them should be responsible for creating @@ -277,11 +279,9 @@ on the host system by the `kata-shim` process. ![kata-oci-create](arch-images/kata-oci-create.svg) +#### `start` - -#### [`start`](https://github.com/kata-containers/runtime/blob/master/cli/start.go) - -With traditional containers, `start` launches a container process in its own set of namespaces. With Kata Containers, the main task of `kata-runtime` is to ask [`kata-agent`](#agent) to start the container workload inside the virtual machine. `kata-runtime` will run through the following steps: +With traditional containers, [`start`](https://github.com/kata-containers/runtime/blob/master/cli/start.go) launches a container process in its own set of namespaces. With Kata Containers, the main task of `kata-runtime` is to ask [`kata-agent`](#agent) to start the container workload inside the virtual machine. `kata-runtime` will run through the following steps: 1. Communicate with `kata-agent` (through the proxy) to start the container workload inside the VM. If, for example, the command to execute inside of the container is `top`, @@ -292,9 +292,9 @@ With traditional containers, `start` launches a container process in its own set ![kata-oci-start](arch-images/kata-oci-start.svg) -#### [`exec`](https://github.com/kata-containers/runtime/blob/master/cli/exec.go) +#### `exec` -OCI `exec` allows you to run an additional command within an already running +OCI [`exec`](https://github.com/kata-containers/runtime/blob/master/cli/exec.go) allows you to run an additional command within an already running container. In Kata Containers, this is handled as follows: 1. A request is sent to the `kata agent` (through the proxy) to start a new process @@ -307,9 +307,9 @@ Now the `exec`'ed process is running within the VM, sharing `uts`, `pid`, `mnt` ![kata-oci-exec](arch-images/kata-oci-exec.svg) -#### [`kill`](https://github.com/kata-containers/runtime/blob/master/cli/kill.go) +#### `kill` -When sending the OCI `kill` command, the container runtime should send a +When sending the OCI [`kill`](https://github.com/kata-containers/runtime/blob/master/cli/kill.go) command, the container runtime should send a [UNIX signal](https://en.wikipedia.org/wiki/Unix_signal) to the container process. A `kill` sending a termination signal such as `SIGKILL` or `SIGTERM` is expected to terminate the container process. In the context of a traditional container, @@ -334,9 +334,9 @@ the container and the VM associated with it. If `kill` was invoked with a non-termination signal, this simply signals the container process. Otherwise, everything has been torn down, and the VM has been removed. -#### [`delete`](https://github.com/kata-containers/runtime/blob/master/cli/delete.go) +#### `delete` -`delete` removes all internal resources related to a container. A running container +[`delete`](https://github.com/kata-containers/runtime/blob/master/cli/delete.go) removes all internal resources related to a container. A running container cannot be deleted unless the OCI runtime is explicitly being asked to, by using `--force` flag. @@ -349,9 +349,10 @@ would go through for a termination signal. After this process, or if the sandbox At this point, everything related to the container should have been removed from the host system, and no related process should be running. -#### [`state`](https://github.com/kata-containers/runtime/blob/master/cli/state.go) +#### `state` -`state` returns the status of the container. For `kata-runtime`, this means being +[`state`](https://github.com/kata-containers/runtime/blob/master/cli/state.go) +returns the status of the container. For `kata-runtime`, this means being able to detect if the container is still running by looking at the state of `kata-shim` process representing this container process. @@ -582,8 +583,8 @@ The CRI-O will catch up soon ([kubernetes-sigs/cri-o#2024](https://github.com/ku Refer to the following how-to guides: -- [How to use Kata Containers and Containerd](how-to/containerd-kata.md) -- [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](how-to/how-to-use-k8s-with-cri-containerd-and-kata.md) +- [How to use Kata Containers and Containerd](/how-to/containerd-kata.md) +- [How to use Kata Containers and CRI (containerd plugin) with Kubernetes](/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md) ### CRI-O @@ -630,7 +631,7 @@ with a Kubernetes pod: #### Mixing VM based and namespace based runtimes -> **Note:** Since Kubernetes 1.12, the `[Kubernetes RuntimeClass](how-to/containerd-kata.md#kubernetes-runtimeclass)` +> **Note:** Since Kubernetes 1.12, the [`Kubernetes RuntimeClass`](/how-to/containerd-kata.md#kubernetes-runtimeclass) > has been supported and the user can specify runtime without the non-standardized annotations. One interesting evolution of the CRI-O support for `kata-runtime` is the ability diff --git a/how-to/containerd-kata.md b/how-to/containerd-kata.md index 7ec435de6..e3d6df06f 100644 --- a/how-to/containerd-kata.md +++ b/how-to/containerd-kata.md @@ -1,19 +1,22 @@ # How to use Kata Containers and Containerd - [Concepts](#concepts) - - [Kubernetes RuntimeClass](#kubernetes-runtimeclass) - - [Containerd Runtime V2 API (Shim V2 API)](#containerd-runtime-v2-api-(shim-v2-api)) + - [Kubernetes RuntimeClass](#kubernetes-runtimeclass) + - [Containerd Runtime V2 API: Shim V2 API](#containerd-runtime-v2-api-shim-v2-api) - [Install](#install) - - [Install Kata Containers](#install-kata-containers) - - [Install containerd with cri plugin](#install-containerd-with-cri-plugin) - - [Install CNI plugins](#install-cni-plugins) - - [Install cri-tools](#install-cri-tools) + - [Install Kata Containers](#install-kata-containers) + - [Install containerd with cri plugin](#install-containerd-with-cri-plugin) + - [Install CNI plugins](#install-cni-plugins) + - [Install cri-tools](#install-cri-tools) - [Configuration](#configuration) - - [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers) - - [Configuration for cri-tools](#configuration-for-cri-tools) + - [Configure containerd to use Kata Containers](#configure-containerd-to-use-kata-containers) + - [Kata Containers as a RuntimeClass](#kata-containers-as-a-runtimeclass) + - [Kata Containers as the runtime for untrusted workload](#kata-containers-as-the-runtime-for-untrusted-workload) + - [Kata Containers as the default runtime](#kata-containers-as-the-default-runtime) + - [Configuration for cri-tools](#configuration-for-cri-tools) - [Run](#run) - - [Launch containers with ctr command line](#launch-containers-with-ctr-command-line) - - [Launch Pods with crictl command line](#launch-pods-with-crictl-command-line) + - [Launch containers with ctr command line](#launch-containers-with-ctr-command-line) + - [Launch Pods with crictl command line](#launch-pods-with-crictl-command-line) This document covers the installation and configuration of [containerd](https://containerd.io/) and [Kata Containers](https://katacontainers.io). The containerd provides not only the `ctr` @@ -52,7 +55,7 @@ extensibility, `RuntimeClass` was introduced. This gives users the ability to af through 'RuntimeClass' without the knowledge of the CRI daemons. We suggest that users with multiple runtimes use 'RuntimeClass' instead of the deprecated annotations. -### Containerd Runtime V2 API (Shim V2 API) +### Containerd Runtime V2 API: Shim V2 API The [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](https://github.com/kata-containers/runtime/tree/master/containerd-shim-v2) implements the [Containerd Runtime V2 (Shim API)](https://github.com/containerd/containerd/tree/master/runtime/v2) for Kata. diff --git a/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md index 9f60d57c5..7ebcf46f9 100644 --- a/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md +++ b/how-to/how-to-use-k8s-with-cri-containerd-and-kata.md @@ -9,7 +9,7 @@ * [Start Kubernetes](#start-kubernetes) * [Install a Pod Network](#install-a-pod-network) * [Allow pods to run in the master node](#allow-pods-to-run-in-the-master-node) -* [Create an unstrusted pod using Kata Containers](#create-an-unstrusted-pod-using-kata-containers) +* [Create an untrusted pod using Kata Containers](#create-an-untrusted-pod-using-kata-containers) * [Delete created pod](#delete-created-pod) This document describes how to set up a single-machine Kubernetes (k8s) cluster. @@ -18,7 +18,7 @@ The Kubernetes cluster will use the [CRI containerd plugin](https://github.com/containerd/cri) and [Kata Containers](https://katacontainers.io) to launch untrusted workloads. -For Kata Containers 1.5.0-rc2 and above, we will use [`containerd-shim-kata-v2` (short as `shimv2` in this documentation)](../design/shimv2.md) +For Kata Containers 1.5.0-rc2 and above, we will use `containerd-shim-kata-v2` (short as `shimv2` in this documentation) to launch Kata Containers. For the previous version of Kata Containers, the Pods are launched with `kata-runtime`. ## Requirements @@ -159,6 +159,7 @@ A pod network plugin is needed to allow pods to communicate with each other. ``` - Check the pod network is running + ```bash $ sudo -E kubectl get pods --all-namespaces | grep dns | grep Running && echo "OK" || ( echo "FAIL" && false ) ``` diff --git a/how-to/how-to-use-kata-containers-with-nemu.md b/how-to/how-to-use-kata-containers-with-nemu.md index 4a3450e20..dc3568525 100644 --- a/how-to/how-to-use-kata-containers-with-nemu.md +++ b/how-to/how-to-use-kata-containers-with-nemu.md @@ -5,7 +5,7 @@ * [Pre-requisites](#pre-requisites) * [NEMU](#nemu) * [Download and build](#download-and-build) - * [x86_64](#x86-64) + * [x86_64](#x86_64) * [aarch64](#aarch64) * [Configure Kata Containers](#configure-kata-containers) diff --git a/how-to/what-is-vm-cache-and-how-do-I-use-it.md b/how-to/what-is-vm-cache-and-how-do-I-use-it.md index cb2cd278f..971c03e2d 100644 --- a/how-to/what-is-vm-cache-and-how-do-I-use-it.md +++ b/how-to/what-is-vm-cache-and-how-do-I-use-it.md @@ -1,11 +1,12 @@ # What Is VMCache and How To Enable It -* [What is VMCache](#what-is-vmcache-and-how-to-enable-it) +* [What is VMCache](#what-is-vmcache) * [How is this different to VM templating](#how-is-this-different-to-vm-templating) * [How to enable VMCache](#how-to-enable-vmcache) * [Limitations](#limitations) ### What is VMCache + VMCache is a new function that creates VMs as caches before using it. It helps speed up new container creation. The function consists of a server and some clients communicating @@ -18,12 +19,14 @@ VM and convert it back to a VM. If VMCache function is enabled, kata-runtime will request VM from factory grpccache when it creates a new sandbox. -### How is this different from VM templating +### How is this different to VM templating + Both [VM templating](https://github.com/kata-containers/documentation/blob/master/how-to/what-is-vm-templating-and-how-do-I-use-it.md) and VMCache help speed up new container creation. When VM templating enabled, new VMs are created by cloning from a pre-created template VM, and they will share the same initramfs, kernel and agent memory in readonly mode. So it saves a lot of memory if there are many Kata Containers running on the same host. VMCache is not vulnerable to [share memory CVE](https://github.com/kata-containers/documentation/blob/master/how-to/what-is-vm-templating-and-how-do-I-use-it.md#what-are-the-cons) because each VM doesn't share the memory. ### How to enable VMCache + VMCache can be enabled by changing your Kata Containers config file (`/usr/share/defaults/kata-containers/configuration.toml`, overridden by `/etc/kata-containers/configuration.toml` if provided) such that: * `vm_cache_number` specifies the number of caches of VMCache: diff --git a/install/installing-with-kata-manager.md b/install/installing-with-kata-manager.md index 265284801..5f51e2687 100644 --- a/install/installing-with-kata-manager.md +++ b/install/installing-with-kata-manager.md @@ -1,6 +1,6 @@ # Installing with kata-manager -* [Introduction](#Introduction) +* [Introduction](#introduction) * [Full Installation](#full-installation) * [Install the Kata packages only](#install-the-kata-packages-only) * [Further Information](#further-information) diff --git a/svc-mesh/service-mesh.md b/svc-mesh/service-mesh.md index da85ffb62..128dfcd2c 100644 --- a/svc-mesh/service-mesh.md +++ b/svc-mesh/service-mesh.md @@ -1,20 +1,20 @@ # Kata Containers and service mesh for Kubernetes * [Assumptions](#assumptions) -* [How do they work?](#how-do-they-work) +* [How they work](#how-they-work) * [Prerequisites](#prerequisites) * [Kata and Kubernetes](#kata-and-kubernetes) * [Restrictions](#restrictions) * [Install and deploy your service mesh](#install-and-deploy-your-service-mesh) - * [Istio](#istio) - * [Conduit](#conduit) + * [Service Mesh Istio](#service-mesh-istio) + * [Service Mesh Linkerd](#service-mesh-linkerd) * [Inject your services with sidecars](#inject-your-services-with-sidecars) - * [Istio](#istio-1) - * [Conduit](#conduit-1) + * [Sidecar Istio](#sidecar-istio) + * [Sidecar Linkerd](#sidecar-linkerd) * [Run your services with Kata](#run-your-services-with-kata) * [Lower privileges](#lower-privileges) * [Add annotations](#add-annotations) - * [Deploy!](#deploy) + * [Deploy](#deploy) A service mesh is a way to monitor and control the traffic between micro-services running in your Kubernetes cluster. It is a powerful @@ -26,9 +26,9 @@ brought by Kata Containers. You are expected to be familiar with concepts such as __pods__, __containers__, __control plane__, __data plane__, and __sidecar__. -## How do they work? +## How they work -Istio and Conduit both rely on the same model, where they run controller +Istio and Linkerd both rely on the same model, where they run controller applications in the control plane, and inject a proxy as a sidecar inside the pod running the service. The proxy registers in the control plane as a first step, and it constantly sends different sorts of information about @@ -59,7 +59,7 @@ you choose to annotate run with Kata Containers. ### Restrictions -As documented [here](https://github.com/runconduit/conduit/issues/982), +As documented [here](https://github.com/linkerd/linkerd2/issues/982), a kernel version between 4.14.22 and 4.14.40 causes a deadlock when `getsockopt()` gets called with the `SO_ORIGINAL_DST` option. Unfortunately, both service meshes use this system call with this same option from the @@ -74,7 +74,7 @@ is not able to perform a proper setup of the rules. ## Install and deploy your service mesh -### Istio +### Service Mesh Istio As a reference, you can follow Istio [instructions](https://istio.io/docs/setup/kubernetes/quick-start/#download-and-prepare-for-the-installation). @@ -97,26 +97,26 @@ $ kubectl get svc -n istio-system $ kubectl get pods -n istio-system -o wide ``` -### Conduit +### Service Mesh Linkerd -As a reference, follow the Conduit [instructions](https://conduit.io/getting-started/index.html). +As a reference, follow the Linkerd [instructions](https://linkerd.io/2/getting-started/index.html). -The following is a summary of what you need to install Conduit on your system: +The following is a summary of what you need to install Linkerd on your system: ``` -$ curl https://run.conduit.io/install | sh -$ export PATH=$PATH:$HOME/.conduit/bin +$ curl https://run.linkerd.io/install | sh +$ export PATH=$PATH:$HOME/.linkerd/bin ``` -Now deploy Conduit in the control plane of your cluster with the following: +Now deploy Linkerd in the control plane of your cluster with the following: ``` -$ conduit install | kubectl apply -f - +$ linkerd install | kubectl apply -f - ``` To verify that the control plane is properly deployed, you can use both of the following commands: ``` -$ kubectl get svc -n conduit -$ kubectl get pods -n conduit -o wide +$ kubectl get svc -n linkerd +$ kubectl get pods -n linkerd -o wide ``` ## Inject your services with sidecars @@ -128,31 +128,29 @@ sidecar proxy using the tools provided by each service mesh. If you do not have such a deployment ready, refer to the samples provided by each project. -### Istio +### Sidecar Istio -Istio provides a [bookinfo](https://istio.io/docs/guides/bookinfo/) +Istio provides a [bookinfo](https://istio.io/docs/examples/bookinfo/) sample, which you can rely on to inject their `envoy` proxy as a sidecar. You need to use their tool called `istioctl kube-inject` to inject -your YAML file. We use their [bookinfo](https://istio.io/docs/guides/bookinfo/) -sample as example: +your YAML file. We use their bookinfo sample as an example: ``` $ istioctl kube-inject -f samples/bookinfo/kube/bookinfo.yaml -o bookinfo-injected.yaml ``` -### Conduit +### Sidecar Linkerd -Conduit provides an [emojivoto](https://conduit.io/getting-started/index.html) -sample, which you can rely on to inject their `conduit` proxy as a +Linkerd provides an [emojivoto](https://linkerd.io/2/getting-started/index.html) +sample, which you can rely on to inject their `linkerd` proxy as a sidecar. -You need to use their tool called `conduit inject` to inject your YAML -file. We use their [emojivoto](https://conduit.io/getting-started/index.html) -sample as example: +You need to use their tool called `linkerd inject` to inject your YAML +file. We use their emojivoto sample as example: ``` $ wget https://raw.githubusercontent.com/runconduit/conduit-examples/master/emojivoto/emojivoto.yml -$ conduit inject emojivoto.yml > emojivoto-injected.yaml +$ linkerd inject emojivoto.yml > emojivoto-injected.yaml ``` ## Run your services with Kata @@ -165,7 +163,7 @@ containers, manually edit your deployment to make it work with Kata. In Kubernetes, the __init__ container is often `privileged` as it needs to setup the environment, which often needs some root privileges. In the case of those services meshes, all they need is the `NET_ADMIN` capability to -modify the underlying network rules. Conduit, by default, does not use +modify the underlying network rules. Linkerd, by default, does not use `privileged` container, but Istio does. Because of the previous reason, if you use Istio you need to switch all @@ -173,7 +171,7 @@ containers with `privileged: true` to `privileged: false`. ### Add annotations -There is no difference between Istio and Conduit in this section. It is +There is no difference between Istio and Linkerd in this section. It is about which CRI implementation you use. For both CRI-O and CRI-containerd, you have to add an annotation indicating @@ -240,7 +238,7 @@ spec: ... ``` -### Deploy! +### Deploy Deploy your application by using the following: ```