From 1fc7b7641dc7021a554c1049094b2b14a57ea958 Mon Sep 17 00:00:00 2001 From: Christophe de Dinechin Date: Wed, 28 Oct 2020 17:06:41 +0100 Subject: [PATCH 1/4] docs: Repair inconsistencies between 2.0 and 1.x The documentation `how-to/how-to-set-sandbox-config-kata.md` contains a number of differences relative to the 1.x variant, which do not seem to correspond to missing features in the actual code. Fixes: #1046 Signed-off-by: Christophe de Dinechin --- docs/how-to/how-to-set-sandbox-config-kata.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/how-to/how-to-set-sandbox-config-kata.md b/docs/how-to/how-to-set-sandbox-config-kata.md index 4d8d1d8457..ba347bca67 100644 --- a/docs/how-to/how-to-set-sandbox-config-kata.md +++ b/docs/how-to/how-to-set-sandbox-config-kata.md @@ -26,6 +26,7 @@ There are several kinds of Kata configurations and they are listed below. | Key | Value Type | Comments | |-------| ----- | ----- | | `io.katacontainers.config.agent.enable_tracing` | `boolean` | enable tracing for the agent | +| `io.katacontainers.config.agent.container_pipe_size` | uint32 | specify the size of the std(in/out) pipes created for containers | | `io.katacontainers.config.agent.kernel_modules` | string | the list of kernel modules and their parameters that will be loaded in the guest kernel. Semicolon separated list of kernel modules and their parameters. These modules will be loaded in the guest kernel using `modprobe`(8). E.g., `e1000e InterruptThrottleRate=3000,3000,3000 EEE=1; i915 enable_ppgtt=0` | | `io.katacontainers.config.agent.trace_mode` | string | the trace mode for the agent | | `io.katacontainers.config.agent.trace_type` | string | the trace type for the agent | @@ -38,15 +39,21 @@ There are several kinds of Kata configurations and they are listed below. | `io.katacontainers.config.hypervisor.block_device_cache_noflush` | `boolean` | Denotes whether flush requests for the device are ignored | | `io.katacontainers.config.hypervisor.block_device_cache_set` | `boolean` | cache-related options will be set to block devices or not | | `io.katacontainers.config.hypervisor.block_device_driver` | string | the driver to be used for block device, valid values are `virtio-blk`, `virtio-scsi`, `nvdimm`| +| `io.katacontainers.config.hypervisor.cpu_features` | `string` | Comma-separated list of CPU features to pass to the CPU (QEMU) | | `io.katacontainers.config.hypervisor.default_max_vcpus` | uint32| the maximum number of vCPUs allocated for the VM by the hypervisor | | `io.katacontainers.config.hypervisor.default_memory` | uint32| the memory assigned for a VM by the hypervisor in `MiB` | | `io.katacontainers.config.hypervisor.default_vcpus` | uint32| the default vCPUs assigned for a VM by the hypervisor | | `io.katacontainers.config.hypervisor.disable_block_device_use` | `boolean` | disallow a block device from being used | +| `io.katacontainers.config.hypervisor.disable_image_nvdimm` | `boolean` | specify if a `nvdimm` device should be used as rootfs for the guest (QEMU) | | `io.katacontainers.config.hypervisor.disable_vhost_net` | `boolean` | specify if `vhost-net` is not available on the host | | `io.katacontainers.config.hypervisor.enable_hugepages` | `boolean` | if the memory should be `pre-allocated` from huge pages | +| `io.katacontainers.config.hypervisor.enable_iommu_platform` | `boolean` | enable `iommu` on CCW devices (QEMU s390x) | +| `io.katacontainers.config.hypervisor.enable_iommu` | `boolean` | enable `iommu` on Q35 (QEMU x86_64) | | `io.katacontainers.config.hypervisor.enable_iothreads` | `boolean`| enable IO to be processed in a separate thread. Supported currently for virtio-`scsi` driver | | `io.katacontainers.config.hypervisor.enable_mem_prealloc` | `boolean` | the memory space used for `nvdimm` device by the hypervisor | | `io.katacontainers.config.hypervisor.enable_swap` | `boolean` | enable swap of VM memory | +| `io.katacontainers.config.hypervisor.enable_vhost_user_store` | `boolean` | enable vhost-user storage device (QEMU) | +| `io.katacontainers.config.hypervisor.enable_virtio_mem` | `boolean` | enable virtio-mem (QEMU) | | `io.katacontainers.config.hypervisor.entropy_source` | string| the path to a host source of entropy (`/dev/random`, `/dev/urandom` or real hardware RNG device) | | `io.katacontainers.config.hypervisor.file_mem_backend` | string | file based memory backend root directory | | `io.katacontainers.config.hypervisor.firmware_hash` | string | container firmware SHA-512 hash value | @@ -69,8 +76,10 @@ There are several kinds of Kata configurations and they are listed below. | `io.katacontainers.config.hypervisor.memory_slots` | uint32| the memory slots assigned to the VM by the hypervisor | | `io.katacontainers.config.hypervisor.msize_9p` | uint32 | the `msize` for 9p shares | | `io.katacontainers.config.hypervisor.path` | string | the hypervisor that will run the container VM | +| `io.katacontainers.config.hypervisor.pcie_root_port` | specify the number of PCIe Root Port devices. The PCIe Root Port device is used to hot-plug a PCIe device (QEMU) | | `io.katacontainers.config.hypervisor.shared_fs` | string | the shared file system type, either `virtio-9p` or `virtio-fs` | | `io.katacontainers.config.hypervisor.use_vsock` | `boolean` | specify use of `vsock` for agent communication | +| `io.katacontainers.config.hypervisor.vhost_user_store_path` | `string` | specify the directory path where vhost-user devices related folders, sockets and device nodes should be (QEMU) | | `io.katacontainers.config.hypervisor.virtio_fs_cache_size` | uint32 | virtio-fs DAX cache size in `MiB` | | `io.katacontainers.config.hypervisor.virtio_fs_cache` | string | the cache mode for virtio-fs, valid values are `always`, `auto` and `none` | | `io.katacontainers.config.hypervisor.virtio_fs_daemon` | string | virtio-fs `vhost-user` daemon path | @@ -101,7 +110,7 @@ $ cat /etc/containerd/config ``` -Additional documentation on the above configuration can be found in the +Additional documentation on the above configuration can be found in the [containerd docs](https://github.com/containerd/cri/blob/8d5a8355d07783ba2f8f451209f6bdcc7c412346/docs/config.md). # Example - Using annotations From d67921a2af247c5dac93773d037ea9aad1ea8cd0 Mon Sep 17 00:00:00 2001 From: Christophe de Dinechin Date: Tue, 27 Oct 2020 15:57:27 +0100 Subject: [PATCH 2/4] docs: Document restricted annotations Document restricted annotations, as implemented in https://github.com/kata-containers/kata-containers/pull/902 Fixes: #1044 Forward-port-of: https://github.com/kata-containers/documentation/pull/755 Signed-off-by: Christophe de Dinechin --- docs/how-to/how-to-set-sandbox-config-kata.md | 40 +++++++++++++++++-- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/docs/how-to/how-to-set-sandbox-config-kata.md b/docs/how-to/how-to-set-sandbox-config-kata.md index ba347bca67..f58b90e6e4 100644 --- a/docs/how-to/how-to-set-sandbox-config-kata.md +++ b/docs/how-to/how-to-set-sandbox-config-kata.md @@ -3,6 +3,11 @@ Kata Containers gives users freedom to customize at per-pod level, by setting a wide range of Kata specific annotations in the pod specification. +Some annotations may be [restricted](#restricted-annotations) by the +configuration file for security reasons, notably annotations that could lead the +runtime to execute programs on the host. Such annotations are marked with _(R)_ in +the tables below. + # Kata Configuration Annotations There are several kinds of Kata configurations and they are listed below. @@ -40,6 +45,7 @@ There are several kinds of Kata configurations and they are listed below. | `io.katacontainers.config.hypervisor.block_device_cache_set` | `boolean` | cache-related options will be set to block devices or not | | `io.katacontainers.config.hypervisor.block_device_driver` | string | the driver to be used for block device, valid values are `virtio-blk`, `virtio-scsi`, `nvdimm`| | `io.katacontainers.config.hypervisor.cpu_features` | `string` | Comma-separated list of CPU features to pass to the CPU (QEMU) | +| `io.katacontainers.config.hypervisor.ctlpath` (R) | `string` | Path to the `acrnctl` binary for the ACRN hypervisor | | `io.katacontainers.config.hypervisor.default_max_vcpus` | uint32| the maximum number of vCPUs allocated for the VM by the hypervisor | | `io.katacontainers.config.hypervisor.default_memory` | uint32| the memory assigned for a VM by the hypervisor in `MiB` | | `io.katacontainers.config.hypervisor.default_vcpus` | uint32| the default vCPUs assigned for a VM by the hypervisor | @@ -55,7 +61,7 @@ There are several kinds of Kata configurations and they are listed below. | `io.katacontainers.config.hypervisor.enable_vhost_user_store` | `boolean` | enable vhost-user storage device (QEMU) | | `io.katacontainers.config.hypervisor.enable_virtio_mem` | `boolean` | enable virtio-mem (QEMU) | | `io.katacontainers.config.hypervisor.entropy_source` | string| the path to a host source of entropy (`/dev/random`, `/dev/urandom` or real hardware RNG device) | -| `io.katacontainers.config.hypervisor.file_mem_backend` | string | file based memory backend root directory | +| `io.katacontainers.config.hypervisor.file_mem_backend` (R) | string | file based memory backend root directory | | `io.katacontainers.config.hypervisor.firmware_hash` | string | container firmware SHA-512 hash value | | `io.katacontainers.config.hypervisor.firmware` | string | the guest firmware that will run the container VM | | `io.katacontainers.config.hypervisor.guest_hook_path` | string | the path within the VM that will be used for drop in hooks | @@ -66,7 +72,7 @@ There are several kinds of Kata configurations and they are listed below. | `io.katacontainers.config.hypervisor.initrd_hash` | string | container guest initrd SHA-512 hash value | | `io.katacontainers.config.hypervisor.initrd` | string | the guest initrd image that will run in the container VM | | `io.katacontainers.config.hypervisor.jailer_hash` | string | container jailer SHA-512 hash value | -| `io.katacontainers.config.hypervisor.jailer_path` | string | the jailer that will constrain the container VM | +| `io.katacontainers.config.hypervisor.jailer_path` (R) | string | the jailer that will constrain the container VM | | `io.katacontainers.config.hypervisor.kernel_hash` | string | container kernel image SHA-512 hash value | | `io.katacontainers.config.hypervisor.kernel_params` | string | additional guest kernel parameters | | `io.katacontainers.config.hypervisor.kernel` | string | the kernel used to boot the container VM | @@ -79,7 +85,7 @@ There are several kinds of Kata configurations and they are listed below. | `io.katacontainers.config.hypervisor.pcie_root_port` | specify the number of PCIe Root Port devices. The PCIe Root Port device is used to hot-plug a PCIe device (QEMU) | | `io.katacontainers.config.hypervisor.shared_fs` | string | the shared file system type, either `virtio-9p` or `virtio-fs` | | `io.katacontainers.config.hypervisor.use_vsock` | `boolean` | specify use of `vsock` for agent communication | -| `io.katacontainers.config.hypervisor.vhost_user_store_path` | `string` | specify the directory path where vhost-user devices related folders, sockets and device nodes should be (QEMU) | +| `io.katacontainers.config.hypervisor.vhost_user_store_path` (R) | `string` | specify the directory path where vhost-user devices related folders, sockets and device nodes should be (QEMU) | | `io.katacontainers.config.hypervisor.virtio_fs_cache_size` | uint32 | virtio-fs DAX cache size in `MiB` | | `io.katacontainers.config.hypervisor.virtio_fs_cache` | string | the cache mode for virtio-fs, valid values are `always`, `auto` and `none` | | `io.katacontainers.config.hypervisor.virtio_fs_daemon` | string | virtio-fs `vhost-user` daemon path | @@ -168,3 +174,31 @@ spec: stdin: true tty: true ``` + +# Restricted annotations + +Some annotations are _restricted_, meaning that the configuration file specifies +the acceptable values. Currently, only hypervisor annotations are restricted, +for security reason, with the intent to control which binaries the Kata +Containers runtime will launch on your behalf. + +The configuration file validates the annotation _name_ as well as the annotation +_value_. + +The acceptable annotation names are defined by the `enable_annotations` entry in +the configuration file. + +For restricted annotations, an additional configuration entry provides a list of +acceptable values. Since most restricted annotations are intended to control +which binaries the runtime can execute, the valid value is generally provided by +a shell pattern, as defined by `glob(3)`. The table below provides the name of +the configuration entry: + +| Key | Config file entry | Comments | +|-------| ----- | ----- | +| `ctlpath` | `valid_ctlpaths` | Valid paths for `acrnctl` binary | +| `file_mem_backend` | `valid_file_mem_backends` | Valid locations for the file-based memory backend root directory | +| `jailer_path` | `valid_jailer_paths`| Valid paths for the jailer constraining the container VM (Firecracker) | +| `path` | `valid_hypervisor_paths` | Valid hypervisors to run the container VM | +| `vhost_user_store_path` | `valid_vhost_user_store_paths` | Valid paths for vhost-user related files| +| `virtio_fs_daemon` | `valid_virtio_fs_daemon_paths` | Valid paths for the `virtiofsd` daemon | From 6c083d9410796b3c52ccd558d2c7f83fbf3b9e7e Mon Sep 17 00:00:00 2001 From: Christophe de Dinechin Date: Tue, 27 Oct 2020 16:06:49 +0100 Subject: [PATCH 3/4] docs: Add a link to document describing how to use annotations Add a link to the document listing the available annotations Fixes: #1044 Forward-port-of: https://github.com/kata-containers/documentation/pull/757 Signed-off-by: Christophe de Dinechin --- docs/how-to/how-to-load-kernel-modules-with-kata.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/how-to/how-to-load-kernel-modules-with-kata.md b/docs/how-to/how-to-load-kernel-modules-with-kata.md index 12e5431493..77ade97972 100644 --- a/docs/how-to/how-to-load-kernel-modules-with-kata.md +++ b/docs/how-to/how-to-load-kernel-modules-with-kata.md @@ -56,8 +56,9 @@ There are some limitations with this approach: As was mentioned above, not all containers need the same modules, therefore using the configuration file for specifying the list of kernel modules per [POD][3] can -be a pain. Unlike the configuration file, annotations provide a way to specify -custom configurations per POD. +be a pain. +Unlike the configuration file, [annotations](how-to-set-sandbox-config-kata.md) +provide a way to specify custom configurations per POD. The list of kernel modules and parameters can be set using the annotation `io.katacontainers.config.agent.kernel_modules` as a semicolon separated From 4c78814bdaa8b72b09d5bafa6d587f843ff5f0f9 Mon Sep 17 00:00:00 2001 From: Christophe de Dinechin Date: Mon, 9 Nov 2020 16:52:33 +0100 Subject: [PATCH 4/4] docs: Fix pre-existing spelling mistakes caught by the CI The documentation contains existing spelling mistakes that are caught by the CI and prevent checking in. The errors include: INFO: Spell checking file 'docs/how-to/how-to-load-kernel-modules-with-kata.md' WARNING: Word 'configurated': did you mean one of the following?: configuration, reconfigured, Confederate, confederate WARNING: Word 'cri': did you mean one of the following?: cir, crib, chi, cry, Fri, crier ERROR: Spell check failed for file: 'docs/how-to/how-to-load-kernel-modules-with-kata.md' INFO: spell check failed for document docs/how-to/how-to-load-kernel-modules-with-kata.md INFO: Spell checking file 'docs/how-to/how-to-set-sandbox-config-kata.md' INFO: Spell check successful for file: 'docs/how-to/how-to-set-sandbox-config-kata.md' ERROR: spell check failed, See https://github.com/kata-containers/documentation/blob/master/Documentation-Requirements.md#spelling for more information. Signed-off-by: Christophe de Dinechin --- docs/how-to/how-to-load-kernel-modules-with-kata.md | 2 +- docs/how-to/how-to-set-sandbox-config-kata.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/how-to/how-to-load-kernel-modules-with-kata.md b/docs/how-to/how-to-load-kernel-modules-with-kata.md index 77ade97972..24a3546012 100644 --- a/docs/how-to/how-to-load-kernel-modules-with-kata.md +++ b/docs/how-to/how-to-load-kernel-modules-with-kata.md @@ -102,7 +102,7 @@ spec: tty: true ``` -> **Note**: To pass annotations to Kata containers, [cri must to be configurated correctly](how-to-set-sandbox-config-kata.md#cri-configuration) +> **Note**: To pass annotations to Kata containers, [CRI-O must be configured correctly](how-to-set-sandbox-config-kata.md#cri-o-configuration) [1]: ../../src/runtime [2]: ../../src/agent diff --git a/docs/how-to/how-to-set-sandbox-config-kata.md b/docs/how-to/how-to-set-sandbox-config-kata.md index f58b90e6e4..72cdf1304c 100644 --- a/docs/how-to/how-to-set-sandbox-config-kata.md +++ b/docs/how-to/how-to-set-sandbox-config-kata.md @@ -91,7 +91,7 @@ There are several kinds of Kata configurations and they are listed below. | `io.katacontainers.config.hypervisor.virtio_fs_daemon` | string | virtio-fs `vhost-user` daemon path | | `io.katacontainers.config.hypervisor.virtio_fs_extra_args` | string | extra options passed to `virtiofs` daemon | -# CRI Configuration +# CRI-O Configuration In case of CRI-O, all annotations specified in the pod spec are passed down to Kata.