mirror of
https://github.com/kata-containers/kata-containers.git
synced 2025-04-28 19:54:35 +00:00
kata-deploy: QEMU, for 2.x, already includes virtiofs
There's no reason to ship qemu & qemu-virtiofs when the former already includes vitiofs support (and that's the default for 2.x deployments). In case we will enable experimental qemu DAX support, we should add a new target, a "qemu-experimental" target, as Carlos has been working on. Fixes: #1424 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
This commit is contained in:
Fabiano Fidêncio
parent
657bd789a3
commit
f21c54a985
29
.github/workflows/main.yaml
vendored
29
.github/workflows/main.yaml
vendored
@ -129,33 +129,6 @@ jobs:
|
||||
name: kata-artifacts
|
||||
path: kata-static-nemu.tar.gz
|
||||
|
||||
# Job for building the QEMU binaries with virtiofs support
|
||||
build-qemu-virtiofsd:
|
||||
runs-on: ubuntu-16.04
|
||||
needs: get-artifact-list
|
||||
env:
|
||||
buildstr: "install_qemu_virtiofsd"
|
||||
steps:
|
||||
- uses: actions/checkout@v1
|
||||
- name: get-artifact-list
|
||||
uses: actions/download-artifact@master
|
||||
with:
|
||||
name: artifact-list
|
||||
- name: build-qemu-virtiofsd
|
||||
run: |
|
||||
if grep -q $buildstr ./artifact-list/artifact-list.txt; then
|
||||
$GITHUB_WORKSPACE/.github/workflows/generate-artifact-tarball.sh $buildstr
|
||||
echo "artifact-built=true" >> $GITHUB_ENV
|
||||
else
|
||||
echo "artifact-built=false" >> $GITHUB_ENV
|
||||
fi
|
||||
- name: store-artifacts
|
||||
if: ${{ env.artifact-built }} == 'true'
|
||||
uses: actions/upload-artifact@master
|
||||
with:
|
||||
name: kata-artifacts
|
||||
path: kata-static-qemu-virtiofsd.tar.gz
|
||||
|
||||
# Job for building the image
|
||||
build-image:
|
||||
runs-on: ubuntu-16.04
|
||||
@ -266,7 +239,7 @@ jobs:
|
||||
|
||||
gather-artifacts:
|
||||
runs-on: ubuntu-16.04
|
||||
needs: [build-experimental-kernel, build-kernel, build-qemu, build-qemu-virtiofsd, build-image, build-firecracker, build-kata-components, build-nemu, build-clh]
|
||||
needs: [build-experimental-kernel, build-kernel, build-qemu, build-image, build-firecracker, build-kata-components, build-nemu, build-clh]
|
||||
steps:
|
||||
- uses: actions/checkout@v1
|
||||
- name: get-artifacts
|
||||
|
||||
28
.github/workflows/release.yaml
vendored
28
.github/workflows/release.yaml
vendored
@ -104,32 +104,6 @@ jobs:
|
||||
name: kata-artifacts
|
||||
path: kata-static-qemu.tar.gz
|
||||
|
||||
build-qemu-virtiofsd:
|
||||
runs-on: ubuntu-16.04
|
||||
needs: get-artifact-list
|
||||
env:
|
||||
buildstr: "install_qemu_virtiofsd"
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
- name: get-artifact-list
|
||||
uses: actions/download-artifact@v2
|
||||
with:
|
||||
name: artifact-list
|
||||
- name: build-qemu-virtiofsd
|
||||
run: |
|
||||
if grep -q $buildstr artifact-list.txt; then
|
||||
$GITHUB_WORKSPACE/.github/workflows/generate-local-artifact-tarball.sh $buildstr
|
||||
echo "artifact-built=true" >> $GITHUB_ENV
|
||||
else
|
||||
echo "artifact-built=false" >> $GITHUB_ENV
|
||||
fi
|
||||
- name: store-artifacts
|
||||
if: ${{ env.artifact-built }} == 'true'
|
||||
uses: actions/upload-artifact@v2
|
||||
with:
|
||||
name: kata-artifacts
|
||||
path: kata-static-qemu-virtiofsd.tar.gz
|
||||
|
||||
build-image:
|
||||
runs-on: ubuntu-16.04
|
||||
needs: get-artifact-list
|
||||
@ -237,7 +211,7 @@ jobs:
|
||||
|
||||
gather-artifacts:
|
||||
runs-on: ubuntu-16.04
|
||||
needs: [build-experimental-kernel, build-kernel, build-qemu, build-qemu-virtiofsd, build-image, build-firecracker, build-kata-components, build-clh]
|
||||
needs: [build-experimental-kernel, build-kernel, build-qemu, build-image, build-firecracker, build-kata-components, build-clh]
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
- name: get-artifacts
|
||||
|
||||
@ -1,61 +1,12 @@
|
||||
# Kata Containers with virtio-fs
|
||||
|
||||
- [Introduction](#introduction)
|
||||
- [Pre-requisites](#pre-requisites)
|
||||
- [Install Kata Containers with virtio-fs support](#install-kata-containers-with-virtio-fs-support)
|
||||
- [Run a Kata Container utilizing virtio-fs](#run-a-kata-container-utilizing-virtio-fs)
|
||||
- [Kata Containers with virtio-fs](#kata-containers-with-virtio-fs)
|
||||
- [Introduction](#introduction)
|
||||
|
||||
## Introduction
|
||||
|
||||
Container deployments utilize explicit or implicit file sharing between host filesystem and containers. From a trust perspective, avoiding a shared file-system between the trusted host and untrusted container is recommended. This is not always feasible. In Kata Containers, block-based volumes are preferred as they allow usage of either device pass through or `virtio-blk` for access within the virtual machine.
|
||||
|
||||
As of the 1.7 release of Kata Containers, [9pfs](https://www.kernel.org/doc/Documentation/filesystems/9p.txt) is the default filesystem sharing mechanism. While this does allow for workload compatibility, it does so with degraded performance and potential for POSIX compliance limitations.
|
||||
As of the 2.0 release of Kata Containers, [virtio-fs](https://virtio-fs.gitlab.io/) is the default filesystem sharing mechanism.
|
||||
|
||||
To help address these limitations, [virtio-fs](https://virtio-fs.gitlab.io/) has been developed. virtio-fs is a shared file system that lets virtual machines access a directory tree on the host. In Kata Containers, virtio-fs can be used to share container volumes, secrets, config-maps, configuration files (hostname, hosts, `resolv.conf`) and the container rootfs on the host with the guest. virtio-fs provides significant performance and POSIX compliance improvements compared to 9pfs.
|
||||
|
||||
Enabling of virtio-fs requires changes in the guest kernel as well as the VMM. For Kata Containers, experimental virtio-fs support is enabled through `qemu` and `cloud-hypervisor` VMMs.
|
||||
|
||||
**Note: virtio-fs support is experimental in the 1.7 release of Kata Containers. Work is underway to improve stability, performance and upstream integration. This is available for early preview - use at your own risk**
|
||||
|
||||
This document describes how to get Kata Containers to work with virtio-fs.
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
Before Kata 1.8 this feature required the host to have hugepages support enabled. Enable this with the `sysctl vm.nr_hugepages=1024` command on the host.In later versions of Kata, virtio-fs leverages `/dev/shm` as the shared memory backend. The default size of `/dev/shm` on a system is typically half of the total system memory. This can pose a physical limit to the maximum number of pods that can be launched with virtio-fs. This can be overcome by increasing the size of `/dev/shm` as shown below:
|
||||
|
||||
```bash
|
||||
$ mount -o remount,size=${desired_shm_size} /dev/shm
|
||||
```
|
||||
|
||||
## Install Kata Containers with virtio-fs support
|
||||
|
||||
The Kata Containers `qemu` configuration with virtio-fs and the `virtiofs` daemon are available in the [Kata Container release](https://github.com/kata-containers/runtime/releases) artifacts starting with the 1.9 release. Installation is available through [distribution packages](https://github.com/kata-containers/documentation/blob/master/install/README.md#supported-distributions) as well through [`kata-deploy`](https://github.com/kata-containers/packaging/tree/master/kata-deploy).
|
||||
|
||||
**Note: Support for virtio-fs was first introduced in `NEMU` hypervisor in Kata 1.8 release. This hypervisor has been deprecated.**
|
||||
|
||||
Install the latest release of Kata with `kata-deploy` as follows:
|
||||
```
|
||||
docker run --runtime=runc -v /opt/kata:/opt/kata -v /var/run/dbus:/var/run/dbus -v /run/systemd:/run/systemd -v /etc/docker:/etc/docker -it katadocker/kata-deploy kata-deploy-docker install
|
||||
```
|
||||
|
||||
This will place the Kata release artifacts in `/opt/kata`, and update Docker's configuration to include a runtime target, `kata-qemu-virtiofs`. Learn more about `kata-deploy` and how to use `kata-deploy` in Kubernetes [here](https://github.com/kata-containers/packaging/tree/master/kata-deploy#kubernetes-quick-start).
|
||||
|
||||
## Run a Kata Container utilizing virtio-fs
|
||||
|
||||
Once installed, start a new container, utilizing `qemu` + `virtiofs`:
|
||||
```bash
|
||||
$ docker run --runtime=kata-qemu-virtiofs -it busybox
|
||||
```
|
||||
|
||||
Verify the new container is running with the `qemu` hypervisor as well as using `virtiofsd`. To do this look for the hypervisor path and the `virtiofs` daemon process on the host:
|
||||
```bash
|
||||
$ ps -aux | grep virtiofs
|
||||
root ... /home/foo/build-x86_64_virt/x86_64_virt-softmmu/qemu-system-x86_64_virt
|
||||
... -machine virt,accel=kvm,kernel_irqchip,nvdimm ...
|
||||
root ... /home/foo/build-x86_64_virt/virtiofsd-x86_64 ...
|
||||
```
|
||||
|
||||
You can also try out virtio-fs using `cloud-hypervisor` VMM:
|
||||
```bash
|
||||
$ docker run --runtime=kata-clh -it busybox
|
||||
```
|
||||
virtio-fs support works out of the box for `cloud-hypervisor` and `qemu`, when Kata Containers is deployed using `kata-deploy`. Learn more about `kata-deploy` and how to use `kata-deploy` in Kubernetes [here](https://github.com/kata-containers/packaging/tree/master/kata-deploy#kubernetes-quick-start).
|
||||
1
src/runtime/.gitignore
vendored
1
src/runtime/.gitignore
vendored
@ -10,7 +10,6 @@ coverage.html
|
||||
/cli/config/configuration-fc.toml
|
||||
/cli/config/configuration-nemu.toml
|
||||
/cli/config/configuration-qemu.toml
|
||||
/cli/config/configuration-qemu-virtiofs.toml
|
||||
/cli/config/configuration-clh.toml
|
||||
/cli/config-generated.go
|
||||
/cli/containerd-shim-kata-v2/config-generated.go
|
||||
|
||||
@ -12,9 +12,6 @@ CPUFEATURES := pmu=off
|
||||
|
||||
QEMUCMD := qemu-system-x86_64
|
||||
|
||||
# Qemu experimental with virtiofs
|
||||
QEMUVIRTIOFSCMD := qemu-virtiofs-system-x86_64
|
||||
|
||||
# Firecracker binary name
|
||||
FCCMD := firecracker
|
||||
# Firecracker's jailer binary name
|
||||
|
||||
@ -17,7 +17,6 @@ supported_artifacts=(
|
||||
"install_kata_components"
|
||||
"install_kernel"
|
||||
"install_qemu"
|
||||
"install_qemu_virtiofsd"
|
||||
)
|
||||
|
||||
for c in ${supported_artifacts[@]}; do echo $c; done
|
||||
|
||||
@ -133,7 +133,7 @@ This image contains all the necessary artifacts for running Kata Containers, all
|
||||
from the [Kata Containers release page](https://github.com/kata-containers/runtime/releases).
|
||||
|
||||
Host artifacts:
|
||||
* `cloud-hypervisor`, `firecracker`, `qemu-system-x86_64`, `qemu-virtiofs-system-x86_64` and supporting binaries
|
||||
* `cloud-hypervisor`, `firecracker`, `qemu-system-x86_64`, and supporting binaries
|
||||
* `containerd-shim-kata-v2`
|
||||
* `kata-collect-data.sh`
|
||||
* `kata-runtime`
|
||||
|
||||
@ -116,7 +116,6 @@ function test_kata() {
|
||||
# apply runtime classes:
|
||||
kubectl apply -f "$YAMLPATH/k8s-1.14/kata-clh-runtimeClass.yaml"
|
||||
kubectl apply -f "$YAMLPATH/k8s-1.14/kata-qemu-runtimeClass.yaml"
|
||||
kubectl apply -f "$YAMLPATH/k8s-1.14/kata-qemu-virtiofs-runtimeClass.yaml"
|
||||
|
||||
kubectl get runtimeclasses
|
||||
|
||||
|
||||
@ -1,42 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
labels:
|
||||
run: php-apache-kata-qemu-virtiofs
|
||||
name: php-apache-kata-qemu-virtiofs
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
run: php-apache-kata-qemu-virtiofs
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
run: php-apache-kata-qemu-virtiofs
|
||||
spec:
|
||||
runtimeClassName: kata-qemu-virtiofs
|
||||
containers:
|
||||
- image: k8s.gcr.io/hpa-example
|
||||
imagePullPolicy: Always
|
||||
name: php-apache
|
||||
ports:
|
||||
- containerPort: 80
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
restartPolicy: Always
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: php-apache-kata-qemu-virtiofs
|
||||
spec:
|
||||
ports:
|
||||
- port: 80
|
||||
protocol: TCP
|
||||
targetPort: 80
|
||||
selector:
|
||||
run: php-apache-kata-qemu-virtiofs
|
||||
sessionAffinity: None
|
||||
type: ClusterIP
|
||||
@ -1,14 +0,0 @@
|
||||
---
|
||||
kind: RuntimeClass
|
||||
apiVersion: node.k8s.io/v1alpha1
|
||||
metadata:
|
||||
name: kata-qemu-virtiofs
|
||||
spec:
|
||||
runtimeHandler: kata-qemu-virtiofs
|
||||
---
|
||||
kind: RuntimeClass
|
||||
apiVersion: node.k8s.io/v1alpha1
|
||||
metadata:
|
||||
name: kata
|
||||
spec:
|
||||
runtimeHandler: kata
|
||||
@ -1,12 +0,0 @@
|
||||
---
|
||||
kind: RuntimeClass
|
||||
apiVersion: node.k8s.io/v1beta1
|
||||
metadata:
|
||||
name: kata
|
||||
handler: kata
|
||||
---
|
||||
kind: RuntimeClass
|
||||
apiVersion: node.k8s.io/v1beta1
|
||||
metadata:
|
||||
name: kata-qemu-virtiofs
|
||||
handler: kata-qemu-virtiofs
|
||||
@ -147,13 +147,6 @@ install_qemu() {
|
||||
kata_version="${kata_version}" "${pkg_root_dir}/static-build/qemu/build-static-qemu.sh"
|
||||
}
|
||||
|
||||
# Install static qemu-virtiofsd asset
|
||||
install_qemu_virtiofsd() {
|
||||
kata_version=${1:-$kata_version}
|
||||
info "build static qemu-virtiofs"
|
||||
kata_version="${kata_version}" "${pkg_root_dir}/static-build/qemu-virtiofs/build-static-qemu-virtiofs.sh"
|
||||
}
|
||||
|
||||
# Install static firecracker asset
|
||||
install_firecracker() {
|
||||
kata_version=${1:-$kata_version}
|
||||
@ -211,8 +204,6 @@ install_kata_components() {
|
||||
untar_qemu_binaries() {
|
||||
info "Install static qemu"
|
||||
tar xf kata-static-qemu.tar.gz -C "${destdir}"
|
||||
info "Install static qemu-virtiofs"
|
||||
tar xf kata-static-qemu-virtiofsd.tar.gz -C "${destdir}"
|
||||
}
|
||||
|
||||
main() {
|
||||
@ -242,7 +233,6 @@ main() {
|
||||
install_kernel
|
||||
install_clh
|
||||
install_qemu
|
||||
install_qemu_virtiofsd
|
||||
install_firecracker
|
||||
install_image
|
||||
|
||||
|
||||
@ -1,78 +0,0 @@
|
||||
# Copyright (c) 2019 Intel Corporation
|
||||
# Copyright (c) 2020 Ant Group
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
from ubuntu:20.04
|
||||
|
||||
ARG QEMU_DESTDIR
|
||||
ARG QEMU_VIRTIOFS_REPO
|
||||
# commit/tag/branch
|
||||
ARG QEMU_VIRTIOFS_TAG
|
||||
ARG QEMU_TARBALL
|
||||
ARG PREFIX
|
||||
|
||||
WORKDIR /root/qemu-virtiofs
|
||||
RUN apt-get update && apt-get upgrade -y
|
||||
RUN apt-get --no-install-recommends install -y \
|
||||
apt-utils \
|
||||
autoconf \
|
||||
automake \
|
||||
bc \
|
||||
bison \
|
||||
ca-certificates \
|
||||
cpio \
|
||||
flex \
|
||||
gawk \
|
||||
libaudit-dev \
|
||||
libblkid-dev \
|
||||
libcap-dev \
|
||||
libcap-ng-dev \
|
||||
libdw-dev \
|
||||
libelf-dev \
|
||||
libffi-dev \
|
||||
libglib2.0-0 \
|
||||
libglib2.0-dev \
|
||||
libglib2.0-dev git \
|
||||
libltdl-dev \
|
||||
libmount-dev \
|
||||
libpixman-1-dev \
|
||||
libpmem-dev \
|
||||
libseccomp-dev \
|
||||
libseccomp2 \
|
||||
libselinux1-dev \
|
||||
libtool \
|
||||
make \
|
||||
pkg-config \
|
||||
pkg-config \
|
||||
python \
|
||||
python-dev \
|
||||
rsync \
|
||||
seccomp \
|
||||
zlib1g-dev
|
||||
|
||||
RUN cd .. && git clone "${QEMU_VIRTIOFS_REPO}" qemu-virtiofs
|
||||
RUN git checkout "${QEMU_VIRTIOFS_TAG}"
|
||||
|
||||
ADD scripts/configure-hypervisor.sh /root/configure-hypervisor.sh
|
||||
ADD qemu /root/kata_qemu
|
||||
ADD scripts/apply_patches.sh /root/apply_patches.sh
|
||||
ADD static-build /root/static-build
|
||||
|
||||
# Apply experimental specific patches
|
||||
# Patches to quick fix virtiofs fork
|
||||
ENV VIRTIOFS_PATCHES_DIR=/root/kata_qemu/patches/${QEMU_VIRTIOFS_TAG}/
|
||||
RUN /root/apply_patches.sh ${VIRTIOFS_PATCHES_DIR}
|
||||
# Apply the stable branch patches
|
||||
RUN stable_branch=$(cat VERSION | awk 'BEGIN{FS=OFS="."}{print $1 "." $2 ".x"}') && \
|
||||
/root/apply_patches.sh "/root/kata_qemu/patches/${stable_branch}"
|
||||
|
||||
RUN PREFIX="${PREFIX}" /root/configure-hypervisor.sh -s kata-qemu | sed -e 's|--disable-seccomp||g' | xargs ./configure \
|
||||
--with-pkgversion=kata-static
|
||||
|
||||
RUN make -j$(nproc)
|
||||
RUN make -j$(nproc) virtiofsd
|
||||
RUN make install DESTDIR="${QEMU_DESTDIR}"
|
||||
RUN cd "${QEMU_DESTDIR}/${PREFIX}" && \
|
||||
mv bin/qemu-system-x86_64 bin/qemu-virtiofs-system-x86_64 && \
|
||||
mv libexec/kata-qemu/virtiofsd bin/virtiofsd-dax
|
||||
RUN /root/static-build/scripts/qemu-build-post.sh
|
||||
@ -1,7 +0,0 @@
|
||||
MK_DIR :=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
|
||||
CONFIG_DIR := $(MK_DIR)/../../scripts/
|
||||
|
||||
build:
|
||||
"$(MK_DIR)/build-static-qemu-virtiofs.sh"
|
||||
clean:
|
||||
rm -f kata-qemu-static.tar.gz
|
||||
@ -1,56 +0,0 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# Copyright (c) 2019 Intel Corporation
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
set -o errexit
|
||||
set -o nounset
|
||||
set -o pipefail
|
||||
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "${script_dir}/../../scripts/lib.sh"
|
||||
source "${script_dir}/../qemu.blacklist"
|
||||
|
||||
DOCKER_CLI="docker"
|
||||
|
||||
if ! command -v docker &>/dev/null && command -v podman &>/dev/null; then
|
||||
DOCKER_CLI="podman"
|
||||
fi
|
||||
|
||||
kata_version="${kata_version:-}"
|
||||
packaging_dir="${script_dir}/../.."
|
||||
qemu_virtiofs_repo=$(get_from_kata_deps "assets.hypervisor.qemu-experimental.url" "${kata_version}")
|
||||
# This tag will be supported on the runtime versions.yaml
|
||||
qemu_virtiofs_tag=$(get_from_kata_deps "assets.hypervisor.qemu-experimental.tag" "${kata_version}")
|
||||
qemu_virtiofs_tar="kata-static-qemu-virtiofsd.tar.gz"
|
||||
qemu_tmp_tar="kata-static-qemu-virtiofsd-tmp.tar.gz"
|
||||
qemu_destdir="/tmp/qemu-virtiofs-static"
|
||||
|
||||
info "Build ${qemu_virtiofs_repo} tag: ${qemu_virtiofs_tag}"
|
||||
|
||||
http_proxy="${http_proxy:-}"
|
||||
https_proxy="${https_proxy:-}"
|
||||
prefix="${prefix:-"/opt/kata"}"
|
||||
|
||||
sudo "${DOCKER_CLI}" build \
|
||||
--no-cache \
|
||||
--build-arg http_proxy="${http_proxy}" \
|
||||
--build-arg https_proxy="${https_proxy}" \
|
||||
--build-arg QEMU_DESTDIR="${qemu_destdir}" \
|
||||
--build-arg QEMU_VIRTIOFS_REPO="${qemu_virtiofs_repo}" \
|
||||
--build-arg QEMU_VIRTIOFS_TAG="${qemu_virtiofs_tag}" \
|
||||
--build-arg QEMU_TARBALL="${qemu_virtiofs_tar}" \
|
||||
--build-arg PREFIX="${prefix}" \
|
||||
"${packaging_dir}" \
|
||||
-f "${script_dir}/Dockerfile" \
|
||||
-t qemu-virtiofs-static
|
||||
|
||||
sudo "${DOCKER_CLI}" run \
|
||||
--rm \
|
||||
-i \
|
||||
-v "${PWD}":/share qemu-virtiofs-static \
|
||||
mv "${qemu_destdir}/${qemu_virtiofs_tar}" /share/
|
||||
|
||||
sudo chown ${USER}:${USER} "${PWD}/${qemu_virtiofs_tar}"
|
||||
Loading…
Reference in New Issue
Block a user