kata-deploy: Remove kustomize yamls, rely on helm-chart only

As the kata-deploy helm chart has been the only way we've been testing
kata-containers deployment as part of our CI, it's time to finally get
rid of the kustomize yamls and avoid us having to maintain two different
methods (with one of those not being tested).

Here I removed:
* kata-deploy yamls and kustomize yamls
* kata-cleanup yamls and kustomize yamls
* kata-rbac yals and kustomize yamls
* README.md for the kustomize yamls was removed

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
This commit is contained in:
Fabiano Fidêncio
2025-10-07 14:26:36 +02:00
committed by Fabiano Fidêncio
parent 3418cedacc
commit 8c4bad68a8
28 changed files with 5 additions and 612 deletions

View File

@@ -450,7 +450,7 @@ You can build and install the guest kernel image as shown [here](../tools/packag
# Install a hypervisor
When setting up Kata using a [packaged installation method](install/README.md#installing-on-a-linux-system), the
`QEMU` VMM is installed automatically. Cloud-Hypervisor, Firecracker and StratoVirt VMMs are available from the [release tarballs](https://github.com/kata-containers/kata-containers/releases), as well as through [`kata-deploy`](../tools/packaging/kata-deploy/README.md).
`QEMU` VMM is installed automatically. Cloud-Hypervisor, Firecracker and StratoVirt VMMs are available from the [release tarballs](https://github.com/kata-containers/kata-containers/releases), as well as through [`kata-deploy`](../tools/packaging/kata-deploy/helm-chart/README.md).
You may choose to manually build your VMM/hypervisor.
## Build a custom QEMU

View File

@@ -318,7 +318,7 @@ Finally, an operational kata container with IBM Secure Execution is now running.
It is reasonable to expect that the manual steps mentioned above can be easily executed.
Typically, you can use
[kata-deploy](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/README.md)
[kata-deploy](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/helm-chart/README.md)
to install Kata Containers on a Kubernetes cluster. However, when leveraging IBM Secure Execution,
you need to employ the confidential container's
[operator](https://github.com/confidential-containers/operator).

View File

@@ -6,4 +6,4 @@ Container deployments utilize explicit or implicit file sharing between host fil
As of the 2.0 release of Kata Containers, [virtio-fs](https://virtio-fs.gitlab.io/) is the default filesystem sharing mechanism.
virtio-fs support works out of the box for `cloud-hypervisor` and `qemu`, when Kata Containers is deployed using `kata-deploy`. Learn more about `kata-deploy` and how to use `kata-deploy` in Kubernetes [here](../../tools/packaging/kata-deploy/README.md#kubernetes-quick-start).
virtio-fs support works out of the box for `cloud-hypervisor` and `qemu`, when Kata Containers is deployed using `kata-deploy`. Learn more about `kata-deploy` and how to use `kata-deploy` in Kubernetes [here](../../tools/packaging/kata-deploy/helm-chart/README.md).

View File

@@ -32,7 +32,7 @@ architectures:
### Kata Deploy Installation
Follow the [`kata-deploy`](../../tools/packaging/kata-deploy/README.md).
Follow the [`kata-deploy`](../../tools/packaging/kata-deploy/helm-chart/README.md).
### Official packages
`ToDo`
### Automatic Installation

View File

@@ -419,7 +419,7 @@ You might need to disable Docker before initializing Kubernetes. Be aware
that the OpenSSL container image built above will need to be exported from
Docker and imported into containerd.
If Kata is installed through [`kata-deploy`](../../tools/packaging/kata-deploy/README.md)
If Kata is installed through [`kata-deploy`](../../tools/packaging/kata-deploy/helm-chart/README.md)
there will be multiple `configuration.toml` files associated with different
hypervisors. Rather than add in the custom Kata kernel, Kata rootfs, and
kernel modules to each `configuration.toml` as the default, instead use

View File

@@ -1,275 +0,0 @@
# `kata-deploy`
[`kata-deploy`](.) provides a Dockerfile, which contains all of the binaries
and artifacts required to run Kata Containers, as well as reference DaemonSets, which can
be utilized to install Kata Containers on a running Kubernetes cluster.
> **Note**: installation through DaemonSets successfully installs `katacontainers.io/kata-runtime`
> on a node only if it uses either containerd or CRI-O CRI-shims.
## Kubernetes quick start
### Install Kata on a running Kubernetes cluster
#### k3s cluster
For your [k3s](https://k3s.io/) cluster, run:
```sh
$ git clone https://github.com/kata-containers/kata-containers.git
```
Check and switch to the stable branch of your choice, if wanted, and then run:
```bash
$ cd kata-containers/tools/packaging/kata-deploy
$ kubectl apply -f kata-rbac/base/kata-rbac.yaml
$ kubectl apply -k kata-deploy/overlays/k3s
$ kubectl apply -f kata-deploy/base/kata-deploy.yaml
```
#### RKE2 cluster
For your [RKE2](https://docs.rke2.io/) cluster, run:
```sh
$ git clone https://github.com/kata-containers/kata-containers.git
```
Check and switch to the stable branch of your choice, if wanted, and then run:
```bash
$ cd kata-containers/tools/packaging/kata-deploy
$ kubectl apply -f kata-rbac/base/kata-rbac.yaml
$ kubectl apply -k kata-deploy/overlays/rke2
$ kubectl apply -f kata-deploy/base/kata-deploy.yaml
```
#### k0s cluster
> [!IMPORTANT]
> As in this section, when following the rest of these instructions, you must use
> `sudo k0s kubectl` instead of `kubectl` for k0s.
> [!NOTE]
> The supported version of k0s is **v1.27.1+k0s** and above, since k0s support in Kata leverages
[dynamic runtime configuration](https://docs.k0sproject.io/v1.29.1+k0s.1/runtime/#k0s-managed-dynamic-runtime-configuration),
which was introduced in that version.
>
> Dynamic runtime configuration is enabled by default in k0s, and you can make sure it is enabled by verifying that `/etc/k0s/containerd.toml` contains the following line:
>
> ```toml
> # k0s_managed=true
> ```
For your [k0s](https://k0sproject.io/) cluster, run:
```sh
$ git clone https://github.com/kata-containers/kata-containers.git
```
Check and switch to "main", and then run:
```bash
$ cd kata-containers/tools/packaging/kata-deploy
$ sudo k0s kubectl apply -f kata-rbac/base/kata-rbac.yaml
$ sudo k0s kubectl apply -k kata-deploy/overlays/k0s
$ sudo k0s kubectl apply -f kata-deploy/base/kata-deploy.yaml
```
#### Microk8s Kubernetes cluster
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
$ kubectl apply -k https://github.com/kata-containers/kata-containers//tools/packaging/kata-deploy/kata-deploy/overlays/microk8s
```
#### Vanilla Kubernetes cluster
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
```
### Ensure Kata has been installed
```bash
$ kubectl -n kube-system wait --timeout=10m --for=condition=Ready -l name=kata-deploy pod
```
### Run a sample workload
Workloads specify the runtime they'd like to utilize by setting the appropriate `runtimeClass` object within
the `Pod` specification. The `runtimeClass` examples provided define a node selector to match node label `katacontainers.io/kata-runtime:"true"`,
which will ensure the workload is only scheduled on a node that has Kata Containers installed
`runtimeClass` is a built-in type in Kubernetes, and are created by kata-deploy during the deployment.
The following YAML snippet shows how to specify a workload should use Kata with `Dragonball`:
```yaml
spec:
template:
spec:
runtimeClassName: kata-dragonball
```
The following YAML snippet shows how to specify a workload should use Kata with Cloud Hypervisor:
```yaml
spec:
template:
spec:
runtimeClassName: kata-clh
```
The following YAML snippet shows how to specify a workload should use Kata with Firecracker:
```yaml
spec:
template:
spec:
runtimeClassName: kata-fc
```
The following YAML snippet shows how to specify a workload should use Kata with StratoVirt:
```yaml
spec:
template:
spec:
runtimeClassName: kata-stratovirt
```
The following YAML snippet shows how to specify a workload should use Kata with QEMU:
```yaml
spec:
template:
spec:
runtimeClassName: kata-qemu
```
To run an example with `kata-dragonball`:
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-dragonball.yaml
```
To run an example with `kata-clh`:
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-clh.yaml
```
To run an example with `kata-fc`:
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-fc.yaml
```
To run an example with `kata-stratovirt`:
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-stratovirt.yaml
```
To run an example with `kata-qemu`:
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-qemu.yaml
```
The following removes the test pods:
```bash
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-dragonball.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-clh.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-fc.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-stratovirt.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/examples/test-deploy-kata-qemu.yaml
```
### Remove Kata from the Kubernetes cluster
#### Removing the latest image
```sh
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
$ kubectl -n kube-system wait --timeout=10m --for=delete -l name=kata-deploy pod
```
After ensuring kata-deploy has been deleted, cleanup the cluster:
```sh
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup.yaml
```
The cleanup daemon-set will run a single time, cleaning up the node-label, which makes it difficult to check in an automated fashion.
This process should take, at most, 5 minutes.
After that, let's delete the cleanup daemon-set, the added RBAC and runtime classes:
```sh
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
```
#### Removing the stable image
```bash
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml
$ kubectl -n kube-system wait --timeout=10m --for=delete -l name=kata-deploy pod
```
After ensuring kata-deploy has been deleted, cleanup the cluster:
```bash
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
```
The cleanup daemon-set will run a single time, cleaning up the node-label, which makes it difficult to check in an automated fashion.
This process should take, at most, 5 minutes.
After that, let's delete the cleanup daemon-set, the added RBAC and runtime classes:
```bash
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
$ kubectl delete -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
```
## `kata-deploy` details
### Dockerfile
The [Dockerfile](Dockerfile) used to create the container image deployed in the DaemonSet is provided here.
This image contains all the necessary artifacts for running Kata Containers, all of which are pulled
from the [Kata Containers release page](https://github.com/kata-containers/kata-containers/releases).
Host artifacts:
* `cloud-hypervisor`, `firecracker`, `qemu`, `stratovirt` and supporting binaries
* `containerd-shim-kata-v2` (go runtime and rust runtime)
* `kata-collect-data.sh`
* `kata-runtime`
Virtual Machine artifacts:
* `kata-containers.img` and `kata-containers-initrd.img`: pulled from Kata GitHub releases page
* `vmlinuz.container` and `vmlinuz-virtiofs.container`: pulled from Kata GitHub releases page
### DaemonSets and RBAC
Two DaemonSets are introduced for `kata-deploy`, as well as an RBAC to facilitate
applying labels to the nodes.
#### Kata deploy
This DaemonSet installs the necessary Kata binaries, configuration files, and virtual machine artifacts on
the node. Once installed, the DaemonSet adds a node label `katacontainers.io/kata-runtime=true` and reconfigures
either CRI-O or containerd to register three `runtimeClasses`: `kata-clh` (for Cloud Hypervisor isolation), `kata-qemu` (for QEMU isolation),
`kata-fc` (for Firecracker isolation) and `kata-stratovirt` (for StratoVirt isolation).
As a final step the DaemonSet restarts either CRI-O or containerd. Upon deletion,
the DaemonSet removes the Kata binaries and VM artifacts and updates the node label to `katacontainers.io/kata-runtime=cleanup`.
#### Kata cleanup
This DaemonSet runs of the node has the label `katacontainers.io/kata-runtime=cleanup`. These DaemonSets removes
the `katacontainers.io/kata-runtime` label as well as restarts either CRI-O or `containerd` `systemctl`
daemon. You cannot execute these resets during the `preStopHook` of the Kata installer DaemonSet,
which necessitated this final cleanup DaemonSet.

View File

@@ -1,35 +0,0 @@
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kubelet-kata-cleanup
namespace: kube-system
spec:
selector:
matchLabels:
name: kubelet-kata-cleanup
template:
metadata:
labels:
name: kubelet-kata-cleanup
spec:
serviceAccountName: kata-deploy-sa
hostPID: true
nodeSelector:
katacontainers.io/kata-runtime: cleanup
containers:
- name: kube-kata-cleanup
image: quay.io/kata-containers/kata-deploy:latest
imagePullPolicy: Always
command: ["bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh reset"]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
privileged: true
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kata-cleanup.yaml

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_k0s_conf.yaml

View File

@@ -1,17 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kubelet-kata-cleanup
namespace: kube-system
spec:
template:
spec:
containers:
- name: kube-kata-cleanup
volumeMounts:
- name: containerd-conf
mountPath: /etc/containerd/
volumes:
- name: containerd-conf
hostPath:
path: /etc/k0s/

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_k3s_conf.yaml

View File

@@ -1,17 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kubelet-kata-cleanup
namespace: kube-system
spec:
template:
spec:
containers:
- name: kube-kata-cleanup
volumeMounts:
- name: containerd-conf
mountPath: /etc/containerd/
volumes:
- name: containerd-conf
hostPath:
path: /var/lib/rancher/k3s/agent/etc/containerd/

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_microk8s_conf.yaml

View File

@@ -1,17 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kubelet-kata-cleanup
namespace: kube-system
spec:
template:
spec:
containers:
- name: kube-kata-cleanup
volumeMounts:
- name: containerd-conf
mountPath: /etc/containerd/
volumes:
- name: containerd-conf
hostPath:
path: /var/snap/microk8s/current/args/

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_rke2_conf.yaml

View File

@@ -1,17 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kubelet-kata-cleanup
namespace: kube-system
spec:
template:
spec:
containers:
- name: kube-kata-cleanup
volumeMounts:
- name: containerd-conf
mountPath: /etc/containerd/
volumes:
- name: containerd-conf
hostPath:
path: /var/lib/rancher/rke2/agent/etc/containerd/

View File

@@ -1,79 +0,0 @@
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kata-deploy
namespace: kube-system
spec:
selector:
matchLabels:
name: kata-deploy
template:
metadata:
labels:
name: kata-deploy
spec:
serviceAccountName: kata-deploy-sa
hostPID: true
containers:
- name: kube-kata
image: quay.io/kata-containers/kata-deploy:latest
imagePullPolicy: Always
lifecycle:
preStop:
exec:
command: ["bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh cleanup"]
command: ["bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh install"]
# NOTE: Please don't change the order of the environment variables below.
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: DEBUG
value: "false"
- name: SHIMS
value: "clh cloud-hypervisor dragonball fc qemu qemu-coco-dev qemu-runtime-rs qemu-se-runtime-rs qemu-snp qemu-tdx stratovirt qemu-nvidia-gpu qemu-nvidia-gpu-snp qemu-nvidia-gpu-tdx"
- name: DEFAULT_SHIM
value: "qemu"
- name: CREATE_RUNTIMECLASSES
value: "true"
- name: CREATE_DEFAULT_RUNTIMECLASS
value: "false"
- name: ALLOWED_HYPERVISOR_ANNOTATIONS
value: ""
- name: SNAPSHOTTER_HANDLER_MAPPING
value: ""
- name: AGENT_HTTPS_PROXY
value: ""
- name: AGENT_NO_PROXY
value: ""
- name: PULL_TYPE_MAPPING
value: ""
- name: INSTALLATION_PREFIX
value: ""
- name: MULTI_INSTALL_SUFFIX
value: ""
securityContext:
privileged: true
volumeMounts:
- name: crio-conf
mountPath: /etc/crio/
- name: containerd-conf
mountPath: /etc/containerd/
- name: host
mountPath: /host/
volumes:
- name: crio-conf
hostPath:
path: /etc/crio/
- name: containerd-conf
hostPath:
path: /etc/containerd/
- name: host
hostPath:
path: /
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kata-deploy.yaml

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_k0s_conf.yaml

View File

@@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kata-deploy
namespace: kube-system
spec:
template:
spec:
volumes:
- name: containerd-conf
hostPath:
path: /etc/k0s/

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_k3s_conf.yaml

View File

@@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kata-deploy
namespace: kube-system
spec:
template:
spec:
volumes:
- name: containerd-conf
hostPath:
path: /var/lib/rancher/k3s/agent/etc/containerd/

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_microk8s_conf.yaml

View File

@@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kata-deploy
namespace: kube-system
spec:
template:
spec:
volumes:
- name: containerd-conf
hostPath:
path: /var/snap/microk8s/current/args/

View File

@@ -1,7 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- path: mount_rke2_conf.yaml

View File

@@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kata-deploy
namespace: kube-system
spec:
template:
spec:
volumes:
- name: containerd-conf
hostPath:
path: /var/lib/rancher/rke2/agent/etc/containerd/

View File

@@ -1,34 +0,0 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kata-deploy-sa
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kata-deploy-role
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "patch"]
- apiGroups: ["node.k8s.io"]
resources: ["runtimeclasses"]
verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
- apiGroups: ["apps"]
resources: ["daemonsets"]
verbs: ["list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kata-deploy-rb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kata-deploy-role
subjects:
- kind: ServiceAccount
name: kata-deploy-sa
namespace: kube-system

View File

@@ -1,4 +0,0 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kata-rbac.yaml