diff --git a/src/tools/csi-kata-directvolume/README.md b/src/tools/csi-kata-directvolume/README.md new file mode 100644 index 0000000000..7c9eccb040 --- /dev/null +++ b/src/tools/csi-kata-directvolume/README.md @@ -0,0 +1,79 @@ +# CSI Direct Volume Driver + +The `Direct Volume CSI driver` is heavily inspired by the [`K8s CSI HostPath driver`](https://github.com/kubernetes-csi/csi-driver-host-path). It aims to provide a production-ready implementation and a reference implementation for Kubernetes to connect to `Direct Volume`. + +This repository houses the `Direct Volume CSI driver`, along with all build and dependent configuration files needed for deployment. + +*WARNING* It is important to note that it is still under development. + +## Pre-requisite + +- K8S cluster +- Running version 1.20 or later +- Access to terminal with `kubectl` installed + +## Features + +The driver can provision volumes based on direct block devices, eliminating the need for loop devices and relying solely on single files stored on the host. + +## Deployment + +[Deployment for K8S 1.20+](docs/deploy-csi-kata-directvol.md) + +## Building the Binary + +If you want to build the driver yourself, you can do so with the following command from `csi-kata-directvolume` path: + +```shell +cd tools/csi-kata-directvolume/ && make +``` + +## Building the Container Image + +If you want to build the container image yourself, you can do so with the following command from a specified path. +Here, we just use `buildah/podman` as an example: + +```shell +$ tree -L 2 buildah-directv/ +buildah-directv/ +├── bin +│   └── directvolplugin +└── Dockerfile + +$ buildah bud -t kata-directvolume:v1.0.19 +STEP 1/7: FROM alpine +STEP 2/7: LABEL maintainers="Kata Containers Authors" +STEP 3/7: LABEL description="Kata DirectVolume Driver" +STEP 4/7: ARG binary=./bin/directvolplugin +STEP 5/7: RUN apk add util-linux coreutils e2fsprogs xfsprogs xfsprogs-extra btrfs-progs && apk update && apk upgrade +fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz +fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz +(1/66) Installing libblkid (2.39.3-r0) +... +(66/66) Installing xfsprogs-extra (6.5.0-r0) +Executing busybox-1.36.1-r15.trigger +OK: 64 MiB in 81 packages +fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz +fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz +v3.19.0-19-ga0ddaee500e [https://dl-cdn.alpinelinux.org/alpine/v3.19/main] +v3.19.0-18-gec62a609516 [https://dl-cdn.alpinelinux.org/alpine/v3.19/community] +OK: 22983 distinct packages available +OK: 64 MiB in 81 packages +STEP 6/7: COPY ${binary} /kata-directvol-plugin +STEP 7/7: ENTRYPOINT ["/kata-directvol-plugin"] +COMMIT kata-directvolume:v1.0.19 +Getting image source signatures +Copying blob 5af4f8f59b76 skipped: already exists +Copying blob a55645705de3 done +Copying config 244001cc51 done +Writing manifest to image destination +Storing signatures +--> 244001cc51d +Successfully tagged localhost/kata-directvolume:v1.0.19 +244001cc51d77302c4ed5e1a0ec347d12d85dec4576ea1313f700f66e2a7d36d +$ podman save localhost/kata-directvolume:v1.0.19 -o kata-directvolume-v1.0.19.tar +$ ctr -n k8s.io image import kata-directvolume-v1.0.19.tar +unpacking localhost/kata-directvolume:v1.0.19 (sha256:1bdc33ff7f9cee92e74cbf77a9d79d00dce6dbb9ba19b9811f683e1a087f8fbf)...done +$ crictl images |grep 1.0.19 +localhost/kata-directvolume v1.0.19 244001cc51d77 83.8MB +``` diff --git a/src/tools/csi-kata-directvolume/docs/deploy-csi-kata-directvol.md b/src/tools/csi-kata-directvolume/docs/deploy-csi-kata-directvol.md new file mode 100644 index 0000000000..2baaac9802 --- /dev/null +++ b/src/tools/csi-kata-directvolume/docs/deploy-csi-kata-directvol.md @@ -0,0 +1,169 @@ +# Deploy Kata Direct Volume CSI and Do Validation + +## How to Deploy Kata Direct Volume CSI + +First, you need to make sure you have a healthy Kubernetes(1.20+) cluster and have the permissions to create Kata pods. + +*WARNING* If you select a `K8S` with lower version, It cannot ensure that it will work well. + +The `CSI driver` is deployed as a `daemonset` and the pods of the `daemonset` contain 4 containers: + +1. `Kata Direct Volume CSI Driver`, which is the key implementation in it +2. [CSI-External-Provisioner](https://github.com/kubernetes-csi/external-provisioner) +3. [CSI-Liveness-Probe](https://github.com/kubernetes-csi/livenessprobe) +4. [CSI-Node-Driver-Registrar](https://github.com/kubernetes-csi/node-driver-registrar) + +The easiest way to deploy the `Direct Volume CSI driver` is to run the `deploy.sh` script for the Kubernetes version used by +the cluster as shown below for Kubernetes 1.28.2. + +```shell +sudo deploy/deploy.sh +``` + +You'll get an output similar to the following, indicating the application of `RBAC rules` and the successful deployment of `csi-provisioner`, `node-driver-registrar`, `kata directvolume csi driver`(`csi-kata-directvol-plugin`), liveness-probe. Please note that the following output is specific to Kubernetes 1.28.2. + +```shell +Creating Namespace kata-directvolume ... +kubectl apply -f /tmp/tmp.kN43BWUGQ5/kata-directvol-ns.yaml +namespace/kata-directvolume created +Namespace kata-directvolume created Done ! +Applying RBAC rules ... +curl https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v3.6.0/deploy/kubernetes/rbac.yaml --output /tmp/tmp.kN43BWUGQ5/rbac.yaml --silent --location +kubectl apply -f ./kata-directvolume/kata-directvol-rbac.yaml +serviceaccount/csi-provisioner created +clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created +clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created +role.rbac.authorization.k8s.io/external-provisioner-cfg created +rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created + +$ ./directvol-deploy.sh +deploying kata directvolume components + ./csi-directvol-driverinfo.yaml +csidriver.storage.k8s.io/directvolume.csi.katacontainers.io created + ./csi-directvol-plugin.yaml + using image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0 + using image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0 + using image: localhost/kata-directvolume:v1.0.18 + using image: registry.k8s.io/sig-storage/livenessprobe:v2.8.0 +daemonset.apps/csi-kata-directvol-plugin created +NAMESPACE NAME READY STATUS RESTARTS AGE +default pod/csi-kata-directvol-plugin-dlphw 4/4 Running 0 3s +kube-flannel pod/kube-flannel-ds-vq796 1/1 Running 1 (52d ago) 52d +kube-system pod/coredns-66f779496c-9bmp2 1/1 Running 3 (52d ago) 52d +kube-system pod/coredns-66f779496c-qlq6d 1/1 Running 1 (52d ago) 52d +kube-system pod/etcd-node001 1/1 Running 19 (52d ago) 52d +kube-system pod/kube-apiserver-node001 1/1 Running 5 (52d ago) 52d +kube-system pod/kube-controller-manager-node001 1/1 Running 8 (52d ago) 52d +kube-system pod/kube-proxy-p9t6t 1/1 Running 6 (52d ago) 52d +kube-system pod/kube-scheduler-node001 1/1 Running 8 (52d ago) 52d + +NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE +default daemonset.apps/csi-kata-directvol-plugin 1 1 1 1 1 3s +kube-flannel daemonset.apps/kube-flannel-ds 1 1 1 1 1 52d +kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 52d +``` + + +## How to Run a Kata Pod and Validate it + + +First, ensure all expected pods are running properly, including `csi-provisioner`, `node-driver-registrar`, `kata-directvolume` `csi driver(csi-kata-directvol-plugin)`, liveness-probe: + +```shell +$ kubectl get po -A +NAMESPACE NAME READY STATUS RESTARTS AGE +default csi-kata-directvol-plugin-dlphw 4/4 Running 0 68m +kube-flannel kube-flannel-ds-vq796 1/1 Running 1 (52d ago) 52d +kube-system coredns-66f779496c-9bmp2 1/1 Running 3 (52d ago) 52d +kube-system coredns-66f779496c-qlq6d 1/1 Running 1 (52d ago) 52d +kube-system etcd-node001 1/1 Running 19 (52d ago) 52d +kube-system kube-apiserver-node001 1/1 Running 5 (52d ago) 52d +kube-system kube-controller-manager-node001 1/1 Running 8 (52d ago) 52d +kube-system kube-proxy-p9t6t 1/1 Running 6 (52d ago) 52d +kube-system kube-scheduler-node001 1/1 Running 8 (52d ago) 52d +``` + +From the root directory, deploy the application pods including a storage class, a `PVC`, and a pod which uses direct block device based volume. The details can be seen in `/examples/pod-with-directvol/*.yaml`: + +```shell +kubectl apply -f ${BASE_DIR}/csi-storageclass.yaml +kubectl apply -f ${BASE_DIR}/csi-pvc.yaml +kubectl apply -f ${BASE_DIR}/csi-app.yaml +``` + +Let's validate the components are deployed: + +```shell +$ kubectl get po -A +NAMESPACE NAME READY STATUS RESTARTS AGE +kata-directvolume csi-kata-directvol-plugin-dlphw 4/4 Running 0 68m +default kata-driectvol-01 1/1 Running 0 67m + +$ kubectl get sc,pvc -A +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +storageclass.storage.k8s.io/csi-kata-directvolume-sc directvolume.csi.katacontainers.io Delete Immediate false 71m + +NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +default persistentvolumeclaim/csi-directvolume-pvc Bound pvc-d7644547-f850-4bdf-8c93-aa745c7f31b5 1Gi RWO csi-kata-directvolume-sc 71m + +``` + +Finally, inspect the application pod `kata-driectvol-01` which running with direct block device based volume: + +```shell +$ kubectl describe po kata-driectvol-01 +Name: kata-driectvol-01 +Namespace: kata-directvolume +Priority: 0 +Runtime Class Name: kata +Service Account: default +Node: node001/10.10.1.19 +Start Time: Sat, 09 Dec 2023 23:06:49 +0800 +Labels: +Annotations: +Status: Running +IP: 10.244.0.232 +IPs: + IP: 10.244.0.232 +Containers: + first-container: + Container ID: containerd://c5eec9d645a67b982549321f382d83c56297d9a2a705857e8f3eaa6c6676908e + Image: ubuntu:22.04 + Image ID: docker.io/library/ubuntu@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f + Port: + Host Port: + Command: + sleep + 1000000 + State: Running + Started: Sat, 09 Dec 2023 23:06:51 +0800 + Ready: True + Restart Count: 0 + Environment: + Mounts: + /data from kata-driectvol0-volume (rw) + /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zs9tm (ro) +Conditions: + Type Status + Initialized True + Ready True + ContainersReady True + PodScheduled True +Volumes: + kata-driectvol0-volume: + Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) + ClaimName: csi-directvolume-pvc + ReadOnly: false + kube-api-access-zs9tm: + Type: Projected (a volume that contains injected data from multiple sources) + TokenExpirationSeconds: 3607 + ConfigMapName: kube-root-ca.crt + ConfigMapOptional: + DownwardAPI: true +QoS Class: BestEffort +Node-Selectors: +Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s + node.kubernetes.io/unreachable:NoExecute op=Exists for 300s +Events: + +```