github/kata-containers
1
0
Fork 1
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-07-21 10:51:05 +00:00
Code Issues Projects Releases Wiki Activity

runtime-rs: add README and user guide to deploy directvol CSI Driver

Browse Source 
Fixes: #8602

Signed-off-by: alex.lyn <alex.lyn@antgroup.com>
This commit is contained in:
alex.lyn 2023-12-13 20:21:35 +08:00
parent 1c42d94550
commit 3b317e69e2
2 changed files with 248 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

79
src/tools/csi-kata-directvolume/README.md Normal file
View File

@ -0,0 +1,79 @@
# CSI Direct Volume Driver
The `Direct Volume CSI driver` is heavily inspired by the [`K8s CSI HostPath driver`](https://github.com/kubernetes-csi/csi-driver-host-path). It aims to provide a production-ready implementation and a reference implementation for Kubernetes to connect to `Direct Volume`.
This repository houses the `Direct Volume CSI driver`, along with all build and dependent configuration files needed for deployment.
*WARNING* It is important to note that it is still under development.
## Pre-requisite
- K8S cluster
- Running version 1.20 or later
- Access to terminal with `kubectl` installed
## Features
The driver can provision volumes based on direct block devices, eliminating the need for loop devices and relying solely on single files stored on the host.
## Deployment
[Deployment for K8S 1.20+](docs/deploy-csi-kata-directvol.md)
## Building the Binary
If you want to build the driver yourself, you can do so with the following command from `csi-kata-directvolume` path:
```shell
cd tools/csi-kata-directvolume/ && make
```
## Building the Container Image
If you want to build the container image yourself, you can do so with the following command from a specified path.
Here, we just use `buildah/podman` as an example:
```shell
$ tree -L 2 buildah-directv/
buildah-directv/
├── bin
│   └── directvolplugin
└── Dockerfile
$ buildah bud -t kata-directvolume:v1.0.19
STEP 1/7: FROM alpine
STEP 2/7: LABEL maintainers="Kata Containers Authors"
STEP 3/7: LABEL description="Kata DirectVolume Driver"
STEP 4/7: ARG binary=./bin/directvolplugin
STEP 5/7: RUN apk add util-linux coreutils e2fsprogs xfsprogs xfsprogs-extra btrfs-progs && apk update && apk upgrade
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/66) Installing libblkid (2.39.3-r0)
...
(66/66) Installing xfsprogs-extra (6.5.0-r0)
Executing busybox-1.36.1-r15.trigger
OK: 64 MiB in 81 packages
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
v3.19.0-19-ga0ddaee500e [https://dl-cdn.alpinelinux.org/alpine/v3.19/main]
v3.19.0-18-gec62a609516 [https://dl-cdn.alpinelinux.org/alpine/v3.19/community]
OK: 22983 distinct packages available
OK: 64 MiB in 81 packages
STEP 6/7: COPY ${binary} /kata-directvol-plugin
STEP 7/7: ENTRYPOINT ["/kata-directvol-plugin"]
COMMIT kata-directvolume:v1.0.19
Getting image source signatures
Copying blob 5af4f8f59b76 skipped: already exists
Copying blob a55645705de3 done
Copying config 244001cc51 done
Writing manifest to image destination
Storing signatures
--> 244001cc51d
Successfully tagged localhost/kata-directvolume:v1.0.19
244001cc51d77302c4ed5e1a0ec347d12d85dec4576ea1313f700f66e2a7d36d
$ podman save localhost/kata-directvolume:v1.0.19 -o kata-directvolume-v1.0.19.tar
$ ctr -n k8s.io image import kata-directvolume-v1.0.19.tar
unpacking localhost/kata-directvolume:v1.0.19 (sha256:1bdc33ff7f9cee92e74cbf77a9d79d00dce6dbb9ba19b9811f683e1a087f8fbf)...done
$ crictl images |grep 1.0.19
localhost/kata-directvolume v1.0.19 244001cc51d77 83.8MB
```

169
src/tools/csi-kata-directvolume/docs/deploy-csi-kata-directvol.md Normal file
View File

@ -0,0 +1,169 @@
# Deploy Kata Direct Volume CSI and Do Validation
## How to Deploy Kata Direct Volume CSI
First, you need to make sure you have a healthy Kubernetes(1.20+) cluster and have the permissions to create Kata pods.
*WARNING* If you select a `K8S` with lower version, It cannot ensure that it will work well.
The `CSI driver` is deployed as a `daemonset` and the pods of the `daemonset` contain 4 containers:
1. `Kata Direct Volume CSI Driver`, which is the key implementation in it
2. [CSI-External-Provisioner](https://github.com/kubernetes-csi/external-provisioner)
3. [CSI-Liveness-Probe](https://github.com/kubernetes-csi/livenessprobe)
4. [CSI-Node-Driver-Registrar](https://github.com/kubernetes-csi/node-driver-registrar)
The easiest way to deploy the `Direct Volume CSI driver` is to run the `deploy.sh` script for the Kubernetes version used by
the cluster as shown below for Kubernetes 1.28.2.
```shell
sudo deploy/deploy.sh
```
You'll get an output similar to the following, indicating the application of `RBAC rules` and the successful deployment of `csi-provisioner`, `node-driver-registrar`, `kata directvolume csi driver`(`csi-kata-directvol-plugin`), liveness-probe. Please note that the following output is specific to Kubernetes 1.28.2.
```shell
Creating Namespace kata-directvolume ...
kubectl apply -f /tmp/tmp.kN43BWUGQ5/kata-directvol-ns.yaml
namespace/kata-directvolume created
Namespace kata-directvolume created Done !
Applying RBAC rules ...
curl https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v3.6.0/deploy/kubernetes/rbac.yaml --output /tmp/tmp.kN43BWUGQ5/rbac.yaml --silent --location
kubectl apply -f ./kata-directvolume/kata-directvol-rbac.yaml
serviceaccount/csi-provisioner created
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created
role.rbac.authorization.k8s.io/external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created
$ ./directvol-deploy.sh
deploying kata directvolume components
./csi-directvol-driverinfo.yaml
csidriver.storage.k8s.io/directvolume.csi.katacontainers.io created
./csi-directvol-plugin.yaml
using image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
using image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
using image: localhost/kata-directvolume:v1.0.18
using image: registry.k8s.io/sig-storage/livenessprobe:v2.8.0
daemonset.apps/csi-kata-directvol-plugin created
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/csi-kata-directvol-plugin-dlphw 4/4 Running 0 3s
kube-flannel pod/kube-flannel-ds-vq796 1/1 Running 1 (52d ago) 52d
kube-system pod/coredns-66f779496c-9bmp2 1/1 Running 3 (52d ago) 52d
kube-system pod/coredns-66f779496c-qlq6d 1/1 Running 1 (52d ago) 52d
kube-system pod/etcd-node001 1/1 Running 19 (52d ago) 52d
kube-system pod/kube-apiserver-node001 1/1 Running 5 (52d ago) 52d
kube-system pod/kube-controller-manager-node001 1/1 Running 8 (52d ago) 52d
kube-system pod/kube-proxy-p9t6t 1/1 Running 6 (52d ago) 52d
kube-system pod/kube-scheduler-node001 1/1 Running 8 (52d ago) 52d
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
default daemonset.apps/csi-kata-directvol-plugin 1 1 1 1 1 <none> 3s
kube-flannel daemonset.apps/kube-flannel-ds 1 1 1 1 1 <none> 52d
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 52d
```
## How to Run a Kata Pod and Validate it
First, ensure all expected pods are running properly, including `csi-provisioner`, `node-driver-registrar`, `kata-directvolume` `csi driver(csi-kata-directvol-plugin)`, liveness-probe:
```shell
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default csi-kata-directvol-plugin-dlphw 4/4 Running 0 68m
kube-flannel kube-flannel-ds-vq796 1/1 Running 1 (52d ago) 52d
kube-system coredns-66f779496c-9bmp2 1/1 Running 3 (52d ago) 52d
kube-system coredns-66f779496c-qlq6d 1/1 Running 1 (52d ago) 52d
kube-system etcd-node001 1/1 Running 19 (52d ago) 52d
kube-system kube-apiserver-node001 1/1 Running 5 (52d ago) 52d
kube-system kube-controller-manager-node001 1/1 Running 8 (52d ago) 52d
kube-system kube-proxy-p9t6t 1/1 Running 6 (52d ago) 52d
kube-system kube-scheduler-node001 1/1 Running 8 (52d ago) 52d
```
From the root directory, deploy the application pods including a storage class, a `PVC`, and a pod which uses direct block device based volume. The details can be seen in `/examples/pod-with-directvol/*.yaml`:
```shell
kubectl apply -f ${BASE_DIR}/csi-storageclass.yaml
kubectl apply -f ${BASE_DIR}/csi-pvc.yaml
kubectl apply -f ${BASE_DIR}/csi-app.yaml
```
Let's validate the components are deployed:
```shell
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kata-directvolume csi-kata-directvol-plugin-dlphw 4/4 Running 0 68m
default kata-driectvol-01 1/1 Running 0 67m
$ kubectl get sc,pvc -A
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/csi-kata-directvolume-sc directvolume.csi.katacontainers.io Delete Immediate false 71m
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
default persistentvolumeclaim/csi-directvolume-pvc Bound pvc-d7644547-f850-4bdf-8c93-aa745c7f31b5 1Gi RWO csi-kata-directvolume-sc 71m
```
Finally, inspect the application pod `kata-driectvol-01` which running with direct block device based volume:
```shell
$ kubectl describe po kata-driectvol-01
Name: kata-driectvol-01
Namespace: kata-directvolume
Priority: 0
Runtime Class Name: kata
Service Account: default
Node: node001/10.10.1.19
Start Time: Sat, 09 Dec 2023 23:06:49 +0800
Labels: <none>
Annotations: <none>
Status: Running
IP: 10.244.0.232
IPs:
IP: 10.244.0.232
Containers:
first-container:
Container ID: containerd://c5eec9d645a67b982549321f382d83c56297d9a2a705857e8f3eaa6c6676908e
Image: ubuntu:22.04
Image ID: docker.io/library/ubuntu@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f
Port: <none>
Host Port: <none>
Command:
sleep
1000000
State: Running
Started: Sat, 09 Dec 2023 23:06:51 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/data from kata-driectvol0-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zs9tm (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kata-driectvol0-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: csi-directvolume-pvc
ReadOnly: false
kube-api-access-zs9tm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
```