diff --git a/docs/use-cases/using-SPDK-vhostuser-and-kata.md b/docs/use-cases/using-SPDK-vhostuser-and-kata.md index ae75930aeb..da07b71196 100644 --- a/docs/use-cases/using-SPDK-vhostuser-and-kata.md +++ b/docs/use-cases/using-SPDK-vhostuser-and-kata.md @@ -1,19 +1,15 @@ # Setup to run SPDK vhost-user devices with Kata Containers -> **Note:** This guide only applies to QEMU, since the vhost-user storage -> device is only available for QEMU now. The enablement work on other -> hypervisors is still ongoing. +> **Note:** This guide applies to both **runtime-rs with Dragonball** and **QEMU** hypervisors. For runtime-rs, the procedure is simplified as there is no need to manually create device nodes. ## SPDK vhost-user Target Overview -The Storage Performance Development Kit (SPDK) provides a set of tools and -libraries for writing high performance, scalable, user-mode storage applications. +The Storage Performance Development Kit (SPDK) provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications. virtio, vhost and vhost-user: -- virtio is an efficient way to transport data for virtual environments and -guests. It is most commonly used in QEMU VMs, where the VM itself exposes a -virtual PCI device and the guest OS communicates with it using a specific virtio -PCI driver. Its diagram is: + +- virtio is an efficient way to transport data for virtual environments and guests. It is most commonly used in QEMU VMs, where the VM itself exposes a virtual PCI device and the guest OS communicates with it using a specific virtio PCI driver. Its diagram is: + ``` +---------+------+--------+----------+--+ | +------+-------------------+ | @@ -42,6 +38,7 @@ uses the same virtio queue layout as virtio to allow vhost devices to be mapped directly to virtio devices. The initial vhost implementation is a part of the Linux kernel and uses an ioctl interface to communicate with userspace applications. Its diagram is: + ``` +---------+------+--------+----------+--+ | +------+-------------------+ | @@ -65,9 +62,8 @@ applications. Its diagram is: +---------------------------------------+ ``` -- vhost-user implements the control plane through Unix domain socket to establish -virtio queue sharing with a user space process on the same host. SPDK exposes -vhost devices via the vhost-user protocol. Its diagram is: +- vhost-user implements the control plane through Unix domain socket to establish virtio queue sharing with a user space process on the same host. SPDK exposes vhost devices via the vhost-user protocol. Its diagram is: + ``` +----------------+------+--+----------+-+ | +------+-------------+ | @@ -95,169 +91,159 @@ vhost devices via the vhost-user protocol. Its diagram is: +---------------------------------------+ ``` -SPDK vhost is a vhost-user slave server. It exposes Unix domain sockets and -allows external applications to connect. It is capable of exposing virtualized -storage devices to QEMU instances or other arbitrary processes. +SPDK vhost is a vhost-user slave server. It exposes Unix domain sockets and allows external applications to connect. It is capable of exposing virtualized storage devices to QEMU instances or other arbitrary processes. -Currently, the SPDK vhost-user target can exposes these types of virtualized -devices: +Currently, the SPDK vhost-user target can expose several types of virtualized devices, but the most commonly used one in Kata Containers is the block device, which is supported by both runtime-rs with Dragonball and QEMU hypervisors: - `vhost-user-blk` -- `vhost-user-scsi` -- `vhost-user-nvme` (deprecated from SPDK 21.07 release) + +A block device that can be used as a regular block device in the guest. It is suitable for workloads that require high performance and low latency, such as databases or high I/O applications. For more information, visit [SPDK](https://spdk.io) and [SPDK vhost-user target](https://spdk.io/doc/vhost.html). -## Install and setup SPDK vhost-user target +## Prerequisites -### Get source code and build SPDK +- A Kubernetes cluster with Kata Containers enabled (runtime-rs with Dragonball or QEMU) +- SPDK built and `spdk_tgt` available +- For Kubernetes CSI integration: `csi-kata-directvolume` deployed -Following the SPDK [getting started guide](https://spdk.io/doc/getting_started.html). +## Method 1: Using CSI Driver (Recommended for Kubernetes) -### Run SPDK vhost-user target +This is the recommended method for Kubernetes environments, leveraging the `csi-kata-directvolume` CSI driver. -First, run the SPDK `setup.sh` script to setup some hugepages for the SPDK vhost -target application. We recommend you use a minimum of 4GiB, enough for the SPDK -vhost target and the virtual machine. -This will allocate 4096MiB (4GiB) of hugepages, and avoid binding PCI devices: +### 1. Start SPDK Service + +```sh +export SPDK_DEVEL= +export VHU_UDS_PATH=/var/lib/spdk/vhost + +# Reset and allocate hugepages +$ cd $SPDK_DEVEL +$ sudo ./scripts/setup.sh reset +$ sudo sysctl -w vm.nr_hugepages=2048 +$ sudo HUGEMEM=4096 ./scripts/setup.sh + +# Start SPDK vhost target +$ sudo mkdir -p $VHU_UDS_PATH +$ sudo $SPDK_DEVEL/build/bin/spdk_tgt -S $VHU_UDS_PATH -s 1024 -m 0x3 & +``` + +> **Notes:** + +> - `-s 1024`: size of the hugepage memory pool in MB. +> - `-m 0x3`: CPU mask specifying which cores SPDK will use. +> - If `vfio-pci` driver is supported, use `DRIVER_OVERRIDE=vfio-pci` with setup.sh. + +### 2. Deploy CSI Driver and Kubernetes Resources + +Deploy the CSI driver following the [deployment guide](../../src/tools/csi-kata-directvolume/docs/deploy-csi-kata-directvol.md). + +Create StorageClass, PVC, and Pod: + +```sh +$ cd kata-containers/src/tools/csi-kata-directvolume/examples/pod-with-spdkvol +$ kubectl apply -f csi-storageclass.yaml +$ kubectl apply -f csi-pvc.yaml +$ kubectl apply -f csi-app.yaml +``` + +This creates: + +- Storage Class `spdk-test-adapted` with `volumetype=spdkvol` +- PVC `kata-spdk-directvolume-pvc` +- Pod `spdk-pod-test` + +### 3. Verify the Volume + +Check the mounted block device inside the pod: + +```sh +$ kubectl exec -it spdk-pod-test -- /bin/sh + +$ lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +vda 254:0 0 256M 1 disk +└─vda1 254:1 0 253M 1 part +vdb 254:16 0 2G 0 disk /data + +$ echo "hello spdk" > /data/test.txt +$ cat /data/test.txt +hello spdk +``` + +The SPDK-backed volume `/dev/vdb` is mounted to `/data` inside the container. + +### 4. Cleanup + +```sh +$ kubectl delete -f csi-app.yaml +$ kubectl delete -f csi-pvc.yaml +$ kubectl delete -f csi-storageclass.yaml +``` + +## Method 2: Using kata-ctl direct-volume (For Manual Setup) + +This method is suitable for manual testing or non-Kubernetes environments using containerd. + +### 1. Start SPDK vhost target and Create Block Device ```bash -$ sudo HUGEMEM=4096 PCI_WHITELIST="none" scripts/setup.sh +$ export SPDK_DEVEL= +$ export VHU_UDS_PATH=/tmp/vhu-targets +$ export RAW_DISKS= # e.g., export RAW_DISKS=/tmp/rawdisks + +# Reset and setup hugepages +$ sudo ${SPDK_DEVEL}/scripts/setup.sh reset +$ sudo sysctl -w vm.nr_hugepages=2048 +$ sudo HUGEMEM=4096 DRIVER_OVERRIDE=vfio-pci ${SPDK_DEVEL}/scripts/setup.sh + +# Start SPDK vhost target +$ sudo ${SPDK_DEVEL}/build/bin/spdk_tgt -S $VHU_UDS_PATH -s 1024 -m 0x3 & ``` -Then, take directory `/var/run/kata-containers/vhost-user` as Kata's vhost-user -device directory. Make subdirectories for vhost-user sockets and device nodes: +Create a vhost controller: ```bash -$ sudo mkdir -p /var/run/kata-containers/vhost-user/ -$ sudo mkdir -p /var/run/kata-containers/vhost-user/block/ -$ sudo mkdir -p /var/run/kata-containers/vhost-user/block/sockets/ -$ sudo mkdir -p /var/run/kata-containers/vhost-user/block/devices/ +# Create raw disk +$ mkdir -p "${RAW_DISKS}" # ensure the directory exists +$ sudo dd if=/dev/zero of=${RAW_DISKS}/rawdisk01.20g bs=1M count=20480 + +# Create AIO bdev +$ sudo ${SPDK_DEVEL}/scripts/rpc.py bdev_aio_create ${RAW_DISKS}/rawdisk01.20g vhu-rawdisk01.20g 512 + +# Create vhost-user-blk controller +$ sudo ${SPDK_DEVEL}/scripts/rpc.py vhost_create_blk_controller vhost-blk-rawdisk01.sock vhu-rawdisk01.20g ``` -For more details, see section [Host setup for vhost-user devices](#host-setup-for-vhost-user-devices). +A vhost controller `vhost-blk-rawdisk01.sock` is created under `$VHU_UDS_PATH/`. -Next, start the SPDK vhost target application. The following command will start -vhost on the first CPU core with all future socket files placed in -`/var/run/kata-containers/vhost-user/block/sockets/`: +### 2. Configure Direct Volume with kata-ctl + +For runtime-rs with Dragonball, there is no need to manually create device nodes. Use `kata-ctl direct-volume add`: ```bash -$ sudo app/spdk_tgt/spdk_tgt -S /var/run/kata-containers/vhost-user/block/sockets/ & +# Add direct volume +$ sudo kata-ctl direct-volume add /kubelet/kata-test-vol-001/volume001 "{\"device\": \"${VHU_UDS_PATH}/vhost-blk-rawdisk01.sock\", \"volume_type\":\"spdkvol\", \"fs_type\": \"ext4\", \"metadata\":{}, \"options\": []}" ``` -To list all available vhost options run the following command: +The volume info is stored at `/run/kata-containers/shared/direct-volumes/` with encoded path. + +### 3. Run a Kata Container ```bash -$ app/spdk_tgt/spdk_tgt -h +# For runtime-rs with Dragonball +# IMAGE=docker.io/library/ubuntu:latest +$ sudo ctr run -t --rm --runtime io.containerd.kata.v2 \ + --mount type=spdkvol,src=/kubelet/kata-test-vol-001/volume001,dst=/disk001,options=rbind:rw \ + "$IMAGE" kata-spdk-vol-test /bin/bash ``` -Create an experimental `vhost-user-blk` device based on memory directly: +Inside the container, the SPDK volume will be available at `/disk001`. -- The following RPC will create a 64MB memory block device named `Malloc0` -with 4096-byte block size: +## Additional Resources -```bash -$ sudo scripts/rpc.py bdev_malloc_create 64 4096 -b Malloc0 -``` +- [How to run Kata Containers with Kinds of Block Volumes](../how-to/how-to-run-kata-containers-with-kinds-of-Block-Volumes.md) +- [CSI Direct Volume Driver README](../../src/tools/csi-kata-directvolume/README.md) +- [SPDK Usage Guide for CSI](../../src/tools/csi-kata-directvolume/docs/spdk-usage.md) +- [Direct Block Device Assignment Design](../design/direct-blk-device-assignment.md) -- The following RPC will create a `vhost-user-blk` device exposing `Malloc0` -block device. The device will be accessible via -`/var/run/kata-containers/vhost-user/block/sockets/vhostblk0`: - -```bash -$ sudo scripts/rpc.py vhost_create_blk_controller vhostblk0 Malloc0 -``` - -## Host setup for vhost-user devices - -Considering the OCI specification and characteristics of vhost-user device, -Kata has chosen to use Linux reserved the block major range `240-254` -to map each vhost-user block type to a major. Also a specific directory is -used for vhost-user devices. - -The base directory for vhost-user device is a configurable value, -with the default being `/var/run/kata-containers/vhost-user`. It can be -configured by parameter `vhost_user_store_path` in [Kata TOML configuration file](../../src/runtime/README.md#configuration). - -Currently, the vhost-user storage device is not enabled by default, so -the user should enable it explicitly inside the Kata TOML configuration -file by setting `enable_vhost_user_store = true`. Since SPDK vhost-user target -requires hugepages, hugepages should also be enabled inside the Kata TOML -configuration file by setting `enable_hugepages = true`. -Here is the conclusion of parameter setting for vhost-user storage device: - -```toml -enable_hugepages = true -enable_vhost_user_store = true -vhost_user_store_path = "" -``` - -> **Note:** These parameters are under `[hypervisor.qemu]` section in Kata -> TOML configuration file. If they are absent, users should still add them -> under `[hypervisor.qemu]` section. - - -For the subdirectories of `vhost_user_store_path`: -- `block` is used for block device; -- `block/sockets` is where we expect UNIX domain sockets for vhost-user -block devices to live; -- `block/devices` is where simulated block device nodes for vhost-user -block devices are created. - -For example, if using the default directory `/var/run/kata-containers/vhost-user`, -UNIX domain sockets for vhost-user block device are under `/var/run/kata-containers/vhost-user/block/sockets/`. -Device nodes for vhost-user block device are under `/var/run/kata-containers/vhost-user/block/devices/`. - -Currently, Kata has chosen major number 241 to map to `vhost-user-blk` devices. -For `vhost-user-blk` device named `vhostblk0`, a UNIX domain socket is already -created by SPDK vhost target, and a block device node with major `241` and -minor `0` should be created for it, in order to be recognized by Kata runtime: - -```bash -$ sudo mknod /var/run/kata-containers/vhost-user/block/devices/vhostblk0 b 241 0 -``` - -## Launch a Kata container with SPDK vhost-user block device - -To use `vhost-user-blk` device, use `ctr` to pass a host `vhost-user-blk` -device to the container. In your `config.json`, you should use `devices` -to pass a host device to the container. - -For example (only `vhost-user-blk` listed): - -```json -{ - "linux": { - "devices": [ - { - "path": "/dev/vda", - "type": "b", - "major": 241, - "minor": 0, - "fileMode": 420, - "uid": 0, - "gid": 0 - } - ] - } -} -``` - -With `rootfs` provisioned under `bundle` directory, you can run your SPDK container: - -```bash -$ sudo ctr run -d --runtime io.containerd.run.kata.v2 --config bundle/config.json spdk_container -``` - -Example of performing I/O operations on the `vhost-user-blk` device inside -container: - -``` -$ sudo ctr t exec --exec-id 1 -t spdk_container sh -/ # ls -l /dev/vda -brw-r--r-- 1 root root 254, 0 Jan 20 03:54 /dev/vda -/ # dd if=/dev/vda of=/tmp/ddtest bs=4k count=20 -20+0 records in -20+0 records out -81920 bytes (80.0KB) copied, 0.002996 seconds, 26.1MB/s -```