github/kata-containers
1
0
Fork 2
mirror of https://github.com/kata-containers/kata-containers.git synced 2026-08-08 15:44:38 +00:00
Code Issues Projects Releases Wiki Activity

docs: Update SPDK vhost-user guide with CSI driver

Browse Source 
- Add support for runtime-rs with Dragonball
- Add CSI driver integration method for Kubernetes
- Add kata-ctl direct-volume method for manual setup
- Preserve SPDK vhost-user Target Overview principles
- Fix minor typo (can exposes -> can expose)

Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
This commit is contained in:
Alex Lyn
2026-03-23 14:55:04 +08:00
parent 0c80372cf5
commit d6308ffb8c
1 changed files with 134 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files

282
docs/use-cases/using-SPDK-vhostuser-and-kata.md
View File

@@ -1,19 +1,15 @@
# Setup to run SPDK vhost-user devices with Kata Containers
> **Note:** This guide only applies to QEMU, since the vhost-user storage
> device is only available for QEMU now. The enablement work on other
> hypervisors is still ongoing.
> **Note:** This guide applies to both **runtime-rs with Dragonball** and **QEMU** hypervisors. For runtime-rs, the procedure is simplified as there is no need to manually create device nodes.
## SPDK vhost-user Target Overview
The Storage Performance Development Kit (SPDK) provides a set of tools and
libraries for writing high performance, scalable, user-mode storage applications.
The Storage Performance Development Kit (SPDK) provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications.
virtio, vhost and vhost-user:
- virtio is an efficient way to transport data for virtual environments and
guests. It is most commonly used in QEMU VMs, where the VM itself exposes a
virtual PCI device and the guest OS communicates with it using a specific virtio
PCI driver. Its diagram is:
- virtio is an efficient way to transport data for virtual environments and guests. It is most commonly used in QEMU VMs, where the VM itself exposes a virtual PCI device and the guest OS communicates with it using a specific virtio PCI driver. Its diagram is:
```
+---------+------+--------+----------+--+
| +------+-------------------+ |
@@ -42,6 +38,7 @@ uses the same virtio queue layout as virtio to allow vhost devices to be mapped
directly to virtio devices. The initial vhost implementation is a part of the
Linux kernel and uses an ioctl interface to communicate with userspace
applications. Its diagram is:
```
+---------+------+--------+----------+--+
| +------+-------------------+ |
@@ -65,9 +62,8 @@ applications. Its diagram is:
+---------------------------------------+
```
- vhost-user implements the control plane through Unix domain socket to establish
virtio queue sharing with a user space process on the same host. SPDK exposes
vhost devices via the vhost-user protocol. Its diagram is:
- vhost-user implements the control plane through Unix domain socket to establish virtio queue sharing with a user space process on the same host. SPDK exposes vhost devices via the vhost-user protocol. Its diagram is:
```
+----------------+------+--+----------+-+
| +------+-------------+ |
@@ -95,169 +91,159 @@ vhost devices via the vhost-user protocol. Its diagram is:
+---------------------------------------+
```
SPDK vhost is a vhost-user slave server. It exposes Unix domain sockets and
allows external applications to connect. It is capable of exposing virtualized
storage devices to QEMU instances or other arbitrary processes.
SPDK vhost is a vhost-user slave server. It exposes Unix domain sockets and allows external applications to connect. It is capable of exposing virtualized storage devices to QEMU instances or other arbitrary processes.
Currently, the SPDK vhost-user target can exposes these types of virtualized
devices:
Currently, the SPDK vhost-user target can expose several types of virtualized devices, but the most commonly used one in Kata Containers is the block device, which is supported by both runtime-rs with Dragonball and QEMU hypervisors:
- `vhost-user-blk`
- `vhost-user-scsi`
- `vhost-user-nvme` (deprecated from SPDK 21.07 release)
A block device that can be used as a regular block device in the guest. It is suitable for workloads that require high performance and low latency, such as databases or high I/O applications.
For more information, visit [SPDK](https://spdk.io) and [SPDK vhost-user target](https://spdk.io/doc/vhost.html).
## Install and setup SPDK vhost-user target
## Prerequisites
### Get source code and build SPDK
- A Kubernetes cluster with Kata Containers enabled (runtime-rs with Dragonball or QEMU)
- SPDK built and `spdk_tgt` available
- For Kubernetes CSI integration: `csi-kata-directvolume` deployed
Following the SPDK [getting started guide](https://spdk.io/doc/getting_started.html).
## Method 1: Using CSI Driver (Recommended for Kubernetes)
### Run SPDK vhost-user target
This is the recommended method for Kubernetes environments, leveraging the `csi-kata-directvolume` CSI driver.
First, run the SPDK `setup.sh` script to setup some hugepages for the SPDK vhost
target application. We recommend you use a minimum of 4GiB, enough for the SPDK
vhost target and the virtual machine.
This will allocate 4096MiB (4GiB) of hugepages, and avoid binding PCI devices:
### 1. Start SPDK Service
```sh
export SPDK_DEVEL=<path-to-your-spdk>
export VHU_UDS_PATH=/var/lib/spdk/vhost
# Reset and allocate hugepages
$ cd $SPDK_DEVEL
$ sudo ./scripts/setup.sh reset
$ sudo sysctl -w vm.nr_hugepages=2048
$ sudo HUGEMEM=4096 ./scripts/setup.sh
# Start SPDK vhost target
$ sudo mkdir -p $VHU_UDS_PATH
$ sudo $SPDK_DEVEL/build/bin/spdk_tgt -S $VHU_UDS_PATH -s 1024 -m 0x3 &
```
> **Notes:**
> - `-s 1024`: size of the hugepage memory pool in MB.
> - `-m 0x3`: CPU mask specifying which cores SPDK will use.
> - If `vfio-pci` driver is supported, use `DRIVER_OVERRIDE=vfio-pci` with setup.sh.
### 2. Deploy CSI Driver and Kubernetes Resources
Deploy the CSI driver following the [deployment guide](../../src/tools/csi-kata-directvolume/docs/deploy-csi-kata-directvol.md).
Create StorageClass, PVC, and Pod:
```sh
$ cd kata-containers/src/tools/csi-kata-directvolume/examples/pod-with-spdkvol
$ kubectl apply -f csi-storageclass.yaml
$ kubectl apply -f csi-pvc.yaml
$ kubectl apply -f csi-app.yaml
```
This creates:
- Storage Class `spdk-test-adapted` with `volumetype=spdkvol`
- PVC `kata-spdk-directvolume-pvc`
- Pod `spdk-pod-test`
### 3. Verify the Volume
Check the mounted block device inside the pod:
```sh
$ kubectl exec -it spdk-pod-test -- /bin/sh
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 254:0 0 256M 1 disk
└─vda1 254:1 0 253M 1 part
vdb 254:16 0 2G 0 disk /data
$ echo "hello spdk" > /data/test.txt
$ cat /data/test.txt
hello spdk
```
The SPDK-backed volume `/dev/vdb` is mounted to `/data` inside the container.
### 4. Cleanup
```sh
$ kubectl delete -f csi-app.yaml
$ kubectl delete -f csi-pvc.yaml
$ kubectl delete -f csi-storageclass.yaml
```
## Method 2: Using kata-ctl direct-volume (For Manual Setup)
This method is suitable for manual testing or non-Kubernetes environments using containerd.
### 1. Start SPDK vhost target and Create Block Device
```bash
$ sudo HUGEMEM=4096 PCI_WHITELIST="none" scripts/setup.sh
$ export SPDK_DEVEL=<path-to-your-spdk>
$ export VHU_UDS_PATH=/tmp/vhu-targets
$ export RAW_DISKS=<your-rawdisk-path> # e.g., export RAW_DISKS=/tmp/rawdisks
# Reset and setup hugepages
$ sudo ${SPDK_DEVEL}/scripts/setup.sh reset
$ sudo sysctl -w vm.nr_hugepages=2048
$ sudo HUGEMEM=4096 DRIVER_OVERRIDE=vfio-pci ${SPDK_DEVEL}/scripts/setup.sh
# Start SPDK vhost target
$ sudo ${SPDK_DEVEL}/build/bin/spdk_tgt -S $VHU_UDS_PATH -s 1024 -m 0x3 &
```
Then, take directory `/var/run/kata-containers/vhost-user` as Kata's vhost-user
device directory. Make subdirectories for vhost-user sockets and device nodes:
Create a vhost controller:
```bash
$ sudo mkdir -p /var/run/kata-containers/vhost-user/
$ sudo mkdir -p /var/run/kata-containers/vhost-user/block/
$ sudo mkdir -p /var/run/kata-containers/vhost-user/block/sockets/
$ sudo mkdir -p /var/run/kata-containers/vhost-user/block/devices/
# Create raw disk
$ mkdir -p "${RAW_DISKS}" # ensure the directory exists
$ sudo dd if=/dev/zero of=${RAW_DISKS}/rawdisk01.20g bs=1M count=20480
# Create AIO bdev
$ sudo ${SPDK_DEVEL}/scripts/rpc.py bdev_aio_create ${RAW_DISKS}/rawdisk01.20g vhu-rawdisk01.20g 512
# Create vhost-user-blk controller
$ sudo ${SPDK_DEVEL}/scripts/rpc.py vhost_create_blk_controller vhost-blk-rawdisk01.sock vhu-rawdisk01.20g
```
For more details, see section [Host setup for vhost-user devices](#host-setup-for-vhost-user-devices).
A vhost controller `vhost-blk-rawdisk01.sock` is created under `$VHU_UDS_PATH/`.
Next, start the SPDK vhost target application. The following command will start
vhost on the first CPU core with all future socket files placed in
`/var/run/kata-containers/vhost-user/block/sockets/`:
### 2. Configure Direct Volume with kata-ctl
For runtime-rs with Dragonball, there is no need to manually create device nodes. Use `kata-ctl direct-volume add`:
```bash
$ sudo app/spdk_tgt/spdk_tgt -S /var/run/kata-containers/vhost-user/block/sockets/ &
# Add direct volume
$ sudo kata-ctl direct-volume add /kubelet/kata-test-vol-001/volume001 "{\"device\": \"${VHU_UDS_PATH}/vhost-blk-rawdisk01.sock\", \"volume_type\":\"spdkvol\", \"fs_type\": \"ext4\", \"metadata\":{}, \"options\": []}"
```
To list all available vhost options run the following command:
The volume info is stored at `/run/kata-containers/shared/direct-volumes/` with encoded path.
### 3. Run a Kata Container
```bash
$ app/spdk_tgt/spdk_tgt -h
# For runtime-rs with Dragonball
# IMAGE=docker.io/library/ubuntu:latest
$ sudo ctr run -t --rm --runtime io.containerd.kata.v2 \
--mount type=spdkvol,src=/kubelet/kata-test-vol-001/volume001,dst=/disk001,options=rbind:rw \
"$IMAGE" kata-spdk-vol-test /bin/bash
```
Create an experimental `vhost-user-blk` device based on memory directly:
Inside the container, the SPDK volume will be available at `/disk001`.
- The following RPC will create a 64MB memory block device named `Malloc0`
with 4096-byte block size:
## Additional Resources
```bash
$ sudo scripts/rpc.py bdev_malloc_create 64 4096 -b Malloc0
```
- [How to run Kata Containers with Kinds of Block Volumes](../how-to/how-to-run-kata-containers-with-kinds-of-Block-Volumes.md)
- [CSI Direct Volume Driver README](../../src/tools/csi-kata-directvolume/README.md)
- [SPDK Usage Guide for CSI](../../src/tools/csi-kata-directvolume/docs/spdk-usage.md)
- [Direct Block Device Assignment Design](../design/direct-blk-device-assignment.md)
- The following RPC will create a `vhost-user-blk` device exposing `Malloc0`
block device. The device will be accessible via
`/var/run/kata-containers/vhost-user/block/sockets/vhostblk0`:
```bash
$ sudo scripts/rpc.py vhost_create_blk_controller vhostblk0 Malloc0
```
## Host setup for vhost-user devices
Considering the OCI specification and characteristics of vhost-user device,
Kata has chosen to use Linux reserved the block major range `240-254`
to map each vhost-user block type to a major. Also a specific directory is
used for vhost-user devices.
The base directory for vhost-user device is a configurable value,
with the default being `/var/run/kata-containers/vhost-user`. It can be
configured by parameter `vhost_user_store_path` in [Kata TOML configuration file](../../src/runtime/README.md#configuration).
Currently, the vhost-user storage device is not enabled by default, so
the user should enable it explicitly inside the Kata TOML configuration
file by setting `enable_vhost_user_store = true`. Since SPDK vhost-user target
requires hugepages, hugepages should also be enabled inside the Kata TOML
configuration file by setting `enable_hugepages = true`.
Here is the conclusion of parameter setting for vhost-user storage device:
```toml
enable_hugepages = true
enable_vhost_user_store = true
vhost_user_store_path = "<Path of the base directory for vhost-user device>"
```
> **Note:** These parameters are under `[hypervisor.qemu]` section in Kata
> TOML configuration file. If they are absent, users should still add them
> under `[hypervisor.qemu]` section.
For the subdirectories of `vhost_user_store_path`:
- `block` is used for block device;
- `block/sockets` is where we expect UNIX domain sockets for vhost-user
block devices to live;
- `block/devices` is where simulated block device nodes for vhost-user
block devices are created.
For example, if using the default directory `/var/run/kata-containers/vhost-user`,
UNIX domain sockets for vhost-user block device are under `/var/run/kata-containers/vhost-user/block/sockets/`.
Device nodes for vhost-user block device are under `/var/run/kata-containers/vhost-user/block/devices/`.
Currently, Kata has chosen major number 241 to map to `vhost-user-blk` devices.
For `vhost-user-blk` device named `vhostblk0`, a UNIX domain socket is already
created by SPDK vhost target, and a block device node with major `241` and
minor `0` should be created for it, in order to be recognized by Kata runtime:
```bash
$ sudo mknod /var/run/kata-containers/vhost-user/block/devices/vhostblk0 b 241 0
```
## Launch a Kata container with SPDK vhost-user block device
To use `vhost-user-blk` device, use `ctr` to pass a host `vhost-user-blk`
device to the container. In your `config.json`, you should use `devices`
to pass a host device to the container.
For example (only `vhost-user-blk` listed):
```json
{
"linux": {
"devices": [
{
"path": "/dev/vda",
"type": "b",
"major": 241,
"minor": 0,
"fileMode": 420,
"uid": 0,
"gid": 0
}
]
}
}
```
With `rootfs` provisioned under `bundle` directory, you can run your SPDK container:
```bash
$ sudo ctr run -d --runtime io.containerd.run.kata.v2 --config bundle/config.json spdk_container
```
Example of performing I/O operations on the `vhost-user-blk` device inside
container:
```
$ sudo ctr t exec --exec-id 1 -t spdk_container sh
/ # ls -l /dev/vda
brw-r--r-- 1 root root 254, 0 Jan 20 03:54 /dev/vda
/ # dd if=/dev/vda of=/tmp/ddtest bs=4k count=20
20+0 records in
20+0 records out
81920 bytes (80.0KB) copied, 0.002996 seconds, 26.1MB/s
```