This patch updates the `iothread` option to specify the CPU affinity
of the iothread. Setting the iothread's CPU affinity could benefit the
Service VM's CPU utilization when Service VM owns limited dedicated CPUs.
It could be helpful to ensure the I/O mediator Quality of Service (QoS).
Once the performance tuning is done, the specific CPU affinity config could
pass to acrn-dm directly, letting the deployment more easily.
The format looks like below:
iothread=<num_iothread>@<cpu_affinity>
"@" is used to separate the following two settings:
- the number of iothread instances
- the CPU affinity settings for each iothread instance.
The format of `cpu_affinity` looks like below:
<cpu_affinity_0>/<cpu_affinity_1>/<cpu_affinity_2>/...
1. "/" is used to separate the CPU affinity setting for each iothread instance
(sequentially).
2. char '*' can be used to skip the setting for the specific iothread instance.
3. the number of cpu_affinity_x vs. the number of iothread instances
- If # of cpu_affinity_x is less than # of iothread instances,
no CPU affinity settings for the last few iothread instances.
- If # of cpu_affinity_x is more than # of iothread instances,
the extra cpu_affinity_x are discarded.
4. ":" is used to separate different CPU cores for each CPU affinity setting.
Examples to specify the CPU affinity of the iothread:
1. iothread=3@0:1:2/0:1
`add_virtual_device 9 virtio-blk iothread=3@0:1:2/0:1,mq=3,/dev/nvme1n1`
a) 3 iothread instances are created.
b) CPU affinity of iothread instances for this virtio-blk device:
- 1st iothread instance <-> pins to Service VM CPU 0,1,2
- 2nd iothread instance <-> pins to Service VM CPU 0,1
- 3rd iothread instance <-> No CPU affinity settings
2. iothread=3@0/*/1
`add_virtual_device 9 virtio-blk iothread=3@0/*/1,mq=3,/dev/nvme1n1`
a) 3 iothread instances are created.
b) CPU affinity of iothread instances for this virtio-blk device:
- 1st iothread instance <-> pins to Service VM CPU 0
- 2nd iothread instance <-> No CPU affinity settings
- 3rd iothread instance <-> pins to Service VM CPU 1
v1 -> v2:
* encapsulate one API in iothread.c to parse the iothread options, so that
other BE can also use it.
v2 -> v3:
* introduce one API iothread_free_options to free the elements that
are allocated dynamically in iothread_parse_options().
Tracked-On: #8612
Signed-off-by: Shiqing Gao <shiqing.gao@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
This patch renames the iothread for better readability. For instance,
the new name of the iothread for virtio-blk device looks like `iothr-0-blk9:0`.
It could be helpful when tuning the performance and the CPU utilization.
v1 -> v2:
* add `const` qualifier for the input parameter of `iothread_create`
Tracked-On: #8612
Signed-off-by: Shiqing Gao <shiqing.gao@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
To improve the performance of the virtual device who utilizes iothread
(such as virtio-blk), this patch sets iothread nice value to PRIO_MIN,
so that it could get higher priority on scheduling.
This patch does:
- introduce `set_thread_priority` to set the priority of the current running
thread.
The priority could be any value in the range PRIO_MIN to PRIO_MAX.
Lower numerical value causes more favorable scheduling.
- set iothread nice value to PRIO_MIN.
Tracked-On: #8612
Signed-off-by: Jian Jun Chen <jian.jun.chen@intel.com>
Signed-off-by: Shiqing Gao <shiqing.gao@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
Prior to this patch, one single iothread instance is created and initialized
in the `main` function. This single iothread monitors all the registered fds
and handles all the corresponding requests. It leads to the limited flexibility
of the iothread support.
To improve the flexibility of the iothread support, this patch does:
- add the support of multiple iothread instances.
`iothread_create` is introduced to create a certain number of iothread
instances. It shall be called at first by each virtual device owner (such as
virtio-blk BE) on initialization phase. Then, `iothread_add` can be called
to add the to be monitored fd to the specified iothread.
- update virtio-blk BE to let the acrn-dm option `iothread` accept a number
as the number of iothread instances to be created.
If `iothread` is contained in the parameters, but the number is not specified,
one iothread instance would be created by default.
Examples to specify the number of iothread instances:
1. Create 2 iothread instances
`add_virtual_device 9 virtio-blk iothread=2,mq=2,/dev/nvme1n1,writeback,aio=io_uring`
2. Create 1 iothread instances (by default)
`add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=io_uring`
- update virtio-blk BE to separate the request handling of different virtqueues
to different iothreads.
The request from one or more virtqueues can be handled in one iothread.
The mapping between virtqueues and iothreads is based on round robin.
v1 -> v2:
* add a mutex to protect the free ioctx slot allocation
Tracked-On: #8612
Signed-off-by: Shiqing Gao <shiqing.gao@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
io_uring is a high-performance asynchronous I/O framework, primarily designed
to improve the efficiency of input and output (I/O) operations in user-space
applications.
This patch enables io_uring in block_if module. It utilizes the interfaces
provided by the user-space library `liburing` to interact with io_uring
in kernel-space.
To build the acrn-dm with io_uring support, `liburing-dev` package needs to be
installed. For example, it can be installed like below in Ubuntu 22.04.
sudo apt install liburing-dev
In order to support both the thread pool mechanism and the io_uring mechanism,
an acrn-dm option `aio` is introduced. By default, thread pool mechanism is
selected.
- Example to use io_uring:
`add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=io_uring`
- Example to use thread pool:
`add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback,aio=threads`
- Example to use thread pool (by default):
`add_virtual_device 9 virtio-blk iothread,mq=2,/dev/nvme1n1,writeback`
v2 -> v3:
* Update iothread_handler
- Use the unified eventfd interfaces to read the counter value of the
ioeventfd.
- Remove the while loop to read the ioeventfd. It is not necessary
because one read would reset the counter value to 0.
* Update iou_submit_sqe to return an error code
The caller of iou_submit_sqe shall check the return value.
If there is NO available submission queue entry in the submission queue,
need to break the while loop. Request can only be submitted when SQE is
available.
v1 -> v2:
* move the logic of reading out ioeventfd from iothread.c to virtio.c, because
it is specific to the virtqueue handling.
Tracked-On: #8612
Signed-off-by: Shiqing Gao <shiqing.gao@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
Fix the bug in iothread handler, the event should be read out so that the
next epoll_wait not return directly as the fd can still readable.
Tracked-On: #8181
Signed-off-by: Conghui <conghui.chen@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
As the type of pthread_t is unsigned long, so the init value for it
should not be '-1'. When the tid is '-1', it will continue to call
pthread_kill/pthread_join. This is incorrect.
Tracked-On: #7960
Signed-off-by: Conghui <conghui.chen@intel.com>
Reviewed-by: Zhao Yakui <yakui.zhao@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
Supply a decidate thread, which can moniter a set of fds with epoll,
when the data is ready, call the corresponding callback.
This iothread will be created automatically with the first successful
call to iothread_add, and will be destroyed in iothread_deinit if it
was created.
Note, currenlty only support one iothread.
Tracked-On: #7940
Signed-off-by: Conghui <conghui.chen@intel.com>
Acked-by: Wang, Yu1 <yu1.wang@intel.com>
