mirror of
https://github.com/projectacrn/acrn-hypervisor.git
synced 2025-05-31 03:15:42 +00:00
184 lines
6.7 KiB
ReStructuredText
184 lines
6.7 KiB
ReStructuredText
.. _cpu_sharing:
|
|
|
|
Enable CPU Sharing in ACRN
|
|
##########################
|
|
|
|
Introduction
|
|
************
|
|
|
|
The goal of CPU Sharing is to fully utilize the physical CPU resource to
|
|
support more virtual machines. Currently, ACRN only supports 1 to 1
|
|
mapping mode between virtual CPUs (vCPUs) and physical CPUs (pCPUs).
|
|
Because of the lack of CPU sharing ability, the number of VMs is
|
|
limited. To support CPU Sharing, we have introduced a scheduling
|
|
framework and implemented two simple small scheduling algorithms to
|
|
satisfy embedded device requirements. Note that, CPU Sharing is not
|
|
available for VMs with local APIC passthrough (``--lapic_pt`` option).
|
|
|
|
Scheduling Framework
|
|
********************
|
|
|
|
To satisfy the modularization design concept, the scheduling framework
|
|
layer isolates the vCPU layer and scheduler algorithm. It does not have
|
|
a vCPU concept so it is only aware of the thread object instance. The
|
|
thread object state machine is maintained in the framework. The
|
|
framework abstracts the scheduler algorithm object, so this architecture
|
|
can easily extend to new scheduler algorithms.
|
|
|
|
.. figure:: images/cpu_sharing_framework.png
|
|
:align: center
|
|
|
|
The below diagram shows that the vCPU layer invokes APIs provided by
|
|
scheduling framework for vCPU scheduling. The scheduling framework also
|
|
provides some APIs for schedulers. The scheduler mainly implements some
|
|
callbacks in an ``acrn_scheduler`` instance for scheduling framework.
|
|
Scheduling initialization is invoked in the hardware management layer.
|
|
|
|
.. figure:: images/cpu_sharing_api.png
|
|
:align: center
|
|
|
|
vCPU affinity
|
|
*************
|
|
|
|
Currently, we do not support vCPU migration; the assignment of vCPU
|
|
mapping to pCPU is statically configured by acrn-dm through
|
|
``--cpu_affinity``. Use these rules to configure the vCPU affinity:
|
|
|
|
- Only one bit can be set for each affinity item of vCPU.
|
|
- vCPUs in the same VM cannot be assigned to the same pCPU.
|
|
|
|
Here is an example for affinity:
|
|
|
|
- VM0: 2 vCPUs, pinned to pCPU0 and pCPU1
|
|
- VM1: 2 vCPUs, pinned to pCPU2 and pCPU3
|
|
- VM2: 2 vCPUs, pinned to pCPU0 and pCPU1
|
|
|
|
.. figure:: images/cpu_sharing_affinity.png
|
|
:align: center
|
|
|
|
Thread object state
|
|
*******************
|
|
|
|
The thread object contains three states: RUNNING, RUNNABLE, and BLOCKED.
|
|
|
|
.. figure:: images/cpu_sharing_state.png
|
|
:align: center
|
|
|
|
After a new vCPU is created, the corresponding thread object is
|
|
initiated. The vCPU layer invokes a wakeup operation. After wakeup, the
|
|
state for the new thread object is set to RUNNABLE, and then follows its
|
|
algorithm to determine whether or not to preempt the current running
|
|
thread object. If yes, it turns to the RUNNING state. In RUNNING state,
|
|
the thread object may turn back to the RUNNABLE state when it runs out
|
|
of its timeslice, or it might yield the pCPU by itself, or be preempted.
|
|
The thread object under RUNNING state may trigger sleep to transfer to
|
|
BLOCKED state.
|
|
|
|
Scheduler
|
|
*********
|
|
|
|
The below block diagram shows the basic concept for the scheduler. There
|
|
are two kinds of schedulers in the diagram: NOOP (No-Operation) scheduler
|
|
and BVT (Borrowed Virtual Time) scheduler.
|
|
|
|
|
|
- **No-Operation scheduler**:
|
|
|
|
The NOOP (No-operation) scheduler has the same policy as the original
|
|
1-1 mapping previously used; every pCPU can run only two thread objects:
|
|
one is the idle thread, and another is the thread of the assigned vCPU.
|
|
With this scheduler, vCPU works in Work-Conserving mode, which always
|
|
try to keep resource busy, and will run once it is ready. Idle thread
|
|
can run when the vCPU thread is blocked.
|
|
|
|
- **Borrowed Virtual Time scheduler**:
|
|
|
|
BVT (Borrowed Virtual time) is a virtual time based scheduling
|
|
algorithm, it dispatching the runnable thread with the earliest
|
|
effective virtual time.
|
|
|
|
TODO: BVT scheduler will be built on top of prioritized scheduling
|
|
mechanism, i.e. higher priority threads get scheduled first, and same
|
|
priority tasks are scheduled per BVT.
|
|
|
|
- **Virtual time**: The thread with the earliest effective virtual
|
|
time (EVT) is dispatched first.
|
|
- **Warp**: a latency-sensitive thread is allowed to warp back in
|
|
virtual time to make it appear earlier. It borrows virtual time from
|
|
its future CPU allocation and thus does not disrupt long-term CPU
|
|
sharing
|
|
- **MCU**: minimum charging unit, the scheduler account for running time
|
|
in units of MCU.
|
|
- **Weighted fair sharing**: each runnable thread receives a share of
|
|
the processor in proportion to its weight over a scheduling
|
|
window of some number of MCU.
|
|
- **C**: context switch allowance. Real time by which the current
|
|
thread is allowed to advance beyond another runnable thread with
|
|
equal claim on the CPU. C is similar to the quantum in conventional
|
|
timesharing.
|
|
|
|
|
|
Scheduler configuration
|
|
***********************
|
|
|
|
Two places in the code decide the usage for the scheduler.
|
|
|
|
* The option in Kconfig decides the only scheduler used in runtime.
|
|
``hypervisor/arch/x86/Kconfig``
|
|
|
|
.. code-block:: none
|
|
|
|
config SCHED_BVT
|
|
bool "BVT scheduler"
|
|
help
|
|
BVT (Borrowed Virtual time) is virtual time based scheduling algorithm. It
|
|
dispatches the runnable thread with the earliest effective virtual time.
|
|
TODO: BVT scheduler will be built on top of prioritized scheduling mechanism,
|
|
i.e. higher priority threads get scheduled first, and same priority tasks are
|
|
scheduled per BVT.
|
|
|
|
The default scheduler is **SCHED_NOOP**. To use the BVT, change it to
|
|
**SCHED_BVT** in the **ACRN Scheduler**.
|
|
|
|
* The cpu_affinity is configured by acrn-dm command.
|
|
|
|
For example, assign physical CPUs (pCPUs) 1 and 3 to this VM using::
|
|
|
|
--cpu_affinity 1,3
|
|
|
|
|
|
Example
|
|
*******
|
|
|
|
Use the following settings to support this configuration in the industry scenario:
|
|
|
|
+---------+-------+-------+-------+
|
|
|pCPU0 |pCPU1 |pCPU2 |pCPU3 |
|
|
+=========+=======+=======+=======+
|
|
|SOS + WaaG |RT Linux |
|
|
+-----------------+---------------+
|
|
|
|
- offline pcpu2-3 in SOS.
|
|
|
|
- launch guests.
|
|
|
|
- launch WaaG with "--cpu_affinity=0,1"
|
|
- launch RT with "--cpu_affinity=2,3"
|
|
|
|
|
|
After you start all VMs, check the vCPU affinities from the Hypervisor
|
|
console with the ``vcpu_list`` command:
|
|
|
|
.. code-block:: console
|
|
|
|
ACRN:\>vcpu_list
|
|
|
|
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE THREAD STATE
|
|
===== ======= ======= ========= ========== ==========
|
|
0 0 0 PRIMARY Running BLOCKED
|
|
0 1 0 SECONDARY Running BLOCKED
|
|
1 0 0 PRIMARY Running RUNNING
|
|
1 1 0 SECONDARY Running RUNNING
|
|
2 2 0 PRIMARY Running RUNNING
|
|
2 3 1 SECONDARY Running RUNNING
|