github/acrn-hypervisor
1
0
Fork 0
mirror of https://github.com/projectacrn/acrn-hypervisor.git synced 2025-09-23 01:37:44 +00:00
Code Issues Projects Releases Wiki Activity

doc: Add MBA documentation and update doc for CAT

Browse Source 
This patch updates CAT documentation as well as adds
documentation support for newly added MBA feature.

Signed-off-by: Vijay Dhanraj <vijay.dhanraj@intel.com>
This commit is contained in:
Vijay Dhanraj
2020-03-25 21:20:08 -07:00
committed by deb-intel
parent 86957d702a
commit a2763ab7f0
20 changed files with 374 additions and 275 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
doc/tutorials/acrn_configuration_tool.rst
View File

@@ -122,8 +122,6 @@ Additional scenario XML elements:
``GUEST_FLAG_IO_COMPLETION_POLLING`` specify whether the hypervisor needs
IO polling to completion
``GUEST_FLAG_CLOS_REQUIRED`` specify whether CLOS is required
``GUEST_FLAG_HIDE_MTRR`` specify whether to hide MTRR from the VM
``GUEST_FLAG_RT`` specify whether the vm is RT-VM

BIN
doc/tutorials/images/menuconfig-rdt.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

194
doc/tutorials/rdt_configuration.rst Normal file
View File

@@ -0,0 +1,194 @@
.. _rdt_configuration:
RDT Configuration
#################
On x86 platforms that support Intel Resource Director Technology(RDT) Allocation features such as Cache Allocation Technology (CAT) and Memory Bandwidth Allocation(MBA), ACRN hypervisor can be used to limit regular VMs which may be over-utilizing common resources such cache and memory bandwidth relative to their priority so that performance of other higher priority VMs (such as RTVMs) are not impacted.
There are basically 3 steps to use RDT,
1. Detect and enumerate RDT allocation capabilites on supported resources such as cache and memory bandwidth.
2. Setup resource mask array MSRs for each CLOS (Class of Service), basically to limit or allow access to resource usage.
3. Select CLOS for the CPU associated with VM, that will apply the resource mask on the CP
Steps #2 and #3 configure RDT resources for a VM and can be done in 2 ways,
a) Using HV debug shell
b) Using VM configuration
The following sections discuss how to detect, enumerate capabilities and configure RDT resources for VMs in ACRN hypervisor.
For futher details, please refer to ACRN RDT high-level design :ref:`hv_rdt` and `Intel (R) 64 and IA-32 Architectures Software Developer's Manual, (Section 17.19 INTEL® RESOURCE DIRECTOR TECHNOLOGY ALLOCATION FEATURES) <https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-3a-3b-3c-and-3d-system-programming-guide>`_
.. _rdt_detection_capabilities:
RDT detection and resource capabilites
**************************************
From the ACRN HV debug shell, you can use ``cpuid`` to detect and identify the resource capabilities. You can use the platform's serial port for the HV shell (refer to :ref:`getting-started-up2` for setup instructions).
Check if the platform supports RDT with ``cpuid``. First run ``cpuid 0x7 0x0``, the return value ebx[bit 15] should be set to 1 if the platform supports RDT. Then run ``cpuid 0x10 0x0``, and check EBX[3-1] bits. EBX[bit 1] set, represents L3 CAT is supported, EBX[bit 2] set, represents L2 CAT is supported and EBX[bit 3] set, represents MBA is supported. To query the capabilties of the supported resources, use the bit posistion as subleaf index. For example, run ``cpuid 0x10 0x2`` to query L2 CAT capability.
.. code-block:: none
ACRN:\>cpuid 0x7 0x0
cpuid leaf: 0x7, subleaf: 0x0, 0x0:0xd39ffffb:0x00000818:0xbc000400
For L3/L2, the following are the bit encoding,
* EAX[bit 4:0] reports the length of cache mask minus one. For example a value 0xa, means cache mask is 0x7ff.
* EBX[bit 31:0] reports a bit mask. Each set bit indicates the corresponding unit of the cache allocation may be used by other entities in the platform. (e.g. integrated graphics engine)
* ECX[bit 2] if set, indicates cache Code and Data Prioritization Technology is supported.
* EDX[bit 15:0] reports the maximum CLOS supported for the resource minus one. For example a value of 0xf, means the max CLOS supported is 0x10.
.. code-block:: none
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0xa:0x0:0x0
ACRN:\>cpuid 0x10 **0x1**
cpuid leaf: 0x10, subleaf: 0x1, 0xa:0x600:0x4:0xf
For MBA, the following are the bit encoding,
* EAX[bit 11:0] reports the maximum MBA throttling value minus one. For example a value 0x59, means max delay value is 0x60.
* EBX[bit 31:0] reserved
* ECX[bit 2] reports whether the response of the delay values is linear.
* EDX[bit 15:0] reports the maximum CLOS supported for the resource minus one. For example a value of 0x7, means the max CLOS supported is 0x8.
.. code-block:: none
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0xa:0x0:0x0
ACRN:\>cpuid 0x10 **0x3**
cpuid leaf: 0x10, subleaf: 0x3, 0x59:0x0:0x4:0x7
Tuning RDT resources in HV debug shell
**************************************
This section explains how to configure the RDT resources from HV debug shell.
#. Check PCPU IDs of each VM, the ``vcpu_list`` shows that VM0 is running on PCPU0,
and VM1 is running on PCPU1:
.. code-block:: none
ACRN:\>vcpu_list
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE
===== ======= ======= ========= ==========
0 0 0 PRIMARY Running
1 1 0 PRIMARY Running
#. Set resource mask array MSRs for each CLOS with ``wrmsr <reg_num> <value>``. For example if we want to restrict VM1 to use the lower 4 ways of LLC cache and allocate upper 7 ways of LLC access to VM0, first a CLOS is assigned for each VM. (say VM0 is assigned CLOS 0 and VM1 CLOS1). Then resource mask MSR corresponding to the CLOS0, in this case IA32_L3_MASK_BASE + 0 is programmed to 0x7f0 and resouce mask MSR corresponding to CLOS1, IA32_L3_MASK_BASE + 1 is set to 0xf.
.. code-block:: none
ACRN:\>wrmsr -p1 0xc90 0x7f0
ACRN:\>wrmsr -p1 0xc91 0xf
#. Assign CLOS1 to PCPU1 by programming the MSR IA32_PQR_ASSOC [bit 63:32] (0xc8f) to 0x100000000 to use CLOS1 and assign CLOS0 to PCPU 0 by programming MSR IA32_PQR_ASSOC [bit 63:32] to 0x0. Note that, IA32_PQR_ASSOC is per LP MSR and CLOS has to programmed on each LP.
.. code-block:: none
ACRN:\>wrmsr -p0 0xc8f 0x000000000 (this is default and can be skipped)
ACRN:\>wrmsr -p1 0xc8f 0x100000000
.. _rdt_vm_configuration:
Configure RDT for VM using VM Configuration
*******************************************
#. RDT on ACRN is enabled by default on platforms that have the support. Thanks to offline tool approach which generates a platform specific xml file using which ACRN identifies if RDT is supported on the platform or not. But the feature can be also be toggled using CONFIG_RDT_ENABLED flag with ``make menuconfig`` command. The first step is to clone the ACRN source code (if you haven't done it already):
.. code-block:: none
$ git clone https://github.com/projectacrn/acrn-hypervisor.git
$ cd acrn-hypervisor/
.. figure:: images/menuconfig-rdt.png
:align: center
#. The predefined cache masks can be found at ``hypervisor/arch/x86/configs/$(CONFIG_BOARD)/board.c``, for respective board. For example for apl-up2, it can found at ``hypervisor/arch/x86/configs/apl-up2/board.c``.
.. code-block:: none
:emphasize-lines: 3,7,11,15
struct platform_clos_info platform_l2_clos_array[MAX_PLATFORM_CLOS_NUM] = {
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 0,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 1,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 2,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 3,
},
};
.. note::
User can change the mask values, but cache mask must have **continuous bits**, or a #GP fault can be triggered. Similary when programming MBA delay value, care should taken to set the value to less than or equal to MAX dealy value.
#. Set up CLOS in the VM config. Please follow `RDT detection and resource capabilites` to identify the MAX CLOS that can be used. In ACRN we a value between 0 to **the lowest common MAX CLOS** among all the RDT resources to avoid resource misconfigurations. We will take SOS on sharing mode as an example. Its configuration data can be found at ``hypervisor/arch/x86/configs/vm_config.c``
.. code-block:: none
:emphasize-lines: 6
struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
{
.type = SOS_VM,
.name = SOS_VM_CONFIG_NAME,
.guest_flags = 0UL,
.clos = 1,
.memory = {
.start_hpa = 0x0UL,
.size = CONFIG_SOS_RAM_SIZE,
},
.os_config = {
.name = SOS_VM_CONFIG_OS_NAME,
},
},
};
.. note::
In ACRN, Lower CLOS always means higher priority (clos 0 > clos 1 > clos 2>...clos n). So care should be taken to program the VMs CLOS accordingly.
#. Careful consideration should be made in assigning vCPU affinity. In cache isolation configuration, not only need to isolate CAT-capable caches, but need to isolate lower-level caches as well. In the following example, logical processor #0 and #2 share L1 and L2 caches. In this case, do not assign LP #0 and LP #2 to different VMs that need to do cache isolation. Assign LP #1 and LP #3 with similar consideration.
.. code-block:: none
:emphasize-lines: 3
# lstopo-no-graphics -v
Package L#0 (P#0 CPUVendor=GenuineIntel CPUFamilyNumber=6 CPUModelNumber=142)
L3Cache L#0 (size=3072KB linesize=64 ways=12 Inclusive=1)
L2Cache L#0 (size=256KB linesize=64 ways=4 Inclusive=0)
L1dCache L#0 (size=32KB linesize=64 ways=8 Inclusive=0)
L1iCache L#0 (size=32KB linesize=64 ways=8 Inclusive=0)
Core L#0 (P#0)
PU L#0 (P#0)
PU L#1 (P#2)
L2Cache L#1 (size=256KB linesize=64 ways=4 Inclusive=0)
L1dCache L#1 (size=32KB linesize=64 ways=8 Inclusive=0)
L1iCache L#1 (size=32KB linesize=64 ways=8 Inclusive=0)
Core L#1 (P#1)
PU L#2 (P#1)
PU L#3 (P#3)
#. Similary bandwidth control is per-core (not per LP), so max delay values of per-LP CLOS is applied to the core. If HT is turned on, don’t place high priority threads on sibling LP running lower priority threads.
#. Based on scenario, build the ACRN hypervisor and copy the artifact ``acrn.efi`` to the
``/boot/EFI/acrn`` directory. If needed, update the devicemodel ``acrn-dm`` as well in ``/usr/bin`` directory. see :ref:`getting-started-building` for building instructions.
.. code-block:: none
$ make hypervisor BOARD=apl-up2 FIRMWARE=uefi
...
# these operations are done on UP2 board
$ mount /dev/mmcblk0p0 /boot
$ scp <acrn.efi-at-your-compile-PC> /boot/EFI/acrn
#. Restart the platform

4
doc/tutorials/rtvm_workload_design_guideline.rst
View File

@@ -15,8 +15,8 @@ Run RTVM with dedicated resources/devices
*****************************************
For best practice, ACRN allocates dedicated CPU, memory resources, and cache resources (using Intel
Cache Allocation Technology, aka CAT) to RTVMs. For best real time performance of I/O devices,
we recommend using dedicated (pass-thru) PCIe devices to avoid VM-Exit at run time.
Resource Directory allocation Technology such as CAT, MBA) to RTVMs. For best real time performance
of I/O devices, we recommend using dedicated (pass-thru) PCIe devices to avoid VM-Exit at run time.
.. note::
The configuration space for pass-thru PCI devices is still emulated and accessing it will

166
doc/tutorials/using_cat_on_up2.rst
View File

@@ -1,166 +0,0 @@
.. _using_cat_up2:
Using CAT on UP2
################
The UP Squared board `(UP2) <https://up-board.org/upsquared/specifications/>`_
is an x86 maker board based on the Intel Apollo Lake platform and supports
Cache Allocation Technology (CAT). With this feature, the usage of cache can be
restricted to each VM. ACRN hypervisor can do that by giving each VM a cache mask,
so that the VM will not evict the masked cache lines, as shown in :numref:
`cache-mask-vm`:
.. figure:: images/using_cat_up2.png
:align: center
:name: cache-mask-vm
Cache usage with Cache Mask
CAT Support on ACRN
*******************
As described at `Intel (R) 64 and IA-32 Architectures Software Developer's Manual
<https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-3a-3b-3c-and-3d-system-programming-guide>`_,
chapter 17.19, volume 3. There are 3 steps to use CAT:
1. Detect the CAT capability.
2. Setup cache mask array MSRs, which is referred to as Class of Service (CLOS) array.
3. Select one of the CLOS array for the CPU, that will be the cache mask of the CPU.
ACRN integrates the usage of CAT into VM configuration. The CLOS array must be defined
in the board config source code, and the VM needs to specify which CLOS to use in its VM
config data. If the platform supports CAT, the CLOS array will be written to the CLOS MSRs
at CPU init time, and the VM will set CLOS for VCPU at VM launch time.
The details are shown in :numref:`acrn-cat-hld`:
.. figure:: images/acrn_cat_hld.png
:align: center
:name: acrn-cat-hld
CAT Support on ACRN
Tuning CAT in HV debug shell
############################
From the ACRN HV debug shell, you can use ``cpuid``, ``wrmsr/rdmsr`` debug commands to
enumerate CAT capability and tune CAT parameters. You can use the UP2 board's serial port
for the HV shell (refer to :ref:`getting-started-up2` for setup instructions).
#. Check CAT ability with ``cupid``. First run ``cpuid 0x10 0x0``, the return value ebx[bit 2]
reports the L2 CAT is supported. Then run ``cpuid 0x10 0x2`` to query L2 CAT capability,
the return value eax[bit 4:0] reports the cache mask has 8 bit, and edx[bit 15:0] reports 4 CLOS are
supported, as shown below:
.. code-block:: none
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0x4:0x0:0x0
ACRN:\>cpuid 0x10 0x2
cpuid leaf: 0x10, subleaf: 0x2, 0x7:0x0:0x0:0x3
#. Check PCPU IDs of each VM, the ``vcpu_list`` shows that VM0 is running on PCPU0,
and VM1 is running on PCPU1:
.. code-block:: none
ACRN:\>vcpu_list
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE
===== ======= ======= ========= ==========
0 0 0 PRIMARY Running
1 1 0 PRIMARY Running
#. Set CLOS with ``wrmsr <reg_num> <value>``, we want VM1 to use the lower 6 ways of cache,
so CLOS0 is set to 0xc0 for the upper 2 ways, and CLOS1 is set to 0x3f for the lower
6 ways:
.. code-block:: none
ACRN:\>wrmsr -p1 0xd10 0xc0
ACRN:\>wrmsr -p1 0xd11 0x3f
#. Attach COS1 to PCPU1. Because MSR IA32_PQR_ASSOC [bit 63:32], we'll write 0x100000000
to it to use CLOS1
.. code-block:: none
ACRN:\>wrmsr -p1 0xc8f 0x100000000
.. _configure_cat_vm:
Configure CAT for VM with VM Configuration
##########################################
#. CAT on ACRN can be enabled and configured by modifying source code, so the first
step is to clone the ACRN source code (if you haven't already):
.. code-block:: none
$ git clone https://github.com/projectacrn/acrn-hypervisor.git
$ cd acrn-hypervisor/
#. The predefined cache masks can be found at
``hypervisor/arch/x86/configs/$(CONFIG_BOARD)/board.c``, for UP2 board, that is
``hypervisor/arch/x86/configs/apl-up2/board.c``, you can change the mask values,
but note that the CLOS mask must have continuous bits, or a #GP fault can be triggered.
.. code-block:: none
:emphasize-lines: 3,7,11,15
struct platform_clos_info platform_clos_array[4] = {
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_0,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_1,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_2,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_3,
},
};
#. Set up CLOS in the VM config. If you want a VM to use one of the CLOSs, you need to find its
configuration data and modify it. We will take SOS on sharing mode as an example. Its
configuration data can be found at ``hypervisor/arch/x86/configs/vm_config.c``
.. code-block:: none
:emphasize-lines: 5,6
struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
{
.type = SOS_VM,
.name = SOS_VM_CONFIG_NAME,
.guest_flags = SOS_VM_CONFIG_GUEST_FLAGS | CLOS_REQUIRED,
.clos = 1,
.memory = {
.start_hpa = 0x0UL,
.size = CONFIG_SOS_RAM_SIZE,
},
.os_config = {
.name = SOS_VM_CONFIG_OS_NAME,
},
},
};
#. Build the ACRN hypervisor and copy the artifact ``acrn.efi`` to the
``/boot/EFI/acrn`` directory, see :ref:`getting-started-building` for building instructions.
.. code-block:: none
$ make hypervisor BOARD=apl-up2 FIRMWARE=uefi
...
# these operations are done on UP2 board
$ mount /dev/mmcblk0p0 /boot
$ scp <acrn.efi-at-your-compile-PC> /boot/EFI/acrn
#. Restart the UP2 board