github/acrn-hypervisor
1
0
Fork 0
mirror of https://github.com/projectacrn/acrn-hypervisor.git synced 2025-08-07 03:04:49 +00:00
Code Issues Projects Releases Wiki Activity

doc: Add MBA documentation and update doc for CAT

Browse Source 
This patch updates CAT documentation as well as adds
documentation support for newly added MBA feature.

Signed-off-by: Vijay Dhanraj <vijay.dhanraj@intel.com>
This commit is contained in:
Vijay Dhanraj 2020-03-25 21:20:08 -07:00 committed by wenlingz
parent 05f6b11f14
commit 143f9bbf5b
20 changed files with 374 additions and 275 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
doc/develop.rst
View File

@ -57,7 +57,7 @@ Enable ACRN Features
tutorials/sgx_virtualization
tutorials/vuart_configuration
tutorials/skl-nuc
tutorials/using_cat_on_up2
tutorials/rdt_configuration
tutorials/using_sbl_on_up2
tutorials/trustyACRN
tutorials/waag-secure-boot

2
doc/developer-guides/hld/hld-hypervisor.rst
View File

@ -23,4 +23,4 @@ Hypervisor high-level design
Console, Shell, and vUART <hv-console>
Hypercall / VHM upcall <hv-hypercall>
Compile-time configuration <hv-config>
RDT support, CAT <hv-rdt>
RDT support <hv-rdt>

1
doc/developer-guides/hld/hld-overview.rst
View File

@ -60,6 +60,7 @@ series features to enhance its real-time performance then meet hard-RT KPI
for its RT VM:
- CAT (Cache Allocation Technology)
- MBA (Memory Bandwidth Allocation)
- LAPIC pass-thru
- Polling mode driver
- ART (always running timer)

16
doc/developer-guides/hld/hv-cpu-virt.rst
View File

@ -955,13 +955,17 @@ pass-through directly:
- x2APIC related MSRs (offset from 0x800 to 0x900)
- emulates with vlapic
* - MSR_IA32_L2_MASK_n
- L2 CAT mask for COSn
- emulates with vCAT
* - MSR_IA32_L2_MASK_BASE~n
- L2 CAT mask for CLOSn
- disabled for guest access
* - MSR_IA32_L3_MASK_n
- L3 CAT mask for COSn
- emulates with vCAT
* - MSR_IA32_L3_MASK_BASE~n
- L3 CAT mask for CLOSn
- disabled for guest access
* - MSR_IA32_MBA_MASK_BASE~n
- MBA delay mask for CLOSn
- disabled for guest access
* - MSR_IA32_VMX_BASIC~VMX_TRUE_ENTRY_CTLS
- VMX related MSRs

12
doc/developer-guides/hld/hv-partitionmode.rst
View File

@ -23,11 +23,13 @@ In partition mode, ACRN provides guests with exclusive access to cores,
memory, cache, and peripheral devices. Partition mode enables developers
to dedicate resources exclusively among the guests. However there is no
support today in x86 hardware or in ACRN to partition resources such as
peripheral buses (e.g. PCI) or memory bandwidth. Cache partitioning
technology, such as Cache Allocation Technology (CAT) in x86, can be
used by developers to partition Last Level Cache (LLC) among the guests.
(Note: ACRN support for x86 CAT is on the roadmap, but not currently
supported).
peripheral buses (e.g. PCI). On x86 platforms that support Cache
Allocation Technology (CAT) and Memory Bandwidth Allocation(MBA), resources
such as Cache and memory bandwidth can be used by developers to partition
L2, Last Level Cache (LLC) and memory bandwidth among the guests. Refer to
:ref:`hv_rdt` for more details on ACRN RDT high-level design and
:ref:`rdt_configuration` for RDT configuration.
ACRN expects static partitioning of resources either by code
modification for guest configuration or through compile-time config

181
doc/developer-guides/hld/hv-rdt.rst
View File

@ -3,45 +3,158 @@
RDT Allocation Feature Supported by Hypervisor
##############################################
The hypervisor uses RDT (Resource Director Technology) allocation features to optimize VM performance. There are 2 sub-features: CAT (Cache Allocation Technology) and MBA (Memory Bandwidth Allocation). CAT is for cache resources and MBA is for memory bandwidth resources. Code and Data Prioritization (CDP) is an extension of CAT. Only CAT is enabled due to the feature availability on an ACRN-supported platform. In ACRN, the CAT is configured via the "VM-Configuration". The resources allocated for VMs are determined in the VM configuration.
The hypervisor uses RDT (Resource Director Technology) allocation features such as CAT(Cache Allocation Technology) and MBA(Memory Bandwidth Allocation) to control VMs which may be over-utilizing cache resources or memory bandwidth relative to their priority. By setting limits to critical resources, ACRN can optimize RTVM performance over regular VMs. In ACRN, the CAT and MBA are configured via the "VM-Configuration". The resources allocated for VMs are determined in the VM configuration(:ref:`rdt_vm_configuration`).
CAT Support in ACRN
*******************
For futher details on Intel RDT, please refer to `Intel (R) 64 and IA-32 Architectures Software Developer's Manual, (Section 17.19 INTEL® RESOURCE DIRECTOR TECHNOLOGY ALLOCATION FEATURES) <https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-3a-3b-3c-and-3d-system-programming-guide>`_
Introduction to CAT Capabilities
================================
On a platform which supports CAT, each CPU can mask last-level-cache (LLC) with a cache mask, the masked cache ways cannot be evicted by this CPU. In terms of SDM, please see chapter 17, volume 3, CAT capabilities are enumerated via CPUID, and configured via MSR registers, these are:
* CPUID.10H contains CAT capabilities, such as cache mask type(L2/L3), mask bit
length, number of masks.
* Cache masks are set in IA32_type_MASK_n, each one of these MSRs can
hold a cache mask. They are shared by CPUs who share the same LLC; CPU sets
the RMID field of its IA32_PQR_ASSOC MSR with class-of-service (CLOS) ID, to
select a cache mask to take effect.
Objective of CAT
================
****************
The CAT feature in the hypervisor can isolate the cache for a VM from other VMs. It can also isolate the cache usage between VMX root mode and VMX non-root mode. Generally, certain cache resources will be allocated for the RT VMs in order to reduce the performance interference through the shared cache access from the neighbor VMs.
The CAT feature in the hypervisor can isolate the cache for a VM from other VMs. It can also isolate the cache usage between VMX root mode and VMX non-root mode. Generally, certain cache resources will be allocated for the RT VMs in order to reduce the performance interference through the shared cache access from the neighbour VMs.
The figure below shows that with CAT, the cache ways can be isolated vs default where high priority VMs can be impacted by a noisy neighbor.
CAT Workflow
=============
.. figure:: images/cat-objective.png
:align: center
The hypervisor enumerates CAT capabilities and setup cache mask arrays; it also sets up CLOS for VMs and hypervisor itself per the "vm configuration".
CAT Support in ACRN
===================
On x86 platforms that support CAT, ACRN hypervisor automatically enables the support and by default shares the cache ways equally between all the VMs. This is done by setting max cache mask in MSR_IA32_type_MASK_n (where type: L2 or L3) MSR corresponding to each CLOS and setting IA32_PQR_ASSOC MSR with CLOS 0. The user can check the cache capabilities such as cache mask, max supported CLOS as described in :ref:`rdt_detection_capabilities` and program the IA32_type_MASK_n and IA32_PQR_ASSOC MSR with class-of-service (CLOS) ID, to select a cache mask to take effect. ACRN uses VMCS MSR loads on every VM Entry/VM Exit for non-root and root modes to enforce the settings.
* The CAT capabilities are enumerated on boot-strap processor (BSP), at the
PCPU pre-initialize stage. The global data structure cat_cap_info holds the
result.
* If CAT is supported, then setup cache masks array on all APs, at the PCPU
post-initialize stage. The mask values are written to IA32_type_MASK_n. In
fact, for CPUs which share LLC, they share the same IA32_type_MASK_n MSRs too,
only need to do that on one CPU of them. The hypervisor does not detect
hierarchy of LLCs.
* If CAT is supported. The CLOS of a VM will be stored into its vCPU
msr_store_area data structure guest part. It will be loaded to
MSR IA32_PQR_ASSOC at each VM entry.
* If CAT is supported, The CLOS of hypervisor is stored for all VMs, in their
vCPU msr_store_area data structure host part. It will be loaded to MSR
IA32_PQR_ASSOC at each VM exit.
.. code-block:: none
:emphasize-lines: 3,7,11,15
struct platform_clos_info platform_l2_clos_array[MAX_PLATFORM_CLOS_NUM] = {
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 0,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 1,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 2,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 3,
},
};
.. code-block:: none
:emphasize-lines: 6
struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
{
.type = SOS_VM,
.name = SOS_VM_CONFIG_NAME,
.guest_flags = 0UL,
.clos = 0,
.memory = {
.start_hpa = 0x0UL,
.size = CONFIG_SOS_RAM_SIZE,
},
.os_config = {
.name = SOS_VM_CONFIG_OS_NAME,
},
},
};
.. note::
ACRN takes the lowest common CLOS max value between the supported resources and sets the MAX_PLATFORM_CLOS_NUM. For example, if max CLOS supported by L3 is 16 and L2 is 8, ACRN programs MAX_PLATFORM_CLOS_NUM to 8. ACRN recommends to have consistent capabilities across all RDT resource by using common subset CLOS. This is done in order to minimize misconfiguration errors.
Objective of MBA
****************
The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate control over memory bandwidth available per-core. It provides a method to control VMs which may be over-utilizing bandwidth relative to their priority and thus improving performance of high priority VMs. MBA introduces a programmable request rate controller (PRRC) between cores and high-speed interconnect. Throttling values can be programmed via MSRs to the PRRC to limit bandwidth availability.
The following figure shows memory bandwidth impact without MBA which cause bottleneck for high priority VMs vs with MBA support,
.. figure:: images/no_mba_objective.png
:align: center
:name: without-mba-support
Without MBA Support
.. figure:: images/mba_objective.png
:align: center
:name: with-mba-support
With MBA Support
MBA Support in ACRN
===================
On x86 platforms that support MBA, ACRN hypervisor automatically enables the support and by default sets no limits to the memory bandwidth access by VMs. This is done by setting 0 mba delay value in MSR_IA32_MBA_MASK_n MSR corresponding to each CLOS and setting IA32_PQR_ASSOC MSR with CLOS 0. The user can check the MBA capabilities such as mba delay values, max supported CLOS as described in :ref:`rdt_detection_capabilities` and program the IA32_MBA_MASK_n and IA32_PQR_ASSOC MSR with class-of-service (CLOS) ID, to select a delay to take effect for restricting memory bandwidth. ACRN uses VMCS MSR loads on every VM Entry/VM Exit for non-root and root modes to enforce the settings.
.. code-block:: none
:emphasize-lines: 3,7,11,15
struct platform_clos_info platform_mba_clos_array[MAX_PLATFORM_CLOS_NUM] = {
{
.mba_delay = 0,
.msr_index = MSR_IA32_MBA_MASK_BASE + 0,
},
{
.mba_delay = 0,
.msr_index = MSR_IA32_MBA_MASK_BASE + 1,
},
{
.mba_delay = 0,
.msr_index = MSR_IA32_MBA_MASK_BASE + 2,
},
{
.mba_delay = 0,
.msr_index = MSR_IA32_MBA_MASK_BASE + 3,
},
};
.. code-block:: none
:emphasize-lines: 6
struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
{
.type = SOS_VM,
.name = SOS_VM_CONFIG_NAME,
.guest_flags = 0UL,
.clos = 0,
.memory = {
.start_hpa = 0x0UL,
.size = CONFIG_SOS_RAM_SIZE,
},
.os_config = {
.name = SOS_VM_CONFIG_OS_NAME,
},
},
};
.. note::
ACRN takes the lowest common CLOS max value between the supported resources and sets the MAX_PLATFORM_CLOS_NUM. For example, if max CLOS supported by L3 is 16 and MBA is 8, ACRN programs MAX_PLATFORM_CLOS_NUM to 8. ACRN recommends to have consistent capabilities across all RDT resource by using common subset CLOS. This is done in order to minimize misconfiguration errors.
CAT and MBA high-level design in ACRN
*************************************
Data structures
===============
The below figure shows the RDT data structure to store the enumerated resources.
.. figure:: images/mba_data_structures.png
:align: center
Enabling CAT, MBA software flow
===============================
The hypervisor enumerates RDT capabilities and sets up mask arrays; it also sets up CLOS for VMs and hypervisor itself per the "vm configuration"(:ref:`rdt_vm_configuration`).
* The RDT capabilities are enumerated on boot-strap processor (BSP), at the pCPU pre-initialize stage. The global data structure ``res_cap_info`` stores the capabilites of the supported resources.
* If CAT or/and MBA is supported, then setup masks array on all APs, at the pCPU post-initialize stage. The mask values are written to IA32_type_MASK_n. Refer :ref:`rdt_detection_capabilities` for details on identifying values to program the mask/delay MRSs as well as max CLOS.
* If CAT or/and is supported, the CLOS of a **VM** will be stored into its vCPU ``msr_store_area`` data structure guest part. It will be loaded to MSR IA32_PQR_ASSOC at each VM entry.
* If CAT or/and MBA is supported, the CLOS of **hypervisor** is stored for all VMs, in their vCPU ``msr_store_area`` data structure host part. It will be loaded to MSR IA32_PQR_ASSOC at each VM exit.
The figure below shows the high level overview of RDT resource flow in ACRN hypervisor.
.. figure:: images/cat_mba_software_flow.png
:align: center

BIN
doc/developer-guides/hld/images/cat-objective.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

BIN
doc/developer-guides/hld/images/cat_mba_software_flow.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

BIN
doc/developer-guides/hld/images/mba_data_structures.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

BIN
doc/developer-guides/hld/images/mba_objective.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 76 KiB

BIN
doc/developer-guides/hld/images/no_mba_objective.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 52 KiB

57
doc/getting-started/rt_industry.rst
View File

@ -339,63 +339,10 @@ Recommended BIOS settings
.. note:: BIOS settings depend on the platform and BIOS version; some may not be applicable.
Configure CAT
Configure RDT
-------------
.. _Apollo Lake NUC:
https://www.intel.com/content/www/us/en/products/boards-kits/nuc/kits/nuc6cayh.html
.. note:: CAT configuration is only supported on the `Apollo Lake NUC`_.
With the ACRN Hypervisor shell, we can use ``cpuid`` and ``wrmsr``/``rdmsr`` debug commands to enumerate the CAT capability and set the CAT configuration without rebuilding binaries. Because ``lapic`` is a pass-through to the RTVM, the CAT configuration must be set before launching the RTVM.
Check CAT ability with cupid
````````````````````````````
First run ``cpuid 0x10 0x0``. The return value of ``ebx[bit 2]`` reports that the L2 CAT is supported.
Next, run ``cpuid 0x10 0x2`` to query the L2 CAT capability; the return value of ``eax[bit 4:0]``
reports that the cache mask has 8 bits, and ``edx[bit 15:0]`` reports that 04 CLOS are supported,
as shown below. The reported data is in the format of ``[ eax:ebx:ecx:edx ]``::
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0x4:0x0:0x0
ACRN:\>cpuid 0x10 0x2
cpuid leaf: 0x10, subleaf: 0x2, 0x7:0x0:0x0:0x3
Set CLOS (QOS MASK) and PQR_ASSOC MSRs to configure the CAT
```````````````````````````````````````````````````````````
Apollo Lake doesn't have L3 cache and it supports L2 CAT. The CLOS MSRs are per L2 cache and starts from 0x00000D10. In the case of 4 CLOS MSRs, the address is as follows::
MSR_IA32_L2_QOS_MASK_0 0x00000D10
MSR_IA32_L2_QOS_MASK_1 0x00000D11
MSR_IA32_L2_QOS_MASK_2 0x00000D12
MSR_IA32_L2_QOS_MASK_3 0x00000D13
The PQR_ASSOC MSR is per CPU core; each core has its own PQR_ASSOC::
MSR_IA32_PQR_ASSOC 0x00000C8F
To set the CAT, first set the CLOS MSRs. Next, set the PQR_ASSOC of each CPU
so that the CPU of the RTVM uses dedicated cache and other CPUs use other cache. Taking a Quad Core Apollo Lake platform for example, CPU0 and CPU1 share L2 cache while CPU2 and CPU3 share the other L2 cache.
- If we allocate CPU2 and CPU3, no extra action is required.
- If we allocate only CPU1 to the RTVM, we need to set the CAT as follows.
These commands actually set the CAT configuration for L2 cache shared by CPU0 and CPU1.
a. Set CLOS with ``wrmsr <reg_num> <value>``, we want VM1 to use the lower 6 ways of cache,
so CLOS0 is set to 0xf0 for the upper 4 ways, and CLOS1 is set to 0x0f for the lower 4 ways::
ACRN:\>wrmsr -p1 0xd10 0xf0
ACRN:\>wrmsr -p1 0xd11 0x0f
#. Attach COS1 to PCPU1. Because MSR is IA32_PQR_ASSOC [bit 63:32], we’ll write 0x100000000 to it to use CLOS1::
ACRN:\>wrmsr -p0 0xc8f 0x000000000
ACRN:\>wrmsr -p1 0xc8f 0x100000000
In addition to setting the CAT configuration via HV commands, we allow developers to add the CAT configurations to the VM config and do the configure automatically at the time of RTVM creation. Refer to :ref:`configure_cat_vm` for details.
In addition to setting the CAT configuration via HV commands, we allow developers to add the CAT configurations to the VM config and do the configure automatically at the time of RTVM creation. Refer to :ref:`rdt_configuration` for details on RDT configuration and :ref:`hv_rdt` for details on RDT high-level design.
Set up the core allocation for the RTVM
---------------------------------------

8
doc/glossary.rst
View File

@ -165,11 +165,17 @@ Glossary of Terms
RSE
Rear Seat Entertainment
RDT
Intel Resource Director Technology (Intel RDT) provides a set of
monitoring and allocation capabilities to control resources such as
Cache, Memory. ACRN supports, Cache Allocation Technology (CAT) and
Memory Bandwidth Allocation (MBA).
RTVM
Real-time VM. A specially designed VM to run hard real-time or
soft real-time workloads (or application) much more efficiently
than the typical User VM through the use of pass-through interrupt
controller, polling-mode Virtio, Intel Cache Allocation Technology (CAT)
controller, polling-mode Virtio, Intel RDT allocation features(CAT, MBA)
and I/O prioritization. RTVMs are typically a :term:`pre-launched VM`.
A non-:term:`safety VM` with real-time requirements can a
:term:`post-launched VM`.

2
doc/release_notes/release_notes_0.8.rst
View File

@ -63,7 +63,7 @@ Document updates
Several new documents have been added in this release, including:
* :ref:`Zephyr RTOS as Guest OS <using_zephyr_as_uos>`
* :ref:`Enable cache QoS with CAT <using_cat_up2>`
* :ref:`Enable cache QoS with CAT <rdt_configuration>`
* :ref:`ACRN kernel parameter introduction <kernel-parameters>`
* :ref:`faq` update for two issues
* :ref:`ACRN Debug introduction <acrn-debug>`

2
doc/release_notes/release_notes_1.0.rst
View File

@ -221,7 +221,7 @@ We have many reference documents `available
* Additional project `coding guidelines
<coding_guidelines.html>`_
* :ref:`Zephyr RTOS as Guest OS <using_zephyr_as_uos>`
* :ref:`Enable cache QoS with CAT <using_cat_up2>`
* :ref:`Enable cache QoS with CAT <rdt_configuration>`
* :ref:`ACRN kernel parameter introduction <kernel-parameters>`
* :ref:`FAQ update for two issues <faq>`
* :ref:`ACRN Debug introduction <acrn-debug>`

2
doc/tutorials/acrn_configuration_tool.rst
View File

@ -122,8 +122,6 @@ Additional scenario XML elements:
``GUEST_FLAG_IO_COMPLETION_POLLING`` specify whether the hypervisor needs
IO polling to completion
``GUEST_FLAG_CLOS_REQUIRED`` specify whether CLOS is required
``GUEST_FLAG_HIDE_MTRR`` specify whether to hide MTRR from the VM
``GUEST_FLAG_RT`` specify whether the vm is RT-VM

BIN
doc/tutorials/images/menuconfig-rdt.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

194
doc/tutorials/rdt_configuration.rst Normal file
View File

@ -0,0 +1,194 @@
.. _rdt_configuration:
RDT Configuration
#################
On x86 platforms that support Intel Resource Director Technology(RDT) Allocation features such as Cache Allocation Technology (CAT) and Memory Bandwidth Allocation(MBA), ACRN hypervisor can be used to limit regular VMs which may be over-utilizing common resources such cache and memory bandwidth relative to their priority so that performance of other higher priority VMs (such as RTVMs) are not impacted.
There are basically 3 steps to use RDT,
1. Detect and enumerate RDT allocation capabilites on supported resources such as cache and memory bandwidth.
2. Setup resource mask array MSRs for each CLOS (Class of Service), basically to limit or allow access to resource usage.
3. Select CLOS for the CPU associated with VM, that will apply the resource mask on the CP
Steps #2 and #3 configure RDT resources for a VM and can be done in 2 ways,
a) Using HV debug shell
b) Using VM configuration
The following sections discuss how to detect, enumerate capabilities and configure RDT resources for VMs in ACRN hypervisor.
For futher details, please refer to ACRN RDT high-level design :ref:`hv_rdt` and `Intel (R) 64 and IA-32 Architectures Software Developer's Manual, (Section 17.19 INTEL® RESOURCE DIRECTOR TECHNOLOGY ALLOCATION FEATURES) <https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-3a-3b-3c-and-3d-system-programming-guide>`_
.. _rdt_detection_capabilities:
RDT detection and resource capabilites
**************************************
From the ACRN HV debug shell, you can use ``cpuid`` to detect and identify the resource capabilities. You can use the platform's serial port for the HV shell (refer to :ref:`getting-started-up2` for setup instructions).
Check if the platform supports RDT with ``cpuid``. First run ``cpuid 0x7 0x0``, the return value ebx[bit 15] should be set to 1 if the platform supports RDT. Then run ``cpuid 0x10 0x0``, and check EBX[3-1] bits. EBX[bit 1] set, represents L3 CAT is supported, EBX[bit 2] set, represents L2 CAT is supported and EBX[bit 3] set, represents MBA is supported. To query the capabilties of the supported resources, use the bit posistion as subleaf index. For example, run ``cpuid 0x10 0x2`` to query L2 CAT capability.
.. code-block:: none
ACRN:\>cpuid 0x7 0x0
cpuid leaf: 0x7, subleaf: 0x0, 0x0:0xd39ffffb:0x00000818:0xbc000400
For L3/L2, the following are the bit encoding,
* EAX[bit 4:0] reports the length of cache mask minus one. For example a value 0xa, means cache mask is 0x7ff.
* EBX[bit 31:0] reports a bit mask. Each set bit indicates the corresponding unit of the cache allocation may be used by other entities in the platform. (e.g. integrated graphics engine)
* ECX[bit 2] if set, indicates cache Code and Data Prioritization Technology is supported.
* EDX[bit 15:0] reports the maximum CLOS supported for the resource minus one. For example a value of 0xf, means the max CLOS supported is 0x10.
.. code-block:: none
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0xa:0x0:0x0
ACRN:\>cpuid 0x10 **0x1**
cpuid leaf: 0x10, subleaf: 0x1, 0xa:0x600:0x4:0xf
For MBA, the following are the bit encoding,
* EAX[bit 11:0] reports the maximum MBA throttling value minus one. For example a value 0x59, means max delay value is 0x60.
* EBX[bit 31:0] reserved
* ECX[bit 2] reports whether the response of the delay values is linear.
* EDX[bit 15:0] reports the maximum CLOS supported for the resource minus one. For example a value of 0x7, means the max CLOS supported is 0x8.
.. code-block:: none
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0xa:0x0:0x0
ACRN:\>cpuid 0x10 **0x3**
cpuid leaf: 0x10, subleaf: 0x3, 0x59:0x0:0x4:0x7
Tuning RDT resources in HV debug shell
**************************************
This section explains how to configure the RDT resources from HV debug shell.
#. Check PCPU IDs of each VM, the ``vcpu_list`` shows that VM0 is running on PCPU0,
and VM1 is running on PCPU1:
.. code-block:: none
ACRN:\>vcpu_list
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE
===== ======= ======= ========= ==========
0 0 0 PRIMARY Running
1 1 0 PRIMARY Running
#. Set resource mask array MSRs for each CLOS with ``wrmsr <reg_num> <value>``. For example if we want to restrict VM1 to use the lower 4 ways of LLC cache and allocate upper 7 ways of LLC access to VM0, first a CLOS is assigned for each VM. (say VM0 is assigned CLOS 0 and VM1 CLOS1). Then resource mask MSR corresponding to the CLOS0, in this case IA32_L3_MASK_BASE + 0 is programmed to 0x7f0 and resouce mask MSR corresponding to CLOS1, IA32_L3_MASK_BASE + 1 is set to 0xf.
.. code-block:: none
ACRN:\>wrmsr -p1 0xc90 0x7f0
ACRN:\>wrmsr -p1 0xc91 0xf
#. Assign CLOS1 to PCPU1 by programming the MSR IA32_PQR_ASSOC [bit 63:32] (0xc8f) to 0x100000000 to use CLOS1 and assign CLOS0 to PCPU 0 by programming MSR IA32_PQR_ASSOC [bit 63:32] to 0x0. Note that, IA32_PQR_ASSOC is per LP MSR and CLOS has to programmed on each LP.
.. code-block:: none
ACRN:\>wrmsr -p0 0xc8f 0x000000000 (this is default and can be skipped)
ACRN:\>wrmsr -p1 0xc8f 0x100000000
.. _rdt_vm_configuration:
Configure RDT for VM using VM Configuration
*******************************************
#. RDT on ACRN is enabled by default on platforms that have the support. Thanks to offline tool approach which generates a platform specific xml file using which ACRN identifies if RDT is supported on the platform or not. But the feature can be also be toggled using CONFIG_RDT_ENABLED flag with ``make menuconfig`` command. The first step is to clone the ACRN source code (if you haven't done it already):
.. code-block:: none
$ git clone https://github.com/projectacrn/acrn-hypervisor.git
$ cd acrn-hypervisor/
.. figure:: images/menuconfig-rdt.png
:align: center
#. The predefined cache masks can be found at ``hypervisor/arch/x86/configs/$(CONFIG_BOARD)/board.c``, for respective board. For example for apl-up2, it can found at ``hypervisor/arch/x86/configs/apl-up2/board.c``.
.. code-block:: none
:emphasize-lines: 3,7,11,15
struct platform_clos_info platform_l2_clos_array[MAX_PLATFORM_CLOS_NUM] = {
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 0,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 1,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 2,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L3_MASK_BASE + 3,
},
};
.. note::
User can change the mask values, but cache mask must have **continuous bits**, or a #GP fault can be triggered. Similary when programming MBA delay value, care should taken to set the value to less than or equal to MAX dealy value.
#. Set up CLOS in the VM config. Please follow `RDT detection and resource capabilites` to identify the MAX CLOS that can be used. In ACRN we a value between 0 to **the lowest common MAX CLOS** among all the RDT resources to avoid resource misconfigurations. We will take SOS on sharing mode as an example. Its configuration data can be found at ``hypervisor/arch/x86/configs/vm_config.c``
.. code-block:: none
:emphasize-lines: 6
struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
{
.type = SOS_VM,
.name = SOS_VM_CONFIG_NAME,
.guest_flags = 0UL,
.clos = 1,
.memory = {
.start_hpa = 0x0UL,
.size = CONFIG_SOS_RAM_SIZE,
},
.os_config = {
.name = SOS_VM_CONFIG_OS_NAME,
},
},
};
.. note::
In ACRN, Lower CLOS always means higher priority (clos 0 > clos 1 > clos 2>...clos n). So care should be taken to program the VMs CLOS accordingly.
#. Careful consideration should be made in assigning vCPU affinity. In cache isolation configuration, not only need to isolate CAT-capable caches, but need to isolate lower-level caches as well. In the following example, logical processor #0 and #2 share L1 and L2 caches. In this case, do not assign LP #0 and LP #2 to different VMs that need to do cache isolation. Assign LP #1 and LP #3 with similar consideration.
.. code-block:: none
:emphasize-lines: 3
# lstopo-no-graphics -v
Package L#0 (P#0 CPUVendor=GenuineIntel CPUFamilyNumber=6 CPUModelNumber=142)
L3Cache L#0 (size=3072KB linesize=64 ways=12 Inclusive=1)
L2Cache L#0 (size=256KB linesize=64 ways=4 Inclusive=0)
L1dCache L#0 (size=32KB linesize=64 ways=8 Inclusive=0)
L1iCache L#0 (size=32KB linesize=64 ways=8 Inclusive=0)
Core L#0 (P#0)
PU L#0 (P#0)
PU L#1 (P#2)
L2Cache L#1 (size=256KB linesize=64 ways=4 Inclusive=0)
L1dCache L#1 (size=32KB linesize=64 ways=8 Inclusive=0)
L1iCache L#1 (size=32KB linesize=64 ways=8 Inclusive=0)
Core L#1 (P#1)
PU L#2 (P#1)
PU L#3 (P#3)
#. Similary bandwidth control is per-core (not per LP), so max delay values of per-LP CLOS is applied to the core. If HT is turned on, don’t place high priority threads on sibling LP running lower priority threads.
#. Based on scenario, build the ACRN hypervisor and copy the artifact ``acrn.efi`` to the
``/boot/EFI/acrn`` directory. If needed, update the devicemodel ``acrn-dm`` as well in ``/usr/bin`` directory. see :ref:`getting-started-building` for building instructions.
.. code-block:: none
$ make hypervisor BOARD=apl-up2 FIRMWARE=uefi
...
# these operations are done on UP2 board
$ mount /dev/mmcblk0p0 /boot
$ scp <acrn.efi-at-your-compile-PC> /boot/EFI/acrn
#. Restart the platform

4
doc/tutorials/rtvm_workload_design_guideline.rst
View File

@ -15,8 +15,8 @@ Run RTVM with dedicated resources/devices
*****************************************
For best practice, ACRN allocates dedicated CPU, memory resources, and cache resources (using Intel
Cache Allocation Technology, aka CAT) to RTVMs. For best real time performance of I/O devices,
we recommend using dedicated (pass-thru) PCIe devices to avoid VM-Exit at run time.
Resource Directory allocation Technology such as CAT, MBA) to RTVMs. For best real time performance
of I/O devices, we recommend using dedicated (pass-thru) PCIe devices to avoid VM-Exit at run time.
.. note::
The configuration space for pass-thru PCI devices is still emulated and accessing it will

166
doc/tutorials/using_cat_on_up2.rst
View File

@ -1,166 +0,0 @@
.. _using_cat_up2:
Using CAT on UP2
################
The UP Squared board `(UP2) <https://up-board.org/upsquared/specifications/>`_
is an x86 maker board based on the Intel Apollo Lake platform and supports
Cache Allocation Technology (CAT). With this feature, the usage of cache can be
restricted to each VM. ACRN hypervisor can do that by giving each VM a cache mask,
so that the VM will not evict the masked cache lines, as shown in :numref:
`cache-mask-vm`:
.. figure:: images/using_cat_up2.png
:align: center
:name: cache-mask-vm
Cache usage with Cache Mask
CAT Support on ACRN
*******************
As described at `Intel (R) 64 and IA-32 Architectures Software Developer's Manual
<https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-3a-3b-3c-and-3d-system-programming-guide>`_,
chapter 17.19, volume 3. There are 3 steps to use CAT:
1. Detect the CAT capability.
2. Setup cache mask array MSRs, which is referred to as Class of Service (CLOS) array.
3. Select one of the CLOS array for the CPU, that will be the cache mask of the CPU.
ACRN integrates the usage of CAT into VM configuration. The CLOS array must be defined
in the board config source code, and the VM needs to specify which CLOS to use in its VM
config data. If the platform supports CAT, the CLOS array will be written to the CLOS MSRs
at CPU init time, and the VM will set CLOS for VCPU at VM launch time.
The details are shown in :numref:`acrn-cat-hld`:
.. figure:: images/acrn_cat_hld.png
:align: center
:name: acrn-cat-hld
CAT Support on ACRN
Tuning CAT in HV debug shell
############################
From the ACRN HV debug shell, you can use ``cpuid``, ``wrmsr/rdmsr`` debug commands to
enumerate CAT capability and tune CAT parameters. You can use the UP2 board's serial port
for the HV shell (refer to :ref:`getting-started-up2` for setup instructions).
#. Check CAT ability with ``cupid``. First run ``cpuid 0x10 0x0``, the return value ebx[bit 2]
reports the L2 CAT is supported. Then run ``cpuid 0x10 0x2`` to query L2 CAT capability,
the return value eax[bit 4:0] reports the cache mask has 8 bit, and edx[bit 15:0] reports 4 CLOS are
supported, as shown below:
.. code-block:: none
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0x4:0x0:0x0
ACRN:\>cpuid 0x10 0x2
cpuid leaf: 0x10, subleaf: 0x2, 0x7:0x0:0x0:0x3
#. Check PCPU IDs of each VM, the ``vcpu_list`` shows that VM0 is running on PCPU0,
and VM1 is running on PCPU1:
.. code-block:: none
ACRN:\>vcpu_list
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE
===== ======= ======= ========= ==========
0 0 0 PRIMARY Running
1 1 0 PRIMARY Running
#. Set CLOS with ``wrmsr <reg_num> <value>``, we want VM1 to use the lower 6 ways of cache,
so CLOS0 is set to 0xc0 for the upper 2 ways, and CLOS1 is set to 0x3f for the lower
6 ways:
.. code-block:: none
ACRN:\>wrmsr -p1 0xd10 0xc0
ACRN:\>wrmsr -p1 0xd11 0x3f
#. Attach COS1 to PCPU1. Because MSR IA32_PQR_ASSOC [bit 63:32], we'll write 0x100000000
to it to use CLOS1
.. code-block:: none
ACRN:\>wrmsr -p1 0xc8f 0x100000000
.. _configure_cat_vm:
Configure CAT for VM with VM Configuration
##########################################
#. CAT on ACRN can be enabled and configured by modifying source code, so the first
step is to clone the ACRN source code (if you haven't already):
.. code-block:: none
$ git clone https://github.com/projectacrn/acrn-hypervisor.git
$ cd acrn-hypervisor/
#. The predefined cache masks can be found at
``hypervisor/arch/x86/configs/$(CONFIG_BOARD)/board.c``, for UP2 board, that is
``hypervisor/arch/x86/configs/apl-up2/board.c``, you can change the mask values,
but note that the CLOS mask must have continuous bits, or a #GP fault can be triggered.
.. code-block:: none
:emphasize-lines: 3,7,11,15
struct platform_clos_info platform_clos_array[4] = {
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_0,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_1,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_2,
},
{
.clos_mask = 0xff,
.msr_index = MSR_IA32_L2_MASK_3,
},
};
#. Set up CLOS in the VM config. If you want a VM to use one of the CLOSs, you need to find its
configuration data and modify it. We will take SOS on sharing mode as an example. Its
configuration data can be found at ``hypervisor/arch/x86/configs/vm_config.c``
.. code-block:: none
:emphasize-lines: 5,6
struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
{
.type = SOS_VM,
.name = SOS_VM_CONFIG_NAME,
.guest_flags = SOS_VM_CONFIG_GUEST_FLAGS | CLOS_REQUIRED,
.clos = 1,
.memory = {
.start_hpa = 0x0UL,
.size = CONFIG_SOS_RAM_SIZE,
},
.os_config = {
.name = SOS_VM_CONFIG_OS_NAME,
},
},
};
#. Build the ACRN hypervisor and copy the artifact ``acrn.efi`` to the
``/boot/EFI/acrn`` directory, see :ref:`getting-started-building` for building instructions.
.. code-block:: none
$ make hypervisor BOARD=apl-up2 FIRMWARE=uefi
...
# these operations are done on UP2 board
$ mount /dev/mmcblk0p0 /boot
$ scp <acrn.efi-at-your-compile-PC> /boot/EFI/acrn
#. Restart the UP2 board