acrn-hypervisor/doc/tutorials/rdt_configuration.rst

.. _rdt_configuration:

Enable Intel Resource Director Technology (RDT) Configurations
###############################################################

About Intel Resource Director Technology (RDT)
**********************************************

On x86 platforms that support Intel Resource Director Technology (RDT)
allocation features, the ACRN hypervisor can partition the shared cache among
VMs to minimize performance impacts on higher-priority VMs, such as real-time
VMs (RTVMs). “Shared cache” refers to cache that is shared among multiple CPU
cores. By default, VMs running on these cores are configured to use the entire
cache, effectively sharing the cache among all VMs without any partitions. This
design may cause too many cache misses for applications running in
higher-priority VMs, negatively affecting their performance. The ACRN hypervisor
can help minimize cache misses by isolating a portion of the shared cache for
a specific VM.

ACRN supports the following features:

* Cache Allocation Technology (CAT)
* Code and Data Prioritization (CDP)
* Virtual Cache Allocation Technology (vCAT)

The CAT support in the hypervisor isolates a portion of the cache for a VM from
other VMs. Generally, certain cache resources are allocated for the RTVMs to
reduce performance interference by other VMs attempting to use the same cache.

The CDP feature in RDT is an extension of CAT that enables separate control over
code and data placement in the cache. The CDP support in the hypervisor isolates
a portion of the cache for code and another portion for data for the same VM.

ACRN also supports the virtualization of CAT, referred to as vCAT. With
vCAT enabled, the hypervisor presents CAT to a selected set of VMs to allow the
guest OSes to further isolate the cache used by higher-priority processes in
those VMs.

Dependencies and Constraints
*****************************

Consider the following dependencies and constraints:

* The hardware must support RDT in order for ACRN to enable RDT support in the
  hypervisor.

* The cache must be shared cache (cache shared across multiple CPU cores), as
  opposed to private cache (cache that is owned by only one CPU core). If the
  cache is private, CAT, CDP, and vCAT have no benefit because the cache is
  already exclusively used by one core. For this reason, the ACRN Configurator
  will not allow you to configure private cache.

* The ACRN Configurator relies on the board configuration file to provide CAT
  information that it can use to display configuration parameters. On Tiger Lake
  systems, L3 CAT, also known as LLC CAT, is model specific and
  non-architectural. For these reasons, the Board Inspector doesn't detect LLC
  CAT, and therefore doesn't provide LLC CAT information in the board
  configuration file even if the board has LLC CAT capabilities. The Board
  Inspector offers a way to manually add LLC CAT information to the board
  configuration file via a command-line option described in
  :ref:`board_inspector_tool`. Run the Board Inspector with the command-line
  option, then import the board configuration file into the ACRN Configurator.

* The guest OS in a VM with vCAT enabled requires utilities in that OS for
  further cache allocation configurations. An example is the `resctrl
  <https://docs.kernel.org/x86/resctrl.html>`__ framework in Linux.

Configuration Overview
**********************

You can allocate cache to each VM at the virtual CPU (vCPU) level. For example,
you can create a post-launched real-time VM and assign three physical CPU
cores to it. ACRN assigns a vCPU ID to each physical CPU. Furthermore, you can
specify a vCPU as a real-time vCPU. Then you can allocate a portion of the cache
to the real-time vCPU and allocate the rest of the cache to be shared among the
other vCPUs. This type of configuration allows the real-time vCPU to use its
assigned cache without interference from the other vCPUs, thus improving the
performance of applications running on the real-time vCPU. The following
documentation is a general overview of the configuration process.

The :ref:`acrn_configurator_tool` provides a user interface to help you allocate
cache to vCPUs. The configuration process requires setting VM parameters, then
allocating cache to the VMs via an interface in the hypervisor parameters. This
documentation presents the configuration process as a linear flow, but in
reality you may find yourself moving back and forth between setting the
hypervisor parameters and the VM parameters until you are satisfied with the
entire configuration.

For a real-time VM, you must set the following parameters in the VM's **Basic
Parameters**: 

* **VM type**: Select **Real-time**.

* **pCPU ID**: Select the physical CPU affinity for the VM.

* **Virtual CPU ID**: Note the vCPU ID that the tool assigns to each physical
  CPU. You will need to know the ID when you are ready to allocate cache.

* **Real-time vCPU**: Select the Real-time vCPU check box next to each real-time
  vCPU. The ACRN Configurator uses this information to create default cache
  configurations, as you will see later in this documentation. If you change the
  VM type from Real-time to Standard, the ACRN Configurator disables the
  Real-time vCPU check box.

.. image:: images/configurator-rt01.png
   :align: center
   :class: drop-shadow

To use vCAT for the VM, you must also set the following parameters in the VM's
**Advanced Parameters**:

* **Maximum virtual CLOS**: Select the maximum number of virtual CLOS masks.
  This parameter defines the number of cache chunks that you will see in the
  hypervisor parameters.

* Select **VM Virtual Cache Allocation Tech**.

.. image:: images/configurator-vcat01.png
   :align: center
   :class: drop-shadow

Next, you can enable Intel RDT features in **Hypervisor Global Settings >
Advanced Parameters > Memory Isolation for Performance**. You can enable one of
the following combinations of features:

* Cache Allocation Technology (CAT) alone

* Cache Allocation Technology plus Code and Data Prioritization (CDP)

* Cache Allocation Technology plus Virtual Cache Allocation Technology (vCAT)

The following figure shows Cache Allocation Technology enabled:

.. image:: images/configurator-cache01.png
   :align: center
   :class: drop-shadow

When CDP or vCAT is enabled, CAT must be enabled too. The tool selects CAT if it's not already selected.

.. image:: images/configurator-cache02.png
   :align: center
   :class: drop-shadow

CDP and vCAT can't be enabled at the same time, so the tool clears the vCAT check box when CDP is selected and vice versa.

Based on your selection, the tool displays the available cache in tables.
Example:

.. image:: images/configurator-cache03.png
   :align: center
   :class: drop-shadow

The table title shows important information:

* Cache level, such as Level 3 (L3) or Level 2 (L2)

* Physical CPU cores that can access the cache

The above example shows an L2 cache table. VMs assigned to any CPU cores 2-6 can
have cache allocated to them.

The table's y-axis shows the names of all VMs that are assigned to the CPU cores
noted in the table title, as well as their vCPU IDs. The table categorizes the
vCPUs as either standard or real-time. The real-time vCPUs are those that are
set as real-time in the VM's parameters. All other vCPUs are considered
standard. The above example shows one real-time vCPU (VM1 vCPU 2) and two
standard vCPUs (VM0 vCPU 2 and 6).

.. note::

   The Service VM is automatically assigned to all CPUs, so it appears in the standard category in all cache tables. 

The table's x-axis shows the number of available cache chunks. You can see the
size of each cache chunk in the note below the table. In the above example, 20
cache chunks are available to allocate to the VMs, and each cache chunk is 64KB.
All cache chunks are yellow, which means all of them are allocated to all VMs.
All VMs share the entire cache.

The **Apply basic real-time defaults** button creates a basic real-time
configuration if real-time vCPUs exist. If there are no real-time vCPUs, the
button will not do anything.

If you select Cache Allocation Technology (CAT) alone, the **Apply basic
real-time defaults** button allocates a different cache chunk to each real-time
vCPU, making sure it doesn't overlap the cache of any other vCPU. The rest of
the cache is shared among the standard vCPUs. In the following example, only VM1
vCPU 2 can use cache chunk19, while all other vCPUs share the rest of the cache.

.. image:: images/configurator-cache04.png
   :align: center
   :class: drop-shadow

If you select CAT with Code and Data Prioritization, you can allocate different
cache chunks to code or data on the same vCPU. The **Apply basic real-time
defaults** button allocates one cache chunk to code on the real-time vCPU and a
different cache chunk to data on the same vCPU, making sure the cache chunks
don't overlap any others. In the following example, VM1 vCPU 2 can use cache
chunk19 for code and chunk18 for data, while all other vCPUs share the rest of
the cache.

.. image:: images/configurator-cache05.png
   :align: center
   :class: drop-shadow

To further customize the cache allocation, you can drag the right or left edges
of the yellow boxes to cover the cache chunks that you want to allocate to
specific VMs.

.. note::

   If you have a real-time VM, ensure its cache chunks do not overlap with any
   other VM's cache chunks.

The tool helps you create valid configurations based on the underlying platform
architecture. For example, it is only possible to assign consecutive cache
chunks to a vCPU; there can be no gaps. Also, a vCPU must have access to at
least one cache chunk.

Learn More
**********

For details on the ACRN RDT high-level design, see :ref:`hv_rdt`.

For details about RDT, see
`Intel 64 and IA-32 Architectures Software Developer's Manual (SDM), Volume 3,
(Section 17.19 Intel Resource Director Technology Allocation Features)
<https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html>`_.

For details on the ACRN vCAT high-level design, see :ref:`hv_vcat`.