acrn-hypervisor/doc/tutorials/using_cat_on_up2.rst

.. _using_cat_up2:

Using CAT on UP2
################

The UP Squared board `(UP2) <https://up-board.org/upsquared/specifications/>`_
is an x86 maker board based on the Intel Apollo Lake platform and supports
Cache Allocation Technology (CAT). With this feature, the usage of cache can be
restricted to each VM. ACRN hypervisor can do that by giving each VM a cache mask,
so that the VM will not evict the masked cache lines, as shown in :numref:
`cache-mask-vm`:

.. figure:: images/using_cat_up2.png
   :align: center
   :name: cache-mask-vm

   Cache usage with Cache Mask


CAT Support on ACRN
*******************

As described at `Intel (R) 64 and IA-32 Architectures Software Developer's Manual
<https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-3a-3b-3c-and-3d-system-programming-guide>`_,
chapter 17.19, volume 3. There are 3 steps to use CAT:

1. Detect the CAT capability.
2. Setup cache mask array MSRs, which is referred to as Class of Service (CLOS) array.
3. Select one of the CLOS array for the CPU, that will be the cache mask of the CPU.

ACRN integrates the usage of CAT into VM configuration. The CLOS array must be defined
in the board config source code, and the VM needs to specify which CLOS to use in its VM
config data. If the platform supports CAT, the CLOS array will be written to the CLOS MSRs
at CPU init time, and the VM will set CLOS for VCPU at VM launch time.
The details are shown in :numref:`acrn-cat-hld`:

.. figure:: images/acrn_cat_hld.png
   :align: center
   :name: acrn-cat-hld

   CAT Support on ACRN

Tuning CAT in HV debug shell
############################

From the ACRN HV debug shell, you can use ``cpuid``, ``wrmsr/rdmsr`` debug commands to
enumerate CAT capability and tune CAT parameters. You can use the UP2 board's serial port
for the HV shell (refer to :ref:`getting-started-up2` for setup instructions).

#. Check CAT ability with ``cupid``. First run ``cpuid 0x10 0x0``, the return value ebx[bit 2]
   reports the L2 CAT is supported. Then run ``cpuid 0x10 0x2`` to query L2 CAT capability,
   the return value eax[bit 4:0] reports the cache mask has 8 bit, and edx[bit 15:0] reports 4 CLOS are
   supported, as shown below:

   .. code-block:: none

      ACRN:\>cpuid 0x10 0x0
      cpuid leaf: 0x10, subleaf: 0x0, 0x0:0x4:0x0:0x0

      ACRN:\>cpuid 0x10 0x2
      cpuid leaf: 0x10, subleaf: 0x2, 0x7:0x0:0x0:0x3

#. Check PCPU IDs of each VM, the ``vcpu_list`` shows that VM0 is running on PCPU0,
   and VM1 is running on PCPU1:

   .. code-block:: none

      ACRN:\>vcpu_list

      VM ID    PCPU ID    VCPU ID    VCPU ROLE    VCPU STATE
      =====    =======    =======    =========    ==========
        0         0          0        PRIMARY       Running
        1         1          0        PRIMARY       Running

#. Set CLOS with ``wrmsr <reg_num> <value>``, we want VM1 to use the lower 6 ways of cache,
   so CLOS0 is set to 0xc0 for the upper 2 ways, and CLOS1 is set to 0x3f for the lower
   6 ways:

   .. code-block:: none

      ACRN:\>wrmsr  -p1 0xd10  0xc0
      ACRN:\>wrmsr  -p1 0xd11  0x3f

#. Attach COS1 to PCPU1. Because MSR IA32_PQR_ASSOC [bit 63:32], we'll write 0x100000000
   to it to use CLOS1

   .. code-block:: none

      ACRN:\>wrmsr   -p1   0xc8f    0x100000000

Configure CAT for VM with VM Configuration
##########################################

#. CAT on ACRN can be enabled and configured by modifying source code, so the first
   step is to clone the ACRN source code (if you haven't already):

   .. code-block:: none

      $ git clone https://github.com/projectacrn/acrn-hypervisor.git
      $ cd acrn-hypervisor/

#. The predefined cache masks can be found at
   ``hypervisor/arch/x86/configs/$(CONFIG_BOARD)/board.c``, for UP2 board, that is
   ``hypervisor/arch/x86/configs/apl-up2/board.c``, you can change the mask values,
   but note that the CLOS mask must have continuous bits, or a #GP fault can be triggered.

   .. code-block:: none
      :emphasize-lines: 3,7,11,15

      struct platform_clos_info platform_clos_array[4] = {
              {
                      .clos_mask = 0xff,
                      .msr_index = MSR_IA32_L2_MASK_0,
              },
              {
                      .clos_mask = 0xff,
                      .msr_index = MSR_IA32_L2_MASK_1,
              },
              {
                      .clos_mask = 0xff,
                      .msr_index = MSR_IA32_L2_MASK_2,
              },
              {
                      .clos_mask = 0xff,
                      .msr_index = MSR_IA32_L2_MASK_3,
              },
      };

#. Set up CLOS in the VM config. If you want a VM to use one of the CLOSs, you need to find its
   configuration data and modify it. We will take SOS on sharing mode as an example. Its
   configuration data can be found at ``hypervisor/arch/x86/configs/vm_config.c``

   .. code-block:: none
      :emphasize-lines: 5,6

      struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
              {
                      .type = SOS_VM,
                      .name = SOS_VM_CONFIG_NAME,
                      .guest_flags = SOS_VM_CONFIG_GUEST_FLAGS | CLOS_REQUIRED,
                      .clos = 1,
                      .memory = {
                              .start_hpa = 0x0UL,
                              .size = CONFIG_SOS_RAM_SIZE,
                      },
                      .os_config = {
                              .name = SOS_VM_CONFIG_OS_NAME,
                      },
              },
      };

#. Build the ACRN hypervisor and copy the artifact ``acrn.efi`` to the
   ``/boot/EFI/acrn`` directory, see :ref:`getting-started-building` for building instructions.

   .. code-block:: none

      $ make hypervisor BOARD=apl-up2 FIRMWARE=uefi
      ...

      # these operations are done on UP2 board
      $ mount /dev/mmcblk0p0 /boot
      $ scp <acrn.efi-at-your-compile-PC> /boot/EFI/acrn

#. Restart the UP2 board