diff --git a/doc/api/GVT-g_api.rst b/doc/api/GVT-g_api.rst index 45bf3581b..6176a830d 100644 --- a/doc/api/GVT-g_api.rst +++ b/doc/api/GVT-g_api.rst @@ -83,6 +83,7 @@ responses to user space modules, notified by vIRQ injections. .. kernel-doc:: include/linux/vhm/acrn_vhm_mm.h :functions: update_memmap_attr +.. _MPT_interface: ACRN-GT mediated pass-through (MPT) interface ************************************************** @@ -143,6 +144,7 @@ This section describes the wrap functions: intel_gvt_hypervisor_unpause_domain intel_gvt_hypervisor_dom0_ready +.. _intel_gvt_ops_interface: GVT-g intel_gvt_ops interface ***************************** @@ -183,6 +185,8 @@ in the `ACRN kernel GitHub repo`_ intel_gvt_activate_vgpu intel_gvt_deactivate_vgpu +.. _sysfs_interface: + ACRN-GT sysfs interface *********************** diff --git a/doc/developer-guides/GVT-G-porting.rst b/doc/developer-guides/GVT-G-porting.rst new file mode 100644 index 000000000..9698d3a0b --- /dev/null +++ b/doc/developer-guides/GVT-G-porting.rst @@ -0,0 +1,174 @@ +.. _GVT-G-porting: + +GVT-G Enabling and Porting Guide +################################ + +Introduction +************ + +GVT-g is Intel's Open Source GPU virtualization solution, up-streamed to +the Linux kernel. Its implementation over KVM is named KVMGT, over Xen +is named XenGT, and over ACRN is named ACRN-GT. GVT-g can export +multiple virtual-GPU (vGPU) instances for virtual machine (VM) system. A +VM can be assigned one instance of a vGPU. The guest OS graphic driver +needs only minor modifications to drive the vGPU adapter in a VM. Every +vGPU instance adopts the full HW GPU's acceleration capability for +media, 3D renderingi, and display. + +ACRN-GT refers to the glue layer between the ACRN hypervisor and GVT-g +core device model. It works as the agent of hypervisor-related services. +It is the only layer that needs to be rewritten when porting GVT-g to +other specific hypervisors. + +For simplicity, in the rest of this document, the term GVT is used to +refer to the core device model component of GVT-g, specifically +corresponding to ``gvt.ko`` when build as a module. + +Purpose of this document +************************ + +This document explains the relationship between components of GVT-g in +the ACRN hypervisor, shows how to enable GVT-g on ACRN, and guides +developers porting GVT-g to work on other hypervisors. + +This document describes: + +- the overall components of GVT-g +- interaction interface of each components +- core interaction scenarios + +APIs of each component interface can be found in the :ref:`GVT-g_api` +documentation. + + +Overall Components +****************** + +For the GVT-g solution for the ACRN hypervisor, there are two +key modules: ACRN-GT and GVT: + +ACRN-GT module + Compiled from ``drivers/gpu/drm/i915/gvt/acrn_gvt.c``, the ACRN-GT + module acts as a glue layer between the ACRN hypervisor and the + interface to the ACRN-DM in user space. + + ACRN-GT is the agent of hypervisor related services, including I/O trap + request, IRQ injection, address translation, VM controls, etc. It also + listens to ACRN hypervisor in ``acrngt_emulation_thread``, and informs GVT + module of I/O traps. + + It calls into the GVT module's :ref:`intel_gvt_ops_interface` to invoke + Device Model's routines, and receives request from GVT module through + the :ref:`MPT_interface`. + + User space programs, such as ACRN-DM, communicate with ACRN-GT through + the :ref:`sysfs_interface` by writing to sysfs node + ``/sys/kernel/gvt/control/create_gvt_instance``. + + This is the only module that needs to be rewritten when porting to + another embedded device hypervisor. + +GVT module + This Device Model service is the central part of all + GVT-g components. It receives workloads from each vGPU, shadows the + workloads, and dispatches the workloads to Dom0's i915 module to deliver + workloads to real hardware. It also emulates the virtual display to each + VM. + +VHM module + This is a kernel module that requires an interrupt (vIRQ) number, + and exposes APIs to external kernel modules such as GVT-g, and the + virtIO BE service running in kernel space. It exposes a char device node + in user space, and only interacts with the DM. The DM routes I/O + requests and responses between other modules to and from the VHM module + via the char device. DM may use the VHM for hypervisor service + (including remote memory map), and VHM may directly service the request + such as for the remote memory map, or invoking hypercall. It also sends + I/O responses to user space modules, notified by vIRQ injections. + + +.. figure:: images/GVT-G-porting-image1.png + :width: 700px + :align: center + :name: GVT-g_components + + GVT-g components and interfaces + +Core scenario interaction sequences +*********************************** + + +vGPU creation scenario +====================== + +In this scenario, ACRN-GT receives a create request from ACRN-DM. It +calls GVT's :ref:`intel_gvt_ops_interface` to inform GVT of vGPU +creation. This interface sets up all vGPU resources such as MMIO, GMA, +PVINFO, GTT, DISPLAY, and Execlists, and calls back to the ACRN-GT +module through the :ref:`MPT_interface` ``attach_vgpu``. Then, the +ACRN-GT module sets up an I/O request server and asks to trap the PCI +configure space of the vGPU (virtual device 0:2:0) via VHM's APIs. +Finally, the ACRN-GT module launches a ACRN-GT emulation thread to +listen to I/O trap notifications from HVM and ACRN hypervisor. + +vGPU destroy scenario +===================== + +In this scenario, ACRN-GT receives a destroy request from ACRN-DM. It +calls GVT's :ref:`intel_gvt_ops_interface` to inform GVT of the vGPU destroy +request, and cleans up all vGPU resources. + +vGPU pci configure space write scenario +======================================= + +ACRN traps the vGPU's PCI config space write, notifies ACRN-GT's +``acrngt_emulation_thread``, which calls ``acrngt_hvm_pio_emulation`` to +handle all I/O trap notifications. This routine calls the GVT's +:ref:`intel_gvt_ops_interface` ``emulate_cfg_write`` to emulate the vGPU PCI +config space write: + +#. If it's BAR0 (GTTMMIO) write, turn on/off GTTMMIO trap, according to + the write value. +#. If it's BAR1 (Aperture) write, maps/unmaps vGPU's aperture to its + corresponding part in the host's aperture. +#. Otherwise, write to the virtual PCI configuration space of the vGPU. + +pci configure space read scenario +================================= + +Call sequence is almost the same as the write scenario above, +but instead it calls the GVT's :ref:`intel_gvt_ops_interface` +``emulate_cfg_read`` to emulate the vGPU PCI config space read. + +GGTT read/write scenario +======================== + +GGTT's trap is set up in the pci configure space write +scenario above. + +MMIO read/write scenario +======================== + +MMIO's trap is set up in the pci configure space write +scenario above. + +PPGTT write protection page set/unset scenario +============================================== + +PPGTT write protection page is set by calling ``acrn_ioreq_add_iorange`` +with range type as ``REQ_WP``, and trap its write to device model while +allowing read without trap. + +PPGTT write protection page is unset by calling ``acrn_ioreq_del_range``. + +PPGTT write protection page write +================================= + +In the VHM module, ioreq for PPGTT WP and MMIO trap is the same. It will +also be trapped into the routine ``intel_vgpu_emulate_mmio_write()``. + +API details +*********** + +APIs of each component interface can be found in the :ref:`GVT-g_api` +documentation. diff --git a/doc/developer-guides/images/GVT-G-porting-image1.png b/doc/developer-guides/images/GVT-G-porting-image1.png new file mode 100644 index 000000000..8174f9708 Binary files /dev/null and b/doc/developer-guides/images/GVT-G-porting-image1.png differ diff --git a/doc/developer-guides/index.rst b/doc/developer-guides/index.rst index 605e03a42..6169e5561 100644 --- a/doc/developer-guides/index.rst +++ b/doc/developer-guides/index.rst @@ -24,6 +24,7 @@ specific areas within the ACRN hypervisor system. ACPI-virt-hld.rst APL_GVT-g-hld.rst uart-virtualization.rst + GVT-G-porting.rst Contributing to the project *************************** diff --git a/doc/glossary.rst b/doc/glossary.rst index dfa736773..6689ff2fb 100644 --- a/doc/glossary.rst +++ b/doc/glossary.rst @@ -20,7 +20,7 @@ Glossary of Terms A user mode device model application running in Service OS to provide device emulations in ACRN hypervisor. - aperture + aperture, Low GM CPU-visible graphics memory API @@ -36,6 +36,9 @@ Glossary of Terms BIOS Basic Input/Output System. + Dom0 i915 + The Intel Graphics driver running in Domain 0 + ELSP GPU's ExecList submission port @@ -71,6 +74,9 @@ Glossary of Terms GVT-s Virtual shared graphics acceleration (multiple VMs to one physical GPU) + Hidden GM, High GM + Hidden or High graphics memory, not visible to the CPU. + I2C Inter-Integrated Circuit