diff --git a/doc/developer-guides/hld/hld-hypervisor.rst b/doc/developer-guides/hld/hld-hypervisor.rst index 5001bdbfc..17fd4f4f9 100644 --- a/doc/developer-guides/hld/hld-hypervisor.rst +++ b/doc/developer-guides/hld/hld-hypervisor.rst @@ -13,3 +13,4 @@ Hypervisor high-level design I/O Emulation Physical Interrupt Timer + Virtual Interrupt diff --git a/doc/developer-guides/hld/hv-cpu-virt.rst b/doc/developer-guides/hld/hv-cpu-virt.rst index 291446546..c6be41871 100644 --- a/doc/developer-guides/hld/hv-cpu-virt.rst +++ b/doc/developer-guides/hld/hv-cpu-virt.rst @@ -514,6 +514,8 @@ that will trigger an error message and return without handling: Details of each vm exit reason handler are described in other sections. +.. _pending-request-handlers: + Pending Request Handlers ======================== diff --git a/doc/developer-guides/hld/hv-interrupt.rst b/doc/developer-guides/hld/hv-interrupt.rst index 995912204..0a8c41a1a 100644 --- a/doc/developer-guides/hld/hv-interrupt.rst +++ b/doc/developer-guides/hld/hv-interrupt.rst @@ -348,6 +348,8 @@ conditions: Request IRQ for different conditions +.. _ipi-management: + IPI Management ************** diff --git a/doc/developer-guides/hld/hv-virt-interrupt.rst b/doc/developer-guides/hld/hv-virt-interrupt.rst new file mode 100644 index 000000000..cd0997a7e --- /dev/null +++ b/doc/developer-guides/hld/hv-virt-interrupt.rst @@ -0,0 +1,259 @@ +.. _virtual-interrupt-hld: + +Virtual Interrupt +################# + +This section introduces ACRN guest virtual interrupt +management, which includes: + +- VCPU request for virtual interrupt kick off, +- vPIC/vIOAPIC/vLAPIC for virtual interrupt injection interfaces, +- physical-to-virtual interrupt mapping for a pass-thru device, and +- the process of VMX interrupt/exception injection. + +A guest VM never owns any physical interrupts. All interrupts received by +Guest OS come from a virtual interrupt injected by vLAPIC, vIOAPIC or +vPIC. Such virtual interrupts are triggered either from a pass-through +device or from I/O mediators in SOS via hypercalls. Section 3.8.6 +introduces how the hypervisor manages the mapping between physical and +virtual interrupts for pass-through devices. + +Emulation for devices is inside SOS user space device model, i.e., +acrn-dm. However for performance consideration: vLAPIC, vIOAPIC, and vPIC +are emulated inside HV directly. + +From guest OS point of view, vPIC is Virtual Wire Mode via vIOAPIC. The +symmetric I/O Mode is shown in :numref:`pending-virt-interrupt` later in +this section. + +The following command line +options to guest Linux affects whether it uses PIC or IOAPIC: + +- **Kernel boot param with vPIC**: add "maxcpu=0" Guest OS will use PIC +- **Kernel boot param with vIOAPIC**: add "maxcpu=1" (as long as not "0") + Guest OS will use IOAPIC. And Keep IOAPIC pin2 as source of PIC. + +vCPU Request for Interrupt Injection +************************************ + +The vCPU request mechanism (described in :ref:`pending-request-handlers`) is leveraged +to inject interrupts to a certain vCPU. As mentioned in +:ref:`ipi-management`, +physical vector 0xF0 is used to kick VCPU out of its VMX non-root mode, +used to make a request for virtual interrupt injection or other +requests such as flush EPT. + +The eventid supported for virtual interrupt injection includes: + +.. code-block:: c + + #define ACRN_REQUEST_EXCP 0 /* request for exception injection */ + #define ACRN_REQUEST_EVENT 1 /* vLAPIC event */ + #define ACRN_REQUEST_EXTINT 2 /* external interrupt from vPIC */ + #define ACRN_REQUEST_NMI 3 /* non-maskable interrupt */ + + +The *vcpu_make_request* is necessary for a virtual interrupt +injection. If the target vCPU is running under VMX non-root mode, it +will send an IPI to kick it out, which leads to an external-interrupt +VM-Exit. For some cases there is no need to send IPI when making a request, +because the CPU making the request itself is the target VCPU. For +example, the #GP exception request always happens on the current CPU when it +finds an invalid emulation has happened. An external interrupt for a pass-thru +device always happens on the VCPUs this device belonging to, so after it +triggers an external-interrupt VM-Exit, the current CPU is also the +target VCPU. + +Virtual LAPIC +************* + +LAPIC is virtualized for all Guest types: SOS and UOS. Given support by +the +physical processor, APICv Virtual Interrupt Delivery (VID) is enabled +and will support Posted-Interrupt feature. Otherwise, it will fall back to legacy +virtual interrupt injection mode. + +vLAPIC provides the same features as the native LAPIC: + +- Vector mask/unmask +- Virtual vector injections (Level or Edge trigger mode) to vCPU +- vIOAPIC notification of EOI processing +- TSC Timer service +- vLAPIC support CR8 to update TPR +- INIT/STARTUP handling + +vLAPIC APIs +=========== + +APIs are provided when an interrupt source from vLAPIC needs to inject +an interrupt, for example: + +- from LVT like LAPIC timer +- from vIOAPIC for a pass-thru device interrupt +- from an emulated device for a MSI + +These APIs will finish by making a request for *ACRN_REQUEST_EVENT.* + +lapic_intr_level + used by HV to issue a level triggered vector to vLAPIC. This vector + will update into IRR registers + +lapic_intr_edge + used by HV to issue an edge triggered vector to vLAPIC. This vector + will update into IRR registers + +lapic_intr_msi + Used by HV to issue a MSI to vLAPIC. This vector will update into IRR + registers + +vlapic_set_local_intr + Triggers the LAPIC local interrupt (LVT) 'vector' on 'cpu'. 'cpu' can + be set to -1 to trigger the interrupt on all CPUs. + +vlapic_pending_intr + Check if vLAPIC has pending vector and return the vector with highest + priority if there is pending virtual interrupts. + +vlapic_intr_accepted + Used in case of no APICv support. If a vector triggers the vLAPIC, + this vector is then moved from IRR registers to ISR registers. It + indicates the vector is accepted by Guest and Guest is processing it. + +EOI processing +============== + +EOI virtualization is enabled if APICv virtual interrupt delivery is +supported. Except for level triggered interrupts, VM will not exit in +case of EOI. + +In case of no APICv virtual interrupt delivery support, vLAPIC requires +EOI from Guest OS whenever a vector was acknowledged and processed by +guest. vLAPIC behavior is the same as HW LAPIC. Once an EOI is received, +it clears the highest priority vector in ISR and TMR, and updates PPR +status. vLAPIC will then notify vIOAPIC if the corresponding vector +comes from vIOAPIC. This only occurs for the level triggered interrupts. + +Virtual IOAPIC +************** + +vIOAPIC is emulated by HV when Guest accesses MMIO GPA range: +0xFEC00000-0xFEC01000. vIOAPIC for SOS should match to the native HW +IOAPIC Pin numbers. vIOAPIC for UOS provides 48 Pins. As the vIOAPIC is +always associated with vLAPIC, the virtual interrupt injection from +vIOAPIC will finally trigger a request for vLAPIC event by calling +vLAPIC APIs. + +**Supported APIs:** + +vioapic_assert_irq + assert a PIN according to active low or active high. + +vioapic_deassert_irq + de-assert a PIN according to assigned device or emulated device + status. + +vioapic_pulse_irq + trigger an edge interrupt. + +Virtual PIC +*********** + +vPIC is required for TSC calculation. Normally UOS will boot with +vIOAPIC and vPIC as the source of external interrupts to Guest. On every +VM Exit, HV will check if there are any pending external PIC interrupts. +vPIC APIs usage are similar to vIOAPIC. + +ACRN hypervisor emulates a vPIC for each VM based on IO range 0x20~0x21, +0xa0~0xa1 and 0x4d0~0x4d1. + +If an interrupt source from vPIC need to inject an interrupt, the +following APIs need be called, which will finally make a request for +*ACRN_REQUEST_EXTINT or ACRN_REQUEST_EVENT*: + +.. code-block:: c + + int vpic_assert_irq(struct vm *vm, uint32_t irq); + int vpic_deassert_irq(struct vm *vm, uint32_t irq); + int vpic_pulse_irq(struct vm *vm, uint32_t irq); + +The following APIs are used to query the vector needed to be injected and ACK +the service (means move the interrupt from request service - IRR to in +service - ISR): + +.. code-block:: c + + void vpic_pending_intr(struct vm *vm, uint32_t *vecptr); + void vpic_intr_accepted(struct vm *vm, uint32_t vector); + +Virtual Exception +***************** + +When doing emulation, an exception may need to be triggered in +hypervisor, for example: + +- if guest accesses an invalid vMSR register, +- hypervisor needs to inject a #GP, or +- during instruction emulation, an instruction fetch may access + a non-exist page from rip_gva, at that time a #PF need be injected. + +ACRN hypervisor implements virtual exception injection using these APIs: + +.. code-block:: c + + int vcpu_queue_exception(struct vcpu *vcpu, uint32_t vector, uint32_t err_code); + int vcpu_inject_gp(struct vcpu *vcpu, uint32_t err_code); + int vcpu_inject_pf(struct vcpu *vcpu, uint64_t addr, uint32_t err_code); + +ACRN hypervisor uses the *vcpu_inject_gp/vcpu_inject_pf* functions +to queue exception request, and follows SDM vol3 - 6.15, Table 6-5 to +generate double fault if the condition is met. + +Virtual Interrupt Injection +*************************** + +The source of virtual interrupts comes from either DM or assigned +devices. + +- **For SOS assigned devices**: as all devices are assigned to SOS + directly. Whenever there is a device's physical interrupt, the + corresponding virtual interrupts are injected to SOS via + vLAPIC/vIOAPIC. SOS does not use vPIC and does not have emulated + devices. See section 3.8.5 Device assignment. + +- **For UOS assigned devices**: only PCI devices could be assigned to + UOS. Virtual interrupt injection follows the same way as SOS. A + virtual interrupt injection operation is triggered when a + device's physical interrupt occurs. + +- **For UOS emulated devices**: DM (acrn-dm) is responsible for UOS + emulated devices' interrupt lifecycle management. DM knows when + an emulated device needs to assert a virtual IOPAIC/PIC Pin or + needs to send a virtual MSI vector to Guest. These logic is + entirely handled by DM. + +.. figure:: images/virtint-image64.png + :align: center + :name: pending-virt-interrupt + + Handle pending virtual interrupt + +Before APICv virtual interrupt delivery, a virtual interrupt can be +injected only if guest interrupt is allowed. There are many cases +that Guest ``RFLAGS.IF`` gets cleared and it would not accept any further +interrupts. HV will check for the available Guest IRQ windows before +injection. + +NMI is unmasked interrupt and its injection is always allowed +regardless of the guest IRQ window status. If current IRQ +windows is not present, HV would enable +``MSR_IA32_VMX_PROCBASED_CTLS_IRQ_WIN (PROCBASED_CTRL.bit[2])`` and +VM Enter directly. The injection will be done on next VM Exit once Guest +issues ``STI (GuestRFLAG.IF=1)``. + +Data structures and interfaces +****************************** + +There is no data structure exported to the other components in the +hypervisor for virtual interrupts. The APIs listed in the previous +sections are meant to be called whenever a virtual interrupt should be +injected or acknowledged. diff --git a/doc/developer-guides/hld/images/virtint-image64.png b/doc/developer-guides/hld/images/virtint-image64.png new file mode 100644 index 000000000..3c2186073 Binary files /dev/null and b/doc/developer-guides/hld/images/virtint-image64.png differ