diff --git a/doc/tutorials/sriov_virtualization.rst b/doc/tutorials/sriov_virtualization.rst index 77375fdb1..c8336f92d 100644 --- a/doc/tutorials/sriov_virtualization.rst +++ b/doc/tutorials/sriov_virtualization.rst @@ -4,174 +4,166 @@ SR-IOV Virtualization ===================== SR-IOV (Single Root Input/Output Virtualization) can isolate PCIe devices -and make the devices to have good performance, close to bare-metal. The -SR-IOV consists of two basic units, one is called PF(Physical Function) -that supports SR-IOV PCIe extended capability and manages entire -physical device, another is called VF (Virtual Function) and is a -"lightweight" PCIe function which is meant to be given to a VM as a -pass-through device. +to improve performance that is similar to bare-metal levels. SR-IOV consists +of two basic units: PF (Physical Function), which supports SR-IOV PCIe +extended capability and manages entire physical devices; and VF (Virtual +Function), a "lightweight" PCIe function which is a passthrough device for +VMs. -For more details, please refer to PCI Express Base Specification Revision 4.0, -Version 1.0 `PCI-SIG Specification ` Chapter 9. +For details, refer to Chapter 9 of PCI-SIG's `PCI Express Base SpecificationRevision 4.0, Version 1.0 `_. SR-IOV Architectural Overview ----------------------------- .. figure:: images/sriov-image1.png :align: center - :width: 900px :name: SR-IOV-architecture-overview SR-IOV Architectural Overview -- **SI** - System Image known as a VM. +- **SI** - A System Image known as a VM. -- **VI** - Virtualization Intermediary known as Hypervisor. +- **VI** - A Virtualization Intermediary known as a hypervisor. -- **SR-PCIM** - Single Root PCI Manager is a software entity for SR-IOV - management. +- **SR-PCIM** - A Single Root PCI Manager; it is a software entity for + SR-IOV management. -- **PF** - A PF is a PCIe Function that supports the SR-IOV capability +- **PF** - A PCIe Function that supports the SR-IOV capability and is accessible to an SR-PCIM, a VI, or an SI. -- **VF** - A VF is a “light-weight” PCIe Function that is directly - accessible by an SI. +- **VF** - A “light-weight” PCIe Function that is directly accessible by an + SI. SR-IOV Extended Capability -------------------------- The SR-IOV Extended Capability defined here is a PCIe extended -capability that must be implemented in each PF device that supports -SR-IOV feature. This Capability is used to describe and control a PF’s +capability that must be implemented in each PF device that supports the +SR-IOV feature. This capability is used to describe and control a PF’s SR-IOV Capabilities. .. figure:: images/sriov-image2.png :align: center - :width: 900px :name: SR-IOV-extended-capability SR-IOV Extended Capability - **PCIe Extended Capability ID** - 0010h. -- **SR-IOV Capabilities** - VF Migration Capable and ARI Capable. +- **SR-IOV Capabilities** - VF Migration-Capable and ARI-Capable. -- **SR-IOV Control** - Enable/Disable VFs, VF migration state query. +- **SR-IOV Control** - Enable/Disable VFs; VF migration state query. -- **SR-IOV Status** - VF Migration Status +- **SR-IOV Status** - VF Migration Status. -- **InitialVFs** - Indicates to SR-PCIM the number of VFs that are - initially associated with the PF +- **InitialVFs** - Indicates to the SR-PCIM the number of VFs that are + initially associated with the PF. -- **TotalVFs** - Indicates the maximum number of VFs that could be - associated with the PF +- **TotalVFs** - Indicates the maximum number of VFs that can be + associated with the PF. - **NumVFs** - Controls the number of VFs that are visible. *NumVFs* <= *InitialVFs* = *TotalVFs*. -- **Function Link Dependency** - The field is used to describe these +- **Function Link Dependency** - The field used to describe dependencies between PFs. VF dependencies are the same as the dependencies of their associated PFs. -- **First VF Offset** - It is a constant and defines the Routing ID +- **First VF Offset** - A constant that defines the Routing ID offset of the first VF that is associated with the PF that contains - this Capability structure + this Capability structure. -- **VF Stride** - It defines the Routing ID offset from one VF to the +- **VF Stride** - Defines the Routing ID offset from one VF to the next one for all VFs associated with the PF that contains this Capability structure. -- **VF Device ID** - This field contains the Device ID that should be +- **VF Device ID** - The field that contains the Device ID that should be presented for every VF to the SI. -- **Supported Page Sizes** - This field indicates the page sizes +- **Supported Page Sizes** - The field that indicates the page sizes supported by the PF. -- **System Page Size** - This field defines the page size the system +- **System Page Size** - The field that defines the page size the system will use to map the VFs’ memory addresses. Software must set the value of the *System Page Size* to one of the page sizes set in the - *Supported Page Sizes* field + *Supported Page Sizes* field. -- **VF BARs** - These fields must define the VF’s Base Address +- **VF BARs** - Fields that must define the VF’s Base Address Registers (BARs). These fields behave as normal PCI BARs. -- **VF Migration State Array Offset** - This register shall contain a +- **VF Migration State Array Offset** - Register that contains a PF BAR relative pointer to the VF Migration State Array. -- **VF Migration State Array** – It is located using the VF Migration +- **VF Migration State Array** – Located using the VF Migration State Array Offset register of the SR-IOV Capability block. -For more details, please refer to *PCI Express Base Specification -Revision 4.0, Version 1.0 Chapter 9.3.3* +For details, refer to the *PCI Express Base Specification Revision 4.0, Version 1.0 Chapter 9.3.3*. SR-IOV Architecture In ACRN --------------------------- .. figure:: images/sriov-image3.png :align: center - :width: 900px :name: SR-IOV-architecure-in-acrn SR-IOV Architectural in ACRN -1. Hypervisor detects SR-IOV capable PCIe device in physical PCI device - enumeration phase. +1. A hypervisor detects a SR-IOV capable PCIe device in the physical PCI + device enumeration phase. -2. Hypervisor intercepts the PF’s SR-IOV capability accessing to - enable/disable VF devices based on *VF\_ENABLE* state, and all - read/write requests for PF device are passthrough to PF physical +2. The hypervisor intercepts the PF’s SR-IOV capability and accesses whether + to enable/disable VF devices based on the *VF\_ENABLE* state. All + read/write requests for a PF device passthrough to the PF physical device. -3. Hypervisor waits for 100ms after *VF\_ENABLE* is set and initialize - VF devices. The differences between normal passthrough device and - SR-IOV VF device are physical device detection, BARs and MSIx - initializations. Hypervisor uses *Subsystem Vendor ID* to detect - SR-IOV VF physical device instead *Vendor ID* since there is no valid - *Vendor ID* for SR-IOV VF physical device, the VF BARs are - initialized by its associated PF’s SR-IOV capabilities not PCI - standard BAR registers, MSIx mapping base address is also from PF’s - SR-IOV capabilities not PCI standard BAR registers. +3. The hypervisor waits for 100ms after *VF\_ENABLE* is set and initializes + VF devices. The differences between a normal passthrough device and + SR-IOV VF device are physical device detection, BARs, and MSIx + initialization. The hypervisor uses *Subsystem Vendor ID* to detect the + SR-IOV VF physical device instead of *Vendor ID* since no valid + *Vendor ID* exists for the SR-IOV VF physical device. The VF BARs are + initialized by its associated PF’s SR-IOV capabilities, not PCI + standard BAR registers. The MSIx mapping base address is also from the + PF’s SR-IOV capabilities, not PCI standard BAR registers. SR-IOV Passthrough VF Architecture In ACRN ------------------------------------------ .. figure:: images/sriov-image4.png :align: center - :width: 900px :name: SR-IOV-vf-passthrough SR-IOV VF Passthrough Architecture In ACRN -1. The SR-IOV VF device needs to bind PCI-stud driver instead - vendor-specific VF driver before passthrough the device. +1. The SR-IOV VF device needs to bind the PCI-stud driver instead of the + vendor-specific VF driver before the device passthrough. -2. User configures ``acrn-dm`` boot parameter with passthrough SR-IOV VF - device. When User VM starts, ``acrn-dm`` invokes a hypercall to set the - *vdev-VF0* device in User VM. +2. The user configures the ``acrn-dm`` boot parameter with the passthrough + SR-IOV VF device. When the User VM starts, ``acrn-dm`` invokes a + hypercall to set the *vdev-VF0* device in the User VM. -3. Hypervisor emulates *Device ID/Vendor ID* and *Memory Space Enable - (MSE*) in configuration space for an assigned SR-IOV VF device. The - assigned VF *Device ID* comes from its associated PF’s capability, - *Vendor ID* is same with PF’s *Vendor ID* and *MSE* is always set - when reading the SR-IOV VF device *CONTROL* register +3. The hypervisor emulates *Device ID/Vendor ID* and *Memory Space Enable + (MSE)* in the configuration space for an assigned SR-IOV VF device. The + assigned VF *Device ID* comes from its associated PF’s capability. The + *Vendor ID* is the same as the PF’s *Vendor ID* and the *MSE* is always + set when reading the SR-IOV VF device's *CONTROL* register. -4. Vendor-specific VF driver in target VM probes the assigned SR-IOV VF - device. +4. The vendor-specific VF driver in the target VM probes the assigned SR-IOV + VF device. SR-IOV Initialization Flow -------------------------- .. figure:: images/sriov-image5.png :align: center - :width: 900px :name: SR-IOV-init-flow SR-IOV Initialization Flow When a SR-IOV capable device is initialized, all access to the -configuration space will pass through to the physical device directly. -Service VM can identify all capabilities of the device from SR-IOV extended -capability then create sysfs node for SR-IOV management. +configuration space will passthrough to the physical device directly. +The Service VM can identify all capabilities of the device from the SR-IOV +extended capability and then create an *sysfs* node for SR-IOV management. SR-IOV VF Enable Flow --------------------- @@ -183,94 +175,87 @@ SR-IOV VF Enable Flow SR-IOV VF Enable Flow -Application enables n VF devices via a SR-IOV PF device *sysfs* node. -Hypervisor intercepts all SR-IOV capability accesses and check -*VF\_ENABLE* state, if the *VF\_ENABLE* is set, hypervisor creates n -virtual devices after 100ms that VF physical devices have enough time to -be created. Service VM waits 100ms then only accesses the first VF device’s -configuration space including *Class Code, Reversion ID, Subsystem -Vendor ID, Subsystem ID,* Service VM uses the first VF device information to -initialize subsequent VF devices. +The application enables n VF devices via a SR-IOV PF device *sysfs* node. +The hypervisor intercepts all SR-IOV capability access and checks the +*VF\_ENABLE* state. If *VF\_ENABLE* is set, the hypervisor creates n +virtual devices after 100ms so that VF physical devices have enough time to +be created. The Service VM waits 100ms and then only accesses the first VF +device’s configuration space including *Class Code, Reversion ID, Subsystem +Vendor ID, Subsystem ID*. The Service VM uses the first VF device +information to initialize subsequent VF devices. SR-IOV VF Disable Flow ---------------------- .. figure:: images/sriov-image7.png :align: center - :width: 900px :name: SR-IOV-disable-flow SR-IOV VF Disable Flow -Application disables SR-IOV VF devices by writing zero to SR-IOV PF -device *sysfs* node. Hypervisor intercepts all SR-IOV capability -accesses and check *VF\_ENABLE* state, if *VF\_ENABLE* is clear, -hypervisor makes VF virtual devices invisible from Service VM that all accesses -for VF will return 0xFFFFFFFF as an error, VF physical devices are -removed within 1s of *VF\_ENABLE* is clear. +The application disables SR-IOV VF devices by writing zero to the SR-IOV PF +device *sysfs* node. The hypervisor intercepts all SR-IOV capability +accesses and checks the *VF\_ENABLE* state. If *VF\_ENABLE* is clear, the +hypervisor makes VF virtual devices invisible from the Service VM so that all +access to VF devices will return 0xFFFFFFFF as an error. The VF physical +devices are removed within 1s of when *VF\_ENABLE* is clear. SR-IOV VF Assignment Policy --------------------------- .. figure:: images/sriov-image8.png :align: center - :width: 900px :name: SR-IOV-vf-assignment SR-IOV VF Assignment -1. All SR-IOV PF devices are managed by Service VM. +1. All SR-IOV PF devices are managed by the Service VM. -2. Currently, SR-IOV PF can’t passthrough to User VM. +2. Currently, the SR-IOV PF cannot passthrough to the User VM. -3. All VFs can be passthrough to User VM, but we do not recommend - passthrough to high privilege VMs, because PF device may impact - assigned VFs including functionality and stability. +3. All VFs can passthrough to the User VM, but we do not recommend + a passthrough to high privilege VMs because the PF device may impact + the assigned VFs' functionality and stability. SR-IOV Usage Guide In ACRN -------------------------- -We use Intel 82576 NIC as an example in here and only support LaaG -(Linux as a Guest). +We use the Intel 82576 NIC as an example in the following instructions. We +only support LaaG (Linux as a Guest). -1. Please ensure the 82576 VF driver is compiled into User VM Kernel - (Set *CONFIG\_IGBVF=y* in Kernel Config). +1. Ensure that the 82576 VF driver is compiled into the User VM Kernel + (set *CONFIG\_IGBVF=y* in the Kernel Config). -2. When the Service VM boot up, “\ *lspci -v*\ ” command can show Intel 82576 - NIC devices have SR-IOV capability and their PF drivers are ``igb``. +#. When the Service VM boots up, the ``\ *lspci -v*\`` command indicates + that the Intel 82576 NIC devices have SR-IOV capability and their PF + drivers are ``igb``. -.. figure:: images/sriov-image9.png - :align: center - :width: 900px - :name: 82576-pf + .. figure:: images/sriov-image9.png + :align: center + :name: 82576-pf - 82576 SR-IOV PF devices + 82576 SR-IOV PF devices -1. Input "\ *echo n > /sys/class/net/enp109s0f0/device/sriov\_numvfs*\ " - command in Service VM to enable n VF devices for the first PF - device(\ *enp109s0f0)*. The number *n* can’t more than *TotalVFs* - that comes from the return value of command +#. Input the ``\ *echo n > /sys/class/net/enp109s0f0/device/sriov\_numvfs*\`` + command in the Service VM to enable n VF devices for the first PF + device (\ *enp109s0f0)*. The number *n* can’t be more than *TotalVFs* + which comes from the return value of command ``cat /sys/class/net/enp109s0f0/device/sriov\_totalvfs``. Here we use *n = 2* as an example. - *“cat /sys/class/net/enp109s0f0/device/sriov\_totalvfs”*. Here we use - *n = 2* for the example. + .. figure:: images/sriov-image10.png + :align: center + :name: 82576-vf -.. figure:: images/sriov-image10.png - :align: center - :width: 900px - :name: 82576-vf + 82576 SR-IOV VF devices - 82576 SR-IOV VF devices + .. figure:: images/sriov-image11.png + :align: center + :name: 82576-vf-nic -.. figure:: images/sriov-image11.png - :align: center - :width: 900px - :name: 82576-vf-nic + 82576 SR-IOV VF NIC - 82576 SR-IOV VF NIC +#. Passthrough a SR-IOV VF device to guest. -1. Passthrough a SR-IOV VF device to guest - - a. Unbind igbvf driver in Service VM + a. Unbind the igbvf driver in the Service VM. i. *modprobe pci\_stub* @@ -282,25 +267,25 @@ We use Intel 82576 NIC as an example in here and only support LaaG iv. *echo "0000:6d:10.0" > /sys/bus/pci/drivers/pci-stub/bind* - b. Add SR-IOV VF device parameter (“*-s X, passthru,6d/10/0*\ ”) in - launch User VM script + b. Add the SR-IOV VF device parameter (“*-s X, passthru,6d/10/0*\ ”) in + the launch User VM script -.. figure:: images/sriov-image12.png - :align: center - :width: 900px - :name: 82576-nic-passthru + .. figure:: images/sriov-image12.png + :align: center + :name: 82576-nic-passthru - Configure 82576 NIC as Passthrough Device + Configure 82576 NIC as a Passthrough Device -a. Boot User VM + c. Boot the User VM SR-IOV Limitations In ACRN -------------------------- 1. The SR-IOV migration feature is not supported. -2. If one SR-IOV PF device is detected during enumeration phase, but there is no - enough room for its total VF devices, the PF device will be dropped. The platform - uses *MAX_PCI_DEV_NUM* ACRN configuration to support the maximum number of PCI - devices, please make sure *MAX_PCI_DEV_NUM* is more than the number of all PCI - devices including total SR-IOV VF devices. +2. If one SR-IOV PF device is detected during the enumeration phase, but + not enough room exists for its total VF devices, the PF device will be + dropped. The platform uses the *MAX_PCI_DEV_NUM* ACRN configuration to + support the maximum number of PCI devices. Make sure *MAX_PCI_DEV_NUM* is + more than the number of all PCI devices, including the total SR-IOV VF + devices.