doc: Grammatical updates to SR-IOV virt doc

Signed-off-by: Deb Taylor <deb.taylor@intel.com>
This commit is contained in:
Deb Taylor 2020-03-18 20:59:39 -04:00 committed by deb-intel
parent 7b3c3c6bcf
commit 537f59f751

View File

@ -4,174 +4,166 @@ SR-IOV Virtualization
===================== =====================
SR-IOV (Single Root Input/Output Virtualization) can isolate PCIe devices SR-IOV (Single Root Input/Output Virtualization) can isolate PCIe devices
and make the devices to have good performance, close to bare-metal. The to improve performance that is similar to bare-metal levels. SR-IOV consists
SR-IOV consists of two basic units, one is called PF(Physical Function) of two basic units: PF (Physical Function), which supports SR-IOV PCIe
that supports SR-IOV PCIe extended capability and manages entire extended capability and manages entire physical devices; and VF (Virtual
physical device, another is called VF (Virtual Function) and is a Function), a "lightweight" PCIe function which is a passthrough device for
"lightweight" PCIe function which is meant to be given to a VM as a VMs.
pass-through device.
For more details, please refer to PCI Express Base Specification Revision 4.0, For details, refer to Chapter 9 of PCI-SIG's `PCI Express Base SpecificationRevision 4.0, Version 1.0 <https://pcisig.com/pci-express-architecture-configuration-space-test-specification-revision-40-version-10>`_.
Version 1.0 `PCI-SIG Specification <https://pcisig.com/specifications>` Chapter 9.
SR-IOV Architectural Overview SR-IOV Architectural Overview
----------------------------- -----------------------------
.. figure:: images/sriov-image1.png .. figure:: images/sriov-image1.png
:align: center :align: center
:width: 900px
:name: SR-IOV-architecture-overview :name: SR-IOV-architecture-overview
SR-IOV Architectural Overview SR-IOV Architectural Overview
- **SI** - System Image known as a VM. - **SI** - A System Image known as a VM.
- **VI** - Virtualization Intermediary known as Hypervisor. - **VI** - A Virtualization Intermediary known as a hypervisor.
- **SR-PCIM** - Single Root PCI Manager is a software entity for SR-IOV - **SR-PCIM** - A Single Root PCI Manager; it is a software entity for
management. SR-IOV management.
- **PF** - A PF is a PCIe Function that supports the SR-IOV capability - **PF** - A PCIe Function that supports the SR-IOV capability
and is accessible to an SR-PCIM, a VI, or an SI. and is accessible to an SR-PCIM, a VI, or an SI.
- **VF** - A VF is a “light-weight” PCIe Function that is directly - **VF** - A “light-weight” PCIe Function that is directly accessible by an
accessible by an SI. SI.
SR-IOV Extended Capability SR-IOV Extended Capability
-------------------------- --------------------------
The SR-IOV Extended Capability defined here is a PCIe extended The SR-IOV Extended Capability defined here is a PCIe extended
capability that must be implemented in each PF device that supports capability that must be implemented in each PF device that supports the
SR-IOV feature. This Capability is used to describe and control a PFs SR-IOV feature. This capability is used to describe and control a PFs
SR-IOV Capabilities. SR-IOV Capabilities.
.. figure:: images/sriov-image2.png .. figure:: images/sriov-image2.png
:align: center :align: center
:width: 900px
:name: SR-IOV-extended-capability :name: SR-IOV-extended-capability
SR-IOV Extended Capability SR-IOV Extended Capability
- **PCIe Extended Capability ID** - 0010h. - **PCIe Extended Capability ID** - 0010h.
- **SR-IOV Capabilities** - VF Migration Capable and ARI Capable. - **SR-IOV Capabilities** - VF Migration-Capable and ARI-Capable.
- **SR-IOV Control** - Enable/Disable VFs, VF migration state query. - **SR-IOV Control** - Enable/Disable VFs; VF migration state query.
- **SR-IOV Status** - VF Migration Status - **SR-IOV Status** - VF Migration Status.
- **InitialVFs** - Indicates to SR-PCIM the number of VFs that are - **InitialVFs** - Indicates to the SR-PCIM the number of VFs that are
initially associated with the PF initially associated with the PF.
- **TotalVFs** - Indicates the maximum number of VFs that could be - **TotalVFs** - Indicates the maximum number of VFs that can be
associated with the PF associated with the PF.
- **NumVFs** - Controls the number of VFs that are visible. *NumVFs* <= - **NumVFs** - Controls the number of VFs that are visible. *NumVFs* <=
*InitialVFs* = *TotalVFs*. *InitialVFs* = *TotalVFs*.
- **Function Link Dependency** - The field is used to describe these - **Function Link Dependency** - The field used to describe
dependencies between PFs. VF dependencies are the same as the dependencies between PFs. VF dependencies are the same as the
dependencies of their associated PFs. dependencies of their associated PFs.
- **First VF Offset** - It is a constant and defines the Routing ID - **First VF Offset** - A constant that defines the Routing ID
offset of the first VF that is associated with the PF that contains offset of the first VF that is associated with the PF that contains
this Capability structure this Capability structure.
- **VF Stride** - It defines the Routing ID offset from one VF to the - **VF Stride** - Defines the Routing ID offset from one VF to the
next one for all VFs associated with the PF that contains this next one for all VFs associated with the PF that contains this
Capability structure. Capability structure.
- **VF Device ID** - This field contains the Device ID that should be - **VF Device ID** - The field that contains the Device ID that should be
presented for every VF to the SI. presented for every VF to the SI.
- **Supported Page Sizes** - This field indicates the page sizes - **Supported Page Sizes** - The field that indicates the page sizes
supported by the PF. supported by the PF.
- **System Page Size** - This field defines the page size the system - **System Page Size** - The field that defines the page size the system
will use to map the VFs memory addresses. Software must set the will use to map the VFs memory addresses. Software must set the
value of the *System Page Size* to one of the page sizes set in the value of the *System Page Size* to one of the page sizes set in the
*Supported Page Sizes* field *Supported Page Sizes* field.
- **VF BARs** - These fields must define the VFs Base Address - **VF BARs** - Fields that must define the VFs Base Address
Registers (BARs). These fields behave as normal PCI BARs. Registers (BARs). These fields behave as normal PCI BARs.
- **VF Migration State Array Offset** - This register shall contain a - **VF Migration State Array Offset** - Register that contains a
PF BAR relative pointer to the VF Migration State Array. PF BAR relative pointer to the VF Migration State Array.
- **VF Migration State Array** It is located using the VF Migration - **VF Migration State Array** Located using the VF Migration
State Array Offset register of the SR-IOV Capability block. State Array Offset register of the SR-IOV Capability block.
For more details, please refer to *PCI Express Base Specification For details, refer to the *PCI Express Base Specification Revision 4.0, Version 1.0 Chapter 9.3.3*.
Revision 4.0, Version 1.0 Chapter 9.3.3*
SR-IOV Architecture In ACRN SR-IOV Architecture In ACRN
--------------------------- ---------------------------
.. figure:: images/sriov-image3.png .. figure:: images/sriov-image3.png
:align: center :align: center
:width: 900px
:name: SR-IOV-architecure-in-acrn :name: SR-IOV-architecure-in-acrn
SR-IOV Architectural in ACRN SR-IOV Architectural in ACRN
1. Hypervisor detects SR-IOV capable PCIe device in physical PCI device 1. A hypervisor detects a SR-IOV capable PCIe device in the physical PCI
enumeration phase. device enumeration phase.
2. Hypervisor intercepts the PFs SR-IOV capability accessing to 2. The hypervisor intercepts the PFs SR-IOV capability and accesses whether
enable/disable VF devices based on *VF\_ENABLE* state, and all to enable/disable VF devices based on the *VF\_ENABLE* state. All
read/write requests for PF device are passthrough to PF physical read/write requests for a PF device passthrough to the PF physical
device. device.
3. Hypervisor waits for 100ms after *VF\_ENABLE* is set and initialize 3. The hypervisor waits for 100ms after *VF\_ENABLE* is set and initializes
VF devices. The differences between normal passthrough device and VF devices. The differences between a normal passthrough device and
SR-IOV VF device are physical device detection, BARs and MSIx SR-IOV VF device are physical device detection, BARs, and MSIx
initializations. Hypervisor uses *Subsystem Vendor ID* to detect initialization. The hypervisor uses *Subsystem Vendor ID* to detect the
SR-IOV VF physical device instead *Vendor ID* since there is no valid SR-IOV VF physical device instead of *Vendor ID* since no valid
*Vendor ID* for SR-IOV VF physical device, the VF BARs are *Vendor ID* exists for the SR-IOV VF physical device. The VF BARs are
initialized by its associated PFs SR-IOV capabilities not PCI initialized by its associated PFs SR-IOV capabilities, not PCI
standard BAR registers, MSIx mapping base address is also from PFs standard BAR registers. The MSIx mapping base address is also from the
SR-IOV capabilities not PCI standard BAR registers. PFs SR-IOV capabilities, not PCI standard BAR registers.
SR-IOV Passthrough VF Architecture In ACRN SR-IOV Passthrough VF Architecture In ACRN
------------------------------------------ ------------------------------------------
.. figure:: images/sriov-image4.png .. figure:: images/sriov-image4.png
:align: center :align: center
:width: 900px
:name: SR-IOV-vf-passthrough :name: SR-IOV-vf-passthrough
SR-IOV VF Passthrough Architecture In ACRN SR-IOV VF Passthrough Architecture In ACRN
1. The SR-IOV VF device needs to bind PCI-stud driver instead 1. The SR-IOV VF device needs to bind the PCI-stud driver instead of the
vendor-specific VF driver before passthrough the device. vendor-specific VF driver before the device passthrough.
2. User configures ``acrn-dm`` boot parameter with passthrough SR-IOV VF 2. The user configures the ``acrn-dm`` boot parameter with the passthrough
device. When User VM starts, ``acrn-dm`` invokes a hypercall to set the SR-IOV VF device. When the User VM starts, ``acrn-dm`` invokes a
*vdev-VF0* device in User VM. hypercall to set the *vdev-VF0* device in the User VM.
3. Hypervisor emulates *Device ID/Vendor ID* and *Memory Space Enable 3. The hypervisor emulates *Device ID/Vendor ID* and *Memory Space Enable
(MSE*) in configuration space for an assigned SR-IOV VF device. The (MSE)* in the configuration space for an assigned SR-IOV VF device. The
assigned VF *Device ID* comes from its associated PFs capability, assigned VF *Device ID* comes from its associated PFs capability. The
*Vendor ID* is same with PFs *Vendor ID* and *MSE* is always set *Vendor ID* is the same as the PFs *Vendor ID* and the *MSE* is always
when reading the SR-IOV VF device *CONTROL* register set when reading the SR-IOV VF device's *CONTROL* register.
4. Vendor-specific VF driver in target VM probes the assigned SR-IOV VF 4. The vendor-specific VF driver in the target VM probes the assigned SR-IOV
device. VF device.
SR-IOV Initialization Flow SR-IOV Initialization Flow
-------------------------- --------------------------
.. figure:: images/sriov-image5.png .. figure:: images/sriov-image5.png
:align: center :align: center
:width: 900px
:name: SR-IOV-init-flow :name: SR-IOV-init-flow
SR-IOV Initialization Flow SR-IOV Initialization Flow
When a SR-IOV capable device is initialized, all access to the When a SR-IOV capable device is initialized, all access to the
configuration space will passthrough to the physical device directly. configuration space will passthrough to the physical device directly.
Service VM can identify all capabilities of the device from SR-IOV extended The Service VM can identify all capabilities of the device from the SR-IOV
capability then create sysfs node for SR-IOV management. extended capability and then create an *sysfs* node for SR-IOV management.
SR-IOV VF Enable Flow SR-IOV VF Enable Flow
--------------------- ---------------------
@ -183,94 +175,87 @@ SR-IOV VF Enable Flow
SR-IOV VF Enable Flow SR-IOV VF Enable Flow
Application enables n VF devices via a SR-IOV PF device *sysfs* node. The application enables n VF devices via a SR-IOV PF device *sysfs* node.
Hypervisor intercepts all SR-IOV capability accesses and check The hypervisor intercepts all SR-IOV capability access and checks the
*VF\_ENABLE* state, if the *VF\_ENABLE* is set, hypervisor creates n *VF\_ENABLE* state. If *VF\_ENABLE* is set, the hypervisor creates n
virtual devices after 100ms that VF physical devices have enough time to virtual devices after 100ms so that VF physical devices have enough time to
be created. Service VM waits 100ms then only accesses the first VF devices be created. The Service VM waits 100ms and then only accesses the first VF
configuration space including *Class Code, Reversion ID, Subsystem devices configuration space including *Class Code, Reversion ID, Subsystem
Vendor ID, Subsystem ID,* Service VM uses the first VF device information to Vendor ID, Subsystem ID*. The Service VM uses the first VF device
initialize subsequent VF devices. information to initialize subsequent VF devices.
SR-IOV VF Disable Flow SR-IOV VF Disable Flow
---------------------- ----------------------
.. figure:: images/sriov-image7.png .. figure:: images/sriov-image7.png
:align: center :align: center
:width: 900px
:name: SR-IOV-disable-flow :name: SR-IOV-disable-flow
SR-IOV VF Disable Flow SR-IOV VF Disable Flow
Application disables SR-IOV VF devices by writing zero to SR-IOV PF The application disables SR-IOV VF devices by writing zero to the SR-IOV PF
device *sysfs* node. Hypervisor intercepts all SR-IOV capability device *sysfs* node. The hypervisor intercepts all SR-IOV capability
accesses and check *VF\_ENABLE* state, if *VF\_ENABLE* is clear, accesses and checks the *VF\_ENABLE* state. If *VF\_ENABLE* is clear, the
hypervisor makes VF virtual devices invisible from Service VM that all accesses hypervisor makes VF virtual devices invisible from the Service VM so that all
for VF will return 0xFFFFFFFF as an error, VF physical devices are access to VF devices will return 0xFFFFFFFF as an error. The VF physical
removed within 1s of *VF\_ENABLE* is clear. devices are removed within 1s of when *VF\_ENABLE* is clear.
SR-IOV VF Assignment Policy SR-IOV VF Assignment Policy
--------------------------- ---------------------------
.. figure:: images/sriov-image8.png .. figure:: images/sriov-image8.png
:align: center :align: center
:width: 900px
:name: SR-IOV-vf-assignment :name: SR-IOV-vf-assignment
SR-IOV VF Assignment SR-IOV VF Assignment
1. All SR-IOV PF devices are managed by Service VM. 1. All SR-IOV PF devices are managed by the Service VM.
2. Currently, SR-IOV PF cant passthrough to User VM. 2. Currently, the SR-IOV PF cannot passthrough to the User VM.
3. All VFs can be passthrough to User VM, but we do not recommend 3. All VFs can passthrough to the User VM, but we do not recommend
passthrough to high privilege VMs, because PF device may impact a passthrough to high privilege VMs because the PF device may impact
assigned VFs including functionality and stability. the assigned VFs' functionality and stability.
SR-IOV Usage Guide In ACRN SR-IOV Usage Guide In ACRN
-------------------------- --------------------------
We use Intel 82576 NIC as an example in here and only support LaaG We use the Intel 82576 NIC as an example in the following instructions. We
(Linux as a Guest). only support LaaG (Linux as a Guest).
1. Please ensure the 82576 VF driver is compiled into User VM Kernel 1. Ensure that the 82576 VF driver is compiled into the User VM Kernel
(Set *CONFIG\_IGBVF=y* in Kernel Config). (set *CONFIG\_IGBVF=y* in the Kernel Config).
2. When the Service VM boot up, “\ *lspci -v*\ ” command can show Intel 82576 #. When the Service VM boots up, the ``\ *lspci -v*\`` command indicates
NIC devices have SR-IOV capability and their PF drivers are ``igb``. that the Intel 82576 NIC devices have SR-IOV capability and their PF
drivers are ``igb``.
.. figure:: images/sriov-image9.png .. figure:: images/sriov-image9.png
:align: center :align: center
:width: 900px
:name: 82576-pf :name: 82576-pf
82576 SR-IOV PF devices 82576 SR-IOV PF devices
1. Input "\ *echo n > /sys/class/net/enp109s0f0/device/sriov\_numvfs*\ " #. Input the ``\ *echo n > /sys/class/net/enp109s0f0/device/sriov\_numvfs*\``
command in Service VM to enable n VF devices for the first PF command in the Service VM to enable n VF devices for the first PF
device(\ *enp109s0f0)*. The number *n* cant more than *TotalVFs* device (\ *enp109s0f0)*. The number *n* cant be more than *TotalVFs*
that comes from the return value of command which comes from the return value of command ``cat /sys/class/net/enp109s0f0/device/sriov\_totalvfs``. Here we use *n = 2* as an example.
*“cat /sys/class/net/enp109s0f0/device/sriov\_totalvfs”*. Here we use
*n = 2* for the example.
.. figure:: images/sriov-image10.png .. figure:: images/sriov-image10.png
:align: center :align: center
:width: 900px
:name: 82576-vf :name: 82576-vf
82576 SR-IOV VF devices 82576 SR-IOV VF devices
.. figure:: images/sriov-image11.png .. figure:: images/sriov-image11.png
:align: center :align: center
:width: 900px
:name: 82576-vf-nic :name: 82576-vf-nic
82576 SR-IOV VF NIC 82576 SR-IOV VF NIC
1. Passthrough a SR-IOV VF device to guest #. Passthrough a SR-IOV VF device to guest.
a. Unbind igbvf driver in Service VM a. Unbind the igbvf driver in the Service VM.
i. *modprobe pci\_stub* i. *modprobe pci\_stub*
@ -282,25 +267,25 @@ We use Intel 82576 NIC as an example in here and only support LaaG
iv. *echo "0000:6d:10.0" > iv. *echo "0000:6d:10.0" >
/sys/bus/pci/drivers/pci-stub/bind* /sys/bus/pci/drivers/pci-stub/bind*
b. Add SR-IOV VF device parameter (“*-s X, passthru,6d/10/0*\ ”) in b. Add the SR-IOV VF device parameter (“*-s X, passthru,6d/10/0*\ ”) in
launch User VM script the launch User VM script
.. figure:: images/sriov-image12.png .. figure:: images/sriov-image12.png
:align: center :align: center
:width: 900px
:name: 82576-nic-passthru :name: 82576-nic-passthru
Configure 82576 NIC as Passthrough Device Configure 82576 NIC as a Passthrough Device
a. Boot User VM c. Boot the User VM
SR-IOV Limitations In ACRN SR-IOV Limitations In ACRN
-------------------------- --------------------------
1. The SR-IOV migration feature is not supported. 1. The SR-IOV migration feature is not supported.
2. If one SR-IOV PF device is detected during enumeration phase, but there is no 2. If one SR-IOV PF device is detected during the enumeration phase, but
enough room for its total VF devices, the PF device will be dropped. The platform not enough room exists for its total VF devices, the PF device will be
uses *MAX_PCI_DEV_NUM* ACRN configuration to support the maximum number of PCI dropped. The platform uses the *MAX_PCI_DEV_NUM* ACRN configuration to
devices, please make sure *MAX_PCI_DEV_NUM* is more than the number of all PCI support the maximum number of PCI devices. Make sure *MAX_PCI_DEV_NUM* is
devices including total SR-IOV VF devices. more than the number of all PCI devices, including the total SR-IOV VF
devices.