diff --git a/doc/develop.rst b/doc/develop.rst index 885794f72..2ef59c7bc 100644 --- a/doc/develop.rst +++ b/doc/develop.rst @@ -57,7 +57,6 @@ Advanced Features .. toctree:: :maxdepth: 1 - tutorials/nvmx_virtualization tutorials/vuart_configuration tutorials/rdt_configuration tutorials/vcat_configuration diff --git a/doc/tutorials/nvmx_virtualization.rst b/doc/tutorials/nvmx_virtualization.rst deleted file mode 100644 index 1c02e7758..000000000 --- a/doc/tutorials/nvmx_virtualization.rst +++ /dev/null @@ -1,351 +0,0 @@ -.. _nested_virt: - -Enable Nested Virtualization -############################ - -With nested virtualization enabled in ACRN, you can run virtual machine -instances inside of a guest VM (also called a User VM) running on the ACRN hypervisor. -Although both "level 1" guest VMs and nested guest VMs can be launched -from the Service VM, the following distinction is worth noting: - -* The VMX feature (``CPUID01.01H:ECX[5]``) does not need to be visible to the Service VM - in order to launch guest VMs. A guest VM not running on top of the - Service VM is considered a level 1 (L1) guest. - -* The VMX feature must be visible to an L1 guest to launch a nested VM. An instance - of a guest hypervisor (KVM) runs on the L1 guest and works with the - L0 ACRN hypervisor to run the nested VM. - -The conventional single-level virtualization has two levels - the L0 host -(ACRN hypervisor) and the L1 guest VMs. With nested virtualization enabled, -ACRN can run guest VMs with their associated virtual machines that define a -third level: - -* The host (ACRN hypervisor), which we call the L0 hypervisor -* The guest hypervisor (KVM), which we call the L1 hypervisor -* The nested guest VMs, which we call the L2 guest VMs - -.. figure:: images/nvmx_1.png - :width: 700px - :align: center - - Generic Nested Virtualization - - -High-Level ACRN Nested Virtualization Design -******************************************** - -The high-level design of nested virtualization in ACRN is shown in :numref:`nested_virt_hld`. -Nested VMX is enabled by allowing a guest VM to use VMX instructions, -and emulating them using the single level of VMX available in the hardware. - -In x86, a logical processor uses a VM control structure (named VMCS in Intel -processors) to manage the state for each vCPU of its guest VMs. These VMCSs -manage VM entries and VM exits as well as processor behavior in VMX non-root -operation. We'll suffix each VMCS with two digits, the hypervisor level managing -it, and the VM level it represents. For example, L0 stores the state of L1 in -VMCS01. The trick of nVMX emulation is ACRN builds a VMCS02 out of the VMCS01, -which is the VMCS ACRN uses to run the L1 VM, and VMCS12 which is built by L1 -hypervisor to actually run the L2 guest. - -.. figure:: images/nvmx_arch_1.png - :width: 400px - :align: center - :name: nested_virt_hld - - Nested Virtualization in ACRN - -#. L0 hypervisor (ACRN) runs L1 guest with VMCS01 - -#. L1 hypervisor (KVM) creates VMCS12 to run a L2 guest - -#. VMX instructions from L1 hypervisor trigger VMExits to L0 hypervisor: - -#. L0 hypervisor runs a L2 guest with VMCS02 - - a. L0 caches VMCS12 in host memory - #. L0 merges VMCS01 and VMCS12 to create VMCS02 - -#. L2 guest runs until triggering VMExits to L0 - - a. L0 reflects most VMExits to L1 hypervisor - #. L0 runs L1 guest with VMCS01 and VMCS02 as the shadow VMCS - - -Restrictions and Constraints -**************************** - -Nested virtualization is considered an experimental feature, and only tested -on Tiger Lake and Kaby Lake platforms (see :ref:`hardware`). - -L1 VMs have the following restrictions: - -* KVM is the only L1 hypervisor supported by ACRN -* KVM runs in 64-bit mode -* KVM enables EPT for L2 guests -* QEMU is used to launch L2 guests - -Constraints on L1 guest configuration: - -* Local APIC passthrough must be enabled -* Only the ``SCHED_NOOP`` scheduler is supported. ACRN can't receive timer interrupts - on LAPIC passthrough pCPUs - -VPID Allocation -=============== - -ACRN doesn't emulate L2 VPIDs and allocates VPIDs for L1 VMs from the reserved top -16-bit VPID range (``0x10000U - CONFIG_MAX_VM_NUM * MAX_VCPUS_PER_VM`` and up). -If the L1 hypervisor enables VPID for L2 VMs and allocates L2 VPIDs not in this -range, ACRN doesn't need to flush L2 VPID during L2 VMX transitions. - -This is the expected behavior most of the time. But in special cases where a -L2 VPID allocated by L1 hypervisor is within this reserved range, it's possible -that this L2 VPID may conflict with a L1 VPID. In this case, ACRN flushes VPID -on L2 VMExit/VMEntry that are associated with this L2 VPID, which may significantly -negatively impact performances of this L2 VM. - - -Service VM Configuration -************************* - -ACRN only supports enabling the nested virtualization feature on the Service VM, not on pre-launched -VMs. - -The nested virtualization feature is disabled by default in ACRN. You can -enable it using the :ref:`ACRN Configurator ` -with these settings: - -.. note:: Normally you'd use the ACRN Configurator GUI to edit the scenario XML file. - The tool wasn't updated in time for the v2.5 release, so you'll need to manually edit - the ACRN scenario XML configuration file to edit the ``SCHEDULER``, ``pcpu_id``, - ``guest_flags``, ``legacy_vuart``, and ``console_vuart`` settings for - the Service VM, as shown below. - -#. Configure system level features: - - - Edit ``hv.features.scheduler`` to ``SCHED_NOOP`` to disable CPU sharing - - .. code-block:: xml - :emphasize-lines: 3,18 - - - y - SCHED_NOOP - y - y - - n - y - 0xfff - 0xfff - 0xfff - 0xfff - 0xfff - 0xfff - 0xfff - 0xfff - - y - -#. In each guest VM configuration: - - - Edit ``vm.nested_virtualization_support`` on the Service VM section and set it to `y` - to enable the nested virtualization feature on the Service VM. - - Edit ``vm.lapic_passthrough`` and set it to `y` to enable local - APIC passthrough on the Service VM. - - Edit ``vm.cpu_affinity.pcpu_id`` to assign ``pCPU`` IDs to run the Service VM. If you are - using debug build and need the hypervisor console, don't assign - ``pCPU0`` to the Service VM. - - .. code-block:: xml - :emphasize-lines: 5,6,7,10,11 - - - SERVICE_VM - ACRN_Service_VM - - 1 - 2 - 3 - - - GUEST_FLAG_NVMX_ENABLED - GUEST_FLAG_LAPIC_PASSTHROUGH - - - The Service VM's virtual legacy UART interrupt doesn't work with LAPIC - passthrough, which may prevent the Service VM from booting. Instead, we need to use - the PCI-vUART for the Service VM. Refer to :ref:`Enable vUART Configurations ` - for more details about VUART configuration. - - - Set ``vm.console_vuart`` to ``PCI`` - - .. code-block:: xml - :emphasize-lines: 1 - - PCI - -#. Remove CPU sharing VMs - - Since CPU sharing is disabled, you may need to delete all ``POST_STD_VM`` and - ``KATA_VM`` VMs from the scenario configuration file, which may share a pCPU - with the Service VM. - -#. Follow instructions in :ref:`gsg` and build with this XML configuration. - - -Prepare for Service VM Kernel and rootfs -**************************************** - -The Service VM can run Ubuntu or other Linux distributions. -Instructions on how to boot Ubuntu as the Service VM can be found in -:ref:`gsg`. - -The Service VM kernel needs to be built from the ``acrn-kernel`` repo, and some changes -to the kernel ``.config`` are needed. -Instructions on how to build and install the Service VM kernel can be found -in :ref:`gsg`. - -Here is a summary of how to modify and build the kernel: - -.. code-block:: none - - git clone https://github.com/projectacrn/acrn-kernel - cd acrn-kernel - cp kernel_config_service_vm .config - make olddefconfig - -The following configuration entries are needed to launch nested -guests on the Service VM: - -.. code-block:: none - - CONFIG_KVM=y - CONFIG_KVM_INTEL=y - CONFIG_ACRN_GUEST=y - -After you make these configuration modifications, build and install the kernel -as described in :ref:`gsg`. - - -Launch a Nested Guest VM -************************ - -Create an Ubuntu KVM Image -========================== - -Refer to :ref:`Build the Ubuntu KVM Image ` -on how to create an Ubuntu KVM image as the nested guest VM's root filesystem. -There is no particular requirement for this image, e.g., it could be of either -qcow2 or raw format. - -Prepare for Launch Scripts -========================== - -Install QEMU on the Service VM that will launch the nested guest VM: - -.. code-block:: none - - sudo apt-get install qemu-kvm qemu virt-manager virt-viewer libvirt-bin - -.. important:: The QEMU ``-cpu host`` option is needed to launch a nested guest VM, and ``-nographics`` - is required to run nested guest VMs reliably. - -You can prepare the script just like the one you use to launch a VM -on native Linux. For example, other than ``-hda``, you can use the following option to launch -a virtio block based RAW image:: - - -drive format=raw,file=/root/ubuntu-20.04.img,if=virtio - -Use the following option to enable Ethernet on the guest VM:: - - -netdev tap,id=net0 -device virtio-net-pci,netdev=net0,mac=a6:cd:47:5f:20:dc - -The following is a simple example for the script to launch a nested guest VM. - -.. code-block:: bash - :emphasize-lines: 2-4 - - sudo qemu-system-x86_64 \ - -enable-kvm \ - -cpu host \ - -nographic \ - -m 2G -smp 2 -hda /root/ubuntu-20.04.qcow2 \ - -net nic,macaddr=00:16:3d:60:0a:80 -net tap,script=/etc/qemu-ifup - -Launch the Guest VM -=================== - -You can launch the nested guest VM from the Service VM's virtual serial console -or from an SSH remote login. - -If the nested VM is launched successfully, you should see the nested -VM's login prompt: - -.. code-block:: console - - [ OK ] Started Terminate Plymouth Boot Screen. - [ OK ] Started Hold until boot process finishes up. - [ OK ] Starting Set console scheme... - [ OK ] Started Serial Getty on ttyS0. - [ OK ] Started LXD - container startup/shutdown. - [ OK ] Started Set console scheme. - [ OK ] Started Getty on tty1. - [ OK ] Reached target Login Prompts. - [ OK ] Reached target Multi-User System. - [ OK ] Started Update UTMP about System Runlevel Changes. - - Ubuntu 20.04 LTS ubuntu_vm ttyS0 - - ubuntu_vm login: - -You won't see the nested guest from a ``vcpu_list`` or ``vm_list`` command -on the ACRN hypervisor console because these commands only show level 1 VMs. - -.. code-block:: console - - ACRN:\>vm_list - - VM_UUID VM_ID VM_NAME VM_STATE - ================================ ===== ========================== - dbbbd4347a574216a12c2201f1ab0240 0 ACRN_Service_VM Running - ACRN:\>vcpu_list - - VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE THREAD STATE - ===== ======= ======= ========= ========== ============ - 0 1 0 PRIMARY Running RUNNING - 0 2 1 SECONDARY Running RUNNING - 0 3 2 SECONDARY Running RUNNING - -On the nested guest VM console, run an ``lshw`` or ``dmidecode`` command -and you'll see that this is a QEMU-managed virtual machine: - -.. code-block:: console - :emphasize-lines: 4,5 - - $ sudo lshw -c system - ubuntu_vm - description: Computer - product: Standard PC (i440FX + PIIX, 1996) - vendor: QEMU - version: pc-i440fx-5.2 - width: 64 bits - capabilities: smbios-2.8 dmi-2.8 smp vsyscall32 - configuration: boot=normal - -For example, compare this to the same command run on the L1 guest (Service VM): - -.. code-block:: console - :emphasize-lines: 4,5 - - $ sudo lshw -c system - localhost.localdomain - description: Computer - product: NUC7i5DNHE - vendor: Intel Corporation - version: J57828-507 - serial: DW1710099900081 - width: 64 bits - capabilities: smbios-3.1 dmi-3.1 smp vsyscall32 - configuration: boot=normal family=Intel NUC uuid=36711CA2-A784-AD49-B0DC-54B2030B16AB