mirror of
https://github.com/projectacrn/acrn-hypervisor.git
synced 2025-08-02 00:08:43 +00:00
doc: Remove outdated nested virtualization tutorial
Signed-off-by: Reyes, Amy <amy.reyes@intel.com>
This commit is contained in:
parent
17d8eaa7ae
commit
9e2b9ffa58
@ -57,7 +57,6 @@ Advanced Features
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
tutorials/nvmx_virtualization
|
||||
tutorials/vuart_configuration
|
||||
tutorials/rdt_configuration
|
||||
tutorials/vcat_configuration
|
||||
|
@ -1,351 +0,0 @@
|
||||
.. _nested_virt:
|
||||
|
||||
Enable Nested Virtualization
|
||||
############################
|
||||
|
||||
With nested virtualization enabled in ACRN, you can run virtual machine
|
||||
instances inside of a guest VM (also called a User VM) running on the ACRN hypervisor.
|
||||
Although both "level 1" guest VMs and nested guest VMs can be launched
|
||||
from the Service VM, the following distinction is worth noting:
|
||||
|
||||
* The VMX feature (``CPUID01.01H:ECX[5]``) does not need to be visible to the Service VM
|
||||
in order to launch guest VMs. A guest VM not running on top of the
|
||||
Service VM is considered a level 1 (L1) guest.
|
||||
|
||||
* The VMX feature must be visible to an L1 guest to launch a nested VM. An instance
|
||||
of a guest hypervisor (KVM) runs on the L1 guest and works with the
|
||||
L0 ACRN hypervisor to run the nested VM.
|
||||
|
||||
The conventional single-level virtualization has two levels - the L0 host
|
||||
(ACRN hypervisor) and the L1 guest VMs. With nested virtualization enabled,
|
||||
ACRN can run guest VMs with their associated virtual machines that define a
|
||||
third level:
|
||||
|
||||
* The host (ACRN hypervisor), which we call the L0 hypervisor
|
||||
* The guest hypervisor (KVM), which we call the L1 hypervisor
|
||||
* The nested guest VMs, which we call the L2 guest VMs
|
||||
|
||||
.. figure:: images/nvmx_1.png
|
||||
:width: 700px
|
||||
:align: center
|
||||
|
||||
Generic Nested Virtualization
|
||||
|
||||
|
||||
High-Level ACRN Nested Virtualization Design
|
||||
********************************************
|
||||
|
||||
The high-level design of nested virtualization in ACRN is shown in :numref:`nested_virt_hld`.
|
||||
Nested VMX is enabled by allowing a guest VM to use VMX instructions,
|
||||
and emulating them using the single level of VMX available in the hardware.
|
||||
|
||||
In x86, a logical processor uses a VM control structure (named VMCS in Intel
|
||||
processors) to manage the state for each vCPU of its guest VMs. These VMCSs
|
||||
manage VM entries and VM exits as well as processor behavior in VMX non-root
|
||||
operation. We'll suffix each VMCS with two digits, the hypervisor level managing
|
||||
it, and the VM level it represents. For example, L0 stores the state of L1 in
|
||||
VMCS01. The trick of nVMX emulation is ACRN builds a VMCS02 out of the VMCS01,
|
||||
which is the VMCS ACRN uses to run the L1 VM, and VMCS12 which is built by L1
|
||||
hypervisor to actually run the L2 guest.
|
||||
|
||||
.. figure:: images/nvmx_arch_1.png
|
||||
:width: 400px
|
||||
:align: center
|
||||
:name: nested_virt_hld
|
||||
|
||||
Nested Virtualization in ACRN
|
||||
|
||||
#. L0 hypervisor (ACRN) runs L1 guest with VMCS01
|
||||
|
||||
#. L1 hypervisor (KVM) creates VMCS12 to run a L2 guest
|
||||
|
||||
#. VMX instructions from L1 hypervisor trigger VMExits to L0 hypervisor:
|
||||
|
||||
#. L0 hypervisor runs a L2 guest with VMCS02
|
||||
|
||||
a. L0 caches VMCS12 in host memory
|
||||
#. L0 merges VMCS01 and VMCS12 to create VMCS02
|
||||
|
||||
#. L2 guest runs until triggering VMExits to L0
|
||||
|
||||
a. L0 reflects most VMExits to L1 hypervisor
|
||||
#. L0 runs L1 guest with VMCS01 and VMCS02 as the shadow VMCS
|
||||
|
||||
|
||||
Restrictions and Constraints
|
||||
****************************
|
||||
|
||||
Nested virtualization is considered an experimental feature, and only tested
|
||||
on Tiger Lake and Kaby Lake platforms (see :ref:`hardware`).
|
||||
|
||||
L1 VMs have the following restrictions:
|
||||
|
||||
* KVM is the only L1 hypervisor supported by ACRN
|
||||
* KVM runs in 64-bit mode
|
||||
* KVM enables EPT for L2 guests
|
||||
* QEMU is used to launch L2 guests
|
||||
|
||||
Constraints on L1 guest configuration:
|
||||
|
||||
* Local APIC passthrough must be enabled
|
||||
* Only the ``SCHED_NOOP`` scheduler is supported. ACRN can't receive timer interrupts
|
||||
on LAPIC passthrough pCPUs
|
||||
|
||||
VPID Allocation
|
||||
===============
|
||||
|
||||
ACRN doesn't emulate L2 VPIDs and allocates VPIDs for L1 VMs from the reserved top
|
||||
16-bit VPID range (``0x10000U - CONFIG_MAX_VM_NUM * MAX_VCPUS_PER_VM`` and up).
|
||||
If the L1 hypervisor enables VPID for L2 VMs and allocates L2 VPIDs not in this
|
||||
range, ACRN doesn't need to flush L2 VPID during L2 VMX transitions.
|
||||
|
||||
This is the expected behavior most of the time. But in special cases where a
|
||||
L2 VPID allocated by L1 hypervisor is within this reserved range, it's possible
|
||||
that this L2 VPID may conflict with a L1 VPID. In this case, ACRN flushes VPID
|
||||
on L2 VMExit/VMEntry that are associated with this L2 VPID, which may significantly
|
||||
negatively impact performances of this L2 VM.
|
||||
|
||||
|
||||
Service VM Configuration
|
||||
*************************
|
||||
|
||||
ACRN only supports enabling the nested virtualization feature on the Service VM, not on pre-launched
|
||||
VMs.
|
||||
|
||||
The nested virtualization feature is disabled by default in ACRN. You can
|
||||
enable it using the :ref:`ACRN Configurator <acrn_configurator_tool>`
|
||||
with these settings:
|
||||
|
||||
.. note:: Normally you'd use the ACRN Configurator GUI to edit the scenario XML file.
|
||||
The tool wasn't updated in time for the v2.5 release, so you'll need to manually edit
|
||||
the ACRN scenario XML configuration file to edit the ``SCHEDULER``, ``pcpu_id``,
|
||||
``guest_flags``, ``legacy_vuart``, and ``console_vuart`` settings for
|
||||
the Service VM, as shown below.
|
||||
|
||||
#. Configure system level features:
|
||||
|
||||
- Edit ``hv.features.scheduler`` to ``SCHED_NOOP`` to disable CPU sharing
|
||||
|
||||
.. code-block:: xml
|
||||
:emphasize-lines: 3,18
|
||||
|
||||
<FEATURES>
|
||||
<RELOC>y</RELOC>
|
||||
<SCHEDULER>SCHED_NOOP</SCHEDULER>
|
||||
<MULTIBOOT2>y</MULTIBOOT2>
|
||||
<ENFORCE_TURNOFF_AC>y</ENFORCE_TURNOFF_AC>
|
||||
<RDT>
|
||||
<RDT_ENABLED>n</RDT_ENABLED>
|
||||
<CDP_ENABLED>y</CDP_ENABLED>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
<CLOS_MASK>0xfff</CLOS_MASK>
|
||||
</RDT>
|
||||
<HYPERV_ENABLED>y</HYPERV_ENABLED>
|
||||
|
||||
#. In each guest VM configuration:
|
||||
|
||||
- Edit ``vm.nested_virtualization_support`` on the Service VM section and set it to `y`
|
||||
to enable the nested virtualization feature on the Service VM.
|
||||
- Edit ``vm.lapic_passthrough`` and set it to `y` to enable local
|
||||
APIC passthrough on the Service VM.
|
||||
- Edit ``vm.cpu_affinity.pcpu_id`` to assign ``pCPU`` IDs to run the Service VM. If you are
|
||||
using debug build and need the hypervisor console, don't assign
|
||||
``pCPU0`` to the Service VM.
|
||||
|
||||
.. code-block:: xml
|
||||
:emphasize-lines: 5,6,7,10,11
|
||||
|
||||
<vm id="1">
|
||||
<vm_type>SERVICE_VM</vm_type>
|
||||
<name>ACRN_Service_VM</name>
|
||||
<cpu_affinity>
|
||||
<pcpu_id>1</pcpu_id>
|
||||
<pcpu_id>2</pcpu_id>
|
||||
<pcpu_id>3</pcpu_id>
|
||||
</cpu_affinity>
|
||||
<guest_flags>
|
||||
<guest_flag>GUEST_FLAG_NVMX_ENABLED</guest_flag>
|
||||
<guest_flag>GUEST_FLAG_LAPIC_PASSTHROUGH</guest_flag>
|
||||
</guest_flags>
|
||||
|
||||
The Service VM's virtual legacy UART interrupt doesn't work with LAPIC
|
||||
passthrough, which may prevent the Service VM from booting. Instead, we need to use
|
||||
the PCI-vUART for the Service VM. Refer to :ref:`Enable vUART Configurations <vuart_config>`
|
||||
for more details about VUART configuration.
|
||||
|
||||
- Set ``vm.console_vuart`` to ``PCI``
|
||||
|
||||
.. code-block:: xml
|
||||
:emphasize-lines: 1
|
||||
|
||||
<console_vuart>PCI</console_vuart>
|
||||
|
||||
#. Remove CPU sharing VMs
|
||||
|
||||
Since CPU sharing is disabled, you may need to delete all ``POST_STD_VM`` and
|
||||
``KATA_VM`` VMs from the scenario configuration file, which may share a pCPU
|
||||
with the Service VM.
|
||||
|
||||
#. Follow instructions in :ref:`gsg` and build with this XML configuration.
|
||||
|
||||
|
||||
Prepare for Service VM Kernel and rootfs
|
||||
****************************************
|
||||
|
||||
The Service VM can run Ubuntu or other Linux distributions.
|
||||
Instructions on how to boot Ubuntu as the Service VM can be found in
|
||||
:ref:`gsg`.
|
||||
|
||||
The Service VM kernel needs to be built from the ``acrn-kernel`` repo, and some changes
|
||||
to the kernel ``.config`` are needed.
|
||||
Instructions on how to build and install the Service VM kernel can be found
|
||||
in :ref:`gsg`.
|
||||
|
||||
Here is a summary of how to modify and build the kernel:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
git clone https://github.com/projectacrn/acrn-kernel
|
||||
cd acrn-kernel
|
||||
cp kernel_config_service_vm .config
|
||||
make olddefconfig
|
||||
|
||||
The following configuration entries are needed to launch nested
|
||||
guests on the Service VM:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
CONFIG_KVM=y
|
||||
CONFIG_KVM_INTEL=y
|
||||
CONFIG_ACRN_GUEST=y
|
||||
|
||||
After you make these configuration modifications, build and install the kernel
|
||||
as described in :ref:`gsg`.
|
||||
|
||||
|
||||
Launch a Nested Guest VM
|
||||
************************
|
||||
|
||||
Create an Ubuntu KVM Image
|
||||
==========================
|
||||
|
||||
Refer to :ref:`Build the Ubuntu KVM Image <build-the-ubuntu-kvm-image>`
|
||||
on how to create an Ubuntu KVM image as the nested guest VM's root filesystem.
|
||||
There is no particular requirement for this image, e.g., it could be of either
|
||||
qcow2 or raw format.
|
||||
|
||||
Prepare for Launch Scripts
|
||||
==========================
|
||||
|
||||
Install QEMU on the Service VM that will launch the nested guest VM:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
sudo apt-get install qemu-kvm qemu virt-manager virt-viewer libvirt-bin
|
||||
|
||||
.. important:: The QEMU ``-cpu host`` option is needed to launch a nested guest VM, and ``-nographics``
|
||||
is required to run nested guest VMs reliably.
|
||||
|
||||
You can prepare the script just like the one you use to launch a VM
|
||||
on native Linux. For example, other than ``-hda``, you can use the following option to launch
|
||||
a virtio block based RAW image::
|
||||
|
||||
-drive format=raw,file=/root/ubuntu-20.04.img,if=virtio
|
||||
|
||||
Use the following option to enable Ethernet on the guest VM::
|
||||
|
||||
-netdev tap,id=net0 -device virtio-net-pci,netdev=net0,mac=a6:cd:47:5f:20:dc
|
||||
|
||||
The following is a simple example for the script to launch a nested guest VM.
|
||||
|
||||
.. code-block:: bash
|
||||
:emphasize-lines: 2-4
|
||||
|
||||
sudo qemu-system-x86_64 \
|
||||
-enable-kvm \
|
||||
-cpu host \
|
||||
-nographic \
|
||||
-m 2G -smp 2 -hda /root/ubuntu-20.04.qcow2 \
|
||||
-net nic,macaddr=00:16:3d:60:0a:80 -net tap,script=/etc/qemu-ifup
|
||||
|
||||
Launch the Guest VM
|
||||
===================
|
||||
|
||||
You can launch the nested guest VM from the Service VM's virtual serial console
|
||||
or from an SSH remote login.
|
||||
|
||||
If the nested VM is launched successfully, you should see the nested
|
||||
VM's login prompt:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
[ OK ] Started Terminate Plymouth Boot Screen.
|
||||
[ OK ] Started Hold until boot process finishes up.
|
||||
[ OK ] Starting Set console scheme...
|
||||
[ OK ] Started Serial Getty on ttyS0.
|
||||
[ OK ] Started LXD - container startup/shutdown.
|
||||
[ OK ] Started Set console scheme.
|
||||
[ OK ] Started Getty on tty1.
|
||||
[ OK ] Reached target Login Prompts.
|
||||
[ OK ] Reached target Multi-User System.
|
||||
[ OK ] Started Update UTMP about System Runlevel Changes.
|
||||
|
||||
Ubuntu 20.04 LTS ubuntu_vm ttyS0
|
||||
|
||||
ubuntu_vm login:
|
||||
|
||||
You won't see the nested guest from a ``vcpu_list`` or ``vm_list`` command
|
||||
on the ACRN hypervisor console because these commands only show level 1 VMs.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ACRN:\>vm_list
|
||||
|
||||
VM_UUID VM_ID VM_NAME VM_STATE
|
||||
================================ ===== ==========================
|
||||
dbbbd4347a574216a12c2201f1ab0240 0 ACRN_Service_VM Running
|
||||
ACRN:\>vcpu_list
|
||||
|
||||
VM ID PCPU ID VCPU ID VCPU ROLE VCPU STATE THREAD STATE
|
||||
===== ======= ======= ========= ========== ============
|
||||
0 1 0 PRIMARY Running RUNNING
|
||||
0 2 1 SECONDARY Running RUNNING
|
||||
0 3 2 SECONDARY Running RUNNING
|
||||
|
||||
On the nested guest VM console, run an ``lshw`` or ``dmidecode`` command
|
||||
and you'll see that this is a QEMU-managed virtual machine:
|
||||
|
||||
.. code-block:: console
|
||||
:emphasize-lines: 4,5
|
||||
|
||||
$ sudo lshw -c system
|
||||
ubuntu_vm
|
||||
description: Computer
|
||||
product: Standard PC (i440FX + PIIX, 1996)
|
||||
vendor: QEMU
|
||||
version: pc-i440fx-5.2
|
||||
width: 64 bits
|
||||
capabilities: smbios-2.8 dmi-2.8 smp vsyscall32
|
||||
configuration: boot=normal
|
||||
|
||||
For example, compare this to the same command run on the L1 guest (Service VM):
|
||||
|
||||
.. code-block:: console
|
||||
:emphasize-lines: 4,5
|
||||
|
||||
$ sudo lshw -c system
|
||||
localhost.localdomain
|
||||
description: Computer
|
||||
product: NUC7i5DNHE
|
||||
vendor: Intel Corporation
|
||||
version: J57828-507
|
||||
serial: DW1710099900081
|
||||
width: 64 bits
|
||||
capabilities: smbios-3.1 dmi-3.1 smp vsyscall32
|
||||
configuration: boot=normal family=Intel NUC uuid=36711CA2-A784-AD49-B0DC-54B2030B16AB
|
Loading…
Reference in New Issue
Block a user