github/acrn-hypervisor
1
0
Fork 0
mirror of https://github.com/projectacrn/acrn-hypervisor.git synced 2025-06-24 14:33:38 +00:00
Code Issues Projects Releases Wiki Activity

[doc] Enhance Using partition mode on UP2 tutorial

Browse Source
This commit is contained in:
Tonny Tzeng 2018-11-16 15:16:03 +08:00 committed by David Kinder
parent d3d474cf1e
commit 6d6c5b959c
3 changed files with 333 additions and 151 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
doc/tutorials/images/menuconfig-partition-mode.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 73 KiB

BIN
doc/tutorials/images/partition_mode_up2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.9 KiB

484
doc/tutorials/using_partition_mode_on_up2.rst
View File

@ -2,154 +2,309 @@
Using partition mode on UP2
###########################
This tutorial describes the steps needed to use partition mode with the ACRN
hypervisor on UP2.
Build kernel and modules for partition mode guest
*************************************************
ACRN hypervisor supports partition mode, in which the User OS running in a
privileged VM can bypass the ACRN hypervisor and directly access isolated
PCI devices. This tutorial provides step by step instructions on how to set up
the ACRN hypervisor partition mode on
`UP2 <https://up-board.org/upsquared/specifications/>`_ boards running two
privileged VMs as shown in :numref:`two-priv-vms`:
* Build kernel and modules
.. figure:: images/partition_mode_up2.png
:align: center
:name: two-priv-vms
Two privileged VMs running in partition mode
.. code-block:: none
Prerequisites
*************
git clone https://github.com/projectacrn/acrn-kernel.git
cd ~/acrn-kernel
cp kernel_config_partition_os .config
make oldconfig
make
In this tutorial two Linux privileged VMs are started by the ACRN hypervisor.
To set up the Linux root filesystems for each VM, follow the Clear Linux
`bare metal installation guide <https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install#bare-metal-install>`_
to install Clear Linux on a **SATA disk** and a **USB flash disk** prior the setup,
as the two privileged VMs will mount the root filesystems via the SATA controller
and the USB controller respectively.
* Install modules
Build kernel and modules for partition mode UOS
***********************************************
First of all, please prepare one USB flash drive and one SATA disk. Plug them
in one PC and follow Clear Linux's `online instruction <https://clearlinux.org/documentation/clear-linux/get-started/bare-metal-install#bare-metal-install>` to install Clear Linux on them.
#. On your development workstation, clone the ACRN kernel source tree, and
build the Linux kernel image that will be used to boot the privileged VMs:
Plug the USB flash drive and SATA disk into your PC. Assuming Clear Linux
rootfs is installed into /dev/sdx3.
.. code-block:: none
.. code-block:: none
$ git clone https://github.com/projectacrn/acrn-kernel.git
Cloning into 'acrn-kernel'...
...
$ cd acrn-kernel
$ cp kernel_config_uos .config
$ make olddefconfig
scripts/kconfig/conf --olddefconfig Kconfig
#
# configuration written to .config
#
mount /dev/sdx3 /mnt/sdx3
cd ~/acrn-kernel
make INSTALL_MOD_PATH=/mnt/sdx3/ modules_install
$ make
$ make modules_install INSTALL_MOD_PATH=out/
* Install kernel
The last two commands built the bootable kernel image ``arch/x86/boot/bzImage``,
and the loadable kernel modules under the ``./out/`` folder. Copy these files
to a removable disk for installing on the UP2 board later.
First, please following Ubuntu's `online instruction <https://tutorials.ubuntu.com/tutorial/tutorial-install-ubuntu-desktop?_ga=2.114179015.1954550575.1530817291-1278304647.1523530035>` to install it on your UP2's eMMC.
#. Current ACRN partition mode implementation requires a multi-boot capable
bootloader to boot both ACRN hypervisor and the bootable kernel image
built from the previous step. You could install Ubuntu OS to the UP2 board
by following
`this Ubuntu tutorial <https://tutorials.ubuntu.com/tutorial/tutorial-install-ubuntu-desktop>`_.
The Ubuntu installer creates 3 disk partitions on the on-board eMMC memory.
By default, the GRUB bootloader is installed on the ESP (EFI System Partition)
partition, which will be used to bootstrap the partiton mode ACRN hypervisor.
Copy kernel image to EFI System Partition
#. After installing the Ubuntu OS, power off the UP2 board, attach the SATA disk
and the USB flash disk to the board. Power on the board and make sure
it boots the Ubuntu OS from the eMMC, then copy the loadable kernel modules
built in Step 1 to the ``/lib/modules/`` folder on both the mounted SATA
disk and USB disk. For example, assuming the SATA disk and USB flash disk
are assigned to ``/dev/sda`` and ``/dev/sdb`` respectively, the following
commands set up the partition mode loadable kernel modules onto the root
filesystems to be loaded by the privileged VMs:
.. code-block:: none
.. code-block:: none
scp username@IP:~/acrn-kernel/arch/x86/boot/bzImage /boot/efi/
# Mount the Clear Linux root filesystem on the SATA disk
$ sudo mount /dev/sda3 /mnt
$ sudo cp -r <kernel-modules-folder-built-in-step1>/lib/modules/* /mnt/lib/modules
$ sudo umount /mnt
# Mount the Clear Linux root filesystem on the USB flash disk
$ sudo mount /dev/sdb3 /mnt
$ sudo cp -r <path-to-kernel-module-folder-built-in-step1>/lib/modules/* /mnt/lib/modules
$ sudo umount /mnt
Enabling partition mode
***********************
#. Copy the bootable kernel image to the ``/boot`` directory:
* Clone ACRN source code
.. code-block:: none
.. code-block:: none
$ sudo cp <path-to-kernel-image-built-in-step1>/bzImage /boot/
git clone https://github.com/projectacrn/acrn-hypervisor.git
Enable partition mode in ACRN hypervisor
****************************************
* Configure the ACRN hypervisor to use partition mode
#. Before building the ACRN hypervisor, you need to figure out the I/O address
of the serial port, and the PCI BAR addresses of the SATA controller and
the USB controller on your UP2 board.
.. code-block:: none
Enter the following command to get the I/O addresses of the serial ports.
UP2 boards support two serial ports. The addresses from the command
output represent the I/O addresses of the serial port of the 10-pin side
connector and the 40-pin expansion header respectively. You will need to
connect the serial port to the development host, in order to access
the ACRN serial console to switch between privileged VMs.
cd ~/acrn-hypervisor/hypervisor
make menuconfig
.. code-block:: none
:emphasize-lines: 1
Select entry ``Hypervisor mode (Sharing mode)`` and press ``Enter``.
Then select ``Partition mode`` sub-entry.
$ dmesg | grep ttyS
[ 10.552369] dw-apb-uart.8: ttyS4 at MMIO 0x91524000 (irq = 4, base_baud = 115200) is a 16550A
[ 10.552568] serial serial0: tty port ttyS4 registered
[ 10.572344] dw-apb-uart.9: ttyS5 at MMIO 0x91522000 (irq = 5, base_baud = 115200) is a 16550A
[ 10.572545] serial serial1: tty port ttyS5 registered
* Configure the ACRN hypervisor's ``serial base address``
The following command prints detailed information about all PCI buses
and devices in the system. Look up the PCI BAR addresses of the SATA
controller and the USB controller, record these addresses (highlighted below) for
editing the source code later.
.. code-block:: none
.. code-block:: none
:emphasize-lines: 1,9,10,14,22
cd ~/acrn-hypervisor/hypervisor
make menuconfig
$ sudo lspci -vv
...
00:12.0 SATA controller: Intel Corporation Device 5ae3 (rev 0b) (prog-if 01 [AHCI 1.0])
Subsystem: Intel Corporation Device 7270
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 123
Region 0: Memory at 91514000 (32-bit, non-prefetchable) [size=8K]
Region 1: Memory at 91537000 (32-bit, non-prefetchable) [size=256]
Region 2: I/O ports at f090 [size=8]
Region 3: I/O ports at f080 [size=4]
Region 4: I/O ports at f060 [size=32]
Region 5: Memory at 91536000 (32-bit, non-prefetchable) [size=2K]
...
00:15.0 USB controller: Intel Corporation Device 5aa8 (rev 0b) (prog-if 30 [XHCI])
Subsystem: Intel Corporation Device 7270
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 122
Region 0: Memory at 91500000 (64-bit, non-prefetchable) [size=64K]
Select entry ``Base address of serial MMIO region`` and press ``Enter``.
Then we will see a prompt msgbox with title ``Value for `Base address of serial MMIO region` (hex)``.
Please type the serial base mmio address of your UP2 board and press ``Enter``.
#. Clone the ACRN source code and configure the build options with
``make menuconfig`` command:
* Configure the PCI info about devices assigned to vm.
.. code-block:: none
For the current code, we can only configure it by modifying the ``vpci_vdev_arrary``
structure (one per vm) in source code file ``hypervisor/partition/vm_description.c``.
$ git clone https://github.com/projectacrn/acrn-hypervisor.git
$ cd acrn-hypervisor/hypervisor
$ make menuconfig
Here is an reference patch for configuring vm1's assgined PCI devices based on
UP2 board. This patch can also apply to other vms you try to configure.
Set the ``Hypervisor mode`` option to ``Partition mode``, and depending
on the serial port you are using, enter its MMIO address to the configuration
menu as shown in this screenshot. Finally, save the configuration.
.. code-block:: none
.. figure:: images/menuconfig-partition-mode.png
:align: center
diff --git a/hypervisor/partition/vm_description.c b/hypervisor/partition/vm_description.c
index 12818185..88799b00 100644
--- a/hypervisor/partition/vm_description.c
+++ b/hypervisor/partition/vm_description.c
@@ -41,6 +41,7 @@ static struct vpci_vdev_array vpci_vdev_array1 = {
}
},
#. Configure the PCI device info for each VM
{/*vdev 1: SATA controller*/
.vbdf.bits = {.b = 0x00U, .d = 0x01U, .f = 0x0U},
.ops = &pci_ops_vdev_pt,
@@ -65,23 +66,24 @@ static struct vpci_vdev_array vpci_vdev_array1 = {
.bdf.bits = {.b = 0x00U, .d = 0x12U, .f = 0x0U},
.bar = {
[0] = {
- .base = 0xb3f10000UL,
+ .base = 0x91514000UL,
.size = 0x2000UL,
.type = PCIBAR_MEM32
},
[1] = {
- .base = 0xb3f53000UL,
+ .base = 0x91537000UL,
.size = 0x100UL,
.type = PCIBAR_MEM32
},
[5] = {
- .base = 0xb3f52000UL,
+ .base = 0x91536000UL,
.size = 0x800UL,
.type = PCIBAR_MEM32
},
}
}
},
PCI devices that are available to the privileged VMs
are hardcoded in the source file ``hypervisor/partition/apl-mrb/vm_description.c``.
You need to review and modify the ``vpci_vdev_array1`` and ``vpci_vdev_array2``
structures in the source code to match the PCI BAR addresses of the SATA
controller and the USB controller noted in step 1:
.. code-block:: none
:emphasize-lines: 3,39,44,49,56,72
:caption: hypervisor/partition/vm_description.c
...
static struct vpci_vdev_array vpci_vdev_array1 = {
.num_pci_vdev = 2,
.vpci_vdev_list = {
{/*vdev 0: hostbridge */
.vbdf.bits = {.b = 0x00U, .d = 0x00U, .f = 0x0U},
.ops = &pci_ops_vdev_hostbridge,
.bar = {},
.pdev = {
.bdf.bits = {.b = 0x00U, .d = 0x00U, .f = 0x0U},
}
};
},
* Configure ``.bootargs``
{/*vdev 1: SATA controller*/
.vbdf.bits = {.b = 0x00U, .d = 0x01U, .f = 0x0U},
.ops = &pci_ops_vdev_pt,
.bar = {
[0] = {
.base = 0UL,
.size = 0x2000UL,
.type = PCIBAR_MEM32
},
[1] = {
.base = 0UL,
.size = 0x1000UL,
.type = PCIBAR_MEM32
},
[5] = {
.base = 0UL,
.size = 0x1000UL,
.type = PCIBAR_MEM32
},
},
.pdev = {
.bdf.bits = {.b = 0x00U, .d = 0x12U, .f = 0x0U},
.bar = {
[0] = {
.base = 0x91514000UL,
.size = 0x2000UL,
.type = PCIBAR_MEM32
},
[1] = {
.base = 0x91537000UL,
.size = 0x100UL,
.type = PCIBAR_MEM32
},
[5] = {
.base = 0x91536000UL,
.size = 0x800UL,
.type = PCIBAR_MEM32
},
}
...
static struct vpci_vdev_array vpci_vdev_array2 = {
.num_pci_vdev = 3,
...
{/*vdev 1: USB controller*/
.vbdf.bits = {.b = 0x00U, .d = 0x01U, .f = 0x0U},
.ops = &pci_ops_vdev_pt,
.bar = {
[0] = {
.base = 0UL,
.size = 0x10000UL,
.type = PCIBAR_MEM32
},
},
.pdev = {
.bdf.bits = {.b = 0x00U, .d = 0x15U, .f = 0x0U},
.bar = {
[0] = {
.base = 0x91500000UL,
.size = 0x10000UL,
.type = PCIBAR_MEM64
},
}
...
A default command line is pre-configured in the source code. Here we just
configure ``root=`` option to help kernel locate the right rootfs and keep the
other options as default.
#. Optionally, configure the ``.bootargs`` kernel command line arguments
.. code-block:: none
The kernel command line arguments used to boot the privileged VMs are
hardcoded as ``/dev/sda3`` to meet the Clear Linux automatic installation.
In case you plan to use your customized root
filesystem, you may optionally edit the ``root=`` parameter specified
in the ``.bootargs`` field of the ``.vm_desc_array`` structure, to
instruct the Linux kernel to mount the right disk partition:
.. code-block:: none
:emphasize-lines: 9-11,20-22
:caption: hypervisor/partition/vm_description.c
@@ -171,7 +180,8 @@ struct vm_description_array vm_desc_partition = {
.start_hpa = 0x100000000UL,
.mem_size = 0x20000000UL, /* uses contiguous memory from host */
.vm_vuart = true,
- .bootargs = "root=/dev/sda rw rootwait noxsave maxcpus=2 nohpet console=hvc0 \
+ .bootargs = "root=PARTUUID=<Your rootfs PARTUUID> rw rootwait noxsave maxcpus=2 nohpet console=hvc0 \
console=ttyS0 no_timer_check ignore_loglevel log_buf_len=16M \
consoleblank=0 tsc=reliable xapic_phys",
.vpci_vdev_array = &vpci_vdev_array1,
...
/* Virtual Machine descriptions */
.vm_desc_array = {
{
/* Internal variable, MUSTBE init to -1 */
.vm_hw_num_cores = VM1_NUM_CPUS,
.vm_pcpu_ids = &VM1_CPUS[0],
...
.bootargs = "root=/dev/sda3 rw rootwait noxsave maxcpus=2 nohpet console=hvc0 \
console=ttyS2 no_timer_check ignore_loglevel log_buf_len=16M \
consoleblank=0 tsc=reliable xapic_phys",
.vpci_vdev_array = &vpci_vdev_array1,
.mptable = &mptable_vm1,
},
{
/* Internal variable, MUSTBE init to -1 */
.vm_hw_num_cores = VM2_NUM_CPUS,
.vm_pcpu_ids = &VM2_CPUS[0],
...
.bootargs = "root=/dev/sda3 rw rootwait noxsave maxcpus=2 nohpet console=hvc0 \
console=ttyS2 no_timer_check ignore_loglevel log_buf_len=16M \
consoleblank=0 tsc=reliable xapic_phys",
.vpci_vdev_array = &vpci_vdev_array2,
.mptable = &mptable_vm2,
.lapic_pt = true,
},
}
* Build ACRN with the above changes.
#. Build the ACRN hypervisor and copy the artifact ``acrn.32.out`` to the
``/boot`` directory:
.. code-block:: none
.. code-block:: none
$ make PLATFORM=sbl
...
$ sudo cp build/acrn.32.out /boot
#. Modify the ``/etc/grub.d/40_custom`` file to create a new GRUB entry
that will multi-boot the ACRN hypervisor and the UOS kernel image
* Install ACRN
Append the following configuration to the ``/etc/grub.d/40_custom`` file:
First we should add one menuentry in ``/boot/efi/grub/grub.cfg`` on UP2 board
.. code-block:: none
.. code-block:: none
menuentry 'ACRN Partition Mode' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-e23c76ae-b06d-4a6e-ad42-46b8eedfd7d3' {
recordfail
@ -160,61 +315,88 @@ Enabling partition mode
insmod ext2
echo 'Loading partition mode hypervisor ...'
multiboot /acrn.32.out
module /bzImage
multiboot /boot/acrn.32.out
module /boot/bzImage
}
Copy ``acrn.32.out`` and bzImage to ``/boot/efi``
.. code-block:: none
scp username@IP:~/acrn-hypervisor/hypervisor/build/acrn.32.out /boot/efi
* Boot ACRN
Plug the USB flash drive and SATA Disk into your UP2 board and then reboot the
board to the built-in EFI shell. Then execute the following command to load
the ACRN hypervisor.
.. code-block:: none
shell> fs1:
fs1:\> EFI\ubuntu\grubx64.efi
Here we will see the following GRUB menu. Select entry ``ACRN Partition Mode`` and
press ``Enter`` and then the ACRN hypervisor will be loaded automatically.
.. code-block:: console
:emphasize-lines: 2
*Ubuntu
ACRN Partition Mode
Advanced options for Ubuntu
System setup
Restore Ubuntu 16.04 to factory state
Playing with ACRN hypervisor with partition mode
************************************************
* Connect the serial port of the UP2 board with another PC and use the following
command to check if vms booted successfully. If vms are booted successfully,
we will see the following message.
Modify the ``/etc/default/grub`` file as follows to make the GRUB menu visible
when booting:
.. code-block:: none
ACRN:\>sos_console 1
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=false
----- Entering Guest 1 Shell -----
Re-generate the GRUB configuration file and reboot the UP2 board. Select
the ``ACRN Partition Mode`` entry to boot the partition mode of the ACRN
hypervisor, the hypervisor will start the privileged VMs automatically.
root@clr-e8216ad453ad4f08914d83cbc50f838c ~ #
.. code-block:: none
---Entering ACRN SHELL---
ACRN:\>sos_console 2
$ sudo update-grub
----- Entering Guest 2 Shell -----
.. code-block:: console
:emphasize-lines: 4
root@clr-bdbe97b256864a92b44a9449bf229769 ~ #
Ubuntu
Advanced options for Ubuntu
System setup
*ACRN Partition Mode
---Entering ACRN SHELL---
ACRN:\>
Switch between privileged VMs
*****************************
Connect the serial port on the UP2 board to the development workstation.
If you set the MMIO address of the serial port right while building the
ACRN hypervisor, you should see the output from the ACRN serial console as below.
You could then log in to the privileged VMs by ``sos_console`` command,
and press :kbd:`CTRL+Space` keys to return to the ACRN serial console.
.. code-block:: console
:emphasize-lines: 14,31
ACRN Hypervisor
calibrate_tsc, tsc_khz=1094400
[21017289us][cpu=0][sev=2][seq=1]:HV version 0.3-unstable-2018-11-08 12:41:24-ef974d1a-dirty DBG (daily tag:acrn-2018w45.3-140000p) build by clear, start time 20997424us
[21034127us][cpu=0][sev=2][seq=2]:API version 1.0
[21039218us][cpu=0][sev=2][seq=3]:Detect processor: Intel(R) Pentium(R) CPU N4200 @ 1.10GHz
[21048422us][cpu=0][sev=2][seq=4]:hardware support HV
[21053897us][cpu=0][sev=1][seq=5]:SECURITY WARNING!!!!!!
[21059672us][cpu=0][sev=1][seq=6]:Please apply the latest CPU uCode patch!
[21074487us][cpu=0][sev=2][seq=28]:Start VM1
[21074488us][cpu=3][sev=2][seq=29]:Start VM2
[21885195us][cpu=0][sev=3][seq=34]:vlapic: Start Secondary VCPU1 for VM[1]...
[21889889us][cpu=3][sev=3][seq=35]:vlapic: Start Secondary VCPU1 for VM[2]...
ACRN:\>
ACRN:\>sos_console 1
----- Entering Guest 1 Shell -----
[ 1.997439] systemd[1]: Listening on Network Service Netlink Socket.
[ OK ] Listening on Network Service Netlink Socket.
[ 1.999347] systemd[1]: Created slice system-serial\x2dgetty.slice.
[ OK ] Created slice system-serial\x2dgetty.slice.
[ OK ] Listening on Journal Socket (/dev/log).
...
clr-932c8a3012ec4dc6af53790b7afbf6ba login: root
Password:
root@clr-932c8a3012ec4dc6af53790b7afbf6ba ~ # lspci
00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
00:01.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
root@clr-932c8a3012ec4dc6af53790b7afbf6ba ~ #
---Entering ACRN SHELL---
ACRN:\>sos_console 2
----- Entering Guest 2 Shell -----
[ 1.490122] usb 1-4: new full-speed USB device number 2 using xhci_hcd
[ 1.621311] usb 1-4: not running at top speed; connect to a high speed hub
[ 1.627824] usb 1-4: New USB device found, idVendor=058f, idProduct=6387, bcdDevice= 1.01
[ 1.628438] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
...
clr-2e8082cd4fc24d57a3c2d3db43368d36 login: root
Password:
root@clr-2e8082cd4fc24d57a3c2d3db43368d36 ~ # lspci
00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
00:01.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
00:02.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
root@clr-2e8082cd4fc24d57a3c2d3db43368d36 ~ #