diff --git a/doc/tutorials/images/menuconfig-partition-mode.png b/doc/tutorials/images/menuconfig-partition-mode.png
new file mode 100644
index 000000000..d647ac6c3
Binary files /dev/null and b/doc/tutorials/images/menuconfig-partition-mode.png differ
diff --git a/doc/tutorials/images/partition_mode_up2.png b/doc/tutorials/images/partition_mode_up2.png
new file mode 100644
index 000000000..a8d9c8a71
Binary files /dev/null and b/doc/tutorials/images/partition_mode_up2.png differ
diff --git a/doc/tutorials/using_partition_mode_on_up2.rst b/doc/tutorials/using_partition_mode_on_up2.rst
index 58c1e5bb5..4f43b2c35 100644
--- a/doc/tutorials/using_partition_mode_on_up2.rst
+++ b/doc/tutorials/using_partition_mode_on_up2.rst
@@ -2,154 +2,309 @@
Using partition mode on UP2
###########################
-This tutorial describes the steps needed to use partition mode with the ACRN
-hypervisor on UP2.
-Build kernel and modules for partition mode guest
-*************************************************
+ACRN hypervisor supports partition mode, in which the User OS running in a
+privileged VM can bypass the ACRN hypervisor and directly access isolated
+PCI devices. This tutorial provides step by step instructions on how to set up
+the ACRN hypervisor partition mode on
+`UP2 `_ boards running two
+privileged VMs as shown in :numref:`two-priv-vms`:
-* Build kernel and modules
+.. figure:: images/partition_mode_up2.png
+ :align: center
+ :name: two-priv-vms
+
+ Two privileged VMs running in partition mode
- .. code-block:: none
+Prerequisites
+*************
- git clone https://github.com/projectacrn/acrn-kernel.git
- cd ~/acrn-kernel
- cp kernel_config_partition_os .config
- make oldconfig
- make
+In this tutorial two Linux privileged VMs are started by the ACRN hypervisor.
+To set up the Linux root filesystems for each VM, follow the Clear Linux
+`bare metal installation guide `_
+to install Clear Linux on a **SATA disk** and a **USB flash disk** prior the setup,
+as the two privileged VMs will mount the root filesystems via the SATA controller
+and the USB controller respectively.
-* Install modules
+Build kernel and modules for partition mode UOS
+***********************************************
- First of all, please prepare one USB flash drive and one SATA disk. Plug them
- in one PC and follow Clear Linux's `online instruction ` to install Clear Linux on them.
+#. On your development workstation, clone the ACRN kernel source tree, and
+ build the Linux kernel image that will be used to boot the privileged VMs:
- Plug the USB flash drive and SATA disk into your PC. Assuming Clear Linux
- rootfs is installed into /dev/sdx3.
+ .. code-block:: none
- .. code-block:: none
+ $ git clone https://github.com/projectacrn/acrn-kernel.git
+ Cloning into 'acrn-kernel'...
+ ...
+ $ cd acrn-kernel
+ $ cp kernel_config_uos .config
+
+ $ make olddefconfig
+ scripts/kconfig/conf --olddefconfig Kconfig
+ #
+ # configuration written to .config
+ #
- mount /dev/sdx3 /mnt/sdx3
- cd ~/acrn-kernel
- make INSTALL_MOD_PATH=/mnt/sdx3/ modules_install
+ $ make
+ $ make modules_install INSTALL_MOD_PATH=out/
-* Install kernel
+ The last two commands built the bootable kernel image ``arch/x86/boot/bzImage``,
+ and the loadable kernel modules under the ``./out/`` folder. Copy these files
+ to a removable disk for installing on the UP2 board later.
- First, please following Ubuntu's `online instruction ` to install it on your UP2's eMMC.
+#. Current ACRN partition mode implementation requires a multi-boot capable
+ bootloader to boot both ACRN hypervisor and the bootable kernel image
+ built from the previous step. You could install Ubuntu OS to the UP2 board
+ by following
+ `this Ubuntu tutorial `_.
+ The Ubuntu installer creates 3 disk partitions on the on-board eMMC memory.
+ By default, the GRUB bootloader is installed on the ESP (EFI System Partition)
+ partition, which will be used to bootstrap the partiton mode ACRN hypervisor.
- Copy kernel image to EFI System Partition
+#. After installing the Ubuntu OS, power off the UP2 board, attach the SATA disk
+ and the USB flash disk to the board. Power on the board and make sure
+ it boots the Ubuntu OS from the eMMC, then copy the loadable kernel modules
+ built in Step 1 to the ``/lib/modules/`` folder on both the mounted SATA
+ disk and USB disk. For example, assuming the SATA disk and USB flash disk
+ are assigned to ``/dev/sda`` and ``/dev/sdb`` respectively, the following
+ commands set up the partition mode loadable kernel modules onto the root
+ filesystems to be loaded by the privileged VMs:
- .. code-block:: none
+ .. code-block:: none
- scp username@IP:~/acrn-kernel/arch/x86/boot/bzImage /boot/efi/
+ # Mount the Clear Linux root filesystem on the SATA disk
+ $ sudo mount /dev/sda3 /mnt
+ $ sudo cp -r /lib/modules/* /mnt/lib/modules
+ $ sudo umount /mnt
+
+ # Mount the Clear Linux root filesystem on the USB flash disk
+ $ sudo mount /dev/sdb3 /mnt
+ $ sudo cp -r /lib/modules/* /mnt/lib/modules
+ $ sudo umount /mnt
-Enabling partition mode
-***********************
+#. Copy the bootable kernel image to the ``/boot`` directory:
-* Clone ACRN source code
+ .. code-block:: none
- .. code-block:: none
+ $ sudo cp /bzImage /boot/
- git clone https://github.com/projectacrn/acrn-hypervisor.git
+Enable partition mode in ACRN hypervisor
+****************************************
-* Configure the ACRN hypervisor to use partition mode
+#. Before building the ACRN hypervisor, you need to figure out the I/O address
+ of the serial port, and the PCI BAR addresses of the SATA controller and
+ the USB controller on your UP2 board.
- .. code-block:: none
+ Enter the following command to get the I/O addresses of the serial ports.
+ UP2 boards support two serial ports. The addresses from the command
+ output represent the I/O addresses of the serial port of the 10-pin side
+ connector and the 40-pin expansion header respectively. You will need to
+ connect the serial port to the development host, in order to access
+ the ACRN serial console to switch between privileged VMs.
- cd ~/acrn-hypervisor/hypervisor
- make menuconfig
+ .. code-block:: none
+ :emphasize-lines: 1
- Select entry ``Hypervisor mode (Sharing mode)`` and press ``Enter``.
- Then select ``Partition mode`` sub-entry.
+ $ dmesg | grep ttyS
+ [ 10.552369] dw-apb-uart.8: ttyS4 at MMIO 0x91524000 (irq = 4, base_baud = 115200) is a 16550A
+ [ 10.552568] serial serial0: tty port ttyS4 registered
+ [ 10.572344] dw-apb-uart.9: ttyS5 at MMIO 0x91522000 (irq = 5, base_baud = 115200) is a 16550A
+ [ 10.572545] serial serial1: tty port ttyS5 registered
-* Configure the ACRN hypervisor's ``serial base address``
+ The following command prints detailed information about all PCI buses
+ and devices in the system. Look up the PCI BAR addresses of the SATA
+ controller and the USB controller, record these addresses (highlighted below) for
+ editing the source code later.
- .. code-block:: none
+ .. code-block:: none
+ :emphasize-lines: 1,9,10,14,22
- cd ~/acrn-hypervisor/hypervisor
- make menuconfig
+ $ sudo lspci -vv
+ ...
+ 00:12.0 SATA controller: Intel Corporation Device 5ae3 (rev 0b) (prog-if 01 [AHCI 1.0])
+ Subsystem: Intel Corporation Device 7270
+ Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
+ Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- rw rootwait noxsave maxcpus=2 nohpet console=hvc0 \
- console=ttyS0 no_timer_check ignore_loglevel log_buf_len=16M \
- consoleblank=0 tsc=reliable xapic_phys",
- .vpci_vdev_array = &vpci_vdev_array1,
+ ...
+ /* Virtual Machine descriptions */
+ .vm_desc_array = {
+ {
+ /* Internal variable, MUSTBE init to -1 */
+ .vm_hw_num_cores = VM1_NUM_CPUS,
+ .vm_pcpu_ids = &VM1_CPUS[0],
+ ...
+ .bootargs = "root=/dev/sda3 rw rootwait noxsave maxcpus=2 nohpet console=hvc0 \
+ console=ttyS2 no_timer_check ignore_loglevel log_buf_len=16M \
+ consoleblank=0 tsc=reliable xapic_phys",
+ .vpci_vdev_array = &vpci_vdev_array1,
+ .mptable = &mptable_vm1,
+ },
+ {
+ /* Internal variable, MUSTBE init to -1 */
+ .vm_hw_num_cores = VM2_NUM_CPUS,
+ .vm_pcpu_ids = &VM2_CPUS[0],
+ ...
+ .bootargs = "root=/dev/sda3 rw rootwait noxsave maxcpus=2 nohpet console=hvc0 \
+ console=ttyS2 no_timer_check ignore_loglevel log_buf_len=16M \
+ consoleblank=0 tsc=reliable xapic_phys",
+ .vpci_vdev_array = &vpci_vdev_array2,
+ .mptable = &mptable_vm2,
+ .lapic_pt = true,
+ },
+ }
-* Build ACRN with the above changes.
+#. Build the ACRN hypervisor and copy the artifact ``acrn.32.out`` to the
+ ``/boot`` directory:
- .. code-block:: none
+ .. code-block:: none
$ make PLATFORM=sbl
+ ...
+ $ sudo cp build/acrn.32.out /boot
+#. Modify the ``/etc/grub.d/40_custom`` file to create a new GRUB entry
+ that will multi-boot the ACRN hypervisor and the UOS kernel image
-* Install ACRN
+ Append the following configuration to the ``/etc/grub.d/40_custom`` file:
- First we should add one menuentry in ``/boot/efi/grub/grub.cfg`` on UP2 board
-
- .. code-block:: none
+ .. code-block:: none
menuentry 'ACRN Partition Mode' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-e23c76ae-b06d-4a6e-ad42-46b8eedfd7d3' {
recordfail
@@ -160,61 +315,88 @@ Enabling partition mode
insmod ext2
echo 'Loading partition mode hypervisor ...'
- multiboot /acrn.32.out
- module /bzImage
+ multiboot /boot/acrn.32.out
+ module /boot/bzImage
}
- Copy ``acrn.32.out`` and bzImage to ``/boot/efi``
-
- .. code-block:: none
-
- scp username@IP:~/acrn-hypervisor/hypervisor/build/acrn.32.out /boot/efi
-
-* Boot ACRN
-
- Plug the USB flash drive and SATA Disk into your UP2 board and then reboot the
- board to the built-in EFI shell. Then execute the following command to load
- the ACRN hypervisor.
-
- .. code-block:: none
-
- shell> fs1:
- fs1:\> EFI\ubuntu\grubx64.efi
-
- Here we will see the following GRUB menu. Select entry ``ACRN Partition Mode`` and
- press ``Enter`` and then the ACRN hypervisor will be loaded automatically.
-
- .. code-block:: console
- :emphasize-lines: 2
-
- *Ubuntu
- ACRN Partition Mode
- Advanced options for Ubuntu
- System setup
- Restore Ubuntu 16.04 to factory state
-
-
-Playing with ACRN hypervisor with partition mode
-************************************************
-
-* Connect the serial port of the UP2 board with another PC and use the following
- command to check if vms booted successfully. If vms are booted successfully,
- we will see the following message.
+ Modify the ``/etc/default/grub`` file as follows to make the GRUB menu visible
+ when booting:
.. code-block:: none
- ACRN:\>sos_console 1
+ #GRUB_HIDDEN_TIMEOUT=0
+ GRUB_HIDDEN_TIMEOUT_QUIET=false
- ----- Entering Guest 1 Shell -----
+ Re-generate the GRUB configuration file and reboot the UP2 board. Select
+ the ``ACRN Partition Mode`` entry to boot the partition mode of the ACRN
+ hypervisor, the hypervisor will start the privileged VMs automatically.
- root@clr-e8216ad453ad4f08914d83cbc50f838c ~ #
+ .. code-block:: none
- ---Entering ACRN SHELL---
- ACRN:\>sos_console 2
+ $ sudo update-grub
- ----- Entering Guest 2 Shell -----
+ .. code-block:: console
+ :emphasize-lines: 4
- root@clr-bdbe97b256864a92b44a9449bf229769 ~ #
+ Ubuntu
+ Advanced options for Ubuntu
+ System setup
+ *ACRN Partition Mode
- ---Entering ACRN SHELL---
- ACRN:\>
+Switch between privileged VMs
+*****************************
+
+Connect the serial port on the UP2 board to the development workstation.
+If you set the MMIO address of the serial port right while building the
+ACRN hypervisor, you should see the output from the ACRN serial console as below.
+You could then log in to the privileged VMs by ``sos_console`` command,
+and press :kbd:`CTRL+Space` keys to return to the ACRN serial console.
+
+.. code-block:: console
+ :emphasize-lines: 14,31
+
+ ACRN Hypervisor
+ calibrate_tsc, tsc_khz=1094400
+ [21017289us][cpu=0][sev=2][seq=1]:HV version 0.3-unstable-2018-11-08 12:41:24-ef974d1a-dirty DBG (daily tag:acrn-2018w45.3-140000p) build by clear, start time 20997424us
+ [21034127us][cpu=0][sev=2][seq=2]:API version 1.0
+ [21039218us][cpu=0][sev=2][seq=3]:Detect processor: Intel(R) Pentium(R) CPU N4200 @ 1.10GHz
+ [21048422us][cpu=0][sev=2][seq=4]:hardware support HV
+ [21053897us][cpu=0][sev=1][seq=5]:SECURITY WARNING!!!!!!
+ [21059672us][cpu=0][sev=1][seq=6]:Please apply the latest CPU uCode patch!
+ [21074487us][cpu=0][sev=2][seq=28]:Start VM1
+ [21074488us][cpu=3][sev=2][seq=29]:Start VM2
+ [21885195us][cpu=0][sev=3][seq=34]:vlapic: Start Secondary VCPU1 for VM[1]...
+ [21889889us][cpu=3][sev=3][seq=35]:vlapic: Start Secondary VCPU1 for VM[2]...
+ ACRN:\>
+ ACRN:\>sos_console 1
+
+ ----- Entering Guest 1 Shell -----
+ [ 1.997439] systemd[1]: Listening on Network Service Netlink Socket.
+ [ OK ] Listening on Network Service Netlink Socket.
+ [ 1.999347] systemd[1]: Created slice system-serial\x2dgetty.slice.
+ [ OK ] Created slice system-serial\x2dgetty.slice.
+ [ OK ] Listening on Journal Socket (/dev/log).
+ ...
+ clr-932c8a3012ec4dc6af53790b7afbf6ba login: root
+ Password:
+ root@clr-932c8a3012ec4dc6af53790b7afbf6ba ~ # lspci
+ 00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
+ 00:01.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b)
+ root@clr-932c8a3012ec4dc6af53790b7afbf6ba ~ #
+
+ ---Entering ACRN SHELL---
+ ACRN:\>sos_console 2
+
+ ----- Entering Guest 2 Shell -----
+ [ 1.490122] usb 1-4: new full-speed USB device number 2 using xhci_hcd
+ [ 1.621311] usb 1-4: not running at top speed; connect to a high speed hub
+ [ 1.627824] usb 1-4: New USB device found, idVendor=058f, idProduct=6387, bcdDevice= 1.01
+ [ 1.628438] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
+ ...
+ clr-2e8082cd4fc24d57a3c2d3db43368d36 login: root
+ Password:
+ root@clr-2e8082cd4fc24d57a3c2d3db43368d36 ~ # lspci
+ 00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
+ 00:01.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b)
+ 00:02.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
+ root@clr-2e8082cd4fc24d57a3c2d3db43368d36 ~ #