sample_app: Add scripts to create RT and HMI VM images

Creating VM images is always a pain to users, and it is especially the case
for those who want to set up the ACRN sample application which needs two
different VM images, one with graphical desktop and the other optimized for
real-time.

This patch introduces the so-called "image builder" utility which is a set
of scripts that can automate the creation of those VM images. The scripts
will take care of:

  - Forking image files based on Ubuntu cloud images and enlarge the root
    file system per needs.

  - Setting up users and passwords.

  - Installing necessary packages to run either the graphical desktop or
    real-time applications.

  - Specific to the RT VM image, disabling services and tweaking kernel
    command line for optimized real-time performance.

  - Copying the sample applications into the images so that users can
    start them directly, after they launch the VMs.

Tracked-On: #7820
Signed-off-by: Junjie Mao <junjie.mao@intel.com>

misc/sample_application/image_builder/README.rst

@@ -0,0 +1,51 @@
+:orphan:
+
+ACRN Sample Application Image Builder
+#####################################
+
+This directory contains the scripts to create VM images for ACRN sample
+application.
+
+Prerequisites
+*************
+
+Make sure you have Ubuntu installed with network access on your development
+computer. Then execute the following command to install the prerequisites.
+
+   .. code-block:: bash
+
+      sudo apt install -y kpartx \
+           schroot \
+           mount \
+	   wget \
+	   qemu-utils
+
+Also you'll need the Debian packages of Linux RT kernels built from
+https://github.com/projectacrn/acrn-kernel when (and only when) building a
+real-time VM image. After the kernel is built, copy those Debian packages (whose
+names looks like ``linux-libc-*``, ``linux-headers-*`` and ``linux-image-*``) to
+this directory (i.e. misc/sample_application/image_builder/).
+
+Build images
+************
+
+To build the VM image for graphical HMI, run the following command under this
+directory:
+
+   .. code-block:: bash
+
+      ./create_image.sh hmi-vm
+
+This will generate an image named ``hmi_vm.img`` under the this directory, which
+can be used as the file of a virtio-blk device of a post-launched VM. Installing
+the GNOME desktop system will take some time depending on your network and
+storage speed.
+
+To build the VM image for running real-time applications, run the following
+command under this directory:
+
+   .. code-block:: bash
+
+      ./create_image.sh rt-vm
+
+This will generate an image named ``rt_vm.img`` under the this directory.

misc/sample_application/image_builder/configRTcores.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Copyright (C) 2020-2022 Intel Corporation.
+# SPDX-License-Identifier: BSD-3-Clause
+
+#Move all IRQs to core 0.
+for i in `cat /proc/interrupts | grep '^ *[0-9]*[0-9]:' | awk {'print $1'} | sed 's/:$//' `;
+do
+    echo setting $i to affine for core zero
+    echo 1 > /proc/irq/$i/smp_affinity
+done
+
+#Move all rcu tasks to core 0.
+for i in `pgrep rcu`; do taskset -pc 0 $i; done
+
+#Change realtime attribute of all rcu tasks to SCHED_OTHER and priority 0
+for i in `pgrep rcu`; do chrt -v -o -p 0 $i; done
+
+#Change realtime attribute of all tasks on core 1 to SCHED_OTHER and priority 0
+for i in `pgrep /1`; do chrt -v -o -p 0 $i; done
+
+#Change realtime attribute of all tasks to SCHED_OTHER and priority 0
+for i in `ps -A -o pid`; do chrt -v -o -p 0 $i; done
+
+echo disabling timer migration
+echo 0 > /proc/sys/kernel/timer_migration

misc/sample_application/image_builder/create_image.sh

@@ -0,0 +1,202 @@
+#!/bin/bash
+# Copyright (C) 2020-2022 Intel Corporation.
+# SPDX-License-Identifier: BSD-3-Clause
+
+cloud_image=focal-server-cloudimg-amd64.img
+cloud_image_url=https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
+hmi_vm_image=hmi_vm.img
+rt_vm_image=rt_vm.img
+rt_kernel=(linux-libc linux-headers linux-image)
+
+vm_type=$1
+if [[ ${vm_type} != "rt-vm" ]] && [[ ${vm_type} != "hmi-vm" ]]; then
+    cat <<EOT
+Usage: $0 <vm_type>
+This script creates VM images based on Ubuntu cloud images.
+
+VM type options:
+  hmi-vm        create a VM with GNOME desktop
+  rt-vm         create a VM with a preempt-RT-patched kernel and real-time test utilities
+EOT
+    exit 1
+fi
+
+########################################
+# Environment checks
+########################################
+
+if [[ ! -d /etc/schroot/chroot.d ]]; then
+    echo "Package schroot is not installed."
+    exit 1
+fi
+
+if [[ ! -d ../build ]]; then
+    echo "Please make the SampleApplication at first."
+    exit 1
+fi
+
+if [ ! -d mnt ]; then
+    mkdir mnt
+fi
+
+########################################
+# Helper functions
+########################################
+
+source logger.sh
+
+########################################
+# Actions defined as functions
+########################################
+
+function check_rt_kernel() {
+    for file in ${rt_kernel[@]}
+    do
+        ls *.deb | grep ${file}
+        if [ $? -eq 1 ]; then
+            echo "RT VM kernel package ${file} is not found."
+            exit
+        fi
+    done
+}
+
+function download_image() {
+    local dest=$1
+    local url=$2
+
+    if [[ -f ${dest} ]]; then
+        print_info "${dest} already exists. Do not redownload."
+    else
+        wget -O ${dest} ${url}
+    fi
+}
+
+function copy_and_enlarge_image() {
+    local source_image=$1
+    local dest_image=$2
+    local size_modifier=$3
+
+    if [[ -f ${dest_image} ]]; then
+        echo -n "${dest_image} already exists! Regenerate the image? (y/N)> "
+        read answer
+        if [[ $answer =~ ^[Yy]$ ]]; then
+            rm ${dest_image}
+        else
+            exit 1
+        fi
+    fi
+
+    qemu-img convert -f qcow2 -O raw ${source_image} ${dest_image} && \
+        qemu-img resize -f raw ${dest_image} ${size_modifier} && \
+        growpart ${dest_image} 1
+}
+
+function resizing_guest_root() {
+    local part_file=$1
+
+    sudo e2fsck -f ${part_file} && \
+        sudo resize2fs ${part_file}
+}
+
+function mount_filesystem() {
+    local part_file=$1
+    local mount_point=$2
+
+    # The symlink /etc/resolv.conf in a fresh cloud image is broken, which will
+    # prevent schroot from working. Touch that linked file to work it around.
+    mkdir -p ${mount_point} && \
+        sudo mount ${part_file} ${mount_point} && \
+        sudo mkdir -p ${mount_point}/run/systemd/resolve/ && \
+        sudo touch ${mount_point}/run/systemd/resolve/stub-resolv.conf
+}
+
+function create_schroot_config() {
+    local mount_point=$1
+    local temp_file=$(mktemp /tmp/acrn-guest.XXXX)
+
+    cat << EOF > ${temp_file}
+[acrn-guest]
+description=Contains ACRN guest root file system.
+type=directory
+directory=${mount_point}
+users=root
+root-groups=root
+profile=desktop
+personality=linux
+preserve-environment=true
+EOF
+
+    sudo mv ${temp_file} /etc/schroot/chroot.d/acrn-guest && \
+        sudo chown root:root /etc/schroot/chroot.d/acrn-guest
+}
+
+function setup_hmi_vm_rootfs() {
+    local mount_point=$1
+
+    sudo cp setup_hmi_vm.sh logger.sh ${mount_point}/ && \
+	sudo cp ../build/userApp ${mount_point}/root && \
+	sudo cp ../build/histapp.py ${mount_point}/root && \
+        sudo schroot -c acrn-guest bash /setup_hmi_vm.sh && \
+        sudo rm ${mount_point}/setup_hmi_vm.sh ${mount_point}/logger.sh
+}
+
+function setup_rt_vm_rootfs() {
+    local mount_point=$1
+
+    sudo cp *.deb ${mount_point}/root && \
+	sudo cp ../build/rtApp ${mount_point}/root && \
+        sudo mkdir ${mount_point}/root/scripts && \
+        sudo cp configRTcores.sh ${mount_point}/root/scripts/ && \
+        sudo cp setup_rt_vm.sh logger.sh ${mount_point}/ && \
+        sudo schroot -c acrn-guest bash /setup_rt_vm.sh && \
+        sudo rm ${mount_point}/setup_rt_vm.sh ${mount_point}/logger.sh
+}
+
+function cleanup() {
+    local mount_point=$1
+    local loop_dev=$2
+
+    sudo umount ${mount_point}
+    sudo rmdir ${mount_point}
+    sudo kpartx -vd /dev/${loop_dev}
+    sudo losetup -vd /dev/${loop_dev}
+    true
+}
+
+########################################
+# Do it!
+########################################
+
+mount_point=$(pwd)/mnt
+if [[ ${vm_type} == "hmi-vm" ]]; then
+    target_image=${hmi_vm_image}
+    size_modifier="+4G"
+elif [[ ${vm_type} == "rt-vm" ]]; then
+    target_image=${rt_vm_image}
+    size_modifier="+1G"
+else
+    echo "Internal error: undefined VM type '${vm_type}'"
+    exit 1
+fi
+
+try_step "Download Ubuntu Focal cloud image" download_image ${cloud_image} ${cloud_image_url}
+if [[ ${vm_type} == "rt-vm" ]]; then
+    try_step "Check availability of RT kernel image" check_rt_kernel
+fi
+try_step "Creating an enlarged copy of ${cloud_image}" copy_and_enlarge_image ${cloud_image} ${target_image} ${size_modifier}
+
+loop_dev=$(sudo kpartx -va ${target_image} 2>&1 | egrep -o -m 1 "loop[0-9]+")
+print_info "Guest image loop-mounted at /dev/${loop_dev}"
+
+try_step "Resizing guest root file system" resizing_guest_root /dev/mapper/${loop_dev}p1
+try_step "Mounting guest root file system at ${mount_point}" mount_filesystem /dev/mapper/${loop_dev}p1 ${mount_point}
+try_step "Preparing schroot configuration" create_schroot_config ${mount_point}
+
+if [[ ${vm_type} == "hmi-vm" ]]; then
+    try_step "Initializing guest root file system for HMI VM" setup_hmi_vm_rootfs ${mount_point}
+else
+    try_step "Initializing guest root file system for RT VM" setup_rt_vm_rootfs ${mount_point}
+fi
+
+do_step  "Cleaning up" cleanup ${mount_point} ${loop_dev}
+print_info "VM image created at ${target_image}."

misc/sample_application/image_builder/logger.sh

@@ -0,0 +1,40 @@
+# Copyright (C) 2020-2022 Intel Corporation.
+# SPDX-License-Identifier: BSD-3-Clause
+
+RED="\033[0;31m"
+YELLOW="\033[1;33m"
+GREEN="\033[0;32m"
+NO_COLOR="\033[0m"
+
+has_error=0
+
+function do_step() {
+    local prompt=$1
+    local func=$2
+    shift 2
+
+    echo -e "$(date -Iseconds) ${logger_prefix}${YELLOW}[ Starting ]${NO_COLOR} ${prompt}"
+    if $func $*; then
+        echo -e "$(date -Iseconds) ${logger_prefix}${GREEN}[   Done   ]${NO_COLOR} ${prompt}"
+    else
+        echo -e "$(date -Iseconds) ${logger_prefix}${RED}[  Failed  ]${NO_COLOR} ${prompt}"
+        has_error=1
+    fi
+}
+
+function try_step() {
+    local prompt=$1
+    shift 1
+
+    if [[ ${has_error} != 0 ]]; then
+        echo -e "$(date -Iseconds) ${logger_prefix}${YELLOW}[ Skipped  ]${NO_COLOR} ${prompt}"
+    else
+        do_step "$prompt" $*
+    fi
+}
+
+function print_info() {
+    if [[ ${has_error} == 0 ]]; then
+        echo -e "$(date -Iseconds) ${logger_prefix}${YELLOW}[   Info   ]${NO_COLOR} $*"
+    fi
+}

misc/sample_application/image_builder/setup_hmi_vm.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# Copyright (C) 2020-2022 Intel Corporation.
+# SPDX-License-Identifier: BSD-3-Clause
+
+logger_prefix="(hmi-vm-rootfs) "
+source logger.sh
+
+function umount_directory() {
+    target_dir=$1
+    umount -q ${target_dir} || true
+}
+
+function update_package_info() {
+    apt update
+    apt install python3 python3-pip net-tools python3-matplotlib
+    pip3 install flask numpy pandas posix_ipc
+
+}
+
+function install_desktop() {
+    apt install ubuntu-gnome-desktop
+}
+
+function change_root_password() {
+    passwd root
+}
+
+function add_normal_user() {
+    useradd -s /bin/bash -d /home/acrn/ -m -G sudo acrn && \
+        passwd acrn
+}
+
+# Change current working directory to the root to avoid "target is busy" errors
+# on unmounting.
+cd /
+
+try_step "Unmounting /root" umount_directory /root
+try_step "Unmounting /home" umount_directory /home
+try_step "Updating package information" update_package_info
+try_step "Installing GNOME desktop" install_desktop
+try_step "Changing the password of the root user" change_root_password
+try_step "Adding the normal user acrn" add_normal_user

misc/sample_application/image_builder/setup_rt_vm.sh

@@ -0,0 +1,78 @@
+#!/bin/bash
+# Copyright (C) 2020-2022 Intel Corporation.
+# SPDX-License-Identifier: BSD-3-Clause
+
+logger_prefix="(rt-vm-rootfs) "
+source logger.sh
+
+function umount_directory() {
+    target_dir=$1
+    umount -q ${target_dir} || true
+}
+
+function disable_os_prober() {
+    if [[ -f /etc/grub.d/30_os-prober ]]; then
+        mv /etc/grub.d/30_os-prober /etc/grub.d/.30_os-prober
+    fi
+}
+
+function update_package_info() {
+    apt update
+}
+
+function install_tools() {
+    apt install rt-tests
+}
+
+function update_kernel_cmdline() {
+    cat <<EOF >> /etc/default/grub
+
+GRUB_CMDLINE_LINUX="rootwait rootfstype=ext4 console=ttyS0,115200 console=tty0 rw nohpet console=hvc0 no_timer_check ignore_loglevel log_buf_len=16M consoleblank=0 tsc=reliable clocksource=tsc tsc=reliable x2apic_phys processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=disable mce=ignore_ce audit=0 isolcpus=nohz,domain,1 nohz_full=1 rcu_nocbs=1 nosoftlockup idle=poll irqaffinity=0 no_ipi_broadcast=1"
+EOF
+}
+
+function install_rt_kernel() {
+    search_dir=$1
+    for file in $(ls -r ${search_dir}/*acrn-kernel-*.deb)
+    do
+        apt install ${file}
+    done
+}
+
+function change_root_password() {
+    passwd root
+}
+
+function disable_services() {
+    services=(systemd-timesyncd.service \
+                  systemd-journald.service \
+                  systemd-journal-flush.service \
+                  serial-getty@ttyS2.service \
+                  apt-daily.service \
+                  apt-daily-upgrade.service)
+    for service in ${services[*]}
+    do
+        systemctl disable ${service}
+        systemctl mask ${service}
+    done
+
+    for timer in $(systemctl list-unit-files | grep -o "^.*\.timer"); do
+        systemctl disable ${timer}
+    done
+
+    apt-get remove unattended-upgrades
+ }
+
+# Change current working directory to the root to avoid "target is busy" errors
+# on unmounting.
+cd /
+
+try_step "Unmounting /root" umount_directory /root
+try_step "Unmounting /home" umount_directory /home
+try_step "Disabling GRUB OS prober" disable_os_prober
+try_step "Updating package information" update_package_info
+try_step "Installing tools" install_tools
+try_step "Updating kernel command line" update_kernel_cmdline
+try_step "Installing RT kernel" install_rt_kernel /root
+try_step "Changing the password of the root user" change_root_password
+try_step "Disabling services that impact real-time performance" disable_services