.. _setup_openstack_libvirt:

Configure ACRN Using OpenStack and Libvirt
##########################################

Introduction
************

This document provides instructions for setting up libvirt to configure
ACRN. We use OpenStack to use libvirt. We'll show how to install OpenStack in a
container to avoid crashing your system and to take advantage of easy
snapshots and restores so that you can quickly roll back your system in the
event of setup failure. (You should only install OpenStack directly on Ubuntu if
you have a dedicated testing machine.) This setup utilizes LXC/LXD on
Ubuntu 20.04.

Install ACRN
************

#. Install ACRN using Ubuntu 20.04 as its Service VM. Refer to
   :ref:`gsg`.

#. Make the acrn-kernel using the `kernel_config_service_vm
   <https://raw.githubusercontent.com/projectacrn/acrn-kernel/master/kernel_config_service_vm>`_
   configuration file (from the ``acrn-kernel`` repo).

#. Append the following kernel boot arguments to the ``multiboot2`` line in
   :file:`/etc/grub.d/40_custom` and run ``sudo update-grub`` before rebooting the system.
   It will give the Service VM  more memory and more loop devices::

      hugepagesz=1G hugepages=10 max_loop=16

#. Boot the Service VM with this new ``acrn-kernel`` using the ACRN
   hypervisor.
#. Use the command: ``losetup -a`` to verify that Ubuntu's snap service is **not**
   using all available loop devices. Typically, OpenStack needs at least 4
   available loop devices. Follow the `snaps guide
   <https://maslosoft.com/kb/how-to-clean-old-snaps/>`_ to clean up old
   snap revisions if you're running out of loop devices.
#. Make sure the networking bridge ``acrn-br0`` is created. See
   :ref:`hostbridge_virt_hld` for more information.

Set Up and Launch LXC/LXD
*************************

1. Set up the LXC/LXD Linux container engine::

      $ sudo snap install lxd
      $ lxd init --auto

   Use all default values if running ``lxd init`` in interactive mode.

2. Create an Ubuntu 18.04 container named ``openstack``::

      $ lxc init ubuntu:18.04 openstack

3. Export the kernel interfaces necessary to launch a Service VM in the
   ``openstack`` container:

   a. Edit the ``openstack`` config file using the command::

      $ lxc config edit openstack

      In the editor, add the following lines in the **config** section:

      .. code-block:: none

         linux.kernel_modules: iptable_nat, ip6table_nat, ebtables, openvswitch
         raw.lxc: |-
           lxc.cgroup.devices.allow = c 10:237 rwm
           lxc.cgroup.devices.allow = b 7:* rwm
           lxc.cgroup.devices.allow = c 243:0 rwm
           lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file 0 0
           lxc.mount.auto=proc:rw sys:rw cgroup:rw
           lxc.apparmor.profile=unconfined
         security.nesting: "true"
         security.privileged: "true"

      Save and exit the editor.

      .. note::

         Make sure to respect the indentation as to keep these options within
         the **config** section. After saving your changes,
         check that they have been correctly recorded (``lxc config show openstack``).

   b. Run the following commands to configure ``openstack``::

         $ lxc config device add openstack eth1 nic name=eth1 nictype=bridged parent=acrn-br0
         $ lxc config device add openstack acrn_hsm unix-char path=/dev/acrn_hsm
         $ lxc config device add openstack loop-control unix-char path=/dev/loop-control
         $ for n in {0..15}; do lxc config device add openstack loop$n unix-block path=/dev/loop$n; done;

4. Launch the ``openstack`` container::

     $ lxc start openstack

5. Log in to the ``openstack`` container::

     $ lxc exec openstack -- su -l

6. Let ``systemd`` manage **eth1** in the container, with **eth0** as the
   default route:

   Edit ``/etc/netplan/50-cloud-init.yaml`` as follows:

   .. code-block:: none

      network:
          version: 2
          ethernets:
              eth0:
                  dhcp4: true
              eth1:
                  dhcp4: true
                  dhcp4-overrides:
                      route-metric: 200


7. Log off and restart the ``openstack`` container::

     $ lxc restart openstack

8. Log in to the ``openstack`` container again::

     $ lxc exec openstack -- su -l

9. If needed, set up the proxy inside the ``openstack`` container via
   ``/etc/environment`` and make sure ``no_proxy`` is properly set up.
   Both IP addresses assigned to **eth0** and
   **eth1** and their subnets must be included. For example::

     no_proxy=xcompany.com,.xcompany.com,10.0.0.0/8,192.168.0.0/16,localhost,.local,127.0.0.0/8,134.134.0.0/16

10. Add a new user named **stack** and set permissions:

    .. code-block:: none

       # useradd -s /bin/bash -d /opt/stack -m stack
       # echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

11. Log off and restart the ``openstack`` container::

     $ lxc restart openstack

The ``openstack`` container is now properly configured for OpenStack.
Use the ``lxc list`` command to verify that both **eth0** and **eth1**
appear in the container.

Set Up ACRN Prerequisites Inside the Container
**********************************************

1. Log in to the ``openstack`` container as the **stack** user::

     $ lxc exec openstack -- su -l stack

2. Download and compile ACRN's source code. Refer to :ref:`gsg`.

   .. note::
      All tools and build dependencies must be installed before you run the first ``make`` command.

   .. code-block:: none

      $ cd ~
      $ git clone https://github.com/projectacrn/acrn-hypervisor
      $ cd acrn-hypervisor
      $ make
      $ sudo make devicemodel-install

   Install only the user-space component: ``acrn-dm`` as shown above.

   .. note:: Use the tag that matches the version of the ACRN hypervisor (``acrn.bin``)
      that runs on your system.

Set Up Libvirt
**************

1. Install the required packages::

     $ sudo apt install libdevmapper-dev libnl-route-3-dev libnl-3-dev python \
       automake autoconf autopoint libtool xsltproc libxml2-utils gettext \
       libxml2-dev libpciaccess-dev gnutls-dev python3-docutils libyajl-dev


2. Download libvirt/ACRN::

     $ cd ~
     $ git clone https://github.com/projectacrn/acrn-libvirt.git

3. Build and install libvirt::

     $ cd acrn-libvirt
     $ mkdir build
     $ cd build
     $ ../autogen.sh --prefix=/usr --disable-werror --with-test-suite=no \
       --with-qemu=no --with-openvz=no --with-vmware=no --with-phyp=no \
       --with-vbox=no --with-lxc=no --with-uml=no --with-esx=no --with-yajl

     $ make
     $ sudo make install

   .. note:: The ``dev-acrn-v6.1.0`` branch is used in this tutorial and is
      the default branch.

4. Edit and enable these options in ``/etc/libvirt/libvirtd.conf``::

     unix_sock_ro_perms = "0777"
     unix_sock_rw_perms = "0777"
     unix_sock_admin_perms = "0777"

5. Restart the libvirt daemon::

     $ sudo systemctl daemon-reload


Set Up OpenStack
****************

Use DevStack to install OpenStack. Refer to the `DevStack instructions <https://docs.openstack.org/devstack/>`_.

1. Use the latest maintenance branch **stable/train** to ensure OpenStack
   stability::

      $ cd ~
      $ git clone https://opendev.org/openstack/devstack.git -b stable/train

2. Go into the ``devstack`` directory, and apply the
   :file:`doc/tutorials/0001-devstack-installation-for-acrn.patch`::

      $ cd devstack
      $ git apply ~/acrn-hypervisor/doc/tutorials/0001-devstack-installation-for-acrn.patch

3. Edit ``lib/nova_plugins/hypervisor-libvirt``:

   Change ``xen_hvmloader_path`` to the location of your OVMF image
   file: ``/usr/share/acrn/bios/OVMF.fd``. Or use the stock image that is included
   in the ACRN source tree (``devicemodel/bios/OVMF.fd``).

4. Create a ``devstack/local.conf`` file as shown below (setting the
   passwords as appropriate):

   .. code-block:: none


      [[local|localrc]]
      PUBLIC_INTERFACE=eth1

      ADMIN_PASSWORD=<password>
      DATABASE_PASSWORD=<password>
      RABBIT_PASSWORD=<password>
      SERVICE_PASSWORD=<password>

      ENABLE_KSM=False
      VIRT_DRIVER=libvirt
      LIBVIRT_TYPE=acrn
      DEBUG_LIBVIRT=True
      DEBUG_LIBVIRT_COREDUMPS=True
      USE_PYTHON3=True

   .. note::

      Now is a great time to take a snapshot of the container using ``lxc
      snapshot``. If the OpenStack installation fails, manually rolling back
      to the previous state can be difficult. No step exists to
      reliably restart OpenStack after restarting the container.

5. Install OpenStack::

      $ ./stack.sh

   The installation should take about 20-30 minutes. Upon successful
   installation, the installer reports the URL of OpenStack's management
   interface. This URL is accessible from the native Ubuntu.

   .. code-block:: console

      ...

      Horizon is now available at http://<IP_address>/dashboard

      ...

      2020-04-09 01:21:37.504 | stack.sh completed in 1755 seconds.

6. Verify using the command ``systemctl status libvirtd.service`` that libvirtd is active
   and running.

7. Set up SNAT for OpenStack instances to connect to the external network.

   a. Inside the container, use the command ``ip a`` to identify the ``br-ex`` bridge
      interface. ``br-ex`` should have two IPs. One should be visible to
      the native Ubuntu's ``acrn-br0`` interface (for example, iNet 192.168.1.104/24).
      The other one is internal to OpenStack (for example, iNet 172.24.4.1/24). The
      latter corresponds to the public network in OpenStack.

   b. Set up SNAT to establish a link between ``acrn-br0`` and OpenStack.
      For example::

        $ sudo iptables -t nat -A POSTROUTING -s 172.24.4.1/24 -o br-ex -j SNAT --to-source 192.168.1.104

Configure and Create OpenStack Instance
***************************************

We'll be using the Ubuntu 20.04 (Focal) Cloud image as the OS image (qcow2
format). Download the Cloud image from https://cloud-images.ubuntu.com/releases/focal,
for example::

   $ wget https://cloud-images.ubuntu.com/releases/focal/release-20210201/ubuntu-20.04-server-cloudimg-amd64.img

Use the OpenStack management interface URL reported in a previous step
to finish setting up the network and configure and create an OpenStack
instance.

1. Begin by using your browser to log in as **admin** to the OpenStack management
   dashboard (using the URL reported previously). Use the admin
   password you set in the ``devstack/local.conf`` file:

   .. figure:: images/OpenStack-01-login.png
      :align: center
      :width: 1200px
      :name: os-01-login

   Click **Project / Network Topology** and then the **Topology** tab
   to view the existing **public** (external) and **shared** (internal) networks:

   .. figure:: images/OpenStack-02-topology.png
      :align: center
      :width: 1200px
      :name: os-02-topology

#. A **router** acts as a bridge between the internal and external
   networks. Create a router using **Project / Network / Routers /
   +Create Router**:

   .. figure:: images/OpenStack-03-create-router.png
      :align: center
      :width: 1200px
      :name: os-03-router

   Give it a name (**acrn_router**), select **public** for the external network,
   and select **Create Router**:

   .. figure:: images/OpenStack-03a-create-router.png
      :align: center
      :width: 1200px
      :name: os-03a-router

   That added the external network to the router.  Now add
   the internal network too. Click the acrn_router name:

   .. figure:: images/OpenStack-03b-created-router.png
      :align: center
      :width: 1200px
      :name: os-03b-router

   Go to the **Interfaces** tab, and click **+Add interface**:

   .. figure:: images/OpenStack-04a-add-interface.png
      :align: center
      :width: 1200px
      :name: os-04a-add-interface

   Select the subnet of the shared (private) network and click **Submit**:

   .. figure:: images/OpenStack-04b-add-interface.png
      :align: center
      :width: 1200px
      :name: os-04b-add-interface

   The router now has interfaces between the external and internal
   networks:

   .. figure:: images/OpenStack-04c-add-interface.png
      :align: center
      :width: 1200px
      :name: os-04c-add-interface

   View the router graphically by clicking the **Network Topology** tab:

   .. figure:: images/OpenStack-05-topology.png
      :align: center
      :width: 1200px
      :name: os-05-topology

   With the router set up, we've completed configuring the OpenStack
   networking.

#. Next, we'll prepare for launching an OpenStack instance.
   Click the **Admin / Compute / Image** tab and then the **+Create
   Image** button:

   .. figure:: images/OpenStack-06-create-image.png
      :align: center
      :width: 1200px
      :name: os-06-create-image

   Browse for and select the Ubuntu Cloud image file we
   downloaded earlier:

   .. figure:: images/OpenStack-06a-create-image-browse.png
      :align: center
      :width: 1200px
      :name: os-06a-create-image

   .. figure:: images/OpenStack-06b-create-image-select.png
      :align: center
      :name: os-06b-create-image

   Give the image a name (**Ubuntu20.04**), select the **QCOW2 - QEMU
   Emulator** format, and click **Create Image**:

   .. figure:: images/OpenStack-06e-create-image.png
      :align: center
      :width: 900px
      :name: os-063-create-image

   This task will take a few minutes to complete.

#. Next, click the **Admin / Compute / Flavors** tab and then the
   **+Create Flavor** button.  Define a machine flavor name
   (**UbuntuCloud**), and specify its resource requirements: the number of vCPUs (**2**), RAM size
   (**512MB**), and root disk size (**4GB**):

   .. figure:: images/OpenStack-07a-create-flavor.png
      :align: center
      :width: 700px
      :name: os-07a-create-flavor

   Click **Create Flavor** and you'll return to see a list of
   available flavors plus the new one you created (**UbuntuCloud**):

   .. figure:: images/OpenStack-07b-flavor-created.png
      :align: center
      :width: 1200px
      :name: os-07b-create-flavor

#. OpenStack security groups act as a virtual firewall controlling
   connections between instances, allowing connections such as SSH and
   HTTPS. These next steps create a security group allowing SSH and ICMP
   connections.

   Go to **Project / Network / Security Groups** and click the **+Create
   Security Group** button:

   .. figure:: images/OpenStack-08-security-group.png
      :align: center
      :width: 1200px
      :name: os-08-security-group

   Name this security group (**acrnSecuGroup**) and click **Create
   Security Group**:

   .. figure:: images/OpenStack-08a-create-security-group.png
      :align: center
      :width: 1200px
      :name: os-08a-security-group

   You'll return to a rule management screen for this new group.  Click
   the **+Add Rule** button:

   .. figure:: images/OpenStack-08b-add-rule.png
      :align: center
      :width: 1200px
      :name: os-08b-security-group

   Select **SSH** from the Rule list and click **Add**:

   .. figure:: images/OpenStack-08c-add-SSH-rule.png
      :align: center
      :width: 1200px
      :name: os-08c-security-group

   Similarly, add another rule to add an **All ICMP** rule too:

   .. figure:: images/OpenStack-08d-add-All-ICMP-rule.png
      :align: center
      :width: 1200px
      :name: os-08d-security-group

#. Create a public/private keypair used to access the created instance.
   Go to **Project / Compute / Key Pairs** and click **+Create Key
   Pair**, give the keypair a name (**acrnKeyPair**) and Key Type
   (**SSH Key**) and click **Create Key Pair**:

   .. figure:: images/OpenStack-09a-create-key-pair.png
      :align: center
      :width: 1200px
      :name: os-09a-key-pair

   Save the **private** keypair file safely,
   for future use:

   .. figure:: images/OpenStack-09c-key-pair-private-key.png
      :align: center
      :width: 1200px
      :name: os-09c-key-pair

#. Now we're ready to launch an instance.  Go to **Project / Compute /
   Instance**, click the **Launch Instance** button, give it a name
   (**UbuntuOnACRN**) and click **Next**:

   .. figure:: images/OpenStack-10a-launch-instance-name.png
      :align: center
      :width: 900px
      :name: os-10a-launch

   Select **No** for "Create New Volume", and click the up-arrow button
   for uploaded (**Ubuntu20.04**) image as the "Available source" for this
   instance:

   .. figure:: images/OpenStack-10b-no-new-vol-select-allocated.png
      :align: center
      :width: 900px
      :name: os-10b-launch

   Click **Next**, and select the machine flavor you created earlier
   (**UbuntuCloud**):

   .. figure:: images/OpenStack-10c-select-flavor.png
      :align: center
      :width: 900px
      :name: os-10c-launch

   Click **>** next to the Allocated **UbuntuCloud** flavor and see
   details about your choice:

   .. figure:: images/OpenStack-10d-flavor-selected.png
      :align: center
      :width: 900px
      :name: os-10d-launch

   Click the **Networks** tab, and select the internal **shared**
   network from the "Available" list:

   .. figure:: images/OpenStack-10e-select-network.png
      :align: center
      :width: 1200px
      :name: os-10e-launch

   Click the **Security Groups** tab and select
   the **acrnSecuGroup**  security group you created earlier. Remove the
   **default** security group if it's in the "Allocated" list:

   .. figure:: images/OpenStack-10d-only-acrn-security-group.png
      :align: center
      :width: 1200px
      :name: os-10d-security

   Click the **Key Pair** tab and verify the **acrnKeyPair** you
   created earlier is in the "Allocated" list, and click **Launch
   Instance**:

   .. figure:: images/OpenStack-10g-show-keypair-launch.png
      :align: center
      :width: 1200px
      :name: os-10g-launch

   It will take a few minutes to complete launching the instance.

#. Click the **Project / Compute / Instances** tab to monitor
   progress. When the instance status is "Active" and power state is
   "Running", associate a floating IP to the instance
   so you can access it:

   .. figure:: images/OpenStack-11-wait-for-running-create-snapshot.png
      :align: center
      :width: 1200px
      :name: os-11-running

   On the **Manage Floating IP Associations** screen, click the **+**
   to add an association:

   .. figure:: images/OpenStack-11a-manage-floating-ip.png
      :align: center
      :width: 700px
      :name: os-11a-running

   Select **public** pool, and click **Allocate IP**:

   .. figure:: images/OpenStack-11b-allocate-floating-ip.png
      :align: center
      :width: 1200px
      :name: os-11b-running

   Finally, click **Associate** after the IP address is assigned:

   .. figure:: images/OpenStack-11c-allocate-floating-ip-success-associate.png
      :align: center
      :width: 1200px
      :name: os-11c-running


Final Steps
***********

The OpenStack instance is now running and connected to the
network.  You can confirm by returning to the **Project /
Network / Network Topology** view:

.. figure:: images/OpenStack-12b-running-topology-instance.png
   :align: center
   :width: 1200px
   :name: os-12b-running

You can also see a hypervisor summary by clicking **Admin / Compute /
Hypervisors**:

.. figure:: images/OpenStack-12d-compute-hypervisor.png
   :align: center
   :width: 1200px
   :name: os-12d-running

.. note::
   OpenStack logs to the ``systemd`` journal and ``libvirt`` logs to
   ``/var/log/libvirt/libvirtd.log``.

Here are some other tasks you can try when the instance is created and
running:

* Use the hypervisor console to verify the instance is running by using
  the ``vm_list`` command.

* Ping the instance inside the container using the instance's floating IP
  address.

For more advanced CLI usage, refer to this `OpenStack cheat sheet
<https://docs.openstack.org/ocata/user-guide/cli-cheat-sheet.html>`_.