diff --git a/doc/developer-guides/hld/hld-hypervisor.rst b/doc/developer-guides/hld/hld-hypervisor.rst index edc0de8bf..0bb42008c 100644 --- a/doc/developer-guides/hld/hld-hypervisor.rst +++ b/doc/developer-guides/hld/hld-hypervisor.rst @@ -9,6 +9,6 @@ Hypervisor high-level design hv-startup hv-cpu-virt - static-core-hld Memory management + Interrupt management diff --git a/doc/developer-guides/hld/hv-cpu-virt.rst b/doc/developer-guides/hld/hv-cpu-virt.rst index 204a76ff0..d65161ced 100644 --- a/doc/developer-guides/hld/hv-cpu-virt.rst +++ b/doc/developer-guides/hld/hv-cpu-virt.rst @@ -9,6 +9,9 @@ CPU Virtualization ACRN Hypervisor CPU Virtualization Components +The following sections discuss the major modules (shown in blue) in the +CPU virtualization overview shown in :numref:`hv-cpu-virt-components`. + Based on Intel VT-x virtualization technology, ACRN emulates a virtual CPU (vCPU) with the following methods: @@ -16,7 +19,8 @@ Based on Intel VT-x virtualization technology, ACRN emulates a virtual CPU physical CPU (pCPU), making much of hardware register emulation simply pass-through and provides good isolation for physical interrupt - and guest execution. + and guest execution. (See `Static CPU Partitioning`_ for more + information.) - **simple schedule**: only two thread loops are maintained for a CPU - vCPU thread and default idle thread. A CPU runs most of the time in @@ -25,8 +29,137 @@ Based on Intel VT-x virtualization technology, ACRN emulates a virtual CPU operation needs it to stay in VMX root mode, such as when waiting for an I/O request from DM or ready to destroy. -The following sections discuss the major modules (shown in blue) in the -CPU virtualization overview shown in :numref:`hv-cpu-virt-components`. +Static CPU Partitioning +*********************** + +CPU partitioning is a policy for mapping a virtual +CPU (VCPU) to a physical CPU. The current ACRN implementation forces a +static 1:1 mapping between VCPUs and physical CPUs and does +not support multiple VCPUs running on a physical CPU and does not +support VCPU migration from one physical CPU to another. + +ACRN forces a fixed 1:1 mapping between a VCPU and a physical CPU when +creating a VCPU for the guest Operating System. This makes the VCPU +management code much simpler. + +An array is used to track the physical CPU allocation information. When +a VCPU is created, we query, allocate, and update the array to map the +VCPU to an available physical CPU. + +The physical CPU number assignment for each guest is pre-defined. For +example, on a platform with four CPU cores, one physical CPU is assigned +to run the Service Operating System (SOS) and other three physical CPUs +are assigned to run the User Operating System (UOS) instances. + +.. note:: + + To improvement SOS boot time, all physical CPUs are assigned to the SOS + during the SOS boot. Afterward, the physical CPUs defined for the UOS + are allocated by the Device Model (DM) by running the launch_uos.sh + script. + +CPU management in SOS +===================== + +With ACRN, all ACPI table entries are pass-thru to the SOS, including +the Multiple Interrupt Controller Table (MADT). The SOS sees all +physical CPUs by parsing the MADT when the SOS kernel boots. All +physical CPUs are initially assigned to the SOS by creating the same +number of virtual CPUs. + +When the SOS boot is finished, it releases the physical CPUs intended +for UOS use. + +Here is an example flow of CPU allocation on a multi-core platform. + +.. figure:: images/static-core-image2.png + :width: 600px + :align: center + :name: static-core-cpu-allocation + + CPU allocation on a multi-core platform + +CPU management in UOS +===================== + +From the UOS point of view, CPU management is very simple, using a +hypercall to create the virtual CPUs. Here's an example from from the DM +code: + +.. code-block:: c + + int vm_create_vcpu(struct vmctx *ctx, uint16_t vcpu_id) + { + struct acrn_create_vcpu cv; + int error; + + bzero(&cv, sizeof(struct acrn_create_vcpu)); + cv.vcpu_id = vcpu_id; + error = ioctl(ctx->fd, IC_CREATE_VCPU, &cv); + return error; + } + +The VHM will respond to the ioctl: + +.. code-block:: c + + case IC_CREATE_VCPU: { + struct acrn_create_vcpu cv; + + if (copy_from_user(&cv, (void *)ioctl_param, + sizeof(struct acrn_create_vcpu))) + return -EFAULT; + + ret = acrn_hypercall2(HC_CREATE_VCPU, vm->vmid, + virt_to_phys(&cv)); + if (ret < 0) { + pr_err("vhm: failed to create vcpu %d!\\n", + cv.vcpu_id); + return -EFAULT; + } + + atomic_inc(&vm->vcpu_num); + return ret; + } + +The hypercall ``HC_CREATE_VCPU`` is handled in the hypervisor with +the parameter: + +.. code-block:: c + + struct acrn_create_vcpu { + /** the virtual CPU ID for the VCPU created */ + uint16_t vcpu_id; + /** the physical CPU ID for the VCPU created */ + uint16_t pcpu_id; + } __attribute__((aligned(8))); + +CPU assignment management in HV +=============================== + +When we create a VCPU in the hypervisor, an available physical CPU is +picked and marked as used. When we destroy the VCPU, we mark the +physical CPU as available again. + +.. figure:: images/static-core-image1.png + :width: 600px + :align: center + :name: static-core-cpu-assign + + HV CPU Assignment Management + +#. ``allocate_pcpu()`` queries the physical CPU allocation info to get an + available physical CPU and marks physical CPU as not available +#. Physical CPU info is passed to ``create_vcpu()`` and a mapping is built + between the physical CPU and virtual CPU +#. When the VCPU is destroyed VCPU, the physical CPU is passed to the + ``free_pcpu()`` function +#. ``free_pcpu()`` marks the physical CPU available again. + +Currently, the ACRN hypervisor does not support virtual CPU migration to +different physical CPUs. This means no changes to the virtual CPU to +physical CPU can happen without first calling destroy_vcpu. + .. _vCPU_lifecycle: diff --git a/doc/developer-guides/hld/static-core-hld.rst b/doc/developer-guides/hld/static-core-hld.rst deleted file mode 100644 index 1a4b19cec..000000000 --- a/doc/developer-guides/hld/static-core-hld.rst +++ /dev/null @@ -1,138 +0,0 @@ -.. cpu-partitioning-hld: - -Static CPU Partitioning -####################### - -CPU partitioning is a policy for mapping a virtual -CPU (VCPU) to a physical CPU. The current ACRN implementation forces a -static 1:1 mapping between VCPUs and physical CPUs and does -not support multiple VCPUs running on a physical CPU and does not -support VCPU migration from one physical CPU to another. - -This section describes the static CPU partitioning implementation in -ACRN. - -Overview -******** - -ACRN forces a fixed 1:1 mapping between a VCPU and a physical CPU when -creating a VCPU for the guest Operating System. This makes the VCPU -management code much simpler. - -An array is used to track the physical CPU allocation information. When -a VCPU is created, we query, allocate, and update the array to map the -VCPU to an available physical CPU. - -The physical CPU number assignment for each guest is pre-defined. For -example, on a platform with four CPU cores, one physical CPU is assigned -to run the Service Operating System (SOS) and other three physical CPUs -are assigned to run the User Operating System (UOS) instances. - -.. note:: - - To improvement SOS boot time, all physical CPUs are assigned to the SOS - during the SOS boot. Afterward, the physical CPUs defined for the UOS - are allocated by the Device Model (DM) by running the launch_uos.sh - script. - -CPU management in SOS -********************* - -With ACRN, all ACPI table entries are pass-thru to the SOS, including -the Multiple Interrupt Controller Table (MADT). The SOS sees all -physical CPUs by parsing the MADT when the SOS kernel boots. All -physical CPUs are initially assigned to the SOS by creating the same -number of virtual CPUs. - -When the SOS boot is finished, it releases the physical CPUs intended -for UOS use. - -Here is an example flow of CPU allocation on a multi-core platform. - -.. figure:: images/static-core-image2.png - :width: 600px - :align: center - :name: static-core-cpu-allocation - - CPU allocation on a multi-core platform - -CPU management in UOS -********************* - -From the UOS point of view, CPU management is very simple, using a -hypercall to create the virtual CPUs. Here's an example from from the DM -code: - -.. code-block:: c - - int vm_create_vcpu(struct vmctx *ctx, uint16_t vcpu_id) - { - struct acrn_create_vcpu cv; - int error; - - bzero(&cv, sizeof(struct acrn_create_vcpu)); - cv.vcpu_id = vcpu_id; - error = ioctl(ctx->fd, IC_CREATE_VCPU, &cv); - return error; - } - -The VHM will respond to the ioctl: - -.. code-block:: c - - case IC_CREATE_VCPU: { - struct acrn_create_vcpu cv; - - if (copy_from_user(&cv, (void *)ioctl_param, - sizeof(struct acrn_create_vcpu))) - return -EFAULT; - - ret = acrn_hypercall2(HC_CREATE_VCPU, vm->vmid, - virt_to_phys(&cv)); - if (ret < 0) { - pr_err("vhm: failed to create vcpu %d!\\n", - cv.vcpu_id); - return -EFAULT; - } - - atomic_inc(&vm->vcpu_num); - return ret; - } - -The hypercall ``HC_CREATE_VCPU`` is handled in the hypervisor with -the parameter: - -.. code-block:: c - - struct acrn_create_vcpu { - /** the virtual CPU ID for the VCPU created */ - uint16_t vcpu_id; - /** the physical CPU ID for the VCPU created */ - uint16_t pcpu_id; - } __attribute__((aligned(8))); - -CPU assignment management in HV -******************************* - -When we create a VCPU in the hypervisor, an available physical CPU is -picked and marked as used. When we destroy the VCPU, we mark the -physical CPU as available again. - -.. figure:: images/static-core-image1.png - :width: 600px - :align: center - :name: static-core-cpu-assign - - HV CPU Assignment Management - -#. ``allocate_pcpu()`` queries the physical CPU allocation info to get an - available physical CPU and marks physical CPU as not available -#. Physical CPU info is passed to ``create_vcpu()`` and a mapping is built - between the physical CPU and virtual CPU -#. When the VCPU is destroyed VCPU, the physical CPU is passed to the - ``free_pcpu()`` function -#. ``free_pcpu()`` marks the physical CPU available again. - -Currently, the ACRN hypervisor does not support virtual CPU migration to -different physical CPUs. This means no changes to the virtual CPU to -physical CPU can happen without first calling destroy_vcpu.