hv: support halt in hv idle

When all vCPU threads on one pCPU are put to sleep (e.g., when all
guests execute HLT), hv would schedule to idle thread. Currently the
idle thread executes PAUSE which does not enter any c-state and consumes
a lot of power. This patch is to support HLT in the idle thread.

When we switch to HLT, we have to make sure events that would wake a
vCPU must also be able to wake the pCPU. Those events are either
generated by local interrupt or issued by other pCPUs followed by an
ipi kick.

Each of them have an interrupt involved, so they are also able to wake
the halted pCPU. Except when the pCPU has just scheduled to idle thread
but not yet halted, interrupts could be missed.

sleep-------schedule to idle------IRQ ON---HLT--(kick missed)
                                         ^
                              wake---kick|

This areas should be protected. This is done by a safe halt
mechanism leveraging STI instruction’s delay effect (same as Linux).

vCPUs with lapic_pt or hv with CONFIG_KEEP_IRQ_DISABLED=y does not allow
interrupts in root mode, so they could never wake from HLT (INIT kick
does not wake HLT in root mode either). They should continue using PAUSE
in idle.

Tracked-On: #8507
Signed-off-by: Wu Zhou <wu.zhou@intel.com>
Reviewed-by: Junjie Mao <junjie.mao@intel.com>

hypervisor/include/arch/x86/asm/cpu.h

@@ -533,6 +533,15 @@ static inline void asm_hlt(void)
 	asm volatile ("hlt");
 }
 
+/* interrupts remain inhibited on the instruction boundary following
+ * an execution of STI. This will make sure pending interrupts will
+ * wake hlt.
+ */
+static inline void asm_safe_hlt(void)
+{
+	asm volatile ("sti; hlt; cli" : : : "cc");
+}
+
 /* Disables interrupts on the current CPU */
 #ifdef CONFIG_KEEP_IRQ_DISABLED
 #define CPU_IRQ_DISABLE_ON_CONFIG()		do { } while (0)

hypervisor/include/arch/x86/asm/per_cpu.h

@@ -55,6 +55,7 @@ struct per_cpu_region {
 	uint32_t lapic_ldr;
 	uint32_t softirq_servicing;
 	uint32_t mode_to_kick_pcpu;
+	uint32_t mode_to_idle;
 	struct smp_call_info_data smp_call_info;
 	struct list_head softirq_dev_entry_list;
 #ifdef PROFILING_ON

hypervisor/include/common/schedule.h

@@ -15,6 +15,9 @@
 #define DEL_MODE_INIT		(1U)
 #define DEL_MODE_IPI		(2U)
 
+#define IDLE_MODE_PAUSE		(1U)
+#define IDLE_MODE_HLT		(2U)
+
 #define THREAD_DATA_SIZE	(256U)
 
 enum thread_object_state {