Commit Graph

5330 Commits

Author SHA1 Message Date
Conghui Chen
2abbb99f6a hv: make thread status more accurate
1. Update thread status after switch_in/switch_out.
2. Add 'be_blocking' to represent the intermediate state during
sleep_thread and switch_out. After switch_out, the thread status
update to THREAD_STS_BLOCKED.

Tracked-On: #5057
Signed-off-by: Conghui Chen <conghui.chen@intel.com>
Reviewed-by: Shuo A Liu <shuo.a.liu@intel.com>
Acked-by: Eddie Dong <eddie.dong@intel.com>
2020-07-22 13:38:41 +08:00
Long Liu
312ea0b374 DM: xHCI: Release all device resource when received HCRST.
There has one bug that the pci_xhci_dev_emu objects recorded in
xdev->devices haven't destroy when OVMF transfer to guest OS.
This will cause the the fds of usbfs can't be freed in libusb.
The OVMF xHCI driver needs to disable all enabled slots during
graceful existing flow, but it doesn't.
The USBCMD.HCRST bit is used for xHC reset that will be triggered
in xHCI driver initialization. We can use this chance to clear
all pci_xhci_dev_emu objects

Tracked-On: #4897
Signed-off-by: Long Liu <long.liu@intel.com>
Acked-by: Yu Wang <yu1.wang@intel.com>
2020-07-22 12:54:37 +08:00
acrnsi-robot
28c9e35771
Merge pull request #5054 from mingqiangchi/add_vm_vcpu_state_lock
hv:add per-vm lock for vm & vcpu state change
2020-07-21 14:51:01 +08:00
zhanqi
288efd31b4 create acrn hypervisor/kernel deb packaging
Tracked-On: #5022

Signed-off-by: zhanqi <sherry.qi.zhang@intel.com>
2020-07-20 11:44:07 +08:00
Mingqiang Chi
aa89eb3541 hv:add per-vm lock for vm & vcpu state change
-- replace global hypercall lock with per-vm lock
-- add spinlock protection for vm & vcpu state change

v1-->v2:
   change get_vm_lock/put_vm_lock parameter from vm_id to vm
   move lock obtain before vm state check
   move all lock from vmcall.c to hypercall.c

Tracked-On: #4958
Signed-off-by: Mingqiang Chi <mingqiang.chi@intel.com
Reviewed-by: Jason Chen CJ <jason.cj.chen@intel.com>
Acked-by: Eddie Dong <eddie.dong@intel.com>
2020-07-20 11:22:17 +08:00
Wei Liu
e5c5337886 acrn-config: generate '-s 1:0,lpc ' for non-hart rt in launch script
Generate '-s 1:0,lpc ' for none Hart RT in launch script.

Tracked-On: #5049
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
2020-07-20 09:52:57 +08:00
yuhong.tao@intel.com
6992d00e45 HV: ptdev hide SRIOV capability for VM
Hide sriov capability of passthrough devices for VMs at init_vdev_pt().
And for post-launched VM, allow assign PF.

Tracked-On: #5041
Signed-off-by: Tao Yuhong <yuhong.tao@intel.com>
Acked-by: Eddie Dong <eddie.dong@intel.com>
2020-07-16 17:27:18 +08:00
yuhong.tao@intel.com
eb36337622 HV: vdev passthough hidding SRIOV
Support hide SRIOV extend capability for passthough device

Tracked-On: #5041
Signed-off-by: Tao Yuhong <yuhong.tao@intel.com>
Acked-by: Eddie Dong <eddie.dong@intel.com>
Reviewed-by: Fei Li <fei1.li@intel.com>
2020-07-16 17:27:18 +08:00
Shuang Zheng
585c652f81 acrn-config: update board xml for ehl-crb-b
add tsn devices and the NVME device in the board xml

Tracked-On: #4831
Signed-off-by: Shuang Zheng <shuang.zheng@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@intel.com>
2020-07-16 14:50:04 +08:00
Sun Peng
e677f9413b dm: Increase OpRegion size for bigger VBT
OpRegion: 8KB(0x2000)
[ OpRegion Header      ] Offset: 0x0
[ Mailbox #1: ACPI     ] Offset: 0x100
[ Mailbox #2: SWSCI    ] Offset: 0x200
[ Mailbox #3: ASLE     ] Offset: 0x300
[ Mailbox #4: VBT      ] Offset: 0x400
[ Mailbox #5: ASLE EXT ] Offset: 0x1C00
Extended OpRegion: 8KB(0x2000)
[ Raw VBT              ] Offset: 0x0

Generally VBT stores in MailBox4 in OpRegion which max size is 6KB.
If VBT larger than 6KB, it will be stored in extended OpRegion which
is neighborhood with legacy OpRegion. In this case, we need to
passthrough extended OpRegion also to support GVT-d feature. The
OpRegion size that we passthrough should be (OpRegion+Extended)=16KB
ASLE.rvda stores the location of VBT.
For OpRegion 2.1+: ASLE.rvda = offset to OpRegion base address
For OpRegion 2.0:  ASLE.rvda = physical address

To-do: Add support for OpRegion on some platforms(eg. APL)

Tracked-On: #5029
Signed-off-by: Sun Peng <peng.p.sun@intel.com>
2020-07-15 16:13:03 +08:00
Wei Liu
76ec5f69f8 acrn-config: minor fix mac seed for launch config
Minor fix for mac seed when generating launch script.

Tracked-On: #5039
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
2020-07-15 16:06:32 +08:00
Peter Fang
f4860859ef OVMF release v2.1
- Enable AcrnLapicTimerDxe
- Enable AcrnTscTimerLib
- Lapic Timer Driver
- Change LVT timer register structure of Lapic
- Add AcrnTscTimerLib

Tracked-On: #5036
Signed-off-by: Peter Fang <peter.fang@intel.com>
2020-07-15 15:41:11 +08:00
Long Liu
f9c44369a0 DM USB: xHCI: Fix XHCI_GET_SLOT value check issue
Fix XHCI_GET_SLOT macro check slot valid function, when the
slot value is bigger than XHCI_MAX_SLOT set the slot value
to zero.

Tracked-On: #4711

Signed-off-by: Long Liu <long.liu@intel.com>
Acked-by: Yu Wang <yu1.wang@intel.com>
2020-07-15 10:34:53 +08:00
Vijay Dhanraj
3c9469d98a DM: Add macvtap support to virtio-net
This patch does the following,
1. Fix an explicit interface name check for tapX preventing
any other interface name like kata_tap to be setup.
2. Add support for macvtap interface.
3. Identify macvtap vs tap interface and
if it is macvtap, identify character device (/dev/tapXX)
to be used.

Tracked-On: #4945
Signed-off-by: Vijay Dhanraj <vijay.dhanraj@intel.com>
Acked-by: Yu Wang <yu1.wang@intel.com>
2020-07-14 13:02:21 +08:00
Conghui Chen
fcc9efec8e acrn-config: enable only 4 vms for TGL
Enable only 4 vms for TGL.

Tracked-On: #5013
Signed-off-by: Conghui Chen <conghui.chen@intel.com>
2020-07-14 12:43:34 +08:00
David B. Kinder
c455085f6d doc: update intro link to project roadmap
Instead of linking to a specific doc on projectacrn.org, update to just
link to the area on projectacrn.org/#resources where the roadmap doc can
be found.  Also remove mention of 2020 to keep it generic so it won't
need updating in 2021.

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
2020-07-13 15:40:34 -07:00
David B. Kinder
2fd1a79ef0 doc: keep doxygen group descriptions in doc output
Propagated fix from other doxygen/breathe project to keep the
description found in the doxygen comments for the group being displayed
(in case there actually is a nice description given).

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
2020-07-10 13:50:58 -07:00
Mostafa Naeem
e43b6c7854 doc: Update Inter-VM Communication with Security Hardening Guidelines
ACRN 2.0 introduced Inter-VM communication feature by enabling the ivshmem v1.0 protocol/channel to communication
between VMs. To support the community's application Security Development Lifecycle (SDL), we provide a security hardening
guideline with some pointers to consider when using this channel by userspace application in case of additional security
requirments for Confidentiality, Integrity, or Authenticity.

Signed-off-by: Mostafa Naeem <mostafa.elsaid@intel.com>
2020-07-10 13:50:24 -07:00
Yin Fengwei
fcec5a94be kconfig: extend the max msix table number to 64
There are some devices (like Samsung NVMe SSD SM981/PM981 which has 33 MSIX tables)
which have more than 16 MSIX tables. Extend the default value to 64 to handle them.

Tracked-On: #4994
Signed-off-by: Yin Fengwei <fengwei.yin@intel.com>
2020-07-10 19:39:11 +08:00
Wei Liu
e413e23bdd acrn-config: extend the max msix table number to 64
Config tool should keep aligning with Kconfig default value for
MAX_MSIX_TABLE_NUM.

Note: Remain the same configuration for the board which does not have
PCIe slot or NVME slot.

Tracked-On: #4994
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
2020-07-10 19:39:11 +08:00
Wei Liu
70d98da042 acrn-config: add max MSI-X table number for board xmls
1.add max MSI-X table number in board xmls.
2.leave MAX_MSIX_TABLE_NUM item to blank.

Tracked-On: #4994
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Acked-by: Victor Sun <victor.sun@intel.com>
2020-07-10 19:39:11 +08:00
Wei Liu
1cb68b2cda acrn-config: detect and parse MSI-X table number
Detect and get MSI-X table number in board xmls.
Parse and generate the number for board config while 'MAX_MSIX_TABLE_NUM'
item is blank.

Tracked-On: #4994
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Acked-by: Victor Sun <victor.sun@intel.com>
2020-07-10 19:39:11 +08:00
Li Fei1
80c7da8f1c hv: vioapic: expose ioapic to guest unconditionally
Some OSes assume the platform must have the IOAPIC. For example:
Linux Kernel allocates IRQ force from GSI (0 if there's no PIC and IOAPIC) on x86.
And it thinks IRQ 0 is an architecture special IRQ, not for device driver. As a
result, the device driver may goes wrong if the allocated IRQ is 0 for RTVM.

This patch expose vIOAPIC to RTVM with LAPIC passthru even though the RTVM can't
use IOAPIC, it servers as a place holder to fullfil the guest assumption.

After vIOAPIC has exposed to guest unconditionally, the 'ready' field could be
removed since we do vIOAPIC initialization for each guest.

Tracked-On: #4691
Signed-off-by: Li Fei1 <fei1.li@intel.com>
Acked-by: Eddie Dong <eddie.dong@intel.com>
2020-07-10 19:33:46 +08:00
Wei Liu
f2479f6489 acrn-config: update passtrough device config for ehl-crb-b launch xmls
1.Update passtrough device config for ehl-crb-b launch xmls.

Tracked-On: #5016
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Acked-by: Victor Sun <victor.sun@intel.com>
2020-07-10 18:58:55 +08:00
Wei Liu
83d64c506f acrn-config: support 6 VMs for ehl-crb-b industry xml
Add support 6 VMs for ehl-crb-b industry xml.

Tracked-On: #5015
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Acked-by: Victor Sun <victor.sun@intel.com>
2020-07-10 18:58:55 +08:00
David B. Kinder
aa45937860 doc: Update contribution guide instructions
While changes to documentation can be submitted directly as PRs, changes
to code must be first submitted for approval to the developer mailing
list.  Update the contribution guidelines to talk about this.

Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
2020-07-08 10:52:57 -07:00
Shuo A Liu
bdbf135708 dm: Fix wrong hugetlb_lv_max
hugetlb_lv_max will get wrong value if the HUGETLB_LV2 mount failed.
Once hugetlb_lv_max is wrong, the following code logic messes up.

Tracked-On: #4937
Signed-off-by: Shuo A Liu <shuo.a.liu@intel.com>
Acked-by: Yu Wang <yu1.wang@intel.com>
2020-07-08 15:32:15 +08:00
Sun Peng
6edd21c1dc dm: Data Stolen Memory (DSM) passthrough support for GVT-d on TGL
The way of passing DSM address on TGL is the same with on EHL.
Adding these code to support GVT-d on TGL.

Tracked-On: #5020
Signed-off-by: Sun Peng <peng.p.sun@intel.com>
Acked-by: Yu Wang yu1.wang@intel.com
2020-07-08 13:19:05 +08:00
Conghui Chen
6722132233 acrn-config: enable more VMs in TGL xml
Add more VMs in xml.
Enable vuart0 for VM1.

Tracked-On: #5013
Signed-off-by: Conghui Chen <conghui.chen@intel.com>
2020-07-08 10:16:34 +08:00
Conghui Chen
213cd4e2b2 acrn-config: enlarge ram size
enlarge ram size, otherwise, there would be compile issue for tgl.

Tracked-On: #5013
Signed-off-by: Conghui Chen <conghui.chen@intel.com>
2020-07-08 10:16:34 +08:00
Shuo A Liu
0b03a2a75a acrn-config: update EHL CRB configs
BIOS version: EHLSFWI1.R00.2224.A00.2005281500

Tracked-On: #4937
Signed-off-by: Shuo A Liu <shuo.a.liu@intel.com>
2020-07-06 13:48:12 +08:00
Shuang Zheng
2c6fad00ee acrn-config: add MBA delay support in acrn config app
MBA_DELAY/CLOS_MASK show be exposed only if "MBA"/"L2" or "L3" existed
in rdt resource supoorted in board xml;
The default value of MBA_DELAY is 0;
The numbers of MAB_DELAY/CLOS_MASK entries is determined by:
If CDP is not enabled, the number of entries for CLOS_MASK and MBA_DELAY
is the min of CLOS_MAX of all RDT resources;
If CDP is enabled,  divide the CLOS_MAX values for L3 and L2 resources
by 2 and then find the min of all RDT resources to get common_clos_max,
the number of entries for CLOS_MASK is common_clos_max*2,
the number of entries for MBA_DELAY is comm_clos_max.

Tracked-On: #4943
Signed-off-by: Shuang Zheng <shuang.zheng@intel.com>
Reviewed-by: Vijay Dhanraj <vijay.dhanraj@intel.com>
2020-07-06 13:48:12 +08:00
Wei Liu
6e2f8e2a03 acrn-config: refine sanity check for RDT/MBA
Refine sanity check for RDT CLOS and MBA Delay.

Tracked-On: #4943
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Reviewed-by: Vijay Dhanraj <vijay.dhanraj@intel.com>
2020-07-06 13:48:12 +08:00
Wei Liu
30750fa7d5 acrn-config: update the LICENSE year in config tool
Update the LICENSE year for hv files which generate by config tool.

Tracked-On: #5004
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Acked-by: Victor Sun <victor.sun@intel.com>
2020-07-06 13:43:15 +08:00
Wei Liu
f716d8a2ad acrn-config: remove unnecessary check for pci.ids
The pci.ids database should be already prepared while tools of 'lspci'
were correctly installed and this check for pci.ids should be removed.

Tracked-On: #4989
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Acked-by: Victor Sun <victor.sun@intel.com>
2020-07-06 13:43:15 +08:00
Wei Liu
76745ccc18 acrn-config: improvement sanity check for vuart1 target id settings
For the base of vuart 1 is not an invalid com base, the tools will check
the target vuart id and it's VM id if matches the other VM's. If they do
not match the error message will report to re-configuration.

Tracked-On: #4991
Signed-off-by: Wei Liu <weix.w.liu@intel.com>
Acked-by: Victor Sun <victor.sun@intel.com>
2020-07-06 13:43:15 +08:00
David B. Kinder
854d8fa46e doc: tweak link in waag tutorial
Signed-off-by: David B. Kinder <david.b.kinder@intel.com>
2020-07-04 14:35:56 -07:00
fangfang.shen
880be9f7ef doc:update WaaG GSG based on Ubuntu.
Signed-off-by: fangfang.shen <fangfang.shen@intel.com>
2020-07-04 14:05:10 -07:00
Mingqiang Chi
b1357cdc0d hv:use spinlock_irqsave_obtain api for uart
replace spinlock_obtain/spinlock_release with spinlock_irqsave_obtain
and spinlock_irqrestore_release to avoid dead lock for uart module.

this uart lock may be accessed in ISR context like this path:
   dispatch_interrupt->pr_err/pr_xxx or printf
   ->console_write->uart16550_puts

Tracked-On: #4958

Signed-off-by: Mingqiang Chi <mingqiang.chi@intel.com>
2020-07-03 17:41:17 +08:00
Minggui Cao
0080c6ca72 tools: improve Makefile to build life-mngr
1. add life-mngr as a target in misc/Makefile, so it is
convenient to build and used in Yocto system.
2. add install target in life-mngr/Makefile to be packaged
into device file-system.

Tracked-On: #4870
Signed-off-by: Minggui Cao <minggui.cao@intel.com>
Reviewed-by: Binbin Wu <binbin.wu@intel.com>
2020-07-02 13:51:34 +08:00
Li Fei1
41805eb2e8 hv: vpci: minor refine about MSI/MSI-X de-initialization
About the MSI/MSI-X Capability, there're some fields of it would never been changed
once they had been initialized. So it's no need to reset them once the vdev instance
is still used. What need to reset are the fields which would been changed by guest
at runtime.

Tracked-On: #4550
Signed-off-by: Li Fei1 <fei1.li@intel.com>
2020-07-02 13:03:36 +08:00
Mingqiang Chi
3b120807c9 hv:rename vioapic.mtx to vioapic.lock
rename vioapic.mtx to vioapic.lock

Tracked-On: #4958
Signed-off-by: Mingqiang Chi <mingqiang.chi@intel.com>
2020-07-02 09:40:29 +08:00
Mingqiang Chi
7751c7933d hv:unify spin_lock initialization
will follow this convention for spin lock initialization:
-- for simple global variable locks, use this style:
   static spinlock_t xxx_spinlock = {.head = 0U, .tail = 0U,}
-- for the locks inside a data structure, need to call
   spinlock_init to initialize.

Tracked-On: #4958
Signed-off-by: Mingqiang Chi <mingqiang.chi@intel.com>
2020-07-02 09:40:29 +08:00
Mingqiang Chi
7b32fce06f hv:use spinlock_irqsave_obtain api for vpic
replace spinlock_obtain/spinlock_release with spinlock_irqsave_obtain
and spinlock_irqrestore_release to avoid dead lock for vpic module.

this vpic lock may be accessed in ISR context like this path:
  dispatch_interrupt->do_softirq->softirq_handlers
  ->ptirq_softirq->ptirq_handle_intx->vpic_set_irqline

Tracked-On: #4958
Signed-off-by: Mingqiang Chi <mingqiang.chi@intel.com>
2020-07-02 09:40:29 +08:00
Qian Wang
aee4515ff0 HV: restrict conditions to assign/deassign pcidev
hv: hypercall: restrict the condition to assign/deassign a pci device to
a post-launched VM for safety

For the safety of post-launched VMs, pci devices assignments should
occur only when VM is being created (at VM_CREATED STATUS), and pci
devices de-assignment should occur only when VM is being created or
shutdown/reset (at VM_CREATED or VM_PAUSED status)

Tracked-On: #4995
Acked-by: Eddie Done <eddie.dong@intel.com>
Reviewed-by: Li Fei <Fei1.Li@intel.com>
Signed-off-by: Wang Qian <qian1.wang@intel.com>
2020-07-01 16:19:05 +08:00
Shuo A Liu
2276f1c43d hv: Change to a permissive check with broken DMAR table
From the VT-d spec 8.3:
If a DRHD structure with INCLUDE_PCI_ALL flag Set is reported for a
Segment, it must be enumerated by BIOS after all other DRHD structures
for the same Segment.

However, some broken BIOS violate the rules. To bring up ACRN with them,
change the ASSERT to a permissive check to unblock the BIOS limitation.
Also, scan the DRHD list to find the one who has INCLUDE_PCI_ALL flag.

Tracked-On: #4937
Signed-off-by: Shuo A Liu <shuo.a.liu@intel.com>
2020-06-30 13:59:38 +08:00
Shuo A Liu
03e18ec492 hv: Refine dmar parsing code
Replace dmar_iterate_tbl() by a direct for loop. Handle the
dmar_unit_cnt and handle_one_drhd() of each DRHD in the direct for loop.

Also tune some function definitions to save LOC.

Tracked-On: #4937
Signed-off-by: Shuo A Liu <shuo.a.liu@intel.com>
2020-06-30 13:59:38 +08:00
wenlingz
eb14ebd967 version:2.1-unstable
Signed-off-by: wenlingz <wenling.zhang@intel.com>
2020-06-28 11:13:14 +08:00
Shuo A Liu
025df6d44c hv: use SELF IPI Register for self IPI in X2APIC mode
According to SDM 10.12.11, we can know this register is dedicated to the
purpose of sending self-IPIs with the intent of enabling a highly
optimized path for sending self-IPIs. Also sending the IPI via the Self
Interrupt Register ensures that interrupt is delivered to the processor
core. Specifically completion of the WRMSR instruction to the SELF IPI
register implies that the interrupt has been logged into the IRR.

Tracked-On: #4937
Signed-off-by: Shuo A Liu <shuo.a.liu@intel.com>
Acked-by: Eddie Dong <eddie.dong@intel.com>
2020-06-28 10:33:22 +08:00
Shuo A Liu
0397cb7174 hv: Fix the interrupts lost issue with PI support
Currently, not all platforms support posted interrupt processing of both
VT-x and VT-d. On EHL, VT-d doesn't support posted interrupt processing.
So in such scenario, is_pi_capable() in vcpu_handle_pi_notification()
will bypass the PIR pending bits check which might cause a self-NV-IPI
lost.

With commit "bf1ff8c98 (hv: Offload syncing PIR to vIRR to processor
hardware)", the syncing PIR to vIRR is postponed and it is handled by a
self-NV-IPI in the following VMEnter. The process looks like,
a) vcpu A accepts a virtual interrupt ->
   1) ACRN_REQUEST_EVENT is set
   2) corresponding bit in PIR is set
   3) Posted Interrupt ON bit is set
b) vcpu A does virtual interrupt injection on resume path due to
   the pending ACRN_REQUEST_EVENT ->
   1) hypervisor disables host interrupt
   2) ACRN_REQUEST_EVENT is cleared
   3) a self-NV-IPI is sent via ICR of LAPIC.
   4) IRR bit of the self-NV-IPI is set
c) (VM-ENTRY) vcpu A returns into non-root mode
   1) host interrupt enable(by HW)
   2) posted interrupt processing clears the ON bit, sync PIR to vIRR
   3) deliver the virtual interrupt if guest rflags.IF=1
d) (VM-EXIT) vcpu A traps due to a instruction execution (e.g. HLT)
   1) host interrupt disable(by HW)
   2) hypervisor enable host interrupt

Above illustrates a normal process of the virtual interrupt injection
with cpu PI support. However, a failing case is observed. The failing
case is that the self-NV-IPI from b-3 is not accepted by the core until
a timing between d-1 and d-2. b-4 happening between d-1 and d-2 is
observed by debug trace. So the self-NV-IPI will be handled in root-mode
which cannot do the syncing PIR to vIRR processing. Due to the bug
described in the first paragraph, vcpu_handle_pi_notification() cannot
succeed the virtual interrupt injection request. This patch fix it by
removing the wrong check in vcpu_handle_pi_notification() because
vcpu_handle_pi_notification() only happens on platform with cpu PI
support.

Here are some cost data for sending IPI via LAPIC ICR regsiter.
Normally, the cycles between ICR write and IRR got set is 140~260,
which is not accurate due to the MSR read overhead.
And from b-3 to c is about 560 cycles. So b-4 happens during this
period. But in bad case, b-4 doesn't happen even c is triggered.
The worse case i captured is that ICR write and IRR got set costs more
than 1900 cycles. Now, the best GUESS of the huge cost of IPI via ICR is
the ACPI bus arbitration(refer to SDM 10.6.3, 10.7 and Figure 10-17).

Tracked-On: #4937
Signed-off-by: Shuo A Liu <shuo.a.liu@intel.com>
Acked-by: Eddie Dong <eddie.dong@intel.com>
2020-06-28 10:33:22 +08:00