github/kata-containers
1
0
Fork 1
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-08-18 07:58:36 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #6182 from GeorginaKin/CCv0

Browse Source 
CCv0: Merge main into CCv0 branch
This commit is contained in:
Georgina Kinge 2023-02-07 15:53:41 +00:00 committed by GitHub
commit b95440712d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
26 changed files with 735 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.github/workflows/kata-deploy-push.yaml vendored
View File

@ -18,6 +18,7 @@ jobs:
matrix: matrix:
asset: asset:
- kernel - kernel
- kernel-dragonball-experimental
- shim-v2 - shim-v2
- qemu - qemu
- cloud-hypervisor - cloud-hypervisor

1
.github/workflows/kata-deploy-test.yaml vendored
View File

@ -50,6 +50,7 @@ jobs:
- cloud-hypervisor - cloud-hypervisor
- firecracker - firecracker
- kernel - kernel
- kernel-dragonball-experimental
- nydus - nydus
- qemu - qemu
- rootfs-image - rootfs-image

1
.github/workflows/release.yaml vendored
View File

@ -13,6 +13,7 @@ jobs:
- cloud-hypervisor - cloud-hypervisor
- firecracker - firecracker
- kernel - kernel
- kernel-dragonball-experimental
- nydus - nydus
- qemu - qemu
- rootfs-image - rootfs-image

29
docs/Developer-Guide.md
View File

@ -232,10 +232,6 @@ $ rustup target add "${ARCH}-unknown-linux-${LIBC}"
To build the agent: To build the agent:
```bash
$ make -C kata-containers/src/agent
```
The agent is built with seccomp capability by default. The agent is built with seccomp capability by default.
If you want to build the agent without the seccomp capability, you need to run `make` with `SECCOMP=no` as follows. If you want to build the agent without the seccomp capability, you need to run `make` with `SECCOMP=no` as follows.
@ -243,6 +239,31 @@ If you want to build the agent without the seccomp capability, you need to run `
$ make -C kata-containers/src/agent SECCOMP=no $ make -C kata-containers/src/agent SECCOMP=no
``` ```
For building the agent with seccomp support using `musl`, set the environment
variables for the [`libseccomp` crate](https://github.com/libseccomp-rs/libseccomp-rs).
```bash
$ export LIBSECCOMP_LINK_TYPE=static
$ export LIBSECCOMP_LIB_PATH="the path of the directory containing libseccomp.a"
$ make -C kata-containers/src/agent
```
If the compilation fails when the agent tries to link the `libseccomp` library statically
against `musl`, you will need to build `libseccomp` manually with `-U_FORTIFY_SOURCE`.
You can use [our script](https://github.com/kata-containers/kata-containers/blob/main/ci/install_libseccomp.sh)
to install `libseccomp` for the agent.
```bash
$ mkdir -p ${seccomp_install_path} ${gperf_install_path}
$ kata-containers/ci/install_libseccomp.sh ${seccomp_install_path} ${gperf_install_path}
$ export LIBSECCOMP_LIB_PATH="${seccomp_install_path}/lib"
```
On `ppc64le` and `s390x`, `glibc` is used. You will need to install the `libseccomp` library
provided by your distribution.
> e.g. `libseccomp-dev` for Ubuntu, or `libseccomp-devel` for CentOS
> **Note:** > **Note:**
> >
> - If you enable seccomp in the main configuration file but build the agent without seccomp capability, > - If you enable seccomp in the main configuration file but build the agent without seccomp capability,

2
docs/use-cases/using-Intel-QAT-and-kata.md
View File

@ -49,7 +49,7 @@ the latest driver.
$ export QAT_DRIVER_VER=qat1.7.l.4.14.0-00031.tar.gz $ export QAT_DRIVER_VER=qat1.7.l.4.14.0-00031.tar.gz
$ export QAT_DRIVER_URL=https://downloadmirror.intel.com/30178/eng/${QAT_DRIVER_VER} $ export QAT_DRIVER_URL=https://downloadmirror.intel.com/30178/eng/${QAT_DRIVER_VER}
$ export QAT_CONF_LOCATION=~/QAT_conf $ export QAT_CONF_LOCATION=~/QAT_conf
$ export QAT_DOCKERFILE=https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/master/demo/openssl-qat-engine/Dockerfile $ export QAT_DOCKERFILE=https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/main/demo/openssl-qat-engine/Dockerfile
$ export QAT_SRC=~/src/QAT $ export QAT_SRC=~/src/QAT
$ export GOPATH=~/src/go $ export GOPATH=~/src/go
$ export KATA_KERNEL_LOCATION=~/kata $ export KATA_KERNEL_LOCATION=~/kata

1
src/dragonball/README.md
View File

@ -19,6 +19,7 @@ and configuration process.
Device: [Device Document](docs/device.md) Device: [Device Document](docs/device.md)
vCPU: [vCPU Document](docs/vcpu.md) vCPU: [vCPU Document](docs/vcpu.md)
API: [API Document](docs/api.md) API: [API Document](docs/api.md)
`Upcall`: [`Upcall` Document](docs/upcall.md)
Currently, the documents are still actively adding. Currently, the documents are still actively adding.
You could see the [official documentation](docs/) page for more details. You could see the [official documentation](docs/) page for more details.

177
src/dragonball/docs/images/upcall-architecture.svg Normal file
View File

@ -0,0 +1,177 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" xmlns:dc="http://purl.org/dc/elements/1.1/" version="1.1" viewBox="51 242 818 479" width="818" height="479">
<defs>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black">
<g>
<path d="M 8 0 L 0 -3 L 0 3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
<marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker_2" stroke-linejoin="miter" stroke-miterlimit="10" viewBox="-9 -4 10 8" markerWidth="10" markerHeight="8" color="black">
<g>
<path d="M -8 0 L 0 3 L 0 -3 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/>
</g>
</marker>
</defs>
<g id="Canvas_1" fill="none" fill-opacity="1" stroke="none" stroke-opacity="1" stroke-dasharray="none">
<title>Canvas 1</title>
<rect fill="white" x="51" y="242" width="818" height="479"/>
<g id="Canvas_1_Layer_1">
<title>Layer 1</title>
<g id="Line_4">
<line x1="153" y1="279.5" x2="856.1097" y2="279.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_5">
<text transform="translate(56 247.5)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="0" y="17">Guest User</tspan>
</text>
</g>
<g id="Graphic_6">
<text transform="translate(56 286)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="0" y="17">Guest Kernel</tspan>
</text>
</g>
<g id="Line_7">
<line x1="153" y1="592" x2="856.1097" y2="592" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_8">
<text transform="translate(62.76 597.5)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="7531753e-19" y="17">Hypervisor</tspan>
</text>
</g>
<g id="Graphic_10">
<path d="M 264 328 L 347.456 328 C 354.0834 328 359.456 333.3726 359.456 340 L 359.456 524.5 C 359.456 531.1274 354.0834 536.5 347.456 536.5 L 264 536.5 C 257.37258 536.5 252 531.1274 252 524.5 L 252 340 C 252 333.3726 257.37258 328 264 328 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_11">
<text transform="translate(276.776 333)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="0" y="17">socket</tspan>
</text>
</g>
<g id="Graphic_12">
<path d="M 582 294.5 L 672 294.5 C 678.6274 294.5 684 299.8726 684 306.5 L 684 354 C 684 360.6274 678.6274 366 672 366 L 582 366 C 575.3726 366 570 360.6274 570 354 L 570 306.5 C 570 299.8726 575.3726 294.5 582 294.5 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(575 302.578)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="27.704" y="15">Device </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="20.44" y="33.448">Manager</tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="25.488" y="51.895996">Service</tspan>
</text>
</g>
<g id="Graphic_13">
<text transform="translate(284.824 374)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="8135714e-19" y="17">bind</tspan>
</text>
</g>
<g id="Graphic_14">
<text transform="translate(280.528 416.25)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="0" y="17">listen</tspan>
</text>
</g>
<g id="Graphic_15">
<text transform="translate(274.92 459.5)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="0" y="17">accept</tspan>
</text>
</g>
<g id="Graphic_16">
<text transform="translate(256.372 503.5)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="8668621e-19" y="17">new kthread</tspan>
</text>
</g>
<g id="Graphic_17">
<path d="M 268 566.5 L 807.5 566.5 C 813.0228 566.5 817.5 570.97715 817.5 576.5 L 817.5 576.5 C 817.5 582.02285 813.0228 586.5 807.5 586.5 L 268 586.5 C 262.47715 586.5 258 582.02285 258 576.5 L 258 576.5 C 258 570.97715 262.47715 566.5 268 566.5 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(263 567.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="226.454" y="15">virtio-vsocket</tspan>
</text>
</g>
<g id="Graphic_18">
<path d="M 268 598.5 L 807.5 598.5 C 813.0228 598.5 817.5 602.97715 817.5 608.5 L 817.5 608.5 C 817.5 614.02285 813.0228 618.5 807.5 618.5 L 268 618.5 C 262.47715 618.5 258 614.02285 258 608.5 L 258 608.5 C 258 602.97715 262.47715 598.5 268 598.5 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(263 599.276)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="193.254" y="15">virtio-vsocket backend</tspan>
</text>
</g>
<g id="Line_20">
<line x1="301.9" y1="352" x2="301.9" y2="369.84976" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_21">
<line x1="300.828" y1="394.6251" x2="300.828" y2="412.4749" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_22">
<line x1="300.828" y1="437.56256" x2="300.828" y2="455.4123" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_23">
<line x1="299.9" y1="480.1251" x2="299.9" y2="497.9749" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_24">
<rect x="266.5" y="541.5" width="71.188" height="20" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(271.5 540.5)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="13.858" y="17">Port</tspan>
</text>
</g>
<g id="Graphic_27">
<path d="M 582 648.5 L 672 648.5 C 678.6274 648.5 684 653.8726 684 660.5 L 684 708 C 684 714.6274 678.6274 720 672 720 L 582 720 C 575.3726 720 570 714.6274 570 708 L 570 660.5 C 570 653.8726 575.3726 648.5 582 648.5 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(575 656.578)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="27.704" y="15">Device </tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="20.44" y="33.448">Manager</tspan>
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="20.288" y="51.895996">Backend</tspan>
</text>
</g>
<g id="Line_28">
<line x1="627" y1="375.9" x2="627" y2="638.6" marker-end="url(#FilledArrow_Marker)" marker-start="url(#FilledArrow_Marker_2)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_31">
<path d="M 711 294.5 L 801 294.5 C 807.6274 294.5 813 299.8726 813 306.5 L 813 354 C 813 360.6274 807.6274 366 801 366 L 711 366 C 704.3726 366 699 360.6274 699 354 L 699 306.5 C 699 299.8726 704.3726 294.5 711 294.5 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(704 321.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="17.784" y="15">Service B</tspan>
</text>
</g>
<g id="Graphic_30">
<path d="M 711 648.5 L 801 648.5 C 807.6274 648.5 813 653.8726 813 660.5 L 813 708 C 813 714.6274 807.6274 720 801 720 L 711 720 C 704.3726 720 699 714.6274 699 708 L 699 660.5 C 699 653.8726 704.3726 648.5 711 648.5 Z" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(704 675.026)" fill="black">
<tspan font-family="Helvetica Neue" font-size="16" fill="black" x="12.584" y="15">Backend B</tspan>
</text>
</g>
<g id="Line_29">
<line x1="756" y1="375.9" x2="756" y2="638.6" marker-end="url(#FilledArrow_Marker)" marker-start="url(#FilledArrow_Marker_2)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="4.0,4.0" stroke-width="1"/>
</g>
<g id="Graphic_32">
<text transform="translate(833 319.25)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="58264504e-20" y="17">……</tspan>
</text>
</g>
<g id="Graphic_33">
<text transform="translate(833 673.25)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="58264504e-20" y="17">……</tspan>
</text>
</g>
<g id="Graphic_34">
<text transform="translate(252.616 296)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="0" y="17">Upcall Server</tspan>
</text>
</g>
<g id="Line_39">
<path d="M 251.372 514.94444 L 196.16455 515.40173 L 196.2135 443.25 L 290.92825 443.92903" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Graphic_41">
<text transform="translate(417 503.5)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="0" y="17">Service handler</tspan>
</text>
</g>
<g id="Graphic_42">
<rect x="591.406" y="540.4723" width="71.188" height="20" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(596.406 539.4723)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="10.386" y="17">Conn</tspan>
</text>
</g>
<g id="Graphic_43">
<rect x="720.406" y="541.4723" width="71.188" height="20" stroke="gray" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
<text transform="translate(725.406 540.4723)" fill="black">
<tspan font-family="Alibaba PuHuiTi" font-size="16" fill="black" x="10.386" y="17">Conn</tspan>
</text>
</g>
<g id="Line_44">
<line x1="358.684" y1="514.5" x2="402.1" y2="514.5" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
<g id="Line_46">
<path d="M 479.2467 498.5 L 480 328 L 560.10116 329.22604" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/>
</g>
</g>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

30
src/dragonball/docs/upcall.md Normal file
View File

@ -0,0 +1,30 @@
# `Upcall`
## What is `Upcall`?
`Upcall` is a direct communication tool between VMM and guest developed upon `vsock`. The server side of the `upcall` is a driver in guest kernel (kernel patches are needed for this feature) and it'll start to serve the requests after the kernel starts. And the client side is in Dragonball VMM , it'll be a thread that communicates with `vsock` through `uds`.
We want to keep the lightweight of the VM through the implementation of the `upcall`.
![architecture overview](images/upcall-architecture.svg)
## What can `upcall` do?
We define specific operations in the device manager service (one of the services in `upcall` we developed) to perform device hotplug / hot-unplug including vCPU hotplug, `virtio-mmio` hotplug, and memory hotplug. We have accomplished device hotplug / hot-unplug directly through `upcall` in order to avoid the virtualization of ACPI to minimize virtual machines overhead. And there could be many other uses if other services are implemented.
## How to enable `upcall`?
`Upcall` needs a server in the guest kernel which will be several kernel patches for the `upcall` server itself and different services registered in the `upcall` server. It's currently tested on upstream Linux kernel 5.10.
To make it easy for users to use, we have open-source the `upcall` guest patches in [Dragonball experimental guest patches](../../../tools/packaging/kernel/patches/5.10.x/dragonball-experimental) and develop `upcall` support in [Kata guest kernel building script](../../../tools/packaging/kernel/build-kernel.sh).
You could use following command to download the upstream kernel (currently Dragonball uses 5.10.25) and put the `upcall` patches and other Kata patches into kernel code.
`sh build-kernel.sh -e -t dragonball -f setup`
`-e` here means experimental, mainly because `upcall` patches are not in upstream Linux kernel.
`-t dragonball` is for specifying hypervisor type
`-f` is for generating `.config` file
After this command, the kernel code with `upcall` and related `.config` file are all set up in the directory `kata-linux-dragonball-experimental-5.10.25-[config version]`. You can either manually compile the kernel with `make` command or following [Document for build-kernel.sh](../../../tools/packaging/kernel/README.md) to build and use this guest kernel.
Also, a client-side is also needed in VMM. Dragonball has already open-source the way to implement `upcall` client and Dragonball compiled with `dbs-upcall` feature will enable Dragonball client side.

5
src/dragonball/src/address_space_manager.rs
View File

@ -250,6 +250,11 @@ impl AddressSpaceMgr {
self.address_space.as_ref() self.address_space.as_ref()
} }
/// Get the guest memory.
pub fn vm_memory(&self) -> Option<<GuestAddressSpaceImpl as GuestAddressSpace>::T> {
self.get_vm_as().map(|m| m.memory())
}
/// Create the address space for a virtual machine. /// Create the address space for a virtual machine.
/// ///
/// This method is designed to be called when starting up a virtual machine instead of at /// This method is designed to be called when starting up a virtual machine instead of at

12
src/dragonball/src/api/v1/vmm_action.rs
View File

@ -656,10 +656,10 @@ mod tests {
let (to_vmm, from_api) = channel(); let (to_vmm, from_api) = channel();
let (to_api, from_vmm) = channel(); let (to_api, from_vmm) = channel();
let vmm = Arc::new(Mutex::new(create_vmm_instance())); let epoll_mgr = EpollManager::default();
let vmm = Arc::new(Mutex::new(create_vmm_instance(epoll_mgr.clone())));
let mut vservice = VmmService::new(from_api, to_api); let mut vservice = VmmService::new(from_api, to_api);
let epoll_mgr = EpollManager::default();
let mut event_mgr = EventManager::new(&vmm, epoll_mgr).unwrap(); let mut event_mgr = EventManager::new(&vmm, epoll_mgr).unwrap();
let mut v = vmm.lock().unwrap(); let mut v = vmm.lock().unwrap();
@ -681,9 +681,9 @@ mod tests {
let (_to_vmm, from_api) = channel(); let (_to_vmm, from_api) = channel();
let (to_api, _from_vmm) = channel(); let (to_api, _from_vmm) = channel();
let vmm = Arc::new(Mutex::new(create_vmm_instance()));
let mut vservice = VmmService::new(from_api, to_api);
let epoll_mgr = EpollManager::default(); let epoll_mgr = EpollManager::default();
let vmm = Arc::new(Mutex::new(create_vmm_instance(epoll_mgr.clone())));
let mut vservice = VmmService::new(from_api, to_api);
let mut event_mgr = EventManager::new(&vmm, epoll_mgr).unwrap(); let mut event_mgr = EventManager::new(&vmm, epoll_mgr).unwrap();
let mut v = vmm.lock().unwrap(); let mut v = vmm.lock().unwrap();
@ -695,9 +695,9 @@ mod tests {
fn test_vmm_action_disconnected() { fn test_vmm_action_disconnected() {
let (to_vmm, from_api) = channel(); let (to_vmm, from_api) = channel();
let (to_api, _from_vmm) = channel(); let (to_api, _from_vmm) = channel();
let vmm = Arc::new(Mutex::new(create_vmm_instance()));
let mut vservice = VmmService::new(from_api, to_api);
let epoll_mgr = EpollManager::default(); let epoll_mgr = EpollManager::default();
let vmm = Arc::new(Mutex::new(create_vmm_instance(epoll_mgr.clone())));
let mut vservice = VmmService::new(from_api, to_api);
let mut event_mgr = EventManager::new(&vmm, epoll_mgr).unwrap(); let mut event_mgr = EventManager::new(&vmm, epoll_mgr).unwrap();
let mut v = vmm.lock().unwrap(); let mut v = vmm.lock().unwrap();

4
src/dragonball/src/device_manager/mod.rs
View File

@ -1065,7 +1065,9 @@ mod tests {
use crate::vm::VmConfigInfo; use crate::vm::VmConfigInfo;
let epoll_manager = EpollManager::default(); let epoll_manager = EpollManager::default();
let vmm = Arc::new(Mutex::new(crate::vmm::tests::create_vmm_instance())); let vmm = Arc::new(Mutex::new(crate::vmm::tests::create_vmm_instance(
epoll_manager.clone(),
)));
let event_mgr = crate::event_manager::EventManager::new(&vmm, epoll_manager).unwrap(); let event_mgr = crate::event_manager::EventManager::new(&vmm, epoll_manager).unwrap();
let mut vm = crate::vm::tests::create_vm_instance(); let mut vm = crate::vm::tests::create_vm_instance();
let vm_config = VmConfigInfo { let vm_config = VmConfigInfo {

207
src/dragonball/src/vm/mod.rs
View File

@ -825,7 +825,14 @@ impl Vm {
#[cfg(test)] #[cfg(test)]
pub mod tests { pub mod tests {
use kvm_ioctls::VcpuExit;
use linux_loader::cmdline::Cmdline;
use test_utils::skip_if_not_root;
use vm_memory::GuestMemory;
use vmm_sys_util::tempfile::TempFile;
use super::*; use super::*;
use crate::test_utils::tests::create_vm_for_test;
impl Vm { impl Vm {
pub fn set_instance_state(&mut self, mstate: InstanceState) { pub fn set_instance_state(&mut self, mstate: InstanceState) {
@ -841,4 +848,204 @@ pub mod tests {
let epoll_manager = EpollManager::default(); let epoll_manager = EpollManager::default();
Vm::new(None, instance_info, epoll_manager).unwrap() Vm::new(None, instance_info, epoll_manager).unwrap()
} }
#[test]
fn test_create_vm_instance() {
skip_if_not_root!();
let vm = create_vm_instance();
assert!(vm.check_health().is_err());
assert!(vm.kernel_config.is_none());
assert!(vm.get_reset_eventfd().is_none());
assert!(!vm.is_vm_initialized());
assert!(!vm.is_vm_running());
assert!(vm.reset_console().is_ok());
}
#[test]
fn test_vm_init_guest_memory() {
skip_if_not_root!();
let vm_config = VmConfigInfo {
vcpu_count: 1,
max_vcpu_count: 3,
cpu_pm: "off".to_string(),
mem_type: "shmem".to_string(),
mem_file_path: "".to_string(),
mem_size_mib: 16,
serial_path: None,
cpu_topology: CpuTopology {
threads_per_core: 1,
cores_per_die: 1,
dies_per_socket: 1,
sockets: 1,
},
vpmu_feature: 0,
};
let mut vm = create_vm_instance();
vm.set_vm_config(vm_config);
assert!(vm.init_guest_memory().is_ok());
let vm_memory = vm.address_space.vm_memory().unwrap();
assert_eq!(vm_memory.num_regions(), 1);
assert_eq!(vm_memory.last_addr(), GuestAddress(0xffffff));
// Reconfigure an already configured vm will be ignored and just return OK.
let vm_config = VmConfigInfo {
vcpu_count: 1,
max_vcpu_count: 3,
cpu_pm: "off".to_string(),
mem_type: "shmem".to_string(),
mem_file_path: "".to_string(),
mem_size_mib: 16,
serial_path: None,
cpu_topology: CpuTopology {
threads_per_core: 1,
cores_per_die: 1,
dies_per_socket: 1,
sockets: 1,
},
vpmu_feature: 0,
};
vm.set_vm_config(vm_config);
assert!(vm.init_guest_memory().is_ok());
let vm_memory = vm.address_space.vm_memory().unwrap();
assert_eq!(vm_memory.num_regions(), 1);
assert_eq!(vm_memory.last_addr(), GuestAddress(0xffffff));
let obj_addr = GuestAddress(0xf0);
vm_memory.write_obj(67u8, obj_addr).unwrap();
let read_val: u8 = vm_memory.read_obj(obj_addr).unwrap();
assert_eq!(read_val, 67u8);
}
#[test]
fn test_vm_create_devices() {
skip_if_not_root!();
let epoll_mgr = EpollManager::default();
let vmm = Arc::new(Mutex::new(crate::vmm::tests::create_vmm_instance(
epoll_mgr.clone(),
)));
let mut guard = vmm.lock().unwrap();
let vm = guard.get_vm_mut().unwrap();
let vm_config = VmConfigInfo {
vcpu_count: 1,
max_vcpu_count: 3,
cpu_pm: "off".to_string(),
mem_type: "shmem".to_string(),
mem_file_path: "".to_string(),
mem_size_mib: 16,
serial_path: None,
cpu_topology: CpuTopology {
threads_per_core: 1,
cores_per_die: 1,
dies_per_socket: 1,
sockets: 1,
},
vpmu_feature: 0,
};
vm.set_vm_config(vm_config);
assert!(vm.init_guest_memory().is_ok());
assert!(vm.setup_interrupt_controller().is_ok());
let vm_memory = vm.address_space.vm_memory().unwrap();
assert_eq!(vm_memory.num_regions(), 1);
assert_eq!(vm_memory.last_addr(), GuestAddress(0xffffff));
let kernel_file = TempFile::new().unwrap();
let cmd_line = Cmdline::new(64);
vm.set_kernel_config(KernelConfigInfo::new(
kernel_file.into_file(),
None,
cmd_line,
));
vm.init_devices(epoll_mgr).unwrap();
}
#[test]
fn test_vm_delete_devices() {
skip_if_not_root!();
let mut vm = create_vm_for_test();
let epoll_mgr = EpollManager::default();
vm.setup_interrupt_controller().unwrap();
vm.init_devices(epoll_mgr).unwrap();
assert!(vm.remove_devices().is_ok());
}
#[test]
fn test_run_code() {
skip_if_not_root!();
use std::io::{self, Write};
// This example is based on https://lwn.net/Articles/658511/
let code = [
0xba, 0xf8, 0x03, /* mov $0x3f8, %dx */
0x00, 0xd8, /* add %bl, %al */
0x04, b'0', /* add $'0', %al */
0xee, /* out %al, (%dx) */
0xb0, b'\n', /* mov $'\n', %al */
0xee, /* out %al, (%dx) */
0xf4, /* hlt */
];
let load_addr = GuestAddress(0x1000);
let instance_info = Arc::new(RwLock::new(InstanceInfo::default()));
let epoll_manager = EpollManager::default();
let mut vm = Vm::new(None, instance_info, epoll_manager).unwrap();
let vcpu_count = 1;
let vm_config = VmConfigInfo {
vcpu_count,
max_vcpu_count: 1,
cpu_pm: "off".to_string(),
mem_type: "shmem".to_string(),
mem_file_path: "".to_string(),
mem_size_mib: 10,
serial_path: None,
cpu_topology: CpuTopology {
threads_per_core: 1,
cores_per_die: 1,
dies_per_socket: 1,
sockets: 1,
},
vpmu_feature: 0,
};
vm.set_vm_config(vm_config);
vm.init_guest_memory().unwrap();
let vm_memory = vm.address_space.vm_memory().unwrap();
vm_memory.write_obj(code, load_addr).unwrap();
let vcpu_fd = vm.vm_fd().create_vcpu(0).unwrap();
let mut vcpu_sregs = vcpu_fd.get_sregs().unwrap();
assert_ne!(vcpu_sregs.cs.base, 0);
assert_ne!(vcpu_sregs.cs.selector, 0);
vcpu_sregs.cs.base = 0;
vcpu_sregs.cs.selector = 0;
vcpu_fd.set_sregs(&vcpu_sregs).unwrap();
let mut vcpu_regs = vcpu_fd.get_regs().unwrap();
vcpu_regs.rip = 0x1000;
vcpu_regs.rax = 2;
vcpu_regs.rbx = 3;
vcpu_regs.rflags = 2;
vcpu_fd.set_regs(&vcpu_regs).unwrap();
match vcpu_fd.run().expect("run failed") {
VcpuExit::IoOut(0x3f8, data) => {
assert_eq!(data.len(), 1);
io::stdout().write_all(data).unwrap();
}
VcpuExit::Hlt => {
io::stdout().write_all(b"KVM_EXIT_HLT\n").unwrap();
}
r => panic!("unexpected exit reason: {:?}", r),
}
}
} }

5
src/dragonball/src/vmm.rs
View File

@ -200,11 +200,10 @@ pub(crate) mod tests {
use super::*; use super::*;
pub fn create_vmm_instance() -> Vmm { pub fn create_vmm_instance(epoll_manager: EpollManager) -> Vmm {
let info = Arc::new(RwLock::new(InstanceInfo::default())); let info = Arc::new(RwLock::new(InstanceInfo::default()));
let event_fd = EventFd::new(libc::EFD_NONBLOCK).unwrap(); let event_fd = EventFd::new(libc::EFD_NONBLOCK).unwrap();
let seccomp_filter: BpfProgram = Vec::new(); let seccomp_filter: BpfProgram = Vec::new();
let epoll_manager = EpollManager::default();
Vmm::new_with_epoll_manager( Vmm::new_with_epoll_manager(
info, info,
@ -221,6 +220,6 @@ pub(crate) mod tests {
fn test_create_vmm_instance() { fn test_create_vmm_instance() {
skip_if_not_root!(); skip_if_not_root!();
create_vmm_instance(); create_vmm_instance(EpollManager::default());
} }
} }

10
src/libs/kata-types/src/config/hypervisor/mod.rs
View File

@ -32,7 +32,7 @@ use regex::RegexSet;
use super::{default, ConfigOps, ConfigPlugin, TomlConfig}; use super::{default, ConfigOps, ConfigPlugin, TomlConfig};
use crate::annotations::KATA_ANNO_CFG_HYPERVISOR_PREFIX; use crate::annotations::KATA_ANNO_CFG_HYPERVISOR_PREFIX;
use crate::{eother, resolve_path, validate_path}; use crate::{eother, resolve_path, sl, validate_path};
mod dragonball; mod dragonball;
pub use self::dragonball::{DragonballConfig, HYPERVISOR_NAME_DRAGONBALL}; pub use self::dragonball::{DragonballConfig, HYPERVISOR_NAME_DRAGONBALL};
@ -779,6 +779,10 @@ pub struct SharedFsInfo {
#[serde(default)] #[serde(default)]
pub virtio_fs_cache_size: u32, pub virtio_fs_cache_size: u32,
/// Default size of virtqueues
#[serde(default)]
pub virtio_fs_queue_size: u32,
/// Enable virtio-fs DAX window if true. /// Enable virtio-fs DAX window if true.
#[serde(default)] #[serde(default)]
pub virtio_fs_is_dax: bool, pub virtio_fs_is_dax: bool,
@ -846,6 +850,10 @@ impl SharedFsInfo {
if self.virtio_fs_cache.is_empty() { if self.virtio_fs_cache.is_empty() {
self.virtio_fs_cache = default::DEFAULT_VIRTIO_FS_CACHE_MODE.to_string(); self.virtio_fs_cache = default::DEFAULT_VIRTIO_FS_CACHE_MODE.to_string();
} }
if self.virtio_fs_cache == *"none" {
warn!(sl!(), "virtio-fs cache mode `none` is deprecated since Kata Containers 2.5.0 and will be removed in the future release, please use `never` instead. For more details please refer to https://github.com/kata-containers/kata-containers/issues/4234.");
self.virtio_fs_cache = default::DEFAULT_VIRTIO_FS_CACHE_MODE.to_string();
}
if self.virtio_fs_is_dax && self.virtio_fs_cache_size == 0 { if self.virtio_fs_is_dax && self.virtio_fs_cache_size == 0 {
self.virtio_fs_cache_size = default::DEFAULT_VIRTIO_FS_DAX_SIZE_MB; self.virtio_fs_cache_size = default::DEFAULT_VIRTIO_FS_DAX_SIZE_MB;
} }

2
src/runtime-rs/Makefile
View File

@ -119,6 +119,7 @@ DEFVALIDVIRTIOFSDAEMONPATHS := [\"$(DEFVIRTIOFSDAEMON)\"]
# if value is 0, DAX is not enabled # if value is 0, DAX is not enabled
DEFVIRTIOFSCACHESIZE ?= 0 DEFVIRTIOFSCACHESIZE ?= 0
DEFVIRTIOFSCACHE ?= auto DEFVIRTIOFSCACHE ?= auto
DEFVIRTIOFSQUEUESIZE ?= 1024
# Format example: # Format example:
# [\"-o\", \"arg1=xxx,arg2\", \"-o\", \"hello world\", \"--arg3=yyy\"] # [\"-o\", \"arg1=xxx,arg2\", \"-o\", \"hello world\", \"--arg3=yyy\"]
# #
@ -256,6 +257,7 @@ USER_VARS += DEFVIRTIOFSDAEMON
USER_VARS += DEFVALIDVIRTIOFSDAEMONPATHS USER_VARS += DEFVALIDVIRTIOFSDAEMONPATHS
USER_VARS += DEFVIRTIOFSCACHESIZE USER_VARS += DEFVIRTIOFSCACHESIZE
USER_VARS += DEFVIRTIOFSCACHE USER_VARS += DEFVIRTIOFSCACHE
USER_VARS += DEFVIRTIOFSQUEUESIZE
USER_VARS += DEFVIRTIOFSEXTRAARGS USER_VARS += DEFVIRTIOFSEXTRAARGS
USER_VARS += DEFENABLEANNOTATIONS USER_VARS += DEFENABLEANNOTATIONS
USER_VARS += DEFENABLEIOTHREADS USER_VARS += DEFENABLEIOTHREADS

28
src/runtime-rs/config/configuration-dragonball.toml.in
View File

@ -136,6 +136,34 @@ block_device_driver = "@DEFBLOCKSTORAGEDRIVER_DB@"
# of shim, does not need an external virtiofsd process. # of shim, does not need an external virtiofsd process.
shared_fs = "@DBSHAREDFS@" shared_fs = "@DBSHAREDFS@"
# Default size of DAX cache in MiB
virtio_fs_cache_size = @DEFVIRTIOFSCACHESIZE@
# Extra args for virtiofsd daemon
#
# Format example:
# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]
# Examples:
# Set virtiofsd log level to debug : ["-o", "log_level=debug"] or ["-d"]
#
# see `virtiofsd -h` for possible options.
virtio_fs_extra_args = @DEFVIRTIOFSEXTRAARGS@
# Cache mode:
#
# - never
# Metadata, data, and pathname lookup are not cached in guest. They are
# always fetched from host and any changes are immediately pushed to host.
#
# - auto
# Metadata and pathname lookup cache expires after a configured amount of
# time (default is 1 second). Data is cached while the file is open (close
# to open consistency).
#
# - always
# Metadata, data, and pathname lookup are cached in guest and never expire.
virtio_fs_cache = "@DEFVIRTIOFSCACHE@"
# Enable huge pages for VM RAM, default false # Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory # Enabling this will result in the VM memory
# being allocated using huge pages. # being allocated using huge pages.

2
src/runtime/config/configuration-qemu.toml.in
View File

@ -336,7 +336,7 @@ valid_file_mem_backends = @DEFVALIDFILEMEMBACKENDS@
pflashes = [] pflashes = []
# This option changes the default hypervisor and kernel parameters # This option changes the default hypervisor and kernel parameters
# to enable debug output where available. # to enable debug output where available. And Debug also enable the hmp socket.
# #
# Default false # Default false
#enable_debug = true #enable_debug = true

11
src/runtime/pkg/govmm/qemu/examples_test.go
View File

@ -27,13 +27,16 @@ func Example() {
// resources // resources
params = append(params, "-m", "370", "-smp", "cpus=2") params = append(params, "-m", "370", "-smp", "cpus=2")
// LaunchCustomQemu should return as soon as the instance has launched as we // LaunchCustomQemu should return immediately. We must then wait
// are using the --daemonize flag. It will set up a unix domain socket // the returned process to terminate as we are using the --daemonize
// called /tmp/qmp-socket that we can use to manage the instance. // flag.
_, err := qemu.LaunchCustomQemu(context.Background(), "", params, nil, nil, nil) // It will set up a unix domain socket called /tmp/qmp-socket that we
// can use to manage the instance.
proc, _, err := qemu.LaunchCustomQemu(context.Background(), "", params, nil, nil, nil)
if err != nil { if err != nil {
panic(err) panic(err)
} }
proc.Wait()
// This channel will be closed when the instance dies. // This channel will be closed when the instance dies.
disconnectedCh := make(chan struct{}) disconnectedCh := make(chan struct{})

73
src/runtime/pkg/govmm/qemu/qemu.go
View File

@ -14,9 +14,9 @@
package qemu package qemu
import ( import (
"bytes"
"context" "context"
"fmt" "fmt"
"io"
"log" "log"
"os" "os"
"os/exec" "os/exec"
@ -2362,10 +2362,17 @@ const (
) )
// QMPSocket represents a qemu QMP socket configuration. // QMPSocket represents a qemu QMP socket configuration.
// nolint: govet
type QMPSocket struct { type QMPSocket struct {
// Type is the socket type (e.g. "unix"). // Type is the socket type (e.g. "unix").
Type QMPSocketType Type QMPSocketType
// Human Monitor Interface (HMP) (true for HMP, false for QMP, default false)
IsHmp bool
// QMP listener file descriptor to be passed to qemu
FD *os.File
// Name is the socket name. // Name is the socket name.
Name string Name string
@ -2378,7 +2385,8 @@ type QMPSocket struct {
// Valid returns true if the QMPSocket structure is valid and complete. // Valid returns true if the QMPSocket structure is valid and complete.
func (qmp QMPSocket) Valid() bool { func (qmp QMPSocket) Valid() bool {
if qmp.Type == "" || qmp.Name == "" { // Exactly one of Name of FD must be set.
if qmp.Type == "" || (qmp.Name == "") == (qmp.FD == nil) {
return false return false
} }
@ -2646,9 +2654,6 @@ type Config struct {
// PidFile is the -pidfile parameter // PidFile is the -pidfile parameter
PidFile string PidFile string
// LogFile is the -D parameter
LogFile string
qemuParams []string qemuParams []string
} }
@ -2718,7 +2723,13 @@ func (config *Config) appendQMPSockets() {
continue continue
} }
qmpParams := append([]string{}, fmt.Sprintf("%s:%s", q.Type, q.Name)) var qmpParams []string
if q.FD != nil {
qemuFDs := config.appendFDs([]*os.File{q.FD})
qmpParams = append([]string{}, fmt.Sprintf("%s:fd=%d", q.Type, qemuFDs[0]))
} else {
qmpParams = append([]string{}, fmt.Sprintf("%s:path=%s", q.Type, q.Name))
}
if q.Server { if q.Server {
qmpParams = append(qmpParams, "server=on") qmpParams = append(qmpParams, "server=on")
if q.NoWait { if q.NoWait {
@ -2726,7 +2737,12 @@ func (config *Config) appendQMPSockets() {
} }
} }
config.qemuParams = append(config.qemuParams, "-qmp") if q.IsHmp {
config.qemuParams = append(config.qemuParams, "-monitor")
} else {
config.qemuParams = append(config.qemuParams, "-qmp")
}
config.qemuParams = append(config.qemuParams, strings.Join(qmpParams, ",")) config.qemuParams = append(config.qemuParams, strings.Join(qmpParams, ","))
} }
} }
@ -2975,13 +2991,6 @@ func (config *Config) appendPidFile() {
} }
} }
func (config *Config) appendLogFile() {
if config.LogFile != "" {
config.qemuParams = append(config.qemuParams, "-D")
config.qemuParams = append(config.qemuParams, config.LogFile)
}
}
func (config *Config) appendFwCfg(logger QMPLog) { func (config *Config) appendFwCfg(logger QMPLog) {
if logger == nil { if logger == nil {
logger = qmpNullLogger{} logger = qmpNullLogger{}
@ -3001,12 +3010,8 @@ func (config *Config) appendFwCfg(logger QMPLog) {
// //
// The Config parameter contains a set of qemu parameters and settings. // The Config parameter contains a set of qemu parameters and settings.
// //
// This function writes its log output via logger parameter. // See LaunchCustomQemu for more information.
// func LaunchQemu(config Config, logger QMPLog) (*exec.Cmd, io.ReadCloser, error) {
// The function will block until the launched qemu process exits. "", nil
// will be returned if the launch succeeds. Otherwise a string containing
// the contents of stderr + a Go error object will be returned.
func LaunchQemu(config Config, logger QMPLog) (string, error) {
config.appendName() config.appendName()
config.appendUUID() config.appendUUID()
config.appendMachine() config.appendMachine()
@ -3024,12 +3029,11 @@ func LaunchQemu(config Config, logger QMPLog) (string, error) {
config.appendIOThreads() config.appendIOThreads()
config.appendIncoming() config.appendIncoming()
config.appendPidFile() config.appendPidFile()
config.appendLogFile()
config.appendFwCfg(logger) config.appendFwCfg(logger)
config.appendSeccompSandbox() config.appendSeccompSandbox()
if err := config.appendCPUs(); err != nil { if err := config.appendCPUs(); err != nil {
return "", err return nil, nil, err
} }
ctx := config.Ctx ctx := config.Ctx
@ -3060,17 +3064,16 @@ func LaunchQemu(config Config, logger QMPLog) (string, error) {
// //
// This function writes its log output via logger parameter. // This function writes its log output via logger parameter.
// //
// The function will block until the launched qemu process exits. "", nil // The function returns cmd, reader, nil where cmd is a Go exec.Cmd object
// will be returned if the launch succeeds. Otherwise a string containing // representing the QEMU process and reader a Go io.ReadCloser object
// the contents of stderr + a Go error object will be returned. // connected to QEMU's stderr, if launched successfully. Otherwise
// nil, nil, err where err is a Go error object is returned.
func LaunchCustomQemu(ctx context.Context, path string, params []string, fds []*os.File, func LaunchCustomQemu(ctx context.Context, path string, params []string, fds []*os.File,
attr *syscall.SysProcAttr, logger QMPLog) (string, error) { attr *syscall.SysProcAttr, logger QMPLog) (*exec.Cmd, io.ReadCloser, error) {
if logger == nil { if logger == nil {
logger = qmpNullLogger{} logger = qmpNullLogger{}
} }
errStr := ""
if path == "" { if path == "" {
path = "qemu-system-x86_64" path = "qemu-system-x86_64"
} }
@ -3084,15 +3087,17 @@ func LaunchCustomQemu(ctx context.Context, path string, params []string, fds []*
cmd.SysProcAttr = attr cmd.SysProcAttr = attr
var stderr bytes.Buffer reader, err := cmd.StderrPipe()
cmd.Stderr = &stderr if err != nil {
logger.Errorf("Unable to connect stderr to a pipe")
return nil, nil, err
}
logger.Infof("launching %s with: %v", path, params) logger.Infof("launching %s with: %v", path, params)
err := cmd.Run() err = cmd.Start()
if err != nil { if err != nil {
logger.Errorf("Unable to launch %s: %v", path, err) logger.Errorf("Unable to launch %s: %v", path, err)
errStr = stderr.String() return nil, nil, err
logger.Errorf("%s", errStr)
} }
return errStr, err return cmd, reader, nil
} }

31
src/runtime/pkg/govmm/qemu/qemu_test.go
View File

@ -698,8 +698,8 @@ func TestFailToAppendCPUs(t *testing.T) {
} }
} }
var qmpSingleSocketServerString = "-qmp unix:cc-qmp,server=on,wait=off" var qmpSingleSocketServerString = "-qmp unix:path=cc-qmp,server=on,wait=off"
var qmpSingleSocketString = "-qmp unix:cc-qmp" var qmpSingleSocketString = "-qmp unix:path=cc-qmp"
func TestAppendSingleQMPSocketServer(t *testing.T) { func TestAppendSingleQMPSocketServer(t *testing.T) {
qmp := QMPSocket{ qmp := QMPSocket{
@ -722,7 +722,27 @@ func TestAppendSingleQMPSocket(t *testing.T) {
testAppend(qmp, qmpSingleSocketString, t) testAppend(qmp, qmpSingleSocketString, t)
} }
var qmpSocketServerString = "-qmp unix:cc-qmp-1,server=on,wait=off -qmp unix:cc-qmp-2,server=on,wait=off" var qmpSocketServerFdString = "-qmp unix:fd=3,server=on,wait=off"
func TestAppendQMPSocketServerFd(t *testing.T) {
foo, _ := os.CreateTemp(os.TempDir(), "govmm-qemu-test")
defer func() {
_ = foo.Close()
_ = os.Remove(foo.Name())
}()
qmp := QMPSocket{
Type: "unix",
FD: foo,
Server: true,
NoWait: true,
}
testAppend(qmp, qmpSocketServerFdString, t)
}
var qmpSocketServerString = "-qmp unix:path=cc-qmp-1,server=on,wait=off -qmp unix:path=cc-qmp-2,server=on,wait=off"
func TestAppendQMPSocketServer(t *testing.T) { func TestAppendQMPSocketServer(t *testing.T) {
qmp := []QMPSocket{ qmp := []QMPSocket{
@ -744,8 +764,7 @@ func TestAppendQMPSocketServer(t *testing.T) {
} }
var pidfile = "/run/vc/vm/iamsandboxid/pidfile" var pidfile = "/run/vc/vm/iamsandboxid/pidfile"
var logfile = "/run/vc/vm/iamsandboxid/logfile" var qemuString = "-name cc-qemu -cpu host -uuid " + agentUUID + " -pidfile " + pidfile
var qemuString = "-name cc-qemu -cpu host -uuid " + agentUUID + " -pidfile " + pidfile + " -D " + logfile
func TestAppendStrings(t *testing.T) { func TestAppendStrings(t *testing.T) {
config := Config{ config := Config{
@ -754,14 +773,12 @@ func TestAppendStrings(t *testing.T) {
UUID: agentUUID, UUID: agentUUID,
CPUModel: "host", CPUModel: "host",
PidFile: pidfile, PidFile: pidfile,
LogFile: logfile,
} }
config.appendName() config.appendName()
config.appendCPUModel() config.appendCPUModel()
config.appendUUID() config.appendUUID()
config.appendPidFile() config.appendPidFile()
config.appendLogFile()
result := strings.Join(config.qemuParams, " ") result := strings.Join(config.qemuParams, " ")
if result != qemuString { if result != qemuString {

10
src/runtime/pkg/govmm/qemu/qmp.go
View File

@ -718,6 +718,16 @@ func QMPStart(ctx context.Context, socket string, cfg QMPConfig, disconnectedCh
return nil, nil, err return nil, nil, err
} }
return QMPStartWithConn(ctx, conn, cfg, disconnectedCh)
}
// Same as QMPStart but with a pre-established connection
func QMPStartWithConn(ctx context.Context, conn net.Conn, cfg QMPConfig, disconnectedCh chan struct{}) (*QMP, *QMPVersion, error) {
if conn == nil {
close(disconnectedCh)
return nil, nil, fmt.Errorf("invalid connection")
}
connectedCh := make(chan *QMPVersion) connectedCh := make(chan *QMPVersion)
q := startQMPLoop(conn, cfg, connectedCh, disconnectedCh) q := startQMPLoop(conn, cfg, connectedCh, disconnectedCh)

16
src/runtime/pkg/govmm/qemu/qmp_test.go
View File

@ -273,6 +273,22 @@ func TestQMPStartBadPath(t *testing.T) {
<-disconnectedCh <-disconnectedCh
} }
// Checks that a call to QMPStartWithConn with a nil connection exits gracefully.
//
// We call QMPStartWithConn with a nil connection.
//
// An error should be returned and the disconnected channel should be closed.
func TestQMPStartWithConnNil(t *testing.T) {
cfg := QMPConfig{Logger: qmpTestLogger{}}
disconnectedCh := make(chan struct{})
q, _, err := QMPStartWithConn(context.Background(), nil, cfg, disconnectedCh)
if err == nil {
t.Errorf("Expected error")
q.Shutdown()
}
<-disconnectedCh
}
// Checks that the qmp_capabilities command is correctly sent. // Checks that the qmp_capabilities command is correctly sent.
// //
// We start a QMPLoop, send the qmp_capabilities command and stop the // We start a QMPLoop, send the qmp_capabilities command and stop the

160
src/runtime/virtcontainers/qemu.go
View File

@ -13,10 +13,14 @@ import (
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"fmt" "fmt"
"io"
"math" "math"
"net"
"os" "os"
"os/exec"
"os/user" "os/user"
"path/filepath" "path/filepath"
"regexp"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
@ -106,6 +110,7 @@ type qemu struct {
const ( const (
consoleSocket = "console.sock" consoleSocket = "console.sock"
qmpSocket = "qmp.sock" qmpSocket = "qmp.sock"
hmpSocket = "hmp.sock"
vhostFSSocket = "vhost-fs.sock" vhostFSSocket = "vhost-fs.sock"
nydusdAPISock = "nydusd-api.sock" nydusdAPISock = "nydusd-api.sock"
@ -313,6 +318,10 @@ func (q *qemu) qmpSocketPath(id string) (string, error) {
return utils.BuildSocketPath(q.config.VMStorePath, id, qmpSocket) return utils.BuildSocketPath(q.config.VMStorePath, id, qmpSocket)
} }
func (q *qemu) hmpSocketPath(id string) (string, error) {
return utils.BuildSocketPath(q.config.VMStorePath, id, hmpSocket)
}
func (q *qemu) getQemuMachine() (govmmQemu.Machine, error) { func (q *qemu) getQemuMachine() (govmmQemu.Machine, error) {
machine := q.arch.machine() machine := q.arch.machine()
@ -354,14 +363,30 @@ func (q *qemu) createQmpSocket() ([]govmmQemu.QMPSocket, error) {
path: monitorSockPath, path: monitorSockPath,
} }
return []govmmQemu.QMPSocket{ var sockets []govmmQemu.QMPSocket
{
sockets = append(sockets, govmmQemu.QMPSocket{
Type: "unix",
Server: true,
NoWait: true,
})
if q.HypervisorConfig().Debug {
humanMonitorSockPath, err := q.hmpSocketPath(q.id)
if err != nil {
return nil, err
}
sockets = append(sockets, govmmQemu.QMPSocket{
Type: "unix", Type: "unix",
Name: q.qmpMonitorCh.path, IsHmp: true,
Name: humanMonitorSockPath,
Server: true, Server: true,
NoWait: true, NoWait: true,
}, })
}, nil }
return sockets, nil
} }
func (q *qemu) buildDevices(ctx context.Context, initrdPath string) ([]govmmQemu.Device, *govmmQemu.IOThread, error) { func (q *qemu) buildDevices(ctx context.Context, initrdPath string) ([]govmmQemu.Device, *govmmQemu.IOThread, error) {
@ -520,7 +545,7 @@ func (q *qemu) CreateVM(ctx context.Context, id string, network Network, hypervi
NoDefaults: true, NoDefaults: true,
NoGraphic: true, NoGraphic: true,
NoReboot: true, NoReboot: true,
Daemonize: true, Daemonize: false,
MemPrealloc: q.config.MemPrealloc, MemPrealloc: q.config.MemPrealloc,
HugePages: q.config.HugePages, HugePages: q.config.HugePages,
IOMMUPlatform: q.config.IOMMUPlatform, IOMMUPlatform: q.config.IOMMUPlatform,
@ -880,6 +905,77 @@ func (q *qemu) AttestVM(ctx context.Context) error {
return nil return nil
} }
// setupEarlyQmpConnection creates a listener socket to be passed to QEMU
// as a QMP listening endpoint. An initial connection is established, to
// be used as the QMP client socket. This allows to detect an early failure
// of QEMU instead of looping on connect until some timeout expires.
func (q *qemu) setupEarlyQmpConnection() (net.Conn, error) {
monitorSockPath := q.qmpMonitorCh.path
qmpListener, err := net.Listen("unix", monitorSockPath)
if err != nil {
q.Logger().WithError(err).Errorf("Unable to listen on unix socket address (%s)", monitorSockPath)
return nil, err
}
// A duplicate fd of this socket will be passed to QEMU. We must
// close the original one when we're done.
defer qmpListener.Close()
if rootless.IsRootless() {
err = syscall.Chown(monitorSockPath, int(q.config.Uid), int(q.config.Gid))
if err != nil {
q.Logger().WithError(err).Errorf("Unable to make unix socket (%s) rootless", monitorSockPath)
return nil, err
}
}
VMFd, err := qmpListener.(*net.UnixListener).File()
if err != nil {
return nil, err
}
defer func() {
if err != nil {
VMFd.Close()
}
}()
// This socket will be used to establish the initial QMP connection
dialer := net.Dialer{Cancel: q.qmpMonitorCh.ctx.Done()}
conn, err := dialer.Dial("unix", monitorSockPath)
if err != nil {
q.Logger().WithError(err).Errorf("Unable to connect to unix socket (%s)", monitorSockPath)
return nil, err
}
// We need to keep the socket file around to be able to re-connect
qmpListener.(*net.UnixListener).SetUnlinkOnClose(false)
// Pass the duplicated fd of the listener socket to QEMU
q.qemuConfig.QMPSockets[0].FD = VMFd
q.fds = append(q.fds, q.qemuConfig.QMPSockets[0].FD)
return conn, nil
}
func (q *qemu) LogAndWait(qemuCmd *exec.Cmd, reader io.ReadCloser) {
pid := qemuCmd.Process.Pid
q.Logger().Infof("Start logging QEMU (qemuPid=%d)", pid)
scanner := bufio.NewScanner(reader)
warnRE := regexp.MustCompile("(^[^:]+: )warning: ")
for scanner.Scan() {
text := scanner.Text()
if warnRE.MatchString(text) {
text = warnRE.ReplaceAllString(text, "$1")
q.Logger().WithField("qemuPid", pid).Warning(text)
} else {
q.Logger().WithField("qemuPid", pid).Error(text)
}
}
q.Logger().Infof("Stop logging QEMU (qemuPid=%d)", pid)
qemuCmd.Wait()
}
// StartVM will start the Sandbox's VM. // StartVM will start the Sandbox's VM.
func (q *qemu) StartVM(ctx context.Context, timeout int) error { func (q *qemu) StartVM(ctx context.Context, timeout int) error {
span, ctx := katatrace.Trace(ctx, q.Logger(), "StartVM", qemuTracingTags, map[string]string{"sandbox_id": q.id}) span, ctx := katatrace.Trace(ctx, q.Logger(), "StartVM", qemuTracingTags, map[string]string{"sandbox_id": q.id})
@ -912,10 +1008,6 @@ func (q *qemu) StartVM(ctx context.Context, timeout int) error {
return err return err
} }
q.Logger().WithField("vm path", vmPath).Info("created vm path") q.Logger().WithField("vm path", vmPath).Info("created vm path")
// append logfile only on debug
if q.config.Debug {
q.qemuConfig.LogFile = filepath.Join(vmPath, "qemu.log")
}
defer func() { defer func() {
if err != nil { if err != nil {
@ -925,6 +1017,12 @@ func (q *qemu) StartVM(ctx context.Context, timeout int) error {
} }
}() }()
var qmpConn net.Conn
qmpConn, err = q.setupEarlyQmpConnection()
if err != nil {
return err
}
// This needs to be done as late as possible, just before launching // This needs to be done as late as possible, just before launching
// virtiofsd are executed by kata-runtime after this call, run with // virtiofsd are executed by kata-runtime after this call, run with
// the SELinux label. If these processes require privileged, we do // the SELinux label. If these processes require privileged, we do
@ -950,20 +1048,23 @@ func (q *qemu) StartVM(ctx context.Context, timeout int) error {
} }
var strErr string qemuCmd, reader, err := govmmQemu.LaunchQemu(q.qemuConfig, newQMPLogger())
strErr, err = govmmQemu.LaunchQemu(q.qemuConfig, newQMPLogger())
if err != nil { if err != nil {
if q.config.Debug && q.qemuConfig.LogFile != "" { q.Logger().WithError(err).Error("failed to launch qemu")
b, err := os.ReadFile(q.qemuConfig.LogFile) return fmt.Errorf("failed to launch qemu: %s", err)
if err == nil { }
strErr += string(b) if q.qemuConfig.Knobs.Daemonize {
} // LaunchQemu returns a handle on the upper QEMU process.
} // Wait for it to exit to assume that the QEMU daemon was
q.Logger().WithError(err).Errorf("failed to launch qemu: %s", strErr) // actually started.
return fmt.Errorf("failed to launch qemu: %s, error messages from qemu log: %s", err, strErr) qemuCmd.Wait()
} else {
// Log QEMU errors and ensure the QEMU process is reaped after
// termination.
go q.LogAndWait(qemuCmd, reader)
} }
err = q.waitVM(ctx, timeout) err = q.waitVM(ctx, qmpConn, timeout)
if err != nil { if err != nil {
return err return err
} }
@ -1001,7 +1102,7 @@ func (q *qemu) bootFromTemplate() error {
} }
// waitVM will wait for the Sandbox's VM to be up and running. // waitVM will wait for the Sandbox's VM to be up and running.
func (q *qemu) waitVM(ctx context.Context, timeout int) error { func (q *qemu) waitVM(ctx context.Context, qmpConn net.Conn, timeout int) error {
span, _ := katatrace.Trace(ctx, q.Logger(), "waitVM", qemuTracingTags, map[string]string{"sandbox_id": q.id}) span, _ := katatrace.Trace(ctx, q.Logger(), "waitVM", qemuTracingTags, map[string]string{"sandbox_id": q.id})
defer span.End() defer span.End()
@ -1021,7 +1122,7 @@ func (q *qemu) waitVM(ctx context.Context, timeout int) error {
timeStart := time.Now() timeStart := time.Now()
for { for {
disconnectCh = make(chan struct{}) disconnectCh = make(chan struct{})
qmp, ver, err = govmmQemu.QMPStart(q.qmpMonitorCh.ctx, q.qmpMonitorCh.path, cfg, disconnectCh) qmp, ver, err = govmmQemu.QMPStartWithConn(q.qmpMonitorCh.ctx, qmpConn, cfg, disconnectCh)
if err == nil { if err == nil {
break break
} }
@ -1071,19 +1172,6 @@ func (q *qemu) StopVM(ctx context.Context, waitOnly bool) (err error) {
} }
}() }()
if q.config.Debug && q.qemuConfig.LogFile != "" {
f, err := os.OpenFile(q.qemuConfig.LogFile, os.O_RDONLY, 0)
if err == nil {
scanner := bufio.NewScanner(f)
for scanner.Scan() {
q.Logger().WithField("file", q.qemuConfig.LogFile).Debug(scanner.Text())
}
if err := scanner.Err(); err != nil {
q.Logger().WithError(err).Debug("read qemu log failed")
}
}
}
if err := q.qmpSetup(); err != nil { if err := q.qmpSetup(); err != nil {
return err return err
} }

4
tools/packaging/kata-deploy/local-build/Makefile
View File

@ -37,6 +37,7 @@ all-parallel: $(MK_DIR)/dockerbuild/install_yq.sh
all: serial-targets \ all: serial-targets \
firecracker-tarball \ firecracker-tarball \
kernel-tarball \ kernel-tarball \
kernel-dragonball-experimental-tarball \
nydus-tarball \ nydus-tarball \
qemu-tarball \ qemu-tarball \
shim-v2-tarball \ shim-v2-tarball \
@ -60,6 +61,9 @@ firecracker-tarball:
kernel-tarball: kernel-tarball:
${MAKE} $@-build ${MAKE} $@-build
kernel-dragonball-experimental-tarball:
${MAKE} $@-build
kernel-experimental-tarball: kernel-experimental-tarball:
${MAKE} $@-build ${MAKE} $@-build

10
tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh
View File

@ -82,6 +82,7 @@ options:
cloud-hypervisor cloud-hypervisor
firecracker firecracker
kernel kernel
kernel-dragonball-experimental
kernel-experimental kernel-experimental
nydus nydus
qemu qemu
@ -509,6 +510,13 @@ install_kernel() {
DESTDIR="${destdir}" PREFIX="${prefix}" "${kernel_builder}" -f -v "${kernel_version}" DESTDIR="${destdir}" PREFIX="${prefix}" "${kernel_builder}" -f -v "${kernel_version}"
} }
#Install dragonball experimental kernel asset
install_dragonball_experimental_kernel() {
info "build dragonball experimental kernel"
export kernel_version="$(yq r $versions_yaml assets.dragonball-kernel-experimental.version)"
info "kernel version ${kernel_version}"
DESTDIR="${destdir}" PREFIX="${prefix}" "${kernel_builder}" -e -t dragonball -v ${kernel_version}
}
#Install experimental kernel asset #Install experimental kernel asset
install_experimental_kernel() { install_experimental_kernel() {
@ -653,6 +661,8 @@ handle_build() {
nydus) install_nydus ;; nydus) install_nydus ;;
kernel-dragonball-experimental) install_dragonball_experimental_kernel;;
kernel-experimental) install_experimental_kernel;; kernel-experimental) install_experimental_kernel;;
qemu) install_qemu ;; qemu) install_qemu ;;

2
versions.yaml
View File

@ -196,7 +196,7 @@ externals:
cni-plugins: cni-plugins:
description: "CNI network plugins" description: "CNI network plugins"
url: "https://github.com/containernetworking/plugins" url: "https://github.com/containernetworking/plugins"
version: "v1.1.1" version: "v1.2.0"
conmon: conmon:
description: "An OCI container runtime monitor" description: "An OCI container runtime monitor"