From 02cc4fe9dbdb5d634e654a2890fbf8fc5a7e39bd Mon Sep 17 00:00:00 2001
From: Yuan-Zhuo <yuanzhuo0118@outlook.com>
Date: Fri, 28 Jul 2023 17:16:51 +0800
Subject: [PATCH 01/49] runtime-rs: add support for gather metrics in
 runtime-rs

1. Implemented metrics collection for runtime-rs shim and dragonball hypervisor.
2. Described the current supported metrics in runtime-rs.(docs/design/kata-metrics-in-runtime-rs.md)

Fixes: #5017

Signed-off-by: Yuan-Zhuo <yuanzhuo0118@outlook.com>
---
 docs/design/README.md                         |    1 +
 docs/design/kata-metrics-in-runtime-rs.md     |   50 +
 src/dragonball/Cargo.lock                     | 1705 -----------------
 src/dragonball/Cargo.toml                     |    5 +-
 src/dragonball/src/api/v1/vmm_action.rs       |   20 +
 src/dragonball/src/hypervisor_metrics.rs      |  102 +
 src/dragonball/src/lib.rs                     |    5 +
 src/runtime-rs/Cargo.lock                     |  590 ++++--
 src/runtime-rs/crates/agent/src/kata/agent.rs |    3 +-
 src/runtime-rs/crates/agent/src/kata/trans.rs |   32 +-
 src/runtime-rs/crates/agent/src/lib.rs        |   16 +-
 src/runtime-rs/crates/agent/src/types.rs      |    5 +
 .../hypervisor/src/ch/inner_hypervisor.rs     |    4 +
 .../crates/hypervisor/src/ch/mod.rs           |    5 +
 .../src/dragonball/inner_hypervisor.rs        |    5 +
 .../crates/hypervisor/src/dragonball/mod.rs   |    5 +
 .../hypervisor/src/dragonball/vmm_instance.rs |    9 +
 src/runtime-rs/crates/hypervisor/src/lib.rs   |    1 +
 .../crates/hypervisor/src/qemu/inner.rs       |    4 +
 .../crates/hypervisor/src/qemu/mod.rs         |    5 +
 src/runtime-rs/crates/runtimes/Cargo.toml     |    2 +
 .../crates/runtimes/common/src/sandbox.rs     |    4 +
 src/runtime-rs/crates/runtimes/src/lib.rs     |    4 +
 .../crates/runtimes/src/shim_metrics.rs       |  235 +++
 .../crates/runtimes/src/shim_mgmt/handlers.rs |   20 +-
 .../runtimes/virt_container/src/sandbox.rs    |   12 +
 26 files changed, 914 insertions(+), 1935 deletions(-)
 create mode 100644 docs/design/kata-metrics-in-runtime-rs.md
 delete mode 100644 src/dragonball/Cargo.lock
 create mode 100644 src/dragonball/src/hypervisor_metrics.rs
 create mode 100644 src/runtime-rs/crates/runtimes/src/shim_metrics.rs

diff --git a/docs/design/README.md b/docs/design/README.md
index 63d52c655c..60a1de910f 100644
--- a/docs/design/README.md
+++ b/docs/design/README.md
@@ -14,6 +14,7 @@ Kata Containers design documents:
 - [`Inotify` support](inotify.md)
 - [`Hooks` support](hooks-handling.md)
 - [Metrics(Kata 2.0)](kata-2-0-metrics.md)
+- [Metrics in Rust Runtime(runtime-rs)](kata-metrics-in-runtime-rs.md)
 - [Design for Kata Containers `Lazyload` ability with `nydus`](kata-nydus-design.md)
 - [Design for direct-assigned volume](direct-blk-device-assignment.md)
 - [Design for core-scheduling](core-scheduling.md)
diff --git a/docs/design/kata-metrics-in-runtime-rs.md b/docs/design/kata-metrics-in-runtime-rs.md
new file mode 100644
index 0000000000..0376958942
--- /dev/null
+++ b/docs/design/kata-metrics-in-runtime-rs.md
@@ -0,0 +1,50 @@
+# Kata Metrics in Rust Runtime(runtime-rs)
+
+Rust Runtime(runtime-rs) is responsible for:
+
+- Gather metrics about `shim`.
+- Gather metrics from `hypervisor` (through `channel`).
+- Get metrics from `agent` (through `ttrpc`).
+
+---
+
+Here are listed all the metrics gathered by `runtime-rs`.
+
+> * Current status of each entry is marked as:
+>  * ✅：DONE
+>   * 🚧：TODO
+
+### Kata Shim
+
+| STATUS | Metric name                                                  | Type        | Units          | Labels                                                       |
+| ------ | ------------------------------------------------------------ | ----------- | -------------- | ------------------------------------------------------------ |
+| 🚧      | `kata_shim_agent_rpc_durations_histogram_milliseconds`: <br> RPC latency distributions. | `HISTOGRAM` | `milliseconds` | <ul><li>`action` (RPC actions of Kata agent)<ul><li>`grpc.CheckRequest`</li><li>`grpc.CloseStdinRequest`</li><li>`grpc.CopyFileRequest`</li><li>`grpc.CreateContainerRequest`</li><li>`grpc.CreateSandboxRequest`</li><li>`grpc.DestroySandboxRequest`</li><li>`grpc.ExecProcessRequest`</li><li>`grpc.GetMetricsRequest`</li><li>`grpc.GuestDetailsRequest`</li><li>`grpc.ListInterfacesRequest`</li><li>`grpc.ListProcessesRequest`</li><li>`grpc.ListRoutesRequest`</li><li>`grpc.MemHotplugByProbeRequest`</li><li>`grpc.OnlineCPUMemRequest`</li><li>`grpc.PauseContainerRequest`</li><li>`grpc.RemoveContainerRequest`</li><li>`grpc.ReseedRandomDevRequest`</li><li>`grpc.ResumeContainerRequest`</li><li>`grpc.SetGuestDateTimeRequest`</li><li>`grpc.SignalProcessRequest`</li><li>`grpc.StartContainerRequest`</li><li>`grpc.StatsContainerRequest`</li><li>`grpc.TtyWinResizeRequest`</li><li>`grpc.UpdateContainerRequest`</li><li>`grpc.UpdateInterfaceRequest`</li><li>`grpc.UpdateRoutesRequest`</li><li>`grpc.WaitProcessRequest`</li><li>`grpc.WriteStreamRequest`</li></ul></li><li>`sandbox_id`</li></ul> |
+| ✅      | `kata_shim_fds`: <br> Kata containerd shim v2 open FDs.      | `GAUGE`     |                | <ul><li>`sandbox_id`</li></ul>                               |
+| ✅      | `kata_shim_io_stat`: <br> Kata containerd shim v2 process IO statistics. | `GAUGE`     |                | <ul><li>`item` (see `/proc/<pid>/io`)<ul><li>`cancelledwritebytes`</li><li>`rchar`</li><li>`readbytes`</li><li>`syscr`</li><li>`syscw`</li><li>`wchar`</li><li>`writebytes`</li></ul></li><li>`sandbox_id`</li></ul> |
+| ✅      | `kata_shim_netdev`: <br> Kata containerd shim v2 network devices statistics. | `GAUGE`     |                | <ul><li>`interface` (network device name)</li><li>`item` (see `/proc/net/dev`)<ul><li>`recv_bytes`</li><li>`recv_compressed`</li><li>`recv_drop`</li><li>`recv_errs`</li><li>`recv_fifo`</li><li>`recv_frame`</li><li>`recv_multicast`</li><li>`recv_packets`</li><li>`sent_bytes`</li><li>`sent_carrier`</li><li>`sent_colls`</li><li>`sent_compressed`</li><li>`sent_drop`</li><li>`sent_errs`</li><li>`sent_fifo`</li><li>`sent_packets`</li></ul></li><li>`sandbox_id`</li></ul> |
+| 🚧      | `kata_shim_pod_overhead_cpu`: <br> Kata Pod overhead for CPU resources(percent). | `GAUGE`     | percent        | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_pod_overhead_memory_in_bytes`: <br> Kata Pod overhead for memory resources(bytes). | `GAUGE`     | `bytes`        | <ul><li>`sandbox_id`</li></ul>                               |
+| ✅      | `kata_shim_proc_stat`: <br> Kata containerd shim v2 process statistics. | `GAUGE`     |                | <ul><li>`item` (see `/proc/<pid>/stat`)<ul><li>`cstime`</li><li>`cutime`</li><li>`stime`</li><li>`utime`</li></ul></li><li>`sandbox_id`</li></ul> |
+| ✅      | `kata_shim_proc_status`: <br> Kata containerd shim v2 process status. | `GAUGE`     |                | <ul><li>`item` (see `/proc/<pid>/status`)<ul><li>`hugetlbpages`</li><li>`nonvoluntary_ctxt_switches`</li><li>`rssanon`</li><li>`rssfile`</li><li>`rssshmem`</li><li>`vmdata`</li><li>`vmexe`</li><li>`vmhwm`</li><li>`vmlck`</li><li>`vmlib`</li><li>`vmpeak`</li><li>`vmpin`</li><li>`vmpmd`</li><li>`vmpte`</li><li>`vmrss`</li><li>`vmsize`</li><li>`vmstk`</li><li>`vmswap`</li><li>`voluntary_ctxt_switches`</li></ul></li><li>`sandbox_id`</li></ul> |
+| 🚧      | `kata_shim_process_cpu_seconds_total`: <br> Total user and system CPU time spent in seconds. | `COUNTER`   | `seconds`      | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_process_max_fds`: <br> Maximum number of open file descriptors. | `GAUGE`     |                | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_process_open_fds`: <br> Number of open file descriptors. | `GAUGE`     |                | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_process_resident_memory_bytes`: <br> Resident memory size in bytes. | `GAUGE`     | `bytes`        | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_process_start_time_seconds`: <br> Start time of the process since `unix` epoch in seconds. | `GAUGE`     | `seconds`      | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_process_virtual_memory_bytes`: <br> Virtual memory size in bytes. | `GAUGE`     | `bytes`        | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_process_virtual_memory_max_bytes`: <br> Maximum amount of virtual memory available in bytes. | `GAUGE`     | `bytes`        | <ul><li>`sandbox_id`</li></ul>                               |
+| 🚧      | `kata_shim_rpc_durations_histogram_milliseconds`: <br> RPC latency distributions. | `HISTOGRAM` | `milliseconds` | <ul><li>`action` (Kata shim v2 actions)<ul><li>`checkpoint`</li><li>`close_io`</li><li>`connect`</li><li>`create`</li><li>`delete`</li><li>`exec`</li><li>`kill`</li><li>`pause`</li><li>`pids`</li><li>`resize_pty`</li><li>`resume`</li><li>`shutdown`</li><li>`start`</li><li>`state`</li><li>`stats`</li><li>`update`</li><li>`wait`</li></ul></li><li>`sandbox_id`</li></ul> |
+| ✅      | `kata_shim_threads`: <br> Kata containerd shim v2 process threads. | `GAUGE`     |                | <ul><li>`sandbox_id`</li></ul>                               |
+
+### Kata Hypervisor
+
+Different from golang runtime, hypervisor and shim in runtime-rs belong to the **same process**, so all previous metrics for hypervisor and shim only need to be gathered once. Thus, we currently only collect previous metrics in kata shim.
+
+At the same time, we added the interface(`VmmAction::GetHypervisorMetrics`) to gather hypervisor metrics, in case we design tailor-made metrics for hypervisor in the future. Here're metrics exposed from [src/dragonball/src/metric.rs](https://github.com/kata-containers/kata-containers/blob/main/src/dragonball/src/metric.rs).
+
+| Metric name                                                  | Type       | Units | Labels                                                       |
+| ------------------------------------------------------------ | ---------- | ----- | ------------------------------------------------------------ |
+| `kata_hypervisor_scrape_count`: <br> Metrics scrape count    | `COUNTER`  |       | <ul><li>`sandbox_id`</li></ul>                               |
+| `kata_hypervisor_vcpu`: <br>Hypervisor metrics specific to VCPUs' mode of functioning. | `IntGauge` |       | <ul><li>`item`<ul><li>`exit_io_in`</li><li>`exit_io_out`</li><li>`exit_mmio_read`</li><li>`exit_mmio_write`</li><li>`failures`</li><li>`filter_cpuid`</li></ul></li><li>`sandbox_id`</li></ul> |
+| `kata_hypervisor_seccomp`: <br> Hypervisor metrics for the seccomp filtering. | `IntGauge` |       | <ul><li>`item`<ul><li>`num_faults`</li></ul></li><li>`sandbox_id`</li></ul> |
+| `kata_hypervisor_seccomp`: <br> Hypervisor metrics for the seccomp filtering. | `IntGauge` |       | <ul><li>`item`<ul><li>`sigbus`</li><li>`sigsegv`</li></ul></li><li>`sandbox_id`</li></ul> |
diff --git a/src/dragonball/Cargo.lock b/src/dragonball/Cargo.lock
deleted file mode 100644
index b71455729c..0000000000
--- a/src/dragonball/Cargo.lock
+++ /dev/null
@@ -1,1705 +0,0 @@
-# This file is automatically @generated by Cargo.
-# It is not intended for manual editing.
-version = 3
-
-[[package]]
-name = "addr2line"
-version = "0.19.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97"
-dependencies = [
- "gimli",
-]
-
-[[package]]
-name = "adler"
-version = "1.0.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe"
-
-[[package]]
-name = "anyhow"
-version = "1.0.69"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "224afbd727c3d6e4b90103ece64b8d1b67fbb1973b1046c2281eed3f3803f800"
-
-[[package]]
-name = "arc-swap"
-version = "1.6.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bddcadddf5e9015d310179a59bb28c4d4b9920ad0f11e8e14dbadf654890c9a6"
-
-[[package]]
-name = "arrayref"
-version = "0.3.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a4c527152e37cf757a3f78aae5a06fbeefdb07ccc535c980a3208ee3060dd544"
-
-[[package]]
-name = "arrayvec"
-version = "0.7.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
-
-[[package]]
-name = "atty"
-version = "0.2.14"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8"
-dependencies = [
- "hermit-abi 0.1.19",
- "libc",
- "winapi",
-]
-
-[[package]]
-name = "autocfg"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
-
-[[package]]
-name = "backtrace"
-version = "0.3.67"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "233d376d6d185f2a3093e58f283f60f880315b6c60075b01f36b3b85154564ca"
-dependencies = [
- "addr2line",
- "cc",
- "cfg-if",
- "libc",
- "miniz_oxide",
- "object",
- "rustc-demangle",
-]
-
-[[package]]
-name = "bitflags"
-version = "1.3.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
-
-[[package]]
-name = "blake3"
-version = "1.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "42ae2468a89544a466886840aa467a25b766499f4f04bf7d9fcd10ecee9fccef"
-dependencies = [
- "arrayref",
- "arrayvec",
- "cc",
- "cfg-if",
- "constant_time_eq",
- "digest",
-]
-
-[[package]]
-name = "block-buffer"
-version = "0.10.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "69cce20737498f97b993470a6e536b8523f0af7892a4f928cceb1ac5e52ebe7e"
-dependencies = [
- "generic-array",
-]
-
-[[package]]
-name = "byteorder"
-version = "1.4.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
-
-[[package]]
-name = "bytes"
-version = "1.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "89b2fd2a0dcf38d7971e2194b6b6eebab45ae01067456a7fd93d5547a61b70be"
-
-[[package]]
-name = "caps"
-version = "0.5.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "190baaad529bcfbde9e1a19022c42781bdb6ff9de25721abdb8fd98c0807730b"
-dependencies = [
- "libc",
- "thiserror",
-]
-
-[[package]]
-name = "cc"
-version = "1.0.79"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f"
-dependencies = [
- "jobserver",
-]
-
-[[package]]
-name = "cfg-if"
-version = "1.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
-
-[[package]]
-name = "cmake"
-version = "0.1.49"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "db34956e100b30725f2eb215f90d4871051239535632f84fea3bc92722c66b7c"
-dependencies = [
- "cc",
-]
-
-[[package]]
-name = "constant_time_eq"
-version = "0.2.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f3ad85c1f65dc7b37604eb0e89748faf0b9653065f2a8ef69f96a687ec1e9279"
-
-[[package]]
-name = "core-foundation-sys"
-version = "0.8.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5827cebf4670468b8772dd191856768aedcb1b0278a04f989f7766351917b9dc"
-
-[[package]]
-name = "cpufeatures"
-version = "0.2.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "28d997bd5e24a5928dd43e46dc529867e207907fe0b239c3477d924f7f2ca320"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "crc32fast"
-version = "1.3.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b540bd8bc810d3885c6ea91e2018302f68baba2129ab3e88f32389ee9370880d"
-dependencies = [
- "cfg-if",
-]
-
-[[package]]
-name = "crossbeam-channel"
-version = "0.5.8"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a33c2bf77f2df06183c3aa30d1e96c0695a313d4f9c453cc3762a6db39f99200"
-dependencies = [
- "cfg-if",
- "crossbeam-utils",
-]
-
-[[package]]
-name = "crossbeam-utils"
-version = "0.8.15"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3c063cd8cc95f5c377ed0d4b49a4b21f632396ff690e8470c29b3359b346984b"
-dependencies = [
- "cfg-if",
-]
-
-[[package]]
-name = "crypto-common"
-version = "0.1.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3"
-dependencies = [
- "generic-array",
- "typenum",
-]
-
-[[package]]
-name = "dbs-address-space"
-version = "0.3.0"
-dependencies = [
- "arc-swap",
- "lazy_static",
- "libc",
- "nix 0.23.2",
- "thiserror",
- "vm-memory",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "dbs-allocator"
-version = "0.1.1"
-dependencies = [
- "thiserror",
-]
-
-[[package]]
-name = "dbs-arch"
-version = "0.2.3"
-dependencies = [
- "kvm-bindings",
- "kvm-ioctls",
- "libc",
- "memoffset",
- "thiserror",
- "vm-memory",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "dbs-boot"
-version = "0.4.0"
-dependencies = [
- "dbs-arch",
- "kvm-bindings",
- "kvm-ioctls",
- "lazy_static",
- "libc",
- "thiserror",
- "vm-fdt",
- "vm-memory",
-]
-
-[[package]]
-name = "dbs-device"
-version = "0.2.0"
-dependencies = [
- "thiserror",
-]
-
-[[package]]
-name = "dbs-interrupt"
-version = "0.2.2"
-dependencies = [
- "dbs-arch",
- "dbs-device",
- "kvm-bindings",
- "kvm-ioctls",
- "libc",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "dbs-legacy-devices"
-version = "0.1.1"
-dependencies = [
- "dbs-device",
- "dbs-utils",
- "libc",
- "log",
- "serde",
- "vm-superio",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "dbs-upcall"
-version = "0.3.0"
-dependencies = [
- "anyhow",
- "dbs-utils",
- "dbs-virtio-devices",
- "log",
- "thiserror",
- "timerfd",
-]
-
-[[package]]
-name = "dbs-utils"
-version = "0.2.1"
-dependencies = [
- "anyhow",
- "event-manager",
- "libc",
- "log",
- "serde",
- "thiserror",
- "timerfd",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "dbs-virtio-devices"
-version = "0.3.1"
-dependencies = [
- "byteorder",
- "caps",
- "dbs-device",
- "dbs-interrupt",
- "dbs-utils",
- "epoll",
- "fuse-backend-rs",
- "io-uring",
- "kvm-bindings",
- "kvm-ioctls",
- "libc",
- "log",
- "nix 0.24.3",
- "nydus-api",
- "nydus-rafs",
- "nydus-storage",
- "rlimit",
- "sendfd",
- "serde",
- "serde_json",
- "thiserror",
- "threadpool",
- "virtio-bindings",
- "virtio-queue",
- "vm-memory",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "digest"
-version = "0.10.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8168378f4e5023e7218c89c891c0fd8ecdb5e5e4f18cb78f38cf245dd021e76f"
-dependencies = [
- "block-buffer",
- "crypto-common",
- "subtle",
-]
-
-[[package]]
-name = "dirs-next"
-version = "2.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b98cf8ebf19c3d1b223e151f99a4f9f0690dca41414773390fc824184ac833e1"
-dependencies = [
- "cfg-if",
- "dirs-sys-next",
-]
-
-[[package]]
-name = "dirs-sys-next"
-version = "0.1.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4ebda144c4fe02d1f7ea1a7d9641b6fc6b580adcfa024ae48797ecdeb6825b4d"
-dependencies = [
- "libc",
- "redox_users",
- "winapi",
-]
-
-[[package]]
-name = "dragonball"
-version = "0.1.0"
-dependencies = [
- "arc-swap",
- "bytes",
- "crossbeam-channel",
- "dbs-address-space",
- "dbs-allocator",
- "dbs-arch",
- "dbs-boot",
- "dbs-device",
- "dbs-interrupt",
- "dbs-legacy-devices",
- "dbs-upcall",
- "dbs-utils",
- "dbs-virtio-devices",
- "kvm-bindings",
- "kvm-ioctls",
- "lazy_static",
- "libc",
- "linux-loader",
- "log",
- "nix 0.24.3",
- "seccompiler",
- "serde",
- "serde_derive",
- "serde_json",
- "slog",
- "slog-async",
- "slog-scope",
- "slog-term",
- "test-utils",
- "thiserror",
- "virtio-queue",
- "vm-memory",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "epoll"
-version = "4.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "20df693c700404f7e19d4d6fae6b15215d2913c27955d2b9d6f2c0f537511cd0"
-dependencies = [
- "bitflags",
- "libc",
-]
-
-[[package]]
-name = "errno"
-version = "0.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4bcfec3a70f97c962c307b2d2c56e358cf1d00b558d74262b5f929ee8cc7e73a"
-dependencies = [
- "errno-dragonfly",
- "libc",
- "windows-sys 0.48.0",
-]
-
-[[package]]
-name = "errno-dragonfly"
-version = "0.1.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aa68f1b12764fab894d2755d2518754e71b4fd80ecfb822714a1206c2aab39bf"
-dependencies = [
- "cc",
- "libc",
-]
-
-[[package]]
-name = "event-manager"
-version = "0.2.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "377fa591135fbe23396a18e2655a6d5481bf7c5823cdfa3cc81b01a229cbe640"
-dependencies = [
- "libc",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "filetime"
-version = "0.2.20"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8a3de6e8d11b22ff9edc6d916f890800597d60f8b2da1caf2955c274638d6412"
-dependencies = [
- "cfg-if",
- "libc",
- "redox_syscall",
- "windows-sys 0.45.0",
-]
-
-[[package]]
-name = "flate2"
-version = "1.0.25"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a8a2db397cb1c8772f31494cb8917e48cd1e64f0fa7efac59fbd741a0a8ce841"
-dependencies = [
- "crc32fast",
- "libz-sys",
- "miniz_oxide",
-]
-
-[[package]]
-name = "foreign-types"
-version = "0.3.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1"
-dependencies = [
- "foreign-types-shared",
-]
-
-[[package]]
-name = "foreign-types-shared"
-version = "0.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
-
-[[package]]
-name = "fuse-backend-rs"
-version = "0.10.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dc24820b14267bec37fa87f5c2a32b5f1c5405b8c60cc3aa77afd481bd2628a6"
-dependencies = [
- "arc-swap",
- "bitflags",
- "caps",
- "core-foundation-sys",
- "lazy_static",
- "libc",
- "log",
- "mio",
- "nix 0.24.3",
- "tokio-uring",
- "virtio-queue",
- "vm-memory",
- "vmm-sys-util 0.10.0",
-]
-
-[[package]]
-name = "generic-array"
-version = "0.14.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bff49e947297f3312447abdca79f45f4738097cc82b06e72054d2223f601f1b9"
-dependencies = [
- "typenum",
- "version_check",
-]
-
-[[package]]
-name = "getrandom"
-version = "0.2.9"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c85e1d9ab2eadba7e5040d4e09cbd6d072b76a557ad64e797c2cb9d4da21d7e4"
-dependencies = [
- "cfg-if",
- "libc",
- "wasi",
-]
-
-[[package]]
-name = "gimli"
-version = "0.27.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ad0a93d233ebf96623465aad4046a8d3aa4da22d4f4beba5388838c8a434bbb4"
-
-[[package]]
-name = "hermit-abi"
-version = "0.1.19"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "hermit-abi"
-version = "0.2.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ee512640fe35acbfb4bb779db6f0d80704c2cacfa2e39b601ef3e3f47d1ae4c7"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "hex"
-version = "0.4.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
-
-[[package]]
-name = "httpdate"
-version = "1.0.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c4a1e36c821dbe04574f602848a19f742f4fb3c98d40449f11bcad18d6b17421"
-
-[[package]]
-name = "io-lifetimes"
-version = "1.0.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1abeb7a0dd0f8181267ff8adc397075586500b81b28a73e8a0208b00fc170fb3"
-dependencies = [
- "libc",
- "windows-sys 0.45.0",
-]
-
-[[package]]
-name = "io-uring"
-version = "0.5.13"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dd1e1a01cfb924fd8c5c43b6827965db394f5a3a16c599ce03452266e1cf984c"
-dependencies = [
- "bitflags",
- "libc",
-]
-
-[[package]]
-name = "itoa"
-version = "1.0.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fad582f4b9e86b6caa621cabeb0963332d92eea04729ab12892c2533951e6440"
-
-[[package]]
-name = "jobserver"
-version = "0.1.26"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "936cfd212a0155903bcbc060e316fb6cc7cbf2e1907329391ebadc1fe0ce77c2"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "kvm-bindings"
-version = "0.6.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "efe70e65a5b092161d17f5005b66e5eefe7a94a70c332e755036fc4af78c4e79"
-dependencies = [
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "kvm-ioctls"
-version = "0.12.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c3a321cabd827642499c77e27314f388dd83a717a5ca716b86476fb947f73ae4"
-dependencies = [
- "kvm-bindings",
- "libc",
- "vmm-sys-util 0.11.1",
-]
-
-[[package]]
-name = "lazy_static"
-version = "1.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
-
-[[package]]
-name = "leaky-bucket"
-version = "0.12.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "79e0d52231827974ba682f6257fd42a2f79749689f7ca0d763e198a0f7051c91"
-dependencies = [
- "parking_lot",
- "tokio",
- "tracing",
-]
-
-[[package]]
-name = "libc"
-version = "0.2.139"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "201de327520df007757c1f0adce6e827fe8562fbc28bfd9c15571c66ca1f5f79"
-
-[[package]]
-name = "libz-sys"
-version = "1.1.8"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9702761c3935f8cc2f101793272e202c72b99da8f4224a19ddcf1279a6450bbf"
-dependencies = [
- "cc",
- "cmake",
- "libc",
- "pkg-config",
- "vcpkg",
-]
-
-[[package]]
-name = "linux-loader"
-version = "0.6.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "62a2f912deca034ec34b0a43a390059ea98daac40e440ebe8bea88f3315fe168"
-dependencies = [
- "vm-memory",
-]
-
-[[package]]
-name = "linux-raw-sys"
-version = "0.1.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f051f77a7c8e6957c0696eac88f26b0117e54f52d3fc682ab19397a8812846a4"
-
-[[package]]
-name = "lock_api"
-version = "0.4.9"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "435011366fe56583b16cf956f9df0095b405b82d76425bc8981c0e22e60ec4df"
-dependencies = [
- "autocfg",
- "scopeguard",
-]
-
-[[package]]
-name = "log"
-version = "0.4.17"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e"
-dependencies = [
- "cfg-if",
-]
-
-[[package]]
-name = "lz4"
-version = "1.24.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7e9e2dd86df36ce760a60f6ff6ad526f7ba1f14ba0356f8254fb6905e6494df1"
-dependencies = [
- "libc",
- "lz4-sys",
-]
-
-[[package]]
-name = "lz4-sys"
-version = "1.9.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "57d27b317e207b10f69f5e75494119e391a96f48861ae870d1da6edac98ca900"
-dependencies = [
- "cc",
- "libc",
-]
-
-[[package]]
-name = "memchr"
-version = "2.5.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2dffe52ecf27772e601905b7522cb4ef790d2cc203488bbd0e2fe85fcb74566d"
-
-[[package]]
-name = "memoffset"
-version = "0.6.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5aa361d4faea93603064a027415f07bd8e1d5c88c9fbf68bf56a285428fd79ce"
-dependencies = [
- "autocfg",
-]
-
-[[package]]
-name = "miniz_oxide"
-version = "0.6.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b275950c28b37e794e8c55d88aeb5e139d0ce23fdbbeda68f8d7174abdf9e8fa"
-dependencies = [
- "adler",
-]
-
-[[package]]
-name = "mio"
-version = "0.8.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5b9d9a46eff5b4ff64b45a9e316a6d1e0bc719ef429cbec4dc630684212bfdf9"
-dependencies = [
- "libc",
- "log",
- "wasi",
- "windows-sys 0.45.0",
-]
-
-[[package]]
-name = "nix"
-version = "0.23.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8f3790c00a0150112de0f4cd161e3d7fc4b2d8a5542ffc35f099a2562aecb35c"
-dependencies = [
- "bitflags",
- "cc",
- "cfg-if",
- "libc",
- "memoffset",
-]
-
-[[package]]
-name = "nix"
-version = "0.24.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fa52e972a9a719cecb6864fb88568781eb706bac2cd1d4f04a648542dbf78069"
-dependencies = [
- "bitflags",
- "cfg-if",
- "libc",
- "memoffset",
-]
-
-[[package]]
-name = "num_cpus"
-version = "1.15.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0fac9e2da13b5eb447a6ce3d392f23a29d8694bff781bf03a16cd9ac8697593b"
-dependencies = [
- "hermit-abi 0.2.6",
- "libc",
-]
-
-[[package]]
-name = "num_threads"
-version = "0.1.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2819ce041d2ee131036f4fc9d6ae7ae125a3a40e97ba64d04fe799ad9dabbb44"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "nydus-api"
-version = "0.3.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "33a6ca41dd10813e3d29397550fbb0f15ad149381f312e04659d39e0adcf2002"
-dependencies = [
- "backtrace",
- "libc",
- "log",
- "serde",
- "serde_json",
- "toml",
-]
-
-[[package]]
-name = "nydus-rafs"
-version = "0.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ed21e44a99472850d2afc4fb07427ed46d4e6a8b1cce28b42bd689319e45076d"
-dependencies = [
- "anyhow",
- "arc-swap",
- "bitflags",
- "fuse-backend-rs",
- "lazy_static",
- "libc",
- "log",
- "nix 0.24.3",
- "nydus-api",
- "nydus-storage",
- "nydus-utils",
- "serde",
- "serde_json",
- "vm-memory",
-]
-
-[[package]]
-name = "nydus-storage"
-version = "0.6.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9591fbee1875895bf1f765656695d0be6887fe65372fbf4924b8b3959bd61375"
-dependencies = [
- "arc-swap",
- "bitflags",
- "fuse-backend-rs",
- "hex",
- "lazy_static",
- "leaky-bucket",
- "libc",
- "log",
- "nix 0.24.3",
- "nydus-api",
- "nydus-utils",
- "serde",
- "serde_json",
- "tar",
- "tokio",
- "vm-memory",
-]
-
-[[package]]
-name = "nydus-utils"
-version = "0.4.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fe8b9269e3a370682f272a1b2cac4bdaf6d6657f3f6966560c4fedab36548362"
-dependencies = [
- "blake3",
- "flate2",
- "httpdate",
- "lazy_static",
- "libc",
- "libz-sys",
- "log",
- "lz4",
- "lz4-sys",
- "nix 0.24.3",
- "nydus-api",
- "openssl",
- "serde",
- "serde_json",
- "sha2",
- "tokio",
- "zstd",
-]
-
-[[package]]
-name = "object"
-version = "0.30.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ea86265d3d3dcb6a27fc51bd29a4bf387fae9d2986b823079d4986af253eb439"
-dependencies = [
- "memchr",
-]
-
-[[package]]
-name = "once_cell"
-version = "1.17.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b7e5500299e16ebb147ae15a00a942af264cf3688f47923b8fc2cd5858f23ad3"
-
-[[package]]
-name = "openssl"
-version = "0.10.55"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "345df152bc43501c5eb9e4654ff05f794effb78d4efe3d53abc158baddc0703d"
-dependencies = [
- "bitflags",
- "cfg-if",
- "foreign-types",
- "libc",
- "once_cell",
- "openssl-macros",
- "openssl-sys",
-]
-
-[[package]]
-name = "openssl-macros"
-version = "0.1.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 2.0.26",
-]
-
-[[package]]
-name = "openssl-src"
-version = "111.26.0+1.1.1u"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "efc62c9f12b22b8f5208c23a7200a442b2e5999f8bdf80233852122b5a4f6f37"
-dependencies = [
- "cc",
-]
-
-[[package]]
-name = "openssl-sys"
-version = "0.9.90"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "374533b0e45f3a7ced10fcaeccca020e66656bc03dac384f852e4e5a7a8104a6"
-dependencies = [
- "cc",
- "libc",
- "openssl-src",
- "pkg-config",
- "vcpkg",
-]
-
-[[package]]
-name = "parking_lot"
-version = "0.12.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f"
-dependencies = [
- "lock_api",
- "parking_lot_core",
-]
-
-[[package]]
-name = "parking_lot_core"
-version = "0.9.7"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9069cbb9f99e3a5083476ccb29ceb1de18b9118cafa53e90c9551235de2b9521"
-dependencies = [
- "cfg-if",
- "libc",
- "redox_syscall",
- "smallvec",
- "windows-sys 0.45.0",
-]
-
-[[package]]
-name = "pin-project-lite"
-version = "0.2.9"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e0a7ae3ac2f1173085d398531c705756c94a4c56843785df85a60c1a0afac116"
-
-[[package]]
-name = "pkg-config"
-version = "0.3.26"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6ac9a59f73473f1b8d852421e59e64809f025994837ef743615c6d0c5b305160"
-
-[[package]]
-name = "proc-macro2"
-version = "1.0.66"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "18fb31db3f9bddb2ea821cde30a9f70117e3f119938b5ee630b7403aa6e2ead9"
-dependencies = [
- "unicode-ident",
-]
-
-[[package]]
-name = "quote"
-version = "1.0.31"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5fe8a65d69dd0808184ebb5f836ab526bb259db23c657efa38711b1072ee47f0"
-dependencies = [
- "proc-macro2",
-]
-
-[[package]]
-name = "redox_syscall"
-version = "0.2.16"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a"
-dependencies = [
- "bitflags",
-]
-
-[[package]]
-name = "redox_users"
-version = "0.4.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b033d837a7cf162d7993aded9304e30a83213c648b6e389db233191f891e5c2b"
-dependencies = [
- "getrandom",
- "redox_syscall",
- "thiserror",
-]
-
-[[package]]
-name = "rlimit"
-version = "0.7.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "347703a5ae47adf1e693144157be231dde38c72bd485925cae7407ad3e52480b"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "rustc-demangle"
-version = "0.1.21"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7ef03e0a2b150c7a90d01faf6254c9c48a41e95fb2a8c2ac1c6f0d2b9aefc342"
-
-[[package]]
-name = "rustix"
-version = "0.36.14"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "14e4d67015953998ad0eb82887a0eb0129e18a7e2f3b7b0f6c422fddcd503d62"
-dependencies = [
- "bitflags",
- "errno",
- "io-lifetimes",
- "libc",
- "linux-raw-sys",
- "windows-sys 0.45.0",
-]
-
-[[package]]
-name = "rustversion"
-version = "1.0.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5583e89e108996506031660fe09baa5011b9dd0341b89029313006d1fb508d70"
-
-[[package]]
-name = "ryu"
-version = "1.0.12"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b4b9743ed687d4b4bcedf9ff5eaa7398495ae14e61cba0a295704edbc7decde"
-
-[[package]]
-name = "scoped-tls"
-version = "1.0.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294"
-
-[[package]]
-name = "scopeguard"
-version = "1.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
-
-[[package]]
-name = "seccompiler"
-version = "0.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e01d1292a1131b22ccea49f30bd106f1238b5ddeec1a98d39268dcc31d540e68"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "sendfd"
-version = "0.4.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "604b71b8fc267e13bb3023a2c901126c8f349393666a6d98ac1ae5729b701798"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "serde"
-version = "1.0.156"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "314b5b092c0ade17c00142951e50ced110ec27cea304b1037c6969246c2469a4"
-dependencies = [
- "serde_derive",
-]
-
-[[package]]
-name = "serde_derive"
-version = "1.0.156"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d7e29c4601e36bcec74a223228dce795f4cd3616341a4af93520ca1a837c087d"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 1.0.109",
-]
-
-[[package]]
-name = "serde_json"
-version = "1.0.96"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "057d394a50403bcac12672b2b18fb387ab6d289d957dab67dd201875391e52f1"
-dependencies = [
- "itoa",
- "ryu",
- "serde",
-]
-
-[[package]]
-name = "sha2"
-version = "0.10.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "82e6b795fe2e3b1e845bafcb27aa35405c4d47cdfc92af5fc8d3002f76cebdc0"
-dependencies = [
- "cfg-if",
- "cpufeatures",
- "digest",
-]
-
-[[package]]
-name = "slab"
-version = "0.4.8"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6528351c9bc8ab22353f9d776db39a20288e8d6c37ef8cfe3317cf875eecfc2d"
-dependencies = [
- "autocfg",
-]
-
-[[package]]
-name = "slog"
-version = "2.7.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8347046d4ebd943127157b94d63abb990fcf729dc4e9978927fdf4ac3c998d06"
-
-[[package]]
-name = "slog-async"
-version = "2.7.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "766c59b252e62a34651412870ff55d8c4e6d04df19b43eecb2703e417b097ffe"
-dependencies = [
- "crossbeam-channel",
- "slog",
- "take_mut",
- "thread_local",
-]
-
-[[package]]
-name = "slog-scope"
-version = "4.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2f95a4b4c3274cd2869549da82b57ccc930859bdbf5bcea0424bc5f140b3c786"
-dependencies = [
- "arc-swap",
- "lazy_static",
- "slog",
-]
-
-[[package]]
-name = "slog-term"
-version = "2.9.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "87d29185c55b7b258b4f120eab00f48557d4d9bc814f41713f449d35b0f8977c"
-dependencies = [
- "atty",
- "slog",
- "term",
- "thread_local",
- "time",
-]
-
-[[package]]
-name = "smallvec"
-version = "1.10.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a507befe795404456341dfab10cef66ead4c041f62b8b11bbb92bffe5d0953e0"
-
-[[package]]
-name = "socket2"
-version = "0.4.7"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "02e2d2db9033d13a1567121ddd7a095ee144db4e1ca1b1bda3419bc0da294ebd"
-dependencies = [
- "libc",
- "winapi",
-]
-
-[[package]]
-name = "subtle"
-version = "2.4.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6bdef32e8150c2a081110b42772ffe7d7c9032b606bc226c8260fd97e0976601"
-
-[[package]]
-name = "syn"
-version = "1.0.109"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
-dependencies = [
- "proc-macro2",
- "quote",
- "unicode-ident",
-]
-
-[[package]]
-name = "syn"
-version = "2.0.26"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "45c3457aacde3c65315de5031ec191ce46604304d2446e803d71ade03308d970"
-dependencies = [
- "proc-macro2",
- "quote",
- "unicode-ident",
-]
-
-[[package]]
-name = "take_mut"
-version = "0.2.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f764005d11ee5f36500a149ace24e00e3da98b0158b3e2d53a7495660d3f4d60"
-
-[[package]]
-name = "tar"
-version = "0.4.38"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4b55807c0344e1e6c04d7c965f5289c39a8d94ae23ed5c0b57aabac549f871c6"
-dependencies = [
- "filetime",
- "libc",
- "xattr",
-]
-
-[[package]]
-name = "term"
-version = "0.7.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c59df8ac95d96ff9bede18eb7300b0fda5e5d8d90960e76f8e14ae765eedbf1f"
-dependencies = [
- "dirs-next",
- "rustversion",
- "winapi",
-]
-
-[[package]]
-name = "test-utils"
-version = "0.1.0"
-dependencies = [
- "nix 0.24.3",
-]
-
-[[package]]
-name = "thiserror"
-version = "1.0.38"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6a9cd18aa97d5c45c6603caea1da6628790b37f7a34b6ca89522331c5180fed0"
-dependencies = [
- "thiserror-impl",
-]
-
-[[package]]
-name = "thiserror-impl"
-version = "1.0.38"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1fb327af4685e4d03fa8cbcf1716380da910eeb2bb8be417e7f9fd3fb164f36f"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 1.0.109",
-]
-
-[[package]]
-name = "thread_local"
-version = "1.1.7"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3fdd6f064ccff2d6567adcb3873ca630700f00b5ad3f060c25b5dcfd9a4ce152"
-dependencies = [
- "cfg-if",
- "once_cell",
-]
-
-[[package]]
-name = "threadpool"
-version = "1.8.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d050e60b33d41c19108b32cea32164033a9013fe3b46cbd4457559bfbf77afaa"
-dependencies = [
- "num_cpus",
-]
-
-[[package]]
-name = "time"
-version = "0.3.20"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cd0cbfecb4d19b5ea75bb31ad904eb5b9fa13f21079c3b92017ebdf4999a5890"
-dependencies = [
- "itoa",
- "libc",
- "num_threads",
- "serde",
- "time-core",
- "time-macros",
-]
-
-[[package]]
-name = "time-core"
-version = "0.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2e153e1f1acaef8acc537e68b44906d2db6436e2b35ac2c6b42640fff91f00fd"
-
-[[package]]
-name = "time-macros"
-version = "0.2.8"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd80a657e71da814b8e5d60d3374fc6d35045062245d80224748ae522dd76f36"
-dependencies = [
- "time-core",
-]
-
-[[package]]
-name = "timerfd"
-version = "1.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0664936efa25f2bbe03ca25b62c50f5f492abec07e59d6dcf45131014b33483f"
-dependencies = [
- "rustix",
-]
-
-[[package]]
-name = "tokio"
-version = "1.26.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "03201d01c3c27a29c8a5cee5b55a93ddae1ccf6f08f65365c2c918f8c1b76f64"
-dependencies = [
- "autocfg",
- "libc",
- "mio",
- "num_cpus",
- "pin-project-lite",
- "socket2",
- "tokio-macros",
- "windows-sys 0.45.0",
-]
-
-[[package]]
-name = "tokio-macros"
-version = "1.8.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d266c00fde287f55d3f1c3e96c500c362a2b8c695076ec180f27918820bc6df8"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 1.0.109",
-]
-
-[[package]]
-name = "tokio-uring"
-version = "0.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0d5e02bb137e030b3a547c65a3bd2f1836d66a97369fdcc69034002b10e155ef"
-dependencies = [
- "io-uring",
- "libc",
- "scoped-tls",
- "slab",
- "socket2",
- "tokio",
-]
-
-[[package]]
-name = "toml"
-version = "0.5.11"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f4f7f0dd8d50a853a531c426359045b1998f04219d88799810762cd4ad314234"
-dependencies = [
- "serde",
-]
-
-[[package]]
-name = "tracing"
-version = "0.1.37"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8ce8c33a8d48bd45d624a6e523445fd21ec13d3653cd51f681abf67418f54eb8"
-dependencies = [
- "cfg-if",
- "pin-project-lite",
- "tracing-attributes",
- "tracing-core",
-]
-
-[[package]]
-name = "tracing-attributes"
-version = "0.1.23"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4017f8f45139870ca7e672686113917c71c7a6e02d4924eda67186083c03081a"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn 1.0.109",
-]
-
-[[package]]
-name = "tracing-core"
-version = "0.1.30"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "24eb03ba0eab1fd845050058ce5e616558e8f8d8fca633e6b163fe25c797213a"
-dependencies = [
- "once_cell",
-]
-
-[[package]]
-name = "typenum"
-version = "1.16.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "497961ef93d974e23eb6f433eb5fe1b7930b659f06d12dec6fc44a8f554c0bba"
-
-[[package]]
-name = "unicode-ident"
-version = "1.0.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "84a22b9f218b40614adcb3f4ff08b703773ad44fa9423e4e0d346d5db86e4ebc"
-
-[[package]]
-name = "vcpkg"
-version = "0.2.15"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
-
-[[package]]
-name = "version_check"
-version = "0.9.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "49874b5167b65d7193b8aba1567f5c7d93d001cafc34600cee003eda787e483f"
-
-[[package]]
-name = "virtio-bindings"
-version = "0.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3ff512178285488516ed85f15b5d0113a7cdb89e9e8a760b269ae4f02b84bd6b"
-
-[[package]]
-name = "virtio-queue"
-version = "0.6.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "435dd49c7b38419729afd43675850c7b5dc4728f2fabd70c7a9079a331e4f8c6"
-dependencies = [
- "log",
- "virtio-bindings",
- "vm-memory",
- "vmm-sys-util 0.10.0",
-]
-
-[[package]]
-name = "vm-fdt"
-version = "0.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f43fb5a6bd1a7d423ad72802801036719b7546cf847a103f8fe4575f5b0d45a6"
-
-[[package]]
-name = "vm-memory"
-version = "0.9.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "583f213899e8a5eea23d9c507252d4bed5bc88f0ecbe0783262f80034630744b"
-dependencies = [
- "arc-swap",
- "libc",
- "winapi",
-]
-
-[[package]]
-name = "vm-superio"
-version = "0.5.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a4b5231d334edbc03b22704caa1a022e4c07491d6df736593f26094df8b04a51"
-
-[[package]]
-name = "vmm-sys-util"
-version = "0.10.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08604d7be03eb26e33b3cee3ed4aef2bf550b305d1cca60e84da5d28d3790b62"
-dependencies = [
- "bitflags",
- "libc",
-]
-
-[[package]]
-name = "vmm-sys-util"
-version = "0.11.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dd64fe09d8e880e600c324e7d664760a17f56e9672b7495a86381b49e4f72f46"
-dependencies = [
- "bitflags",
- "libc",
-]
-
-[[package]]
-name = "wasi"
-version = "0.11.0+wasi-snapshot-preview1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
-
-[[package]]
-name = "winapi"
-version = "0.3.9"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
-dependencies = [
- "winapi-i686-pc-windows-gnu",
- "winapi-x86_64-pc-windows-gnu",
-]
-
-[[package]]
-name = "winapi-i686-pc-windows-gnu"
-version = "0.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
-
-[[package]]
-name = "winapi-x86_64-pc-windows-gnu"
-version = "0.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
-
-[[package]]
-name = "windows-sys"
-version = "0.45.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0"
-dependencies = [
- "windows-targets 0.42.1",
-]
-
-[[package]]
-name = "windows-sys"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9"
-dependencies = [
- "windows-targets 0.48.0",
-]
-
-[[package]]
-name = "windows-targets"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8e2522491fbfcd58cc84d47aeb2958948c4b8982e9a2d8a2a35bbaed431390e7"
-dependencies = [
- "windows_aarch64_gnullvm 0.42.1",
- "windows_aarch64_msvc 0.42.1",
- "windows_i686_gnu 0.42.1",
- "windows_i686_msvc 0.42.1",
- "windows_x86_64_gnu 0.42.1",
- "windows_x86_64_gnullvm 0.42.1",
- "windows_x86_64_msvc 0.42.1",
-]
-
-[[package]]
-name = "windows-targets"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b1eb6f0cd7c80c79759c929114ef071b87354ce476d9d94271031c0497adfd5"
-dependencies = [
- "windows_aarch64_gnullvm 0.48.0",
- "windows_aarch64_msvc 0.48.0",
- "windows_i686_gnu 0.48.0",
- "windows_i686_msvc 0.48.0",
- "windows_x86_64_gnu 0.48.0",
- "windows_x86_64_gnullvm 0.48.0",
- "windows_x86_64_msvc 0.48.0",
-]
-
-[[package]]
-name = "windows_aarch64_gnullvm"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8c9864e83243fdec7fc9c5444389dcbbfd258f745e7853198f365e3c4968a608"
-
-[[package]]
-name = "windows_aarch64_gnullvm"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "91ae572e1b79dba883e0d315474df7305d12f569b400fcf90581b06062f7e1bc"
-
-[[package]]
-name = "windows_aarch64_msvc"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4c8b1b673ffc16c47a9ff48570a9d85e25d265735c503681332589af6253c6c7"
-
-[[package]]
-name = "windows_aarch64_msvc"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b2ef27e0d7bdfcfc7b868b317c1d32c641a6fe4629c171b8928c7b08d98d7cf3"
-
-[[package]]
-name = "windows_i686_gnu"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "de3887528ad530ba7bdbb1faa8275ec7a1155a45ffa57c37993960277145d640"
-
-[[package]]
-name = "windows_i686_gnu"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "622a1962a7db830d6fd0a69683c80a18fda201879f0f447f065a3b7467daa241"
-
-[[package]]
-name = "windows_i686_msvc"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bf4d1122317eddd6ff351aa852118a2418ad4214e6613a50e0191f7004372605"
-
-[[package]]
-name = "windows_i686_msvc"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4542c6e364ce21bf45d69fdd2a8e455fa38d316158cfd43b3ac1c5b1b19f8e00"
-
-[[package]]
-name = "windows_x86_64_gnu"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c1040f221285e17ebccbc2591ffdc2d44ee1f9186324dd3e84e99ac68d699c45"
-
-[[package]]
-name = "windows_x86_64_gnu"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ca2b8a661f7628cbd23440e50b05d705db3686f894fc9580820623656af974b1"
-
-[[package]]
-name = "windows_x86_64_gnullvm"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "628bfdf232daa22b0d64fdb62b09fcc36bb01f05a3939e20ab73aaf9470d0463"
-
-[[package]]
-name = "windows_x86_64_gnullvm"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7896dbc1f41e08872e9d5e8f8baa8fdd2677f29468c4e156210174edc7f7b953"
-
-[[package]]
-name = "windows_x86_64_msvc"
-version = "0.42.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "447660ad36a13288b1db4d4248e857b510e8c3a225c822ba4fb748c0aafecffd"
-
-[[package]]
-name = "windows_x86_64_msvc"
-version = "0.48.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"
-
-[[package]]
-name = "xattr"
-version = "0.2.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6d1526bbe5aaeb5eb06885f4d987bcdfa5e23187055de9b83fe00156a821fabc"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "zstd"
-version = "0.11.2+zstd.1.5.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "20cc960326ece64f010d2d2107537f26dc589a6573a316bd5b1dba685fa5fde4"
-dependencies = [
- "zstd-safe",
-]
-
-[[package]]
-name = "zstd-safe"
-version = "5.0.2+zstd.1.5.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1d2a5585e04f9eea4b2a3d1eca508c4dee9592a89ef6f450c11719da0726f4db"
-dependencies = [
- "libc",
- "zstd-sys",
-]
-
-[[package]]
-name = "zstd-sys"
-version = "2.0.7+zstd.1.5.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "94509c3ba2fe55294d752b79842c530ccfab760192521df74a081a78d2b3c7f5"
-dependencies = [
- "cc",
- "libc",
- "pkg-config",
-]
diff --git a/src/dragonball/Cargo.toml b/src/dragonball/Cargo.toml
index 08bd34a8f5..f46ab6a6c6 100644
--- a/src/dragonball/Cargo.toml
+++ b/src/dragonball/Cargo.toml
@@ -10,6 +10,7 @@ license = "Apache-2.0"
 edition = "2018"
 
 [dependencies]
+anyhow = "1.0.32"
 arc-swap = "1.5.0"
 bytes = "1.1.0"
 dbs-address-space =  { path = "./src/dbs_address_space" }
@@ -29,6 +30,8 @@ libc = "0.2.39"
 linux-loader = "0.6.0"
 log = "0.4.14"
 nix = "0.24.2"
+procfs = "0.12.0"
+prometheus = { version = "0.13.0", features = ["process"] }
 seccompiler = "0.2.0"
 serde = "1.0.27"
 serde_derive = "1.0.27"
@@ -42,8 +45,8 @@ vm-memory = { version = "0.9.0", features = ["backend-mmap"] }
 crossbeam-channel = "0.5.6"
 
 [dev-dependencies]
-slog-term = "2.9.0"
 slog-async = "2.7.0"
+slog-term = "2.9.0"
 test-utils = { path = "../libs/test-utils" }
 
 [features]
diff --git a/src/dragonball/src/api/v1/vmm_action.rs b/src/dragonball/src/api/v1/vmm_action.rs
index 247daf10d5..5c1c728fe2 100644
--- a/src/dragonball/src/api/v1/vmm_action.rs
+++ b/src/dragonball/src/api/v1/vmm_action.rs
@@ -16,6 +16,8 @@ use crate::event_manager::EventManager;
 use crate::vm::{CpuTopology, KernelConfigInfo, VmConfigInfo};
 use crate::vmm::Vmm;
 
+use crate::hypervisor_metrics::get_hypervisor_metrics;
+
 use self::VmConfigError::*;
 use self::VmmActionError::MachineConfig;
 
@@ -58,6 +60,11 @@ pub enum VmmActionError {
     #[error("Upcall not ready, can't hotplug device.")]
     UpcallServerNotReady,
 
+    /// Error when get prometheus metrics.
+    /// Currently does not distinguish between error types for metrics.
+    #[error("failed to get hypervisor metrics")]
+    GetHypervisorMetrics,
+
     /// The action `ConfigureBootSource` failed either because of bad user input or an internal
     /// error.
     #[error("failed to configure boot source for VM: {0}")]
@@ -135,6 +142,9 @@ pub enum VmmAction {
     /// Get the configuration of the microVM.
     GetVmConfiguration,
 
+    /// Get Prometheus Metrics.
+    GetHypervisorMetrics,
+
     /// Set the microVM configuration (memory & vcpu) using `VmConfig` as input. This
     /// action can only be called before the microVM has booted.
     SetVmConfiguration(VmConfigInfo),
@@ -208,6 +218,8 @@ pub enum VmmData {
     Empty,
     /// The microVM configuration represented by `VmConfigInfo`.
     MachineConfiguration(Box<VmConfigInfo>),
+    /// Prometheus Metrics represented by String.
+    HypervisorMetrics(String),
 }
 
 /// Request data type used to communicate between the API and the VMM.
@@ -262,6 +274,7 @@ impl VmmService {
             VmmAction::GetVmConfiguration => Ok(VmmData::MachineConfiguration(Box::new(
                 self.machine_config.clone(),
             ))),
+            VmmAction::GetHypervisorMetrics => self.get_hypervisor_metrics(),
             VmmAction::SetVmConfiguration(machine_config) => {
                 self.set_vm_configuration(vmm, machine_config)
             }
@@ -381,6 +394,13 @@ impl VmmService {
         Ok(VmmData::Empty)
     }
 
+    /// Get prometheus metrics.
+    fn get_hypervisor_metrics(&self) -> VmmRequestResult {
+        get_hypervisor_metrics()
+            .map_err(|_| VmmActionError::GetHypervisorMetrics)
+            .map(VmmData::HypervisorMetrics)
+    }
+
     /// Set virtual machine configuration.
     pub fn set_vm_configuration(
         &mut self,
diff --git a/src/dragonball/src/hypervisor_metrics.rs b/src/dragonball/src/hypervisor_metrics.rs
new file mode 100644
index 0000000000..fec2fcb588
--- /dev/null
+++ b/src/dragonball/src/hypervisor_metrics.rs
@@ -0,0 +1,102 @@
+// Copyright 2021-2022 Ant Group
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+extern crate procfs;
+
+use crate::metric::{IncMetric, METRICS};
+use anyhow::{anyhow, Result};
+use prometheus::{Encoder, IntCounter, IntGaugeVec, Opts, Registry, TextEncoder};
+use std::sync::Mutex;
+
+const NAMESPACE_KATA_HYPERVISOR: &str = "kata_hypervisor";
+
+lazy_static! {
+    static ref REGISTERED: Mutex<bool> = Mutex::new(false);
+
+    // custom registry
+    static ref REGISTRY: Registry = Registry::new();
+
+    // hypervisor metrics
+    static ref HYPERVISOR_SCRAPE_COUNT: IntCounter =
+    IntCounter::new(format!("{}_{}",NAMESPACE_KATA_HYPERVISOR,"scrape_count"), "Hypervisor metrics scrape count.").unwrap();
+
+    static ref HYPERVISOR_VCPU: IntGaugeVec =
+    IntGaugeVec::new(Opts::new(format!("{}_{}",NAMESPACE_KATA_HYPERVISOR,"vcpu"), "Hypervisor metrics specific to VCPUs' mode of functioning."), &["item"]).unwrap();
+
+    static ref HYPERVISOR_SECCOMP: IntGaugeVec =
+    IntGaugeVec::new(Opts::new(format!("{}_{}",NAMESPACE_KATA_HYPERVISOR,"seccomp"), "Hypervisor metrics for the seccomp filtering."), &["item"]).unwrap();
+
+    static ref HYPERVISOR_SIGNALS: IntGaugeVec =
+    IntGaugeVec::new(Opts::new(format!("{}_{}",NAMESPACE_KATA_HYPERVISOR,"signals"), "Hypervisor metrics related to signals."), &["item"]).unwrap();
+}
+
+/// get prometheus metrics
+pub fn get_hypervisor_metrics() -> Result<String> {
+    let mut registered = REGISTERED
+        .lock()
+        .map_err(|e| anyhow!("failed to check hypervisor metrics register status {:?}", e))?;
+
+    if !(*registered) {
+        register_hypervisor_metrics()?;
+        *registered = true;
+    }
+
+    update_hypervisor_metrics()?;
+
+    // gather all metrics and return as a String
+    let metric_families = REGISTRY.gather();
+
+    let mut buffer = Vec::new();
+    let encoder = TextEncoder::new();
+    encoder.encode(&metric_families, &mut buffer)?;
+
+    Ok(String::from_utf8(buffer)?)
+}
+
+fn register_hypervisor_metrics() -> Result<()> {
+    REGISTRY.register(Box::new(HYPERVISOR_SCRAPE_COUNT.clone()))?;
+    REGISTRY.register(Box::new(HYPERVISOR_VCPU.clone()))?;
+    REGISTRY.register(Box::new(HYPERVISOR_SECCOMP.clone()))?;
+    REGISTRY.register(Box::new(HYPERVISOR_SIGNALS.clone()))?;
+
+    Ok(())
+}
+
+fn update_hypervisor_metrics() -> Result<()> {
+    HYPERVISOR_SCRAPE_COUNT.inc();
+
+    set_intgauge_vec_vcpu(&HYPERVISOR_VCPU);
+    set_intgauge_vec_seccomp(&HYPERVISOR_SECCOMP);
+    set_intgauge_vec_signals(&HYPERVISOR_SIGNALS);
+
+    Ok(())
+}
+
+fn set_intgauge_vec_vcpu(icv: &prometheus::IntGaugeVec) {
+    icv.with_label_values(&["exit_io_in"])
+        .set(METRICS.vcpu.exit_io_in.count() as i64);
+    icv.with_label_values(&["exit_io_out"])
+        .set(METRICS.vcpu.exit_io_out.count() as i64);
+    icv.with_label_values(&["exit_mmio_read"])
+        .set(METRICS.vcpu.exit_mmio_read.count() as i64);
+    icv.with_label_values(&["exit_mmio_write"])
+        .set(METRICS.vcpu.exit_mmio_write.count() as i64);
+    icv.with_label_values(&["failures"])
+        .set(METRICS.vcpu.failures.count() as i64);
+    icv.with_label_values(&["filter_cpuid"])
+        .set(METRICS.vcpu.filter_cpuid.count() as i64);
+}
+
+fn set_intgauge_vec_seccomp(icv: &prometheus::IntGaugeVec) {
+    icv.with_label_values(&["num_faults"])
+        .set(METRICS.seccomp.num_faults.count() as i64);
+}
+
+fn set_intgauge_vec_signals(icv: &prometheus::IntGaugeVec) {
+    icv.with_label_values(&["sigbus"])
+        .set(METRICS.signals.sigbus.count() as i64);
+    icv.with_label_values(&["sigsegv"])
+        .set(METRICS.signals.sigsegv.count() as i64);
+}
diff --git a/src/dragonball/src/lib.rs b/src/dragonball/src/lib.rs
index 5c06853267..26142a4562 100644
--- a/src/dragonball/src/lib.rs
+++ b/src/dragonball/src/lib.rs
@@ -9,6 +9,9 @@
 //TODO: Remove this, after the rest of dragonball has been committed.
 #![allow(dead_code)]
 
+#[macro_use]
+extern crate lazy_static;
+
 /// Address space manager for virtual machines.
 pub mod address_space_manager;
 /// API to handle vmm requests.
@@ -19,6 +22,8 @@ pub mod config_manager;
 pub mod device_manager;
 /// Errors related to Virtual machine manager.
 pub mod error;
+/// Prometheus Metrics.
+pub mod hypervisor_metrics;
 /// KVM operation context for virtual machines.
 pub mod kvm_context;
 /// Metrics system.
diff --git a/src/runtime-rs/Cargo.lock b/src/runtime-rs/Cargo.lock
index 04a7090068..ab090028d1 100644
--- a/src/runtime-rs/Cargo.lock
+++ b/src/runtime-rs/Cargo.lock
@@ -4,12 +4,12 @@ version = 3
 
 [[package]]
 name = "actix-macros"
-version = "0.2.3"
+version = "0.2.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "465a6172cf69b960917811022d8f29bc0b7fa1398bc4f78b3c466673db1213b6"
+checksum = "e01ed3140b2f8d422c68afa1ed2e85d996ea619c988ac834d255db32138655cb"
 dependencies = [
  "quote",
- "syn 1.0.109",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -25,9 +25,9 @@ dependencies = [
 
 [[package]]
 name = "addr2line"
-version = "0.19.0"
+version = "0.20.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97"
+checksum = "f4fa78e18c64fce05e902adecd7a5eed15a5e0a3439f7b0e169f0252214865e3"
 dependencies = [
  "gimli",
 ]
@@ -99,9 +99,9 @@ dependencies = [
 
 [[package]]
 name = "anyhow"
-version = "1.0.71"
+version = "1.0.72"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c7d0618f0e0b7e8ff11427422b64564d5fb0be1940354bfe2e0529b18a9d9b8"
+checksum = "3b13c32d80ecc7ab747b80c3784bce54ee8a7a0cc4fbda9bf4cda2cf6fe90854"
 
 [[package]]
 name = "api_client"
@@ -125,15 +125,15 @@ checksum = "6b4930d2cb77ce62f89ee5d5289b4ac049559b1c45539271f5ed4fdc7db34545"
 
 [[package]]
 name = "arrayvec"
-version = "0.7.2"
+version = "0.7.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
+checksum = "96d30a06541fbafbc7f82ed10c06164cfbd2c401138f6addd8404629c4b16711"
 
 [[package]]
 name = "async-channel"
-version = "1.8.0"
+version = "1.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cf46fee83e5ccffc220104713af3292ff9bc7c64c7de289f66dae8e38d826833"
+checksum = "81953c529336010edd6d8e358f886d9581267795c61b19475b71314bffa46d35"
 dependencies = [
  "concurrent-queue",
  "event-listener",
@@ -149,7 +149,7 @@ dependencies = [
  "async-lock",
  "async-task",
  "concurrent-queue",
- "fastrand",
+ "fastrand 1.9.0",
  "futures-lite",
  "slab",
 ]
@@ -183,7 +183,7 @@ dependencies = [
  "log",
  "parking",
  "polling",
- "rustix",
+ "rustix 0.37.23",
  "slab",
  "socket2",
  "waker-fn",
@@ -232,13 +232,13 @@ checksum = "ecc7ab41815b3c653ccd2978ec3255c81349336702dfdf62ee6f7069b12a3aae"
 
 [[package]]
 name = "async-trait"
-version = "0.1.68"
+version = "0.1.72"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b9ccdd8f2a161be9bd5c023df56f1b2a0bd1d83872ae53b71a84a12c9bf6e842"
+checksum = "cc6dde6e4ed435a4c1ee4e73592f5ba9da2151af10076cc04858746af9352d09"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -272,15 +272,15 @@ checksum = "cc17ab023b4091c10ff099f9deebaeeb59b5189df07e554c4fef042b70745d68"
 
 [[package]]
 name = "backtrace"
-version = "0.3.67"
+version = "0.3.68"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "233d376d6d185f2a3093e58f283f60f880315b6c60075b01f36b3b85154564ca"
+checksum = "4319208da049c43661739c5fade2ba182f09d1dc2299b32298d3a31692b17e12"
 dependencies = [
  "addr2line",
  "cc",
  "cfg-if 1.0.0",
  "libc",
- "miniz_oxide 0.6.2",
+ "miniz_oxide",
  "object",
  "rustc-demangle",
 ]
@@ -298,20 +298,26 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
 
 [[package]]
-name = "bitmask-enum"
-version = "2.1.0"
+name = "bitflags"
+version = "2.3.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd9e32d7420c85055e8107e5b2463c4eeefeaac18b52359fe9f9c08a18f342b2"
+checksum = "630be753d4e58660abd17930c71b647fe46c27ea6b63cc59e1e3851406972e42"
+
+[[package]]
+name = "bitmask-enum"
+version = "2.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "78d456f91b4c1fdebf2698214e599fec3d7f8b46e3140fb254a9ea88c970ab0a"
 dependencies = [
  "quote",
- "syn 1.0.109",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "blake3"
-version = "1.4.0"
+version = "1.4.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "729b71f35bd3fa1a4c86b85d32c8b9069ea7fe14f7a53cfabb65f62d4265b888"
+checksum = "199c42ab6972d92c9f8995f086273d25c42fc0f7b2a1fcefba465c1352d25ba5"
 dependencies = [
  "arrayref",
  "arrayvec",
@@ -349,7 +355,7 @@ dependencies = [
  "async-lock",
  "async-task",
  "atomic-waker",
- "fastrand",
+ "fastrand 1.9.0",
  "futures-lite",
  "log",
 ]
@@ -521,9 +527,9 @@ dependencies = [
 
 [[package]]
 name = "constant_time_eq"
-version = "0.2.6"
+version = "0.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "21a53c0a4d288377e7415b53dcfc3c04da5cdc2cc95c8d5ac178b58f0b861ad6"
+checksum = "f7144d30dcf0fafbce74250a3963025d8d52177934239851c917d29f1df280c2"
 
 [[package]]
 name = "containerd-shim-protos"
@@ -545,9 +551,9 @@ checksum = "e496a50fda8aacccc86d7529e2c1e0892dbd0f898a6b5645b5561b89c3210efa"
 
 [[package]]
 name = "cpufeatures"
-version = "0.2.7"
+version = "0.2.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3e4c1eaa2012c47becbbad2ab175484c2a84d1185b566fb2cc5b8707343dfe58"
+checksum = "a17b76ff3a4162b0b27f354a0c87015ddad39d35f9c0c36607a3bdd175dde1f1"
 dependencies = [
  "libc",
 ]
@@ -579,9 +585,9 @@ dependencies = [
 
 [[package]]
 name = "crossbeam-utils"
-version = "0.8.15"
+version = "0.8.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3c063cd8cc95f5c377ed0d4b49a4b21f632396ff690e8470c29b3359b346984b"
+checksum = "5a22b2d63d4d1dc0b7f1b6b2747dd0088008a9be28b6ddf0b1e7d335e3037294"
 dependencies = [
  "cfg-if 1.0.0",
 ]
@@ -598,12 +604,12 @@ dependencies = [
 
 [[package]]
 name = "dashmap"
-version = "5.4.0"
+version = "5.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "907076dfda823b0b36d2a1bb5f90c96660a5bbcd7729e10727f07858f22c4edc"
+checksum = "6943ae99c34386c84a470c499d3414f66502a41340aa895406e0d2e4a207b91d"
 dependencies = [
  "cfg-if 1.0.0",
- "hashbrown",
+ "hashbrown 0.14.0",
  "lock_api",
  "once_cell",
  "parking_lot_core 0.9.8",
@@ -808,6 +814,7 @@ checksum = "0688c2a7f92e427f44895cd63841bff7b29f8d7a1648b9e7e07a4a365b2e1257"
 name = "dragonball"
 version = "0.1.0"
 dependencies = [
+ "anyhow",
  "arc-swap",
  "bytes 1.4.0",
  "crossbeam-channel",
@@ -828,6 +835,8 @@ dependencies = [
  "linux-loader",
  "log",
  "nix 0.24.3",
+ "procfs 0.12.0",
+ "prometheus",
  "seccompiler",
  "serde",
  "serde_derive",
@@ -842,9 +851,9 @@ dependencies = [
 
 [[package]]
 name = "either"
-version = "1.8.1"
+version = "1.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7fcaabb2fef8c910e7f4c7ce9f67a1283a1715879a7c230ca9d6d1ae31f16d91"
+checksum = "a26ae43d7bcc3b814de94796a5e736d4029efb0ee900c12e2d54c993ad1a1e07"
 
 [[package]]
 name = "epoll"
@@ -852,7 +861,7 @@ version = "4.3.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "20df693c700404f7e19d4d6fae6b15215d2913c27955d2b9d6f2c0f537511cd0"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "libc",
 ]
 
@@ -864,7 +873,7 @@ checksum = "4bcfec3a70f97c962c307b2d2c56e358cf1d00b558d74262b5f929ee8cc7e73a"
 dependencies = [
  "errno-dragonfly",
  "libc",
- "windows-sys",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -913,6 +922,12 @@ dependencies = [
  "instant",
 ]
 
+[[package]]
+name = "fastrand"
+version = "2.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6999dc1837253364c2ebb0704ba97994bd874e8f195d665c50b7548f6ea92764"
+
 [[package]]
 name = "filetime"
 version = "0.2.21"
@@ -922,7 +937,7 @@ dependencies = [
  "cfg-if 1.0.0",
  "libc",
  "redox_syscall 0.2.16",
- "windows-sys",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -939,7 +954,7 @@ checksum = "3b9429470923de8e8cbd4d2dc513535400b4b3fef0319fb5c4e1f520a7bef743"
 dependencies = [
  "crc32fast",
  "libz-sys",
- "miniz_oxide 0.7.1",
+ "miniz_oxide",
 ]
 
 [[package]]
@@ -985,7 +1000,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dc24820b14267bec37fa87f5c2a32b5f1c5405b8c60cc3aa77afd481bd2628a6"
 dependencies = [
  "arc-swap",
- "bitflags",
+ "bitflags 1.3.2",
  "caps",
  "core-foundation-sys",
  "lazy_static",
@@ -1059,7 +1074,7 @@ version = "1.13.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "49a9d51ce47660b1e808d3c990b4709f2f415d928835a17dfd16991515c46bce"
 dependencies = [
- "fastrand",
+ "fastrand 1.9.0",
  "futures-core",
  "futures-io",
  "memchr",
@@ -1076,7 +1091,7 @@ checksum = "89ca545a94061b6365f2c7355b4b32bd20df3ff95f02da9329b34ccc3bd6ee72"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -1143,9 +1158,9 @@ dependencies = [
 
 [[package]]
 name = "gimli"
-version = "0.27.2"
+version = "0.27.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ad0a93d233ebf96623465aad4046a8d3aa4da22d4f4beba5388838c8a434bbb4"
+checksum = "b6c80984affa11d98d1b88b66ac8853f143217b399d3c74116778ff8fdb4ed2e"
 
 [[package]]
 name = "glob"
@@ -1176,9 +1191,9 @@ dependencies = [
 
 [[package]]
 name = "h2"
-version = "0.3.19"
+version = "0.3.20"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d357c7ae988e7d2182f7d7871d0b963962420b0678b0997ce7de72001aeab782"
+checksum = "97ec8491ebaf99c8eaa73058b045fe58073cd6be7f596ac993ced0b0a0c01049"
 dependencies = [
  "bytes 1.4.0",
  "fnv",
@@ -1202,6 +1217,12 @@ dependencies = [
  "ahash",
 ]
 
+[[package]]
+name = "hashbrown"
+version = "0.14.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2c6201b9ff9fd90a5a3bac2e56a830d0caa509576f0e503818ee82c181b3437a"
+
 [[package]]
 name = "headers"
 version = "0.3.8"
@@ -1209,7 +1230,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f3e372db8e5c0d213e0cd0b9be18be2aca3d44cf2fe30a9d46a65581cd454584"
 dependencies = [
  "base64",
- "bitflags",
+ "bitflags 1.3.2",
  "bytes 1.4.0",
  "headers-core",
  "http",
@@ -1253,18 +1274,9 @@ dependencies = [
 
 [[package]]
 name = "hermit-abi"
-version = "0.2.6"
+version = "0.3.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ee512640fe35acbfb4bb779db6f0d80704c2cacfa2e39b601ef3e3f47d1ae4c7"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "hermit-abi"
-version = "0.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fed44880c466736ef9a5c5b5facefb5ed0785676d0c02d612db14e54f0d84286"
+checksum = "443144c8cdadd93ebf52ddb4056d257f5b52c04d3c804e657d19eb73fc33668b"
 
 [[package]]
 name = "hex"
@@ -1308,9 +1320,9 @@ checksum = "c4a1e36c821dbe04574f602848a19f742f4fb3c98d40449f11bcad18d6b17421"
 
 [[package]]
 name = "hyper"
-version = "0.14.26"
+version = "0.14.27"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ab302d72a6f11a3b910431ff93aae7e773078c769f0a3ef15fb9ec692ed147d4"
+checksum = "ffb1cfd654a8219eaef89881fdb3bb3b1cdc5fa75ded05d6933b2b382e395468"
 dependencies = [
  "bytes 1.4.0",
  "futures-channel",
@@ -1419,7 +1431,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99"
 dependencies = [
  "autocfg",
- "hashbrown",
+ "hashbrown 0.12.3",
 ]
 
 [[package]]
@@ -1443,9 +1455,9 @@ version = "1.0.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "eae7b9aee968036d54dce06cebaefd919e4472e753296daccd6d344e3e2df0c2"
 dependencies = [
- "hermit-abi 0.3.1",
+ "hermit-abi 0.3.2",
  "libc",
- "windows-sys",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -1454,7 +1466,7 @@ version = "0.5.13"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dd1e1a01cfb924fd8c5c43b6827965db394f5a3a16c599ce03452266e1cf984c"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "libc",
 ]
 
@@ -1478,9 +1490,9 @@ dependencies = [
 
 [[package]]
 name = "itoa"
-version = "1.0.6"
+version = "1.0.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "453ad9f582a441959e5f0d088b02ce04cfe8d51a8eaf077f12ac6d3e94164ca6"
+checksum = "af150ab688ff2122fcef229be89cb50dd66af9e01a4ff320cc137eecc9bacc38"
 
 [[package]]
 name = "jobserver"
@@ -1493,9 +1505,9 @@ dependencies = [
 
 [[package]]
 name = "js-sys"
-version = "0.3.63"
+version = "0.3.64"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2f37a4a5928311ac501dee68b3c7613a1037d0edb30c8e5427bd832d55d1b790"
+checksum = "c5f195fe497f702db0f318b07fdd68edb16955aed830df8363d837542f8f935a"
 dependencies = [
  "wasm-bindgen",
 ]
@@ -1594,15 +1606,15 @@ dependencies = [
 
 [[package]]
 name = "libc"
-version = "0.2.146"
+version = "0.2.147"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f92be4933c13fd498862a9e02a3055f8a8d9c039ce33db97306fd5a6caa7f29b"
+checksum = "b4668fb0ea861c1df094127ac5f1da3409a82116a4ba74fca2e58ef927159bb3"
 
 [[package]]
 name = "libz-sys"
-version = "1.1.9"
+version = "1.1.12"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "56ee889ecc9568871456d42f603d6a0ce59ff328d291063a45cbdf0036baf6db"
+checksum = "d97137b25e321a73eef1418d1d5d2eda4d77e12813f8e6dead84bc52c5870a7b"
 dependencies = [
  "cc",
  "cmake",
@@ -1620,12 +1632,24 @@ dependencies = [
  "vm-memory",
 ]
 
+[[package]]
+name = "linux-raw-sys"
+version = "0.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f051f77a7c8e6957c0696eac88f26b0117e54f52d3fc682ab19397a8812846a4"
+
 [[package]]
 name = "linux-raw-sys"
 version = "0.3.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ef53942eb7bf7ff43a617b3e2c1c4a5ecf5944a7c1bc12d7ee39bbb15e5c1519"
 
+[[package]]
+name = "linux-raw-sys"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09fc20d2ca12cb9f044c93e3bd6d32d523e6e2ec3db4f7b2939cd99026ecd3f0"
+
 [[package]]
 name = "linux_container"
 version = "0.1.0"
@@ -1718,15 +1742,6 @@ version = "0.3.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
 
-[[package]]
-name = "miniz_oxide"
-version = "0.6.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b275950c28b37e794e8c55d88aeb5e139d0ce23fdbbeda68f8d7174abdf9e8fa"
-dependencies = [
- "adler",
-]
-
 [[package]]
 name = "miniz_oxide"
 version = "0.7.1"
@@ -1745,7 +1760,7 @@ dependencies = [
  "libc",
  "log",
  "wasi 0.11.0+wasi-snapshot-preview1",
- "windows-sys",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -1773,7 +1788,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f5dee5ed749373c298237fe694eb0a51887f4cc1a27370c8464bac4382348f1a"
 dependencies = [
  "anyhow",
- "bitflags",
+ "bitflags 1.3.2",
  "byteorder",
  "libc",
  "netlink-packet-core",
@@ -1836,7 +1851,7 @@ version = "0.23.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8f3790c00a0150112de0f4cd161e3d7fc4b2d8a5542ffc35f099a2562aecb35c"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cc",
  "cfg-if 1.0.0",
  "libc",
@@ -1849,7 +1864,7 @@ version = "0.24.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fa52e972a9a719cecb6864fb88568781eb706bac2cd1d4f04a648542dbf78069"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if 1.0.0",
  "libc",
  "memoffset 0.6.5",
@@ -1862,7 +1877,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f346ff70e7dbfd675fe90590b92d59ef2de15a8779ae305ebcbfd3f0caf59be4"
 dependencies = [
  "autocfg",
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if 1.0.0",
  "libc",
  "memoffset 0.6.5",
@@ -1875,7 +1890,7 @@ version = "0.26.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bfdda3d196821d6af13126e40375cdf7da646a96114af134d5f417a9a1dc8e1a"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if 1.0.0",
  "libc",
  "memoffset 0.7.1",
@@ -1895,20 +1910,20 @@ dependencies = [
 
 [[package]]
 name = "num-traits"
-version = "0.2.15"
+version = "0.2.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "578ede34cf02f8924ab9447f50c28075b4d3e5b269972345e7e0372b38c6cdcd"
+checksum = "f30b0abd723be7e2ffca1272140fac1a2f084c77ec3e123c192b66af1ee9e6c2"
 dependencies = [
  "autocfg",
 ]
 
 [[package]]
 name = "num_cpus"
-version = "1.15.0"
+version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0fac9e2da13b5eb447a6ce3d392f23a29d8694bff781bf03a16cd9ac8697593b"
+checksum = "4161fcb6d602d4d2081af7c3a45852d875a03dd337a6bfdd6e06407b61342a43"
 dependencies = [
- "hermit-abi 0.2.6",
+ "hermit-abi 0.3.2",
  "libc",
 ]
 
@@ -1943,7 +1958,7 @@ checksum = "ed21e44a99472850d2afc4fb07427ed46d4e6a8b1cce28b42bd689319e45076d"
 dependencies = [
  "anyhow",
  "arc-swap",
- "bitflags",
+ "bitflags 1.3.2",
  "fuse-backend-rs",
  "lazy_static",
  "libc",
@@ -1964,7 +1979,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "9591fbee1875895bf1f765656695d0be6887fe65372fbf4924b8b3959bd61375"
 dependencies = [
  "arc-swap",
- "bitflags",
+ "bitflags 1.3.2",
  "fuse-backend-rs",
  "hex",
  "lazy_static",
@@ -2001,16 +2016,16 @@ dependencies = [
  "openssl",
  "serde",
  "serde_json",
- "sha2 0.10.6",
+ "sha2 0.10.7",
  "tokio",
  "zstd",
 ]
 
 [[package]]
 name = "object"
-version = "0.30.4"
+version = "0.31.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "03b4680b86d9cfafba8fc491dc9b6df26b68cf40e9e6cd73909194759a63c385"
+checksum = "8bda667d9f2b5051b8833f59f3bf748b28ef54f850f4fcb389a252aa383866d1"
 dependencies = [
  "memchr",
 ]
@@ -2043,7 +2058,7 @@ version = "0.10.55"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "345df152bc43501c5eb9e4654ff05f794effb78d4efe3d53abc158baddc0703d"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if 1.0.0",
  "foreign-types",
  "libc",
@@ -2060,7 +2075,7 @@ checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -2193,7 +2208,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ccd746e37177e1711c20dd619a1620f34f5c8b569c53590a72dedd5344d8924a"
 dependencies = [
  "dlv-list",
- "hashbrown",
+ "hashbrown 0.12.3",
 ]
 
 [[package]]
@@ -2253,14 +2268,14 @@ dependencies = [
  "libc",
  "redox_syscall 0.3.5",
  "smallvec",
- "windows-targets",
+ "windows-targets 0.48.1",
 ]
 
 [[package]]
 name = "paste"
-version = "1.0.12"
+version = "1.0.14"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9f746c4065a8fa3fe23974dd82f15431cc8d40779821001404d10d2e79ca7d79"
+checksum = "de3145af08024dea9fa9914f381a17b8fc6034dfb00f3a84013f7ff43f29ed4c"
 
 [[package]]
 name = "path-clean"
@@ -2301,29 +2316,29 @@ dependencies = [
 
 [[package]]
 name = "pin-project"
-version = "1.1.0"
+version = "1.1.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c95a7476719eab1e366eaf73d0260af3021184f18177925b07f54b30089ceead"
+checksum = "030ad2bc4db10a8944cb0d837f158bdfec4d4a4873ab701a95046770d11f8842"
 dependencies = [
  "pin-project-internal",
 ]
 
 [[package]]
 name = "pin-project-internal"
-version = "1.1.0"
+version = "1.1.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "39407670928234ebc5e6e580247dd567ad73a3578460c5990f9503df207e8f07"
+checksum = "ec2e072ecce94ec471b13398d5402c188e76ac03cf74dd1a975161b23a3f6d9c"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "pin-project-lite"
-version = "0.2.9"
+version = "0.2.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e0a7ae3ac2f1173085d398531c705756c94a4c56843785df85a60c1a0afac116"
+checksum = "4c40d25201921e5ff0c862a505c6557ea88568a4e3ace775ab55e93f2f4f9d57"
 
 [[package]]
 name = "pin-utils"
@@ -2344,13 +2359,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4b2d323e8ca7996b3e23126511a523f7e62924d93ecd5ae73b333815b0eb3dce"
 dependencies = [
  "autocfg",
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if 1.0.0",
  "concurrent-queue",
  "libc",
  "log",
  "pin-project-lite",
- "windows-sys",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -2361,13 +2376,58 @@ checksum = "5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de"
 
 [[package]]
 name = "proc-macro2"
-version = "1.0.60"
+version = "1.0.66"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dec2b086b7a862cf4de201096214fa870344cf922b2b30c167badb3af3195406"
+checksum = "18fb31db3f9bddb2ea821cde30a9f70117e3f119938b5ee630b7403aa6e2ead9"
 dependencies = [
  "unicode-ident",
 ]
 
+[[package]]
+name = "procfs"
+version = "0.12.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0941606b9934e2d98a3677759a971756eb821f75764d0e0d26946d08e74d9104"
+dependencies = [
+ "bitflags 1.3.2",
+ "byteorder",
+ "chrono",
+ "flate2",
+ "hex",
+ "lazy_static",
+ "libc",
+]
+
+[[package]]
+name = "procfs"
+version = "0.14.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b1de8dacb0873f77e6aefc6d71e044761fcc68060290f5b1089fcdf84626bb69"
+dependencies = [
+ "bitflags 1.3.2",
+ "byteorder",
+ "hex",
+ "lazy_static",
+ "rustix 0.36.15",
+]
+
+[[package]]
+name = "prometheus"
+version = "0.13.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "449811d15fbdf5ceb5c1144416066429cf82316e2ec8ce0c1f6f8a02e7bbcf8c"
+dependencies = [
+ "cfg-if 1.0.0",
+ "fnv",
+ "lazy_static",
+ "libc",
+ "memchr",
+ "parking_lot 0.12.1",
+ "procfs 0.14.2",
+ "protobuf 2.28.0",
+ "thiserror",
+]
+
 [[package]]
 name = "prost"
 version = "0.8.0"
@@ -2498,9 +2558,9 @@ dependencies = [
 
 [[package]]
 name = "quote"
-version = "1.0.28"
+version = "1.0.32"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1b9ab9c7eadfd8df19006f1cf1a4aed13540ed5cbc047010ece5826e10825488"
+checksum = "50f3b39ccfb720540debaa0164757101c08ecb8d326b15358ce76a62c7e85965"
 dependencies = [
  "proc-macro2",
 ]
@@ -2629,7 +2689,7 @@ version = "0.2.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
 ]
 
 [[package]]
@@ -2638,7 +2698,7 @@ version = "0.3.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "567664f262709473930a4bf9e51bf2ebf3348f2e748ccc50dea20646858f8f29"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
 ]
 
 [[package]]
@@ -2654,9 +2714,21 @@ dependencies = [
 
 [[package]]
 name = "regex"
-version = "1.8.4"
+version = "1.9.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d0ab3ca65655bb1e41f2a8c8cd662eb4fb035e67c3f78da1d61dffe89d07300f"
+checksum = "b2eae68fc220f7cf2532e4494aded17545fce192d59cd996e0fe7887f4ceb575"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "39354c10dd07468c2e73926b23bb9c2caca74c5501e38a35da70406f1d923310"
 dependencies = [
  "aho-corasick",
  "memchr",
@@ -2665,9 +2737,9 @@ dependencies = [
 
 [[package]]
 name = "regex-syntax"
-version = "0.7.2"
+version = "0.7.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "436b050e76ed2903236f032a59761c1eb99e1b0aead2c257922771dab1fc8c78"
+checksum = "e5ea92a5b6195c6ef2a0295ea818b312502c6fc94dde986c5553242e18fd4ce2"
 
 [[package]]
 name = "resource"
@@ -2677,7 +2749,7 @@ dependencies = [
  "agent",
  "anyhow",
  "async-trait",
- "bitflags",
+ "bitflags 1.3.2",
  "byte-unit 4.0.19",
  "cgroups-rs",
  "futures 0.3.28",
@@ -2753,6 +2825,8 @@ dependencies = [
  "opentelemetry",
  "opentelemetry-jaeger",
  "persist",
+ "procfs 0.12.0",
+ "prometheus",
  "resource",
  "serde_json",
  "shim-interface",
@@ -2785,29 +2859,56 @@ checksum = "d626bb9dae77e28219937af045c257c28bfd3f69333c512553507f5f9798cb76"
 
 [[package]]
 name = "rustix"
-version = "0.37.20"
+version = "0.36.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b96e891d04aa506a6d1f318d2771bcb1c7dfda84e126660ace067c9b474bb2c0"
+checksum = "c37f1bd5ef1b5422177b7646cba67430579cfe2ace80f284fee876bca52ad941"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "errno",
  "io-lifetimes",
  "libc",
- "linux-raw-sys",
- "windows-sys",
+ "linux-raw-sys 0.1.4",
+ "windows-sys 0.45.0",
+]
+
+[[package]]
+name = "rustix"
+version = "0.37.23"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4d69718bf81c6127a49dc64e44a742e8bb9213c0ff8869a22c308f84c1d4ab06"
+dependencies = [
+ "bitflags 1.3.2",
+ "errno",
+ "io-lifetimes",
+ "libc",
+ "linux-raw-sys 0.3.8",
+ "windows-sys 0.48.0",
+]
+
+[[package]]
+name = "rustix"
+version = "0.38.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0a962918ea88d644592894bc6dc55acc6c0956488adcebbfb6e273506b7fd6e5"
+dependencies = [
+ "bitflags 2.3.3",
+ "errno",
+ "libc",
+ "linux-raw-sys 0.4.3",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
 name = "rustversion"
-version = "1.0.12"
+version = "1.0.14"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4f3208ce4d8448b3f3e7d168a73f5e0c43a61e32930de3bceeccedb388b6bf06"
+checksum = "7ffc183a10b4478d04cbbbfc96d0873219d962dd5accaff2ffbd4ceb7df837f4"
 
 [[package]]
 name = "ryu"
-version = "1.0.13"
+version = "1.0.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f91339c0467de62360649f8d3e185ca8de4224ff281f66000de5eb2a77a79041"
+checksum = "1ad4cc8da4ef723ed60bced201181d83791ad433213d8c24efffda1eec85d741"
 
 [[package]]
 name = "safe-path"
@@ -2833,9 +2934,9 @@ checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294"
 
 [[package]]
 name = "scopeguard"
-version = "1.1.0"
+version = "1.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
+checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
 
 [[package]]
 name = "seccompiler"
@@ -2857,29 +2958,29 @@ dependencies = [
 
 [[package]]
 name = "serde"
-version = "1.0.164"
+version = "1.0.177"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9e8c8cf938e98f769bc164923b06dce91cea1751522f46f8466461af04c9027d"
+checksum = "63ba2516aa6bf82e0b19ca8b50019d52df58455d3cf9bdaf6315225fdd0c560a"
 dependencies = [
  "serde_derive",
 ]
 
 [[package]]
 name = "serde_derive"
-version = "1.0.164"
+version = "1.0.177"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d9735b638ccc51c28bf6914d90a2e9725b377144fc612c49a611fddd1b631d68"
+checksum = "401797fe7833d72109fedec6bfcbe67c0eed9b99772f26eb8afd261f0abc6fd3"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "serde_json"
-version = "1.0.96"
+version = "1.0.104"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "057d394a50403bcac12672b2b18fb387ab6d289d957dab67dd201875391e52f1"
+checksum = "076066c5f1078eac5b722a31827a8832fe108bed65dfa75e233c89f8206e976c"
 dependencies = [
  "itoa",
  "ryu",
@@ -2953,9 +3054,9 @@ dependencies = [
 
 [[package]]
 name = "sha2"
-version = "0.10.6"
+version = "0.10.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "82e6b795fe2e3b1e845bafcb27aa35405c4d47cdfc92af5fc8d3002f76cebdc0"
+checksum = "479fb9d862239e610720565ca91403019f2f00410f1864c5aa7479b950a76ed8"
 dependencies = [
  "cfg-if 1.0.0",
  "cpufeatures",
@@ -3071,7 +3172,7 @@ dependencies = [
  "serde",
  "serde_json",
  "slog",
- "time 0.3.22",
+ "time 0.3.23",
 ]
 
 [[package]]
@@ -3106,14 +3207,14 @@ dependencies = [
  "slog",
  "term",
  "thread_local",
- "time 0.3.22",
+ "time 0.3.23",
 ]
 
 [[package]]
 name = "smallvec"
-version = "1.10.0"
+version = "1.11.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a507befe795404456341dfab10cef66ead4c041f62b8b11bbb92bffe5d0953e0"
+checksum = "62bb4feee49fdd9f707ef802e22365a35de4b7b299de4763d44bfea899442ff9"
 
 [[package]]
 name = "socket2"
@@ -3182,9 +3283,9 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "2.0.18"
+version = "2.0.27"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "32d41677bcbe24c20c52e7c70b0d8db04134c5d1066bf98662e2871ad200ea3e"
+checksum = "b60f673f44a8255b9c8c657daf66a596d435f2da81a555b06dc644d080ba45e0"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -3199,9 +3300,9 @@ checksum = "f764005d11ee5f36500a149ace24e00e3da98b0158b3e2d53a7495660d3f4d60"
 
 [[package]]
 name = "tar"
-version = "0.4.38"
+version = "0.4.39"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4b55807c0344e1e6c04d7c965f5289c39a8d94ae23ed5c0b57aabac549f871c6"
+checksum = "ec96d2ffad078296368d46ff1cb309be1c23c513b4ab0e22a45de0185275ac96"
 dependencies = [
  "filetime",
  "libc",
@@ -3210,16 +3311,15 @@ dependencies = [
 
 [[package]]
 name = "tempfile"
-version = "3.6.0"
+version = "3.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "31c0432476357e58790aaa47a8efb0c5138f137343f3b5f23bd36a27e3b0a6d6"
+checksum = "5486094ee78b2e5038a6382ed7645bc084dc2ec433426ca4c3cb61e2007b8998"
 dependencies = [
- "autocfg",
  "cfg-if 1.0.0",
- "fastrand",
+ "fastrand 2.0.0",
  "redox_syscall 0.3.5",
- "rustix",
- "windows-sys",
+ "rustix 0.38.4",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -3249,22 +3349,22 @@ dependencies = [
 
 [[package]]
 name = "thiserror"
-version = "1.0.40"
+version = "1.0.44"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "978c9a314bd8dc99be594bc3c175faaa9794be04a5a5e153caba6915336cebac"
+checksum = "611040a08a0439f8248d1990b111c95baa9c704c805fa1f62104b39655fd7f90"
 dependencies = [
  "thiserror-impl",
 ]
 
 [[package]]
 name = "thiserror-impl"
-version = "1.0.40"
+version = "1.0.44"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f9456a42c5b0d803c8cd86e73dd7cc9edd429499f37a3550d286d5e86720569f"
+checksum = "090198534930841fab3a5d1bb637cde49e339654e606195f8d9c76eeb081dc96"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -3312,9 +3412,9 @@ dependencies = [
 
 [[package]]
 name = "time"
-version = "0.3.22"
+version = "0.3.23"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ea9e1b3cf1243ae005d9e74085d4d542f3125458f3a81af210d901dcd7411efd"
+checksum = "59e399c068f43a5d116fedaf73b203fa4f9c519f17e2b34f63221d3792f81446"
 dependencies = [
  "itoa",
  "libc",
@@ -3332,9 +3432,9 @@ checksum = "7300fbefb4dadc1af235a9cef3737cea692a9d97e1b9cbcd4ebdae6f8868e6fb"
 
 [[package]]
 name = "time-macros"
-version = "0.2.9"
+version = "0.2.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "372950940a5f07bf38dbe211d7283c9e6d7327df53794992d293e534c733d09b"
+checksum = "96ba15a897f3c86766b757e5ac7221554c6750054d74d5b28844fce5fb36a6c4"
 dependencies = [
  "time-core",
 ]
@@ -3345,7 +3445,7 @@ version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "3d3fd47d83ad0b5c7be2e8db0b9d712901ef6ce5afbcc6f676761004f5104ea2"
 dependencies = [
- "rustix",
+ "rustix 0.37.23",
 ]
 
 [[package]]
@@ -3365,11 +3465,12 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
 
 [[package]]
 name = "tokio"
-version = "1.28.2"
+version = "1.29.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "94d7b1cfd2aa4011f2de74c2c4c63665e27a71006b0a192dcd2710272e73dfa2"
+checksum = "532826ff75199d5833b9d2c5fe410f29235e25704ee5f0ef599fb51c21f4a4da"
 dependencies = [
  "autocfg",
+ "backtrace",
  "bytes 1.4.0",
  "libc",
  "mio",
@@ -3379,7 +3480,7 @@ dependencies = [
  "signal-hook-registry",
  "socket2",
  "tokio-macros",
- "windows-sys",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -3390,7 +3491,7 @@ checksum = "630bdcf245f78637c13ec01ffae6187cca34625e8c63150d424b59e55af2675e"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -3483,13 +3584,13 @@ dependencies = [
 
 [[package]]
 name = "tracing-attributes"
-version = "0.1.24"
+version = "0.1.26"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0f57e3ca2a01450b1a921183a9c9cbfda207fd822cef4ccb00a65402cbba7a74"
+checksum = "5f4f31f56159e98206da9efd823404b79b6ef3143b4a7ab76e67b1751b25a4ab"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -3607,9 +3708,9 @@ checksum = "92888ba5573ff080736b3648696b70cafad7d250551175acbaa4e0385b3e1460"
 
 [[package]]
 name = "unicode-ident"
-version = "1.0.9"
+version = "1.0.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b15811caf2415fb889178633e7724bad2509101cde276048e013b9def5e51fa0"
+checksum = "301abaae475aa91687eb82514b328ab47a211a533026cb25fc3e519b86adfc3c"
 
 [[package]]
 name = "unicode-normalization"
@@ -3669,9 +3770,9 @@ checksum = "830b7e5d4d90034032940e4ace0d9a9a057e7a45cd94e6c007832e39edb82f6d"
 
 [[package]]
 name = "value-bag"
-version = "1.4.0"
+version = "1.4.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a4d330786735ea358f3bc09eea4caa098569c1c93f342d9aca0514915022fe7e"
+checksum = "d92ccd67fb88503048c01b59152a04effd0782d035a83a6d256ce6085f08f4a3"
 
 [[package]]
 name = "vcpkg"
@@ -3766,7 +3867,7 @@ version = "0.10.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "08604d7be03eb26e33b3cee3ed4aef2bf550b305d1cca60e84da5d28d3790b62"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "libc",
 ]
 
@@ -3776,7 +3877,7 @@ version = "0.11.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dd64fe09d8e880e600c324e7d664760a17f56e9672b7495a86381b49e4f72f46"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "libc",
 ]
 
@@ -3798,11 +3899,10 @@ checksum = "9d5b2c62b4012a3e1eca5a7e077d13b3bf498c4073e33ccd58626607748ceeca"
 
 [[package]]
 name = "want"
-version = "0.3.0"
+version = "0.3.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1ce8a968cb1cd110d136ff8b819a556d6fb6d919363c61534f6860c7eb172ba0"
+checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
 dependencies = [
- "log",
  "try-lock",
 ]
 
@@ -3826,9 +3926,9 @@ checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
 
 [[package]]
 name = "wasm-bindgen"
-version = "0.2.86"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5bba0e8cb82ba49ff4e229459ff22a191bbe9a1cb3a341610c9c33efc27ddf73"
+checksum = "7706a72ab36d8cb1f80ffbf0e071533974a60d0a308d01a5d0375bf60499a342"
 dependencies = [
  "cfg-if 1.0.0",
  "wasm-bindgen-macro",
@@ -3836,24 +3936,24 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-backend"
-version = "0.2.86"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "19b04bc93f9d6bdee709f6bd2118f57dd6679cf1176a1af464fca3ab0d66d8fb"
+checksum = "5ef2b6d3c510e9625e5fe6f509ab07d66a760f0885d858736483c32ed7809abd"
 dependencies = [
  "bumpalo",
  "log",
  "once_cell",
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
  "wasm-bindgen-shared",
 ]
 
 [[package]]
 name = "wasm-bindgen-futures"
-version = "0.4.36"
+version = "0.4.37"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2d1985d03709c53167ce907ff394f5316aa22cb4e12761295c5dc57dacb6297e"
+checksum = "c02dbc21516f9f1f04f187958890d7e6026df8d16540b7ad9492bc34a67cea03"
 dependencies = [
  "cfg-if 1.0.0",
  "js-sys",
@@ -3863,9 +3963,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-macro"
-version = "0.2.86"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "14d6b024f1a526bb0234f52840389927257beb670610081360e5a03c5df9c258"
+checksum = "dee495e55982a3bd48105a7b947fd2a9b4a8ae3010041b9e0faab3f9cd028f1d"
 dependencies = [
  "quote",
  "wasm-bindgen-macro-support",
@@ -3873,22 +3973,22 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-macro-support"
-version = "0.2.86"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e128beba882dd1eb6200e1dc92ae6c5dbaa4311aa7bb211ca035779e5efc39f8"
+checksum = "54681b18a46765f095758388f2d0cf16eb8d4169b639ab575a8f5693af210c7b"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.18",
+ "syn 2.0.27",
  "wasm-bindgen-backend",
  "wasm-bindgen-shared",
 ]
 
 [[package]]
 name = "wasm-bindgen-shared"
-version = "0.2.86"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ed9d5b4305409d1fc9482fee2d7f9bcbf24b3972bf59817ef757e23982242a93"
+checksum = "ca6ad05a4870b2bf5fe995117d3728437bd27d7cd5f06f13c17443ef369775a1"
 
 [[package]]
 name = "wasm_container"
@@ -3903,9 +4003,9 @@ dependencies = [
 
 [[package]]
 name = "web-sys"
-version = "0.3.63"
+version = "0.3.64"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3bdd9ef4e984da1187bf8110c5cf5b845fbc87a23602cdf912386a76fcd3a7c2"
+checksum = "9b85cbef8c220a6abc02aefd892dfc0fc23afb1c6a426316ec33253a3877249b"
 dependencies = [
  "js-sys",
  "wasm-bindgen",
@@ -3950,7 +4050,16 @@ version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e686886bc078bc1b0b600cac0147aadb815089b6e4da64016cbd754b6342700f"
 dependencies = [
- "windows-targets",
+ "windows-targets 0.48.1",
+]
+
+[[package]]
+name = "windows-sys"
+version = "0.45.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0"
+dependencies = [
+ "windows-targets 0.42.2",
 ]
 
 [[package]]
@@ -3959,60 +4068,117 @@ version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9"
 dependencies = [
- "windows-targets",
+ "windows-targets 0.48.1",
 ]
 
 [[package]]
 name = "windows-targets"
-version = "0.48.0"
+version = "0.42.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b1eb6f0cd7c80c79759c929114ef071b87354ce476d9d94271031c0497adfd5"
+checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071"
 dependencies = [
- "windows_aarch64_gnullvm",
- "windows_aarch64_msvc",
- "windows_i686_gnu",
- "windows_i686_msvc",
- "windows_x86_64_gnu",
- "windows_x86_64_gnullvm",
- "windows_x86_64_msvc",
+ "windows_aarch64_gnullvm 0.42.2",
+ "windows_aarch64_msvc 0.42.2",
+ "windows_i686_gnu 0.42.2",
+ "windows_i686_msvc 0.42.2",
+ "windows_x86_64_gnu 0.42.2",
+ "windows_x86_64_gnullvm 0.42.2",
+ "windows_x86_64_msvc 0.42.2",
 ]
 
+[[package]]
+name = "windows-targets"
+version = "0.48.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "05d4b17490f70499f20b9e791dcf6a299785ce8af4d709018206dc5b4953e95f"
+dependencies = [
+ "windows_aarch64_gnullvm 0.48.0",
+ "windows_aarch64_msvc 0.48.0",
+ "windows_i686_gnu 0.48.0",
+ "windows_i686_msvc 0.48.0",
+ "windows_x86_64_gnu 0.48.0",
+ "windows_x86_64_gnullvm 0.48.0",
+ "windows_x86_64_msvc 0.48.0",
+]
+
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8"
+
 [[package]]
 name = "windows_aarch64_gnullvm"
 version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "91ae572e1b79dba883e0d315474df7305d12f569b400fcf90581b06062f7e1bc"
 
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43"
+
 [[package]]
 name = "windows_aarch64_msvc"
 version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b2ef27e0d7bdfcfc7b868b317c1d32c641a6fe4629c171b8928c7b08d98d7cf3"
 
+[[package]]
+name = "windows_i686_gnu"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f"
+
 [[package]]
 name = "windows_i686_gnu"
 version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "622a1962a7db830d6fd0a69683c80a18fda201879f0f447f065a3b7467daa241"
 
+[[package]]
+name = "windows_i686_msvc"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060"
+
 [[package]]
 name = "windows_i686_msvc"
 version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4542c6e364ce21bf45d69fdd2a8e455fa38d316158cfd43b3ac1c5b1b19f8e00"
 
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36"
+
 [[package]]
 name = "windows_x86_64_gnu"
 version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ca2b8a661f7628cbd23440e50b05d705db3686f894fc9580820623656af974b1"
 
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3"
+
 [[package]]
 name = "windows_x86_64_gnullvm"
 version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7896dbc1f41e08872e9d5e8f8baa8fdd2677f29468c4e156210174edc7f7b953"
 
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
+
 [[package]]
 name = "windows_x86_64_msvc"
 version = "0.48.0"
diff --git a/src/runtime-rs/crates/agent/src/kata/agent.rs b/src/runtime-rs/crates/agent/src/kata/agent.rs
index d5f5b2b7fc..ee3e97bf85 100644
--- a/src/runtime-rs/crates/agent/src/kata/agent.rs
+++ b/src/runtime-rs/crates/agent/src/kata/agent.rs
@@ -121,5 +121,6 @@ impl_agent!(
     set_ip_tables | crate::SetIPTablesRequest | crate::SetIPTablesResponse | None,
     get_volume_stats | crate::VolumeStatsRequest | crate::VolumeStatsResponse | None,
     resize_volume | crate::ResizeVolumeRequest | crate::Empty | None,
-    online_cpu_mem | crate::OnlineCPUMemRequest | crate::Empty | None
+    online_cpu_mem | crate::OnlineCPUMemRequest | crate::Empty | None,
+    get_metrics | crate::Empty | crate::MetricsResponse | None
 );
diff --git a/src/runtime-rs/crates/agent/src/kata/trans.rs b/src/runtime-rs/crates/agent/src/kata/trans.rs
index ca9c8f1720..55635c6c59 100644
--- a/src/runtime-rs/crates/agent/src/kata/trans.rs
+++ b/src/runtime-rs/crates/agent/src/kata/trans.rs
@@ -7,7 +7,7 @@
 use std::convert::Into;
 
 use protocols::{
-    agent::{self, OOMEvent},
+    agent::{self, Metrics, OOMEvent},
     csi, empty, health, types,
 };
 
@@ -19,13 +19,13 @@ use crate::{
         Empty, ExecProcessRequest, FSGroup, FSGroupChangePolicy, GetIPTablesRequest,
         GetIPTablesResponse, GuestDetailsResponse, HealthCheckResponse, HugetlbStats, IPAddress,
         IPFamily, Interface, Interfaces, KernelModule, MemHotplugByProbeRequest, MemoryData,
-        MemoryStats, NetworkStats, OnlineCPUMemRequest, PidsStats, ReadStreamRequest,
-        ReadStreamResponse, RemoveContainerRequest, ReseedRandomDevRequest, ResizeVolumeRequest,
-        Route, Routes, SetGuestDateTimeRequest, SetIPTablesRequest, SetIPTablesResponse,
-        SignalProcessRequest, StatsContainerResponse, Storage, StringUser, ThrottlingData,
-        TtyWinResizeRequest, UpdateContainerRequest, UpdateInterfaceRequest, UpdateRoutesRequest,
-        VersionCheckResponse, VolumeStatsRequest, VolumeStatsResponse, WaitProcessRequest,
-        WriteStreamRequest,
+        MemoryStats, MetricsResponse, NetworkStats, OnlineCPUMemRequest, PidsStats,
+        ReadStreamRequest, ReadStreamResponse, RemoveContainerRequest, ReseedRandomDevRequest,
+        ResizeVolumeRequest, Route, Routes, SetGuestDateTimeRequest, SetIPTablesRequest,
+        SetIPTablesResponse, SignalProcessRequest, StatsContainerResponse, Storage, StringUser,
+        ThrottlingData, TtyWinResizeRequest, UpdateContainerRequest, UpdateInterfaceRequest,
+        UpdateRoutesRequest, VersionCheckResponse, VolumeStatsRequest, VolumeStatsResponse,
+        WaitProcessRequest, WriteStreamRequest,
     },
     OomEventResponse, WaitProcessResponse, WriteStreamResponse,
 };
@@ -755,6 +755,14 @@ impl From<agent::WaitProcessResponse> for WaitProcessResponse {
     }
 }
 
+impl From<Empty> for agent::GetMetricsRequest {
+    fn from(_: Empty) -> Self {
+        Self {
+            ..Default::default()
+        }
+    }
+}
+
 impl From<Empty> for agent::GetOOMEventRequest {
     fn from(_: Empty) -> Self {
         Self {
@@ -789,6 +797,14 @@ impl From<health::VersionCheckResponse> for VersionCheckResponse {
     }
 }
 
+impl From<agent::Metrics> for MetricsResponse {
+    fn from(from: Metrics) -> Self {
+        Self {
+            metrics: from.metrics,
+        }
+    }
+}
+
 impl From<agent::OOMEvent> for OomEventResponse {
     fn from(from: OOMEvent) -> Self {
         Self {
diff --git a/src/runtime-rs/crates/agent/src/lib.rs b/src/runtime-rs/crates/agent/src/lib.rs
index 43449ca591..8dd71e8d7b 100644
--- a/src/runtime-rs/crates/agent/src/lib.rs
+++ b/src/runtime-rs/crates/agent/src/lib.rs
@@ -18,13 +18,14 @@ pub use types::{
     CloseStdinRequest, ContainerID, ContainerProcessID, CopyFileRequest, CreateContainerRequest,
     CreateSandboxRequest, Empty, ExecProcessRequest, GetGuestDetailsRequest, GetIPTablesRequest,
     GetIPTablesResponse, GuestDetailsResponse, HealthCheckResponse, IPAddress, IPFamily, Interface,
-    Interfaces, ListProcessesRequest, MemHotplugByProbeRequest, OnlineCPUMemRequest,
-    OomEventResponse, ReadStreamRequest, ReadStreamResponse, RemoveContainerRequest,
-    ReseedRandomDevRequest, ResizeVolumeRequest, Route, Routes, SetGuestDateTimeRequest,
-    SetIPTablesRequest, SetIPTablesResponse, SignalProcessRequest, StatsContainerResponse, Storage,
-    TtyWinResizeRequest, UpdateContainerRequest, UpdateInterfaceRequest, UpdateRoutesRequest,
-    VersionCheckResponse, VolumeStatsRequest, VolumeStatsResponse, WaitProcessRequest,
-    WaitProcessResponse, WriteStreamRequest, WriteStreamResponse,
+    Interfaces, ListProcessesRequest, MemHotplugByProbeRequest, MetricsResponse,
+    OnlineCPUMemRequest, OomEventResponse, ReadStreamRequest, ReadStreamResponse,
+    RemoveContainerRequest, ReseedRandomDevRequest, ResizeVolumeRequest, Route, Routes,
+    SetGuestDateTimeRequest, SetIPTablesRequest, SetIPTablesResponse, SignalProcessRequest,
+    StatsContainerResponse, Storage, TtyWinResizeRequest, UpdateContainerRequest,
+    UpdateInterfaceRequest, UpdateRoutesRequest, VersionCheckResponse, VolumeStatsRequest,
+    VolumeStatsResponse, WaitProcessRequest, WaitProcessResponse, WriteStreamRequest,
+    WriteStreamResponse,
 };
 
 use anyhow::Result;
@@ -86,6 +87,7 @@ pub trait Agent: AgentManager + HealthService + Send + Sync {
 
     // utils
     async fn copy_file(&self, req: CopyFileRequest) -> Result<Empty>;
+    async fn get_metrics(&self, req: Empty) -> Result<MetricsResponse>;
     async fn get_oom_event(&self, req: Empty) -> Result<OomEventResponse>;
     async fn get_ip_tables(&self, req: GetIPTablesRequest) -> Result<GetIPTablesResponse>;
     async fn set_ip_tables(&self, req: SetIPTablesRequest) -> Result<SetIPTablesResponse>;
diff --git a/src/runtime-rs/crates/agent/src/types.rs b/src/runtime-rs/crates/agent/src/types.rs
index 1ba7efd5e3..325d84588d 100644
--- a/src/runtime-rs/crates/agent/src/types.rs
+++ b/src/runtime-rs/crates/agent/src/types.rs
@@ -556,6 +556,11 @@ pub struct VersionCheckResponse {
     pub agent_version: String,
 }
 
+#[derive(PartialEq, Clone, Default, Debug)]
+pub struct MetricsResponse {
+    pub metrics: String,
+}
+
 #[derive(PartialEq, Clone, Default, Debug)]
 pub struct OomEventResponse {
     pub container_id: String,
diff --git a/src/runtime-rs/crates/hypervisor/src/ch/inner_hypervisor.rs b/src/runtime-rs/crates/hypervisor/src/ch/inner_hypervisor.rs
index a54a6a43e2..fc020f0d2a 100644
--- a/src/runtime-rs/crates/hypervisor/src/ch/inner_hypervisor.rs
+++ b/src/runtime-rs/crates/hypervisor/src/ch/inner_hypervisor.rs
@@ -536,6 +536,10 @@ impl CloudHypervisorInner {
         caps.set(CapabilityBits::FsSharingSupport);
         Ok(caps)
     }
+
+    pub(crate) async fn get_hypervisor_metrics(&self) -> Result<String> {
+        todo!()
+    }
 }
 
 // Log all output from the CH process until a shutdown signal is received.
diff --git a/src/runtime-rs/crates/hypervisor/src/ch/mod.rs b/src/runtime-rs/crates/hypervisor/src/ch/mod.rs
index dd95413fc3..72373978ce 100644
--- a/src/runtime-rs/crates/hypervisor/src/ch/mod.rs
+++ b/src/runtime-rs/crates/hypervisor/src/ch/mod.rs
@@ -152,6 +152,11 @@ impl Hypervisor for CloudHypervisor {
         let inner = self.inner.read().await;
         inner.capabilities().await
     }
+
+    async fn get_hypervisor_metrics(&self) -> Result<String> {
+        let inner = self.inner.read().await;
+        inner.get_hypervisor_metrics().await
+    }
 }
 
 #[async_trait]
diff --git a/src/runtime-rs/crates/hypervisor/src/dragonball/inner_hypervisor.rs b/src/runtime-rs/crates/hypervisor/src/dragonball/inner_hypervisor.rs
index 18aa7139dd..5fc1da4f17 100644
--- a/src/runtime-rs/crates/hypervisor/src/dragonball/inner_hypervisor.rs
+++ b/src/runtime-rs/crates/hypervisor/src/dragonball/inner_hypervisor.rs
@@ -92,6 +92,11 @@ impl DragonballInner {
         ))
     }
 
+    pub(crate) async fn get_hypervisor_metrics(&self) -> Result<String> {
+        info!(sl!(), "get hypervisor metrics");
+        self.vmm_instance.get_hypervisor_metrics()
+    }
+
     pub(crate) async fn disconnect(&mut self) {
         self.state = VmmState::NotReady;
     }
diff --git a/src/runtime-rs/crates/hypervisor/src/dragonball/mod.rs b/src/runtime-rs/crates/hypervisor/src/dragonball/mod.rs
index 76e949381e..3b98f38a23 100644
--- a/src/runtime-rs/crates/hypervisor/src/dragonball/mod.rs
+++ b/src/runtime-rs/crates/hypervisor/src/dragonball/mod.rs
@@ -160,6 +160,11 @@ impl Hypervisor for Dragonball {
         let inner = self.inner.read().await;
         inner.capabilities().await
     }
+
+    async fn get_hypervisor_metrics(&self) -> Result<String> {
+        let inner = self.inner.read().await;
+        inner.get_hypervisor_metrics().await
+    }
 }
 
 #[async_trait]
diff --git a/src/runtime-rs/crates/hypervisor/src/dragonball/vmm_instance.rs b/src/runtime-rs/crates/hypervisor/src/dragonball/vmm_instance.rs
index 1010f9d3f6..68dfe41041 100644
--- a/src/runtime-rs/crates/hypervisor/src/dragonball/vmm_instance.rs
+++ b/src/runtime-rs/crates/hypervisor/src/dragonball/vmm_instance.rs
@@ -267,6 +267,15 @@ impl VmmInstance {
         std::process::id()
     }
 
+    pub fn get_hypervisor_metrics(&self) -> Result<String> {
+        if let Ok(VmmData::HypervisorMetrics(metrics)) =
+            self.handle_request(Request::Sync(VmmAction::GetHypervisorMetrics))
+        {
+            return Ok(metrics);
+        }
+        Err(anyhow!("Failed to get hypervisor metrics"))
+    }
+
     pub fn stop(&mut self) -> Result<()> {
         self.handle_request(Request::Sync(VmmAction::ShutdownMicroVm))
             .map_err(|e| {
diff --git a/src/runtime-rs/crates/hypervisor/src/lib.rs b/src/runtime-rs/crates/hypervisor/src/lib.rs
index f9ec5109b4..e9fa3a9bf0 100644
--- a/src/runtime-rs/crates/hypervisor/src/lib.rs
+++ b/src/runtime-rs/crates/hypervisor/src/lib.rs
@@ -97,4 +97,5 @@ pub trait Hypervisor: std::fmt::Debug + Send + Sync {
     async fn get_jailer_root(&self) -> Result<String>;
     async fn save_state(&self) -> Result<HypervisorState>;
     async fn capabilities(&self) -> Result<Capabilities>;
+    async fn get_hypervisor_metrics(&self) -> Result<String>;
 }
diff --git a/src/runtime-rs/crates/hypervisor/src/qemu/inner.rs b/src/runtime-rs/crates/hypervisor/src/qemu/inner.rs
index 3bfd3cbc17..6f59d93391 100644
--- a/src/runtime-rs/crates/hypervisor/src/qemu/inner.rs
+++ b/src/runtime-rs/crates/hypervisor/src/qemu/inner.rs
@@ -136,6 +136,10 @@ impl QemuInner {
         info!(sl!(), "QemuInner::hypervisor_config()");
         self.config.clone()
     }
+
+    pub(crate) async fn get_hypervisor_metrics(&self) -> Result<String> {
+        todo!()
+    }
 }
 
 use crate::device::DeviceType;
diff --git a/src/runtime-rs/crates/hypervisor/src/qemu/mod.rs b/src/runtime-rs/crates/hypervisor/src/qemu/mod.rs
index f8c66d7cf3..1221af26b8 100644
--- a/src/runtime-rs/crates/hypervisor/src/qemu/mod.rs
+++ b/src/runtime-rs/crates/hypervisor/src/qemu/mod.rs
@@ -147,4 +147,9 @@ impl Hypervisor for Qemu {
         let inner = self.inner.read().await;
         inner.capabilities().await
     }
+
+    async fn get_hypervisor_metrics(&self) -> Result<String> {
+        let inner = self.inner.read().await;
+        inner.get_hypervisor_metrics().await
+    }
 }
diff --git a/src/runtime-rs/crates/runtimes/Cargo.toml b/src/runtime-rs/crates/runtimes/Cargo.toml
index 1d0ff168e4..4d5b4dcc97 100644
--- a/src/runtime-rs/crates/runtimes/Cargo.toml
+++ b/src/runtime-rs/crates/runtimes/Cargo.toml
@@ -22,6 +22,8 @@ hyperlocal = "0.8"
 serde_json = "1.0.88"
 nix = "0.25.0"
 url = "2.3.1"
+procfs = "0.12.0"
+prometheus = { version = "0.13.0", features = ["process"] }
 
 agent = { path = "../agent" }
 common = { path = "./common" }
diff --git a/src/runtime-rs/crates/runtimes/common/src/sandbox.rs b/src/runtime-rs/crates/runtimes/common/src/sandbox.rs
index 9134b8c78b..f27c837ab2 100644
--- a/src/runtime-rs/crates/runtimes/common/src/sandbox.rs
+++ b/src/runtime-rs/crates/runtimes/common/src/sandbox.rs
@@ -41,4 +41,8 @@ pub trait Sandbox: Send + Sync {
     async fn direct_volume_stats(&self, volume_path: &str) -> Result<String>;
     async fn direct_volume_resize(&self, resize_req: agent::ResizeVolumeRequest) -> Result<()>;
     async fn agent_sock(&self) -> Result<String>;
+
+    // metrics function
+    async fn agent_metrics(&self) -> Result<String>;
+    async fn hypervisor_metrics(&self) -> Result<String>;
 }
diff --git a/src/runtime-rs/crates/runtimes/src/lib.rs b/src/runtime-rs/crates/runtimes/src/lib.rs
index 63357df013..9a98ed5e8c 100644
--- a/src/runtime-rs/crates/runtimes/src/lib.rs
+++ b/src/runtime-rs/crates/runtimes/src/lib.rs
@@ -4,6 +4,9 @@
 // SPDX-License-Identifier: Apache-2.0
 //
 
+#[macro_use(lazy_static)]
+extern crate lazy_static;
+
 #[macro_use]
 extern crate slog;
 
@@ -12,5 +15,6 @@ logging::logger_with_subsystem!(sl, "runtimes");
 pub mod manager;
 pub use manager::RuntimeHandlerManager;
 pub use shim_interface;
+mod shim_metrics;
 mod shim_mgmt;
 pub mod tracer;
diff --git a/src/runtime-rs/crates/runtimes/src/shim_metrics.rs b/src/runtime-rs/crates/runtimes/src/shim_metrics.rs
new file mode 100644
index 0000000000..62ebf01356
--- /dev/null
+++ b/src/runtime-rs/crates/runtimes/src/shim_metrics.rs
@@ -0,0 +1,235 @@
+// Copyright 2021-2022 Ant Group
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+extern crate procfs;
+
+use anyhow::{anyhow, Result};
+use prometheus::{Encoder, Gauge, GaugeVec, Opts, Registry, TextEncoder};
+use slog::warn;
+use std::sync::Mutex;
+
+const NAMESPACE_KATA_SHIM: &str = "kata_shim";
+
+// Convenience macro to obtain the scope logger
+macro_rules! sl {
+    () => {
+        slog_scope::logger().new(o!("subsystem" => "metrics"))
+    };
+}
+
+lazy_static! {
+    static ref REGISTERED: Mutex<bool> = Mutex::new(false);
+
+    // custom registry
+    static ref REGISTRY: Registry = Registry::new();
+
+    // shim metrics
+    static ref SHIM_THREADS: Gauge = Gauge::new(format!("{}_{}", NAMESPACE_KATA_SHIM, "threads"),"Kata containerd shim v2 process threads.").unwrap();
+
+    static ref SHIM_PROC_STATUS: GaugeVec =
+        GaugeVec::new(Opts::new(format!("{}_{}",NAMESPACE_KATA_SHIM,"proc_status"), "Kata containerd shim v2 process status."), &["item"]).unwrap();
+
+    static ref SHIM_PROC_STAT: GaugeVec = GaugeVec::new(Opts::new(format!("{}_{}",NAMESPACE_KATA_SHIM,"proc_stat"), "Kata containerd shim v2 process statistics."), &["item"]).unwrap();
+
+    static ref SHIM_NETDEV: GaugeVec = GaugeVec::new(Opts::new(format!("{}_{}",NAMESPACE_KATA_SHIM,"netdev"), "Kata containerd shim v2 network devices statistics."), &["interface", "item"]).unwrap();
+
+    static ref SHIM_IO_STAT: GaugeVec = GaugeVec::new(Opts::new(format!("{}_{}",NAMESPACE_KATA_SHIM,"io_stat"), "Kata containerd shim v2 process IO statistics."), &["item"]).unwrap();
+
+    static ref SHIM_OPEN_FDS: Gauge = Gauge::new(format!("{}_{}", NAMESPACE_KATA_SHIM, "fds"), "Kata containerd shim v2 open FDs.").unwrap();
+}
+
+pub fn get_shim_metrics() -> Result<String> {
+    let mut registered = REGISTERED
+        .lock()
+        .map_err(|e| anyhow!("failed to check shim metrics register status {:?}", e))?;
+
+    if !(*registered) {
+        register_shim_metrics()?;
+        *registered = true;
+    }
+
+    update_shim_metrics()?;
+
+    // gather all metrics and return as a String
+    let metric_families = REGISTRY.gather();
+
+    let mut buffer = Vec::new();
+    let encoder = TextEncoder::new();
+    encoder.encode(&metric_families, &mut buffer)?;
+
+    Ok(String::from_utf8(buffer)?)
+}
+
+fn register_shim_metrics() -> Result<()> {
+    REGISTRY.register(Box::new(SHIM_THREADS.clone()))?;
+    REGISTRY.register(Box::new(SHIM_PROC_STATUS.clone()))?;
+    REGISTRY.register(Box::new(SHIM_PROC_STAT.clone()))?;
+    REGISTRY.register(Box::new(SHIM_NETDEV.clone()))?;
+    REGISTRY.register(Box::new(SHIM_IO_STAT.clone()))?;
+    REGISTRY.register(Box::new(SHIM_OPEN_FDS.clone()))?;
+
+    // TODO:
+    // REGISTRY.register(Box::new(RPC_DURATIONS_HISTOGRAM.clone()))?;
+    // REGISTRY.register(Box::new(SHIM_POD_OVERHEAD_CPU.clone()))?;
+    // REGISTRY.register(Box::new(SHIM_POD_OVERHEAD_MEMORY.clone()))?;
+
+    Ok(())
+}
+
+fn update_shim_metrics() -> Result<()> {
+    let me = procfs::process::Process::myself();
+
+    let me = match me {
+        Ok(p) => p,
+        Err(e) => {
+            warn!(sl!(), "failed to create process instance: {:?}", e);
+            return Ok(());
+        }
+    };
+
+    SHIM_THREADS.set(me.stat.num_threads as f64);
+
+    match me.status() {
+        Err(err) => error!(sl!(), "failed to get process status: {:?}", err),
+        Ok(status) => set_gauge_vec_proc_status(&SHIM_PROC_STATUS, &status),
+    }
+
+    match me.stat() {
+        Err(err) => {
+            error!(sl!(), "failed to get process stat: {:?}", err);
+        }
+        Ok(stat) => {
+            set_gauge_vec_proc_stat(&SHIM_PROC_STAT, &stat);
+        }
+    }
+
+    match procfs::net::dev_status() {
+        Err(err) => {
+            error!(sl!(), "failed to get host net::dev_status: {:?}", err);
+        }
+        Ok(devs) => {
+            for (_, status) in devs {
+                set_gauge_vec_netdev(&SHIM_NETDEV, &status);
+            }
+        }
+    }
+
+    match me.io() {
+        Err(err) => {
+            error!(sl!(), "failed to get process io stat: {:?}", err);
+        }
+        Ok(io) => {
+            set_gauge_vec_proc_io(&SHIM_IO_STAT, &io);
+        }
+    }
+
+    match me.fd_count() {
+        Err(err) => {
+            error!(sl!(), "failed to get process open fds number: {:?}", err);
+        }
+        Ok(fds) => {
+            SHIM_OPEN_FDS.set(fds as f64);
+        }
+    }
+
+    // TODO:
+    // RPC_DURATIONS_HISTOGRAM & SHIM_POD_OVERHEAD_CPU & SHIM_POD_OVERHEAD_MEMORY
+
+    Ok(())
+}
+
+fn set_gauge_vec_proc_status(gv: &prometheus::GaugeVec, status: &procfs::process::Status) {
+    gv.with_label_values(&["vmpeak"])
+        .set(status.vmpeak.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmsize"])
+        .set(status.vmsize.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmlck"])
+        .set(status.vmlck.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmpin"])
+        .set(status.vmpin.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmhwm"])
+        .set(status.vmhwm.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmrss"])
+        .set(status.vmrss.unwrap_or(0) as f64);
+    gv.with_label_values(&["rssanon"])
+        .set(status.rssanon.unwrap_or(0) as f64);
+    gv.with_label_values(&["rssfile"])
+        .set(status.rssfile.unwrap_or(0) as f64);
+    gv.with_label_values(&["rssshmem"])
+        .set(status.rssshmem.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmdata"])
+        .set(status.vmdata.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmstk"])
+        .set(status.vmstk.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmexe"])
+        .set(status.vmexe.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmlib"])
+        .set(status.vmlib.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmpte"])
+        .set(status.vmpte.unwrap_or(0) as f64);
+    gv.with_label_values(&["vmswap"])
+        .set(status.vmswap.unwrap_or(0) as f64);
+    gv.with_label_values(&["hugetlbpages"])
+        .set(status.hugetlbpages.unwrap_or(0) as f64);
+    gv.with_label_values(&["voluntary_ctxt_switches"])
+        .set(status.voluntary_ctxt_switches.unwrap_or(0) as f64);
+    gv.with_label_values(&["nonvoluntary_ctxt_switches"])
+        .set(status.nonvoluntary_ctxt_switches.unwrap_or(0) as f64);
+}
+
+fn set_gauge_vec_proc_stat(gv: &prometheus::GaugeVec, stat: &procfs::process::Stat) {
+    gv.with_label_values(&["utime"]).set(stat.utime as f64);
+    gv.with_label_values(&["stime"]).set(stat.stime as f64);
+    gv.with_label_values(&["cutime"]).set(stat.cutime as f64);
+    gv.with_label_values(&["cstime"]).set(stat.cstime as f64);
+}
+
+fn set_gauge_vec_netdev(gv: &prometheus::GaugeVec, status: &procfs::net::DeviceStatus) {
+    gv.with_label_values(&[status.name.as_str(), "recv_bytes"])
+        .set(status.recv_bytes as f64);
+    gv.with_label_values(&[status.name.as_str(), "recv_packets"])
+        .set(status.recv_packets as f64);
+    gv.with_label_values(&[status.name.as_str(), "recv_errs"])
+        .set(status.recv_errs as f64);
+    gv.with_label_values(&[status.name.as_str(), "recv_drop"])
+        .set(status.recv_drop as f64);
+    gv.with_label_values(&[status.name.as_str(), "recv_fifo"])
+        .set(status.recv_fifo as f64);
+    gv.with_label_values(&[status.name.as_str(), "recv_frame"])
+        .set(status.recv_frame as f64);
+    gv.with_label_values(&[status.name.as_str(), "recv_compressed"])
+        .set(status.recv_compressed as f64);
+    gv.with_label_values(&[status.name.as_str(), "recv_multicast"])
+        .set(status.recv_multicast as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_bytes"])
+        .set(status.sent_bytes as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_packets"])
+        .set(status.sent_packets as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_errs"])
+        .set(status.sent_errs as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_drop"])
+        .set(status.sent_drop as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_fifo"])
+        .set(status.sent_fifo as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_colls"])
+        .set(status.sent_colls as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_carrier"])
+        .set(status.sent_carrier as f64);
+    gv.with_label_values(&[status.name.as_str(), "sent_compressed"])
+        .set(status.sent_compressed as f64);
+}
+
+fn set_gauge_vec_proc_io(gv: &prometheus::GaugeVec, io_stat: &procfs::process::Io) {
+    gv.with_label_values(&["rchar"]).set(io_stat.rchar as f64);
+    gv.with_label_values(&["wchar"]).set(io_stat.wchar as f64);
+    gv.with_label_values(&["syscr"]).set(io_stat.syscr as f64);
+    gv.with_label_values(&["syscw"]).set(io_stat.syscw as f64);
+    gv.with_label_values(&["read_bytes"])
+        .set(io_stat.read_bytes as f64);
+    gv.with_label_values(&["write_bytes"])
+        .set(io_stat.write_bytes as f64);
+    gv.with_label_values(&["cancelled_write_bytes"])
+        .set(io_stat.cancelled_write_bytes as f64);
+}
diff --git a/src/runtime-rs/crates/runtimes/src/shim_mgmt/handlers.rs b/src/runtime-rs/crates/runtimes/src/shim_mgmt/handlers.rs
index b0c79cd404..613cca2878 100644
--- a/src/runtime-rs/crates/runtimes/src/shim_mgmt/handlers.rs
+++ b/src/runtime-rs/crates/runtimes/src/shim_mgmt/handlers.rs
@@ -7,6 +7,7 @@
 // This defines the handlers corresponding to the url when a request is sent to destined url,
 // the handler function should be invoked, and the corresponding data will be in the response
 
+use crate::shim_metrics::get_shim_metrics;
 use agent::ResizeVolumeRequest;
 use anyhow::{anyhow, Context, Result};
 use common::Sandbox;
@@ -16,7 +17,7 @@ use url::Url;
 
 use shim_interface::shim_mgmt::{
     AGENT_URL, DIRECT_VOLUME_PATH_KEY, DIRECT_VOLUME_RESIZE_URL, DIRECT_VOLUME_STATS_URL,
-    IP6_TABLE_URL, IP_TABLE_URL,
+    IP6_TABLE_URL, IP_TABLE_URL, METRICS_URL,
 };
 
 // main router for response, this works as a multiplexer on
@@ -43,6 +44,7 @@ pub(crate) async fn handler_mux(
         (&Method::POST, DIRECT_VOLUME_RESIZE_URL) => {
             direct_volume_resize_handler(sandbox, req).await
         }
+        (&Method::GET, METRICS_URL) => metrics_url_handler(sandbox, req).await,
         _ => Ok(not_found(req).await),
     }
 }
@@ -146,3 +148,19 @@ async fn direct_volume_resize_handler(
         _ => Err(anyhow!("handler: Failed to resize volume")),
     }
 }
+
+// returns the url for metrics
+async fn metrics_url_handler(
+    sandbox: Arc<dyn Sandbox>,
+    _req: Request<Body>,
+) -> Result<Response<Body>> {
+    // get metrics from agent, hypervisor, and shim
+    let agent_metrics = sandbox.agent_metrics().await.unwrap_or_default();
+    let hypervisor_metrics = sandbox.hypervisor_metrics().await.unwrap_or_default();
+    let shim_metrics = get_shim_metrics().unwrap_or_default();
+
+    Ok(Response::new(Body::from(format!(
+        "{}{}{}",
+        agent_metrics, hypervisor_metrics, shim_metrics
+    ))))
+}
diff --git a/src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs b/src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs
index 43f8fef873..9ded5033cf 100644
--- a/src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs
+++ b/src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs
@@ -459,6 +459,18 @@ impl Sandbox for VirtSandbox {
             .context("sandbox: failed to get iptables")?;
         Ok(resp.data)
     }
+
+    async fn agent_metrics(&self) -> Result<String> {
+        self.agent
+            .get_metrics(agent::Empty::new())
+            .await
+            .map_err(|err| anyhow!("failed to get agent metrics {:?}", err))
+            .map(|resp| resp.metrics)
+    }
+
+    async fn hypervisor_metrics(&self) -> Result<String> {
+        self.hypervisor.get_hypervisor_metrics().await
+    }
 }
 
 #[async_trait]

From d74639d8c6294a14ec7f2008f74977787c7b509d Mon Sep 17 00:00:00 2001
From: Yuan-Zhuo <yuanzhuo0118@outlook.com>
Date: Fri, 28 Jul 2023 17:23:37 +0800
Subject: [PATCH 02/49] kata-ctl: provide the global TIMEOUT for creating
 MgmtClient

Several functions in kata-ctl need to establish a connection with runtime-rs through MgmtClient.
This PR provides a global TIMEOUT to avoid multiple definitions.

Fixes: #5017

Signed-off-by: Yuan-Zhuo <yuanzhuo0118@outlook.com>
---
 src/tools/kata-ctl/src/ops/exec_ops.rs   | 3 ++-
 src/tools/kata-ctl/src/ops/volume_ops.rs | 5 +++--
 src/tools/kata-ctl/src/utils.rs          | 4 +++-
 3 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/src/tools/kata-ctl/src/ops/exec_ops.rs b/src/tools/kata-ctl/src/ops/exec_ops.rs
index 51651188b9..f7ad0170cb 100644
--- a/src/tools/kata-ctl/src/ops/exec_ops.rs
+++ b/src/tools/kata-ctl/src/ops/exec_ops.rs
@@ -25,6 +25,8 @@ use vmm_sys_util::terminal::Terminal;
 use crate::args::ExecArguments;
 use shim_interface::shim_mgmt::{client::MgmtClient, AGENT_URL};
 
+use crate::utils::TIMEOUT;
+
 const CMD_CONNECT: &str = "CONNECT";
 const CMD_OK: &str = "OK";
 const SCHEME_VSOCK: &str = "VSOCK";
@@ -32,7 +34,6 @@ const SCHEME_HYBRID_VSOCK: &str = "HVSOCK";
 
 const EPOLL_EVENTS_LEN: usize = 16;
 const KATA_AGENT_VSOCK_TIMEOUT: u64 = 5;
-const TIMEOUT: Duration = Duration::from_millis(2000);
 
 type Result<T> = std::result::Result<T, Error>;
 
diff --git a/src/tools/kata-ctl/src/ops/volume_ops.rs b/src/tools/kata-ctl/src/ops/volume_ops.rs
index 94243743cf..a8c6818fe3 100644
--- a/src/tools/kata-ctl/src/ops/volume_ops.rs
+++ b/src/tools/kata-ctl/src/ops/volume_ops.rs
@@ -14,7 +14,7 @@ use kata_types::mount::{
 use nix;
 use reqwest::StatusCode;
 use slog::{info, o};
-use std::{fs, time::Duration};
+use std::fs;
 use url;
 
 use agent::ResizeVolumeRequest;
@@ -23,7 +23,8 @@ use shim_interface::shim_mgmt::{
     DIRECT_VOLUME_PATH_KEY, DIRECT_VOLUME_RESIZE_URL, DIRECT_VOLUME_STATS_URL,
 };
 
-const TIMEOUT: Duration = Duration::from_millis(2000);
+use crate::utils::TIMEOUT;
+
 const CONTENT_TYPE_JSON: &str = "application/json";
 
 macro_rules! sl {
diff --git a/src/tools/kata-ctl/src/utils.rs b/src/tools/kata-ctl/src/utils.rs
index 3b6e4daee0..f020a077e5 100644
--- a/src/tools/kata-ctl/src/utils.rs
+++ b/src/tools/kata-ctl/src/utils.rs
@@ -8,10 +8,12 @@
 use crate::arch::arch_specific;
 
 use anyhow::{anyhow, Context, Result};
-use std::fs;
+use std::{fs, time::Duration};
 
 const NON_PRIV_USER: &str = "nobody";
 
+pub const TIMEOUT: Duration = Duration::from_millis(2000);
+
 pub fn drop_privs() -> Result<()> {
     if nix::unistd::Uid::effective().is_root() {
         privdrop::PrivDrop::default()

From 731e7c763ffaf9c83739529706ef88f13cadee19 Mon Sep 17 00:00:00 2001
From: Yuan-Zhuo <yuanzhuo0118@outlook.com>
Date: Fri, 28 Jul 2023 17:30:08 +0800
Subject: [PATCH 03/49] kata-ctl: add monitor subcommand for runtime-rs The
 previous kata-monitor in golang could not communicate with runtime-rs to
 gather metrics due to different sandbox addresses. This PR adds the
 subcommand monitor in kata-ctl to gather metrics from runtime-rs and monitor
 itself.

Fixes: #5017

Signed-off-by: Yuan-Zhuo <yuanzhuo0118@outlook.com>
---
 src/tools/kata-ctl/Cargo.lock                 | 877 +++++++++++-------
 src/tools/kata-ctl/Cargo.toml                 |   7 +-
 src/tools/kata-ctl/src/args.rs                |   9 +
 src/tools/kata-ctl/src/main.rs                |  10 +-
 src/tools/kata-ctl/src/monitor/http_server.rs | 181 ++++
 src/tools/kata-ctl/src/monitor/metrics.rs     |  91 ++
 src/tools/kata-ctl/src/monitor/mod.rs         |   8 +
 src/tools/kata-ctl/src/ops/check_ops.rs       |  21 +-
 8 files changed, 867 insertions(+), 337 deletions(-)
 create mode 100644 src/tools/kata-ctl/src/monitor/http_server.rs
 create mode 100644 src/tools/kata-ctl/src/monitor/metrics.rs
 create mode 100644 src/tools/kata-ctl/src/monitor/mod.rs

diff --git a/src/tools/kata-ctl/Cargo.lock b/src/tools/kata-ctl/Cargo.lock
index c37d17b5e4..eda34fb174 100644
--- a/src/tools/kata-ctl/Cargo.lock
+++ b/src/tools/kata-ctl/Cargo.lock
@@ -2,6 +2,21 @@
 # It is not intended for manual editing.
 version = 3
 
+[[package]]
+name = "addr2line"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f4fa78e18c64fce05e902adecd7a5eed15a5e0a3439f7b0e169f0252214865e3"
+dependencies = [
+ "gimli",
+]
+
+[[package]]
+name = "adler"
+version = "1.0.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe"
+
 [[package]]
 name = "agent"
 version = "0.1.0"
@@ -20,15 +35,16 @@ dependencies = [
  "slog",
  "slog-scope",
  "tokio",
- "ttrpc",
+ "tracing",
+ "ttrpc 0.7.1",
  "url",
 ]
 
 [[package]]
 name = "aho-corasick"
-version = "0.7.20"
+version = "1.0.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
+checksum = "43f6cb1bf222025340178f382c426f13757b2960e89779dfcb319c32542a5a41"
 dependencies = [
  "memchr",
 ]
@@ -50,49 +66,58 @@ dependencies = [
 
 [[package]]
 name = "anstream"
-version = "0.2.6"
+version = "0.3.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "342258dd14006105c2b75ab1bd7543a03bdf0cfc94383303ac212a04939dff6f"
+checksum = "0ca84f3628370c59db74ee214b3263d58f9aadd9b4fe7e711fd87dc452b7f163"
 dependencies = [
  "anstyle",
  "anstyle-parse",
+ "anstyle-query",
  "anstyle-wincon",
- "concolor-override",
- "concolor-query",
+ "colorchoice",
  "is-terminal",
  "utf8parse",
 ]
 
 [[package]]
 name = "anstyle"
-version = "0.3.5"
+version = "1.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "23ea9e81bd02e310c216d080f6223c179012256e5151c41db88d12c88a1684d2"
+checksum = "3a30da5c5f2d5e72842e00bcb57657162cdabef0931f40e2deb9b4140440cecd"
 
 [[package]]
 name = "anstyle-parse"
-version = "0.1.1"
+version = "0.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a7d1bb534e9efed14f3e5f44e7dd1a4f709384023a4165199a4241e18dff0116"
+checksum = "938874ff5980b03a87c5524b3ae5b59cf99b1d6bc836848df7bc5ada9643c333"
 dependencies = [
  "utf8parse",
 ]
 
 [[package]]
-name = "anstyle-wincon"
-version = "0.2.0"
+name = "anstyle-query"
+version = "1.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c3127af6145b149f3287bb9a0d10ad9c5692dba8c53ad48285e5bec4063834fa"
+checksum = "5ca11d4be1bab0c8bc8734a9aa7bf4ee8316d462a08c6ac5052f888fef5b494b"
+dependencies = [
+ "windows-sys 0.48.0",
+]
+
+[[package]]
+name = "anstyle-wincon"
+version = "1.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "180abfa45703aebe0093f79badacc01b8fd4ea2e35118747e5811127f926e188"
 dependencies = [
  "anstyle",
- "windows-sys 0.45.0",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
 name = "anyhow"
-version = "1.0.69"
+version = "1.0.72"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "224afbd727c3d6e4b90103ece64b8d1b67fbb1973b1046c2281eed3f3803f800"
+checksum = "3b13c32d80ecc7ab747b80c3784bce54ee8a7a0cc4fbda9bf4cda2cf6fe90854"
 
 [[package]]
 name = "arc-swap"
@@ -102,13 +127,13 @@ checksum = "bddcadddf5e9015d310179a59bb28c4d4b9920ad0f11e8e14dbadf654890c9a6"
 
 [[package]]
 name = "async-trait"
-version = "0.1.64"
+version = "0.1.72"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1cd7fce9ba8c3c042128ce72d8b2ddbf3a05747efb67ea0313c635e10bda47a2"
+checksum = "cc6dde6e4ed435a4c1ee4e73592f5ba9da2151af10076cc04858746af9352d09"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -128,6 +153,21 @@ version = "1.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
 
+[[package]]
+name = "backtrace"
+version = "0.3.68"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4319208da049c43661739c5fade2ba182f09d1dc2299b32298d3a31692b17e12"
+dependencies = [
+ "addr2line",
+ "cc",
+ "cfg-if",
+ "libc",
+ "miniz_oxide",
+ "object",
+ "rustc-demangle",
+]
+
 [[package]]
 name = "base64"
 version = "0.13.1"
@@ -136,9 +176,9 @@ checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8"
 
 [[package]]
 name = "base64"
-version = "0.21.0"
+version = "0.21.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a4a4ddaa51a5bc52a6948f74c06d20aaaddb71924eab79b8c97a8c556e942d6a"
+checksum = "604178f6c5c21f02dc555784810edfb88d34ac2c73b2eae109655649ee73ce3d"
 
 [[package]]
 name = "bitflags"
@@ -147,20 +187,26 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
 
 [[package]]
-name = "bitmask-enum"
-version = "2.1.0"
+name = "bitflags"
+version = "2.3.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fd9e32d7420c85055e8107e5b2463c4eeefeaac18b52359fe9f9c08a18f342b2"
+checksum = "630be753d4e58660abd17930c71b647fe46c27ea6b63cc59e1e3851406972e42"
+
+[[package]]
+name = "bitmask-enum"
+version = "2.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "78d456f91b4c1fdebf2698214e599fec3d7f8b46e3140fb254a9ea88c970ab0a"
 dependencies = [
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "bumpalo"
-version = "3.12.0"
+version = "3.13.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0d261e256854913907f67ed06efbc3338dfe6179796deefc1ff763fc1aee5535"
+checksum = "a3e2c3daef883ecc1b5d58c15adae93470a91d425f3532ba1695849656af3fc1"
 
 [[package]]
 name = "byte-unit"
@@ -232,9 +278,9 @@ dependencies = [
 
 [[package]]
 name = "clap"
-version = "4.2.1"
+version = "4.3.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "046ae530c528f252094e4a77886ee1374437744b2bff1497aa898bbddbbb29b3"
+checksum = "5fd304a20bff958a57f04c4e96a2e7594cc4490a0e809cbd48bb6437edaa452d"
 dependencies = [
  "clap_builder",
  "clap_derive",
@@ -243,13 +289,12 @@ dependencies = [
 
 [[package]]
 name = "clap_builder"
-version = "4.2.1"
+version = "4.3.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "223163f58c9a40c3b0a43e1c4b50a9ce09f007ea2cb1ec258a687945b4b7929f"
+checksum = "01c6a3f08f1fe5662a35cfe393aec09c4df95f60ee93b7556505260f75eee9e1"
 dependencies = [
  "anstream",
  "anstyle",
- "bitflags",
  "clap_lex",
  "once_cell",
  "strsim",
@@ -257,21 +302,27 @@ dependencies = [
 
 [[package]]
 name = "clap_derive"
-version = "4.2.0"
+version = "4.3.12"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3f9644cd56d6b87dbe899ef8b053e331c0637664e9e21a33dfcdc36093f5c5c4"
+checksum = "54a9bb5758fc5dfe728d1019941681eccaf0cf8a4189b692a0ee2f2ecf90a050"
 dependencies = [
  "heck 0.4.1",
  "proc-macro2",
  "quote",
- "syn 2.0.13",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "clap_lex"
-version = "0.4.1"
+version = "0.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8a2dd5a6fe8c6e3502f568a6353e5273bbb15193ad9a89e457b9970798efbea1"
+checksum = "2da6da31387c7e4ef160ffab6d5e7f00c42626fe39aea70a7b0f1773f7dd6c1b"
+
+[[package]]
+name = "colorchoice"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "acbf1af155f9b9ef647e42cdc158db4b64a1b61f743629225fde6f3e0be2a7c7"
 
 [[package]]
 name = "common-path"
@@ -279,21 +330,6 @@ version = "1.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2382f75942f4b3be3690fe4f86365e9c853c1587d6ee58212cebf6e2a9ccd101"
 
-[[package]]
-name = "concolor-override"
-version = "1.0.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a855d4a1978dc52fb0536a04d384c2c0c1aa273597f08b77c8c4d3b2eec6037f"
-
-[[package]]
-name = "concolor-query"
-version = "0.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "88d11d52c3d7ca2e6d0040212be9e4dbbcd78b6447f535b6b561f449427944cf"
-dependencies = [
- "windows-sys 0.45.0",
-]
-
 [[package]]
 name = "core-foundation"
 version = "0.9.3"
@@ -306,15 +342,24 @@ dependencies = [
 
 [[package]]
 name = "core-foundation-sys"
-version = "0.8.3"
+version = "0.8.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5827cebf4670468b8772dd191856768aedcb1b0278a04f989f7766351917b9dc"
+checksum = "e496a50fda8aacccc86d7529e2c1e0892dbd0f898a6b5645b5561b89c3210efa"
+
+[[package]]
+name = "crc32fast"
+version = "1.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b540bd8bc810d3885c6ea91e2018302f68baba2129ab3e88f32389ee9370880d"
+dependencies = [
+ "cfg-if",
+]
 
 [[package]]
 name = "crossbeam-channel"
-version = "0.5.6"
+version = "0.5.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c2dd04ddaf88237dc3b8d8f9a3c1004b506b54b3313403944054d23c0870c521"
+checksum = "a33c2bf77f2df06183c3aa30d1e96c0695a313d4f9c453cc3762a6db39f99200"
 dependencies = [
  "cfg-if",
  "crossbeam-utils",
@@ -322,9 +367,9 @@ dependencies = [
 
 [[package]]
 name = "crossbeam-utils"
-version = "0.8.14"
+version = "0.8.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4fb766fa798726286dbbb842f174001dab8abc7b627a1dd86e0b7222a95d929f"
+checksum = "5a22b2d63d4d1dc0b7f1b6b2747dd0088008a9be28b6ddf0b1e7d335e3037294"
 dependencies = [
  "cfg-if",
 ]
@@ -337,7 +382,7 @@ checksum = "3418329ca0ad70234b9735dc4ceed10af4df60eff9c8e7b06cb5e520d92c3535"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 1.0.109",
 ]
 
 [[package]]
@@ -363,9 +408,9 @@ dependencies = [
 
 [[package]]
 name = "either"
-version = "1.8.1"
+version = "1.9.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7fcaabb2fef8c910e7f4c7ce9f67a1283a1715879a7c230ca9d6d1ae31f16d91"
+checksum = "a26ae43d7bcc3b814de94796a5e736d4029efb0ee900c12e2d54c993ad1a1e07"
 
 [[package]]
 name = "encoding_rs"
@@ -378,23 +423,23 @@ dependencies = [
 
 [[package]]
 name = "epoll"
-version = "4.3.1"
+version = "4.3.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "20df693c700404f7e19d4d6fae6b15215d2913c27955d2b9d6f2c0f537511cd0"
+checksum = "74351c3392ea1ff6cd2628e0042d268ac2371cb613252ff383b6dfa50d22fa79"
 dependencies = [
- "bitflags",
+ "bitflags 2.3.3",
  "libc",
 ]
 
 [[package]]
 name = "errno"
-version = "0.3.0"
+version = "0.3.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "50d6a0976c999d473fe89ad888d5a284e55366d9dc9038b1ba2aa15128c4afa0"
+checksum = "4bcfec3a70f97c962c307b2d2c56e358cf1d00b558d74262b5f929ee8cc7e73a"
 dependencies = [
  "errno-dragonfly",
  "libc",
- "windows-sys 0.45.0",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -420,12 +465,9 @@ dependencies = [
 
 [[package]]
 name = "fastrand"
-version = "1.8.0"
+version = "2.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a7a407cfaa3385c4ae6b23e84623d48c2798d06e3e6a1878f7f59f17b3f86499"
-dependencies = [
- "instant",
-]
+checksum = "6999dc1837253364c2ebb0704ba97994bd874e8f195d665c50b7548f6ea92764"
 
 [[package]]
 name = "fixedbitset"
@@ -433,6 +475,16 @@ version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "37ab347416e802de484e4d03c7316c48f1ecb56574dfd4a46a80f173ce1de04d"
 
+[[package]]
+name = "flate2"
+version = "1.0.26"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3b9429470923de8e8cbd4d2dc513535400b4b3fef0319fb5c4e1f520a7bef743"
+dependencies = [
+ "crc32fast",
+ "miniz_oxide",
+]
+
 [[package]]
 name = "fnv"
 version = "1.0.7"
@@ -456,18 +508,18 @@ checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b"
 
 [[package]]
 name = "form_urlencoded"
-version = "1.1.0"
+version = "1.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a9c384f161156f5260c24a097c56119f9be8c798586aecc13afbcbe7b7e26bf8"
+checksum = "a62bc1cf6f830c2ec14a513a9fb124d0a213a629668a4186f329db21fe045652"
 dependencies = [
  "percent-encoding",
 ]
 
 [[package]]
 name = "futures"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "13e2792b0ff0340399d58445b88fd9770e3489eff258a4cbc1523418f12abf84"
+checksum = "23342abe12aba583913b2e62f22225ff9c950774065e4bfb61a19cd9770fec40"
 dependencies = [
  "futures-channel",
  "futures-core",
@@ -480,9 +532,9 @@ dependencies = [
 
 [[package]]
 name = "futures-channel"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2e5317663a9089767a1ec00a487df42e0ca174b61b4483213ac24448e4664df5"
+checksum = "955518d47e09b25bbebc7a18df10b81f0c766eaf4c4f1cccef2fca5f2a4fb5f2"
 dependencies = [
  "futures-core",
  "futures-sink",
@@ -490,15 +542,15 @@ dependencies = [
 
 [[package]]
 name = "futures-core"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ec90ff4d0fe1f57d600049061dc6bb68ed03c7d2fbd697274c41805dcb3f8608"
+checksum = "4bca583b7e26f571124fe5b7561d49cb2868d79116cfa0eefce955557c6fee8c"
 
 [[package]]
 name = "futures-executor"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e8de0a35a6ab97ec8869e32a2473f4b1324459e14c29275d14b10cb1fd19b50e"
+checksum = "ccecee823288125bd88b4d7f565c9e58e41858e47ab72e8ea2d64e93624386e0"
 dependencies = [
  "futures-core",
  "futures-task",
@@ -507,38 +559,38 @@ dependencies = [
 
 [[package]]
 name = "futures-io"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bfb8371b6fb2aeb2d280374607aeabfc99d95c72edfe51692e42d3d7f0d08531"
+checksum = "4fff74096e71ed47f8e023204cfd0aa1289cd54ae5430a9523be060cdb849964"
 
 [[package]]
 name = "futures-macro"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "95a73af87da33b5acf53acfebdc339fe592ecf5357ac7c0a7734ab9d8c876a70"
+checksum = "89ca545a94061b6365f2c7355b4b32bd20df3ff95f02da9329b34ccc3bd6ee72"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "futures-sink"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f310820bb3e8cfd46c80db4d7fb8353e15dfff853a127158425f31e0be6c8364"
+checksum = "f43be4fe21a13b9781a69afa4985b0f6ee0e1afab2c6f454a8cf30e2b2237b6e"
 
 [[package]]
 name = "futures-task"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "dcf79a1bf610b10f42aea489289c5a2c478a786509693b80cd39c44ccd936366"
+checksum = "76d3d132be6c0e6aa1534069c705a74a5997a356c0dc2f86a47765e5617c5b65"
 
 [[package]]
 name = "futures-util"
-version = "0.3.26"
+version = "0.3.28"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c1d6de3acfef38d2be4b1f543f553131788603495be83da675e180c8d6b7bd1"
+checksum = "26b01e40b772d54cf6c6d721c1d1abd0647a0106a12ecaa1c186273392a69533"
 dependencies = [
  "futures-channel",
  "futures-core",
@@ -554,15 +606,21 @@ dependencies = [
 
 [[package]]
 name = "getrandom"
-version = "0.2.9"
+version = "0.2.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c85e1d9ab2eadba7e5040d4e09cbd6d072b76a557ad64e797c2cb9d4da21d7e4"
+checksum = "be4136b2a15dd319360be1c07d9933517ccf0be8f16bf62a3bee4f0d618df427"
 dependencies = [
  "cfg-if",
  "libc",
  "wasi 0.11.0+wasi-snapshot-preview1",
 ]
 
+[[package]]
+name = "gimli"
+version = "0.27.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b6c80984affa11d98d1b88b66ac8853f143217b399d3c74116778ff8fdb4ed2e"
+
 [[package]]
 name = "glob"
 version = "0.3.1"
@@ -571,9 +629,9 @@ checksum = "d2fabcfbdc87f4758337ca535fb41a6d701b65693ce38287d856d1674551ec9b"
 
 [[package]]
 name = "h2"
-version = "0.3.15"
+version = "0.3.20"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5f9f29bc9dda355256b2916cf526ab02ce0aeaaaf2bad60d65ef3f12f11dd0f4"
+checksum = "97ec8491ebaf99c8eaa73058b045fe58073cd6be7f596ac993ced0b0a0c01049"
 dependencies = [
  "bytes 1.4.0",
  "fnv",
@@ -620,18 +678,9 @@ dependencies = [
 
 [[package]]
 name = "hermit-abi"
-version = "0.2.6"
+version = "0.3.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ee512640fe35acbfb4bb779db6f0d80704c2cacfa2e39b601ef3e3f47d1ae4c7"
-dependencies = [
- "libc",
-]
-
-[[package]]
-name = "hermit-abi"
-version = "0.3.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fed44880c466736ef9a5c5b5facefb5ed0785676d0c02d612db14e54f0d84286"
+checksum = "443144c8cdadd93ebf52ddb4056d257f5b52c04d3c804e657d19eb73fc33668b"
 
 [[package]]
 name = "hex"
@@ -641,9 +690,9 @@ checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
 
 [[package]]
 name = "http"
-version = "0.2.8"
+version = "0.2.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "75f43d41e26995c17e71ee126451dd3941010b0514a81a9d11f3b341debc2399"
+checksum = "bd6effc99afb63425aff9b05836f029929e345a6148a14b7ecd5ab67af944482"
 dependencies = [
  "bytes 1.4.0",
  "fnv",
@@ -675,9 +724,9 @@ checksum = "c4a1e36c821dbe04574f602848a19f742f4fb3c98d40449f11bcad18d6b17421"
 
 [[package]]
 name = "hyper"
-version = "0.14.24"
+version = "0.14.27"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5e011372fa0b68db8350aa7a248930ecc7839bf46d8485577d69f117a75f164c"
+checksum = "ffb1cfd654a8219eaef89881fdb3bb3b1cdc5fa75ded05d6933b2b382e395468"
 dependencies = [
  "bytes 1.4.0",
  "futures-channel",
@@ -699,10 +748,11 @@ dependencies = [
 
 [[package]]
 name = "hyper-rustls"
-version = "0.23.2"
+version = "0.24.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1788965e61b367cd03a62950836d5cd41560c3577d90e40e0819373194d1661c"
+checksum = "8d78e1e73ec14cf7375674f74d7dde185c8206fd9dea6fb6295e8a98098aaa97"
 dependencies = [
+ "futures-util",
  "http",
  "hyper",
  "rustls",
@@ -761,9 +811,9 @@ dependencies = [
 
 [[package]]
 name = "idna"
-version = "0.3.0"
+version = "0.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e14ddfc70884202db2244c223200c204c2bda1bc6e0998d11b5e024d657209e6"
+checksum = "7d20d6b07bfbc108882d88ed8e37d39636dcc260e15e30c45e6ba089610b917c"
 dependencies = [
  "unicode-bidi",
  "unicode-normalization",
@@ -771,9 +821,9 @@ dependencies = [
 
 [[package]]
 name = "indexmap"
-version = "1.9.2"
+version = "1.9.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1885e79c1fc4b10f0e172c475f458b7f7b93061064d98c3293e98c5ba0c8b399"
+checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99"
 dependencies = [
  "autocfg",
  "hashbrown",
@@ -790,11 +840,11 @@ dependencies = [
 
 [[package]]
 name = "io-lifetimes"
-version = "1.0.10"
+version = "1.0.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9c66c74d2ae7e79a5a8f7ac924adbe38ee42a859c6539ad869eb51f0b52dc220"
+checksum = "eae7b9aee968036d54dce06cebaefd919e4472e753296daccd6d344e3e2df0c2"
 dependencies = [
- "hermit-abi 0.3.1",
+ "hermit-abi 0.3.2",
  "libc",
  "windows-sys 0.48.0",
 ]
@@ -810,20 +860,19 @@ dependencies = [
 
 [[package]]
 name = "ipnet"
-version = "2.7.1"
+version = "2.8.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "30e22bd8629359895450b59ea7a776c850561b96a3b1d31321c1949d9e6c9146"
+checksum = "28b29a3cd74f0f4598934efe3aeba42bae0eb4680554128851ebbecb02af14e6"
 
 [[package]]
 name = "is-terminal"
-version = "0.4.6"
+version = "0.4.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "256017f749ab3117e93acb91063009e1f1bb56d03965b14c2c8df4eb02c524d8"
+checksum = "cb0889898416213fab133e1d33a0e5858a48177452750691bde3666d0fdbaf8b"
 dependencies = [
- "hermit-abi 0.3.1",
- "io-lifetimes",
- "rustix",
- "windows-sys 0.45.0",
+ "hermit-abi 0.3.2",
+ "rustix 0.38.4",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -837,15 +886,15 @@ dependencies = [
 
 [[package]]
 name = "itoa"
-version = "1.0.5"
+version = "1.0.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fad582f4b9e86b6caa621cabeb0963332d92eea04729ab12892c2533951e6440"
+checksum = "af150ab688ff2122fcef229be89cb50dd66af9e01a4ff320cc137eecc9bacc38"
 
 [[package]]
 name = "js-sys"
-version = "0.3.61"
+version = "0.3.64"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "445dde2150c55e483f3d8416706b97ec8e8237c307e5b7b4b8dd15e6af2a0730"
+checksum = "c5f195fe497f702db0f318b07fdd68edb16955aed830df8363d837542f8f935a"
 dependencies = [
  "wasm-bindgen",
 ]
@@ -863,11 +912,14 @@ dependencies = [
  "hyper",
  "kata-sys-util",
  "kata-types",
+ "lazy_static",
  "libc",
  "logging",
  "micro_http",
  "nix 0.25.1",
  "privdrop",
+ "procfs 0.12.0",
+ "prometheus",
  "reqwest",
  "safe-path",
  "semver",
@@ -885,6 +937,7 @@ dependencies = [
  "thiserror",
  "tokio",
  "toml",
+ "ttrpc 0.6.1",
  "url",
  "vmm-sys-util",
 ]
@@ -943,21 +996,27 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
 
 [[package]]
 name = "libc"
-version = "0.2.139"
+version = "0.2.147"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "201de327520df007757c1f0adce6e827fe8562fbc28bfd9c15571c66ca1f5f79"
+checksum = "b4668fb0ea861c1df094127ac5f1da3409a82116a4ba74fca2e58ef927159bb3"
 
 [[package]]
 name = "linux-raw-sys"
-version = "0.3.1"
+version = "0.1.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d59d8c75012853d2e872fb56bc8a2e53718e2cafe1a4c823143141c6d90c322f"
+checksum = "f051f77a7c8e6957c0696eac88f26b0117e54f52d3fc682ab19397a8812846a4"
+
+[[package]]
+name = "linux-raw-sys"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09fc20d2ca12cb9f044c93e3bd6d32d523e6e2ec3db4f7b2939cd99026ecd3f0"
 
 [[package]]
 name = "lock_api"
-version = "0.4.9"
+version = "0.4.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "435011366fe56583b16cf956f9df0095b405b82d76425bc8981c0e22e60ec4df"
+checksum = "c1cc9717a20b1bb222f333e6a92fd32f7d8a18ddc5a3191a11af45dcbf4dcd16"
 dependencies = [
  "autocfg",
  "scopeguard",
@@ -965,12 +1024,9 @@ dependencies = [
 
 [[package]]
 name = "log"
-version = "0.4.17"
+version = "0.4.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e"
-dependencies = [
- "cfg-if",
-]
+checksum = "b06a4cde4c0f271a446782e3eff8de789548ce57dbc8eca9292c27f4a42004b4"
 
 [[package]]
 name = "logging"
@@ -1019,20 +1075,28 @@ dependencies = [
 
 [[package]]
 name = "mime"
-version = "0.3.16"
+version = "0.3.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2a60c7ce501c71e03a9c9c0d35b861413ae925bd979cc7a4e30d060069aaac8d"
+checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
+
+[[package]]
+name = "miniz_oxide"
+version = "0.7.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e7810e0be55b428ada41041c41f32c9f1a42817901b4ccf45fa3d4b6561e74c7"
+dependencies = [
+ "adler",
+]
 
 [[package]]
 name = "mio"
-version = "0.8.5"
+version = "0.8.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e5d732bc30207a6423068df043e3d02e0735b155ad7ce1a6f76fe2baa5b158de"
+checksum = "927a765cd3fc26206e66b296465fa9d3e5ab003e651c1b3c060e7956d96b19d2"
 dependencies = [
  "libc",
- "log",
  "wasi 0.11.0+wasi-snapshot-preview1",
- "windows-sys 0.42.0",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -1065,7 +1129,7 @@ version = "0.23.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8f3790c00a0150112de0f4cd161e3d7fc4b2d8a5542ffc35f099a2562aecb35c"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cc",
  "cfg-if",
  "libc",
@@ -1078,7 +1142,7 @@ version = "0.24.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fa52e972a9a719cecb6864fb88568781eb706bac2cd1d4f04a648542dbf78069"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if",
  "libc",
  "memoffset 0.6.5",
@@ -1091,7 +1155,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f346ff70e7dbfd675fe90590b92d59ef2de15a8779ae305ebcbfd3f0caf59be4"
 dependencies = [
  "autocfg",
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if",
  "libc",
  "memoffset 0.6.5",
@@ -1104,7 +1168,7 @@ version = "0.26.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bfdda3d196821d6af13126e40375cdf7da646a96114af134d5f417a9a1dc8e1a"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if",
  "libc",
  "memoffset 0.7.1",
@@ -1114,20 +1178,20 @@ dependencies = [
 
 [[package]]
 name = "num-traits"
-version = "0.2.15"
+version = "0.2.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "578ede34cf02f8924ab9447f50c28075b4d3e5b269972345e7e0372b38c6cdcd"
+checksum = "f30b0abd723be7e2ffca1272140fac1a2f084c77ec3e123c192b66af1ee9e6c2"
 dependencies = [
  "autocfg",
 ]
 
 [[package]]
 name = "num_cpus"
-version = "1.15.0"
+version = "1.16.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0fac9e2da13b5eb447a6ce3d392f23a29d8694bff781bf03a16cd9ac8697593b"
+checksum = "4161fcb6d602d4d2081af7c3a45852d875a03dd337a6bfdd6e06407b61342a43"
 dependencies = [
- "hermit-abi 0.2.6",
+ "hermit-abi 0.3.2",
  "libc",
 ]
 
@@ -1140,6 +1204,15 @@ dependencies = [
  "libc",
 ]
 
+[[package]]
+name = "object"
+version = "0.31.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8bda667d9f2b5051b8833f59f3bf748b28ef54f850f4fcb389a252aa383866d1"
+dependencies = [
+ "memchr",
+]
+
 [[package]]
 name = "oci"
 version = "0.1.0"
@@ -1152,17 +1225,17 @@ dependencies = [
 
 [[package]]
 name = "once_cell"
-version = "1.17.0"
+version = "1.18.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6f61fba1741ea2b3d6a1e3178721804bb716a68a6aeba1149b5d52e3d464ea66"
+checksum = "dd8b5dd2ae5ed71462c540258bedcb51965123ad7e7ccf4b9a8cafaa4a63576d"
 
 [[package]]
 name = "openssl"
-version = "0.10.45"
+version = "0.10.55"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b102428fd03bc5edf97f62620f7298614c45cedf287c271e7ed450bbaf83f2e1"
+checksum = "345df152bc43501c5eb9e4654ff05f794effb78d4efe3d53abc158baddc0703d"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "cfg-if",
  "foreign-types",
  "libc",
@@ -1173,13 +1246,13 @@ dependencies = [
 
 [[package]]
 name = "openssl-macros"
-version = "0.1.0"
+version = "0.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b501e44f11665960c7e7fcf062c7d96a14ade4aa98116c004b2e37b5be7d736c"
+checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -1190,11 +1263,10 @@ checksum = "ff011a302c396a5197692431fc1948019154afc178baf7d8e37367442a4601cf"
 
 [[package]]
 name = "openssl-sys"
-version = "0.9.80"
+version = "0.9.90"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "23bbbf7854cd45b83958ebe919f0e8e516793727652e27fda10a8384cfc790b7"
+checksum = "374533b0e45f3a7ced10fcaeccca020e66656bc03dac384f852e4e5a7a8104a6"
 dependencies = [
- "autocfg",
  "cc",
  "libc",
  "pkg-config",
@@ -1209,7 +1281,17 @@ checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99"
 dependencies = [
  "instant",
  "lock_api",
- "parking_lot_core",
+ "parking_lot_core 0.8.6",
+]
+
+[[package]]
+name = "parking_lot"
+version = "0.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f"
+dependencies = [
+ "lock_api",
+ "parking_lot_core 0.9.8",
 ]
 
 [[package]]
@@ -1221,16 +1303,29 @@ dependencies = [
  "cfg-if",
  "instant",
  "libc",
- "redox_syscall",
+ "redox_syscall 0.2.16",
  "smallvec",
  "winapi",
 ]
 
 [[package]]
-name = "percent-encoding"
-version = "2.2.0"
+name = "parking_lot_core"
+version = "0.9.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "478c572c3d73181ff3c2539045f6eb99e5491218eae919370993b890cdbdd98e"
+checksum = "93f00c865fe7cabf650081affecd3871070f26767e7b2070a3ffae14c654b447"
+dependencies = [
+ "cfg-if",
+ "libc",
+ "redox_syscall 0.3.5",
+ "smallvec",
+ "windows-targets 0.48.1",
+]
+
+[[package]]
+name = "percent-encoding"
+version = "2.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9b2a4787296e9989611394c33f193f676704af1686e70b8f8033ab5ba9a35a94"
 
 [[package]]
 name = "petgraph"
@@ -1244,29 +1339,29 @@ dependencies = [
 
 [[package]]
 name = "pin-project"
-version = "1.0.12"
+version = "1.1.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ad29a609b6bcd67fee905812e544992d216af9d755757c05ed2d0e15a74c6ecc"
+checksum = "030ad2bc4db10a8944cb0d837f158bdfec4d4a4873ab701a95046770d11f8842"
 dependencies = [
  "pin-project-internal",
 ]
 
 [[package]]
 name = "pin-project-internal"
-version = "1.0.12"
+version = "1.1.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "069bdb1e05adc7a8990dce9cc75370895fbe4e3d58b9b73bf1aee56359344a55"
+checksum = "ec2e072ecce94ec471b13398d5402c188e76ac03cf74dd1a975161b23a3f6d9c"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "pin-project-lite"
-version = "0.2.9"
+version = "0.2.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e0a7ae3ac2f1173085d398531c705756c94a4c56843785df85a60c1a0afac116"
+checksum = "4c40d25201921e5ff0c862a505c6557ea88568a4e3ace775ab55e93f2f4f9d57"
 
 [[package]]
 name = "pin-utils"
@@ -1276,9 +1371,9 @@ checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
 
 [[package]]
 name = "pkg-config"
-version = "0.3.26"
+version = "0.3.27"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6ac9a59f73473f1b8d852421e59e64809f025994837ef743615c6d0c5b305160"
+checksum = "26072860ba924cbfa98ea39c8c19b4dd6a4a25423dbdf219c1eca91aa0cf6964"
 
 [[package]]
 name = "ppv-lite86"
@@ -1288,9 +1383,9 @@ checksum = "5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de"
 
 [[package]]
 name = "privdrop"
-version = "0.5.3"
+version = "0.5.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "81ed9e5437d82d5f2cde999a21571474c5f09b3d76e33eab94bf0e8e42a4fd96"
+checksum = "9bc12de3935536ed9b69488faea4450a298dac44179b54f71806e63f55034bf9"
 dependencies = [
  "libc",
  "nix 0.26.2",
@@ -1298,13 +1393,58 @@ dependencies = [
 
 [[package]]
 name = "proc-macro2"
-version = "1.0.56"
+version = "1.0.66"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2b63bdb0cd06f1f4dedf69b254734f9b45af66e4a031e42a7480257d9898b435"
+checksum = "18fb31db3f9bddb2ea821cde30a9f70117e3f119938b5ee630b7403aa6e2ead9"
 dependencies = [
  "unicode-ident",
 ]
 
+[[package]]
+name = "procfs"
+version = "0.12.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0941606b9934e2d98a3677759a971756eb821f75764d0e0d26946d08e74d9104"
+dependencies = [
+ "bitflags 1.3.2",
+ "byteorder",
+ "chrono",
+ "flate2",
+ "hex",
+ "lazy_static",
+ "libc",
+]
+
+[[package]]
+name = "procfs"
+version = "0.14.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b1de8dacb0873f77e6aefc6d71e044761fcc68060290f5b1089fcdf84626bb69"
+dependencies = [
+ "bitflags 1.3.2",
+ "byteorder",
+ "hex",
+ "lazy_static",
+ "rustix 0.36.15",
+]
+
+[[package]]
+name = "prometheus"
+version = "0.13.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "449811d15fbdf5ceb5c1144416066429cf82316e2ec8ce0c1f6f8a02e7bbcf8c"
+dependencies = [
+ "cfg-if",
+ "fnv",
+ "lazy_static",
+ "libc",
+ "memchr",
+ "parking_lot 0.12.1",
+ "procfs 0.14.2",
+ "protobuf 2.28.0",
+ "thiserror",
+]
+
 [[package]]
 name = "prost"
 version = "0.8.0"
@@ -1343,7 +1483,7 @@ dependencies = [
  "itertools",
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 1.0.109",
 ]
 
 [[package]]
@@ -1397,6 +1537,16 @@ dependencies = [
  "thiserror",
 ]
 
+[[package]]
+name = "protobuf-codegen-pure"
+version = "2.28.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "95a29399fc94bcd3eeaa951c715f7bea69409b2445356b00519740bcd6ddd865"
+dependencies = [
+ "protobuf 2.28.0",
+ "protobuf-codegen 2.28.0",
+]
+
 [[package]]
 name = "protobuf-parse"
 version = "3.2.0"
@@ -1429,15 +1579,15 @@ dependencies = [
  "async-trait",
  "oci",
  "protobuf 3.2.0",
- "ttrpc",
+ "ttrpc 0.7.1",
  "ttrpc-codegen",
 ]
 
 [[package]]
 name = "quote"
-version = "1.0.26"
+version = "1.0.32"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4424af4bf778aae2051a77b60283332f386554255d722233d09fbfc7e30da2fc"
+checksum = "50f3b39ccfb720540debaa0164757101c08ecb8d326b15358ce76a62c7e85965"
 dependencies = [
  "proc-macro2",
 ]
@@ -1478,7 +1628,16 @@ version = "0.2.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
+]
+
+[[package]]
+name = "redox_syscall"
+version = "0.3.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "567664f262709473930a4bf9e51bf2ebf3348f2e748ccc50dea20646858f8f29"
+dependencies = [
+ "bitflags 1.3.2",
 ]
 
 [[package]]
@@ -1488,15 +1647,27 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b033d837a7cf162d7993aded9304e30a83213c648b6e389db233191f891e5c2b"
 dependencies = [
  "getrandom",
- "redox_syscall",
+ "redox_syscall 0.2.16",
  "thiserror",
 ]
 
 [[package]]
 name = "regex"
-version = "1.7.1"
+version = "1.9.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "48aaa5748ba571fb95cd2c85c09f629215d3a6ece942baa100950af03a34f733"
+checksum = "b2eae68fc220f7cf2532e4494aded17545fce192d59cd996e0fe7887f4ceb575"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "39354c10dd07468c2e73926b23bb9c2caca74c5501e38a35da70406f1d923310"
 dependencies = [
  "aho-corasick",
  "memchr",
@@ -1505,26 +1676,17 @@ dependencies = [
 
 [[package]]
 name = "regex-syntax"
-version = "0.6.28"
+version = "0.7.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "456c603be3e8d448b072f410900c09faf164fbce2d480456f50eea6e25f9c848"
-
-[[package]]
-name = "remove_dir_all"
-version = "0.5.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3acd125665422973a33ac9d3dd2df85edad0f4ae9b00dafb1a05e43a9f5ef8e7"
-dependencies = [
- "winapi",
-]
+checksum = "e5ea92a5b6195c6ef2a0295ea818b312502c6fc94dde986c5553242e18fd4ce2"
 
 [[package]]
 name = "reqwest"
-version = "0.11.14"
+version = "0.11.18"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "21eed90ec8570952d53b772ecf8f206aa1ec9a3d76b2521c56c42973f2d91ee9"
+checksum = "cde824a14b7c14f85caff81225f411faacc04a2013f41670f41443742b1c1c55"
 dependencies = [
- "base64 0.21.0",
+ "base64 0.21.2",
  "bytes 1.4.0",
  "encoding_rs",
  "futures-core",
@@ -1576,51 +1738,80 @@ dependencies = [
 ]
 
 [[package]]
-name = "rustix"
-version = "0.37.3"
+name = "rustc-demangle"
+version = "0.1.23"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "62b24138615de35e32031d041a09032ef3487a616d901ca4db224e7d557efae2"
+checksum = "d626bb9dae77e28219937af045c257c28bfd3f69333c512553507f5f9798cb76"
+
+[[package]]
+name = "rustix"
+version = "0.36.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c37f1bd5ef1b5422177b7646cba67430579cfe2ace80f284fee876bca52ad941"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "errno",
  "io-lifetimes",
  "libc",
- "linux-raw-sys",
+ "linux-raw-sys 0.1.4",
  "windows-sys 0.45.0",
 ]
 
 [[package]]
-name = "rustls"
-version = "0.20.8"
+name = "rustix"
+version = "0.38.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fff78fc74d175294f4e83b28343315ffcfb114b156f0185e9741cb5570f50e2f"
+checksum = "0a962918ea88d644592894bc6dc55acc6c0956488adcebbfb6e273506b7fd6e5"
+dependencies = [
+ "bitflags 2.3.3",
+ "errno",
+ "libc",
+ "linux-raw-sys 0.4.3",
+ "windows-sys 0.48.0",
+]
+
+[[package]]
+name = "rustls"
+version = "0.21.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "79ea77c539259495ce8ca47f53e66ae0330a8819f67e23ac96ca02f50e7b7d36"
 dependencies = [
  "log",
  "ring",
+ "rustls-webpki",
  "sct",
- "webpki",
 ]
 
 [[package]]
 name = "rustls-pemfile"
-version = "1.0.2"
+version = "1.0.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d194b56d58803a43635bdc398cd17e383d6f71f9182b9a192c127ca42494a59b"
+checksum = "2d3987094b1d07b653b7dfdc3f70ce9a1da9c51ac18c1b06b662e4f9a0e9f4b2"
 dependencies = [
- "base64 0.21.0",
+ "base64 0.21.2",
+]
+
+[[package]]
+name = "rustls-webpki"
+version = "0.101.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "513722fd73ad80a71f72b61009ea1b584bcfa1483ca93949c8f290298837fa59"
+dependencies = [
+ "ring",
+ "untrusted",
 ]
 
 [[package]]
 name = "rustversion"
-version = "1.0.11"
+version = "1.0.14"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5583e89e108996506031660fe09baa5011b9dd0341b89029313006d1fb508d70"
+checksum = "7ffc183a10b4478d04cbbbfc96d0873219d962dd5accaff2ffbd4ceb7df837f4"
 
 [[package]]
 name = "ryu"
-version = "1.0.12"
+version = "1.0.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b4b9743ed687d4b4bcedf9ff5eaa7398495ae14e61cba0a295704edbc7decde"
+checksum = "1ad4cc8da4ef723ed60bced201181d83791ad433213d8c24efffda1eec85d741"
 
 [[package]]
 name = "safe-path"
@@ -1631,18 +1822,18 @@ dependencies = [
 
 [[package]]
 name = "schannel"
-version = "0.1.21"
+version = "0.1.22"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "713cfb06c7059f3588fb8044c0fad1d09e3c01d225e25b9220dbfdcf16dbb1b3"
+checksum = "0c3733bf4cf7ea0880754e19cb5a462007c4a8c1914bff372ccc95b464f1df88"
 dependencies = [
- "windows-sys 0.42.0",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
 name = "scopeguard"
-version = "1.1.0"
+version = "1.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
+checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
 
 [[package]]
 name = "sct"
@@ -1656,11 +1847,11 @@ dependencies = [
 
 [[package]]
 name = "security-framework"
-version = "2.8.2"
+version = "2.9.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a332be01508d814fed64bf28f798a146d73792121129962fdf335bb3c49a4254"
+checksum = "05b64fb303737d99b81884b2c63433e9ae28abebe5eb5045dcdd175dc2ecf4de"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "core-foundation",
  "core-foundation-sys",
  "libc",
@@ -1669,9 +1860,9 @@ dependencies = [
 
 [[package]]
 name = "security-framework-sys"
-version = "2.8.0"
+version = "2.9.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "31c9bb296072e961fcbd8853511dd39c2d8be2deb1e17c6860b1d30732b323b4"
+checksum = "e932934257d3b408ed8f30db49d85ea163bfe74961f017f405b025af298f0c7a"
 dependencies = [
  "core-foundation-sys",
  "libc",
@@ -1679,35 +1870,35 @@ dependencies = [
 
 [[package]]
 name = "semver"
-version = "1.0.16"
+version = "1.0.18"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "58bc9567378fc7690d6b2addae4e60ac2eeea07becb2c64b9f218b53865cba2a"
+checksum = "b0293b4b29daaf487284529cc2f5675b8e57c61f70167ba415a463651fd6a918"
 
 [[package]]
 name = "serde"
-version = "1.0.152"
+version = "1.0.177"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bb7d1f0d3021d347a83e556fc4683dea2ea09d87bccdf88ff5c12545d89d5efb"
+checksum = "63ba2516aa6bf82e0b19ca8b50019d52df58455d3cf9bdaf6315225fdd0c560a"
 dependencies = [
  "serde_derive",
 ]
 
 [[package]]
 name = "serde_derive"
-version = "1.0.152"
+version = "1.0.177"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "af487d118eecd09402d70a5d72551860e788df87b464af30e5ea6a38c75c541e"
+checksum = "401797fe7833d72109fedec6bfcbe67c0eed9b99772f26eb8afd261f0abc6fd3"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
 ]
 
 [[package]]
 name = "serde_json"
-version = "1.0.93"
+version = "1.0.104"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cad406b69c91885b5107daf2c29572f6c8cdb3c66826821e286c533490c0bc76"
+checksum = "076066c5f1078eac5b722a31827a8832fe108bed65dfa75e233c89f8206e976c"
 dependencies = [
  "itoa",
  "ryu",
@@ -1733,7 +1924,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e0bccbcf40c8938196944a3da0e133e031a33f4d6b72db3bda3cc556e361905d"
 dependencies = [
  "lazy_static",
- "parking_lot",
+ "parking_lot 0.11.2",
  "serial_test_derive",
 ]
 
@@ -1745,7 +1936,7 @@ checksum = "b2acd6defeddb41eb60bb468f8825d0cfd0c2a76bc03bfd235b6a1dc4f6a1ad5"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 1.0.109",
 ]
 
 [[package]]
@@ -1759,10 +1950,19 @@ dependencies = [
 ]
 
 [[package]]
-name = "slab"
-version = "0.4.7"
+name = "signal-hook-registry"
+version = "1.4.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4614a76b2a8be0058caa9dbbaf66d988527d86d003c11a94fbd335d7661edcef"
+checksum = "d8229b473baa5980ac72ef434c4415e70c4b5e71b423043adb4ba059f89c99a1"
+dependencies = [
+ "libc",
+]
+
+[[package]]
+name = "slab"
+version = "0.4.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6528351c9bc8ab22353f9d776db39a20288e8d6c37ef8cfe3317cf875eecfc2d"
 dependencies = [
  "autocfg",
 ]
@@ -1794,7 +1994,7 @@ dependencies = [
  "serde",
  "serde_json",
  "slog",
- "time 0.3.17",
+ "time 0.3.23",
 ]
 
 [[package]]
@@ -1818,14 +2018,14 @@ dependencies = [
  "slog",
  "term",
  "thread_local",
- "time 0.3.17",
+ "time 0.3.23",
 ]
 
 [[package]]
 name = "smallvec"
-version = "1.10.0"
+version = "1.11.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a507befe795404456341dfab10cef66ead4c041f62b8b11bbb92bffe5d0953e0"
+checksum = "62bb4feee49fdd9f707ef802e22365a35de4b7b299de4763d44bfea899442ff9"
 
 [[package]]
 name = "socket2"
@@ -1871,7 +2071,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "rustversion",
- "syn 1.0.107",
+ "syn 1.0.109",
 ]
 
 [[package]]
@@ -1886,9 +2086,9 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "1.0.107"
+version = "1.0.109"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1f4064b5b16e03ae50984a5a8ed5d4f8803e6bc1fd170a3cda91a1be4b18e3f5"
+checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -1897,9 +2097,9 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "2.0.13"
+version = "2.0.27"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4c9da457c5285ac1f936ebd076af6dac17a61cfe7826f2076b4d015cf47bc8ec"
+checksum = "b60f673f44a8255b9c8c657daf66a596d435f2da81a555b06dc644d080ba45e0"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -1924,16 +2124,15 @@ checksum = "f764005d11ee5f36500a149ace24e00e3da98b0158b3e2d53a7495660d3f4d60"
 
 [[package]]
 name = "tempfile"
-version = "3.3.0"
+version = "3.7.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5cdb1ef4eaeeaddc8fbd371e5017057064af0911902ef36b39801f67cc6d79e4"
+checksum = "5486094ee78b2e5038a6382ed7645bc084dc2ec433426ca4c3cb61e2007b8998"
 dependencies = [
  "cfg-if",
  "fastrand",
- "libc",
- "redox_syscall",
- "remove_dir_all",
- "winapi",
+ "redox_syscall 0.3.5",
+ "rustix 0.38.4",
+ "windows-sys 0.48.0",
 ]
 
 [[package]]
@@ -1956,22 +2155,22 @@ dependencies = [
 
 [[package]]
 name = "thiserror"
-version = "1.0.38"
+version = "1.0.44"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6a9cd18aa97d5c45c6603caea1da6628790b37f7a34b6ca89522331c5180fed0"
+checksum = "611040a08a0439f8248d1990b111c95baa9c704c805fa1f62104b39655fd7f90"
 dependencies = [
  "thiserror-impl",
 ]
 
 [[package]]
 name = "thiserror-impl"
-version = "1.0.38"
+version = "1.0.44"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1fb327af4685e4d03fa8cbcf1716380da910eeb2bb8be417e7f9fd3fb164f36f"
+checksum = "090198534930841fab3a5d1bb637cde49e339654e606195f8d9c76eeb081dc96"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -1997,9 +2196,9 @@ dependencies = [
 
 [[package]]
 name = "time"
-version = "0.3.17"
+version = "0.3.23"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a561bf4617eebd33bca6434b988f39ed798e527f51a1e797d0ee4f61c0a38376"
+checksum = "59e399c068f43a5d116fedaf73b203fa4f9c519f17e2b34f63221d3792f81446"
 dependencies = [
  "itoa",
  "libc",
@@ -2011,15 +2210,15 @@ dependencies = [
 
 [[package]]
 name = "time-core"
-version = "0.1.0"
+version = "0.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2e153e1f1acaef8acc537e68b44906d2db6436e2b35ac2c6b42640fff91f00fd"
+checksum = "7300fbefb4dadc1af235a9cef3737cea692a9d97e1b9cbcd4ebdae6f8868e6fb"
 
 [[package]]
 name = "time-macros"
-version = "0.2.6"
+version = "0.2.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d967f99f534ca7e495c575c62638eebc2898a8c84c119b89e250477bc4ba16b2"
+checksum = "96ba15a897f3c86766b757e5ac7221554c6750054d74d5b28844fce5fb36a6c4"
 dependencies = [
  "time-core",
 ]
@@ -2041,16 +2240,18 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
 
 [[package]]
 name = "tokio"
-version = "1.28.1"
+version = "1.29.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0aa32867d44e6f2ce3385e89dceb990188b8bb0fb25b0cf576647a6f98ac5105"
+checksum = "532826ff75199d5833b9d2c5fe410f29235e25704ee5f0ef599fb51c21f4a4da"
 dependencies = [
  "autocfg",
+ "backtrace",
  "bytes 1.4.0",
  "libc",
  "mio",
  "num_cpus",
  "pin-project-lite",
+ "signal-hook-registry",
  "socket2",
  "tokio-macros",
  "windows-sys 0.48.0",
@@ -2064,7 +2265,7 @@ checksum = "630bdcf245f78637c13ec01ffae6187cca34625e8c63150d424b59e55af2675e"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 2.0.13",
+ "syn 2.0.27",
 ]
 
 [[package]]
@@ -2079,20 +2280,19 @@ dependencies = [
 
 [[package]]
 name = "tokio-rustls"
-version = "0.23.4"
+version = "0.24.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c43ee83903113e03984cb9e5cebe6c04a5116269e900e3ddba8f068a62adda59"
+checksum = "c28327cf380ac148141087fbfb9de9d7bd4e84ab5d2c28fbc911d753de8a7081"
 dependencies = [
  "rustls",
  "tokio",
- "webpki",
 ]
 
 [[package]]
 name = "tokio-util"
-version = "0.7.7"
+version = "0.7.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5427d89453009325de0d8f342c9490009f76e999cb7672d77e46267448f7e6b2"
+checksum = "806fe8c2c87eccc8b3267cbae29ed3ab2d0bd37fca70ab622e46aaa9375ddb7d"
 dependencies = [
  "bytes 1.4.0",
  "futures-core",
@@ -2138,14 +2338,26 @@ checksum = "8ce8c33a8d48bd45d624a6e523445fd21ec13d3653cd51f681abf67418f54eb8"
 dependencies = [
  "cfg-if",
  "pin-project-lite",
+ "tracing-attributes",
  "tracing-core",
 ]
 
 [[package]]
-name = "tracing-core"
-version = "0.1.30"
+name = "tracing-attributes"
+version = "0.1.26"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "24eb03ba0eab1fd845050058ce5e616558e8f8d8fca633e6b163fe25c797213a"
+checksum = "5f4f31f56159e98206da9efd823404b79b6ef3143b4a7ab76e67b1751b25a4ab"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.27",
+]
+
+[[package]]
+name = "tracing-core"
+version = "0.1.31"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0955b8137a1df6f1a2e9a37d8a6656291ff0297c1a97c24e0d8425fe2312f79a"
 dependencies = [
  "once_cell",
 ]
@@ -2156,6 +2368,21 @@ version = "0.2.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "3528ecfd12c466c6f163363caf2d02a71161dd5e1cc6ae7b34207ea2d42d81ed"
 
+[[package]]
+name = "ttrpc"
+version = "0.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2ecfff459a859c6ba6668ff72b34c2f1d94d9d58f7088414c2674ad0f31cc7d8"
+dependencies = [
+ "byteorder",
+ "libc",
+ "log",
+ "nix 0.23.2",
+ "protobuf 2.28.0",
+ "protobuf-codegen-pure",
+ "thiserror",
+]
+
 [[package]]
 name = "ttrpc"
 version = "0.7.1"
@@ -2204,15 +2431,15 @@ dependencies = [
 
 [[package]]
 name = "unicode-bidi"
-version = "0.3.10"
+version = "0.3.13"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d54675592c1dbefd78cbd98db9bacd89886e1ca50692a0692baefffdeb92dd58"
+checksum = "92888ba5573ff080736b3648696b70cafad7d250551175acbaa4e0385b3e1460"
 
 [[package]]
 name = "unicode-ident"
-version = "1.0.6"
+version = "1.0.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "84a22b9f218b40614adcb3f4ff08b703773ad44fa9423e4e0d346d5db86e4ebc"
+checksum = "301abaae475aa91687eb82514b328ab47a211a533026cb25fc3e519b86adfc3c"
 
 [[package]]
 name = "unicode-normalization"
@@ -2237,9 +2464,9 @@ checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a"
 
 [[package]]
 name = "url"
-version = "2.3.1"
+version = "2.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0d68c799ae75762b8c3fe375feb6600ef5602c883c5d21eb51c09f22b83c4643"
+checksum = "50bff7831e19200a85b17131d085c25d7811bc4e186efdaf54bbd132994a88cb"
 dependencies = [
  "form_urlencoded",
  "idna",
@@ -2264,7 +2491,7 @@ version = "0.11.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dd64fe09d8e880e600c324e7d664760a17f56e9672b7495a86381b49e4f72f46"
 dependencies = [
- "bitflags",
+ "bitflags 1.3.2",
  "libc",
 ]
 
@@ -2280,11 +2507,10 @@ dependencies = [
 
 [[package]]
 name = "want"
-version = "0.3.0"
+version = "0.3.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1ce8a968cb1cd110d136ff8b819a556d6fb6d919363c61534f6860c7eb172ba0"
+checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
 dependencies = [
- "log",
  "try-lock",
 ]
 
@@ -2302,9 +2528,9 @@ checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
 
 [[package]]
 name = "wasm-bindgen"
-version = "0.2.84"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "31f8dcbc21f30d9b8f2ea926ecb58f6b91192c17e9d33594b3df58b2007ca53b"
+checksum = "7706a72ab36d8cb1f80ffbf0e071533974a60d0a308d01a5d0375bf60499a342"
 dependencies = [
  "cfg-if",
  "wasm-bindgen-macro",
@@ -2312,24 +2538,24 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-backend"
-version = "0.2.84"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "95ce90fd5bcc06af55a641a86428ee4229e44e07033963a2290a8e241607ccb9"
+checksum = "5ef2b6d3c510e9625e5fe6f509ab07d66a760f0885d858736483c32ed7809abd"
 dependencies = [
  "bumpalo",
  "log",
  "once_cell",
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
  "wasm-bindgen-shared",
 ]
 
 [[package]]
 name = "wasm-bindgen-futures"
-version = "0.4.34"
+version = "0.4.37"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f219e0d211ba40266969f6dbdd90636da12f75bee4fc9d6c23d1260dadb51454"
+checksum = "c02dbc21516f9f1f04f187958890d7e6026df8d16540b7ad9492bc34a67cea03"
 dependencies = [
  "cfg-if",
  "js-sys",
@@ -2339,9 +2565,9 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-macro"
-version = "0.2.84"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4c21f77c0bedc37fd5dc21f897894a5ca01e7bb159884559461862ae90c0b4c5"
+checksum = "dee495e55982a3bd48105a7b947fd2a9b4a8ae3010041b9e0faab3f9cd028f1d"
 dependencies = [
  "quote",
  "wasm-bindgen-macro-support",
@@ -2349,28 +2575,28 @@ dependencies = [
 
 [[package]]
 name = "wasm-bindgen-macro-support"
-version = "0.2.84"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2aff81306fcac3c7515ad4e177f521b5c9a15f2b08f4e32d823066102f35a5f6"
+checksum = "54681b18a46765f095758388f2d0cf16eb8d4169b639ab575a8f5693af210c7b"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 1.0.107",
+ "syn 2.0.27",
  "wasm-bindgen-backend",
  "wasm-bindgen-shared",
 ]
 
 [[package]]
 name = "wasm-bindgen-shared"
-version = "0.2.84"
+version = "0.2.87"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0046fef7e28c3804e5e38bfa31ea2a0f73905319b677e57ebe37e49358989b5d"
+checksum = "ca6ad05a4870b2bf5fe995117d3728437bd27d7cd5f06f13c17443ef369775a1"
 
 [[package]]
 name = "web-sys"
-version = "0.3.61"
+version = "0.3.64"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e33b99f4b23ba3eec1a53ac264e35a755f00e966e0065077d6027c0f575b0b97"
+checksum = "9b85cbef8c220a6abc02aefd892dfc0fc23afb1c6a426316ec33253a3877249b"
 dependencies = [
  "js-sys",
  "wasm-bindgen",
@@ -2434,22 +2660,7 @@ version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e686886bc078bc1b0b600cac0147aadb815089b6e4da64016cbd754b6342700f"
 dependencies = [
- "windows-targets 0.48.0",
-]
-
-[[package]]
-name = "windows-sys"
-version = "0.42.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5a3e1820f08b8513f676f7ab6c1f99ff312fb97b553d30ff4dd86f9f15728aa7"
-dependencies = [
- "windows_aarch64_gnullvm 0.42.2",
- "windows_aarch64_msvc 0.42.2",
- "windows_i686_gnu 0.42.2",
- "windows_i686_msvc 0.42.2",
- "windows_x86_64_gnu 0.42.2",
- "windows_x86_64_gnullvm 0.42.2",
- "windows_x86_64_msvc 0.42.2",
+ "windows-targets 0.48.1",
 ]
 
 [[package]]
@@ -2467,7 +2678,7 @@ version = "0.48.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9"
 dependencies = [
- "windows-targets 0.48.0",
+ "windows-targets 0.48.1",
 ]
 
 [[package]]
@@ -2487,9 +2698,9 @@ dependencies = [
 
 [[package]]
 name = "windows-targets"
-version = "0.48.0"
+version = "0.48.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7b1eb6f0cd7c80c79759c929114ef071b87354ce476d9d94271031c0497adfd5"
+checksum = "05d4b17490f70499f20b9e791dcf6a299785ce8af4d709018206dc5b4953e95f"
 dependencies = [
  "windows_aarch64_gnullvm 0.48.0",
  "windows_aarch64_msvc 0.48.0",
diff --git a/src/tools/kata-ctl/Cargo.toml b/src/tools/kata-ctl/Cargo.toml
index 2c45824321..d308f0271f 100644
--- a/src/tools/kata-ctl/Cargo.toml
+++ b/src/tools/kata-ctl/Cargo.toml
@@ -44,7 +44,12 @@ logging = { path = "../../libs/logging" }
 slog = "2.7.0"
 slog-scope = "4.4.0"
 hyper = "0.14.20"
-tokio = "1.28.1"
+tokio = { version = "1.28.1", features = ["signal"] }
+ttrpc = "0.6.0"
+
+prometheus = { version = "0.13.0", features = ["process"] }
+procfs = "0.12.0"
+lazy_static = "1.2"
 
 [target.'cfg(target_arch = "s390x")'.dependencies]
 reqwest = { version = "0.11", default-features = false, features = ["json", "blocking", "native-tls"] }
diff --git a/src/tools/kata-ctl/src/args.rs b/src/tools/kata-ctl/src/args.rs
index 8049ae31b1..7eb56863d1 100644
--- a/src/tools/kata-ctl/src/args.rs
+++ b/src/tools/kata-ctl/src/args.rs
@@ -56,6 +56,9 @@ pub enum Commands {
     /// Gather metrics associated with infrastructure used to run a sandbox
     Metrics(MetricsCommand),
 
+    /// Start a monitor to get metrics of Kata Containers
+    Monitor(MonitorArgument),
+
     /// Display version details
     Version,
 }
@@ -122,6 +125,12 @@ pub enum IpTablesArguments {
     Metrics,
 }
 
+#[derive(Debug, Args)]
+pub struct MonitorArgument {
+    /// The address to listen on for HTTP requests. (default "127.0.0.1:8090")
+    pub address: Option<String>,
+}
+
 #[derive(Debug, Args)]
 pub struct DirectVolumeCommand {
     #[clap(subcommand)]
diff --git a/src/tools/kata-ctl/src/main.rs b/src/tools/kata-ctl/src/main.rs
index 5495d02792..044c2bd07a 100644
--- a/src/tools/kata-ctl/src/main.rs
+++ b/src/tools/kata-ctl/src/main.rs
@@ -3,9 +3,16 @@
 // SPDX-License-Identifier: Apache-2.0
 //
 
+#[macro_use]
+extern crate lazy_static;
+
+#[macro_use]
+extern crate slog;
+
 mod arch;
 mod args;
 mod check;
+mod monitor;
 mod ops;
 mod types;
 mod utils;
@@ -18,7 +25,7 @@ use std::process::exit;
 use args::{Commands, KataCtlCli};
 
 use ops::check_ops::{
-    handle_check, handle_factory, handle_iptables, handle_metrics, handle_version,
+    handle_check, handle_factory, handle_iptables, handle_metrics, handle_monitor, handle_version,
 };
 use ops::env_ops::handle_env;
 use ops::exec_ops::handle_exec;
@@ -52,6 +59,7 @@ fn real_main() -> Result<()> {
         Commands::Factory => handle_factory(),
         Commands::Iptables(args) => handle_iptables(args),
         Commands::Metrics(args) => handle_metrics(args),
+        Commands::Monitor(args) => handle_monitor(args),
         Commands::Version => handle_version(),
     };
 
diff --git a/src/tools/kata-ctl/src/monitor/http_server.rs b/src/tools/kata-ctl/src/monitor/http_server.rs
new file mode 100644
index 0000000000..4a1a59ff9f
--- /dev/null
+++ b/src/tools/kata-ctl/src/monitor/http_server.rs
@@ -0,0 +1,181 @@
+// Copyright 2022-2023 Ant Group
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+use crate::monitor::metrics::get_monitor_metrics;
+use crate::sl;
+use crate::utils::TIMEOUT;
+
+use anyhow::{anyhow, Context, Result};
+use hyper::body;
+use hyper::service::{make_service_fn, service_fn};
+use hyper::{Body, Method, Request, Response, Server, StatusCode};
+use shim_interface::shim_mgmt::client::MgmtClient;
+use slog::{self, info};
+use std::collections::HashMap;
+use std::net::SocketAddr;
+
+const ROOT_URI: &str = "/";
+const METRICS_URI: &str = "/metrics";
+
+async fn handler_mux(req: Request<Body>) -> Result<Response<Body>> {
+    info!(
+        sl!(),
+        "mgmt-svr(mux): recv req, method: {}, uri: {}",
+        req.method(),
+        req.uri().path()
+    );
+
+    match (req.method(), req.uri().path()) {
+        (&Method::GET, ROOT_URI) => root_uri_handler(req).await,
+        (&Method::GET, METRICS_URI) => metrics_uri_handler(req).await,
+        _ => not_found_uri_handler(req).await,
+    }
+    .map_or_else(
+        |e| {
+            Response::builder()
+                .status(StatusCode::INTERNAL_SERVER_ERROR)
+                .body(Body::from(format!("{:?}\n", e)))
+                .map_err(|e| anyhow!("Failed to Build Response {:?}", e))
+        },
+        Ok,
+    )
+}
+
+pub async fn http_server_setup(socket_addr: &str) -> Result<()> {
+    let addr: SocketAddr = socket_addr
+        .parse()
+        .context("failed to parse http socket address")?;
+
+    let make_svc =
+        make_service_fn(|_conn| async { Ok::<_, anyhow::Error>(service_fn(handler_mux)) });
+
+    Server::bind(&addr).serve(make_svc).await?;
+
+    Ok(())
+}
+
+async fn root_uri_handler(_req: Request<Body>) -> Result<Response<Body>> {
+    Response::builder()
+        .status(StatusCode::OK)
+        .body(Body::from(
+            r#"Available HTTP endpoints:
+    /metrics : Get metrics from sandboxes.
+"#,
+        ))
+        .map_err(|e| anyhow!("Failed to Build Response {:?}", e))
+}
+
+async fn metrics_uri_handler(req: Request<Body>) -> Result<Response<Body>> {
+    let mut response_body = String::new();
+
+    response_body += &get_monitor_metrics().context("Failed to Get Monitor Metrics")?;
+
+    if let Some(uri_query) = req.uri().query() {
+        if let Ok(sandbox_id) = parse_sandbox_id(uri_query) {
+            response_body += &get_runtime_metrics(sandbox_id)
+                .await
+                .context(format!("{}\nFailed to Get Runtime Metrics", response_body))?;
+        }
+    }
+
+    Response::builder()
+        .status(StatusCode::OK)
+        .body(Body::from(response_body))
+        .map_err(|e| anyhow!("Failed to Build Response {:?}", e))
+}
+
+async fn get_runtime_metrics(sandbox_id: &str) -> Result<String> {
+    // build shim client
+    let shim_client =
+        MgmtClient::new(sandbox_id, Some(TIMEOUT)).context("failed to build shim mgmt client")?;
+
+    // get METRICS_URI
+    let shim_response = shim_client
+        .get(METRICS_URI)
+        .await
+        .context("failed to get METRICS_URI")?;
+
+    // get runtime_metrics
+    let runtime_metrics = String::from_utf8(body::to_bytes(shim_response).await?.to_vec())
+        .context("failed to get runtime_metrics")?;
+
+    Ok(runtime_metrics)
+}
+
+async fn not_found_uri_handler(_req: Request<Body>) -> Result<Response<Body>> {
+    Response::builder()
+        .status(StatusCode::NOT_FOUND)
+        .body(Body::from("NOT FOUND"))
+        .map_err(|e| anyhow!("Failed to Build Response {:?}", e))
+}
+
+fn parse_sandbox_id(uri: &str) -> Result<&str> {
+    let uri_pairs: HashMap<_, _> = uri
+        .split_whitespace()
+        .map(|s| s.split_at(s.find('=').unwrap_or(0)))
+        .map(|(key, val)| (key, &val[1..]))
+        .collect();
+
+    match uri_pairs.get("sandbox") {
+        Some(sid) => Ok(sid.to_owned()),
+        None => Err(anyhow!("params sandbox not found")),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_parse_sandbox_id() {
+        assert!(parse_sandbox_id("sandbox=demo_sandbox").unwrap() == "demo_sandbox");
+        assert!(parse_sandbox_id("foo=bar").is_err());
+    }
+
+    #[tokio::test]
+    async fn test_root_uri_handler() {
+        let root_resp = handler_mux(
+            Request::builder()
+                .method("GET")
+                .uri("/")
+                .body(hyper::Body::from(""))
+                .unwrap(),
+        )
+        .await
+        .unwrap();
+
+        assert!(root_resp.status() == StatusCode::OK);
+    }
+
+    #[tokio::test]
+    async fn test_metrics_uri_handler() {
+        let metrics_resp = handler_mux(
+            Request::builder()
+                .method("GET")
+                .uri("/metrics?sandbox=demo_sandbox")
+                .body(hyper::Body::from(""))
+                .unwrap(),
+        )
+        .await
+        .unwrap();
+
+        assert!(metrics_resp.status() == StatusCode::INTERNAL_SERVER_ERROR);
+    }
+
+    #[tokio::test]
+    async fn test_not_found_uri_handler() {
+        let not_found_resp = handler_mux(
+            Request::builder()
+                .method("POST")
+                .uri("/metrics?sandbox=demo_sandbox")
+                .body(hyper::Body::from(""))
+                .unwrap(),
+        )
+        .await
+        .unwrap();
+
+        assert!(not_found_resp.status() == StatusCode::NOT_FOUND);
+    }
+}
diff --git a/src/tools/kata-ctl/src/monitor/metrics.rs b/src/tools/kata-ctl/src/monitor/metrics.rs
new file mode 100644
index 0000000000..28b6b29eaa
--- /dev/null
+++ b/src/tools/kata-ctl/src/monitor/metrics.rs
@@ -0,0 +1,91 @@
+// Copyright 2022-2023 Ant Group
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+extern crate procfs;
+
+use anyhow::{anyhow, Context, Result};
+
+use prometheus::{Encoder, Gauge, IntCounter, Registry, TextEncoder};
+use std::sync::Mutex;
+
+const NAMESPACE_KATA_MONITOR: &str = "kata_ctl_monitor";
+
+lazy_static! {
+
+    static ref REGISTERED: Mutex<bool> = Mutex::new(false);
+
+    // custom registry
+    static ref REGISTRY: Registry = Registry::new();
+
+    // monitor metrics
+    static ref MONITOR_SCRAPE_COUNT: IntCounter =
+    IntCounter::new(format!("{}_{}", NAMESPACE_KATA_MONITOR, "scrape_count"), "Monitor scrape count").unwrap();
+
+    static ref MONITOR_MAX_FDS: Gauge = Gauge::new(format!("{}_{}", NAMESPACE_KATA_MONITOR, "process_max_fds"), "Open FDs for monitor").unwrap();
+
+    static ref MONITOR_OPEN_FDS: Gauge = Gauge::new(format!("{}_{}", NAMESPACE_KATA_MONITOR, "process_open_fds"), "Open FDs for monitor").unwrap();
+
+    static ref MONITOR_RESIDENT_MEMORY: Gauge = Gauge::new(format!("{}_{}", NAMESPACE_KATA_MONITOR, "process_resident_memory_bytes"), "Resident memory size in bytes for monitor").unwrap();
+}
+
+/// get monitor metrics
+pub fn get_monitor_metrics() -> Result<String> {
+    let mut registered = REGISTERED
+        .lock()
+        .map_err(|e| anyhow!("failed to check monitor metrics register status {:?}", e))?;
+
+    if !(*registered) {
+        register_monitor_metrics().context("failed to register monitor metrics")?;
+        *registered = true;
+    }
+
+    update_monitor_metrics().context("failed to update monitor metrics")?;
+
+    // gather all metrics and return as a String
+    let metric_families = REGISTRY.gather();
+
+    let mut buffer = Vec::new();
+    TextEncoder::new()
+        .encode(&metric_families, &mut buffer)
+        .context("failed to encode gathered metrics")?;
+
+    Ok(String::from_utf8(buffer)?)
+}
+
+fn register_monitor_metrics() -> Result<()> {
+    REGISTRY.register(Box::new(MONITOR_SCRAPE_COUNT.clone()))?;
+    REGISTRY.register(Box::new(MONITOR_MAX_FDS.clone()))?;
+    REGISTRY.register(Box::new(MONITOR_OPEN_FDS.clone()))?;
+    REGISTRY.register(Box::new(MONITOR_RESIDENT_MEMORY.clone()))?;
+
+    Ok(())
+}
+
+fn update_monitor_metrics() -> Result<()> {
+    MONITOR_SCRAPE_COUNT.inc();
+
+    let me = match procfs::process::Process::myself() {
+        Ok(p) => p,
+        Err(e) => {
+            eprintln!("failed to create process instance: {:?}", e);
+
+            return Ok(());
+        }
+    };
+
+    if let Ok(fds) = procfs::sys::fs::file_max() {
+        MONITOR_MAX_FDS.set(fds as f64);
+    }
+
+    if let Ok(fds) = me.fd_count() {
+        MONITOR_OPEN_FDS.set(fds as f64);
+    }
+
+    if let Ok(statm) = me.statm() {
+        MONITOR_RESIDENT_MEMORY.set(statm.resident as f64);
+    }
+
+    Ok(())
+}
diff --git a/src/tools/kata-ctl/src/monitor/mod.rs b/src/tools/kata-ctl/src/monitor/mod.rs
new file mode 100644
index 0000000000..ee40b123b8
--- /dev/null
+++ b/src/tools/kata-ctl/src/monitor/mod.rs
@@ -0,0 +1,8 @@
+// Copyright 2022-2023 Ant Group
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+mod metrics;
+
+pub mod http_server;
diff --git a/src/tools/kata-ctl/src/ops/check_ops.rs b/src/tools/kata-ctl/src/ops/check_ops.rs
index 8b19e0d8f8..8388637f1e 100644
--- a/src/tools/kata-ctl/src/ops/check_ops.rs
+++ b/src/tools/kata-ctl/src/ops/check_ops.rs
@@ -5,15 +5,21 @@
 
 use crate::arch::arch_specific::get_checks;
 
-use crate::args::{CheckArgument, CheckSubCommand, IptablesCommand, MetricsCommand};
+use crate::args::{
+    CheckArgument, CheckSubCommand, IptablesCommand, MetricsCommand, MonitorArgument,
+};
 
 use crate::check;
 
+use crate::monitor::http_server;
+
 use crate::ops::version;
 
 use crate::types::*;
 
-use anyhow::{anyhow, Result};
+use anyhow::{anyhow, Context, Result};
+
+const MONITOR_DEFAULT_SOCK_ADDR: &str = "127.0.0.1:8090";
 
 use slog::{info, o, warn};
 
@@ -128,6 +134,17 @@ pub fn handle_metrics(_args: MetricsCommand) -> Result<()> {
     Ok(())
 }
 
+pub fn handle_monitor(monitor_args: MonitorArgument) -> Result<()> {
+    tokio::runtime::Runtime::new()
+        .context("failed to new runtime for aync http server")?
+        .block_on(http_server::http_server_setup(
+            monitor_args
+                .address
+                .as_deref()
+                .unwrap_or(MONITOR_DEFAULT_SOCK_ADDR),
+        ))
+}
+
 pub fn handle_version() -> Result<()> {
     let version = version::get().unwrap();
 

From 151256011117b8d40e3c97f65483c39e835a39ff Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Fri, 28 Jul 2023 19:50:07 +0000
Subject: [PATCH 04/49] metrics: Add sysbench performance test

This PR adds the sysbench performance test for kata CI.

Fixes #7485

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/density/k8s-sysbench.sh | 81 +++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)
 create mode 100755 tests/metrics/density/k8s-sysbench.sh

diff --git a/tests/metrics/density/k8s-sysbench.sh b/tests/metrics/density/k8s-sysbench.sh
new file mode 100755
index 0000000000..9beb2300e1
--- /dev/null
+++ b/tests/metrics/density/k8s-sysbench.sh
@@ -0,0 +1,81 @@
+#!/bin/bash
+#
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+SCRIPT_PATH=$(dirname "$(readlink -f "$0")")
+
+source "${SCRIPT_PATH}/../../.ci/lib.sh"
+source "${SCRIPT_PATH}/../lib/common.bash"
+sysbench_file=$(mktemp sysbenchresults.XXXXXXXXXX)
+TEST_NAME="${TEST_NAME:-sysbench}"
+CI_JOB="${CI_JOB:-}"
+IMAGE="docker.io/library/local-sysbench:latest"
+DOCKERFILE="${SCRIPT_PATH}/sysbench-dockerfile/Dockerfile"
+
+function remove_tmp_file() {
+	rm -rf "${sysbench_file}"
+}
+
+trap remove_tmp_file EXIT
+
+function sysbench_memory() {
+	kubectl exec -i "$pod_name" -- sh -c "sysbench memory --threads=2 run" > "${sysbench_file}"
+	metrics_json_init
+	local memory_latency_sum=$(cat "$sysbench_file" | grep sum | cut -f2 -d':' | sed 's/[[:blank:]]//g')
+	metrics_json_start_array
+	local json="$(cat << EOF
+	{
+		"memory-latency-sum": {
+			"Result" : $memory_latency_sum,
+			"Units" : "ms"
+		}
+	}
+EOF
+)"
+	metrics_json_add_array_element "$json"
+	metrics_json_end_array "Results"
+	metrics_json_save
+}
+
+function sysbench_start_deployment() {
+	cmds=("bc" "jq")
+	check_cmds "${cmds[@]}"
+
+	# Check no processes are left behind
+	check_processes
+
+	if [ -z "${CI_JOB}" ]; then
+		# Start kubernetes
+		start_kubernetes
+	fi
+
+	export KUBECONFIG="$HOME/.kube/config"
+	export pod_name="test-sysbench"
+
+	kubectl create -f "${SCRIPT_PATH}/runtimeclass_workloads/sysbench-pod.yaml"
+	kubectl wait --for=condition=Ready --timeout=120s pod "$pod_name"
+
+}
+
+function sysbench_cleanup() {
+	kubectl delete pod "$pod_name"
+	if [ -z "${CI_JOB}" ]; then
+		end_kubernetes
+		check_processes
+	fi
+}
+
+function main() {
+	init_env
+	sysbench_start_deployment
+	sysbench_memory
+	sysbench_cleanup
+}
+
+main "$@"

From ad413d16469d5065fa6f1c86e2f2a8bf7a088ac2 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Fri, 28 Jul 2023 19:58:10 +0000
Subject: [PATCH 05/49] metrics: Add sysbench dockerfile

This PR adds sysbench dockerfile.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../density/sysbench-dockerfile/Dockerfile      | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 tests/metrics/density/sysbench-dockerfile/Dockerfile

diff --git a/tests/metrics/density/sysbench-dockerfile/Dockerfile b/tests/metrics/density/sysbench-dockerfile/Dockerfile
new file mode 100644
index 0000000000..f6d1ba398b
--- /dev/null
+++ b/tests/metrics/density/sysbench-dockerfile/Dockerfile
@@ -0,0 +1,17 @@
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Usage: FROM [image name]
+FROM ubuntu:20.04
+
+# Version of the Dockerfile
+LABEL DOCKERFILE_VERSION="1.0"
+
+RUN apt-get update && \
+	apt-get install -y build-essential git curl sudo && \
+	apt-get remove -y unattended-upgrades && \
+	curl -OkL https://packagecloud.io/install/repositories/akopytov/sysbench/script.deb.sh && \
+	apt-get install -y sysbench
+
+CMD ["/bin/bash"]

From 5a1b5d36729e8b5791f083bb6d2f5b12174d233d Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Fri, 28 Jul 2023 20:00:12 +0000
Subject: [PATCH 06/49] metrics: Add sysbench pod yaml

This PR adds the sysbench pod yaml for the sysbench performance test.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/density/k8s-sysbench.sh          |  4 +---
 .../runtimeclass_workloads/sysbench-pod.yaml   | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+), 3 deletions(-)
 create mode 100644 tests/metrics/density/runtimeclass_workloads/sysbench-pod.yaml

diff --git a/tests/metrics/density/k8s-sysbench.sh b/tests/metrics/density/k8s-sysbench.sh
index 9beb2300e1..a28daf56a1 100755
--- a/tests/metrics/density/k8s-sysbench.sh
+++ b/tests/metrics/density/k8s-sysbench.sh
@@ -8,9 +8,7 @@ set -o errexit
 set -o nounset
 set -o pipefail
 
-SCRIPT_PATH=$(dirname "$(readlink -f "$0")")
-
-source "${SCRIPT_PATH}/../../.ci/lib.sh"
+SCRIPT_PATH=$(dirname "$(readlink -f "$0")"
 source "${SCRIPT_PATH}/../lib/common.bash"
 sysbench_file=$(mktemp sysbenchresults.XXXXXXXXXX)
 TEST_NAME="${TEST_NAME:-sysbench}"
diff --git a/tests/metrics/density/runtimeclass_workloads/sysbench-pod.yaml b/tests/metrics/density/runtimeclass_workloads/sysbench-pod.yaml
new file mode 100644
index 0000000000..03dc850f46
--- /dev/null
+++ b/tests/metrics/density/runtimeclass_workloads/sysbench-pod.yaml
@@ -0,0 +1,18 @@
+#
+# Copyright (c) 2018-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-sysbench
+spec:
+  terminationGracePeriodSeconds: 0
+  runtimeClassName: kata
+  containers:
+  - name: test-sysbench
+    image: localhost:5000/sysbench-kata:latest
+    command:
+        - sleep
+        - "60"

From cddcde1d40dd7382160c6f41410ccccd5225d770 Mon Sep 17 00:00:00 2001
From: Zvonko Kaiser <zkaiser@nvidia.com>
Date: Mon, 31 Jul 2023 11:26:27 +0000
Subject: [PATCH 07/49] vfio: Fix vfio device ordering

If modeVFIO is enabled we need 1st to attach the VFIO control group
device /dev/vfio/vfio an 2nd the actuall device(s) afterwards.Sort the
devices starting with device #1 being the VFIO control group device and
the next the actuall device(s)
/dev/vfio/<group>

Fixes: #7493

Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
---
 src/runtime/virtcontainers/container.go | 22 ++++++++++++++++++++++
 src/runtime/virtcontainers/sandbox.go   | 10 ++--------
 2 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/src/runtime/virtcontainers/container.go b/src/runtime/virtcontainers/container.go
index 4927bdab3f..0f68810c40 100644
--- a/src/runtime/virtcontainers/container.go
+++ b/src/runtime/virtcontainers/container.go
@@ -851,6 +851,21 @@ func (c *Container) checkBlockDeviceSupport(ctx context.Context) bool {
 	return false
 }
 
+// Sort the devices starting with device #1 being the VFIO control group
+// device and the next the actuall device(s) e.g. /dev/vfio/<group>
+func sortContainerVFIODevices(devices []ContainerDevice) []ContainerDevice {
+	var vfioDevices []ContainerDevice
+
+	for _, device := range devices {
+		if deviceManager.IsVFIOControlDevice(device.ContainerPath) {
+			vfioDevices = append([]ContainerDevice{device}, vfioDevices...)
+			continue
+		}
+		vfioDevices = append(vfioDevices, device)
+	}
+	return vfioDevices
+}
+
 // create creates and starts a container inside a Sandbox. It has to be
 // called only when a new container, not known by the sandbox, has to be created.
 func (c *Container) create(ctx context.Context) (err error) {
@@ -893,6 +908,13 @@ func (c *Container) create(ctx context.Context) (err error) {
 		}
 		c.devices = cntDevices
 	}
+	// If modeVFIO is enabled we need 1st to attach the VFIO control group
+	// device /dev/vfio/vfio an 2nd the actuall device(s) afterwards.
+	// Sort the devices starting with device #1 being the VFIO control group
+	// device and the next the actuall device(s) /dev/vfio/<group>
+	if modeVFIO {
+		c.devices = sortContainerVFIODevices(c.devices)
+	}
 
 	c.Logger().WithFields(logrus.Fields{
 		"devices": c.devices,
diff --git a/src/runtime/virtcontainers/sandbox.go b/src/runtime/virtcontainers/sandbox.go
index a4d7e07302..3a3c275bb5 100644
--- a/src/runtime/virtcontainers/sandbox.go
+++ b/src/runtime/virtcontainers/sandbox.go
@@ -651,7 +651,8 @@ func (s *Sandbox) coldOrHotPlugVFIO(sandboxConfig *SandboxConfig) (bool, error)
 	hotPlugVFIO := (sandboxConfig.HypervisorConfig.HotPlugVFIO != config.NoPort)
 
 	modeIsGK := (sandboxConfig.VfioMode == config.VFIOModeGuestKernel)
-	modeIsVFIO := (sandboxConfig.VfioMode == config.VFIOModeVFIO)
+	// modeIsVFIO is needed at the container level not the sandbox level.
+	// modeIsVFIO := (sandboxConfig.VfioMode == config.VFIOModeVFIO)
 
 	var vfioDevices []config.DeviceInfo
 	// vhost-user-block device is a PCIe device in Virt, keep track of it
@@ -666,13 +667,6 @@ func (s *Sandbox) coldOrHotPlugVFIO(sandboxConfig *SandboxConfig) (bool, error)
 				continue
 			}
 			isVFIODevice := deviceManager.IsVFIODevice(device.ContainerPath)
-			isVFIOControlDevice := deviceManager.IsVFIOControlDevice(device.ContainerPath)
-			// vfio_mode=vfio needs the VFIO control device add it to the list
-			// of devices to be added to the VM.
-			if modeIsVFIO && isVFIOControlDevice && !hotPlugVFIO {
-				vfioDevices = append(vfioDevices, device)
-			}
-
 			if hotPlugVFIO && isVFIODevice {
 				device.ColdPlug = false
 				device.Port = sandboxConfig.HypervisorConfig.HotPlugVFIO

From 87d41b3dfaf10792826da0a3429bc70d629c8a95 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Mon, 31 Jul 2023 16:50:16 +0000
Subject: [PATCH 08/49] metrics: Add FIO test to gha for kata metrics CI

This PR adds FIO test to gha for kata metrics CI.

Fixes #7502

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/storage/fio-k8s/fio-test-ci.sh | 85 ++++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100755 tests/metrics/storage/fio-k8s/fio-test-ci.sh

diff --git a/tests/metrics/storage/fio-k8s/fio-test-ci.sh b/tests/metrics/storage/fio-k8s/fio-test-ci.sh
new file mode 100755
index 0000000000..3ae51bea40
--- /dev/null
+++ b/tests/metrics/storage/fio-k8s/fio-test-ci.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+#
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+
+set -e
+
+# General env
+SCRIPT_PATH=$(dirname "$(readlink -f "$0")")
+source "${SCRIPT_PATH}/../../lib/common.bash"
+FIO_PATH="${GOPATH}/src/${TEST_REPO}/metrics/storage/fio-k8s"
+TEST_NAME="${TEST_NAME:-fio}"
+
+function main() {
+	cmds=("bc" "jq")
+	check_cmds "${cmds[@]}"
+	check_processes
+	init_env
+
+	export KUBECONFIG="$HOME/.kube/config"
+
+	pushd "${FIO_PATH}"
+		echo "INFO: Running K8S FIO test"
+		make test-ci
+	popd
+
+	test_result_file="${FIO_PATH}/cmd/fiotest/test-results/kata/randrw-sync.job/output.json"
+
+	metrics_json_init
+	local read_io=$(cat $test_result_file | grep io_bytes | head -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+	local read_bw=$(cat $test_result_file | grep bw_bytes | head -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+ 	local read_90_percentile=$(cat $test_result_file | grep 90.000000 | head -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+	local read_95_percentile=$(cat $test_result_file | grep 95.000000 | head -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+	local write_io=$(cat $test_result_file | grep io_bytes | head -2 | tail -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+	local write_bw=$(cat $test_result_file | grep bw_bytes | head -2 | tail -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+	local write_90_percentile=$(cat $test_result_file | grep 90.000000 | head -2 | tail -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+	local write_95_percentile=$(cat $test_result_file | grep 95.000000 | head -2 | tail -1 | sed 's/[[:blank:]]//g' | cut -f2 -d ':' | cut -f1 -d ',')
+
+	metrics_json_start_array
+	local json="$(cat << EOF
+	{
+		"readio": {
+			"Result" : $read_io,
+			"Units" : "bytes"
+		},
+		"readbw": {
+			"Result" : $read_bw,
+			"Units" : "bytes/sec"
+		},
+		"read90percentile": {
+			"Result" : $read_90_percentile,
+			"Units" : "ns"
+		},
+		"read95percentile": {
+			"Result" : $read_95_percentile,
+			"Units" : "ns"
+		},
+		"writeio": {
+			"Result" : $write_io,
+			"Units" : "bytes"
+		},
+		"writebw": {
+			"Result" : $write_bw,
+			"Units" : "bytes/sec"
+		},
+		"write90percentile": {
+			"Result" : $write_90_percentile,
+			"Units" : "ns"
+		},
+		"write95percentile": {
+			"Result" : $write_95_percentile,
+			"Units" : "ns"
+		}
+	}
+EOF
+)"
+	metrics_json_add_array_element "$json"
+	metrics_json_end_array "Results"
+	metrics_json_save
+
+	check_processes
+}
+
+main "$@"

From 69f05cf9e6fc700c57bc7e139c1729148fe15cbc Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Mon, 31 Jul 2023 17:30:05 +0000
Subject: [PATCH 09/49] metrics: Add FIO general README

This PR adds FIO general README information.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/storage/fio-k8s/README.md | 26 +++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
 create mode 100644 tests/metrics/storage/fio-k8s/README.md

diff --git a/tests/metrics/storage/fio-k8s/README.md b/tests/metrics/storage/fio-k8s/README.md
new file mode 100644
index 0000000000..8daad84083
--- /dev/null
+++ b/tests/metrics/storage/fio-k8s/README.md
@@ -0,0 +1,26 @@
+# FIO test in Kubernetes
+
+This is an automation to run `fio` with Kubernetes.
+
+## Requirements:
+- Kubernetes cluster running.
+- Kata configured as `runtimeclass`. 
+The test is structured as follow: 
+- [fio-test]: Program wrapper to launch `fio` in a K8s pod. 
+- [pkg]: Library code that could be used for more `fio` automation. 
+- [configs]: Configuration files used by [fio-test]. 
+- [DAX-compare-test]: Script to run [fio-test] to generate `fio` data for Kata with/without `virtio-fs DAX` and K8s bare-metal runtime(`runc`). 
+- [report] Jupyter Notebook to create reports for data generated by [DAX-compare-test].
+
+## Top-level Makefile targets
+- `build`: Build `fio` metrics.
+- `test`: quick test, used to verify changes in [fio-test]. 
+- `run`: Run `fio` metrics and generate reports. 
+- `test-report-interactive`: Run python notebook in `localhost:8888`, useful to edit the report. 
+- `test-report`: Generate report from data generated by `make test`. 
+
+[fio-test]:cmd/fiotest 
+[configs]:configs 
+[pkg]:pkg 
+[report]:scripts/dax-compare-test/report
+[DAX-compare-test]:scripts/dax-compare-test/README.md

From 21f5b652331859abc528964759fc9d7ae525c38f Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Mon, 31 Jul 2023 17:32:40 +0000
Subject: [PATCH 10/49] metrics: Add FIO information in storage general README

This PR adds FIO information in storage general README.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/storage/README.md | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/tests/metrics/storage/README.md b/tests/metrics/storage/README.md
index b9d1cc68a4..305c7e227d 100644
--- a/tests/metrics/storage/README.md
+++ b/tests/metrics/storage/README.md
@@ -1,11 +1,27 @@
 # Kata Containers storage I/O tests
+
 The metrics tests in this directory are designed to be used to assess storage IO.
+
 ## `Blogbench` test
+
 The `blogbench` script is based on the `blogbench` program which is designed to emulate a busy blog server with a number of concurrent 
 threads performing a mixture of reads, writes and rewrites.
+
 ### Running the `blogbench` test
+
 The `blogbench` test can be run by hand, for example:
 ```
 $ cd metrics
 $ bash storage/blogbench.sh
 ```
+## `fio` test
+
+The `fio` test utilises the [fio tool](https://github.com/axboe/fio), configured
+to perform measurements upon a single test file.
+
+The test configuration used by the script can be modified by setting a number of
+environment variables to change or over-ride the test defaults.
+
+## DAX `virtio-fs` `fio` Kubernetes tests
+
+[Test](fio-k8s/README.md) to compare the use of DAX option in `virtio-fs`.

From 8a584589ff39bbaba9796604323f86f8179482e6 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Mon, 31 Jul 2023 17:42:44 +0000
Subject: [PATCH 11/49] metrics: Add DAX FIO README

This PR adds DAX FIO README information.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../metrics/storage/fio-k8s/scripts/README.md | 41 +++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 tests/metrics/storage/fio-k8s/scripts/README.md

diff --git a/tests/metrics/storage/fio-k8s/scripts/README.md b/tests/metrics/storage/fio-k8s/scripts/README.md
new file mode 100644
index 0000000000..71cfdea846
--- /dev/null
+++ b/tests/metrics/storage/fio-k8s/scripts/README.md
@@ -0,0 +1,41 @@
+# FIO in Kubernetes
+
+This test run `fio` jobs to measure how Kata Containers work using virtio-fs DAX. The test works using Kubernetes.  
+The test has to run in a single node cluster, it is needed as the test modifies Kata configuration file. 
+
+The `virtio-fs` options that this test will use are: 
+
+* `cache mode` Only `auto`, this is the most compatible mode for most of the Kata use cases. Today this is default in Kata. 
+* `thread pool size` Restrict the number of worker threads per request queue, zero means no thread pool. 
+* `DAX` 
+``` 
+File contents can be mapped into a memory window on the host, allowing the guest to directly access data from the host page cache. This has several advantages: The guest page cache is bypassed, reducing the memory footprint.  No communication is necessary 
+to access file contents, improving I/O performance. Shared file access is coherent between virtual machines on the same host even with mmap. 
+``` 
+
+This test by default iterates over different `virtio-fs` configurations.
+
+| test name                 | DAX | thread pool size | cache mode |
+|---------------------------|-----|------------------|------------|
+| pool_0_cache_auto_no_DAX  | no | 0 | auto | pool_0_cache_auto_DAX | yes | 0 | auto |
+
+The `fio` options used are: 
+
+`ioengine`: How the IO requests are issued to the kernel. 
+* `libaio`: Supports async IO for both direct and buffered IO. 
+* `mmap`: File is memory mapped with mmap(2) and data copied to/from using memcpy(3). 
+`rw type`: Type of I/O pattern. 
+* `randread`: Random reads. 
+* `randrw`: Random mixed reads and writes. 
+* `randwrite`: Random writes. 
+* `read`: Sequential reads. 
+* `write`: Sequential writes. Additional notes: Some jobs contain a `multi` prefix. This means that the same job runs more than once at the same time using its own file.
+
+### Static `fio` values:
+Some `fio` values are not modified over all the jobs. 
+`runtime`: Tell `fio` to terminate processing after the specified period of time(seconds). 
+`loops`: Run the specified number of iterations of this job. Used to repeat the same workload a given number of times. 
+`iodepth`: Number of I/O units to keep in flight against the file. Note that increasing `iodepth` beyond 1 will not affect synchronous `ioengine`. 
+`size`: The total size of file I/O for each thread of this job. 
+`direct`: If value is true, use non-buffered I/O. This is usually O_`DIRECT`. 
+`blocksize`: The block size in bytes used for I/O units

From 8933d54428d012b6154e45a2d4ca3b8abb4d210b Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Mon, 31 Jul 2023 17:51:11 +0000
Subject: [PATCH 12/49] metrics: Add FIO to gha run script

This PR adds FIO to gha run script.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/gha-run.sh | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/metrics/gha-run.sh b/tests/metrics/gha-run.sh
index 99ec5a1c1a..3a663bf236 100755
--- a/tests/metrics/gha-run.sh
+++ b/tests/metrics/gha-run.sh
@@ -87,6 +87,14 @@ function run_test_tensorflow() {
 	bash tests/metrics/machine_learning/tensorflow.sh 1 20
 }
 
+function run_test_fio() {
+	info "Running FIO test using ${KATA_HYPERVISOR} hypervisor"
+        # ToDo: remove the exit once the metrics workflow is stable
+        exit 0
+
+	bash storage/fio-k8s/fio-test-ci.sh
+}
+
 function main() {
 	action="${1:-}"
 	case "${action}" in
@@ -97,6 +105,7 @@ function main() {
 		run-test-memory-usage-inside-container) run_test_memory_usage_inside_container ;;
 		run-test-blogbench) run_test_blogbench ;;
 		run-test-tensorflow) run_test_tensorflow ;;
+		run-test-fio) run_test_fio ;;
 		*) >&2 die "Invalid argument" ;;
 	esac
 }

From 6328181762835d8a1b80426a3ebf6d22245f0676 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Fri, 28 Jul 2023 20:08:15 +0000
Subject: [PATCH 13/49] metrics: Add k8s sysbench documentation

This PR adds k8s sysbench documentation at general density documentation.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/density/README.md       |  5 +++++
 tests/metrics/density/k8s-sysbench.sh | 14 ++------------
 2 files changed, 7 insertions(+), 12 deletions(-)

diff --git a/tests/metrics/density/README.md b/tests/metrics/density/README.md
index e07ee18b39..d553740156 100644
--- a/tests/metrics/density/README.md
+++ b/tests/metrics/density/README.md
@@ -51,3 +51,8 @@ For more details see the [footprint test documentation](footprint_data.md).
 Measures the memory statistics *inside* the container. This allows evaluation of
 the overhead the VM kernel and rootfs are having on the memory that was requested
 by the container co-ordination system, and thus supplied to the VM.
+
+## `k8s-sysbench`
+
+`Sysbench`is an open-source and multi-purpose benchmark utility that evaluates parameters features tests for `CPU`, memory
+and I/O. Currently the `k8s-sysbench` test is measuring the `CPU` performance.
diff --git a/tests/metrics/density/k8s-sysbench.sh b/tests/metrics/density/k8s-sysbench.sh
index a28daf56a1..288e69957a 100755
--- a/tests/metrics/density/k8s-sysbench.sh
+++ b/tests/metrics/density/k8s-sysbench.sh
@@ -8,7 +8,7 @@ set -o errexit
 set -o nounset
 set -o pipefail
 
-SCRIPT_PATH=$(dirname "$(readlink -f "$0")"
+SCRIPT_PATH=$(dirname "$(readlink -f "$0")")
 source "${SCRIPT_PATH}/../lib/common.bash"
 sysbench_file=$(mktemp sysbenchresults.XXXXXXXXXX)
 TEST_NAME="${TEST_NAME:-sysbench}"
@@ -48,25 +48,15 @@ function sysbench_start_deployment() {
 	# Check no processes are left behind
 	check_processes
 
-	if [ -z "${CI_JOB}" ]; then
-		# Start kubernetes
-		start_kubernetes
-	fi
-
-	export KUBECONFIG="$HOME/.kube/config"
 	export pod_name="test-sysbench"
 
 	kubectl create -f "${SCRIPT_PATH}/runtimeclass_workloads/sysbench-pod.yaml"
 	kubectl wait --for=condition=Ready --timeout=120s pod "$pod_name"
-
 }
 
 function sysbench_cleanup() {
 	kubectl delete pod "$pod_name"
-	if [ -z "${CI_JOB}" ]; then
-		end_kubernetes
-		check_processes
-	fi
+	check_processes
 }
 
 function main() {

From 1b21a46246f5567cd664c428c7599fe5c44560bc Mon Sep 17 00:00:00 2001
From: Manabu Sugimoto <Manabu.Sugimoto@sony.com>
Date: Thu, 27 Jul 2023 15:13:51 +0900
Subject: [PATCH 14/49] docs: Use control-plane term instead of master

Replace `master` with `control-plane` in the context of K8s
because `master` is a legacy term and haven't been used any more.

Ref. https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint

Fixes: #7466

Signed-off-by: Manabu Sugimoto <Manabu.Sugimoto@sony.com>
---
 docs/design/architecture/kubernetes.md                 | 2 +-
 docs/how-to/how-to-use-k8s-with-containerd-and-kata.md | 6 +++---
 docs/how-to/run-kata-with-k8s.md                       | 6 +++---
 docs/install/minikube-installation-guide.md            | 2 +-
 4 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/docs/design/architecture/kubernetes.md b/docs/design/architecture/kubernetes.md
index 8d4d3144c0..a607291ec3 100644
--- a/docs/design/architecture/kubernetes.md
+++ b/docs/design/architecture/kubernetes.md
@@ -12,7 +12,7 @@ only needs to run a container runtime and a container agent (called a
 Kata Containers represents a Kubelet pod as a VM.
 
 A Kubernetes cluster runs a control plane where a scheduler (typically
-running on a dedicated master node) calls into a compute Kubelet. This
+running on a dedicated control-plane node) calls into a compute Kubelet. This
 Kubelet instance is responsible for managing the lifecycle of pods
 within the nodes and eventually relies on a container runtime to
 handle execution. The Kubelet architecture decouples lifecycle
diff --git a/docs/how-to/how-to-use-k8s-with-containerd-and-kata.md b/docs/how-to/how-to-use-k8s-with-containerd-and-kata.md
index de7a34ef61..3cc1ebb126 100644
--- a/docs/how-to/how-to-use-k8s-with-containerd-and-kata.md
+++ b/docs/how-to/how-to-use-k8s-with-containerd-and-kata.md
@@ -139,12 +139,12 @@ By default the CNI plugin binaries is installed under `/opt/cni/bin` (in package
   EOF
   ```
 
-## Allow pods to run in the master node
+## Allow pods to run in the control-plane node
 
-By default, the cluster will not schedule pods in the master node. To enable master node scheduling:
+By default, the cluster will not schedule pods in the control-plane node. To enable control-plane node scheduling:
 
 ```bash
-$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
+$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/control-plane-
 ```
 
 ## Create runtime class for Kata Containers
diff --git a/docs/how-to/run-kata-with-k8s.md b/docs/how-to/run-kata-with-k8s.md
index 4e5c58d5a2..ab700d2a4a 100644
--- a/docs/how-to/run-kata-with-k8s.md
+++ b/docs/how-to/run-kata-with-k8s.md
@@ -115,11 +115,11 @@ $ sudo kubeadm init --ignore-preflight-errors=all --config kubeadm-config.yaml
 $ export KUBECONFIG=/etc/kubernetes/admin.conf
 ```
 
-### Allow pods to run in the master node
+### Allow pods to run in the control-plane node
 
-By default, the cluster will not schedule pods in the master node. To enable master node scheduling:
+By default, the cluster will not schedule pods in the control-plane node. To enable control-plane node scheduling:
 ```bash
-$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/master-
+$ sudo -E kubectl taint nodes --all node-role.kubernetes.io/control-plane-
 ```
 
 ### Create runtime class for Kata Containers
diff --git a/docs/install/minikube-installation-guide.md b/docs/install/minikube-installation-guide.md
index c49e2bac99..6a946a28ba 100644
--- a/docs/install/minikube-installation-guide.md
+++ b/docs/install/minikube-installation-guide.md
@@ -91,7 +91,7 @@ Before you install Kata Containers, check that your Minikube is operating. On yo
 $ kubectl get nodes
 ```
 
-You should see your `master` node listed as being `Ready`.
+You should see your `control-plane` node listed as being `Ready`.
 
 Check you have virtualization enabled inside your Minikube. The following should return
 a number larger than `0` if you have either of the `vmx` or `svm` nested virtualization features

From f910c66d6f44b5b11d4a9af4af494e15c9366083 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= <fabiano.fidencio@intel.com>
Date: Tue, 1 Aug 2023 12:36:33 +0200
Subject: [PATCH 15/49] ci: k8s: Do not fail when gathering info on AKS nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Otherwise the VM deletion may not delete, leaving us with several
machines behind.

Fixes: #7509

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
---
 tests/integration/kubernetes/gha-run.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/integration/kubernetes/gha-run.sh b/tests/integration/kubernetes/gha-run.sh
index 0d02e70820..d756ccfdce 100755
--- a/tests/integration/kubernetes/gha-run.sh
+++ b/tests/integration/kubernetes/gha-run.sh
@@ -173,8 +173,8 @@ function delete_cluster() {
 }
 
 function get_nodes_and_pods_info() {
-    kubectl debug $(kubectl get nodes -o name) -it --image=quay.io/kata-containers/kata-debug:latest
-    kubectl get pods -o name | grep node-debugger | xargs kubectl delete
+    kubectl debug $(kubectl get nodes -o name) -it --image=quay.io/kata-containers/kata-debug:latest || true
+    kubectl get pods -o name | grep node-debugger | xargs kubectl delete || true
 }
 
 function main() {

From f5e9985afe90f55ecc5870938e809768cccbf5b2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= <fabiano.fidencio@intel.com>
Date: Tue, 1 Aug 2023 12:31:34 +0200
Subject: [PATCH 16/49] release: release-note.sh: Fix typos and reference to
 images
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

diferent -> different

And also let's make sure we escape the backticks around the kata-deploy
environment variables, otherwise bash will try to interpret those.

Fixes: #7497

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
---
 tools/packaging/release/release-notes.sh | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/tools/packaging/release/release-notes.sh b/tools/packaging/release/release-notes.sh
index a21d5b1479..254aa255b7 100755
--- a/tools/packaging/release/release-notes.sh
+++ b/tools/packaging/release/release-notes.sh
@@ -141,18 +141,18 @@ build reproducibility we publish those container images, and when those are used
 of the projects listed as part of the "versions.yaml" file, users can get as close to the environment we
 used to build the release artefacts.
 * Kernel (on all its different flavours): $(get_kernel_image_name)
-* OVMF (on all its diferent flavours): $(get_ovmf_image_name)
+* OVMF (on all its different flavours): $(get_ovmf_image_name)
 * QEMU (on all its different flavurs): $(get_qemu_image_name)
 * shim-v2: $(get_shim_v2_image_name)
 * virtiofsd: $(get_virtiofsd_image_name)
 
 The users who want to rebuild the tarballs using exactly the same images can simply use the following environment
 variables:
-* `KERNEL_CONTAINER_BUILDER`
-* `OVMF_CONTAINER_BUILDER`
-* `QEMU_CONTAINER_BUILDER`
-* `SHIM_V2_CONTAINER_BUILDER`
-* `VIRTIOFSD_CONTAINER_BUILDER`
+* \`KERNEL_CONTAINER_BUILDER\`
+* \`OVMF_CONTAINER_BUILDER\`
+* \`QEMU_CONTAINER_BUILDER\`
+* \`SHIM_V2_CONTAINER_BUILDER\`
+* \`VIRTIOFSD_CONTAINER_BUILDER\`
 
 ## Kata Linux Containers Kernel
 Kata Containers ${runtime_version} suggest to use the Linux kernel [${kernel_version}][kernel]

From 27c02367f9752bc4771dc01d0c6ff9bedf2b5671 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Tue, 1 Aug 2023 16:30:46 +0000
Subject: [PATCH 17/49] metrics: Add iperf3 network test

This PR adds the iperf3 benchmark test for kata metrics.

Fixes #7515

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../k8s-network-metrics-iperf3.sh             | 314 ++++++++++++++++++
 1 file changed, 314 insertions(+)
 create mode 100755 tests/metrics/network/iperf3_kubernetes/k8s-network-metrics-iperf3.sh

diff --git a/tests/metrics/network/iperf3_kubernetes/k8s-network-metrics-iperf3.sh b/tests/metrics/network/iperf3_kubernetes/k8s-network-metrics-iperf3.sh
new file mode 100755
index 0000000000..6593bb6515
--- /dev/null
+++ b/tests/metrics/network/iperf3_kubernetes/k8s-network-metrics-iperf3.sh
@@ -0,0 +1,314 @@
+#!/bin/bash
+#
+# Copyright (c) 2021-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+# This test measures the following network essentials:
+# - bandwith simplex
+# - jitter
+#
+# These metrics/results will be got from the interconnection between
+# a client and a server using iperf3 tool.
+# The following cases are covered:
+#
+# case 1:
+#  container-server <----> container-client
+#
+# case 2"
+#  container-server <----> host-client
+
+set -o pipefail
+
+SCRIPT_PATH=$(dirname "$(readlink -f "$0")")
+
+source "${SCRIPT_PATH}/../../lib/common.bash"
+iperf_file=$(mktemp iperfresults.XXXXXXXXXX)
+TEST_NAME="${TEST_NAME:-network-iperf3}"
+COLLECT_ALL="${COLLECT_ALL:-false}"
+
+function remove_tmp_file() {
+	rm -rf "${iperf_file}"
+}
+
+trap remove_tmp_file EXIT
+
+function iperf3_all_collect_results() {
+	metrics_json_init
+	metrics_json_start_array
+	local json="$(cat << EOF
+	{
+		"bandwidth": {
+			"Result" : $bandwidth_result,
+			"Units" : "$bandwidth_units"
+		},
+		"jitter": {
+			"Result" : $jitter_result,
+			"Units" : "$jitter_units"
+		},
+		"cpu": {
+			"Result" : $cpu_result,
+			"Units"  : "$cpu_units"
+		},
+		"parallel": {
+			"Result" : $parallel_result,
+			"Units" : "$parallel_units"
+		}
+	}
+EOF
+)"
+	metrics_json_add_array_element "$json"
+	metrics_json_end_array "Results"
+}
+
+function iperf3_bandwidth() {
+	# Start server
+	local transmit_timeout="30"
+
+	kubectl exec -i "$client_pod_name" -- sh -c "iperf3 -J -c ${server_ip_add} -t ${transmit_timeout}" | jq '.end.sum_received.bits_per_second' > "${iperf_file}"
+	export bandwidth_result=$(cat "${iperf_file}")
+	export bandwidth_units="bits per second"
+
+	if [ "$COLLECT_ALL" == "true" ]; then
+		iperf3_all_collect_results
+	else
+		metrics_json_init
+		metrics_json_start_array
+
+		local json="$(cat << EOF
+		{
+			"bandwidth": {
+				"Result" : $bandwidth_result,
+				"Units" : "$bandwidth_units"
+			}
+		}
+EOF
+)"
+		metrics_json_add_array_element "$json"
+		metrics_json_end_array "Results"
+	fi
+}
+
+function iperf3_jitter() {
+	# Start server
+	local transmit_timeout="30"
+
+	kubectl exec -i "$client_pod_name" -- sh -c "iperf3 -J -c ${server_ip_add} -u -t ${transmit_timeout}" | jq '.end.sum.jitter_ms' > "${iperf_file}"
+	result=$(cat "${iperf_file}")
+	export jitter_result=$(printf "%0.3f\n" $result)
+	export jitter_units="ms"
+
+	if [ "$COLLECT_ALL" == "true" ]; then
+		iperf3_all_collect_results
+	else
+		metrics_json_init
+		metrics_json_start_array
+
+		local json="$(cat << EOF
+		{
+			"jitter": {
+				"Result" : $jitter_result,
+				"Units" : "ms"
+			}
+		}
+EOF
+)"
+		metrics_json_add_array_element "$json"
+		metrics_json_end_array "Results"
+	fi
+}
+
+function iperf3_parallel() {
+	# This will measure four parallel connections with iperf3
+	kubectl exec -i "$client_pod_name" -- sh -c "iperf3 -J -c ${server_ip_add} -P 4" | jq '.end.sum_received.bits_per_second' > "${iperf_file}"
+	export parallel_result=$(cat "${iperf_file}")
+	export parallel_units="bits per second"
+
+	if [ "$COLLECT_ALL" == "true" ]; then
+		iperf3_all_collect_results
+	else
+		metrics_json_init
+		metrics_json_start_array
+
+		local json="$(cat << EOF
+		{
+			"parallel": {
+				"Result" : $parallel_result,
+				"Units" : "$parallel_units"
+			}
+		}
+EOF
+)"
+		metrics_json_add_array_element "$json"
+		metrics_json_end_array "Results"
+	fi
+}
+
+function iperf3_cpu() {
+	# Start server
+	local transmit_timeout="80"
+
+	kubectl exec -i "$client_pod_name" -- sh -c "iperf3 -J -c ${server_ip_add} -t ${transmit_timeout}" | jq '.end.cpu_utilization_percent.host_total' > "${iperf_file}"
+	export cpu_result=$(cat "${iperf_file}")
+	export cpu_units="percent"
+
+	if [ "$COLLECT_ALL" == "true" ]; then
+		iperf3_all_collect_results
+	else
+		metrics_json_init
+		metrics_json_start_array
+
+		local json="$(cat << EOF
+		{
+			"cpu": {
+				"Result" : $cpu_result,
+				"Units"  : "$cpu_units"
+			}
+		}
+EOF
+)"
+
+		metrics_json_add_array_element "$json"
+		metrics_json_end_array "Results"
+	fi
+}
+
+function iperf3_start_deployment() {
+	cmds=("bc" "jq")
+	check_cmds "${cmds[@]}"
+
+	# Check no processes are left behind
+	check_processes
+
+	export service="iperf3-server"
+	export deployment="iperf3-server-deployment"
+
+	wait_time=20
+	sleep_time=2
+
+	# Create deployment
+	kubectl create -f "${SCRIPT_PATH}/runtimeclass_workloads/iperf3-deployment.yaml"
+
+	# Check deployment creation
+	local cmd="kubectl wait --for=condition=Available deployment/${deployment}"
+	waitForProcess "$wait_time" "$sleep_time" "$cmd"
+
+	# Create DaemonSet
+	kubectl create -f "${SCRIPT_PATH}/runtimeclass_workloads/iperf3-daemonset.yaml"
+
+	# Expose deployment
+	kubectl expose deployment/"${deployment}"
+
+	# Get the names of the server pod
+	export server_pod_name=$(kubectl get pods -o name | grep server | cut -d '/' -f2)
+
+	# Verify the server pod is working
+	local cmd="kubectl get pod $server_pod_name -o yaml | grep 'phase: Running'"
+	waitForProcess "$wait_time" "$sleep_time" "$cmd"
+
+	# Get the names of client pod
+	export client_pod_name=$(kubectl get pods -o name | grep client | cut -d '/' -f2)
+
+	# Verify the client pod is working
+	local cmd="kubectl get pod $client_pod_name -o yaml | grep 'phase: Running'"
+	waitForProcess "$wait_time" "$sleep_time" "$cmd"
+
+	# Get the ip address of the server pod
+	export server_ip_add=$(kubectl get pod "$server_pod_name" -o jsonpath='{.status.podIP}')
+}
+
+function iperf3_deployment_cleanup() {
+	kubectl delete pod "$server_pod_name" "$client_pod_name"
+	kubectl delete ds iperf3-clients
+	kubectl delete deployment "$deployment"
+	kubectl delete service "$deployment"
+	check_processes
+}
+
+function help() {
+echo "$(cat << EOF
+Usage: $0 "[options]"
+	Description:
+		This script implements a number of network metrics
+		using iperf3.
+
+	Options:
+		-a	Run all tests
+		-b 	Run bandwidth tests
+		-c	Run cpu metrics tests
+		-h	Help
+		-j	Run jitter tests
+EOF
+)"
+}
+
+function main() {
+	init_env
+	iperf3_start_deployment
+
+	local OPTIND
+	while getopts ":abcjph:" opt
+	do
+		case "$opt" in
+		a)	# all tests
+			test_all="1"
+			;;
+		b)	# bandwith test
+			test_bandwith="1"
+			;;
+		c)
+			# run cpu tests
+			test_cpu="1"
+			;;
+		h)
+			help
+			exit 0;
+			;;
+		j)	# jitter tests
+			test_jitter="1"
+			;;
+		p)
+			# run parallel tests
+			test_parallel="1"
+			;;
+		:)
+			echo "Missing argument for -$OPTARG";
+			help
+			exit 1;
+			;;
+		esac
+	done
+	shift $((OPTIND-1))
+
+	[[ -z "$test_bandwith" ]] && \
+	[[ -z "$test_jitter" ]] && \
+	[[ -z "$test_cpu" ]] && \
+	[[ -z "$test_parallel" ]] && \
+	[[ -z "$test_all" ]] && \
+		help && die "Must choose at least one test"
+
+	if [ "$test_bandwith" == "1" ]; then
+		iperf3_bandwidth
+	fi
+
+	if [ "$test_jitter" == "1" ]; then
+		iperf3_jitter
+	fi
+
+	if [ "$test_cpu" == "1" ]; then
+		iperf3_cpu
+	fi
+
+	if [ "$test_parallel" == "1" ]; then
+		iperf3_parallel
+	fi
+
+	if [ "$test_all" == "1" ]; then
+		export COLLECT_ALL=true && iperf3_bandwidth && iperf3_jitter && iperf3_cpu && iperf3_parallel
+	fi
+
+	metrics_json_save
+	iperf3_deployment_cleanup
+}
+
+main "$@"

From badff23c71513cafbe0ed9bd1d414abadb2737a9 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Tue, 1 Aug 2023 16:37:19 +0000
Subject: [PATCH 18/49] metrics: Add iperf3 service yaml for k8s

This PR adds the iperf3 service yaml for k8s.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../iperf3-service.yaml                       | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml

diff --git a/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml
new file mode 100644
index 0000000000..7e83be11ae
--- /dev/null
+++ b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml
@@ -0,0 +1,44 @@
+#
+# Copyright (c) 2021 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: iperf3-server-deployment
+  labels:
+    app: iperf3-server
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: iperf3-server
+  template:
+    metadata:
+      labels:
+        app: iperf3-server
+    spec:
+      affinity:
+        nodeAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+          - weight: 1
+            preference:
+              matchExpressions:
+              - key: kubernetes.io/role
+                operator: In
+                values:
+                - master
+      tolerations:
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+      - name: iperf3-server
+        image: networkstatic/iperf3
+        args: ['-s']
+        ports:
+        - containerPort: 5201
+          name: server
+      terminationGracePeriodSeconds: 0
+      runtimeClassName: kata

From bda83cee5d719b66034039ed4c8cee2cf49da6b0 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Tue, 1 Aug 2023 16:39:26 +0000
Subject: [PATCH 19/49] metrics: Add iperf3 daemonset for k8s

This PR adds the iperf3 daemonset for k8s.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../iperf3-daemonset.yaml                     | 29 +++++++++++++++++++
 .../iperf3-service.yaml                       |  2 +-
 2 files changed, 30 insertions(+), 1 deletion(-)
 create mode 100644 tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-daemonset.yaml

diff --git a/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-daemonset.yaml b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-daemonset.yaml
new file mode 100644
index 0000000000..f0a02bbe7c
--- /dev/null
+++ b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-daemonset.yaml
@@ -0,0 +1,29 @@
+#
+# Copyright (c) 2021-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: iperf3-clients
+  labels:
+    app: iperf3-client
+spec:
+  selector:
+    matchLabels:
+      app: iperf3-client
+  template:
+    metadata:
+      labels:
+        app: iperf3-client
+    spec:
+      tolerations:
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+      - name: iperf3-client
+        image: networkstatic/iperf3
+        command: ['/bin/sh', '-c', 'sleep infinity']
+      terminationGracePeriodSeconds: 0
diff --git a/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml
index 7e83be11ae..2a4b20ca67 100644
--- a/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml
+++ b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-service.yaml
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 Intel Corporation
+# Copyright (c) 2021-2023 Intel Corporation
 #
 # SPDX-License-Identifier: Apache-2.0
 #

From d8439dba89042e058560b8c6f0ef96fe7dcd2e6c Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Tue, 1 Aug 2023 16:45:01 +0000
Subject: [PATCH 20/49] metrics: Add iperf3 deployment yaml

This PR adds the iperf3 deployment yaml.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../iperf3-deployment.yaml                    | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-deployment.yaml

diff --git a/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-deployment.yaml b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-deployment.yaml
new file mode 100644
index 0000000000..2a4b20ca67
--- /dev/null
+++ b/tests/metrics/network/iperf3_kubernetes/runtimeclass_workloads/iperf3-deployment.yaml
@@ -0,0 +1,44 @@
+#
+# Copyright (c) 2021-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: iperf3-server-deployment
+  labels:
+    app: iperf3-server
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: iperf3-server
+  template:
+    metadata:
+      labels:
+        app: iperf3-server
+    spec:
+      affinity:
+        nodeAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+          - weight: 1
+            preference:
+              matchExpressions:
+              - key: kubernetes.io/role
+                operator: In
+                values:
+                - master
+      tolerations:
+        - key: node-role.kubernetes.io/master
+          operator: Exists
+          effect: NoSchedule
+      containers:
+      - name: iperf3-server
+        image: networkstatic/iperf3
+        args: ['-s']
+        ports:
+        - containerPort: 5201
+          name: server
+      terminationGracePeriodSeconds: 0
+      runtimeClassName: kata

From 07694ef3ae8ec202e77a6010bbc4472fb5f9aa5d Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Tue, 1 Aug 2023 16:49:09 +0000
Subject: [PATCH 21/49] metrics: Add Kata Containers network metrics README

This PR adds the Kata Containers network metrics README.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/network/README.md | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
 create mode 100644 tests/metrics/network/README.md

diff --git a/tests/metrics/network/README.md b/tests/metrics/network/README.md
new file mode 100644
index 0000000000..c109a68411
--- /dev/null
+++ b/tests/metrics/network/README.md
@@ -0,0 +1,21 @@
+# Kata Containers network metrics
+
+Kata Containers provides a series of network performance tests. Running these provides a basic reference for measuring network essentials like 
+bandwidth, jitter, latency and parallel bandwidth.
+
+## Performance tools
+
+- `iperf3` measures bandwidth, jitter, CPU usage and the quality of a network link.
+
+## Networking tests
+
+- `k8s-network-metrics-iperf3.sh` measures bandwidth which is the speed of the data transfer.
+
+## Running the tests
+
+Individual tests can be run by hand, for example:
+
+``` 
+$ cd metrics
+$ bash network/iperf3_kubernetes/k8s-network-metrics-iperf3.sh -b
+```

From 58f9a57c207ebd72bff9f8f63da44d731f3036be Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Tue, 1 Aug 2023 16:54:00 +0000
Subject: [PATCH 22/49] metrics: Add network reference to general README
 metrics

This PR adds network reference to the general metrics README.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/README.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/metrics/README.md b/tests/metrics/README.md
index 825018b016..05ccf38919 100644
--- a/tests/metrics/README.md
+++ b/tests/metrics/README.md
@@ -66,6 +66,8 @@ Tests relating to networking. General items could include:
 - parallel bandwidth
 - write and read percentiles 
 
+For further details see the [network tests documentation](network).
+
 ### Storage
 
 Tests relating to the storage (graph, volume) drivers.

From 2fc5f0e2e0ab18e2d39f6a2a43378a87eb4d98db Mon Sep 17 00:00:00 2001
From: Jianyong Wu <jianyong.wu@arm.com>
Date: Sun, 2 Apr 2023 09:09:13 +0000
Subject: [PATCH 23/49] kata-depoly: prepare env for cross build in lib.sh

We leverage three env, TARGET_ARCH means the buid target tuple;
ARCH nearly the same meaning with TARGET_ARCH but has been widely
used in kata; CROSS_BUILD means if you want to do cross compile.

Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
---
 tools/packaging/scripts/lib.sh | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/tools/packaging/scripts/lib.sh b/tools/packaging/scripts/lib.sh
index 1f0173e473..e7a6fc987d 100644
--- a/tools/packaging/scripts/lib.sh
+++ b/tools/packaging/scripts/lib.sh
@@ -19,6 +19,16 @@ short_commit_length=10
 
 hub_bin="hub-bin"
 
+#for cross build
+CROSS_BUILD=${CROSS_BUILD-:}
+BUILDX=""
+PLATFORM=""
+TARGET_ARCH=${TARGET_ARCH:-$(uname -m)}
+ARCH=${ARCH:-$(uname -m)}
+[ "${TARGET_ARCH}" == "aarch64" ] && TARGET_ARCH=arm64
+TARGET_OS=${TARGET_OS:-linux}
+[ "${CROSS_BUILD}" == "true" ] && BUILDX=buildx && PLATFORM="--platform=${TARGET_OS}/${TARGET_ARCH}"
+
 clone_tests_repo() {
 	# KATA_CI_NO_NETWORK is (has to be) ignored if there is
 	# no existing clone.
@@ -189,7 +199,7 @@ get_ovmf_image_name() {
 }
 
 get_virtiofsd_image_name() {
-	ARCH=$(uname -m)
+	ARCH=${ARCH:-$(uname -m)}
 	case ${ARCH} in
 	        "aarch64")
 	                libc="musl"

From e2c31fce235ffdc42ac3efd6405848d788c7ff8e Mon Sep 17 00:00:00 2001
From: Jianyong Wu <jianyong.wu@arm.com>
Date: Mon, 24 Jul 2023 16:06:22 +0800
Subject: [PATCH 24/49] kata-deploy: enable cross build for kata deploy script

kata-deploy-binaries-in-docker.sh is the entry to build kata components.
set some environment to facilitate the following cross build work.

Fixes: #6557
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
---
 .../kata-deploy-binaries-in-docker.sh         | 26 +++++++++++++++++++
 .../local-build/kata-deploy-binaries.sh       | 14 +++++-----
 2 files changed, 33 insertions(+), 7 deletions(-)

diff --git a/tools/packaging/kata-deploy/local-build/kata-deploy-binaries-in-docker.sh b/tools/packaging/kata-deploy/local-build/kata-deploy-binaries-in-docker.sh
index fc82082c4d..ebcc80c814 100755
--- a/tools/packaging/kata-deploy/local-build/kata-deploy-binaries-in-docker.sh
+++ b/tools/packaging/kata-deploy/local-build/kata-deploy-binaries-in-docker.sh
@@ -19,6 +19,29 @@ gid=$(id -g ${USER})
 http_proxy="${http_proxy:-}"
 https_proxy="${https_proxy:-}"
 
+ARCH=${ARCH:-$(uname -m)}
+CROSS_BUILD=
+BUILDX=""
+PLATFORM=""
+TARGET_ARCH=${TARGET_ARCH:-$(uname -m)}
+[ "$(uname -m)" != "${TARGET_ARCH}" ] && CROSS_BUILD=true
+
+[ "${TARGET_ARCH}" == "aarch64" ] && TARGET_ARCH=arm64
+
+# used for cross build
+TARGET_OS=${TARGET_OS:-linux}
+TARGET_ARCH=${TARGET_ARCH:-$ARCH}
+
+[ "${CROSS_BUILD}" == "true" ] && BUILDX="buildx" && PLATFORM="--platform=${TARGET_OS}/${TARGET_ARCH}"
+if [ "${CROSS_BUILD}" == "true" ]; then
+       # check if the current docker support docker buildx
+       docker buildx ls > /dev/null 2>&1 || true
+       [ $? != 0 ] && echo "no docker buildx support, please upgrad your docker" && exit 1
+       # check if docker buildx support target_arch, if not install it
+       r=$(docker buildx ls | grep "${TARGET_ARCH}" || true)
+       [ -z "$r" ] && sudo docker run --privileged --rm tonistiigi/binfmt --install ${TARGET_ARCH}
+fi
+
 if [ "${script_dir}" != "${PWD}" ]; then
 	ln -sf "${script_dir}/build" "${PWD}/build"
 fi
@@ -66,6 +89,9 @@ docker run \
 	--env VIRTIOFSD_CONTAINER_BUILDER="${VIRTIOFSD_CONTAINER_BUILDER:-}" \
 	--env MEASURED_ROOTFS="${MEASURED_ROOTFS:-}" \
 	--env USE_CACHE="${USE_CACHE:-}" \
+	--env CROSS_BUILD="${CROSS_BUILD}" \
+	--env TARGET_ARCH="${TARGET_ARCH}" \
+	--env ARCH="${ARCH}" \
 	--rm \
 	-w ${script_dir} \
 	build-kata-deploy "${kata_deploy_create}" $@
diff --git a/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh b/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh
index 2af5cd36b8..8f819166af 100755
--- a/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh
+++ b/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh
@@ -38,7 +38,7 @@ readonly rootfs_builder="${repo_root_dir}/tools/packaging/guest-image/build_imag
 readonly jenkins_url="http://jenkins.katacontainers.io"
 readonly cached_artifacts_path="lastSuccessfulBuild/artifact/artifacts"
 
-ARCH=$(uname -m)
+ARCH=${ARCH:-$(uname -m)}
 MEASURED_ROOTFS=${MEASURED_ROOTFS:-no}
 USE_CACHE="${USE_CACHE:-"yes"}"
 
@@ -150,7 +150,7 @@ install_image() {
 		image_type+="-${variant}"
 	fi
 
-	local jenkins="${jenkins_url}/job/kata-containers-main-rootfs-${image_type}-$(uname -m)/${cached_artifacts_path}"
+	local jenkins="${jenkins_url}/job/kata-containers-main-rootfs-${image_type}-${ARCH}/${cached_artifacts_path}"
 	local component="rootfs-${image_type}"
 
 	local osbuilder_last_commit="$(get_last_modification "${repo_root_dir}/tools/osbuilder")"
@@ -197,7 +197,7 @@ install_initrd() {
 		initrd_type+="-${variant}"
 	fi
 
-	local jenkins="${jenkins_url}/job/kata-containers-main-rootfs-${initrd_type}-$(uname -m)/${cached_artifacts_path}"
+	local jenkins="${jenkins_url}/job/kata-containers-main-rootfs-${initrd_type}-${ARCH}/${cached_artifacts_path}"
 	local component="rootfs-${initrd_type}"
 
 	local osbuilder_last_commit="$(get_last_modification "${repo_root_dir}/tools/osbuilder")"
@@ -247,7 +247,7 @@ install_cached_kernel_tarball_component() {
 
 	install_cached_tarball_component \
 		"${kernel_name}" \
-		"${jenkins_url}/job/kata-containers-main-${kernel_name}-$(uname -m)/${cached_artifacts_path}" \
+		"${jenkins_url}/job/kata-containers-main-${kernel_name}-${ARCH}/${cached_artifacts_path}" \
 		"${kernel_version}-${kernel_kata_config_version}-$(get_last_modification $(dirname $kernel_builder))" \
 		"$(get_kernel_image_name)" \
 		"${final_tarball_name}" \
@@ -384,7 +384,7 @@ install_qemu_helper() {
 
 	install_cached_tarball_component \
 		"${qemu_name}" \
-		"${jenkins_url}/job/kata-containers-main-${qemu_name}-$(uname -m)/${cached_artifacts_path}" \
+		"${jenkins_url}/job/kata-containers-main-${qemu_name}-${ARCH}/${cached_artifacts_path}" \
 		"${qemu_version}-$(calc_qemu_files_sha256sum)" \
 		"$(get_qemu_image_name)" \
 		"${final_tarball_name}" \
@@ -495,7 +495,7 @@ install_clh_glibc() {
 install_virtiofsd() {
 	install_cached_tarball_component \
 		"virtiofsd" \
-		"${jenkins_url}/job/kata-containers-main-virtiofsd-$(uname -m)/${cached_artifacts_path}" \
+		"${jenkins_url}/job/kata-containers-main-virtiofsd-${ARCH}/${cached_artifacts_path}" \
 		"$(get_from_kata_deps "externals.virtiofsd.version")-$(get_from_kata_deps "externals.virtiofsd.toolchain")" \
 		"$(get_virtiofsd_image_name)" \
 		"${final_tarball_name}" \
@@ -542,7 +542,7 @@ install_shimv2() {
 
 	install_cached_tarball_component \
 		"shim-v2" \
-		"${jenkins_url}/job/kata-containers-main-shim-v2-$(uname -m)/${cached_artifacts_path}" \
+		"${jenkins_url}/job/kata-containers-main-shim-v2-${ARCH}/${cached_artifacts_path}" \
 		"${shim_v2_version}" \
 		"$(get_shim_v2_image_name)" \
 		"${final_tarball_name}" \

From 7923de8999de055c7f55c568d3354d4c36f830e5 Mon Sep 17 00:00:00 2001
From: Jianyong Wu <jianyong.wu@arm.com>
Date: Wed, 19 Jul 2023 17:31:58 +0800
Subject: [PATCH 25/49] static-build: cross build kernel

Prepare cross build environment based on current Dockerfile.

Fixes: #6557
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
---
 tools/packaging/kernel/build-kernel.sh        |  8 +++++--
 tools/packaging/kernel/kata_config_version    |  2 +-
 .../packaging/static-build/kernel/Dockerfile  |  5 +++-
 tools/packaging/static-build/kernel/build.sh  | 24 +++++++++++++++----
 4 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/tools/packaging/kernel/build-kernel.sh b/tools/packaging/kernel/build-kernel.sh
index 18725c1b87..8119be7047 100755
--- a/tools/packaging/kernel/build-kernel.sh
+++ b/tools/packaging/kernel/build-kernel.sh
@@ -65,6 +65,8 @@ kernel_url=""
 #Linux headers for GPU guest fs module building
 linux_headers=""
 
+CROSS_BUILD_ARG=""
+
 MEASURED_ROOTFS=${MEASURED_ROOTFS:-no}
 
 packaging_scripts_dir="${script_dir}/../scripts"
@@ -436,7 +438,7 @@ setup_kernel() {
 
 	info "Copying config file from: ${kernel_config_path}"
 	cp "${kernel_config_path}" ./.config
-	make oldconfig
+	ARCH=${arch_target}  make oldconfig ${CROSS_BUILD_ARG}
 	)
 }
 
@@ -447,7 +449,7 @@ build_kernel() {
 	[ -n "${arch_target}" ] || arch_target="$(uname -m)"
 	arch_target=$(arch_to_kernel "${arch_target}")
 	pushd "${kernel_path}" >>/dev/null
-	make -j $(nproc ${CI:+--ignore 1}) ARCH="${arch_target}"
+	make -j $(nproc ${CI:+--ignore 1}) ARCH="${arch_target}" ${CROSS_BUILD_ARG}
 	if [ "${conf_guest}" == "sev" ]; then
 		make -j $(nproc ${CI:+--ignore 1}) INSTALL_MOD_STRIP=1 INSTALL_MOD_PATH=${kernel_path} modules_install
 	fi
@@ -658,6 +660,8 @@ main() {
 
 	info "Kernel version: ${kernel_version}"
 
+	[ "${arch_target}" != "" -a "${arch_target}" != $(uname -m) ] && CROSS_BUILD_ARG="CROSS_COMPILE=${arch_target}-linux-gnu-"
+
 	case "${subcmd}" in
 		build)
 			build_kernel "${kernel_path}"
diff --git a/tools/packaging/kernel/kata_config_version b/tools/packaging/kernel/kata_config_version
index 58c9bdf9d0..194b81caae 100644
--- a/tools/packaging/kernel/kata_config_version
+++ b/tools/packaging/kernel/kata_config_version
@@ -1 +1 @@
-111
+112
diff --git a/tools/packaging/static-build/kernel/Dockerfile b/tools/packaging/static-build/kernel/Dockerfile
index 183f8a47e2..c806e35743 100644
--- a/tools/packaging/static-build/kernel/Dockerfile
+++ b/tools/packaging/static-build/kernel/Dockerfile
@@ -5,6 +5,8 @@
 FROM ubuntu:22.04
 ENV DEBIAN_FRONTEND=noninteractive
 
+ARG ARCH
+
 # kernel deps
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
@@ -23,4 +25,5 @@ RUN apt-get update && \
 	    rsync \
 	    cpio \
 	    patch && \
-    apt-get clean && apt-get autoclean
+    if [ "${ARCH}" != "$(uname -m)" ]; then apt-get install --no-install-recommends -y gcc-"${ARCH}"-linux-gnu binutils-"${ARCH}"-linux-gnu; fi && \
+    apt-get clean && apt-get autoclean && rm -rf /var/lib/apt/lists/*
diff --git a/tools/packaging/static-build/kernel/build.sh b/tools/packaging/static-build/kernel/build.sh
index b36f06cff8..b8deea4f09 100755
--- a/tools/packaging/static-build/kernel/build.sh
+++ b/tools/packaging/static-build/kernel/build.sh
@@ -14,12 +14,26 @@ source "${script_dir}/../../scripts/lib.sh"
 
 readonly kernel_builder="${repo_root_dir}/tools/packaging/kernel/build-kernel.sh"
 
+BUILDX=
+PLATFORM=
+
 DESTDIR=${DESTDIR:-${PWD}}
 PREFIX=${PREFIX:-/opt/kata}
 container_image="${KERNEL_CONTAINER_BUILDER:-$(get_kernel_image_name)}"
 
+if [ "${CROSS_BUILD}" == "true" ]; then
+       container_image="${container_image}-${ARCH}-cross-build"
+       # Need to build a s390x image due to an issue at
+       # https://github.com/kata-containers/kata-containers/pull/6586#issuecomment-1603189242
+       if [ ${ARCH} == "s390x" ]; then
+               BUILDX="buildx"
+               PLATFORM="--platform=linux/s390x"
+       fi
+fi
+
 sudo docker pull ${container_image} || \
-	(sudo docker build -t "${container_image}" "${script_dir}" && \
+	(sudo docker ${BUILDX} build ${PLATFORM} \
+	--build-arg ARCH=${ARCH} -t "${container_image}" "${script_dir}" && \
 	 # No-op unless PUSH_TO_REGISTRY is exported as "yes"
 	 push_to_registry "${container_image}")
 
@@ -27,21 +41,21 @@ sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
 	-w "${PWD}" \
 	--env MEASURED_ROOTFS="${MEASURED_ROOTFS:-}" \
 	"${container_image}" \
-	bash -c "${kernel_builder} $* setup"
+	bash -c "${kernel_builder} -a ${ARCH} $* setup"
 
 sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
 	-w "${PWD}" \
 	"${container_image}" \
-	bash -c "${kernel_builder} $* build"
+	bash -c "${kernel_builder} -a ${ARCH} $* build"
 
 sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
 	-w "${PWD}" \
 	--env DESTDIR="${DESTDIR}" --env PREFIX="${PREFIX}" \
 	"${container_image}" \
-	bash -c "${kernel_builder} $* install"
+	bash -c "${kernel_builder} -a ${ARCH} $* install"
 
 sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
 	-w "${PWD}" \
 	--env DESTDIR="${DESTDIR}" --env PREFIX="${PREFIX}" \
 	"${container_image}" \
-	bash -c "${kernel_builder} $* build-headers"
+	bash -c "${kernel_builder} -a ${ARCH} $* build-headers"

From 11631c681ae8f4fafc4a1001a33eba6fcf3ae0bf Mon Sep 17 00:00:00 2001
From: Jianyong Wu <jianyong.wu@arm.com>
Date: Tue, 20 Jun 2023 22:30:37 +0800
Subject: [PATCH 26/49] static-build: enable cross build for shim-v2

shim-v2 has go and rust code. For rust code, we use messense/rust-musl-cross
to build for speed up as it doesn't depends on qemu emulation. Build go
code based on docker buildx as it doesn't support cross build now.

Fixes: #6557
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
---
 tools/packaging/static-build/shim-v2/build.sh | 21 ++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/tools/packaging/static-build/shim-v2/build.sh b/tools/packaging/static-build/shim-v2/build.sh
index 9330445646..aeb48f2da8 100755
--- a/tools/packaging/static-build/shim-v2/build.sh
+++ b/tools/packaging/static-build/shim-v2/build.sh
@@ -16,6 +16,7 @@ VMM_CONFIGS="qemu fc"
 
 GO_VERSION=${GO_VERSION}
 RUST_VERSION=${RUST_VERSION}
+CC=""
 
 DESTDIR=${DESTDIR:-${PWD}}
 PREFIX=${PREFIX:-/opt/kata}
@@ -23,29 +24,43 @@ container_image="${SHIM_V2_CONTAINER_BUILDER:-$(get_shim_v2_image_name)}"
 
 EXTRA_OPTS="${EXTRA_OPTS:-""}"
 
+[ "${CROSS_BUILD}" == "true" ] && container_image_bk="${container_image}" && container_image="${container_image}-cross-build"
 sudo docker pull ${container_image} || \
-       	(sudo docker build  \
+	(sudo docker ${BUILDX} build ${PLATFORM}  \
 		--build-arg GO_VERSION="${GO_VERSION}" \
 		--build-arg RUST_VERSION="${RUST_VERSION}" \
 		-t "${container_image}" \
 		"${script_dir}" && \
 	 push_to_registry "${container_image}")
 
-arch=$(uname -m)
+arch=${ARCH:-$(uname -m)}
+GCC_ARCH=${arch}
 if [ ${arch} = "ppc64le" ]; then
+	GCC_ARCH="powerpc64le"
 	arch="ppc64"
 fi
 
+#Build rust project using cross build musl image to speed up
+[[ "${CROSS_BUILD}" == "true" && ${ARCH} != "s390x" ]] && container_image="messense/rust-musl-cross:${GCC_ARCH}-musl" && CC=${GCC_ARCH}-unknown-linux-musl-gcc
+
 sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
+	--env CROSS_BUILD=${CROSS_BUILD} \
+	--env ARCH=${ARCH} \
+	--env CC="${CC}" \
 	-w "${repo_root_dir}/src/runtime-rs" \
 	"${container_image}" \
 	bash -c "git config --global --add safe.directory ${repo_root_dir} && make PREFIX=${PREFIX} QEMUCMD=qemu-system-${arch}"
 
 sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
+	--env CROSS_BUILD=${CROSS_BUILD} \
+        --env ARCH=${ARCH} \
+        --env CC="${CC}" \
 	-w "${repo_root_dir}/src/runtime-rs" \
 	"${container_image}" \
 	bash -c "git config --global --add safe.directory ${repo_root_dir} && make PREFIX="${PREFIX}" DESTDIR="${DESTDIR}" install"
-	
+
+[ "${CROSS_BUILD}" == "true" ] && container_image="${container_image_bk}-cross-build"
+
 sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
 	-w "${repo_root_dir}/src/runtime" \
 	"${container_image}" \

From 2205fb9d050ad2bdaf77edd3a16f8a543d490333 Mon Sep 17 00:00:00 2001
From: Jianyong Wu <jianyong.wu@arm.com>
Date: Sun, 2 Apr 2023 10:58:29 +0000
Subject: [PATCH 27/49] static-build: enable cross build for virtiofsd

Based on messense/rust-musl-cross which offer cross build musl lib
environment to cross compile virtiofsd.

Fixes: #6557
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
---
 .../static-build/virtiofsd/build-static-virtiofsd.sh        | 2 +-
 tools/packaging/static-build/virtiofsd/build.sh             | 6 ++++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/tools/packaging/static-build/virtiofsd/build-static-virtiofsd.sh b/tools/packaging/static-build/virtiofsd/build-static-virtiofsd.sh
index 6bad74ffb2..a6370f8944 100755
--- a/tools/packaging/static-build/virtiofsd/build-static-virtiofsd.sh
+++ b/tools/packaging/static-build/virtiofsd/build-static-virtiofsd.sh
@@ -8,7 +8,7 @@ set -o errexit
 set -o nounset
 set -o pipefail
 
-ARCH=$(uname -m)
+ARCH=${ARCH:-$(uname -m)}
 ARCH_LIBC=""
 LIBC=""
 
diff --git a/tools/packaging/static-build/virtiofsd/build.sh b/tools/packaging/static-build/virtiofsd/build.sh
index 6eb5ad51bf..7f85c5c0d4 100755
--- a/tools/packaging/static-build/virtiofsd/build.sh
+++ b/tools/packaging/static-build/virtiofsd/build.sh
@@ -13,6 +13,7 @@ readonly virtiofsd_builder="${script_dir}/build-static-virtiofsd.sh"
 
 source "${script_dir}/../../scripts/lib.sh"
 
+ARCH=${ARCH:-$(uname -m)}
 DESTDIR=${DESTDIR:-${PWD}}
 PREFIX=${PREFIX:-/opt/kata}
 kata_version="${kata_version:-}"
@@ -32,7 +33,6 @@ package_output_dir="${package_output_dir:-}"
 [ -n "${virtiofsd_toolchain}" ] || die "Failed to get the rust toolchain to build virtiofsd"
 [ -n "${virtiofsd_zip}" ] || die "Failed to get virtiofsd binary URL"
 
-ARCH=$(uname -m)
 case ${ARCH} in
 	"aarch64")
 		libc="musl"
@@ -49,9 +49,10 @@ case ${ARCH} in
 esac
 
 container_image="${VIRTIOFSD_CONTAINER_BUILDER:-$(get_virtiofsd_image_name)}"
+[ "${CROSS_BUILD}" == "true" ] && container_image="${container_image}-cross-build"
 
 sudo docker pull ${container_image} || \
-	(sudo docker build \
+	(sudo docker $BUILDX build $PLATFORM \
 		--build-arg RUST_TOOLCHAIN="${virtiofsd_toolchain}" \
 		-t "${container_image}" "${script_dir}/${libc}" && \
 	 # No-op unless PUSH_TO_REGISTRY is exported as "yes"
@@ -64,5 +65,6 @@ sudo docker run --rm -i -v "${repo_root_dir}:${repo_root_dir}" \
 	--env virtiofsd_repo="${virtiofsd_repo}" \
 	--env virtiofsd_version="${virtiofsd_version}" \
 	--env virtiofsd_zip="${virtiofsd_zip}" \
+	--env ARCH="${ARCH}" \
 	"${container_image}" \
 	bash -c "${virtiofsd_builder}"

From f7642480951564d0625665a07cd8bd6e4371f3e2 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Mon, 31 Jul 2023 17:54:03 +0000
Subject: [PATCH 28/49] gha: Add FIO test to run metrics yaml

This PR adds FIO test to run metrics yaml.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .github/workflows/run-metrics.yaml            |  5 +++-
 tests/metrics/storage/fio-k8s/README.md       |  6 ++++-
 tests/metrics/storage/fio-k8s/fio-test-ci.sh  |  2 +-
 .../scripts/{ => dax-compare-test}/Makefile   |  0
 .../scripts/{ => dax-compare-test}/README.md  | 24 ++++++++++++-------
 .../compare-virtiofsd-dax.sh                  |  0
 6 files changed, 25 insertions(+), 12 deletions(-)
 rename tests/metrics/storage/fio-k8s/scripts/{ => dax-compare-test}/Makefile (100%)
 rename tests/metrics/storage/fio-k8s/scripts/{ => dax-compare-test}/README.md (60%)
 rename tests/metrics/storage/fio-k8s/scripts/{ => dax-compare-test}/compare-virtiofsd-dax.sh (100%)

diff --git a/.github/workflows/run-metrics.yaml b/.github/workflows/run-metrics.yaml
index f1ac9d61ad..596a3dea46 100644
--- a/.github/workflows/run-metrics.yaml
+++ b/.github/workflows/run-metrics.yaml
@@ -49,9 +49,12 @@ jobs:
       - name: run tensorflow test
         run:  bash tests/metrics/gha-run.sh run-test-tensorflow
 
+      - name: run fio test
+        run:  bash tests/metrics/gha-run.sh run-test-fio
+
       - name: make metrics tarball ${{ matrix.vmm }}
         run: bash tests/metrics/gha-run.sh make-tarball-results
-          
+
       - name: archive metrics results ${{ matrix.vmm }}
         uses: actions/upload-artifact@v3
         with:
diff --git a/tests/metrics/storage/fio-k8s/README.md b/tests/metrics/storage/fio-k8s/README.md
index 8daad84083..96798849a6 100644
--- a/tests/metrics/storage/fio-k8s/README.md
+++ b/tests/metrics/storage/fio-k8s/README.md
@@ -3,9 +3,12 @@
 This is an automation to run `fio` with Kubernetes.
 
 ## Requirements:
+
 - Kubernetes cluster running.
 - Kata configured as `runtimeclass`. 
-The test is structured as follow: 
+
+## Test structure:
+
 - [fio-test]: Program wrapper to launch `fio` in a K8s pod. 
 - [pkg]: Library code that could be used for more `fio` automation. 
 - [configs]: Configuration files used by [fio-test]. 
@@ -13,6 +16,7 @@ The test is structured as follow:
 - [report] Jupyter Notebook to create reports for data generated by [DAX-compare-test].
 
 ## Top-level Makefile targets
+
 - `build`: Build `fio` metrics.
 - `test`: quick test, used to verify changes in [fio-test]. 
 - `run`: Run `fio` metrics and generate reports. 
diff --git a/tests/metrics/storage/fio-k8s/fio-test-ci.sh b/tests/metrics/storage/fio-k8s/fio-test-ci.sh
index 3ae51bea40..a05ca9bf0e 100755
--- a/tests/metrics/storage/fio-k8s/fio-test-ci.sh
+++ b/tests/metrics/storage/fio-k8s/fio-test-ci.sh
@@ -9,7 +9,7 @@ set -e
 # General env
 SCRIPT_PATH=$(dirname "$(readlink -f "$0")")
 source "${SCRIPT_PATH}/../../lib/common.bash"
-FIO_PATH="${GOPATH}/src/${TEST_REPO}/metrics/storage/fio-k8s"
+FIO_PATH="${GOPATH}/src/github.com/kata-containers/kata-containers/tests/metrics/storage/fio-k8s"
 TEST_NAME="${TEST_NAME:-fio}"
 
 function main() {
diff --git a/tests/metrics/storage/fio-k8s/scripts/Makefile b/tests/metrics/storage/fio-k8s/scripts/dax-compare-test/Makefile
similarity index 100%
rename from tests/metrics/storage/fio-k8s/scripts/Makefile
rename to tests/metrics/storage/fio-k8s/scripts/dax-compare-test/Makefile
diff --git a/tests/metrics/storage/fio-k8s/scripts/README.md b/tests/metrics/storage/fio-k8s/scripts/dax-compare-test/README.md
similarity index 60%
rename from tests/metrics/storage/fio-k8s/scripts/README.md
rename to tests/metrics/storage/fio-k8s/scripts/dax-compare-test/README.md
index 71cfdea846..c8b00f0f98 100644
--- a/tests/metrics/storage/fio-k8s/scripts/README.md
+++ b/tests/metrics/storage/fio-k8s/scripts/dax-compare-test/README.md
@@ -1,6 +1,6 @@
 # FIO in Kubernetes
 
-This test run `fio` jobs to measure how Kata Containers work using virtio-fs DAX. The test works using Kubernetes.  
+This test runs `fio` jobs to measure how Kata Containers work using virtio-fs DAX. The test works using Kubernetes.  
 The test has to run in a single node cluster, it is needed as the test modifies Kata configuration file. 
 
 The `virtio-fs` options that this test will use are: 
@@ -17,25 +17,31 @@ This test by default iterates over different `virtio-fs` configurations.
 
 | test name                 | DAX | thread pool size | cache mode |
 |---------------------------|-----|------------------|------------|
-| pool_0_cache_auto_no_DAX  | no | 0 | auto | pool_0_cache_auto_DAX | yes | 0 | auto |
+| pool_0_cache_auto_no_DAX  | no  | 0                | auto       |
+| pool_0_cache_auto_DAX     | yes | 0                | auto       |
 
 The `fio` options used are: 
 
 `ioengine`: How the IO requests are issued to the kernel. 
 * `libaio`: Supports async IO for both direct and buffered IO. 
 * `mmap`: File is memory mapped with mmap(2) and data copied to/from using memcpy(3). 
+
 `rw type`: Type of I/O pattern. 
 * `randread`: Random reads. 
 * `randrw`: Random mixed reads and writes. 
 * `randwrite`: Random writes. 
 * `read`: Sequential reads. 
-* `write`: Sequential writes. Additional notes: Some jobs contain a `multi` prefix. This means that the same job runs more than once at the same time using its own file.
+* `write`: Sequential writes.
+
+Additional notes: Some jobs contain a `multi` prefix. This means that the same job runs more than once at the same time using its own file.
 
 ### Static `fio` values:
+
 Some `fio` values are not modified over all the jobs. 
-`runtime`: Tell `fio` to terminate processing after the specified period of time(seconds). 
-`loops`: Run the specified number of iterations of this job. Used to repeat the same workload a given number of times. 
-`iodepth`: Number of I/O units to keep in flight against the file. Note that increasing `iodepth` beyond 1 will not affect synchronous `ioengine`. 
-`size`: The total size of file I/O for each thread of this job. 
-`direct`: If value is true, use non-buffered I/O. This is usually O_`DIRECT`. 
-`blocksize`: The block size in bytes used for I/O units
+
+* `runtime`: Tell `fio` to terminate processing after the specified period of time(seconds).
+* `loops`: Run the specified number of iterations of this job. Used to repeat the same workload a given number of times.
+* `iodepth`: Number of I/O units to keep in flight against the file. Note that increasing `iodepth` beyond 1 will not affect synchronous `ioengine`.
+* `size`: The total size of file I/O for each thread of this job.
+* `direct`: If value is true, use non-buffered I/O. This is usually O_`DIRECT`.
+* `blocksize`: The block size in bytes used for I/O units.
diff --git a/tests/metrics/storage/fio-k8s/scripts/compare-virtiofsd-dax.sh b/tests/metrics/storage/fio-k8s/scripts/dax-compare-test/compare-virtiofsd-dax.sh
similarity index 100%
rename from tests/metrics/storage/fio-k8s/scripts/compare-virtiofsd-dax.sh
rename to tests/metrics/storage/fio-k8s/scripts/dax-compare-test/compare-virtiofsd-dax.sh

From 35d6d86ab57664b805b8acb0aed5bab94e23055f Mon Sep 17 00:00:00 2001
From: Jianyong Wu <jianyong.wu@arm.com>
Date: Wed, 19 Jul 2023 17:41:42 +0800
Subject: [PATCH 29/49] static-build: enable cross-build for image build

It's too long a time to cross build agent based on docker buildx, thus
we cross build rootfs based on a container with cross compile toolchain
of gcc and rust with musl libc. Then we get fast build just like native
build.

rootfs initrd cross build is disabled as no cross compile tolchain for
rust with musl lib if found for alpine and based on docker buildx takes
too long a time.

Fixes: #6557
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
---
 ci/install_libseccomp.sh                           |  3 ++-
 tools/osbuilder/image-builder/image_builder.sh     | 14 +++++++++++++-
 tools/osbuilder/rootfs-builder/rootfs.sh           | 10 ++++++++++
 .../osbuilder/rootfs-builder/ubuntu/Dockerfile.in  |  2 ++
 tools/packaging/guest-image/build_image.sh         |  8 +++++++-
 .../local-build/kata-deploy-binaries.sh            |  2 ++
 utils.mk                                           |  4 ----
 7 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/ci/install_libseccomp.sh b/ci/install_libseccomp.sh
index 3b1f8aa86f..683d0f65b1 100755
--- a/ci/install_libseccomp.sh
+++ b/ci/install_libseccomp.sh
@@ -87,7 +87,8 @@ build_and_install_libseccomp() {
     curl -sLO "${libseccomp_tarball_url}"
     tar -xf "${libseccomp_tarball}"
     pushd "libseccomp-${libseccomp_version}"
-    ./configure --prefix="${libseccomp_install_dir}" CFLAGS="${cflags}" --enable-static --host="${arch}"
+    [ "${arch}" == $(uname -m) ] && cc_name="" || cc_name="${arch}-linux-gnu-gcc"
+    CC=${cc_name} ./configure --prefix="${libseccomp_install_dir}" CFLAGS="${cflags}" --enable-static --host="${arch}"
     make
     make install
     popd
diff --git a/tools/osbuilder/image-builder/image_builder.sh b/tools/osbuilder/image-builder/image_builder.sh
index 3e7f0babc0..26f37d122a 100755
--- a/tools/osbuilder/image-builder/image_builder.sh
+++ b/tools/osbuilder/image-builder/image_builder.sh
@@ -13,6 +13,16 @@ set -o pipefail
 DOCKER_RUNTIME=${DOCKER_RUNTIME:-runc}
 MEASURED_ROOTFS=${MEASURED_ROOTFS:-no}
 
+#For cross build
+CROSS_BUILD=${CROSS_BUILD:-false}
+BUILDX=""
+PLATFORM=""
+TARGET_ARCH=${TARGET_ARCH:-$(uname -m)}
+ARCH=${ARCH:-$(uname -m)}
+[ "${TARGET_ARCH}" == "aarch64" ] && TARGET_ARCH=arm64
+TARGET_OS=${TARGET_OS:-linux}
+[ "${CROSS_BUILD}" == "true" ] && BUILDX=buildx && PLATFORM="--platform=${TARGET_OS}/${TARGET_ARCH}"
+
 readonly script_name="${0##*/}"
 readonly script_dir=$(dirname "$(readlink -f "$0")")
 readonly lib_file="${script_dir}/../scripts/lib.sh"
@@ -154,7 +164,7 @@ build_with_container() {
 		engine_build_args+=" --runtime ${DOCKER_RUNTIME}"
 	fi
 
-	"${container_engine}" build  \
+	"${container_engine}" ${BUILDX} build ${PLATFORM}  \
 		   ${engine_build_args} \
 		   --build-arg http_proxy="${http_proxy}" \
 		   --build-arg https_proxy="${https_proxy}" \
@@ -189,6 +199,8 @@ build_with_container() {
 		   --env MEASURED_ROOTFS="${MEASURED_ROOTFS}" \
 		   --env SELINUX="${SELINUX}" \
 		   --env DEBUG="${DEBUG}" \
+		   --env ARCH="${ARCH}" \
+		   --env TARGET_ARCH="${TARGET_ARCH}" \
 		   -v /dev:/dev \
 		   -v "${script_dir}":"/osbuilder" \
 		   -v "${script_dir}/../scripts":"/scripts" \
diff --git a/tools/osbuilder/rootfs-builder/rootfs.sh b/tools/osbuilder/rootfs-builder/rootfs.sh
index 22940ee994..89efd388b5 100755
--- a/tools/osbuilder/rootfs-builder/rootfs.sh
+++ b/tools/osbuilder/rootfs-builder/rootfs.sh
@@ -31,6 +31,16 @@ SELINUX=${SELINUX:-"no"}
 lib_file="${script_dir}/../scripts/lib.sh"
 source "$lib_file"
 
+#For cross build
+CROSS_BUILD=${CROSS_BUILD:-false}
+BUILDX=""
+PLATFORM=""
+TARGET_ARCH=${TARGET_ARCH:-$(uname -m)}
+ARCH=${ARCH:-$(uname -m)}
+[ "${TARGET_ARCH}" == "aarch64" ] && TARGET_ARCH=arm64
+TARGET_OS=${TARGET_OS:-linux}
+[ "${CROSS_BUILD}" == "true" ] && BUILDX=buildx && PLATFORM="--platform=${TARGET_OS}/${TARGET_ARCH}"
+
 handle_error() {
 	local exit_code="${?}"
 	local line_number="${1:-}"
diff --git a/tools/osbuilder/rootfs-builder/ubuntu/Dockerfile.in b/tools/osbuilder/rootfs-builder/ubuntu/Dockerfile.in
index 937ea89f43..61307e9560 100644
--- a/tools/osbuilder/rootfs-builder/ubuntu/Dockerfile.in
+++ b/tools/osbuilder/rootfs-builder/ubuntu/Dockerfile.in
@@ -8,6 +8,7 @@ FROM ${IMAGE_REGISTRY}/ubuntu:@OS_VERSION@
 
 # makedev tries to mknod from postinst
 RUN [ -x /usr/bin/systemd-detect-virt ] || ( echo "echo docker" >/usr/bin/systemd-detect-virt && chmod +x /usr/bin/systemd-detect-virt )
+# hadolint ignore=DL3009,SC2046
 RUN apt-get update && \
     DEBIAN_FRONTEND=noninteractive \
     apt-get --no-install-recommends -y install \
@@ -18,6 +19,7 @@ RUN apt-get update && \
          libc_arch="$gcc_arch" && \
          [ "$gcc_arch" = aarch64 ] && libc_arch=arm64; \
          [ "$gcc_arch" = ppc64le ] && gcc_arch=powerpc64le && libc_arch=ppc64el; \
+         [ "$gcc_arch" = s390x ] && gcc_arch=s390x && libc_arch=s390x; \
          [ "$gcc_arch" = x86_64 ] && gcc_arch=x86-64 && libc_arch=amd64; \
          echo "gcc-$gcc_arch-linux-gnu libc6-dev-$libc_arch-cross")) \
     git \
diff --git a/tools/packaging/guest-image/build_image.sh b/tools/packaging/guest-image/build_image.sh
index fad6646517..372ddf3a46 100755
--- a/tools/packaging/guest-image/build_image.sh
+++ b/tools/packaging/guest-image/build_image.sh
@@ -21,7 +21,13 @@ readonly osbuilder_dir="$(cd "${repo_root_dir}/tools/osbuilder" && pwd)"
 
 export GOPATH=${GOPATH:-${HOME}/go}
 
-arch_target="$(uname -m)"
+ARCH=${ARCH:-$(uname -m)}
+if [ $(uname -m) == "${ARCH}" ]; then
+       arch_target="$(uname -m)"
+else
+       arch_target="${ARCH}"
+fi
+
 final_artifact_name="kata-containers"
 image_initrd_extension=".img"
 
diff --git a/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh b/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh
index 8f819166af..a13f9c51f1 100755
--- a/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh
+++ b/tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh
@@ -208,6 +208,8 @@ install_initrd() {
 	local libseccomp_version="$(get_from_kata_deps "externals.libseccomp.version")"
 	local rust_version="$(get_from_kata_deps "languages.rust.meta.newest-version")"
 
+	[[ "${ARCH}" == "aarch64" && "${CROSS_BUILD}" == "true" ]] && echo "warning: Don't cross build initrd for aarch64 as it's too slow" && exit 0
+
 	install_cached_tarball_component \
 		"${component}" \
 		"${jenkins}" \
diff --git a/utils.mk b/utils.mk
index 3c809cf544..1a153e26cc 100644
--- a/utils.mk
+++ b/utils.mk
@@ -157,10 +157,6 @@ endif
 
 
 EXTRA_RUSTFLAGS :=
-ifeq ($(ARCH), aarch64)
-    override EXTRA_RUSTFLAGS = -C link-arg=-lgcc
-    $(warning "WARNING: aarch64-musl needs extra symbols from libgcc")
-endif
 
 ifneq ($(HOST_ARCH),$(ARCH))
     ifeq ($(CC),)

From 7ffc0c1225c3972efd2c227504e5194f7866ddf6 Mon Sep 17 00:00:00 2001
From: Jianyong Wu <jianyong.wu@arm.com>
Date: Sun, 2 Apr 2023 09:25:22 +0000
Subject: [PATCH 30/49] static-build: enable cross build for qemu

Depends on mutiarch feature of ubuntu, we can set up cross build
environment easily and achive as good build performance as native
build.

Fixes: #6557
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
---
 tools/packaging/static-build/qemu/Dockerfile  | 61 +++++++++++++------
 .../static-build/qemu/build-base-qemu.sh      | 14 ++++-
 .../packaging/static-build/qemu/build-qemu.sh | 12 +++-
 3 files changed, 62 insertions(+), 25 deletions(-)

diff --git a/tools/packaging/static-build/qemu/Dockerfile b/tools/packaging/static-build/qemu/Dockerfile
index 930a907817..f5e812c915 100644
--- a/tools/packaging/static-build/qemu/Dockerfile
+++ b/tools/packaging/static-build/qemu/Dockerfile
@@ -8,8 +8,23 @@ from ubuntu:20.04
 # This is required to keep build dependencies with security fixes.
 ARG CACHE_TIMEOUT
 ARG DEBIAN_FRONTEND=noninteractive
+ARG DPKG_ARCH
+ARG ARCH
+ARG GCC_ARCH
+ARG PREFIX
 
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+RUN if [ "${ARCH}" != "$(uname -m)" ]; then sed -i 's/^deb/deb [arch=amd64]/g' /etc/apt/sources.list && \
+    dpkg --add-architecture "${DPKG_ARCH#:}" && \
+    echo "deb [arch=${DPKG_ARCH#:}] http://ports.ubuntu.com/ focal main restricted" >> /etc/apt/sources.list && \
+    echo "deb [arch=${DPKG_ARCH#:}] http://ports.ubuntu.com/ focal-updates main restricted" >> /etc/apt/sources.list && \
+    echo "deb [arch=${DPKG_ARCH#:}] http://ports.ubuntu.com/ focal universe" >> /etc/apt/sources.list && \
+    echo "deb [arch=${DPKG_ARCH#:}] http://ports.ubuntu.com/ focal-updates universe" >> /etc/apt/sources.list && \
+    echo "deb [arch=${DPKG_ARCH#:}] http://ports.ubuntu.com/ focal multiverse" >> /etc/apt/sources.list && \
+    echo "deb [arch=${DPKG_ARCH#:}] http://ports.ubuntu.com/ focal-updates multiverse" >> /etc/apt/sources.list && \
+    echo "deb [arch=${DPKG_ARCH#:}] http://ports.ubuntu.com/ focal-backports main restricted universe multiverse" >> /etc/apt/sources.list; fi
+
 RUN apt-get update && apt-get upgrade -y && \
     apt-get --no-install-recommends install -y \
 	    apt-utils \
@@ -19,37 +34,43 @@ RUN apt-get update && apt-get upgrade -y && \
 	    bison \
 	    ca-certificates \
 	    cpio \
+	    dpkg-dev \
 	    flex \
 	    gawk \
-	    libaudit-dev \
-	    libblkid-dev \
-	    libcap-dev \
-	    libcap-ng-dev \
-	    libdw-dev \
-	    libelf-dev \
-	    libffi-dev \
-	    libglib2.0-0 \
-	    libglib2.0-dev \
-	    libglib2.0-dev git \
-	    libltdl-dev \
-	    libmount-dev \
-	    libpixman-1-dev \
-	    libselinux1-dev \
-	    libtool \
+	    libaudit-dev${DPKG_ARCH} \
+	    libblkid-dev${DPKG_ARCH} \
+	    libcap-dev${DPKG_ARCH} \
+	    libcap-ng-dev${DPKG_ARCH} \
+	    libdw-dev${DPKG_ARCH} \
+	    libelf-dev${DPKG_ARCH} \
+	    libffi-dev${DPKG_ARCH} \
+	    libglib2.0-0${DPKG_ARCH} \
+	    libglib2.0-dev${DPKG_ARCH} \
+	    libglib2.0-dev${DPKG_ARCH} git \
+	    libltdl-dev${DPKG_ARCH} \
+	    libmount-dev${DPKG_ARCH} \
+	    libpixman-1-dev${DPKG_ARCH} \
+	    libselinux1-dev${DPKG_ARCH} \
+	    libtool${DPKG_ARCH} \
 	    make \
 	    ninja-build \
-	    pkg-config \
-	    libseccomp-dev \
-	    libseccomp2 \
+	    pkg-config${DPKG_ARCH} \
+	    libseccomp-dev${DPKG_ARCH} \
+	    libseccomp2${DPKG_ARCH} \
 	    patch \
 	    python \
 	    python-dev \
 	    rsync \
-	    zlib1g-dev && \
-    if [ "$(uname -m)" != "s390x" ]; then apt-get install -y --no-install-recommends libpmem-dev; fi && \
+	    zlib1g-dev${DPKG_ARCH} && \
+    if [ "${ARCH}" != s390x ]; then apt-get install -y --no-install-recommends libpmem-dev${DPKG_ARCH}; fi && \
+    GCC_ARCH="${ARCH}" && if [ "${ARCH}" = "ppc64le" ]; then GCC_ARCH="powerpc64le"; fi && \
+    if [ "${ARCH}" != "$(uname -m)" ]; then apt-get install --no-install-recommends -y gcc-"${GCC_ARCH}"-linux-gnu; fi && \
     apt-get clean && rm -rf /var/lib/apt/lists/
 
 RUN git clone  https://github.com/axboe/liburing/ ~/liburing && \
     cd ~/liburing && \
     git checkout tags/liburing-2.1 && \
+    GCC_ARCH="${ARCH}" && if [ "${ARCH}" = "ppc64le" ]; then GCC_ARCH="powerpc64le"; fi && \
+    if [ "${ARCH}" != "$(uname -m)" ]; then PREFIX="${GCC_ARCH}-linux-gnu"; fi && \
+    ./configure --cc=${GCC_ARCH}-linux-gnu-gcc --cxx=${GCC_ARCH}-linux-gnu-cpp --prefix=/usr/${PREFIX}/ && \
     make && make install && ldconfig
diff --git a/tools/packaging/static-build/qemu/build-base-qemu.sh b/tools/packaging/static-build/qemu/build-base-qemu.sh
index 9767e5d548..1514419d78 100755
--- a/tools/packaging/static-build/qemu/build-base-qemu.sh
+++ b/tools/packaging/static-build/qemu/build-base-qemu.sh
@@ -14,6 +14,12 @@ readonly qemu_builder="${script_dir}/build-qemu.sh"
 source "${script_dir}/../../scripts/lib.sh"
 source "${script_dir}/../qemu.blacklist"
 
+ARCH=${ARCH:-$(uname -m)}
+dpkg_arch=":${ARCH}"
+[ ${dpkg_arch} == ":aarch64" ] && dpkg_arch=":arm64"
+[ ${dpkg_arch} == ":x86_64" ] && dpkg_arch=""
+[ "${dpkg_arch}" == ":ppc64le" ] && dpkg_arch=":ppc64el"
+
 packaging_dir="${script_dir}/../.."
 qemu_destdir="/tmp/qemu-static/"
 container_engine="${USE_PODMAN:+podman}"
@@ -39,11 +45,14 @@ CACHE_TIMEOUT=$(date +"%Y-%m-%d")
 [ -n "${build_suffix}" ] && PKGVERSION="kata-static-${build_suffix}" || PKGVERSION="kata-static"
 
 container_image="${QEMU_CONTAINER_BUILDER:-$(get_qemu_image_name)}"
+[ "${CROSS_BUILD}" == "true" ] && container_image="${container_image}-cross-build"
 
 sudo docker pull ${container_image} || (sudo "${container_engine}" build \
 	--build-arg CACHE_TIMEOUT="${CACHE_TIMEOUT}" \
 	--build-arg http_proxy="${http_proxy}" \
 	--build-arg https_proxy="${https_proxy}" \
+	--build-arg DPKG_ARCH="${dpkg_arch}" \
+	--build-arg ARCH="${ARCH}" \
 	"${packaging_dir}" \
 	-f "${script_dir}/Dockerfile" \
 	-t "${container_image}" && \
@@ -54,13 +63,14 @@ sudo "${container_engine}" run \
 	--rm \
 	-i \
 	--env BUILD_SUFFIX="${build_suffix}" \
-	--env HYPERVISOR_NAME="${HYPERVISOR_NAME}" \
 	--env PKGVERSION="${PKGVERSION}" \
 	--env QEMU_DESTDIR="${qemu_destdir}" \
 	--env QEMU_REPO="${qemu_repo}" \
-	--env QEMU_VERSION="${qemu_version}" \
 	--env QEMU_TARBALL="${qemu_tar}" \
 	--env PREFIX="${prefix}" \
+	--env HYPERVISOR_NAME="${HYPERVISOR_NAME}" \
+	--env QEMU_VERSION_NUM="${qemu_version}" \
+	--env ARCH="${ARCH}" \
 	-v "${repo_root_dir}:/root/kata-containers" \
 	-v "${PWD}":/share "${container_image}" \
 	bash -c "/root/kata-containers/tools/packaging/static-build/qemu/build-qemu.sh"
diff --git a/tools/packaging/static-build/qemu/build-qemu.sh b/tools/packaging/static-build/qemu/build-qemu.sh
index edab348910..cab5f251b1 100755
--- a/tools/packaging/static-build/qemu/build-qemu.sh
+++ b/tools/packaging/static-build/qemu/build-qemu.sh
@@ -14,13 +14,19 @@ kata_packaging_scripts="${kata_packaging_dir}/scripts"
 kata_static_build_dir="${kata_packaging_dir}/static-build"
 kata_static_build_scripts="${kata_static_build_dir}/scripts"
 
+ARCH=${ARCH:-$(uname -m)}
+
 git clone --depth=1 "${QEMU_REPO}" qemu
 pushd qemu
-git fetch --depth=1 origin "${QEMU_VERSION}"
+git fetch --depth=1 origin "${QEMU_VERSION_NUM}"
 git checkout FETCH_HEAD
 scripts/git-submodule.sh update meson capstone
-${kata_packaging_scripts}/patch_qemu.sh "${QEMU_VERSION}" "${kata_packaging_dir}/qemu/patches"
-PREFIX="${PREFIX}" ${kata_packaging_scripts}/configure-hypervisor.sh -s "${HYPERVISOR_NAME}" | xargs ./configure  --with-pkgversion="${PKGVERSION}"
+${kata_packaging_scripts}/patch_qemu.sh "${QEMU_VERSION_NUM}" "${kata_packaging_dir}/qemu/patches"
+if [ "$(uname -m)" != "${ARCH}" ] && [ "${ARCH}" == "s390x" ]; then
+       PREFIX="${PREFIX}" ${kata_packaging_scripts}/configure-hypervisor.sh -s "${HYPERVISOR_NAME}" "${ARCH}" | xargs ./configure  --with-pkgversion="${PKGVERSION}" --cc=s390x-linux-gnu-gcc --cross-prefix=s390x-linux-gnu- --prefix="${PREFIX}" --target-list=s390x-softmmu
+else
+       PREFIX="${PREFIX}" ${kata_packaging_scripts}/configure-hypervisor.sh -s "${HYPERVISOR_NAME}" "${ARCH}" | xargs ./configure  --with-pkgversion="${PKGVERSION}"
+fi
 make -j"$(nproc +--ignore 1)"
 make install DESTDIR="${QEMU_DESTDIR}"
 popd

From ad6e53c399b6ac768b0575ba6c59b8d3c0a90686 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Tue, 1 Aug 2023 23:34:15 +0000
Subject: [PATCH 31/49] metrics: Modify boot time values

This PR modifies boot time values limit.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../ci_worker/checkmetrics-json-clh-kata-metric8.toml           | 2 +-
 .../ci_worker/checkmetrics-json-qemu-kata-metric8.toml          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-clh-kata-metric8.toml b/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-clh-kata-metric8.toml
index 9ca4d139b3..a9199a9e0f 100644
--- a/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-clh-kata-metric8.toml
+++ b/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-clh-kata-metric8.toml
@@ -17,7 +17,7 @@ description = "measure container lifecycle timings"
 checkvar = ".\"boot-times\".Results | .[] | .\"to-workload\".Result"
 checktype = "mean"
 midval = 0.69
-minpercent = 30.0
+minpercent = 40.0
 maxpercent = 30.0
 
 [[metric]]
diff --git a/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-qemu-kata-metric8.toml b/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-qemu-kata-metric8.toml
index 4860fa2563..5437d2d6a9 100644
--- a/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-qemu-kata-metric8.toml
+++ b/tests/metrics/cmd/checkmetrics/ci_worker/checkmetrics-json-qemu-kata-metric8.toml
@@ -17,7 +17,7 @@ description = "measure container lifecycle timings"
 checkvar = ".\"boot-times\".Results | .[] | .\"to-workload\".Result"
 checktype = "mean"
 midval = 0.71
-minpercent = 30.0
+minpercent = 40.0
 maxpercent = 30.0
 
 [[metric]]

From 025596b289872549024ad66ac7705bfc803a2487 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= <fabiano.fidencio@intel.com>
Date: Wed, 2 Aug 2023 09:26:24 +0200
Subject: [PATCH 32/49] ci-on-push: Make the CI also run for the stable-*
 branches
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

As we only support one stable branch, it'll be used as part of the
stable-3.2 and onwards.

Fixes: #7518

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
---
 .github/workflows/ci-on-push.yaml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/ci-on-push.yaml b/.github/workflows/ci-on-push.yaml
index 99d4837209..e1f3eaf968 100644
--- a/.github/workflows/ci-on-push.yaml
+++ b/.github/workflows/ci-on-push.yaml
@@ -3,6 +3,7 @@ on:
   pull_request_target:
     branches:
       - 'main'
+      - 'stable-*'
     types:
       # Adding 'labeled' to the list of activity types that trigger this event
       # (default: opened, synchronize, reopened) so that we can run this

From 3230dec950e41c1a499a552b0a54739d079f9b97 Mon Sep 17 00:00:00 2001
From: Jeremi Piotrowski <jpiotrowski@microsoft.com>
Date: Wed, 2 Aug 2023 11:16:11 +0200
Subject: [PATCH 33/49] kata-deploy: Use host's systemctl

when interacting with systemd. We have occasionally faced issues with
compatibility between the systemctl version used inside the kata-deploy
container and the systemd version on the host. Instead of using a containerized
systemctl with bind mounted sockets, nsenter the host and run systemctl from
there. This provides less coupling between the kata-deploy container and the
host.

Fixes: #7511
Signed-off-by: Jeremi Piotrowski <jpiotrowski@microsoft.com>
---
 .../kata-cleanup/base/kata-cleanup.yaml       | 13 +-----------
 .../kata-deploy/base/kata-deploy.yaml         | 11 +---------
 .../kata-deploy/scripts/kata-deploy.sh        | 20 +++++++++++--------
 3 files changed, 14 insertions(+), 30 deletions(-)

diff --git a/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup.yaml b/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup.yaml
index 36d1b9b5b8..3d9006572e 100644
--- a/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup.yaml
+++ b/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup.yaml
@@ -14,6 +14,7 @@ spec:
           name: kubelet-kata-cleanup
     spec:
       serviceAccountName: kata-deploy-sa
+      hostPID: true
       nodeSelector:
           katacontainers.io/kata-runtime: cleanup
       containers:
@@ -38,18 +39,6 @@ spec:
           value: "false"
         securityContext:
           privileged: true
-        volumeMounts:
-        - name: dbus
-          mountPath: /var/run/dbus/system_bus_socket
-        - name: systemd
-          mountPath: /run/systemd/system
-      volumes:
-        - name: dbus
-          hostPath:
-            path: /var/run/dbus/system_bus_socket
-        - name: systemd
-          hostPath:
-            path: /run/systemd/system
   updateStrategy:
     rollingUpdate:
       maxUnavailable: 1
diff --git a/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml b/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
index 5431a47fae..c10061d902 100644
--- a/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
+++ b/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
@@ -14,6 +14,7 @@ spec:
           name: kata-deploy
     spec:
       serviceAccountName: kata-deploy-sa
+      hostPID: true
       containers:
       - name: kube-kata
         image: quay.io/kata-containers/kata-deploy:latest
@@ -47,10 +48,6 @@ spec:
           mountPath: /etc/containerd/
         - name: kata-artifacts
           mountPath: /opt/kata/
-        - name: dbus
-          mountPath: /var/run/dbus/system_bus_socket
-        - name: systemd
-          mountPath: /run/systemd/system
         - name: local-bin
           mountPath: /usr/local/bin/
       volumes:
@@ -64,12 +61,6 @@ spec:
           hostPath:
             path: /opt/kata/
             type: DirectoryOrCreate
-        - name: dbus
-          hostPath:
-            path: /var/run/dbus/system_bus_socket
-        - name: systemd
-          hostPath:
-            path: /run/systemd/system
         - name: local-bin
           hostPath:
             path: /usr/local/bin/
diff --git a/tools/packaging/kata-deploy/scripts/kata-deploy.sh b/tools/packaging/kata-deploy/scripts/kata-deploy.sh
index 5ed4df0675..fc26921df2 100755
--- a/tools/packaging/kata-deploy/scripts/kata-deploy.sh
+++ b/tools/packaging/kata-deploy/scripts/kata-deploy.sh
@@ -25,6 +25,10 @@ die() {
         exit 1
 }
 
+function host_systemctl() {
+	nsenter --target 1 --mount systemctl "${@}"
+}
+
 function print_usage() {
 	echo "Usage: $0 [install/cleanup/reset]"
 }
@@ -71,11 +75,11 @@ function get_container_runtime() {
                 die "invalid node name"
 	fi
 	if echo "$runtime" | grep -qE 'containerd.*-k3s'; then
-		if systemctl is-active --quiet rke2-agent; then
+		if host_systemctl is-active --quiet rke2-agent; then
 			echo "rke2-agent"
-		elif systemctl is-active --quiet rke2-server; then
+		elif host_systemctl is-active --quiet rke2-server; then
 			echo "rke2-server"
-		elif systemctl is-active --quiet k3s-agent; then
+		elif host_systemctl is-active --quiet k3s-agent; then
 			echo "k3s-agent"
 		else
 			echo "k3s"
@@ -136,8 +140,8 @@ function configure_cri_runtime() {
 		configure_containerd
 		;;
 	esac
-	systemctl daemon-reload
-	systemctl restart "$1"
+	host_systemctl daemon-reload
+	host_systemctl restart "$1"
 
 	wait_till_node_is_ready
 }
@@ -371,10 +375,10 @@ function cleanup_containerd() {
 
 function reset_runtime() {
 	kubectl label node "$NODE_NAME" katacontainers.io/kata-runtime-
-	systemctl daemon-reload
-	systemctl restart "$1"
+	host_systemctl daemon-reload
+	host_systemctl restart "$1"
 	if [ "$1" == "crio" ] || [ "$1" == "containerd" ]; then
-		systemctl restart kubelet
+		host_systemctl restart kubelet
 	fi
 
 	wait_till_node_is_ready

From 6ad16d49772d36b083e546612a3d4a6f19099a1a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= <fabiano.fidencio@intel.com>
Date: Wed, 2 Aug 2023 16:59:41 +0200
Subject: [PATCH 34/49] release: Adapt kata-deploy for 3.2.0-rc0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

kata-deploy files must be adapted to a new release.  The cases where it
happens are when the release goes from -> to:
* main -> stable:
  * kata-deploy-stable / kata-cleanup-stable: are removed

* stable -> stable:
  * kata-deploy / kata-cleanup: bump the release to the new one.

There are no changes when doing an alpha release, as the files on the
"main" branch always point to the "latest" and "stable" tags.

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
---
 .../base/kata-cleanup-stable.yaml             | 46 -------------
 .../kata-deploy/base/kata-deploy-stable.yaml  | 69 -------------------
 2 files changed, 115 deletions(-)
 delete mode 100644 tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
 delete mode 100644 tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml

diff --git a/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml b/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
deleted file mode 100644
index f1d9d0a2f9..0000000000
--- a/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
+++ /dev/null
@@ -1,46 +0,0 @@
----
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: kubelet-kata-cleanup
-  namespace: kube-system
-spec:
-  selector:
-      matchLabels:
-        name: kubelet-kata-cleanup
-  template:
-    metadata:
-        labels:
-          name: kubelet-kata-cleanup
-    spec:
-      serviceAccountName: kata-label-node
-      nodeSelector:
-          katacontainers.io/kata-runtime: cleanup
-      containers:
-      - name: kube-kata-cleanup
-        image: quay.io/kata-containers/kata-deploy:stable
-        imagePullPolicy: Always
-        command: [ "bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh reset" ]
-        env:
-        - name: NODE_NAME
-          valueFrom:
-            fieldRef:
-              fieldPath: spec.nodeName
-        securityContext:
-          privileged: false
-        volumeMounts:
-        - name: dbus
-          mountPath: /var/run/dbus
-        - name: systemd
-          mountPath: /run/systemd
-      volumes:
-        - name: dbus
-          hostPath:
-            path: /var/run/dbus
-        - name: systemd
-          hostPath:
-            path: /run/systemd
-  updateStrategy:
-    rollingUpdate:
-      maxUnavailable: 1
-    type: RollingUpdate
diff --git a/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml b/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml
deleted file mode 100644
index 346e4c0ee2..0000000000
--- a/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml
+++ /dev/null
@@ -1,69 +0,0 @@
----
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: kata-deploy
-  namespace: kube-system
-spec:
-  selector:
-      matchLabels:
-        name: kata-deploy
-  template:
-    metadata:
-        labels:
-          name: kata-deploy
-    spec:
-      serviceAccountName: kata-label-node
-      containers:
-      - name: kube-kata
-        image: quay.io/kata-containers/kata-deploy:stable
-        imagePullPolicy: Always
-        lifecycle:
-          preStop:
-            exec:
-              command: ["bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh cleanup"]
-        command: [ "bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh install" ]
-        env:
-        - name: NODE_NAME
-          valueFrom:
-            fieldRef:
-              fieldPath: spec.nodeName
-        securityContext:
-          privileged: false
-        volumeMounts:
-        - name: crio-conf
-          mountPath: /etc/crio/
-        - name: containerd-conf
-          mountPath: /etc/containerd/
-        - name: kata-artifacts
-          mountPath: /opt/kata/
-        - name: dbus
-          mountPath: /var/run/dbus
-        - name: systemd
-          mountPath: /run/systemd
-        - name: local-bin
-          mountPath: /usr/local/bin/
-      volumes:
-        - name: crio-conf
-          hostPath:
-            path: /etc/crio/
-        - name: containerd-conf
-          hostPath:
-            path: /etc/containerd/
-        - name: kata-artifacts
-          hostPath:
-            path: /opt/kata/
-            type: DirectoryOrCreate
-        - name: dbus
-          hostPath:
-            path: /var/run/dbus
-        - name: systemd
-          hostPath:
-            path: /run/systemd
-        - name: local-bin
-          hostPath:
-            path: /usr/local/bin/
-  updateStrategy:
-    rollingUpdate:
-      maxUnavailable: 1
-    type: RollingUpdate

From 006ecce49a70d57fa49fb823a44a9fcdd26d34f0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= <fabiano.fidencio@intel.com>
Date: Wed, 2 Aug 2023 16:59:41 +0200
Subject: [PATCH 35/49] release: Kata Containers 3.2.0-rc0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- ci-on-push: Make the CI also run for the stable-* branches
- ci: k8s: Do not fail when gathering info on AKS nodes
- kata-deploy: enable cross build for non-x86
- runtime-rs: add support for gather metrics in runtime-rs
- kata-ctl: add monitor subcommand for runtime-rs
- release: release-note.sh: Fix typos and reference to images
- metrics: Add sysbench performance test
- Simplify implementation of runtime-rs/service

6ad16d497 release: Adapt kata-deploy for 3.2.0-rc0
025596b28 ci-on-push: Make the CI also run for the stable-* branches
7ffc0c122 static-build: enable cross build for qemu
35d6d86ab static-build: enable cross-build for image build
2205fb9d0 static-build: enable cross build for virtiofsd
11631c681 static-build: enable cross build for shim-v2
7923de899 static-build: cross build kernel
e2c31fce2 kata-deploy: enable cross build for kata deploy script
2fc5f0e2e kata-depoly: prepare env for cross build in lib.sh
f5e9985af release: release-note.sh: Fix typos and reference to images
f910c66d6 ci: k8s: Do not fail when gathering info on AKS nodes
632818176 metrics: Add k8s sysbench documentation
b3901c46d runtime-rs: ignore errors during clean up sandbox resources
5a1b5d367 metrics: Add sysbench pod yaml
ad413d164 metrics: Add sysbench dockerfile
151256011 metrics: Add sysbench performance test
62e328ca5 runtime-rs: refine implementation of TaskService
458e1bc71 runtime-rs: make send_message() as an method of ServiceManager
1cc1c81c9 runtime-rs: fix possibe bug in ServiceManager::run()
1a5f90dc3 runtime-rs: simplify implementation of service crate
731e7c763 kata-ctl: add monitor subcommand for runtime-rs The previous kata-monitor in golang could not communicate with runtime-rs to gather metrics due to different sandbox addresses. This PR adds the subcommand monitor in kata-ctl to gather metrics from runtime-rs and monitor itself.
d74639d8c kata-ctl: provide the global TIMEOUT for creating MgmtClient
02cc4fe9d runtime-rs: add support for gather metrics in runtime-rs

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
---
 VERSION | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/VERSION b/VERSION
index ed590bd2a9..2f290ae1b6 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-3.2.0-alpha4
+3.2.0-rc0

From a81ad3b58723e71f60fac3e48cc8d045d8bcd4b6 Mon Sep 17 00:00:00 2001
From: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
Date: Fri, 16 Jun 2023 00:19:28 -0700
Subject: [PATCH 36/49] runtime-rs: Add block device handling in cloud
 hypervisor

Add functions for adding a block device to a container for CH.

Fixes #6690

Signed-off-by: Chelsea Mafrica <chelsea.e.mafrica@intel.com>
---
 .../crates/hypervisor/ch-config/src/ch_api.rs | 20 ++++++++-
 .../crates/hypervisor/src/ch/inner_device.rs  | 42 +++++++++++++++++--
 2 files changed, 58 insertions(+), 4 deletions(-)

diff --git a/src/runtime-rs/crates/hypervisor/ch-config/src/ch_api.rs b/src/runtime-rs/crates/hypervisor/ch-config/src/ch_api.rs
index 191cb54578..c1c8685c79 100644
--- a/src/runtime-rs/crates/hypervisor/ch-config/src/ch_api.rs
+++ b/src/runtime-rs/crates/hypervisor/ch-config/src/ch_api.rs
@@ -2,7 +2,7 @@
 //
 // SPDX-License-Identifier: Apache-2.0
 
-use crate::{DeviceConfig, FsConfig, VmConfig};
+use crate::{DeviceConfig, DiskConfig, FsConfig, VmConfig};
 use anyhow::{anyhow, Result};
 use api_client::simple_api_full_command_and_response;
 
@@ -69,6 +69,24 @@ pub async fn cloud_hypervisor_vm_stop(mut socket: UnixStream) -> Result<Option<S
     .await?
 }
 
+pub async fn cloud_hypervisor_vm_blockdev_add(
+    mut socket: UnixStream,
+    blk_config: DiskConfig,
+) -> Result<Option<String>> {
+    task::spawn_blocking(move || -> Result<Option<String>> {
+        let response = simple_api_full_command_and_response(
+            &mut socket,
+            "PUT",
+            "vm.add-disk",
+            Some(&serde_json::to_string(&blk_config)?),
+        )
+        .map_err(|e| anyhow!(e))?;
+
+        Ok(response)
+    })
+    .await?
+}
+
 #[allow(dead_code)]
 pub async fn cloud_hypervisor_vm_device_add(mut socket: UnixStream) -> Result<Option<String>> {
     let device_config = DeviceConfig::default();
diff --git a/src/runtime-rs/crates/hypervisor/src/ch/inner_device.rs b/src/runtime-rs/crates/hypervisor/src/ch/inner_device.rs
index e06832cbd1..d876b31093 100644
--- a/src/runtime-rs/crates/hypervisor/src/ch/inner_device.rs
+++ b/src/runtime-rs/crates/hypervisor/src/ch/inner_device.rs
@@ -6,17 +6,21 @@
 
 use super::inner::CloudHypervisorInner;
 use crate::device::DeviceType;
+use crate::BlockConfig;
 use crate::HybridVsockConfig;
 use crate::ShareFsDeviceConfig;
 use crate::VmmState;
 use anyhow::{anyhow, Context, Result};
-use ch_config::ch_api::cloud_hypervisor_vm_fs_add;
+use ch_config::ch_api::{cloud_hypervisor_vm_blockdev_add, cloud_hypervisor_vm_fs_add};
+use ch_config::DiskConfig;
 use ch_config::FsConfig;
 use safe_path::scoped_join;
 use std::convert::TryFrom;
 use std::path::PathBuf;
 
 const VIRTIO_FS: &str = "virtio-fs";
+const DEFAULT_DISK_QUEUES: usize = 1;
+const DEFAULT_DISK_QUEUE_SIZE: u16 = 1024;
 
 impl CloudHypervisorInner {
     pub(crate) async fn add_device(&mut self, device: DeviceType) -> Result<()> {
@@ -43,6 +47,7 @@ impl CloudHypervisorInner {
         match device {
             DeviceType::ShareFs(sharefs) => self.handle_share_fs_device(sharefs.config).await,
             DeviceType::HybridVsock(hvsock) => self.handle_hvsock_device(&hvsock.config).await,
+            DeviceType::Block(block) => self.handle_block_device(block.config).await,
             _ => Err(anyhow!("unhandled device: {:?}", device)),
         }
     }
@@ -125,6 +130,37 @@ impl CloudHypervisorInner {
         Ok(())
     }
 
+    async fn handle_block_device(&mut self, cfg: BlockConfig) -> Result<()> {
+        let socket = self
+            .api_socket
+            .as_ref()
+            .ok_or("missing socket")
+            .map_err(|e| anyhow!(e))?;
+
+        let num_queues: usize = DEFAULT_DISK_QUEUES;
+        let queue_size: u16 = DEFAULT_DISK_QUEUE_SIZE;
+
+        let block_config = DiskConfig {
+            path: Some(cfg.path_on_host.as_str().into()),
+            readonly: cfg.is_readonly,
+            num_queues,
+            queue_size,
+            ..Default::default()
+        };
+
+        let response = cloud_hypervisor_vm_blockdev_add(
+            socket.try_clone().context("failed to clone socket")?,
+            block_config,
+        )
+        .await?;
+
+        if let Some(detail) = response {
+            debug!(sl!(), "blockdev add response: {:?}", detail);
+        }
+
+        Ok(())
+    }
+
     pub(crate) async fn get_shared_fs_devices(&mut self) -> Result<Option<Vec<FsConfig>>> {
         let pending_root_devices = self.pending_devices.take();
 
@@ -173,13 +209,13 @@ impl TryFrom<ShareFsSettings> for FsConfig {
         let num_queues: usize = if cfg.queue_num > 0 {
             cfg.queue_num as usize
         } else {
-            1
+            DEFAULT_DISK_QUEUES
         };
 
         let queue_size: u16 = if cfg.queue_num > 0 {
             u16::try_from(cfg.queue_size)?
         } else {
-            1024
+            DEFAULT_DISK_QUEUE_SIZE
         };
 
         let socket_path = if cfg.sock_path.starts_with('/') {

From 64fdb98704e68ec91f4683624872f06037c388ed Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Wed, 2 Aug 2023 16:46:48 +0000
Subject: [PATCH 37/49] metrics: Add network latency test

This PR adds network latency test for kata metrics.

Fixes #7526

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 latency_kubernetes/latency-network.sh | 85 +++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)
 create mode 100755 latency_kubernetes/latency-network.sh

diff --git a/latency_kubernetes/latency-network.sh b/latency_kubernetes/latency-network.sh
new file mode 100755
index 0000000000..19c84ef14e
--- /dev/null
+++ b/latency_kubernetes/latency-network.sh
@@ -0,0 +1,85 @@
+#!/bin/bash
+#
+# Copyright (c) 2022 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+
+set -o pipefail
+
+SCRIPT_PATH=$(dirname "$(readlink -f "$0")")
+
+source "${SCRIPT_PATH}/../../lib/common.bash"
+latency_file=$(mktemp latencyresults.XXXXXXXXXX)
+TEST_NAME="${TEST_NAME:-latency}"
+
+function remove_tmp_file() {
+	rm -rf "${latency_file}"
+}
+
+trap remove_tmp_file EXIT
+
+function main() {
+	init_env
+	cmds=("bc" "jq")
+	check_cmds "${cmds[@]}"
+
+	# Check no processes are left behind
+	check_processes
+
+	wait_time=20
+	sleep_time=2
+
+	# Create server
+	kubectl create -f "${SCRIPT_PATH}/runtimeclass_workloads/latency-server.yaml"
+
+	# Get the names of the server pod
+	export server_pod_name="latency-server"
+
+	# Verify the server pod is working
+	local cmd="kubectl get pod $server_pod_name -o yaml | grep 'phase: Running'"
+	waitForProcess "$wait_time" "$sleep_time" "$cmd"
+
+	# Create client
+	kubectl create -f "${SCRIPT_PATH}/runtimeclass_workloads/latency-client.yaml"
+
+	# Get the names of the client pod
+	export client_pod_name="latency-client"
+
+	# Verify the client pod is working
+	local cmd="kubectl get pod $client_pod_name -o yaml | grep 'phase: Running'"
+	waitForProcess "$wait_time" "$sleep_time" "$cmd"
+
+	# Get the ip address of the server pod
+	export server_ip_add=$(kubectl get pod "$server_pod_name" -o jsonpath='{.status.podIP}')
+
+	# Number of packets (sent)
+	local number="${number:-30}"
+
+	local client_command="ping -c ${number} ${server_ip_add}"
+
+	kubectl exec "$client_pod_name" -- sh -c "$client_command" > "$latency_file"
+
+	metrics_json_init
+
+	local latency=$(cat $latency_file | grep avg | cut -f2 -d'=' | sed 's/[[:blank:]]//g' | cut -f2 -d'/')
+
+	metrics_json_start_array
+
+	local json="$(cat << EOF
+	{
+		"latency": {
+			"Result" : $latency,
+			"Units" : "ms"
+		}
+	}
+EOF
+)"
+
+	metrics_json_add_array_element "$json"
+	metrics_json_end_array "Results"
+	metrics_json_save
+
+	kubectl delete pod "$client_pod_name" "$server_pod_name"
+	check_processes
+}
+main "$@"

From 9bb8451df5e278ac31db7ed00156cb77ed9186c8 Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Wed, 2 Aug 2023 16:50:51 +0000
Subject: [PATCH 38/49] metrics: Add latency client yaml

This PR adds latency client yaml for the kubernetes test.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../runtimeclass_workloads/latency-client.yaml  | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 latency_kubernetes/runtimeclass_workloads/latency-client.yaml

diff --git a/latency_kubernetes/runtimeclass_workloads/latency-client.yaml b/latency_kubernetes/runtimeclass_workloads/latency-client.yaml
new file mode 100644
index 0000000000..8e1f3bd0ea
--- /dev/null
+++ b/latency_kubernetes/runtimeclass_workloads/latency-client.yaml
@@ -0,0 +1,17 @@
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+apiVersion: v1
+kind: Pod
+metadata:
+  name: latency-client
+spec:
+  terminationGracePeriodSeconds: 0
+  runtimeClassName: kata
+  containers:
+    - name: client-container
+      image: quay.io/prometheus/busybox:latest
+      command:
+        - sleep
+        - "180"

From 3b45060b6169934943212d399299713b5511f28c Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Wed, 2 Aug 2023 16:52:17 +0000
Subject: [PATCH 39/49] metrics: Add latency server yaml

This PR adds latency server yaml for kubernetes test.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 .../runtimeclass_workloads/latency-server.yaml | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
 create mode 100644 latency_kubernetes/runtimeclass_workloads/latency-server.yaml

diff --git a/latency_kubernetes/runtimeclass_workloads/latency-server.yaml b/latency_kubernetes/runtimeclass_workloads/latency-server.yaml
new file mode 100644
index 0000000000..ce301f6535
--- /dev/null
+++ b/latency_kubernetes/runtimeclass_workloads/latency-server.yaml
@@ -0,0 +1,18 @@
+#
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# SPDX-License-Identifier: Apache-2.0
+#
+apiVersion: v1
+kind: Pod
+metadata:
+  name: latency-server
+spec:
+  terminationGracePeriodSeconds: 0
+  runtimeClassName: kata
+  containers:
+    - name: server-container
+      image: quay.io/prometheus/busybox:latest
+      command:
+        - sleep
+        - "180"

From 5dbe88330f0dfb40ac6e88225c128688a1f9dedc Mon Sep 17 00:00:00 2001
From: David Esparza <david.esparza.borquez@intel.com>
Date: Wed, 2 Aug 2023 12:15:50 -0600
Subject: [PATCH 40/49] metrics: Clean kata components before start a metric
 test.

This PR kills all kata components before start a new
metric test.

Fixes: #7528

Signed-off-by: David Esparza <david.esparza.borquez@intel.com>
---
 tests/common.bash             | 29 +++++++++--------------------
 tests/metrics/lib/common.bash |  2 ++
 2 files changed, 11 insertions(+), 20 deletions(-)

diff --git a/tests/common.bash b/tests/common.bash
index d71b3f43ce..224df23f52 100644
--- a/tests/common.bash
+++ b/tests/common.bash
@@ -186,31 +186,20 @@ function clean_env_ctr()
 	if (( count_tasks > 0 )); then
 		die "Can't remove running containers."
 	fi
-
-	kill_kata_components
 }
 
 # Kills running shim and hypervisor components
+# by using the kata-component file name.
 function kill_kata_components() {
-	local kata_bin_dir="/opt/kata/bin"
-	local shim_path="${kata_bin_dir}/containerd-shim-kata-v2"
-	local hypervisor_path="${kata_bin_dir}/qemu-system-x86_64"
-	local pid_shim_count="$(pgrep -fc ${shim_path} || exit 0)"
+	local PID_NAMES=( "containerd-shim-kata-v2" "qemu-system-x86_64" "cloud-hypervisor" )
 
-	[ ${pid_shim_count} -gt "0" ] && sudo kill -SIGKILL "$(pgrep -f ${shim_path})" > /dev/null 2>&1
-
-        if [ "${KATA_HYPERVISOR}" = 'clh' ]; then
-		hypervisor_path="${kata_bin_dir}/cloud-hypervisor"
-	elif [ "${KATA_HYPERVISOR}" != 'qemu' ]; then
-                echo "Failed to stop the hypervisor: '${KATA_HYPERVISOR}' as it is not recognized"
-		return
-        fi
-
-	local pid_hypervisor_count="$(pgrep -fc ${hypervisor_path} || exit 0)"
-
-	if [ ${pid_hypervisor_count} -gt "0" ]; then
-		sudo kill -SIGKILL "$(pgrep -f ${hypervisor_path})" > /dev/null 2>&1
-	fi
+	sudo systemctl stop containerd
+	# Get the filenames of the kata components
+	# and kill the correspondingt processes
+	for PID_NAME in ${PID_NAMES} ; do
+		sudo killall ${PID_NAME} > /dev/null 2>&1 || true
+	done
+	sudo systemctl start containerd
 }
 
 # Restarts a systemd service while ensuring the start-limit-burst is set to 0.
diff --git a/tests/metrics/lib/common.bash b/tests/metrics/lib/common.bash
index c43019a709..4111d67d40 100755
--- a/tests/metrics/lib/common.bash
+++ b/tests/metrics/lib/common.bash
@@ -192,6 +192,8 @@ function kill_processes_before_start()
 	CTR_PROCS=$(sudo "${CTR_EXE}" t list -q)
 	[[ -n "${CTR_PROCS}" ]] && clean_env_ctr
 
+	kill_kata_components
+
 	check_processes
 }
 

From 1e15369e59faf92b1efcf4b68d099fb4418b73de Mon Sep 17 00:00:00 2001
From: David Esparza <david.esparza.borquez@intel.com>
Date: Wed, 2 Aug 2023 12:23:09 -0600
Subject: [PATCH 41/49] metrics: Improve naming testing containers in launch
 times test

This commit provides a new way to name the containers used
in the launch-times-test in this form:
'kata_launch_times_RANDOM_NUMBER', where RANDOM_NUMBER is
in the 0-1000 range.

Fixes: #7529

Signed-off-by: David Esparza <david.esparza.borquez@intel.com>
---
 tests/metrics/time/launch_times.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/metrics/time/launch_times.sh b/tests/metrics/time/launch_times.sh
index 0458b979ed..3d6b98e348 100755
--- a/tests/metrics/time/launch_times.sh
+++ b/tests/metrics/time/launch_times.sh
@@ -96,6 +96,7 @@ run_workload() {
 	# number of decimal digits after the decimal points
 	# for 'bc' performing math in kernel period estimation
 	L_CALC_SCALE=13
+	local CONTAINER_NAME="kata_launch_times_$(( $RANDOM % 1000 + 1))"
 	start_time=$($DATECMD)
 
 	# Check entropy level of the host
@@ -103,8 +104,7 @@ run_workload() {
 
 	# Run the image and command and capture the results into an array...
 	declare workload_result
-	readarray -n 0 workload_result < <(sudo -E "${CTR_EXE}" run --rm --runtime=${CTR_RUNTIME} ${IMAGE} test bash -c "$DATECMD $DMESGCMD")
-
+	readarray -n 0 workload_result < <(sudo -E "${CTR_EXE}" run --rm --runtime ${CTR_RUNTIME} ${IMAGE} ${CONTAINER_NAME} bash -c "$DATECMD $DMESGCMD")
 	end_time=$($DATECMD)
 
 	# Delay this calculation until after we have run - do not want

From 1163fc9de2c7a6c3ec65afe81cf411f780467c62 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fabiano=20Fid=C3=AAncio?= <fabiano.fidencio@intel.com>
Date: Wed, 2 Aug 2023 16:59:45 +0200
Subject: [PATCH 42/49] release: Revert kata-deploy changes after 3.2.0-rc0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

As 3.2.0-rc0 has been released, let's switch the kata-deploy / kata-cleanup
tags back to "latest", and re-add the kata-deploy-stable and the
kata-cleanup-stable files.

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
---
 .../base/kata-cleanup-stable.yaml             | 46 +++++++++++++
 .../kata-deploy/base/kata-deploy-stable.yaml  | 69 +++++++++++++++++++
 2 files changed, 115 insertions(+)
 create mode 100644 tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
 create mode 100644 tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml

diff --git a/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml b/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
new file mode 100644
index 0000000000..f1d9d0a2f9
--- /dev/null
+++ b/tools/packaging/kata-deploy/kata-cleanup/base/kata-cleanup-stable.yaml
@@ -0,0 +1,46 @@
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kubelet-kata-cleanup
+  namespace: kube-system
+spec:
+  selector:
+      matchLabels:
+        name: kubelet-kata-cleanup
+  template:
+    metadata:
+        labels:
+          name: kubelet-kata-cleanup
+    spec:
+      serviceAccountName: kata-label-node
+      nodeSelector:
+          katacontainers.io/kata-runtime: cleanup
+      containers:
+      - name: kube-kata-cleanup
+        image: quay.io/kata-containers/kata-deploy:stable
+        imagePullPolicy: Always
+        command: [ "bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh reset" ]
+        env:
+        - name: NODE_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        securityContext:
+          privileged: false
+        volumeMounts:
+        - name: dbus
+          mountPath: /var/run/dbus
+        - name: systemd
+          mountPath: /run/systemd
+      volumes:
+        - name: dbus
+          hostPath:
+            path: /var/run/dbus
+        - name: systemd
+          hostPath:
+            path: /run/systemd
+  updateStrategy:
+    rollingUpdate:
+      maxUnavailable: 1
+    type: RollingUpdate
diff --git a/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml b/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml
new file mode 100644
index 0000000000..346e4c0ee2
--- /dev/null
+++ b/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml
@@ -0,0 +1,69 @@
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: kata-deploy
+  namespace: kube-system
+spec:
+  selector:
+      matchLabels:
+        name: kata-deploy
+  template:
+    metadata:
+        labels:
+          name: kata-deploy
+    spec:
+      serviceAccountName: kata-label-node
+      containers:
+      - name: kube-kata
+        image: quay.io/kata-containers/kata-deploy:stable
+        imagePullPolicy: Always
+        lifecycle:
+          preStop:
+            exec:
+              command: ["bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh cleanup"]
+        command: [ "bash", "-c", "/opt/kata-artifacts/scripts/kata-deploy.sh install" ]
+        env:
+        - name: NODE_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: spec.nodeName
+        securityContext:
+          privileged: false
+        volumeMounts:
+        - name: crio-conf
+          mountPath: /etc/crio/
+        - name: containerd-conf
+          mountPath: /etc/containerd/
+        - name: kata-artifacts
+          mountPath: /opt/kata/
+        - name: dbus
+          mountPath: /var/run/dbus
+        - name: systemd
+          mountPath: /run/systemd
+        - name: local-bin
+          mountPath: /usr/local/bin/
+      volumes:
+        - name: crio-conf
+          hostPath:
+            path: /etc/crio/
+        - name: containerd-conf
+          hostPath:
+            path: /etc/containerd/
+        - name: kata-artifacts
+          hostPath:
+            path: /opt/kata/
+            type: DirectoryOrCreate
+        - name: dbus
+          hostPath:
+            path: /var/run/dbus
+        - name: systemd
+          hostPath:
+            path: /run/systemd
+        - name: local-bin
+          hostPath:
+            path: /usr/local/bin/
+  updateStrategy:
+    rollingUpdate:
+      maxUnavailable: 1
+    type: RollingUpdate

From b2ff97aa01db3151b437da99e1da3a24f320d231 Mon Sep 17 00:00:00 2001
From: Wedson Almeida Filho <walmeida@microsoft.com>
Date: Thu, 3 Aug 2023 13:50:50 -0300
Subject: [PATCH 43/49] dragonball: use version 0.10.4 of `fuse-backend-rs`

Version 0.10.5, which was just released, breaks `nydus-storage`.

This is a workaround to fix the CI which is blocking other PRs.

Fixes: #7541

Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
---
 src/dragonball/Cargo.toml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/dragonball/Cargo.toml b/src/dragonball/Cargo.toml
index f46ab6a6c6..68f08060d7 100644
--- a/src/dragonball/Cargo.toml
+++ b/src/dragonball/Cargo.toml
@@ -43,6 +43,7 @@ vmm-sys-util = "0.11.0"
 virtio-queue = { version = "0.6.0", optional = true }
 vm-memory = { version = "0.9.0", features = ["backend-mmap"] }
 crossbeam-channel = "0.5.6"
+fuse-backend-rs = "=0.10.4"
 
 [dev-dependencies]
 slog-async = "2.7.0"

From 6163c3565712aa9ab59ec43a2ddaecc9e4070e82 Mon Sep 17 00:00:00 2001
From: Wedson Almeida Filho <walmeida@microsoft.com>
Date: Thu, 20 Apr 2023 09:42:43 -0300
Subject: [PATCH 44/49] agent: skip mount options that start with
 "io.katacontainers."

This is so that file systems don't fail when we pass kata-specific
options from the snapshotter to kata.

Fixes: #7536

Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
---
 src/agent/src/mount.rs | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/src/agent/src/mount.rs b/src/agent/src/mount.rs
index d80aea4c6d..a6bf6b4ff2 100644
--- a/src/agent/src/mount.rs
+++ b/src/agent/src/mount.rs
@@ -813,10 +813,14 @@ fn parse_mount_flags_and_options(options_vec: Vec<&str>) -> (MsFlags, String) {
                     }
                 }
                 None => {
+                    if opt.starts_with("io.katacontainers.") {
+                        continue;
+                    }
+
                     if !options.is_empty() {
                         options.push_str(format!(",{}", opt).as_str());
                     } else {
-                        options.push_str(opt.to_string().as_str());
+                        options.push_str(opt);
                     }
                 }
             };

From 6c867d9e8640d1f72fa607487e5c918e983a6605 Mon Sep 17 00:00:00 2001
From: Wedson Almeida Filho <walmeida@microsoft.com>
Date: Tue, 13 Jun 2023 01:14:44 -0300
Subject: [PATCH 45/49] agent: add io.katacontainers.fs-opt.overlay-rw option

This causes the overlay-fs driver to add the `upperdir` and `workdir`
options to an overlay-fs mount so that the mount becomes writable using
a discardable directory under the container id.

Fixes: #7536

Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
---
 src/agent/src/mount.rs | 27 ++++++++++++++++++++++++++-
 src/agent/src/rpc.rs   |  2 +-
 2 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/src/agent/src/mount.rs b/src/agent/src/mount.rs
index a6bf6b4ff2..83f624f98c 100644
--- a/src/agent/src/mount.rs
+++ b/src/agent/src/mount.rs
@@ -340,8 +340,33 @@ pub async fn update_ephemeral_mounts(
 async fn overlayfs_storage_handler(
     logger: &Logger,
     storage: &Storage,
+    cid: Option<&str>,
     _sandbox: Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
+    if storage
+        .options
+        .iter()
+        .any(|e| e == "io.katacontainers.fs-opt.overlay-rw")
+    {
+        let cid = cid.ok_or_else(|| anyhow!("No container id in rw overlay"))?;
+        let cpath = Path::new(crate::rpc::CONTAINER_BASE).join(cid);
+        let work = cpath.join("work");
+        let upper = cpath.join("upper");
+
+        fs::create_dir_all(&work).context("Creating overlay work directory")?;
+        fs::create_dir_all(&upper).context("Creating overlay upper directory")?;
+
+        let mut storage = storage.clone();
+        storage.fstype = "overlay".into();
+        storage
+            .options
+            .push(format!("upperdir={}", upper.to_string_lossy()));
+        storage
+            .options
+            .push(format!("workdir={}", work.to_string_lossy()));
+        return common_storage_handler(logger, &storage);
+    }
+
     common_storage_handler(logger, storage)
 }
 
@@ -869,7 +894,7 @@ pub async fn add_storages(
                 ephemeral_storage_handler(&logger, &storage, sandbox.clone()).await
             }
             DRIVER_OVERLAYFS_TYPE => {
-                overlayfs_storage_handler(&logger, &storage, sandbox.clone()).await
+                overlayfs_storage_handler(&logger, &storage, cid.as_deref(), sandbox.clone()).await
             }
             DRIVER_MMIO_BLK_TYPE => {
                 virtiommio_blk_storage_handler(&logger, &storage, sandbox.clone()).await
diff --git a/src/agent/src/rpc.rs b/src/agent/src/rpc.rs
index 6c92eb774e..3b7f0a4720 100644
--- a/src/agent/src/rpc.rs
+++ b/src/agent/src/rpc.rs
@@ -84,7 +84,7 @@ use std::io::{BufRead, BufReader, Write};
 use std::os::unix::fs::FileExt;
 use std::path::PathBuf;
 
-const CONTAINER_BASE: &str = "/run/kata-containers";
+pub const CONTAINER_BASE: &str = "/run/kata-containers";
 const MODPROBE_PATH: &str = "/sbin/modprobe";
 
 /// the iptables seriers binaries could appear either in /sbin

From 7e1b1949d43314f6de3321309cc64cbd7b7c6b1d Mon Sep 17 00:00:00 2001
From: Wedson Almeida Filho <walmeida@microsoft.com>
Date: Thu, 20 Apr 2023 04:52:27 -0300
Subject: [PATCH 46/49] runtime: add support for kata overlays

When at least one `io.katacontainers.fs-opt.layer` option is added to
the rootfs, it gets inserted into the VM as a layer, and the file system
is mounted as an overlay of all layers using the overlayfs driver.

Additionally, if the `io.katacontainers.fs-opt.block_device=file` option
is present in a layer, it is mounted as a block device backed by a file
on the host.

Fixes: #7536

Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
---
 src/runtime/pkg/containerd-shim-v2/create.go  | 35 +++++++++++++++++++
 src/runtime/pkg/device/config/config.go       |  4 +++
 src/runtime/pkg/device/manager/manager.go     | 17 +++++++--
 src/runtime/pkg/govmm/qemu/qmp.go             |  7 +++-
 src/runtime/virtcontainers/container.go       | 16 +++++++--
 src/runtime/virtcontainers/fs_share_linux.go  | 15 ++++++++
 src/runtime/virtcontainers/kata_agent.go      | 20 ++++++++---
 src/runtime/virtcontainers/mount.go           | 18 ++++++++++
 .../pkg/annotations/annotations.go            | 15 ++++++++
 9 files changed, 136 insertions(+), 11 deletions(-)

diff --git a/src/runtime/pkg/containerd-shim-v2/create.go b/src/runtime/pkg/containerd-shim-v2/create.go
index 474925b122..ee9bbb36b6 100644
--- a/src/runtime/pkg/containerd-shim-v2/create.go
+++ b/src/runtime/pkg/containerd-shim-v2/create.go
@@ -16,6 +16,7 @@ import (
 	"path"
 	"path/filepath"
 	"strconv"
+	"strings"
 	"syscall"
 
 	containerd_types "github.com/containerd/containerd/api/types"
@@ -23,6 +24,8 @@ import (
 	taskAPI "github.com/containerd/containerd/runtime/v2/task"
 	"github.com/containerd/typeurl"
 	"github.com/kata-containers/kata-containers/src/runtime/pkg/utils"
+	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers"
+	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/annotations"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/rootless"
 	"github.com/opencontainers/runtime-spec/specs-go"
 	"github.com/pkg/errors"
@@ -48,6 +51,28 @@ var defaultStartManagementServerFunc startManagementServerFunc = func(s *service
 	shimLog.Info("management server started")
 }
 
+func copyLayersToMounts(rootFs *vc.RootFs, spec *specs.Spec) error {
+	for _, o := range rootFs.Options {
+		if !strings.HasPrefix(o, annotations.FileSystemLayer) {
+			continue
+		}
+
+		fields := strings.Split(o[len(annotations.FileSystemLayer):], ",")
+		if len(fields) < 2 {
+			return fmt.Errorf("Missing fields in rootfs layer: %q", o)
+		}
+
+		spec.Mounts = append(spec.Mounts, specs.Mount{
+			Destination: "/run/kata-containers/sandbox/layers/" + filepath.Base(fields[0]),
+			Type:        fields[1],
+			Source:      fields[0],
+			Options:     fields[2:],
+		})
+	}
+
+	return nil
+}
+
 func create(ctx context.Context, s *service, r *taskAPI.CreateTaskRequest) (*container, error) {
 	rootFs := vc.RootFs{}
 	if len(r.Rootfs) == 1 {
@@ -63,6 +88,11 @@ func create(ctx context.Context, s *service, r *taskAPI.CreateTaskRequest) (*con
 	if err != nil {
 		return nil, err
 	}
+
+	if err := copyLayersToMounts(&rootFs, ociSpec); err != nil {
+		return nil, err
+	}
+
 	containerType, err := oci.ContainerType(*ociSpec)
 	if err != nil {
 		return nil, err
@@ -268,6 +298,11 @@ func checkAndMount(s *service, r *taskAPI.CreateTaskRequest) (bool, error) {
 		if katautils.IsBlockDevice(m.Source) && !s.config.HypervisorConfig.DisableBlockDeviceUse {
 			return false, nil
 		}
+
+		if virtcontainers.HasOptionPrefix(m.Options, annotations.FileSystemLayer) {
+			return false, nil
+		}
+
 		if m.Type == vc.NydusRootFSType {
 			// if kata + nydus, do not mount
 			return false, nil
diff --git a/src/runtime/pkg/device/config/config.go b/src/runtime/pkg/device/config/config.go
index ef2a5c4b03..773eaaa2d5 100644
--- a/src/runtime/pkg/device/config/config.go
+++ b/src/runtime/pkg/device/config/config.go
@@ -480,6 +480,10 @@ func GetHostPath(devInfo DeviceInfo, vhostUserStoreEnabled bool, vhostUserStoreP
 		return "", fmt.Errorf("Empty path provided for device")
 	}
 
+	if devInfo.Major == -1 {
+		return devInfo.HostPath, nil
+	}
+
 	// Filter out vhost-user storage devices by device Major numbers.
 	if vhostUserStoreEnabled && devInfo.DevType == "b" &&
 		(devInfo.Major == VhostUserSCSIMajor || devInfo.Major == VhostUserBlkMajor) {
diff --git a/src/runtime/pkg/device/manager/manager.go b/src/runtime/pkg/device/manager/manager.go
index cb5e86a045..ed3708dc9a 100644
--- a/src/runtime/pkg/device/manager/manager.go
+++ b/src/runtime/pkg/device/manager/manager.go
@@ -83,10 +83,21 @@ func NewDeviceManager(blockDriver string, vhostUserStoreEnabled bool, vhostUserS
 	return dm
 }
 
-func (dm *deviceManager) findDeviceByMajorMinor(major, minor int64) api.Device {
+func (dm *deviceManager) findDevice(devInfo *config.DeviceInfo) api.Device {
+	// For devices with a major of -1, we use the host path to find existing instances.
+	if devInfo.Major == -1 {
+		for _, dev := range dm.devices {
+			dma, _ := dev.GetMajorMinor()
+			if dma == -1 && dev.GetHostPath() == devInfo.HostPath {
+				return dev
+			}
+		}
+		return nil
+	}
+
 	for _, dev := range dm.devices {
 		dma, dmi := dev.GetMajorMinor()
-		if dma == major && dmi == minor {
+		if dma == devInfo.Major && dmi == devInfo.Minor {
 			return dev
 		}
 	}
@@ -111,7 +122,7 @@ func (dm *deviceManager) createDevice(devInfo config.DeviceInfo) (dev api.Device
 		}
 	}()
 
-	if existingDev := dm.findDeviceByMajorMinor(devInfo.Major, devInfo.Minor); existingDev != nil {
+	if existingDev := dm.findDevice(&devInfo); existingDev != nil {
 		return existingDev, nil
 	}
 
diff --git a/src/runtime/pkg/govmm/qemu/qmp.go b/src/runtime/pkg/govmm/qemu/qmp.go
index 46eb1c0827..8241d91729 100644
--- a/src/runtime/pkg/govmm/qemu/qmp.go
+++ b/src/runtime/pkg/govmm/qemu/qmp.go
@@ -802,7 +802,12 @@ func (q *QMP) blockdevAddBaseArgs(driver string, blockDevice *BlockDevice) map[s
 // used to name the device.  As this identifier will be passed directly to QMP,
 // it must obey QMP's naming rules, e,g., it must start with a letter.
 func (q *QMP) ExecuteBlockdevAdd(ctx context.Context, blockDevice *BlockDevice) error {
-	args := q.blockdevAddBaseArgs("host_device", blockDevice)
+	var args map[string]interface{}
+	if fi, err := os.Stat(blockDevice.File); err == nil && fi.Mode().IsRegular() {
+		args = q.blockdevAddBaseArgs("file", blockDevice)
+	} else {
+		args = q.blockdevAddBaseArgs("host_device", blockDevice)
+	}
 
 	return q.executeCommand(ctx, "blockdev-add", args, nil)
 }
diff --git a/src/runtime/virtcontainers/container.go b/src/runtime/virtcontainers/container.go
index 0f68810c40..1f8646a63c 100644
--- a/src/runtime/virtcontainers/container.go
+++ b/src/runtime/virtcontainers/container.go
@@ -608,8 +608,9 @@ func (c *Container) createBlockDevices(ctx context.Context) error {
 			continue
 		}
 
-		if c.mounts[i].Type != "bind" {
-			// We only handle for bind-mounts
+		isBlockFile := HasOption(c.mounts[i].Options, vcAnnotations.IsFileBlockDevice)
+		if c.mounts[i].Type != "bind" && !isBlockFile {
+			// We only handle for bind and block device mounts.
 			continue
 		}
 
@@ -671,7 +672,7 @@ func (c *Container) createBlockDevices(ctx context.Context) error {
 
 		// Check if mount is a block device file. If it is, the block device will be attached to the host
 		// instead of passing this as a shared mount.
-		if stat.Mode&unix.S_IFBLK == unix.S_IFBLK {
+		if stat.Mode&unix.S_IFMT == unix.S_IFBLK {
 			di = &config.DeviceInfo{
 				HostPath:      c.mounts[i].Source,
 				ContainerPath: c.mounts[i].Destination,
@@ -680,6 +681,15 @@ func (c *Container) createBlockDevices(ctx context.Context) error {
 				Minor:         int64(unix.Minor(uint64(stat.Rdev))),
 				ReadOnly:      c.mounts[i].ReadOnly,
 			}
+		} else if isBlockFile && stat.Mode&unix.S_IFMT == unix.S_IFREG {
+			di = &config.DeviceInfo{
+				HostPath:      c.mounts[i].Source,
+				ContainerPath: c.mounts[i].Destination,
+				DevType:       "b",
+				Major:         -1,
+				Minor:         0,
+				ReadOnly:      c.mounts[i].ReadOnly,
+			}
 			// Check whether source can be used as a pmem device
 		} else if di, err = config.PmemDeviceInfo(c.mounts[i].Source, c.mounts[i].Destination); err != nil {
 			c.Logger().WithError(err).
diff --git a/src/runtime/virtcontainers/fs_share_linux.go b/src/runtime/virtcontainers/fs_share_linux.go
index 085f3b4f8b..d2f9039726 100644
--- a/src/runtime/virtcontainers/fs_share_linux.go
+++ b/src/runtime/virtcontainers/fs_share_linux.go
@@ -22,6 +22,7 @@ import (
 	"github.com/kata-containers/kata-containers/src/runtime/pkg/device/config"
 	"github.com/kata-containers/kata-containers/src/runtime/pkg/katautils/katatrace"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/agent/protocols/grpc"
+	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/annotations"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/utils"
 )
 
@@ -376,6 +377,20 @@ func (f *FilesystemShare) ShareRootFilesystem(ctx context.Context, c *Container)
 	}
 	rootfsGuestPath := filepath.Join(kataGuestSharedDir(), c.id, c.rootfsSuffix)
 
+	if HasOptionPrefix(c.rootFs.Options, annotations.FileSystemLayer) {
+		path := filepath.Join("/run/kata-containers", c.id, "rootfs")
+		return &SharedFile{
+			storage: &grpc.Storage{
+				MountPoint: path,
+				Source:     "none",
+				Fstype:     c.rootFs.Type,
+				Driver:     kataOverlayDevType,
+				Options:    c.rootFs.Options,
+			},
+			guestPath: path,
+		}, nil
+	}
+
 	if c.state.Fstype != "" && c.state.BlockDeviceID != "" {
 		// The rootfs storage volume represents the container rootfs
 		// mount point inside the guest.
diff --git a/src/runtime/virtcontainers/kata_agent.go b/src/runtime/virtcontainers/kata_agent.go
index a3e5f5d421..e102484c4c 100644
--- a/src/runtime/virtcontainers/kata_agent.go
+++ b/src/runtime/virtcontainers/kata_agent.go
@@ -907,6 +907,8 @@ func (k *kataAgent) removeIgnoredOCIMount(spec *specs.Spec, ignoredMounts map[st
 	for _, m := range spec.Mounts {
 		if _, found := ignoredMounts[m.Source]; found {
 			k.Logger().WithField("removed-mount", m.Source).Debug("Removing OCI mount")
+		} else if HasOption(m.Options, vcAnnotations.IsFileSystemLayer) {
+			k.Logger().WithField("removed-mount", m.Source).Debug("Removing layer")
 		} else {
 			mounts = append(mounts, m)
 		}
@@ -1293,13 +1295,17 @@ func (k *kataAgent) createContainer(ctx context.Context, sandbox *Sandbox, c *Co
 
 	// Block based volumes will require some adjustments in the OCI spec, and creation of
 	// storage objects to pass to the agent.
-	volumeStorages, err := k.handleBlkOCIMounts(c, ociSpec)
+	layerStorages, volumeStorages, err := k.handleBlkOCIMounts(c, ociSpec)
 	if err != nil {
 		return nil, err
 	}
 
 	ctrStorages = append(ctrStorages, volumeStorages...)
 
+	// Layer storage objects are prepended to the list so that they come _before_ the
+	// rootfs because the rootfs depends on them (it's an overlay of the layers).
+	ctrStorages = append(layerStorages, ctrStorages...)
+
 	grpcSpec, err := grpc.OCItoGRPC(ociSpec)
 	if err != nil {
 		return nil, err
@@ -1611,9 +1617,10 @@ func (k *kataAgent) createBlkStorageObject(c *Container, m Mount) (*grpc.Storage
 // handleBlkOCIMounts will create a unique destination mountpoint in the guest for each volume in the
 // given container and will update the OCI spec to utilize this mount point as the new source for the
 // container volume. The container mount structure is updated to store the guest destination mountpoint.
-func (k *kataAgent) handleBlkOCIMounts(c *Container, spec *specs.Spec) ([]*grpc.Storage, error) {
+func (k *kataAgent) handleBlkOCIMounts(c *Container, spec *specs.Spec) ([]*grpc.Storage, []*grpc.Storage, error) {
 
 	var volumeStorages []*grpc.Storage
+	var layerStorages []*grpc.Storage
 
 	for i, m := range c.mounts {
 		id := m.BlockDeviceID
@@ -1629,7 +1636,12 @@ func (k *kataAgent) handleBlkOCIMounts(c *Container, spec *specs.Spec) ([]*grpc.
 		// Create Storage structure
 		vol, err := k.createBlkStorageObject(c, m)
 		if vol == nil || err != nil {
-			return nil, err
+			return nil, nil, err
+		}
+
+		if HasOption(m.Options, vcAnnotations.IsFileSystemLayer) {
+			layerStorages = append(layerStorages, vol)
+			continue
 		}
 
 		// Each device will be mounted at a unique location within the VM only once. Mounting
@@ -1660,7 +1672,7 @@ func (k *kataAgent) handleBlkOCIMounts(c *Container, spec *specs.Spec) ([]*grpc.
 		volumeStorages = append(volumeStorages, vol)
 	}
 
-	return volumeStorages, nil
+	return layerStorages, volumeStorages, nil
 }
 
 // handlePidNamespace checks if Pid namespace for a container needs to be shared with its sandbox
diff --git a/src/runtime/virtcontainers/mount.go b/src/runtime/virtcontainers/mount.go
index 6c2e204208..f261618905 100644
--- a/src/runtime/virtcontainers/mount.go
+++ b/src/runtime/virtcontainers/mount.go
@@ -415,3 +415,21 @@ func isWatchableMount(path string) bool {
 
 	return false
 }
+
+func HasOption(options []string, option string) bool {
+	for _, o := range options {
+		if o == option {
+			return true
+		}
+	}
+	return false
+}
+
+func HasOptionPrefix(options []string, prefix string) bool {
+	for _, o := range options {
+		if strings.HasPrefix(o, prefix) {
+			return true
+		}
+	}
+	return false
+}
diff --git a/src/runtime/virtcontainers/pkg/annotations/annotations.go b/src/runtime/virtcontainers/pkg/annotations/annotations.go
index eb8656cc28..d51447d73e 100644
--- a/src/runtime/virtcontainers/pkg/annotations/annotations.go
+++ b/src/runtime/virtcontainers/pkg/annotations/annotations.go
@@ -310,6 +310,21 @@ const (
 	ContainerResourcesSwapInBytes = kataAnnotContainerResourcePrefix + "swap_in_bytes"
 )
 
+// Annotations related to file system options.
+const (
+	kataAnnotFsOptPrefix = kataAnnotationsPrefix + "fs-opt."
+
+	// FileSystemLayer describes a layer of an overlay filesystem.
+	FileSystemLayer = kataAnnotFsOptPrefix + "layer="
+
+	// IsFileSystemLayer indicates that the annotated filesystem is a layer of an overlay fs.
+	IsFileSystemLayer = kataAnnotFsOptPrefix + "is-layer"
+
+	// IsFileBlockDevice indicates that the annotated filesystem is mounted on a block device
+	// backed by a host file.
+	IsFileBlockDevice = kataAnnotFsOptPrefix + "block_device=file"
+)
+
 const (
 	// SHA512 is the SHA-512 (64) hash algorithm
 	SHA512 string = "sha512"

From 4fbe0a3a5336c28e94e6040fc3ed03bab7c6a5c8 Mon Sep 17 00:00:00 2001
From: Wedson Almeida Filho <walmeida@microsoft.com>
Date: Fri, 16 Jun 2023 00:44:34 -0300
Subject: [PATCH 47/49] runtime: bind-mount mounted block device into container

When the mounted block device isn't a layer, we want to mount it into
containers, but since it's already mounted with the correct fs (e.g.,
tar, ext4, etc.) in the pod, we just bind-mount it into the container.

Fixes: #7536

Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
---
 src/runtime/virtcontainers/kata_agent.go | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/runtime/virtcontainers/kata_agent.go b/src/runtime/virtcontainers/kata_agent.go
index e102484c4c..810deeee54 100644
--- a/src/runtime/virtcontainers/kata_agent.go
+++ b/src/runtime/virtcontainers/kata_agent.go
@@ -1662,6 +1662,10 @@ func (k *kataAgent) handleBlkOCIMounts(c *Container, spec *specs.Spec) ([]*grpc.
 				"new-source":      path,
 			}).Debug("Replacing OCI mount source")
 			spec.Mounts[idx].Source = path
+			if HasOption(spec.Mounts[idx].Options, vcAnnotations.IsFileBlockDevice) {
+				// The device is already mounted, just bind to path in container.
+				spec.Mounts[idx].Options = []string{"bind"}
+			}
 			break
 		}
 

From c36572418f97a298f7647d74c1eb64e8994cb17e Mon Sep 17 00:00:00 2001
From: Wedson Almeida Filho <walmeida@microsoft.com>
Date: Wed, 28 Jun 2023 01:19:09 -0300
Subject: [PATCH 48/49] agent: avoid unnecessary calls to `Arc::clone`

These calls cause two extra atomic instructions each time they're used,
one to increment and another one to decrement the refcount.

Since we don't need them because the referred value is guaranteed to
outlive the function, remove the calls.

Fixes: #7190

Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
---
 src/agent/src/mount.rs | 69 ++++++++++++++-----------------
 src/agent/src/rpc.rs   | 92 ++++++++++++++----------------------------
 2 files changed, 61 insertions(+), 100 deletions(-)

diff --git a/src/agent/src/mount.rs b/src/agent/src/mount.rs
index 83f624f98c..84227b1821 100644
--- a/src/agent/src/mount.rs
+++ b/src/agent/src/mount.rs
@@ -230,7 +230,7 @@ pub fn baremount(
 async fn ephemeral_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     // hugetlbfs
     if storage.fstype == FS_TYPE_HUGETLB {
@@ -278,7 +278,7 @@ async fn ephemeral_storage_handler(
 pub async fn update_ephemeral_mounts(
     logger: Logger,
     storages: Vec<Storage>,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<()> {
     for (_, storage) in storages.iter().enumerate() {
         let handler_name = storage.driver.clone();
@@ -341,7 +341,7 @@ async fn overlayfs_storage_handler(
     logger: &Logger,
     storage: &Storage,
     cid: Option<&str>,
-    _sandbox: Arc<Mutex<Sandbox>>,
+    _sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     if storage
         .options
@@ -374,7 +374,7 @@ async fn overlayfs_storage_handler(
 async fn local_storage_handler(
     _logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     fs::create_dir_all(&storage.mount_point).context(format!(
         "failed to create dir all {:?}",
@@ -414,7 +414,7 @@ async fn local_storage_handler(
 async fn virtio9p_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    _sandbox: Arc<Mutex<Sandbox>>,
+    _sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     common_storage_handler(logger, storage)
 }
@@ -528,11 +528,11 @@ fn get_pagesize_and_size_from_option(options: &[String]) -> Result<(u64, u64)> {
 async fn virtiommio_blk_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     let storage = storage.clone();
     if !Path::new(&storage.source).exists() {
-        get_virtio_mmio_device_name(&sandbox, &storage.source)
+        get_virtio_mmio_device_name(sandbox, &storage.source)
             .await
             .context("failed to get mmio device name")?;
     }
@@ -545,7 +545,7 @@ async fn virtiommio_blk_storage_handler(
 async fn virtiofs_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    _sandbox: Arc<Mutex<Sandbox>>,
+    _sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     common_storage_handler(logger, storage)
 }
@@ -555,7 +555,7 @@ async fn virtiofs_storage_handler(
 async fn virtio_blk_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     let mut storage = storage.clone();
     // If hot-plugged, get the device node path based on the PCI path
@@ -570,7 +570,7 @@ async fn virtio_blk_storage_handler(
         }
     } else {
         let pcipath = pci::Path::from_str(&storage.source)?;
-        let dev_path = get_virtio_blk_pci_device_name(&sandbox, &pcipath).await?;
+        let dev_path = get_virtio_blk_pci_device_name(sandbox, &pcipath).await?;
         storage.source = dev_path;
     }
 
@@ -583,11 +583,11 @@ async fn virtio_blk_storage_handler(
 async fn virtio_blk_ccw_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     let mut storage = storage.clone();
     let ccw_device = ccw::Device::from_str(&storage.source)?;
-    let dev_path = get_virtio_blk_ccw_device_name(&sandbox, &ccw_device).await?;
+    let dev_path = get_virtio_blk_ccw_device_name(sandbox, &ccw_device).await?;
     storage.source = dev_path;
     common_storage_handler(logger, &storage)
 }
@@ -597,7 +597,7 @@ async fn virtio_blk_ccw_storage_handler(
 async fn virtio_blk_ccw_storage_handler(
     _: &Logger,
     _: &Storage,
-    _: Arc<Mutex<Sandbox>>,
+    _: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     Err(anyhow!("CCW is only supported on s390x"))
 }
@@ -607,12 +607,12 @@ async fn virtio_blk_ccw_storage_handler(
 async fn virtio_scsi_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     let mut storage = storage.clone();
 
     // Retrieve the device path from SCSI address.
-    let dev_path = get_scsi_device_name(&sandbox, &storage.source).await?;
+    let dev_path = get_scsi_device_name(sandbox, &storage.source).await?;
     storage.source = dev_path;
 
     common_storage_handler(logger, &storage)
@@ -633,12 +633,12 @@ fn common_storage_handler(logger: &Logger, storage: &Storage) -> Result<String>
 async fn nvdimm_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
 ) -> Result<String> {
     let storage = storage.clone();
 
     // Retrieve the device path from NVDIMM address.
-    wait_for_pmem_device(&sandbox, &storage.source).await?;
+    wait_for_pmem_device(sandbox, &storage.source).await?;
 
     common_storage_handler(logger, &storage)
 }
@@ -646,7 +646,7 @@ async fn nvdimm_storage_handler(
 async fn bind_watcher_storage_handler(
     logger: &Logger,
     storage: &Storage,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
     cid: Option<String>,
 ) -> Result<()> {
     let mut locked = sandbox.lock().await;
@@ -862,7 +862,7 @@ fn parse_mount_flags_and_options(options_vec: Vec<&str>) -> (MsFlags, String) {
 pub async fn add_storages(
     logger: Logger,
     storages: Vec<Storage>,
-    sandbox: Arc<Mutex<Sandbox>>,
+    sandbox: &Arc<Mutex<Sandbox>>,
     cid: Option<String>,
 ) -> Result<Vec<String>> {
     let mut mount_list = Vec::new();
@@ -882,31 +882,22 @@ pub async fn add_storages(
         }
 
         let res = match handler_name.as_str() {
-            DRIVER_BLK_TYPE => virtio_blk_storage_handler(&logger, &storage, sandbox.clone()).await,
-            DRIVER_BLK_CCW_TYPE => {
-                virtio_blk_ccw_storage_handler(&logger, &storage, sandbox.clone()).await
-            }
-            DRIVER_9P_TYPE => virtio9p_storage_handler(&logger, &storage, sandbox.clone()).await,
-            DRIVER_VIRTIOFS_TYPE => {
-                virtiofs_storage_handler(&logger, &storage, sandbox.clone()).await
-            }
-            DRIVER_EPHEMERAL_TYPE => {
-                ephemeral_storage_handler(&logger, &storage, sandbox.clone()).await
-            }
+            DRIVER_BLK_TYPE => virtio_blk_storage_handler(&logger, &storage, sandbox).await,
+            DRIVER_BLK_CCW_TYPE => virtio_blk_ccw_storage_handler(&logger, &storage, sandbox).await,
+            DRIVER_9P_TYPE => virtio9p_storage_handler(&logger, &storage, sandbox).await,
+            DRIVER_VIRTIOFS_TYPE => virtiofs_storage_handler(&logger, &storage, sandbox).await,
+            DRIVER_EPHEMERAL_TYPE => ephemeral_storage_handler(&logger, &storage, sandbox).await,
             DRIVER_OVERLAYFS_TYPE => {
-                overlayfs_storage_handler(&logger, &storage, cid.as_deref(), sandbox.clone()).await
+                overlayfs_storage_handler(&logger, &storage, cid.as_deref(), sandbox).await
             }
             DRIVER_MMIO_BLK_TYPE => {
-                virtiommio_blk_storage_handler(&logger, &storage, sandbox.clone()).await
+                virtiommio_blk_storage_handler(&logger, &storage, sandbox).await
             }
-            DRIVER_LOCAL_TYPE => local_storage_handler(&logger, &storage, sandbox.clone()).await,
-            DRIVER_SCSI_TYPE => {
-                virtio_scsi_storage_handler(&logger, &storage, sandbox.clone()).await
-            }
-            DRIVER_NVDIMM_TYPE => nvdimm_storage_handler(&logger, &storage, sandbox.clone()).await,
+            DRIVER_LOCAL_TYPE => local_storage_handler(&logger, &storage, sandbox).await,
+            DRIVER_SCSI_TYPE => virtio_scsi_storage_handler(&logger, &storage, sandbox).await,
+            DRIVER_NVDIMM_TYPE => nvdimm_storage_handler(&logger, &storage, sandbox).await,
             DRIVER_WATCHABLE_BIND_TYPE => {
-                bind_watcher_storage_handler(&logger, &storage, sandbox.clone(), cid.clone())
-                    .await?;
+                bind_watcher_storage_handler(&logger, &storage, sandbox, cid.clone()).await?;
                 // Don't register watch mounts, they're handled separately by the watcher.
                 Ok(String::new())
             }
diff --git a/src/agent/src/rpc.rs b/src/agent/src/rpc.rs
index 3b7f0a4720..7b92f245e8 100644
--- a/src/agent/src/rpc.rs
+++ b/src/agent/src/rpc.rs
@@ -150,9 +150,6 @@ impl AgentService {
         let mut oci_spec = req.OCI.clone();
         let use_sandbox_pidns = req.sandbox_pidns();
 
-        let sandbox;
-        let mut s;
-
         let mut oci = match oci_spec.as_mut() {
             Some(spec) => rustjail::grpc_to_oci(spec),
             None => {
@@ -184,15 +181,13 @@ impl AgentService {
         let m = add_storages(
             sl(),
             req.storages.to_vec(),
-            self.sandbox.clone(),
+            &self.sandbox,
             Some(req.container_id.clone()),
         )
         .await?;
-        {
-            sandbox = self.sandbox.clone();
-            s = sandbox.lock().await;
-            s.container_mounts.insert(cid.clone(), m);
-        }
+
+        let mut s = self.sandbox.lock().await;
+        s.container_mounts.insert(cid.clone(), m);
 
         update_container_namespaces(&s, &mut oci, use_sandbox_pidns)?;
 
@@ -265,8 +260,7 @@ impl AgentService {
     async fn do_start_container(&self, req: protocols::agent::StartContainerRequest) -> Result<()> {
         let cid = req.container_id;
 
-        let sandbox = self.sandbox.clone();
-        let mut s = sandbox.lock().await;
+        let mut s = self.sandbox.lock().await;
         let sid = s.id.clone();
 
         let ctr = s
@@ -300,8 +294,7 @@ impl AgentService {
         let cid = req.container_id.clone();
 
         if req.timeout == 0 {
-            let s = Arc::clone(&self.sandbox);
-            let mut sandbox = s.lock().await;
+            let mut sandbox = self.sandbox.lock().await;
 
             sandbox.bind_watcher.remove_container(&cid).await;
 
@@ -341,8 +334,7 @@ impl AgentService {
             return Err(anyhow!(nix::Error::UnknownErrno));
         }
 
-        let s = self.sandbox.clone();
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
         remove_container_resources(&mut sandbox, &cid)?;
 
         Ok(())
@@ -355,8 +347,7 @@ impl AgentService {
 
         info!(sl(), "do_exec_process cid: {} eid: {}", cid, exec_id);
 
-        let s = self.sandbox.clone();
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
 
         let mut process = req
             .process
@@ -383,7 +374,6 @@ impl AgentService {
     async fn do_signal_process(&self, req: protocols::agent::SignalProcessRequest) -> Result<()> {
         let cid = req.container_id.clone();
         let eid = req.exec_id.clone();
-        let s = self.sandbox.clone();
 
         info!(
             sl(),
@@ -395,7 +385,7 @@ impl AgentService {
 
         let mut sig: libc::c_int = req.signal as libc::c_int;
         {
-            let mut sandbox = s.lock().await;
+            let mut sandbox = self.sandbox.lock().await;
             let p = sandbox.find_container_process(cid.as_str(), eid.as_str())?;
             // For container initProcess, if it hasn't installed handler for "SIGTERM" signal,
             // it will ignore the "SIGTERM" signal sent to it, thus send it "SIGKILL" signal
@@ -468,8 +458,7 @@ impl AgentService {
     }
 
     async fn freeze_cgroup(&self, cid: &str, state: FreezerState) -> Result<()> {
-        let s = self.sandbox.clone();
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
         let ctr = sandbox
             .get_container(cid)
             .ok_or_else(|| anyhow!("Invalid container id {}", cid))?;
@@ -478,8 +467,7 @@ impl AgentService {
     }
 
     async fn get_pids(&self, cid: &str) -> Result<Vec<i32>> {
-        let s = self.sandbox.clone();
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
         let ctr = sandbox
             .get_container(cid)
             .ok_or_else(|| anyhow!("Invalid container id {}", cid))?;
@@ -494,7 +482,6 @@ impl AgentService {
     ) -> Result<protocols::agent::WaitProcessResponse> {
         let cid = req.container_id.clone();
         let eid = req.exec_id;
-        let s = self.sandbox.clone();
         let mut resp = WaitProcessResponse::new();
         let pid: pid_t;
 
@@ -508,7 +495,7 @@ impl AgentService {
         );
 
         let exit_rx = {
-            let mut sandbox = s.lock().await;
+            let mut sandbox = self.sandbox.lock().await;
             let p = sandbox.find_container_process(cid.as_str(), eid.as_str())?;
 
             p.exit_watchers.push(exit_send);
@@ -523,7 +510,7 @@ impl AgentService {
             info!(sl(), "cid {} eid {} received exit signal", &cid, &eid);
         }
 
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
         let ctr = sandbox
             .get_container(&cid)
             .ok_or_else(|| anyhow!("Invalid container id"))?;
@@ -565,8 +552,7 @@ impl AgentService {
         let eid = req.exec_id.clone();
 
         let writer = {
-            let s = self.sandbox.clone();
-            let mut sandbox = s.lock().await;
+            let mut sandbox = self.sandbox.lock().await;
             let p = sandbox.find_container_process(cid.as_str(), eid.as_str())?;
 
             // use ptmx io
@@ -597,9 +583,7 @@ impl AgentService {
 
         let term_exit_notifier;
         let reader = {
-            let s = self.sandbox.clone();
-            let mut sandbox = s.lock().await;
-
+            let mut sandbox = self.sandbox.lock().await;
             let p = sandbox.find_container_process(cid.as_str(), eid.as_str())?;
 
             term_exit_notifier = p.term_exit_notifier.clone();
@@ -630,7 +614,7 @@ impl AgentService {
             // Poll::Ready so that the term_exit_notifier will never polled
             // before all data were read.
             biased;
-            v = read_stream(reader, req.len as usize)  => {
+            v = read_stream(&reader, req.len as usize)  => {
                 let vector = v?;
                 let mut resp = ReadStreamResponse::new();
                 resp.set_data(vector);
@@ -734,8 +718,7 @@ impl agent_ttrpc::AgentService for AgentService {
         let cid = req.container_id.clone();
         let res = req.resources;
 
-        let s = Arc::clone(&self.sandbox);
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
 
         let ctr = sandbox.get_container(&cid).ok_or_else(|| {
             ttrpc_error(
@@ -768,8 +751,7 @@ impl agent_ttrpc::AgentService for AgentService {
         trace_rpc_call!(ctx, "stats_container", req);
         is_allowed(&req)?;
         let cid = req.container_id;
-        let s = Arc::clone(&self.sandbox);
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
 
         let ctr = sandbox.get_container(&cid).ok_or_else(|| {
             ttrpc_error(
@@ -790,8 +772,7 @@ impl agent_ttrpc::AgentService for AgentService {
         trace_rpc_call!(ctx, "pause_container", req);
         is_allowed(&req)?;
         let cid = req.container_id();
-        let s = Arc::clone(&self.sandbox);
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
 
         let ctr = sandbox.get_container(cid).ok_or_else(|| {
             ttrpc_error(
@@ -814,8 +795,7 @@ impl agent_ttrpc::AgentService for AgentService {
         trace_rpc_call!(ctx, "resume_container", req);
         is_allowed(&req)?;
         let cid = req.container_id();
-        let s = Arc::clone(&self.sandbox);
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
 
         let ctr = sandbox.get_container(cid).ok_or_else(|| {
             ttrpc_error(
@@ -896,8 +876,7 @@ impl agent_ttrpc::AgentService for AgentService {
 
         let cid = req.container_id.clone();
         let eid = req.exec_id;
-        let s = Arc::clone(&self.sandbox);
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
 
         let p = sandbox
             .find_container_process(cid.as_str(), eid.as_str())
@@ -923,8 +902,7 @@ impl agent_ttrpc::AgentService for AgentService {
 
         let cid = req.container_id.clone();
         let eid = req.exec_id.clone();
-        let s = Arc::clone(&self.sandbox);
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
         let p = sandbox
             .find_container_process(cid.as_str(), eid.as_str())
             .map_err(|e| {
@@ -1028,7 +1006,7 @@ impl agent_ttrpc::AgentService for AgentService {
         trace_rpc_call!(ctx, "update_mounts", req);
         is_allowed(&req)?;
 
-        match update_ephemeral_mounts(sl(), req.storages.to_vec(), self.sandbox.clone()).await {
+        match update_ephemeral_mounts(sl(), req.storages.to_vec(), &self.sandbox).await {
             Ok(_) => Ok(Empty::new()),
             Err(e) => Err(ttrpc_error(
                 ttrpc::Code::INTERNAL,
@@ -1251,8 +1229,7 @@ impl agent_ttrpc::AgentService for AgentService {
         is_allowed(&req)?;
 
         {
-            let sandbox = self.sandbox.clone();
-            let mut s = sandbox.lock().await;
+            let mut s = self.sandbox.lock().await;
 
             let _ = fs::remove_dir_all(CONTAINER_BASE);
             let _ = fs::create_dir_all(CONTAINER_BASE);
@@ -1282,19 +1259,16 @@ impl agent_ttrpc::AgentService for AgentService {
                 .map_err(|e| ttrpc_error(ttrpc::Code::INTERNAL, e))?;
         }
 
-        match add_storages(sl(), req.storages.to_vec(), self.sandbox.clone(), None).await {
+        match add_storages(sl(), req.storages.to_vec(), &self.sandbox, None).await {
             Ok(m) => {
-                let sandbox = self.sandbox.clone();
-                let mut s = sandbox.lock().await;
-                s.mounts = m
+                self.sandbox.lock().await.mounts = m;
             }
             Err(e) => return Err(ttrpc_error(ttrpc::Code::INTERNAL, e)),
         };
 
         match setup_guest_dns(sl(), req.dns.to_vec()) {
             Ok(_) => {
-                let sandbox = self.sandbox.clone();
-                let mut s = sandbox.lock().await;
+                let mut s = self.sandbox.lock().await;
                 let _dns = req
                     .dns
                     .to_vec()
@@ -1315,8 +1289,7 @@ impl agent_ttrpc::AgentService for AgentService {
         trace_rpc_call!(ctx, "destroy_sandbox", req);
         is_allowed(&req)?;
 
-        let s = Arc::clone(&self.sandbox);
-        let mut sandbox = s.lock().await;
+        let mut sandbox = self.sandbox.lock().await;
         // destroy all containers, clean up, notify agent to exit
         // etc.
         sandbox
@@ -1383,8 +1356,7 @@ impl agent_ttrpc::AgentService for AgentService {
         req: protocols::agent::OnlineCPUMemRequest,
     ) -> ttrpc::Result<Empty> {
         is_allowed(&req)?;
-        let s = Arc::clone(&self.sandbox);
-        let sandbox = s.lock().await;
+        let sandbox = self.sandbox.lock().await;
         trace_rpc_call!(ctx, "online_cpu_mem", req);
 
         sandbox
@@ -1507,12 +1479,10 @@ impl agent_ttrpc::AgentService for AgentService {
         req: protocols::agent::GetOOMEventRequest,
     ) -> ttrpc::Result<OOMEvent> {
         is_allowed(&req)?;
-        let sandbox = self.sandbox.clone();
-        let s = sandbox.lock().await;
+        let s = self.sandbox.lock().await;
         let event_rx = &s.event_rx.clone();
         let mut event_rx = event_rx.lock().await;
         drop(s);
-        drop(sandbox);
 
         if let Some(container_id) = event_rx.recv().await {
             info!(sl(), "get_oom_event return {}", &container_id);
@@ -1709,7 +1679,7 @@ fn get_agent_details() -> AgentDetails {
     detail
 }
 
-async fn read_stream(reader: Arc<Mutex<ReadHalf<PipeStream>>>, l: usize) -> Result<Vec<u8>> {
+async fn read_stream(reader: &Mutex<ReadHalf<PipeStream>>, l: usize) -> Result<Vec<u8>> {
     let mut content = vec![0u8; l];
 
     let mut reader = reader.lock().await;

From 66db5b53500b5e75be3f25882cd59273f71d0b9a Mon Sep 17 00:00:00 2001
From: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Date: Wed, 2 Aug 2023 16:56:27 +0000
Subject: [PATCH 49/49] metrics: Add latency test to network README

This PR adds latency test to network README for kata metrics.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
---
 tests/metrics/network/README.md                                  | 1 +
 .../metrics/network/latency_kubernetes}/latency-client.yaml      | 0
 .../metrics/network/latency_kubernetes}/latency-network.sh       | 0
 .../metrics/network/latency_kubernetes}/latency-server.yaml      | 0
 4 files changed, 1 insertion(+)
 rename {latency_kubernetes/runtimeclass_workloads => tests/metrics/network/latency_kubernetes}/latency-client.yaml (100%)
 rename {latency_kubernetes => tests/metrics/network/latency_kubernetes}/latency-network.sh (100%)
 rename {latency_kubernetes/runtimeclass_workloads => tests/metrics/network/latency_kubernetes}/latency-server.yaml (100%)

diff --git a/tests/metrics/network/README.md b/tests/metrics/network/README.md
index c109a68411..05acd4cd67 100644
--- a/tests/metrics/network/README.md
+++ b/tests/metrics/network/README.md
@@ -10,6 +10,7 @@ bandwidth, jitter, latency and parallel bandwidth.
 ## Networking tests
 
 - `k8s-network-metrics-iperf3.sh` measures bandwidth which is the speed of the data transfer.
+- `latency-network.sh` measures network latency.
 
 ## Running the tests
 
diff --git a/latency_kubernetes/runtimeclass_workloads/latency-client.yaml b/tests/metrics/network/latency_kubernetes/latency-client.yaml
similarity index 100%
rename from latency_kubernetes/runtimeclass_workloads/latency-client.yaml
rename to tests/metrics/network/latency_kubernetes/latency-client.yaml
diff --git a/latency_kubernetes/latency-network.sh b/tests/metrics/network/latency_kubernetes/latency-network.sh
similarity index 100%
rename from latency_kubernetes/latency-network.sh
rename to tests/metrics/network/latency_kubernetes/latency-network.sh
diff --git a/latency_kubernetes/runtimeclass_workloads/latency-server.yaml b/tests/metrics/network/latency_kubernetes/latency-server.yaml
similarity index 100%
rename from latency_kubernetes/runtimeclass_workloads/latency-server.yaml
rename to tests/metrics/network/latency_kubernetes/latency-server.yaml