Merge pull request #2015 from fidencio/2.2.0-alpha0-branch-bump

# Kata Containers 2.2.0-alpha0

.github/workflows/require-pr-porting-labels.yaml

@@ -12,6 +12,9 @@ on:
       - reopened
       - labeled
       - unlabeled
+   pull_request:
+     branches:
+      - main
 
 jobs:
   check-pr-porting-labels:

.github/workflows/snap-release.yaml

@@ -33,5 +33,5 @@ jobs:
           snap_file="kata-containers_${snap_version}_amd64.snap"
           # Upload the snap if it exists
           if [ -f ${snap_file} ]; then
-            snapcraft upload --release=candidate ${snap_file}
+            snapcraft upload --release=stable ${snap_file}
           fi

VERSION

@@ -1 +1 @@
-2.1.0
+2.2.0-alpha0

docs/Developer-Guide.md

@@ -37,6 +37,7 @@
   - [Set up a debug console](#set-up-a-debug-console)
     - [Simple debug console setup](#simple-debug-console-setup)
       - [Enable agent debug console](#enable-agent-debug-console)
+      - [Start `kata-monitor` - ONLY NEEDED FOR 2.0.x](#start-kata-monitor---only-needed-for-20x)
       - [Connect to debug console](#connect-to-debug-console)
     - [Traditional debug console setup](#traditional-debug-console-setup)
       - [Create a custom image containing a shell](#create-a-custom-image-containing-a-shell)
@@ -304,7 +305,7 @@ You MUST choose one of `alpine`, `centos`, `clearlinux`, `debian`, `euleros`, `f
 > - You should only do this step if you are testing with the latest version of the agent.
 
 ```
-$ sudo install -o root -g root -m 0550 -t ${ROOTFS_DIR}/bin ../../../src/agent/target/x86_64-unknown-linux-musl/release/kata-agent
+$ sudo install -o root -g root -m 0550 -t ${ROOTFS_DIR}/usr/bin ../../../src/agent/target/x86_64-unknown-linux-musl/release/kata-agent
 $ sudo install -o root -g root -m 0440 ../../../src/agent/kata-agent.service ${ROOTFS_DIR}/usr/lib/systemd/system/
 $ sudo install -o root -g root -m 0440 ../../../src/agent/kata-containers.target ${ROOTFS_DIR}/usr/lib/systemd/system/
 ```
@@ -393,14 +394,40 @@ You may choose to manually build your VMM/hypervisor.
 Kata Containers makes use of upstream QEMU branch. The exact version
 and repository utilized can be found by looking at the [versions file](../versions.yaml).
 
-Kata often utilizes patches for not-yet-upstream fixes for components,
-including QEMU. These can be found in the [packaging/QEMU directory](../tools/packaging/qemu/patches)
+Find the correct version of QEMU from the versions file:
+```
+$ source ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/scripts/lib.sh
+$ qemu_version=$(get_from_kata_deps "assets.hypervisor.qemu.version")
+$ echo ${qemu_version}
+```
+Get source from the matching branch of QEMU:
+```
+$ go get -d github.com/qemu/qemu
+$ cd ${GOPATH}/src/github.com/qemu/qemu
+$ git checkout ${qemu_version}
+$ your_qemu_directory=${GOPATH}/src/github.com/qemu/qemu
+```
+
+There are scripts to manage the build and packaging of QEMU. For the examples below, set your
+environment as:
+```
+$ go get -d github.com/kata-containers/kata-containers
+$ packaging_dir="${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging"
+```
+
+Kata often utilizes patches for not-yet-upstream and/or backported fixes for components,
+including QEMU. These can be found in the [packaging/QEMU directory](../tools/packaging/qemu/patches),
+and it's *recommended* that you apply them. For example, suppose that you are going to build QEMU
+version 5.2.0, do:
+```
+$ cd $your_qemu_directory
+$ $packaging_dir/scripts/apply_patches.sh $packaging_dir/qemu/patches/5.2.x/
+```
 
 To build utilizing the same options as Kata, you should make use of the `configure-hypervisor.sh` script. For example:
 ```
-$ go get -d github.com/kata-containers/kata-containers/tools/packaging
 $ cd $your_qemu_directory
-$ ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/scripts/configure-hypervisor.sh kata-qemu > kata.cfg
+$ $packaging_dir/scripts/configure-hypervisor.sh kata-qemu > kata.cfg
 $ eval ./configure "$(cat kata.cfg)"
 $ make -j $(nproc)
 $ sudo -E make install
@@ -475,6 +502,16 @@ debug_console_enabled = true
 
 This will pass `agent.debug_console agent.debug_console_vport=1026` to agent as kernel parameters, and sandboxes created using this parameters will start a shell in guest if new connection is accept from VSOCK.
 
+#### Start `kata-monitor` - ONLY NEEDED FOR 2.0.x
+
+For Kata Containers `2.0.x` releases, the `kata-runtime exec` command depends on the`kata-monitor` running, in order to get the sandbox's `vsock` address to connect to. Thus, first start the `kata-monitor` process.
+
+```
+$ sudo kata-monitor
+```
+
+`kata-monitor` will serve at `localhost:8090` by default.
+
 #### Connect to debug console
 
 Command `kata-runtime exec` is used to connect to the debug console.

docs/Release-Process.md

@@ -29,6 +29,7 @@
 
 ## Release Process
 
+
 ### Bump all Kata repositories
 
   Bump the repositories using a script in the Kata packaging repo, where:
@@ -41,6 +42,23 @@
   $ ./update-repository-version.sh -p "$NEW_VERSION" "$BRANCH"
   ```
 
+### Point tests repository to stable branch
+
+  If you create a new stable branch, i.e. if your release changes a major or minor version number (not a patch release), then
+  you should modify the `tests` repository to point to that newly created stable branch and not the `main` branch.
+  The objective is that changes in the CI on the main branch will not impact the stable branch.
+
+  In the test directory, change references the main branch in:
+  * `README.md`
+  * `versions.yaml`
+  * `cmd/github-labels/labels.yaml.in`
+  * `cmd/pmemctl/pmemctl.sh`
+  * `.ci/lib.sh`
+  * `.ci/static-checks.sh`
+
+  See the commits in [the corresponding PR for stable-2.1](https://github.com/kata-containers/tests/pull/3504) for an example of the changes.
+
+
 ### Merge all bump version Pull requests
 
   - The above step will create a GitHub pull request in the Kata projects. Trigger the CI using `/test` command on each bump Pull request.
@@ -50,7 +68,7 @@
 ### Tag all Kata repositories
 
   Once all the pull requests to bump versions in all Kata repositories are merged,
-  tag all the repositories as shown below.  
+  tag all the repositories as shown below.
   ```
   $ cd ${GOPATH}/src/github.com/kata-containers/kata-containers/tools/packaging/release
   $ git checkout  <kata-branch-to-release>
@@ -60,7 +78,7 @@
 
 ### Check Git-hub Actions
 
-  We make use of [GitHub actions](https://github.com/features/actions) in this [file](https://github.com/kata-containers/kata-containers/blob/master/.github/workflows/main.yaml) in the `kata-containers/kata-containers` repository to build and upload release artifacts. This action is auto triggered with the above step when a new tag is pushed to the `kata-containers/kata-conatiners` repository.
+  We make use of [GitHub actions](https://github.com/features/actions) in this [file](https://github.com/kata-containers/kata-containers/blob/main/.github/workflows/main.yaml) in the `kata-containers/kata-containers` repository to build and upload release artifacts. This action is auto triggered with the above step when a new tag is pushed to the `kata-containers/kata-containers` repository.
 
   Check the [actions status page](https://github.com/kata-containers/kata-containers/actions) to verify all steps in the actions workflow have completed successfully. On success, a static tarball containing Kata release artifacts will be uploaded to the [Release page](https://github.com/kata-containers/kata-containers/releases).
 

docs/Stable-Branch-Strategy.md

@@ -32,16 +32,16 @@ provides additional information regarding release `99.123.77` in the previous ex
   changing the existing behavior*.
 
 - When `MAJOR` increases, the new release adds **new features, bug fixes, or
-  both** and which *changes the behavior from the previous release* (incompatible with previous releases).
+  both** and which **changes the behavior from the previous release** (incompatible with previous releases).
 
   A major release will also likely require a change of the container manager version used, 
-for example Docker\*. Please refer to the release notes for further details.
+for example Containerd or CRI-O. Please refer to the release notes for further details.
 
 ## Release Strategy
 
 Any new features added since the last release will be available in the next minor
 release. These will include bug fixes as well. To facilitate a stable user environment, 
-Kata provides stable branch-based releases and a master branch release.
+Kata provides stable branch-based releases and a main branch release.
 
 ## Stable branch patch criteria
 
@@ -49,9 +49,10 @@ No new features should be introduced to stable branches.  This is intended to li
 providing only bug and security fixes.
 
 ## Branch Management
-Kata Containers will maintain two stable release branches in addition to the master branch.
-Once a new MAJOR or MINOR release is created from master, a new stable branch is created for
-the prior MAJOR or MINOR release and the older stable branch is no longer maintained. End of
+Kata Containers will maintain **one** stable release branch, in addition to the main branch, for
+each active major release.
+Once a new MAJOR or MINOR release is created from main, a new stable branch is created for
+the prior MAJOR or MINOR release and the previous stable branch is no longer maintained. End of
 maintenance for a branch is announced on the Kata Containers mailing list.  Users can determine
 the version currently installed by running `kata-runtime kata-env`. It is recommended to use the
 latest stable branch available.
@@ -61,59 +62,59 @@ A couple of examples follow to help clarify this process.
 ### New bug fix introduced
 
 A bug fix is submitted against the runtime which does not introduce new inter-component dependencies.
-This fix is applied to both the master and stable branches, and there is no need to create a new
+This fix is applied to both the main and stable branches, and there is no need to create a new
 stable branch.
 
 | Branch | Original version | New version |
 |--|--|--|
-| `master` | `1.3.0-rc0` | `1.3.0-rc1` |
-| `stable-1.2` | `1.2.0` | `1.2.1` |
-| `stable-1.1` | `1.1.2` | `1.1.3` |
+| `main` | `2.3.0-rc0` | `2.3.0-rc1` |
+| `stable-2.2` | `2.2.0` | `2.2.1` |
+| `stable-2.1` | (unmaintained) | (unmaintained) |
 
 
 ### New release made feature or change adding new inter-component dependency
 
 A new feature is introduced, which adds a new inter-component dependency. In this case a new stable
-branch is created (stable-1.3) starting from master and the older stable branch (stable-1.1)
+branch is created (stable-2.3) starting from main and the previous stable branch (stable-2.2)
 is dropped from maintenance.
 
 
 | Branch | Original version | New version |
 |--|--|--|
-| `master` | `1.3.0-rc1` | `1.3.0` |
-| `stable-1.3` | N/A| `1.3.0` |
-| `stable-1.2` | `1.2.1` | `1.2.2` |
-| `stable-1.1` | `1.1.3` | (unmaintained) |
+| `main` | `2.3.0-rc1` | `2.3.0` |
+| `stable-2.3` | N/A| `2.3.0` |
+| `stable-2.2` | `2.2.1` | (unmaintained) |
+| `stable-2.1` | (unmaintained) | (unmaintained) |
 
-Note, the stable-1.1 branch will still exist with tag 1.1.3, but under current plans it is
-not maintained further. The next tag applied to master will be 1.4.0-alpha0. We would then
+Note, the stable-2.2 branch will still exist with tag 2.2.1, but under current plans it is
+not maintained further. The next tag applied to main will be 2.4.0-alpha0. We would then
 create a couple of alpha releases gathering features targeted for that particular release (in
-this case 1.4.0), followed by a release candidate. The release candidate marks a feature freeze.
+this case 2.4.0), followed by a release candidate. The release candidate marks a feature freeze.
 A new stable branch is created for the release candidate. Only bug fixes and any security issues
-are added to the branch going forward until release 1.4.0 is made.
+are added to the branch going forward until release 2.4.0 is made.
    
 ## Backporting Process 
 
-Development that occurs against the master branch and applicable code commits should also be submitted
+Development that occurs against the main branch and applicable code commits should also be submitted
 against the stable branches. Some guidelines for this process follow::
   1. Only bug and security fixes which do not introduce inter-component dependencies are
  candidates for stable branches. These PRs should be marked with "bug" in GitHub.
-  2. Once a PR is created against master which meets requirement of (1), a comparable one
+  2. Once a PR is created against main which meets requirement of (1), a comparable one
  should also be submitted against the stable branches. It is the responsibility of the submitter
  to apply their pull request against stable, and it is the responsibility of the
  reviewers to help identify stable-candidate pull requests.
  
 ## Continuous Integration Testing
 
-The test repository is forked to create stable branches from master. Full CI
-runs on each stable and master PR using its respective tests repository branch.
+The test repository is forked to create stable branches from main. Full CI
+runs on each stable and main PR using its respective tests repository branch.
 
 ### An alternative method for CI testing:
 
-Ideally, the continuous integration infrastructure will run the same test suite on both master
+Ideally, the continuous integration infrastructure will run the same test suite on both main
 and the stable branches.  When tests are modified or new feature tests are introduced, explicit
 logic should exist within the testing CI to make sure only applicable tests are executed against
-stable and master. While this is not in place currently, it should be considered in the long term.
+stable and main. While this is not in place currently, it should be considered in the long term.
 
 ## Release Management
 
@@ -121,7 +122,7 @@ stable and master. While this is not in place currently, it should be considered
 
 Releases are made every three weeks, which include a GitHub release as
 well as binary packages. These patch releases are made for both stable branches, and a "release candidate"
-for the next `MAJOR` or `MINOR` is created from master. If there are no changes across all the repositories, no
+for the next `MAJOR` or `MINOR` is created from main. If there are no changes across all the repositories, no
 release is created and an announcement is made on the developer mailing list to highlight this.
 If a release is being made, each repository is tagged for this release, regardless
 of whether changes are introduced. The release schedule can be seen on the
@@ -142,10 +143,10 @@ maturity, we have increased the cadence from six weeks to twelve weeks. The rele
 ### Compatibility
 Kata guarantees compatibility between components that are within one minor release of each other. 
  
-This is critical for dependencies which cross between host (runtime, shim, proxy) and
+This is critical for dependencies which cross between host (shimv2 runtime) and
 the guest (hypervisor, rootfs and agent).  For example, consider a cluster with a long-running
-deployment, workload-never-dies, all on Kata version 1.1.3 components. If the operator updates
-the Kata components to the next new minor release (i.e. 1.2.0), we need to guarantee that the 1.2.0
-runtime still communicates with 1.1.3 agent within workload-never-dies.
+deployment, workload-never-dies, all on Kata version 2.1.3 components. If the operator updates
+the Kata components to the next new minor release (i.e. 2.2.0), we need to guarantee that the 2.2.0
+shimv2 runtime still communicates with 2.1.3 agent within workload-never-dies.
 
 Handling live-update is out of the scope of this document. See this [`kata-runtime` issue](https://github.com/kata-containers/runtime/issues/492) for details.

docs/how-to/README.md

@@ -37,3 +37,4 @@
 - [How to use Kata Containers with `virtio-mem`](how-to-use-virtio-mem-with-kata.md)
 - [How to set sandbox Kata Containers configurations with pod annotations](how-to-set-sandbox-config-kata.md)
 - [How to monitor Kata Containers in K8s](how-to-set-prometheus-in-k8s.md)
+- [How to use hotplug memory on arm64 in Kata Containers](how-to-hotplug-memory-arm64.md)

docs/how-to/how-to-hotplug-memory-arm64.md

@@ -0,0 +1,32 @@
+# How to use memory hotplug feature in Kata Containers on arm64
+
+- [Introduction](#introduction)
+- [Install UEFI ROM](#install-uefi-rom)
+- [Run for test](#run-for-test)
+
+## Introduction
+
+Memory hotplug is a key feature for containers to allocate memory dynamically in deployment.
+As Kata Container bases on VM, this feature needs support both from VMM and guest kernel. Luckily, it has been fully supported for the current default version of QEMU and guest kernel used by Kata on arm64. For other VMMs, e.g, Cloud Hypervisor, the enablement work is on the road. Apart from VMM and guest kernel, memory hotplug also depends on ACPI which depends on firmware either. On x86, you can boot a VM using QEMU with ACPI enabled directly, because it boots up with firmware implicitly. For arm64, however, you need specify firmware explicitly. That is to say, if you are ready to run a normal Kata Container on arm64, what you need extra to do is to install the UEFI ROM before use the memory hotplug feature.
+
+## Install UEFI ROM
+
+We have offered a helper script for you to install the UEFI ROM. If you have installed Kata normally on your host, you just need to run the script as fellows:
+
+```bash
+$ pushd $GOPATH/src/github.com/kata-containers/tests
+$ sudo .ci/aarch64/install_rom_aarch64.sh
+$ popd
+```
+
+## Run for test
+
+Let's test if the memory hotplug is ready for Kata after install the UEFI ROM. Make sure containerd is ready to run Kata before test.
+
+```bash
+$ sudo ctr image pull docker.io/library/ubuntu:latest
+$ sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/ubuntu:latest hello sh -c "free -h"
+$ sudo ctr run --runtime io.containerd.run.kata.v2 -t --memory-limit 536870912 --rm docker.io/library/ubuntu:latest hello sh -c "free -h"
+```
+
+Compare the results between the two tests. If the latter is 0.5G larger than the former, you have done what you want, and congratulation!

docs/how-to/how-to-use-virtio-mem-with-kata.md

@@ -13,26 +13,23 @@ Kata Containers with `virtio-mem` supports memory resize.
 
 ## Requisites
 
-Kata Containers with `virtio-mem` requires Linux and the QEMU that support `virtio-mem`.
-The Linux kernel and QEMU upstream version still not support `virtio-mem`.  @davidhildenbrand is working on them.
-Please use following unofficial version of the Linux kernel and QEMU that support `virtio-mem` with Kata Containers.
+Kata Containers just supports `virtio-mem` with QEMU.
+Install and setup Kata Containers as shown [here](../install/README.md).
 
-The Linux kernel is at https://github.com/davidhildenbrand/linux/tree/virtio-mem-rfc-v4.
-The Linux kernel config that can work with Kata Containers is at https://gist.github.com/teawater/016194ee84748c768745a163d08b0fb9.
-
-The QEMU is at https://github.com/teawater/qemu/tree/kata-virtio-mem. (The original source is at https://github.com/davidhildenbrand/qemu/tree/virtio-mem.  Its base version of QEMU cannot work with Kata Containers.  So merge the commit of `virtio-mem` to upstream QEMU.)
-
-Set Linux and the QEMU that support `virtio-mem` with following line in the Kata Containers QEMU configuration `configuration-qemu.toml`:
-```toml
-[hypervisor.qemu]
-path = "qemu-dir"
-kernel = "vmlinux-dir"
+### With x86_64
+The `virtio-mem` config of the x86_64 Kata Linux kernel is open.
+Enable `virtio-mem` as follows:
+```
+$ sudo sed -i -e 's/^#enable_virtio_mem.*$/enable_virtio_mem = true/g' /etc/kata-containers/configuration.toml
 ```
 
-Enable `virtio-mem` with following line in the Kata Containers configuration:
-```toml
-enable_virtio_mem = true
+### With other architectures
+The `virtio-mem` config of the others Kata Linux kernel is not open.
+You can open `virtio-mem` config as follows:
 ```
+CONFIG_VIRTIO_MEM=y
+```
+Then you can build and install the guest kernel image as shown [here](../../tools/packaging/kernel/README.md#build-kata-containers-kernel).
 
 ## Run a Kata Container utilizing `virtio-mem`
 
@@ -41,13 +38,35 @@ Use following command to enable memory overcommitment of a Linux kernel.  Becaus
 $ echo 1 | sudo tee /proc/sys/vm/overcommit_memory
 ```
 
-Use following command start a Kata Container.
+Use following command to start a Kata Container.
 ```
-$ docker run --rm -it --runtime=kata --name test busybox
+$ pod_yaml=pod.yaml
+$ container_yaml=${REPORT_DIR}/container.yaml
+$ image="quay.io/prometheus/busybox:latest"
+$ cat << EOF > "${pod_yaml}"
+metadata:
+  name: busybox-sandbox1
+EOF
+$ cat << EOF > "${container_yaml}"
+metadata:
+  name: busybox-killed-vmm
+image:
+  image: "$image"
+command:
+- top
+EOF
+$ sudo crictl pull $image
+$ podid=$(sudo crictl runp $pod_yaml)
+$ cid=$(sudo crictl create $podid $container_yaml $pod_yaml)
+$ sudo crictl start $cid
 ```
 
-Use following command set the memory size of test to default_memory + 512m.
+Use the following command to set the container memory limit to 2g and the memory size of the VM to its default_memory + 2g.
 ```
-$ docker update -m 512m --memory-swap -1 test
+$ sudo crictl update --memory $((2*1024*1024*1024)) $cid
 ```
 
+Use the following command to set the container memory limit to 1g and the memory size of the VM to its default_memory + 1g.
+```
+$ sudo crictl update --memory $((1*1024*1024*1024)) $cid
+```

docs/how-to/run-kata-with-k8s.md

@@ -171,10 +171,10 @@ $ sudo systemctl daemon-reload
 $ sudo systemctl restart kubelet
 
 # If using CRI-O
-$ sudo kubeadm init --skip-preflight-checks --cri-socket /var/run/crio/crio.sock --pod-network-cidr=10.244.0.0/16
+$ sudo kubeadm init --ignore-preflight-errors=all --cri-socket /var/run/crio/crio.sock --pod-network-cidr=10.244.0.0/16
 
 # If using CRI-containerd
-$ sudo kubeadm init --skip-preflight-checks --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
+$ sudo kubeadm init --ignore-preflight-errors=all --cri-socket /run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16
 
 $ export KUBECONFIG=/etc/kubernetes/admin.conf
 ```

docs/install/snap-installation-guide.md

@@ -14,7 +14,7 @@ Kata Containers can be installed in any Linux distribution that supports
 Run the following command to install **Kata Containers**:
 
 ```sh
-$ sudo snap install kata-containers --candidate --classic
+$ sudo snap install kata-containers --stable --classic
 ```
 
 ## Configure Kata Containers

src/agent/Cargo.toml

@@ -13,8 +13,8 @@ lazy_static = "1.3.0"
 ttrpc = { version = "0.5.0", features = ["async", "protobuf-codec"], default-features = false }
 protobuf = "=2.14.0"
 libc = "0.2.58"
-nix = "0.17.0"
-prctl = "1.0.0"
+nix = "0.21.0"
+capctl = "0.2.0"
 serde_json = "1.0.39"
 scan_fmt = "0.2.3"
 scopeguard = "1.0.0"
@@ -24,8 +24,11 @@ async-trait = "0.1.42"
 tokio = { version = "1.2.0", features = ["rt", "rt-multi-thread", "sync", "macros", "io-util", "time", "signal", "io-std", "process", "fs"] }
 futures = "0.3.12"
 netlink-sys = { version = "0.6.0", features = ["tokio_socket",]}
-tokio-vsock = "0.3.0"
-rtnetlink = "0.7.0"
+tokio-vsock = "0.3.1"
+# Because the author has no time to maintain the crate, we switch the dependency to github,
+# Once the new version released on crates.io, we switch it back.
+# https://github.com/little-dude/netlink/issues/161
+rtnetlink = { git = "https://github.com/little-dude/netlink", rev = "a9367bc4700496ddebc088110c28f40962923326" }
 netlink-packet-utils = "0.4.0"
 ipnetwork = "0.17.0"
 
@@ -40,13 +43,21 @@ slog-scope = "4.1.2"
 slog-stdlog = "4.0.0"
 log = "0.4.11"
 
-# for testing
-tempfile = "3.1.0"
 prometheus = { version = "0.9.0", features = ["process"] }
 procfs = "0.7.9"
 anyhow = "1.0.32"
 cgroups = { package = "cgroups-rs", version = "0.2.5" }
 
+# Tracing
+tracing = "0.1.26"
+tracing-subscriber = "0.2.18"
+tracing-opentelemetry = "0.13.0"
+opentelemetry = "0.14.0"
+vsock-exporter = { path = "vsock-exporter" }
+
+[dev-dependencies]
+tempfile = "3.1.0"
+
 [workspace]
 members = [
     "oci",

src/agent/rustjail/Cargo.toml

@@ -11,9 +11,9 @@ serde_derive = "1.0.91"
 oci = { path = "../oci" }
 protocols = { path ="../protocols" }
 caps = "0.5.0"
-nix = "0.17.0"
+nix = "0.21.0"
 scopeguard = "1.0.0"
-prctl = "1.0.0"
+capctl = "0.2.0"
 lazy_static = "1.3.0"
 libc = "0.2.58"
 protobuf = "=2.14.0"
@@ -24,7 +24,6 @@ regex = "1.1"
 path-absolutize = "1.2.0"
 anyhow = "1.0.32"
 cgroups = { package = "cgroups-rs", version = "0.2.5" }
-tempfile = "3.1.0"
 rlimit = "0.5.3"
 
 tokio = { version = "1.2.0", features = ["sync", "io-util", "process", "time", "macros"] }
@@ -34,3 +33,4 @@ inotify = "0.9.2"
 
 [dev-dependencies]
 serial_test = "0.5.0"
+tempfile = "3.1.0"

src/agent/rustjail/src/cgroups/fs/mod.rs

@@ -349,14 +349,34 @@ fn set_memory_resources(cg: &cgroups::Cgroup, memory: &LinuxMemory, update: bool
         mem_controller.set_kmem_limit(-1)?;
     }
 
-    set_resource!(mem_controller, set_limit, memory, limit);
-    set_resource!(mem_controller, set_soft_limit, memory, reservation);
-    set_resource!(mem_controller, set_kmem_limit, memory, kernel);
-    set_resource!(mem_controller, set_tcp_limit, memory, kernel_tcp);
+    // If the memory update is set to -1 we should also
+    // set swap to -1, it means unlimited memory.
+    let mut swap = memory.swap.unwrap_or(0);
+    if memory.limit == Some(-1) {
+        swap = -1;
+    }
 
-    if let Some(swap) = memory.swap {
-        // set memory swap
-        let swap = if cg.v2() {
+    if memory.limit.is_some() && swap != 0 {
+        let memstat = get_memory_stats(cg)
+            .into_option()
+            .ok_or_else(|| anyhow!("failed to get the cgroup memory stats"))?;
+        let memusage = memstat.get_usage();
+
+        // When update memory limit, the kernel would check the current memory limit
+        // set against the new swap setting, if the current memory limit is large than
+        // the new swap, then set limit first, otherwise the kernel would complain and
+        // refused to set; on the other hand, if the current memory limit is smaller than
+        // the new swap, then we should set the swap first and then set the memor limit.
+        if swap == -1 || memusage.get_limit() < swap as u64 {
+            mem_controller.set_memswap_limit(swap)?;
+            set_resource!(mem_controller, set_limit, memory, limit);
+        } else {
+            set_resource!(mem_controller, set_limit, memory, limit);
+            mem_controller.set_memswap_limit(swap)?;
+        }
+    } else {
+        set_resource!(mem_controller, set_limit, memory, limit);
+        swap = if cg.v2() {
             convert_memory_swap_to_v2_value(swap, memory.limit.unwrap_or(0))?
         } else {
             swap
@@ -366,6 +386,10 @@ fn set_memory_resources(cg: &cgroups::Cgroup, memory: &LinuxMemory, update: bool
         }
     }
 
+    set_resource!(mem_controller, set_soft_limit, memory, reservation);
+    set_resource!(mem_controller, set_kmem_limit, memory, kernel);
+    set_resource!(mem_controller, set_tcp_limit, memory, kernel_tcp);
+
     if let Some(swappiness) = memory.swappiness {
         if (0..=100).contains(&swappiness) {
             mem_controller.set_swappiness(swappiness as u64)?;

src/agent/rustjail/src/configs/device.rs

@@ -1,56 +0,0 @@
-// Copyright (c) 2019 Ant Financial
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-
-use libc::*;
-use serde;
-#[macro_use]
-use serde_derive;
-use serde_json;
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Device {
-    #[serde(default)]
-    r#type: char,
-    #[serde(default)]
-    path: String,
-    #[serde(default)]
-    major: i64,
-    #[serde(default)]
-    minor: i64,
-    #[serde(default)]
-    permissions: String,
-    #[serde(default)]
-    file_mode: mode_t,
-    #[serde(default)]
-    uid: i32,
-    #[serde(default)]
-    gid: i32,
-    #[serde(default)]
-    allow: bool,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct BlockIODevice {
-    #[serde(default)]
-    major: i64,
-    #[serde(default)]
-    minor: i64,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct WeightDevice {
-    block: BlockIODevice,
-    #[serde(default)]
-    weight: u16,
-    #[serde(default, rename = "leafWeight")]
-    leaf_weight: u16,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct ThrottleDevice {
-    block: BlockIODevice,
-    #[serde(default)]
-    rate: u64,
-}

src/agent/rustjail/src/configs/mod.rs

@@ -1,372 +0,0 @@
-// Copyright (c) 2019 Ant Financial
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-
-use serde;
-#[macro_use]
-use serde_derive;
-use serde_json;
-
-use protocols::oci::State as OCIState;
-
-use std::collections::HashMap;
-use std::fmt;
-use std::path::PathBuf;
-use std::time::Duration;
-
-use nix::unistd;
-
-use self::device::{Device, ThrottleDevice, WeightDevice};
-use self::namespaces::Namespaces;
-use crate::specconv::CreateOpts;
-
-pub mod device;
-pub mod namespaces;
-pub mod validator;
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Rlimit {
-    #[serde(default)]
-    r#type: i32,
-    #[serde(default)]
-    hard: i32,
-    #[serde(default)]
-    soft: i32,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct IDMap {
-    #[serde(default)]
-    container_id: i32,
-    #[serde(default)]
-    host_id: i32,
-    #[serde(default)]
-    size: i32,
-}
-
-type Action = i32;
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Seccomp {
-    #[serde(default)]
-    default_action: Action,
-    #[serde(default)]
-    architectures: Vec<String>,
-    #[serde(default)]
-    flags: Vec<String>,
-    #[serde(default)]
-    syscalls: Vec<Syscall>,
-}
-
-type Operator = i32;
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Arg {
-    #[serde(default)]
-    index: u32,
-    #[serde(default)]
-    value: u64,
-    #[serde(default)]
-    value_two: u64,
-    #[serde(default)]
-    op: Operator,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Syscall {
-    #[serde(default, skip_serializing_if = "String::is_empty")]
-    names: String,
-    #[serde(default)]
-    action: Action,
-    #[serde(default, rename = "errnoRet")]
-    errno_ret: u32,
-    #[serde(default, skip_serializing_if = "Vec::is_empty")]
-    args: Vec<Arg>,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Config<'a> {
-    #[serde(default)]
-    no_pivot_root: bool,
-    #[serde(default)]
-    parent_death_signal: i32,
-    #[serde(default)]
-    rootfs: String,
-    #[serde(default)]
-    readonlyfs: bool,
-    #[serde(default, rename = "rootPropagation")]
-    root_propagation: i32,
-    #[serde(default)]
-    mounts: Vec<Mount>,
-    #[serde(default)]
-    devices: Vec<Device>,
-    #[serde(default)]
-    mount_label: String,
-    #[serde(default)]
-    hostname: String,
-    #[serde(default)]
-    namespaces: Namespaces,
-    #[serde(default)]
-    capabilities: Option<Capabilities>,
-    #[serde(default)]
-    networks: Vec<Network>,
-    #[serde(default)]
-    routes: Vec<Route>,
-    #[serde(default)]
-    cgroups: Option<Cgroup<'a>>,
-    #[serde(default, skip_serializing_if = "String::is_empty")]
-    apparmor_profile: String,
-    #[serde(default, skip_serializing_if = "String::is_empty")]
-    process_label: String,
-    #[serde(default, skip_serializing_if = "Vec::is_empty")]
-    rlimits: Vec<Rlimit>,
-    #[serde(default)]
-    oom_score_adj: Option<i32>,
-    #[serde(default)]
-    uid_mappings: Vec<IDMap>,
-    #[serde(default)]
-    gid_mappings: Vec<IDMap>,
-    #[serde(default)]
-    mask_paths: Vec<String>,
-    #[serde(default)]
-    readonly_paths: Vec<String>,
-    #[serde(default)]
-    sysctl: HashMap<String, String>,
-    #[serde(default)]
-    seccomp: Option<Seccomp>,
-    #[serde(default)]
-    no_new_privileges: bool,
-    hooks: Option<Hooks>,
-    #[serde(default)]
-    version: String,
-    #[serde(default)]
-    labels: Vec<String>,
-    #[serde(default)]
-    no_new_keyring: bool,
-    #[serde(default)]
-    intel_rdt: Option<IntelRdt>,
-    #[serde(default)]
-    rootless_euid: bool,
-    #[serde(default)]
-    rootless_cgroups: bool,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Hooks {
-    prestart: Vec<Box<Hook>>,
-    poststart: Vec<Box<Hook>>,
-    poststop: Vec<Box<Hook>>,
-}
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Capabilities {
-    bounding: Vec<String>,
-    effective: Vec<String>,
-    inheritable: Vec<String>,
-    permitted: Vec<String>,
-    ambient: Vec<String>,
-}
-
-pub trait Hook {
-    fn run(&self, state: &OCIState) -> Result<()>;
-}
-
-pub struct FuncHook {
-    // run: fn(&OCIState) -> Result<()>,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Command {
-    #[serde(default)]
-    path: String,
-    #[serde(default)]
-    args: Vec<String>,
-    #[serde(default)]
-    env: Vec<String>,
-    #[serde(default)]
-    dir: String,
-    #[serde(default)]
-    timeout: Duration,
-}
-
-pub struct CommandHook {
-    command: Command,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Mount {
-    #[serde(default)]
-    source: String,
-    #[serde(default)]
-    destination: String,
-    #[serde(default)]
-    device: String,
-    #[serde(default)]
-    flags: i32,
-    #[serde(default)]
-    propagation_flags: Vec<i32>,
-    #[serde(default)]
-    data: String,
-    #[serde(default)]
-    relabel: String,
-    #[serde(default)]
-    extensions: i32,
-    #[serde(default)]
-    premount_cmds: Vec<Command>,
-    #[serde(default)]
-    postmount_cmds: Vec<Command>,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct HugepageLimit {
-    #[serde(default)]
-    page_size: String,
-    #[serde(default)]
-    limit: u64,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct IntelRdt {
-    #[serde(default, skip_serializing_if = "String::is_empty")]
-    l3_cache_schema: String,
-    #[serde(
-        default,
-        rename = "memBwSchema",
-        skip_serializing_if = "String::is_empty"
-    )]
-    mem_bw_schema: String,
-}
-
-pub type FreezerState = String;
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Cgroup<'a> {
-    #[serde(default, skip_serializing_if = "String::is_empty")]
-    name: String,
-    #[serde(default, skip_serializing_if = "String::is_empty")]
-    parent: String,
-    #[serde(default)]
-    path: String,
-    #[serde(default)]
-    scope_prefix: String,
-    paths: HashMap<String, String>,
-    resource: &'a Resources<'a>,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Resources<'a> {
-    #[serde(default)]
-    allow_all_devices: bool,
-    #[serde(default, skip_serializing_if = "Vec::is_empty")]
-    allowed_devices: Vec<&'a Device>,
-    #[serde(default, skip_serializing_if = "Vec::is_empty")]
-    denied_devices: Vec<&'a Device>,
-    #[serde(default)]
-    devices: Vec<&'a Device>,
-    #[serde(default)]
-    memory: i64,
-    #[serde(default)]
-    memory_reservation: i64,
-    #[serde(default)]
-    memory_swap: i64,
-    #[serde(default)]
-    kernel_memory: i64,
-    #[serde(default)]
-    kernel_memory_tcp: i64,
-    #[serde(default)]
-    cpu_shares: u64,
-    #[serde(default)]
-    cpu_quota: i64,
-    #[serde(default)]
-    cpu_period: u64,
-    #[serde(default)]
-    cpu_rt_quota: i64,
-    #[serde(default)]
-    cpu_rt_period: u64,
-    #[serde(default)]
-    cpuset_cpus: String,
-    #[serde(default)]
-    cpuset_mems: String,
-    #[serde(default)]
-    pids_limit: i64,
-    #[serde(default)]
-    blkio_weight: u64,
-    #[serde(default)]
-    blkio_leaf_weight: u64,
-    #[serde(default)]
-    blkio_weight_device: Vec<&'a WeightDevice>,
-    #[serde(default)]
-    blkio_throttle_read_bps_device: Vec<&'a ThrottleDevice>,
-    #[serde(default)]
-    blkio_throttle_write_bps_device: Vec<&'a ThrottleDevice>,
-    #[serde(default)]
-    blkio_throttle_read_iops_device: Vec<&'a ThrottleDevice>,
-    #[serde(default)]
-    blkio_throttle_write_iops_device: Vec<&'a ThrottleDevice>,
-    #[serde(default)]
-    freezer: FreezerState,
-    #[serde(default)]
-    hugetlb_limit: Vec<&'a HugepageLimit>,
-    #[serde(default)]
-    oom_kill_disable: bool,
-    #[serde(default)]
-    memory_swapiness: u64,
-    #[serde(default)]
-    net_prio_ifpriomap: Vec<&'a IfPrioMap>,
-    #[serde(default)]
-    net_cls_classid_u: u32,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Network {
-    #[serde(default)]
-    r#type: String,
-    #[serde(default)]
-    name: String,
-    #[serde(default)]
-    bridge: String,
-    #[serde(default)]
-    mac_address: String,
-    #[serde(default)]
-    address: String,
-    #[serde(default)]
-    gateway: String,
-    #[serde(default)]
-    ipv6_address: String,
-    #[serde(default)]
-    ipv6_gateway: String,
-    #[serde(default)]
-    mtu: i32,
-    #[serde(default)]
-    txqueuelen: i32,
-    #[serde(default)]
-    host_interface_name: String,
-    #[serde(default)]
-    hairpin_mode: bool,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Route {
-    #[serde(default)]
-    destination: String,
-    #[serde(default)]
-    source: String,
-    #[serde(default)]
-    gateway: String,
-    #[serde(default)]
-    interface_name: String,
-}
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct IfPrioMap {
-    #[serde(default)]
-    interface: String,
-    #[serde(default)]
-    priority: i32,
-}
-
-impl IfPrioMap {
-    fn cgroup_string(&self) -> String {
-        format!("{} {}", self.interface, self.priority)
-    }
-}

src/agent/rustjail/src/configs/namespaces.rs

@@ -1,46 +0,0 @@
-// Copyright (c) 2019 Ant Financial
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-
-use serde;
-#[macro_use]
-use serde_derive;
-use serde_json;
-
-use std::collections::HashMap;
-#[macro_use]
-use lazy_static;
-
-pub type NamespaceType = String;
-pub type Namespaces = Vec<Namespace>;
-
-#[derive(Serialize, Deserialize, Debug)]
-pub struct Namespace {
-    #[serde(default)]
-    r#type: NamespaceType,
-    #[serde(default)]
-    path: String,
-}
-
-pub const NEWNET: &'static str = "NEWNET";
-pub const NEWPID: &'static str = "NEWPID";
-pub const NEWNS: &'static str = "NEWNS";
-pub const NEWUTS: &'static str = "NEWUTS";
-pub const NEWUSER: &'static str = "NEWUSER";
-pub const NEWCGROUP: &'static str = "NEWCGROUP";
-pub const NEWIPC: &'static str = "NEWIPC";
-
-lazy_static! {
-    static ref TYPETONAME: HashMap<&'static str, &'static str> = {
-        let mut m = HashMap::new();
-        m.insert("pid", "pid");
-        m.insert("network", "net");
-        m.insert("mount", "mnt");
-        m.insert("user", "user");
-        m.insert("uts", "uts");
-        m.insert("ipc", "ipc");
-        m.insert("cgroup", "cgroup");
-        m
-    };
-}

src/agent/rustjail/src/configs/validator.rs

@@ -1,23 +0,0 @@
-// Copyright (c) 2019 Ant Financial
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-
-use crate::configs::Config;
-use std::io::Result;
-
-pub trait Validator {
-    fn validate(&self, config: &Config) -> Result<()> {
-        Ok(())
-    }
-}
-
-pub struct ConfigValidator {}
-
-impl Validator for ConfigValidator {}
-
-impl ConfigValidator {
-    fn new() -> Self {
-        ConfigValidator {}
-    }
-}

src/agent/rustjail/src/container.rs

@@ -8,7 +8,7 @@ use libc::pid_t;
 use oci::{ContainerState, LinuxDevice, LinuxIdMapping};
 use oci::{Hook, Linux, LinuxNamespace, LinuxResources, Spec};
 use std::clone::Clone;
-use std::ffi::{CStr, CString};
+use std::ffi::CString;
 use std::fmt::Display;
 use std::fs;
 use std::os::unix::io::RawFd;
@@ -346,7 +346,7 @@ fn do_init_child(cwfd: RawFd) -> Result<()> {
         Err(_e) => sched::unshare(CloneFlags::CLONE_NEWPID)?,
     }
 
-    match fork() {
+    match unsafe { fork() } {
         Ok(ForkResult::Parent { child, .. }) => {
             log_child!(
                 cfd_log,
@@ -469,7 +469,7 @@ fn do_init_child(cwfd: RawFd) -> Result<()> {
     // Ref: https://github.com/opencontainers/runc/commit/50a19c6ff828c58e5dab13830bd3dacde268afe5
     //
     if !nses.is_empty() {
-        prctl::set_dumpable(false)
+        capctl::prctl::set_dumpable(false)
             .map_err(|e| anyhow!(e).context("set process non-dumpable failed"))?;
     }
 
@@ -545,7 +545,7 @@ fn do_init_child(cwfd: RawFd) -> Result<()> {
         // notify parent to run prestart hooks
         write_sync(cwfd, SYNC_SUCCESS, "")?;
         // wait parent run prestart hooks
-        let _ = read_sync(crfd)?;
+        read_sync(crfd)?;
     }
 
     if mount_fd != -1 {
@@ -602,7 +602,7 @@ fn do_init_child(cwfd: RawFd) -> Result<()> {
 
     // NoNewPeiviledges, Drop capabilities
     if oci_process.no_new_privileges {
-        prctl::set_no_new_privileges(true).map_err(|_| anyhow!("cannot set no new privileges"))?;
+        capctl::prctl::set_no_new_privs().map_err(|_| anyhow!("cannot set no new privileges"))?;
     }
 
     if oci_process.capabilities.is_some() {
@@ -612,8 +612,6 @@ fn do_init_child(cwfd: RawFd) -> Result<()> {
 
     if init {
         // notify parent to run poststart hooks
-        // cfd is closed when return from join_namespaces
-        // should retunr cfile instead of cfd?
         write_sync(cwfd, SYNC_SUCCESS, "")?;
     }
 
@@ -822,7 +820,7 @@ impl BaseContainer for LinuxContainer {
             if stat::stat(fifo_file.as_str()).is_ok() {
                 return Err(anyhow!("exec fifo exists"));
             }
-            unistd::mkfifo(fifo_file.as_str(), Mode::from_bits(0o622).unwrap())?;
+            unistd::mkfifo(fifo_file.as_str(), Mode::from_bits(0o644).unwrap())?;
 
             fifofd = fcntl::open(
                 fifo_file.as_str(),
@@ -1081,9 +1079,8 @@ fn do_exec(args: &[String]) -> ! {
         .iter()
         .map(|s| CString::new(s.to_string()).unwrap_or_default())
         .collect();
-    let a: Vec<&CStr> = sa.iter().map(|s| s.as_c_str()).collect();
 
-    let _ = unistd::execvp(p.as_c_str(), a.as_slice()).map_err(|e| match e {
+    let _ = unistd::execvp(p.as_c_str(), &sa).map_err(|e| match e {
         nix::Error::Sys(errno) => {
             std::process::exit(errno as i32);
         }
@@ -1256,7 +1253,7 @@ async fn join_namespaces(
 
     if p.init {
         info!(logger, "notify child parent ready to run prestart hook!");
-        let _ = read_async(pipe_r).await?;
+        read_async(pipe_r).await?;
 
         info!(logger, "get ready to run prestart hook!");
 
@@ -1316,7 +1313,7 @@ fn write_mappings(logger: &Logger, path: &str, maps: &[LinuxIdMapping]) -> Resul
 
 fn setid(uid: Uid, gid: Gid) -> Result<()> {
     // set uid/gid
-    prctl::set_keep_capabilities(true)
+    capctl::prctl::set_keepcaps(true)
         .map_err(|e| anyhow!(e).context("set keep capabilities returned"))?;
 
     {
@@ -1330,7 +1327,7 @@ fn setid(uid: Uid, gid: Gid) -> Result<()> {
         capabilities::reset_effective()?;
     }
 
-    prctl::set_keep_capabilities(false)
+    capctl::prctl::set_keepcaps(false)
         .map_err(|e| anyhow!(e).context("set keep capabilities returned"))?;
 
     Ok(())

src/agent/rustjail/src/lib.rs

@@ -23,7 +23,7 @@ extern crate caps;
 extern crate protocols;
 #[macro_use]
 extern crate scopeguard;
-extern crate prctl;
+extern crate capctl;
 #[macro_use]
 extern crate lazy_static;
 extern crate libc;
@@ -47,16 +47,6 @@ pub mod sync;
 pub mod sync_with_async;
 pub mod utils;
 pub mod validator;
-// pub mod factory;
-//pub mod configs;
-// pub mod devices;
-// pub mod init;
-// pub mod rootfs;
-// pub mod capabilities;
-// pub mod console;
-// pub mod stats;
-// pub mod user;
-//pub mod intelrdt;
 
 use std::collections::HashMap;
 

src/agent/rustjail/src/validator.rs

@@ -28,16 +28,6 @@ fn contain_namespace(nses: &[LinuxNamespace], key: &str) -> bool {
     false
 }
 
-fn get_namespace_path(nses: &[LinuxNamespace], key: &str) -> Result<String> {
-    for ns in nses {
-        if ns.r#type.as_str() == key {
-            return Ok(ns.path.clone());
-        }
-    }
-
-    Err(einval())
-}
-
 fn rootfs(root: &str) -> Result<()> {
     let path = PathBuf::from(root);
     // not absolute path or not exists
@@ -166,31 +156,6 @@ lazy_static! {
     };
 }
 
-fn check_host_ns(path: &str) -> Result<()> {
-    let cpath = PathBuf::from(path);
-    let hpath = PathBuf::from("/proc/self/ns/net");
-
-    let real_hpath = hpath
-        .read_link()
-        .context(format!("read link {:?}", hpath))?;
-    let meta = cpath
-        .symlink_metadata()
-        .context(format!("symlink metadata {:?}", cpath))?;
-    let file_type = meta.file_type();
-
-    if !file_type.is_symlink() {
-        return Ok(());
-    }
-    let real_cpath = cpath
-        .read_link()
-        .context(format!("read link {:?}", cpath))?;
-    if real_cpath == real_hpath {
-        return Err(einval());
-    }
-
-    Ok(())
-}
-
 fn sysctl(oci: &Spec) -> Result<()> {
     let linux = get_linux(oci)?;
 
@@ -334,19 +299,6 @@ mod tests {
         assert_eq!(contain_namespace(&namespaces, ""), false);
         assert_eq!(contain_namespace(&namespaces, "Net"), false);
         assert_eq!(contain_namespace(&namespaces, "ipc"), false);
-
-        assert_eq!(
-            get_namespace_path(&namespaces, "net").unwrap(),
-            "/sys/cgroups/net"
-        );
-        assert_eq!(
-            get_namespace_path(&namespaces, "uts").unwrap(),
-            "/sys/cgroups/uts"
-        );
-
-        get_namespace_path(&namespaces, "").unwrap_err();
-        get_namespace_path(&namespaces, "Uts").unwrap_err();
-        get_namespace_path(&namespaces, "ipc").unwrap_err();
     }
 
     #[test]
@@ -528,12 +480,6 @@ mod tests {
         rootless_euid(&spec).unwrap();
     }
 
-    #[test]
-    fn test_check_host_ns() {
-        check_host_ns("/proc/self/ns/net").unwrap_err();
-        check_host_ns("/proc/sys/net/ipv4/tcp_sack").unwrap();
-    }
-
     #[test]
     fn test_sysctl() {
         let mut spec = Spec::default();

src/agent/src/config.rs

@@ -2,13 +2,16 @@
 //
 // SPDX-License-Identifier: Apache-2.0
 //
+use crate::tracer;
 use anyhow::{anyhow, Result};
 use std::env;
 use std::fs;
 use std::time;
+use tracing::instrument;
 
 const DEBUG_CONSOLE_FLAG: &str = "agent.debug_console";
 const DEV_MODE_FLAG: &str = "agent.devmode";
+const TRACE_MODE_OPTION: &str = "agent.trace";
 const LOG_LEVEL_OPTION: &str = "agent.log";
 const SERVER_ADDR_OPTION: &str = "agent.server_addr";
 const HOTPLUG_TIMOUT_OPTION: &str = "agent.hotplug_timeout";
@@ -26,6 +29,7 @@ const VSOCK_PORT: u16 = 1024;
 // Environment variables used for development and testing
 const SERVER_ADDR_ENV_VAR: &str = "KATA_AGENT_SERVER_ADDR";
 const LOG_LEVEL_ENV_VAR: &str = "KATA_AGENT_LOG_LEVEL";
+const TRACE_TYPE_ENV_VAR: &str = "KATA_AGENT_TRACE_TYPE";
 
 const ERR_INVALID_LOG_LEVEL: &str = "invalid log level";
 const ERR_INVALID_LOG_LEVEL_PARAM: &str = "invalid log level parameter";
@@ -54,6 +58,7 @@ pub struct AgentConfig {
     pub container_pipe_size: i32,
     pub server_addr: String,
     pub unified_cgroup_hierarchy: bool,
+    pub tracing: tracer::TraceType,
 }
 
 // parse_cmdline_param parse commandline parameters.
@@ -98,9 +103,11 @@ impl AgentConfig {
             container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
             server_addr: format!("{}:{}", VSOCK_ADDR, VSOCK_PORT),
             unified_cgroup_hierarchy: false,
+            tracing: tracer::TraceType::Disabled,
         }
     }
 
+    #[instrument]
     pub fn parse_cmdline(&mut self, file: &str) -> Result<()> {
         let cmdline = fs::read_to_string(file)?;
         let params: Vec<&str> = cmdline.split_ascii_whitespace().collect();
@@ -109,6 +116,15 @@ impl AgentConfig {
             parse_cmdline_param!(param, DEBUG_CONSOLE_FLAG, self.debug_console);
             parse_cmdline_param!(param, DEV_MODE_FLAG, self.dev_mode);
 
+            // Support "bare" tracing option for backwards compatibility with
+            // Kata 1.x.
+            if param == &TRACE_MODE_OPTION {
+                self.tracing = tracer::TraceType::Isolated;
+                continue;
+            }
+
+            parse_cmdline_param!(param, TRACE_MODE_OPTION, self.tracing, get_trace_type);
+
             // parse cmdline options
             parse_cmdline_param!(param, LOG_LEVEL_OPTION, self.log_level, get_log_level);
             parse_cmdline_param!(
@@ -167,10 +183,17 @@ impl AgentConfig {
             }
         }
 
+        if let Ok(value) = env::var(TRACE_TYPE_ENV_VAR) {
+            if let Ok(result) = value.parse::<tracer::TraceType>() {
+                self.tracing = result;
+            }
+        }
+
         Ok(())
     }
 }
 
+#[instrument]
 fn get_vsock_port(p: &str) -> Result<i32> {
     let fields: Vec<&str> = p.split('=').collect();
     if fields.len() != 2 {
@@ -185,6 +208,7 @@ fn get_vsock_port(p: &str) -> Result<i32> {
 //
 // Note: Logrus names are used for compatability with the previous
 // golang-based agent.
+#[instrument]
 fn logrus_to_slog_level(logrus_level: &str) -> Result<slog::Level> {
     let level = match logrus_level {
         // Note: different semantics to logrus: log, but don't panic.
@@ -207,6 +231,7 @@ fn logrus_to_slog_level(logrus_level: &str) -> Result<slog::Level> {
     Ok(level)
 }
 
+#[instrument]
 fn get_log_level(param: &str) -> Result<slog::Level> {
     let fields: Vec<&str> = param.split('=').collect();
 
@@ -221,6 +246,28 @@ fn get_log_level(param: &str) -> Result<slog::Level> {
     }
 }
 
+#[instrument]
+fn get_trace_type(param: &str) -> Result<tracer::TraceType> {
+    if param.is_empty() {
+        return Err(anyhow!("invalid trace type parameter"));
+    }
+
+    let fields: Vec<&str> = param.split('=').collect();
+
+    if fields[0] != TRACE_MODE_OPTION {
+        return Err(anyhow!("invalid trace type key name"));
+    }
+
+    if fields.len() == 1 {
+        return Ok(tracer::TraceType::Isolated);
+    }
+
+    let result = fields[1].parse::<tracer::TraceType>()?;
+
+    Ok(result)
+}
+
+#[instrument]
 fn get_hotplug_timeout(param: &str) -> Result<time::Duration> {
     let fields: Vec<&str> = param.split('=').collect();
 
@@ -241,6 +288,7 @@ fn get_hotplug_timeout(param: &str) -> Result<time::Duration> {
     Ok(time::Duration::from_secs(value.unwrap()))
 }
 
+#[instrument]
 fn get_bool_value(param: &str) -> Result<bool> {
     let fields: Vec<&str> = param.split('=').collect();
 
@@ -265,6 +313,7 @@ fn get_bool_value(param: &str) -> Result<bool> {
 // - A value can contain any number of equal signs.
 // - We could/should maybe check if the name is pure whitespace
 //   since this is considered to be invalid.
+#[instrument]
 fn get_string_value(param: &str) -> Result<String> {
     let fields: Vec<&str> = param.split('=').collect();
 
@@ -285,6 +334,7 @@ fn get_string_value(param: &str) -> Result<String> {
     Ok(value)
 }
 
+#[instrument]
 fn get_container_pipe_size(param: &str) -> Result<i32> {
     let fields: Vec<&str> = param.split('=').collect();
 
@@ -319,6 +369,10 @@ mod tests {
     use std::time;
     use tempfile::tempdir;
 
+    const ERR_INVALID_TRACE_TYPE_PARAM: &str = "invalid trace type parameter";
+    const ERR_INVALID_TRACE_TYPE: &str = "invalid trace type";
+    const ERR_INVALID_TRACE_TYPE_KEY: &str = "invalid trace type key name";
+
     // helper function to make errors less crazy-long
     fn make_err(desc: &str) -> Error {
         anyhow!(desc.to_string())
@@ -374,6 +428,7 @@ mod tests {
             container_pipe_size: i32,
             server_addr: &'a str,
             unified_cgroup_hierarchy: bool,
+            tracing: tracer::TraceType,
         }
 
         let tests = &[
@@ -387,6 +442,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.debug_console agent.devmodex",
@@ -398,6 +454,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.logx=debug",
@@ -409,6 +466,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.log=debug",
@@ -420,6 +478,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.log=debug",
@@ -431,6 +490,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -442,6 +502,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo",
@@ -453,6 +514,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo bar",
@@ -464,6 +526,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo bar",
@@ -475,6 +538,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo agent bar",
@@ -486,6 +550,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo debug_console agent bar devmode",
@@ -497,6 +562,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.debug_console",
@@ -508,6 +574,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "   agent.debug_console ",
@@ -519,6 +586,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.debug_console foo",
@@ -530,6 +598,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: " agent.debug_console foo",
@@ -541,6 +610,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo agent.debug_console bar",
@@ -552,6 +622,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo agent.debug_console",
@@ -563,6 +634,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo agent.debug_console ",
@@ -574,6 +646,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode",
@@ -585,6 +658,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "   agent.devmode ",
@@ -596,6 +670,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode foo",
@@ -607,6 +682,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: " agent.devmode foo",
@@ -618,6 +694,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo agent.devmode bar",
@@ -629,6 +706,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo agent.devmode",
@@ -640,6 +718,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "foo agent.devmode ",
@@ -651,6 +730,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode agent.debug_console",
@@ -662,6 +742,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode agent.debug_console agent.hotplug_timeout=100 agent.unified_cgroup_hierarchy=a",
@@ -673,6 +754,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode agent.debug_console agent.hotplug_timeout=0 agent.unified_cgroup_hierarchy=11",
@@ -684,6 +766,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: true,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode agent.debug_console agent.container_pipe_size=2097152 agent.unified_cgroup_hierarchy=false",
@@ -695,6 +778,7 @@ mod tests {
                 container_pipe_size: 2097152,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode agent.debug_console agent.container_pipe_size=100 agent.unified_cgroup_hierarchy=true",
@@ -706,6 +790,7 @@ mod tests {
                 container_pipe_size: 100,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: true,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode agent.debug_console agent.container_pipe_size=0 agent.unified_cgroup_hierarchy=0",
@@ -717,6 +802,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.devmode agent.debug_console agent.container_pip_siz=100 agent.unified_cgroup_hierarchy=1",
@@ -728,6 +814,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: true,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -739,6 +826,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -750,6 +838,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "foo",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -761,6 +850,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "=",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -772,6 +862,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "=foo",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -783,6 +874,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "foo=bar=baz=",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -794,6 +886,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "unix:///tmp/foo.socket",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -805,6 +898,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "unix://@/tmp/foo.socket",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -816,6 +910,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -827,6 +922,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -838,6 +934,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "",
@@ -849,6 +946,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "server_addr=unix:///tmp/foo.socket",
@@ -860,6 +958,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.server_address=unix:///tmp/foo.socket",
@@ -871,6 +970,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: TEST_SERVER_ADDR,
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: "agent.server_addr=unix:///tmp/foo.socket",
@@ -882,6 +982,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "unix:///tmp/foo.socket",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: " agent.server_addr=unix:///tmp/foo.socket",
@@ -893,6 +994,7 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "unix:///tmp/foo.socket",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
             TestData {
                 contents: " agent.server_addr=unix:///tmp/foo.socket a",
@@ -904,6 +1006,115 @@ mod tests {
                 container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
                 server_addr: "unix:///tmp/foo.socket",
                 unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
+            },
+            TestData {
+                contents: "trace",
+                env_vars: Vec::new(),
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
+            },
+            TestData {
+                contents: ".trace",
+                env_vars: Vec::new(),
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
+            },
+            TestData {
+                contents: "agent.tracer",
+                env_vars: Vec::new(),
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
+            },
+            TestData {
+                contents: "agent.trac",
+                env_vars: Vec::new(),
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
+            },
+            TestData {
+                contents: "agent.trace",
+                env_vars: Vec::new(),
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Isolated,
+            },
+            TestData {
+                contents: "agent.trace=isolated",
+                env_vars: Vec::new(),
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Isolated,
+            },
+            TestData {
+                contents: "agent.trace=disabled",
+                env_vars: Vec::new(),
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
+            },
+            TestData {
+                contents: "",
+                env_vars: vec!["KATA_AGENT_TRACE_TYPE=isolated"],
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Isolated,
+            },
+            TestData {
+                contents: "",
+                env_vars: vec!["KATA_AGENT_TRACE_TYPE=disabled"],
+                debug_console: false,
+                dev_mode: false,
+                log_level: DEFAULT_LOG_LEVEL,
+                hotplug_timeout: DEFAULT_HOTPLUG_TIMEOUT,
+                container_pipe_size: DEFAULT_CONTAINER_PIPE_SIZE,
+                server_addr: TEST_SERVER_ADDR,
+                unified_cgroup_hierarchy: false,
+                tracing: tracer::TraceType::Disabled,
             },
         ];
 
@@ -958,6 +1169,7 @@ mod tests {
             );
             assert_eq!(config.container_pipe_size, 0, "{}", msg);
             assert_eq!(config.server_addr, TEST_SERVER_ADDR, "{}", msg);
+            assert_eq!(config.tracing, tracer::TraceType::Disabled, "{}", msg);
 
             let result = config.parse_cmdline(filename);
             assert!(result.is_ok(), "{}", msg);
@@ -973,6 +1185,7 @@ mod tests {
             assert_eq!(d.hotplug_timeout, config.hotplug_timeout, "{}", msg);
             assert_eq!(d.container_pipe_size, config.container_pipe_size, "{}", msg);
             assert_eq!(d.server_addr, config.server_addr, "{}", msg);
+            assert_eq!(d.tracing, config.tracing, "{}", msg);
 
             for v in vars_to_unset {
                 env::remove_var(v);
@@ -1369,4 +1582,62 @@ mod tests {
             assert_result!(d.result, result, msg);
         }
     }
+
+    #[test]
+    fn test_get_trace_type() {
+        #[derive(Debug)]
+        struct TestData<'a> {
+            param: &'a str,
+            result: Result<tracer::TraceType>,
+        }
+
+        let tests = &[
+            TestData {
+                param: "",
+                result: Err(make_err(ERR_INVALID_TRACE_TYPE_PARAM)),
+            },
+            TestData {
+                param: "agent.tracer",
+                result: Err(make_err(ERR_INVALID_TRACE_TYPE_KEY)),
+            },
+            TestData {
+                param: "agent.trac",
+                result: Err(make_err(ERR_INVALID_TRACE_TYPE_KEY)),
+            },
+            TestData {
+                param: "agent.trace=",
+                result: Err(make_err(ERR_INVALID_TRACE_TYPE)),
+            },
+            TestData {
+                param: "agent.trace==",
+                result: Err(make_err(ERR_INVALID_TRACE_TYPE)),
+            },
+            TestData {
+                param: "agent.trace=foo",
+                result: Err(make_err(ERR_INVALID_TRACE_TYPE)),
+            },
+            TestData {
+                param: "agent.trace",
+                result: Ok(tracer::TraceType::Isolated),
+            },
+            TestData {
+                param: "agent.trace=isolated",
+                result: Ok(tracer::TraceType::Isolated),
+            },
+            TestData {
+                param: "agent.trace=disabled",
+                result: Ok(tracer::TraceType::Disabled),
+            },
+        ];
+
+        for (i, d) in tests.iter().enumerate() {
+            let msg = format!("test[{}]: {:?}", i, d);
+
+            let result = get_trace_type(d.param);
+
+            let msg = format!("{}: result: {:?}", msg, result);
+
+            assert_result!(d.result, result, msg);
+        }
+    }
 }

src/agent/src/console.rs

@@ -145,9 +145,10 @@ fn run_in_child(slave_fd: libc::c_int, shell: String) -> Result<()> {
     }
 
     let cmd = CString::new(shell).unwrap();
+    let args: Vec<CString> = Vec::new();
 
     // run shell
-    let _ = unistd::execvp(cmd.as_c_str(), &[]).map_err(|e| match e {
+    let _ = unistd::execvp(cmd.as_c_str(), &args).map_err(|e| match e {
         nix::Error::Sys(errno) => {
             std::process::exit(errno as i32);
         }
@@ -205,7 +206,7 @@ async fn run_debug_console_vsock<T: AsyncRead + AsyncWrite>(
 
     let slave_fd = pseudo.slave;
 
-    match fork() {
+    match unsafe { fork() } {
         Ok(ForkResult::Child) => run_in_child(slave_fd, shell),
         Ok(ForkResult::Parent { child: child_pid }) => {
             run_in_parent(logger.clone(), stream, pseudo, child_pid).await

src/agent/src/device.rs

@@ -22,6 +22,7 @@ use crate::uevent::{wait_for_uevent, Uevent, UeventMatcher};
 use anyhow::{anyhow, Result};
 use oci::{LinuxDeviceCgroup, LinuxResources, Spec};
 use protocols::agent::Device;
+use tracing::instrument;
 
 // Convenience macro to obtain the scope logger
 macro_rules! sl {
@@ -32,17 +33,21 @@ macro_rules! sl {
 
 const VM_ROOTFS: &str = "/";
 
+#[derive(Debug)]
 struct DevIndexEntry {
     idx: usize,
     residx: Vec<usize>,
 }
 
+#[derive(Debug)]
 struct DevIndex(HashMap<String, DevIndexEntry>);
 
+#[instrument]
 pub fn rescan_pci_bus() -> Result<()> {
     online_device(SYSFS_PCI_BUS_RESCAN_FILE)
 }
 
+#[instrument]
 pub fn online_device(path: &str) -> Result<()> {
     fs::write(path, "1")?;
     Ok(())
@@ -51,6 +56,7 @@ pub fn online_device(path: &str) -> Result<()> {
 // pcipath_to_sysfs fetches the sysfs path for a PCI path, relative to
 // the sysfs path for the PCI host bridge, based on the PCI path
 // provided.
+#[instrument]
 fn pcipath_to_sysfs(root_bus_sysfs: &str, pcipath: &pci::Path) -> Result<String> {
     let mut bus = "0000:00".to_string();
     let mut relpath = String::new();
@@ -109,6 +115,7 @@ impl UeventMatcher for ScsiBlockMatcher {
     }
 }
 
+#[instrument]
 pub async fn get_scsi_device_name(
     sandbox: &Arc<Mutex<Sandbox>>,
     scsi_addr: &str,
@@ -141,6 +148,7 @@ impl UeventMatcher for VirtioBlkPciMatcher {
     }
 }
 
+#[instrument]
 pub async fn get_virtio_blk_pci_device_name(
     sandbox: &Arc<Mutex<Sandbox>>,
     pcipath: &pci::Path,
@@ -177,6 +185,7 @@ impl UeventMatcher for PmemBlockMatcher {
     }
 }
 
+#[instrument]
 pub async fn wait_for_pmem_device(sandbox: &Arc<Mutex<Sandbox>>, devpath: &str) -> Result<()> {
     let devname = match devpath.strip_prefix("/dev/") {
         Some(dev) => dev,
@@ -201,6 +210,7 @@ pub async fn wait_for_pmem_device(sandbox: &Arc<Mutex<Sandbox>>, devpath: &str)
 }
 
 /// Scan SCSI bus for the given SCSI address(SCSI-Id and LUN)
+#[instrument]
 fn scan_scsi_bus(scsi_addr: &str) -> Result<()> {
     let tokens: Vec<&str> = scsi_addr.split(':').collect();
     if tokens.len() != 2 {
@@ -235,6 +245,7 @@ fn scan_scsi_bus(scsi_addr: &str) -> Result<()> {
 // the same device in the list of devices provided through the OCI spec.
 // This is needed to update information about minor/major numbers that cannot
 // be predicted from the caller.
+#[instrument]
 fn update_spec_device_list(device: &Device, spec: &mut Spec, devidx: &DevIndex) -> Result<()> {
     let major_id: c_uint;
     let minor_id: c_uint;
@@ -311,6 +322,7 @@ fn update_spec_device_list(device: &Device, spec: &mut Spec, devidx: &DevIndex)
 
 // device.Id should be the predicted device name (vda, vdb, ...)
 // device.VmPath already provides a way to send it in
+#[instrument]
 async fn virtiommio_blk_device_handler(
     device: &Device,
     spec: &mut Spec,
@@ -325,6 +337,7 @@ async fn virtiommio_blk_device_handler(
 }
 
 // device.Id should be a PCI path string
+#[instrument]
 async fn virtio_blk_device_handler(
     device: &Device,
     spec: &mut Spec,
@@ -340,6 +353,7 @@ async fn virtio_blk_device_handler(
 }
 
 // device.Id should be the SCSI address of the disk in the format "scsiID:lunID"
+#[instrument]
 async fn virtio_scsi_device_handler(
     device: &Device,
     spec: &mut Spec,
@@ -351,6 +365,7 @@ async fn virtio_scsi_device_handler(
     update_spec_device_list(&dev, spec, devidx)
 }
 
+#[instrument]
 async fn virtio_nvdimm_device_handler(
     device: &Device,
     spec: &mut Spec,
@@ -389,6 +404,7 @@ impl DevIndex {
     }
 }
 
+#[instrument]
 pub async fn add_devices(
     devices: &[Device],
     spec: &mut Spec,
@@ -403,6 +419,7 @@ pub async fn add_devices(
     Ok(())
 }
 
+#[instrument]
 async fn add_device(
     device: &Device,
     spec: &mut Spec,
@@ -437,6 +454,7 @@ async fn add_device(
 // update_device_cgroup update the device cgroup for container
 // to not allow access to the guest root partition. This prevents
 // the container from being able to access the VM rootfs.
+#[instrument]
 pub fn update_device_cgroup(spec: &mut Spec) -> Result<()> {
     let meta = fs::metadata(VM_ROOTFS)?;
     let rdev = meta.dev();

src/agent/src/main.rs

@@ -5,8 +5,8 @@
 
 #[macro_use]
 extern crate lazy_static;
+extern crate capctl;
 extern crate oci;
-extern crate prctl;
 extern crate prometheus;
 extern crate protocols;
 extern crate regex;
@@ -32,6 +32,7 @@ use std::os::unix::io::AsRawFd;
 use std::path::Path;
 use std::process::exit;
 use std::sync::Arc;
+use tracing::{instrument, span};
 
 mod config;
 mod console;
@@ -55,7 +56,7 @@ mod version;
 use mount::{cgroups_mount, general_mount};
 use sandbox::Sandbox;
 use signal::setup_signal_handler;
-use slog::Logger;
+use slog::{error, info, o, warn, Logger};
 use uevent::watch_uevents;
 
 use futures::future::join_all;
@@ -70,6 +71,7 @@ use tokio::{
 };
 
 mod rpc;
+mod tracer;
 
 const NAME: &str = "kata-agent";
 const KERNEL_CMDLINE_FILE: &str = "/proc/cmdline";
@@ -79,6 +81,7 @@ lazy_static! {
         Arc::new(RwLock::new(config::AgentConfig::new()));
 }
 
+#[instrument]
 fn announce(logger: &Logger, config: &AgentConfig) {
     info!(logger, "announce";
     "agent-commit" => version::VERSION_COMMIT,
@@ -199,6 +202,17 @@ async fn real_main() -> std::result::Result<(), Box<dyn std::error::Error>> {
         ttrpc_log_guard = Ok(slog_stdlog::init().map_err(|e| e)?);
     }
 
+    if config.tracing != tracer::TraceType::Disabled {
+        let _ = tracer::setup_tracing(NAME, &logger, &config)?;
+    }
+
+    let root = span!(tracing::Level::TRACE, "root-span", work_units = 2);
+
+    // XXX: Start the root trace transaction.
+    //
+    // XXX: Note that *ALL* spans needs to start after this point!!
+    let _enter = root.enter();
+
     // Start the sandbox and wait for its ttRPC server to end
     start_sandbox(&logger, &config, init_mode, &mut tasks, shutdown_rx.clone()).await?;
 
@@ -227,6 +241,10 @@ async fn real_main() -> std::result::Result<(), Box<dyn std::error::Error>> {
         }
     }
 
+    if config.tracing != tracer::TraceType::Disabled {
+        tracer::end_tracing();
+    }
+
     eprintln!("{} shutdown complete", NAME);
 
     Ok(())
@@ -248,6 +266,7 @@ fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
     }
 
     if args.len() == 2 && args[1] == "init" {
+        reset_sigpipe();
         rustjail::container::init_child();
         exit(0);
     }
@@ -259,6 +278,7 @@ fn main() -> std::result::Result<(), Box<dyn std::error::Error>> {
     rt.block_on(real_main())
 }
 
+#[instrument]
 async fn start_sandbox(
     logger: &Logger,
     config: &AgentConfig,
@@ -305,7 +325,7 @@ async fn start_sandbox(
     let mut server = rpc::start(sandbox.clone(), config.server_addr.as_str());
     server.start().await?;
 
-    let _ = rx.await?;
+    rx.await?;
     server.shutdown().await?;
 
     Ok(())
@@ -345,6 +365,7 @@ fn init_agent_as_init(logger: &Logger, unified_cgroup_hierarchy: bool) -> Result
     Ok(())
 }
 
+#[instrument]
 fn sethostname(hostname: &OsStr) -> Result<()> {
     let size = hostname.len() as usize;
 
@@ -358,5 +379,16 @@ fn sethostname(hostname: &OsStr) -> Result<()> {
     }
 }
 
+// The Rust standard library had suppressed the default SIGPIPE behavior,
+// see https://github.com/rust-lang/rust/pull/13158.
+// Since the parent's signal handler would be inherited by it's child process,
+// thus we should re-enable the standard SIGPIPE behavior as a workaround to
+// fix the issue of https://github.com/kata-containers/kata-containers/issues/1887.
+fn reset_sigpipe() {
+    unsafe {
+        libc::signal(libc::SIGPIPE, libc::SIG_DFL);
+    }
+}
+
 use crate::config::AgentConfig;
 use std::os::unix::io::{FromRawFd, RawFd};

src/agent/src/metrics.rs

@@ -8,6 +8,7 @@ extern crate procfs;
 use prometheus::{Encoder, Gauge, GaugeVec, IntCounter, TextEncoder};
 
 use anyhow::Result;
+use tracing::instrument;
 
 const NAMESPACE_KATA_AGENT: &str = "kata_agent";
 const NAMESPACE_KATA_GUEST: &str = "kata_guest";
@@ -68,6 +69,7 @@ lazy_static! {
     prometheus::register_gauge_vec!(format!("{}_{}",NAMESPACE_KATA_GUEST,"meminfo").as_ref() , "Statistics about memory usage in the system.", &["item"]).unwrap();
 }
 
+#[instrument]
 pub fn get_metrics(_: &protocols::agent::GetMetricsRequest) -> Result<String> {
     AGENT_SCRAPE_COUNT.inc();
 
@@ -87,6 +89,7 @@ pub fn get_metrics(_: &protocols::agent::GetMetricsRequest) -> Result<String> {
     Ok(String::from_utf8(buffer).unwrap())
 }
 
+#[instrument]
 fn update_agent_metrics() {
     let me = procfs::process::Process::myself();
 
@@ -136,6 +139,7 @@ fn update_agent_metrics() {
     }
 }
 
+#[instrument]
 fn update_guest_metrics() {
     // try get load and task info
     match procfs::LoadAverage::new() {
@@ -218,6 +222,7 @@ fn update_guest_metrics() {
     }
 }
 
+#[instrument]
 fn set_gauge_vec_meminfo(gv: &prometheus::GaugeVec, meminfo: &procfs::Meminfo) {
     gv.with_label_values(&["mem_total"])
         .set(meminfo.mem_total as f64);
@@ -332,6 +337,7 @@ fn set_gauge_vec_meminfo(gv: &prometheus::GaugeVec, meminfo: &procfs::Meminfo) {
         .set(meminfo.k_reclaimable.unwrap_or(0) as f64);
 }
 
+#[instrument]
 fn set_gauge_vec_cpu_time(gv: &prometheus::GaugeVec, cpu: &str, cpu_time: &procfs::CpuTime) {
     gv.with_label_values(&[cpu, "user"])
         .set(cpu_time.user as f64);
@@ -355,6 +361,7 @@ fn set_gauge_vec_cpu_time(gv: &prometheus::GaugeVec, cpu: &str, cpu_time: &procf
         .set(cpu_time.guest_nice.unwrap_or(0.0) as f64);
 }
 
+#[instrument]
 fn set_gauge_vec_diskstat(gv: &prometheus::GaugeVec, diskstat: &procfs::DiskStat) {
     gv.with_label_values(&[diskstat.name.as_str(), "reads"])
         .set(diskstat.reads as f64);
@@ -393,6 +400,7 @@ fn set_gauge_vec_diskstat(gv: &prometheus::GaugeVec, diskstat: &procfs::DiskStat
 }
 
 // set_gauge_vec_netdev set gauge for NetDevLine
+#[instrument]
 fn set_gauge_vec_netdev(gv: &prometheus::GaugeVec, status: &procfs::net::DeviceStatus) {
     gv.with_label_values(&[status.name.as_str(), "recv_bytes"])
         .set(status.recv_bytes as f64);
@@ -429,6 +437,7 @@ fn set_gauge_vec_netdev(gv: &prometheus::GaugeVec, status: &procfs::net::DeviceS
 }
 
 // set_gauge_vec_proc_status set gauge for ProcStatus
+#[instrument]
 fn set_gauge_vec_proc_status(gv: &prometheus::GaugeVec, status: &procfs::process::Status) {
     gv.with_label_values(&["vmpeak"])
         .set(status.vmpeak.unwrap_or(0) as f64);
@@ -469,6 +478,7 @@ fn set_gauge_vec_proc_status(gv: &prometheus::GaugeVec, status: &procfs::process
 }
 
 // set_gauge_vec_proc_io set gauge for ProcIO
+#[instrument]
 fn set_gauge_vec_proc_io(gv: &prometheus::GaugeVec, io_stat: &procfs::process::Io) {
     gv.with_label_values(&["rchar"]).set(io_stat.rchar as f64);
     gv.with_label_values(&["wchar"]).set(io_stat.wchar as f64);
@@ -483,6 +493,7 @@ fn set_gauge_vec_proc_io(gv: &prometheus::GaugeVec, io_stat: &procfs::process::I
 }
 
 // set_gauge_vec_proc_stat set gauge for ProcStat
+#[instrument]
 fn set_gauge_vec_proc_stat(gv: &prometheus::GaugeVec, stat: &procfs::process::Stat) {
     gv.with_label_values(&["utime"]).set(stat.utime as f64);
     gv.with_label_values(&["stime"]).set(stat.stime as f64);

src/agent/src/mount.rs

@@ -32,6 +32,7 @@ use crate::protocols::agent::Storage;
 use crate::Sandbox;
 use anyhow::{anyhow, Context, Result};
 use slog::Logger;
+use tracing::instrument;
 
 pub const DRIVER_9P_TYPE: &str = "9p";
 pub const DRIVER_VIRTIOFS_TYPE: &str = "virtio-fs";
@@ -156,6 +157,7 @@ pub struct BareMount<'a> {
 // * evaluate all symlinks
 // * ensure the source exists
 impl<'a> BareMount<'a> {
+    #[instrument]
     pub fn new(
         s: &'a str,
         d: &'a str,
@@ -174,6 +176,7 @@ impl<'a> BareMount<'a> {
         }
     }
 
+    #[instrument]
     pub fn mount(&self) -> Result<()> {
         let source;
         let dest;
@@ -232,6 +235,7 @@ impl<'a> BareMount<'a> {
     }
 }
 
+#[instrument]
 async fn ephemeral_storage_handler(
     logger: &Logger,
     storage: &Storage,
@@ -278,6 +282,7 @@ async fn ephemeral_storage_handler(
     Ok("".to_string())
 }
 
+#[instrument]
 async fn local_storage_handler(
     _logger: &Logger,
     storage: &Storage,
@@ -324,6 +329,7 @@ async fn local_storage_handler(
     Ok("".to_string())
 }
 
+#[instrument]
 async fn virtio9p_storage_handler(
     logger: &Logger,
     storage: &Storage,
@@ -333,6 +339,7 @@ async fn virtio9p_storage_handler(
 }
 
 // virtiommio_blk_storage_handler handles the storage for mmio blk driver.
+#[instrument]
 async fn virtiommio_blk_storage_handler(
     logger: &Logger,
     storage: &Storage,
@@ -343,6 +350,7 @@ async fn virtiommio_blk_storage_handler(
 }
 
 // virtiofs_storage_handler handles the storage for virtio-fs.
+#[instrument]
 async fn virtiofs_storage_handler(
     logger: &Logger,
     storage: &Storage,
@@ -352,6 +360,7 @@ async fn virtiofs_storage_handler(
 }
 
 // virtio_blk_storage_handler handles the storage for blk driver.
+#[instrument]
 async fn virtio_blk_storage_handler(
     logger: &Logger,
     storage: &Storage,
@@ -377,7 +386,8 @@ async fn virtio_blk_storage_handler(
     common_storage_handler(logger, &storage)
 }
 
-// virtio_scsi_storage_handler handles the storage for scsi driver.
+// virtio_scsi_storage_handler handles the  storage for scsi driver.
+#[instrument]
 async fn virtio_scsi_storage_handler(
     logger: &Logger,
     storage: &Storage,
@@ -392,6 +402,7 @@ async fn virtio_scsi_storage_handler(
     common_storage_handler(logger, &storage)
 }
 
+#[instrument]
 fn common_storage_handler(logger: &Logger, storage: &Storage) -> Result<String> {
     // Mount the storage device.
     let mount_point = storage.mount_point.to_string();
@@ -400,6 +411,7 @@ fn common_storage_handler(logger: &Logger, storage: &Storage) -> Result<String>
 }
 
 // nvdimm_storage_handler handles the storage for NVDIMM driver.
+#[instrument]
 async fn nvdimm_storage_handler(
     logger: &Logger,
     storage: &Storage,
@@ -414,6 +426,7 @@ async fn nvdimm_storage_handler(
 }
 
 // mount_storage performs the mount described by the storage structure.
+#[instrument]
 fn mount_storage(logger: &Logger, storage: &Storage) -> Result<()> {
     let logger = logger.new(o!("subsystem" => "mount"));
 
@@ -464,6 +477,7 @@ fn mount_storage(logger: &Logger, storage: &Storage) -> Result<()> {
 }
 
 /// Looks for `mount_point` entry in the /proc/mounts.
+#[instrument]
 fn is_mounted(mount_point: &str) -> Result<bool> {
     let mount_point = mount_point.trim_end_matches('/');
     let found = fs::metadata(mount_point).is_ok()
@@ -481,6 +495,7 @@ fn is_mounted(mount_point: &str) -> Result<bool> {
     Ok(found)
 }
 
+#[instrument]
 fn parse_mount_flags_and_options(options_vec: Vec<&str>) -> (MsFlags, String) {
     let mut flags = MsFlags::empty();
     let mut options: String = "".to_string();
@@ -509,6 +524,7 @@ fn parse_mount_flags_and_options(options_vec: Vec<&str>) -> (MsFlags, String) {
 // associated operations such as waiting for the device to show up, and mount
 // it to a specific location, according to the type of handler chosen, and for
 // each storage.
+#[instrument]
 pub async fn add_storages(
     logger: Logger,
     storages: Vec<Storage>,
@@ -558,6 +574,7 @@ pub async fn add_storages(
     Ok(mount_list)
 }
 
+#[instrument]
 fn mount_to_rootfs(logger: &Logger, m: &InitMount) -> Result<()> {
     let options_vec: Vec<&str> = m.options.clone();
 
@@ -583,6 +600,7 @@ fn mount_to_rootfs(logger: &Logger, m: &InitMount) -> Result<()> {
     Ok(())
 }
 
+#[instrument]
 pub fn general_mount(logger: &Logger) -> Result<()> {
     let logger = logger.new(o!("subsystem" => "mount"));
 
@@ -600,6 +618,7 @@ pub fn get_mount_fs_type(mount_point: &str) -> Result<String> {
 
 // get_mount_fs_type_from_file returns the FS type corresponding to the passed mount point and
 // any error ecountered.
+#[instrument]
 pub fn get_mount_fs_type_from_file(mount_file: &str, mount_point: &str) -> Result<String> {
     if mount_point.is_empty() {
         return Err(anyhow!("Invalid mount point {}", mount_point));
@@ -630,6 +649,7 @@ pub fn get_mount_fs_type_from_file(mount_file: &str, mount_point: &str) -> Resul
     ))
 }
 
+#[instrument]
 pub fn get_cgroup_mounts(
     logger: &Logger,
     cg_path: &str,
@@ -720,6 +740,7 @@ pub fn get_cgroup_mounts(
     Ok(cg_mounts)
 }
 
+#[instrument]
 pub fn cgroups_mount(logger: &Logger, unified_cgroup_hierarchy: bool) -> Result<()> {
     let logger = logger.new(o!("subsystem" => "mount"));
 
@@ -735,6 +756,7 @@ pub fn cgroups_mount(logger: &Logger, unified_cgroup_hierarchy: bool) -> Result<
     Ok(())
 }
 
+#[instrument]
 pub fn remove_mounts(mounts: &[String]) -> Result<()> {
     for m in mounts.iter() {
         mount::umount(m.as_str()).context(format!("failed to umount {:?}", m))?;
@@ -744,6 +766,7 @@ pub fn remove_mounts(mounts: &[String]) -> Result<()> {
 
 // ensure_destination_exists will recursively create a given mountpoint. If directories
 // are created, their permissions are initialized to mountPerm(0755)
+#[instrument]
 fn ensure_destination_exists(destination: &str, fs_type: &str) -> Result<()> {
     let d = Path::new(destination);
     if !d.exists() {
@@ -764,6 +787,7 @@ fn ensure_destination_exists(destination: &str, fs_type: &str) -> Result<()> {
     Ok(())
 }
 
+#[instrument]
 fn parse_options(option_list: Vec<String>) -> HashMap<String, String> {
     let mut options = HashMap::new();
     for opt in option_list.iter() {

src/agent/src/namespace.rs

@@ -11,6 +11,7 @@ use std::fmt;
 use std::fs;
 use std::fs::File;
 use std::path::{Path, PathBuf};
+use tracing::instrument;
 
 use crate::mount::{BareMount, FLAGS};
 use slog::Logger;
@@ -20,6 +21,7 @@ pub const NSTYPEIPC: &str = "ipc";
 pub const NSTYPEUTS: &str = "uts";
 pub const NSTYPEPID: &str = "pid";
 
+#[instrument]
 pub fn get_current_thread_ns_path(ns_type: &str) -> String {
     format!(
         "/proc/{}/task/{}/ns/{}",
@@ -40,6 +42,7 @@ pub struct Namespace {
 }
 
 impl Namespace {
+    #[instrument]
     pub fn new(logger: &Logger) -> Self {
         Namespace {
             logger: logger.clone(),
@@ -50,11 +53,13 @@ impl Namespace {
         }
     }
 
+    #[instrument]
     pub fn get_ipc(mut self) -> Self {
         self.ns_type = NamespaceType::Ipc;
         self
     }
 
+    #[instrument]
     pub fn get_uts(mut self, hostname: &str) -> Self {
         self.ns_type = NamespaceType::Uts;
         if !hostname.is_empty() {
@@ -63,6 +68,7 @@ impl Namespace {
         self
     }
 
+    #[instrument]
     pub fn get_pid(mut self) -> Self {
         self.ns_type = NamespaceType::Pid;
         self
@@ -76,6 +82,7 @@ impl Namespace {
 
     // setup creates persistent namespace without switching to it.
     // Note, pid namespaces cannot be persisted.
+    #[instrument]
     pub async fn setup(mut self) -> Result<Self> {
         fs::create_dir_all(&self.persistent_ns_dir)?;
 

src/agent/src/random.rs

@@ -9,6 +9,7 @@ use nix::fcntl::{self, OFlag};
 use nix::sys::stat::Mode;
 use std::fs;
 use std::os::unix::io::{AsRawFd, FromRawFd};
+use tracing::instrument;
 
 pub const RNGDEV: &str = "/dev/random";
 pub const RNDADDTOENTCNT: libc::c_int = 0x40045201;
@@ -20,6 +21,7 @@ type IoctlRequestType = libc::c_int;
 #[cfg(target_env = "gnu")]
 type IoctlRequestType = libc::c_ulong;
 
+#[instrument]
 pub fn reseed_rng(data: &[u8]) -> Result<()> {
     let len = data.len() as libc::c_long;
     fs::write(RNGDEV, data)?;
@@ -37,10 +39,10 @@ pub fn reseed_rng(data: &[u8]) -> Result<()> {
             &len as *const libc::c_long,
         )
     };
-    let _ = Errno::result(ret).map(drop)?;
+    Errno::result(ret).map(drop)?;
 
     let ret = unsafe { libc::ioctl(f.as_raw_fd(), RNDRESEEDRNG as IoctlRequestType, 0) };
-    let _ = Errno::result(ret).map(drop)?;
+    Errno::result(ret).map(drop)?;
 
     Ok(())
 }

src/agent/src/rpc.rs

@@ -103,7 +103,7 @@ impl AgentService {
     ) -> Result<()> {
         let cid = req.container_id.clone();
 
-        let _ = verify_cid(&cid)?;
+        verify_cid(&cid)?;
 
         let mut oci_spec = req.OCI.clone();
         let use_sandbox_pidns = req.get_sandbox_pidns();
@@ -926,7 +926,7 @@ impl protocols::agent_ttrpc::AgentService for AgentService {
             }
 
             for m in req.kernel_modules.iter() {
-                let _ = load_kernel_module(m)
+                load_kernel_module(m)
                     .map_err(|e| ttrpc_error(ttrpc::Code::INTERNAL, e.to_string()))?;
             }
 
@@ -1529,22 +1529,22 @@ fn setup_bundle(cid: &str, spec: &mut Spec) -> Result<PathBuf> {
 fn cleanup_process(p: &mut Process) -> Result<()> {
     if p.parent_stdin.is_some() {
         p.close_stream(StreamType::ParentStdin);
-        let _ = unistd::close(p.parent_stdin.unwrap())?;
+        unistd::close(p.parent_stdin.unwrap())?;
     }
 
     if p.parent_stdout.is_some() {
         p.close_stream(StreamType::ParentStdout);
-        let _ = unistd::close(p.parent_stdout.unwrap())?;
+        unistd::close(p.parent_stdout.unwrap())?;
     }
 
     if p.parent_stderr.is_some() {
         p.close_stream(StreamType::ParentStderr);
-        let _ = unistd::close(p.parent_stderr.unwrap())?;
+        unistd::close(p.parent_stderr.unwrap())?;
     }
 
     if p.term_master.is_some() {
         p.close_stream(StreamType::TermMaster);
-        let _ = unistd::close(p.term_master.unwrap())?;
+        unistd::close(p.term_master.unwrap())?;
     }
 
     p.notify_term_close();

src/agent/src/sandbox.rs

@@ -28,6 +28,7 @@ use std::{thread, time};
 use tokio::sync::mpsc::{channel, Receiver, Sender};
 use tokio::sync::oneshot;
 use tokio::sync::Mutex;
+use tracing::instrument;
 
 type UeventWatcher = (Box<dyn UeventMatcher>, oneshot::Sender<Uevent>);
 
@@ -56,6 +57,7 @@ pub struct Sandbox {
 }
 
 impl Sandbox {
+    #[instrument]
     pub fn new(logger: &Logger) -> Result<Self> {
         let fs_type = get_mount_fs_type("/")?;
         let logger = logger.new(o!("subsystem" => "sandbox"));
@@ -94,6 +96,7 @@ impl Sandbox {
     //
     // It's assumed that caller is calling this method after
     // acquiring a lock on sandbox.
+    #[instrument]
     pub fn set_sandbox_storage(&mut self, path: &str) -> bool {
         match self.storages.get_mut(path) {
             None => {
@@ -116,6 +119,7 @@ impl Sandbox {
     //
     // It's assumed that caller is calling this method after
     // acquiring a lock on sandbox.
+    #[instrument]
     pub fn unset_sandbox_storage(&mut self, path: &str) -> Result<bool> {
         match self.storages.get_mut(path) {
             None => Err(anyhow!("Sandbox storage with path {} not found", path)),
@@ -135,6 +139,7 @@ impl Sandbox {
     //
     // It's assumed that caller is calling this method after
     // acquiring a lock on sandbox.
+    #[instrument]
     pub fn remove_sandbox_storage(&self, path: &str) -> Result<()> {
         let mounts = vec![path.to_string()];
         remove_mounts(&mounts)?;
@@ -148,6 +153,7 @@ impl Sandbox {
     //
     // It's assumed that caller is calling this method after
     // acquiring a lock on sandbox.
+    #[instrument]
     pub fn unset_and_remove_sandbox_storage(&mut self, path: &str) -> Result<()> {
         if self.unset_sandbox_storage(path)? {
             return self.remove_sandbox_storage(path);
@@ -156,6 +162,7 @@ impl Sandbox {
         Ok(())
     }
 
+    #[instrument]
     pub async fn setup_shared_namespaces(&mut self) -> Result<bool> {
         // Set up shared IPC namespace
         self.shared_ipcns = Namespace::new(&self.logger)
@@ -174,10 +181,12 @@ impl Sandbox {
         Ok(true)
     }
 
+    #[instrument]
     pub fn add_container(&mut self, c: LinuxContainer) {
         self.containers.insert(c.id.clone(), c);
     }
 
+    #[instrument]
     pub fn update_shared_pidns(&mut self, c: &LinuxContainer) -> Result<()> {
         // Populate the shared pid path only if this is an infra container and
         // sandbox_pidns has not been passed in the create_sandbox request.
@@ -201,10 +210,12 @@ impl Sandbox {
         Ok(())
     }
 
+    #[instrument]
     pub fn get_container(&mut self, id: &str) -> Option<&mut LinuxContainer> {
         self.containers.get_mut(id)
     }
 
+    #[instrument]
     pub fn find_process(&mut self, pid: pid_t) -> Option<&mut Process> {
         for (_, c) in self.containers.iter_mut() {
             if c.processes.get(&pid).is_some() {
@@ -215,6 +226,7 @@ impl Sandbox {
         None
     }
 
+    #[instrument]
     pub async fn destroy(&mut self) -> Result<()> {
         for ctr in self.containers.values_mut() {
             ctr.destroy().await?;
@@ -222,6 +234,7 @@ impl Sandbox {
         Ok(())
     }
 
+    #[instrument]
     pub fn online_cpu_memory(&self, req: &OnlineCPUMemRequest) -> Result<()> {
         if req.nb_cpus > 0 {
             // online cpus
@@ -265,6 +278,7 @@ impl Sandbox {
         Ok(())
     }
 
+    #[instrument]
     pub fn add_hooks(&mut self, dir: &str) -> Result<()> {
         let mut hooks = Hooks::default();
         if let Ok(hook) = self.find_hooks(dir, "prestart") {
@@ -280,6 +294,7 @@ impl Sandbox {
         Ok(())
     }
 
+    #[instrument]
     fn find_hooks(&self, hook_path: &str, hook_type: &str) -> Result<Vec<Hook>> {
         let mut hooks = Vec::new();
         for entry in fs::read_dir(Path::new(hook_path).join(hook_type))? {
@@ -316,6 +331,7 @@ impl Sandbox {
         Ok(hooks)
     }
 
+    #[instrument]
     pub async fn run_oom_event_monitor(&self, mut rx: Receiver<String>, container_id: String) {
         let logger = self.logger.clone();
 
@@ -348,6 +364,7 @@ impl Sandbox {
     }
 }
 
+#[instrument]
 fn online_resources(logger: &Logger, path: &str, pattern: &str, num: i32) -> Result<i32> {
     let mut count = 0;
     let re = Regex::new(pattern)?;
@@ -393,6 +410,7 @@ fn online_resources(logger: &Logger, path: &str, pattern: &str, num: i32) -> Res
 const ONLINE_CPUMEM_WATI_MILLIS: u64 = 50;
 const ONLINE_CPUMEM_MAX_RETRIES: u32 = 100;
 
+#[instrument]
 fn online_cpus(logger: &Logger, num: i32) -> Result<i32> {
     let mut onlined_count: i32 = 0;
 
@@ -422,6 +440,7 @@ fn online_cpus(logger: &Logger, num: i32) -> Result<i32> {
     ))
 }
 
+#[instrument]
 fn online_memory(logger: &Logger) -> Result<()> {
     online_resources(logger, SYSFS_MEMORY_ONLINE_PATH, r"memory[0-9]+", -1)?;
     Ok(())

src/agent/src/signal.rs

@@ -6,10 +6,10 @@
 
 use crate::sandbox::Sandbox;
 use anyhow::{anyhow, Result};
+use capctl::prctl::set_subreaper;
 use nix::sys::wait::WaitPidFlag;
 use nix::sys::wait::{self, WaitStatus};
 use nix::unistd;
-use prctl::set_child_subreaper;
 use slog::{error, info, o, Logger};
 use std::sync::Arc;
 use tokio::select;
@@ -88,7 +88,7 @@ pub async fn setup_signal_handler(
 ) -> Result<()> {
     let logger = logger.new(o!("subsystem" => "signals"));
 
-    set_child_subreaper(true)
+    set_subreaper(true)
         .map_err(|err| anyhow!(err).context("failed to setup agent as a child subreaper"))?;
 
     let mut sigchild_stream = signal(SignalKind::child())?;

src/agent/src/tracer.rs

@@ -0,0 +1,91 @@
+// Copyright (c) 2020-2021 Intel Corporation
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+use crate::config::AgentConfig;
+use anyhow::Result;
+use opentelemetry::{global, sdk::trace::Config, trace::TracerProvider};
+use slog::{info, o, Logger};
+use std::error::Error;
+use std::fmt;
+use std::str::FromStr;
+use tracing_opentelemetry::OpenTelemetryLayer;
+use tracing_subscriber::layer::SubscriberExt;
+use tracing_subscriber::Registry;
+
+#[derive(Debug, PartialEq)]
+pub enum TraceType {
+    Disabled,
+    Isolated,
+}
+
+#[derive(Debug)]
+pub struct TraceTypeError {
+    details: String,
+}
+
+impl TraceTypeError {
+    fn new(msg: &str) -> TraceTypeError {
+        TraceTypeError {
+            details: msg.into(),
+        }
+    }
+}
+
+impl Error for TraceTypeError {}
+
+impl fmt::Display for TraceTypeError {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "{}", self.details)
+    }
+}
+
+impl FromStr for TraceType {
+    type Err = TraceTypeError;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s {
+            "isolated" => Ok(TraceType::Isolated),
+            "disabled" => Ok(TraceType::Disabled),
+            _ => Err(TraceTypeError::new("invalid trace type")),
+        }
+    }
+}
+
+pub fn setup_tracing(name: &'static str, logger: &Logger, _agent_cfg: &AgentConfig) -> Result<()> {
+    let logger = logger.new(o!("subsystem" => "vsock-tracer"));
+
+    let exporter = vsock_exporter::Exporter::builder()
+        .with_logger(&logger)
+        .init();
+
+    let config = Config::default();
+
+    let builder = opentelemetry::sdk::trace::TracerProvider::builder()
+        .with_simple_exporter(exporter)
+        .with_config(config);
+
+    let provider = builder.build();
+
+    // We don't need a versioned tracer.
+    let version = None;
+
+    let tracer = provider.get_tracer(name, version);
+
+    let _global_provider = global::set_tracer_provider(provider);
+
+    let layer = OpenTelemetryLayer::new(tracer);
+
+    let subscriber = Registry::default().with(layer);
+
+    tracing::subscriber::set_global_default(subscriber)?;
+
+    info!(logger, "tracing setup");
+
+    Ok(())
+}
+
+pub fn end_tracing() {
+    global::shutdown_tracer_provider();
+}

src/agent/src/uevent.rs

@@ -18,6 +18,7 @@ use std::sync::Arc;
 use tokio::select;
 use tokio::sync::watch::Receiver;
 use tokio::sync::Mutex;
+use tracing::instrument;
 
 // Convenience macro to obtain the scope logger
 macro_rules! sl {
@@ -64,6 +65,7 @@ impl Uevent {
         event
     }
 
+    #[instrument]
     async fn process_add(&self, logger: &Logger, sandbox: &Arc<Mutex<Sandbox>>) {
         // Special case for memory hot-adds first
         let online_path = format!("{}/{}/online", SYSFS_DIR, &self.devpath);
@@ -95,6 +97,7 @@ impl Uevent {
         }
     }
 
+    #[instrument]
     async fn process(&self, logger: &Logger, sandbox: &Arc<Mutex<Sandbox>>) {
         if self.action == U_EVENT_ACTION_ADD {
             return self.process_add(logger, sandbox).await;
@@ -103,6 +106,7 @@ impl Uevent {
     }
 }
 
+#[instrument]
 pub async fn wait_for_uevent(
     sandbox: &Arc<Mutex<Sandbox>>,
     matcher: impl UeventMatcher,
@@ -145,6 +149,7 @@ pub async fn wait_for_uevent(
     Ok(uev)
 }
 
+#[instrument]
 pub async fn watch_uevents(
     sandbox: Arc<Mutex<Sandbox>>,
     mut shutdown: Receiver<bool>,

src/agent/src/util.rs

@@ -11,6 +11,7 @@ use std::os::unix::io::{FromRawFd, RawFd};
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::sync::watch::Receiver;
 use tokio_vsock::{Incoming, VsockListener, VsockStream};
+use tracing::instrument;
 
 // Size of I/O read buffer
 const BUF_SIZE: usize = 8192;
@@ -56,10 +57,12 @@ where
     Ok(total_bytes)
 }
 
+#[instrument]
 pub fn get_vsock_incoming(fd: RawFd) -> Incoming {
     unsafe { VsockListener::from_raw_fd(fd).incoming() }
 }
 
+#[instrument]
 pub async fn get_vsock_stream(fd: RawFd) -> Result<VsockStream> {
     let stream = get_vsock_incoming(fd).next().await.unwrap()?;
     Ok(stream)

src/agent/vsock-exporter/Cargo.toml

@@ -0,0 +1,19 @@
+[package]
+name = "vsock-exporter"
+version = "0.1.0"
+authors = ["James O. D. Hunt <james.o.hunt@intel.com>"]
+edition = "2018"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+nix = "0.20.0"
+libc = "0.2.94"
+thiserror = "1.0.24"
+opentelemetry = { version = "0.14.0", features=["serialize"] }
+serde = { version = "1.0.126", features = ["derive"] }
+vsock = "0.2.3"
+bincode = "1.3.3"
+byteorder = "1.4.3"
+slog = { version = "2.5.2", features = ["dynamic-keys", "max_level_trace", "release_max_level_info"] }
+async-trait = "0.1.50"

src/agent/vsock-exporter/src/lib.rs

@@ -0,0 +1,196 @@
+// Copyright (c) 2020-2021 Intel Corporation
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+// The VSOCK Exporter sends trace spans "out" to the forwarder running on the
+// host (which then forwards them on to a trace collector). The data is sent
+// via a VSOCK socket that the forwarder process is listening on. To allow the
+// forwarder to know how much data to each for each trace span the simplest
+// protocol is employed which uses a header packet and the payload (trace
+// span) data. The header packet is a simple count of the number of bytes in the
+// payload, which allows the forwarder to know how many bytes it must read to
+// consume the trace span. The payload is a serialised version of the trace span.
+
+use async_trait::async_trait;
+use byteorder::{ByteOrder, NetworkEndian};
+use opentelemetry::sdk::export::trace::{ExportResult, SpanData, SpanExporter};
+use opentelemetry::sdk::export::ExportError;
+use slog::{error, o, Logger};
+use std::io::{ErrorKind, Write};
+use std::net::Shutdown;
+use std::sync::Mutex;
+use vsock::{SockAddr, VsockStream};
+
+const ANY_CID: &str = "any";
+
+// Must match the value of the variable of the same name in the trace forwarder.
+const HEADER_SIZE_BYTES: u64 = std::mem::size_of::<u64>() as u64;
+
+// By default, the VSOCK exporter should talk "out" to the host where the
+// forwarder is running.
+const DEFAULT_CID: u32 = libc::VMADDR_CID_HOST;
+
+// The VSOCK port the forwarders listens on by default
+const DEFAULT_PORT: u32 = 10240;
+
+#[derive(Debug)]
+pub struct Exporter {
+    port: u32,
+    cid: u32,
+    conn: Mutex<VsockStream>,
+    logger: Logger,
+}
+
+impl Exporter {
+    /// Create a new exporter builder.
+    pub fn builder() -> Builder {
+        Builder::default()
+    }
+}
+
+#[derive(thiserror::Error, Debug)]
+pub enum Error {
+    #[error("connection error: {0}")]
+    ConnectionError(String),
+    #[error("serialisation error: {0}")]
+    SerialisationError(#[from] bincode::Error),
+    #[error("I/O error: {0}")]
+    IOError(#[from] std::io::Error),
+}
+
+impl ExportError for Error {
+    fn exporter_name(&self) -> &'static str {
+        "vsock-exporter"
+    }
+}
+
+fn make_io_error(desc: String) -> std::io::Error {
+    std::io::Error::new(ErrorKind::Other, desc)
+}
+
+// Send a trace span to the forwarder running on the host.
+fn write_span(writer: &mut dyn Write, span: &SpanData) -> Result<(), std::io::Error> {
+    let encoded_payload: Vec<u8> =
+        bincode::serialize(&span).map_err(|e| make_io_error(e.to_string()))?;
+
+    let payload_len: u64 = encoded_payload.len() as u64;
+
+    let mut payload_len_as_bytes: [u8; HEADER_SIZE_BYTES as usize] =
+        [0; HEADER_SIZE_BYTES as usize];
+
+    // Encode the header
+    NetworkEndian::write_u64(&mut payload_len_as_bytes, payload_len);
+
+    // Send the header
+    writer
+        .write_all(&payload_len_as_bytes)
+        .map_err(|e| make_io_error(format!("failed to write trace header: {:?}", e)))?;
+
+    writer
+        .write_all(&encoded_payload)
+        .map_err(|e| make_io_error(format!("failed to write trace payload: {:?}", e)))
+}
+
+fn handle_batch(writer: &mut dyn Write, batch: Vec<SpanData>) -> ExportResult {
+    for span_data in batch {
+        write_span(writer, &span_data).map_err(Error::IOError)?;
+    }
+
+    Ok(())
+}
+
+#[async_trait]
+impl SpanExporter for Exporter {
+    async fn export(&mut self, batch: Vec<SpanData>) -> ExportResult {
+        let conn = self.conn.lock();
+
+        match conn {
+            Ok(mut c) => handle_batch(&mut *c, batch),
+            Err(e) => {
+                error!(self.logger, "failed to obtain connection";
+                        "error" => format!("{}", e));
+
+                return Err(Error::ConnectionError(e.to_string()).into());
+            }
+        }
+    }
+
+    fn shutdown(&mut self) {
+        let conn = match self.conn.lock() {
+            Ok(conn) => conn,
+            Err(e) => {
+                error!(self.logger, "failed to obtain connection";
+                        "error" => format!("{}", e));
+                return;
+            }
+        };
+
+        conn.shutdown(Shutdown::Write)
+            .expect("failed to shutdown VSOCK connection");
+    }
+}
+
+#[derive(Debug)]
+pub struct Builder {
+    port: u32,
+    cid: u32,
+    logger: Logger,
+}
+
+impl Default for Builder {
+    fn default() -> Self {
+        let logger = Logger::root(slog::Discard, o!());
+
+        Builder {
+            cid: DEFAULT_CID,
+            port: DEFAULT_PORT,
+            logger,
+        }
+    }
+}
+
+impl Builder {
+    pub fn with_cid(self, cid: u32) -> Self {
+        Builder { cid, ..self }
+    }
+
+    pub fn with_port(self, port: u32) -> Self {
+        Builder { port, ..self }
+    }
+
+    pub fn with_logger(self, logger: &Logger) -> Self {
+        Builder {
+            logger: logger.new(o!()),
+            ..self
+        }
+    }
+
+    pub fn init(self) -> Exporter {
+        let Builder { port, cid, logger } = self;
+
+        let sock_addr = SockAddr::new_vsock(self.cid, self.port);
+
+        let cid_str: String;
+
+        if self.cid == libc::VMADDR_CID_ANY {
+            cid_str = ANY_CID.to_string();
+        } else {
+            cid_str = format!("{}", self.cid);
+        }
+
+        let msg = format!(
+            "failed to connect to VSOCK server (port: {}, cid: {}) - {}",
+            self.port, cid_str, "ensure trace forwarder is running on host"
+        );
+
+        let conn = VsockStream::connect(&sock_addr).expect(&msg);
+
+        Exporter {
+            port,
+            cid,
+            conn: Mutex::new(conn),
+            logger: logger.new(o!("cid" => cid_str, "port" => port)),
+        }
+    }
+}

src/runtime/arch/s390x-options.mk

@@ -11,3 +11,10 @@ MACHINEACCELERATORS :=
 CPUFEATURES :=
 
 QEMUCMD := qemu-system-s390x
+
+# See https://github.com/kata-containers/osbuilder/issues/217
+FEDORA_LIKE = $(shell grep -E "\<fedora\>" /etc/os-release 2> /dev/null)
+ifneq (,$(FEDORA_LIKE))
+	CC := gcc
+	export CC
+endif

src/runtime/cli/config/configuration-qemu.toml.in

@@ -16,6 +16,14 @@ kernel = "@KERNELPATH@"
 image = "@IMAGEPATH@"
 machine_type = "@MACHINETYPE@"
 
+# Enable confidential guest support.
+# Toggling that setting may trigger different hardware features, ranging
+# from memory encryption to both memory and CPU-state encryption and integrity.
+# The Kata Containers runtime dynamically detects the available feature set and
+# aims at enabling the largest possible one.
+# Default false
+# confidential_guest = true
+
 # List of valid annotation names for the hypervisor
 # Each member of the list is a regular expression, which is the base name
 # of the annotation, e.g. "path" for io.katacontainers.config.hypervisor.path"
@@ -536,3 +544,30 @@ experimental=@DEFAULTEXPFEATURES@
 # If enabled, user can run pprof tools with shim v2 process through kata-monitor.
 # (default: false)
 # enable_pprof = true
+
+# WARNING: All the options in the following section have not been implemented yet.
+# This section was added as a placeholder. DO NOT USE IT!
+[image]
+# Container image service.
+#
+# Offload the CRI image management service to the Kata agent.
+# (default: false)
+#service_offload = true
+
+# Container image decryption keys provisioning.
+# Applies only if service_offload is true.
+# Keys can be provisioned locally (e.g. through a special command or
+# a local file) or remotely (usually after the guest is remotely attested).
+# The provision setting is a complete URL that lets the Kata agent decide
+# which method to use in order to fetch the keys.
+#
+# Keys can be stored in a local file, in a measured and attested initrd:
+#provision=data:///local/key/file
+#
+# Keys could be fetched through a special command or binary from the
+# initrd (guest) image, e.g. a firmware call:
+#provision=file:///path/to/bin/fetcher/in/guest
+#
+# Keys can be remotely provisioned. The Kata agent fetches them from e.g.
+# a HTTPS URL:
+#provision=https://my-key-broker.foo/tenant/<tenant-id>

src/runtime/cli/kata-check.go

@@ -25,6 +25,7 @@ import (
 	"strings"
 	"syscall"
 
+	"github.com/containerd/cgroups"
 	"github.com/kata-containers/kata-containers/src/runtime/pkg/katautils"
 	vc "github.com/kata-containers/kata-containers/src/runtime/virtcontainers"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/oci"
@@ -416,6 +417,11 @@ EXAMPLES:
 			return errors.New("check: cannot determine runtime config")
 		}
 
+		// check if cgroup can work use the same logic for creating containers
+		if _, err := vc.V1Constraints(); err != nil && err == cgroups.ErrMountPointNotExist && !runtimeConfig.SandboxCgroupOnly {
+			return fmt.Errorf("Cgroup v2 requires the following configuration: `sandbox_cgroup_only=true`.")
+		}
+
 		err := setCPUtype(runtimeConfig.HypervisorType)
 		if err != nil {
 			return err

src/runtime/cli/main.go

@@ -22,6 +22,7 @@ import (
 	vc "github.com/kata-containers/kata-containers/src/runtime/virtcontainers"
 	exp "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/experimental"
 	vf "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/factory"
+	tl "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/factory/template"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/oci"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/rootless"
 	specs "github.com/opencontainers/runtime-spec/specs-go"
@@ -201,6 +202,9 @@ func setExternalLoggers(ctx context.Context, logger *logrus.Entry) {
 	// Set vm factory logger.
 	vf.SetLogger(ctx, logger)
 
+	// Set vm factory template logger.
+	tl.SetLogger(ctx, logger)
+
 	// Set the OCI package logger.
 	oci.SetLogger(ctx, logger)
 

src/runtime/cli/main_test.go

@@ -20,7 +20,6 @@ import (
 	"strings"
 	"testing"
 
-	"github.com/dlespiau/covertool/pkg/cover"
 	ktu "github.com/kata-containers/kata-containers/src/runtime/pkg/katatestutils"
 	"github.com/kata-containers/kata-containers/src/runtime/pkg/katautils"
 	"github.com/kata-containers/kata-containers/src/runtime/pkg/utils"
@@ -152,16 +151,6 @@ func runUnitTests(m *testing.M) {
 // TestMain is the common main function used by ALL the test functions
 // for this package.
 func TestMain(m *testing.M) {
-	// Parse the command line using the stdlib flag package so the flags defined
-	// in the testing package get populated.
-	cover.ParseAndStripTestFlags()
-
-	// Make sure we have the opportunity to flush the coverage report to disk when
-	// terminating the process.
-	defer func() {
-		cover.FlushProfiles()
-	}()
-
 	// If the test binary name is kata-runtime.coverage, we've are being asked to
 	// run the coverage-instrumented kata-runtime.
 	if path.Base(os.Args[0]) == name+".coverage" ||

src/runtime/containerd-shim-v2/service.go

@@ -294,8 +294,7 @@ func trace(ctx context.Context, name string) (otelTrace.Span, context.Context) {
 		ctx = context.Background()
 	}
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(ctx, name)
-	span.SetAttributes([]label.KeyValue{label.Key("source").String("runtime"), label.Key("package").String("containerdshim")}...)
+	ctx, span := tracer.Start(ctx, name, otelTrace.WithAttributes(label.String("source", "runtime"), label.String("package", "containerdshim")))
 
 	return span, ctx
 }

src/runtime/containerd-shim-v2/start.go

@@ -14,7 +14,7 @@ import (
 )
 
 func startContainer(ctx context.Context, s *service, c *container) error {
-	//start a container
+	// start a container
 	if c.cType == "" {
 		err := fmt.Errorf("Bug, the container %s type is empty", c.id)
 		return err
@@ -37,8 +37,8 @@ func startContainer(ctx context.Context, s *service, c *container) error {
 		}
 		go watchSandbox(ctx, s)
 
-		// We don't rely on the context passed to startContainer as it can be cancelled after
-		// this rpc call.
+		// We use s.ctx(`ctx` derived from `s.ctx`) to check for cancellation of the
+		// shim context and the context passed to startContainer for tracing.
 		go watchOOMEvents(ctx, s)
 	} else {
 		_, err := s.sandbox.StartContainer(ctx, c.id)
@@ -74,10 +74,10 @@ func startContainer(ctx context.Context, s *service, c *container) error {
 		c.ttyio = tty
 		go ioCopy(c.exitIOch, c.stdinCloser, tty, stdin, stdout, stderr)
 	} else {
-		//close the io exit channel, since there is no io for this container,
-		//otherwise the following wait goroutine will hang on this channel.
+		// close the io exit channel, since there is no io for this container,
+		// otherwise the following wait goroutine will hang on this channel.
 		close(c.exitIOch)
-		//close the stdin closer channel to notify that it's safe to close process's
+		// close the stdin closer channel to notify that it's safe to close process's
 		// io.
 		close(c.stdinCloser)
 	}
@@ -88,7 +88,7 @@ func startContainer(ctx context.Context, s *service, c *container) error {
 }
 
 func startExec(ctx context.Context, s *service, containerID, execID string) (*exec, error) {
-	//start an exec
+	// start an exec
 	c, err := s.getContainer(containerID)
 	if err != nil {
 		return nil, err

src/runtime/go.mod

@@ -18,7 +18,6 @@ require (
 	github.com/containerd/typeurl v1.0.1-0.20190228175220-2a93cfde8c20
 	github.com/containernetworking/plugins v0.8.2
 	github.com/cri-o/cri-o v1.0.0-rc2.0.20170928185954-3394b3b2d6af
-	github.com/dlespiau/covertool v0.0.0-20180314162135-b0c4c6d0583a
 	github.com/docker/distribution v2.7.1+incompatible // indirect
 	github.com/docker/docker v1.13.1 // indirect
 	github.com/docker/go-events v0.0.0-20190806004212-e31b211e4f1c // indirect
@@ -31,7 +30,7 @@ require (
 	github.com/gogo/googleapis v1.4.0 // indirect
 	github.com/gogo/protobuf v1.3.1
 	github.com/hashicorp/go-multierror v1.0.0
-	github.com/kata-containers/govmm v0.0.0-20210428163604-f0e9a35308ee
+	github.com/kata-containers/govmm v0.0.0-20210520142420-eb57f004d89f
 	github.com/mdlayher/vsock v0.0.0-20191108225356-d9c65923cb8f
 	github.com/opencontainers/image-spec v1.0.1 // indirect
 	github.com/opencontainers/runc v1.0.0-rc9.0.20200102164712-2b52db75279c

src/runtime/go.sum

@@ -108,8 +108,6 @@ github.com/d2g/hardwareaddr v0.0.0-20190221164911-e7d9fbe030e4/go.mod h1:bMl4RjI
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/dlespiau/covertool v0.0.0-20180314162135-b0c4c6d0583a h1:+cYgqwB++gEE09SluRYGqJyDhWmLmdWZ2cXlOXSGV8w=
-github.com/dlespiau/covertool v0.0.0-20180314162135-b0c4c6d0583a/go.mod h1:/eQMcW3eA1bzKx23ZYI2H3tXPdJB5JWYTHzoUPBvQY4=
 github.com/docker/distribution v2.7.1+incompatible h1:a5mlkVzth6W5A4fOsS3D2EO5BUmsJpcB+cRlLU7cSug=
 github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
 github.com/docker/docker v1.13.1 h1:IkZjBSIc8hBjLpqeAbeE5mca5mNgeatLHBy3GO78BWo=
@@ -200,7 +198,6 @@ github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt
 github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
 github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
 github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
 github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
@@ -218,9 +215,7 @@ github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Z
 github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
 github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
-github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg=
 github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
-github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
 github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
@@ -242,7 +237,6 @@ github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hf
 github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
 github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
 github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
-github.com/google/uuid v1.1.0 h1:Jf4mxPC/ziBnoPIdpQdPJ9OeiomAUHLvxmPRSPH9m4s=
 github.com/google/uuid v1.1.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
 github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -274,12 +268,10 @@ github.com/juju/errors v0.0.0-20180806074554-22422dad46e1/go.mod h1:W54LbzXuIE0b
 github.com/juju/loggo v0.0.0-20190526231331-6e530bcce5d8/go.mod h1:vgyd7OREkbtVEN/8IXZe5Ooef3LQePvuBm9UWj6ZL8U=
 github.com/juju/testing v0.0.0-20190613124551-e81189438503/go.mod h1:63prj8cnj0tU0S9OHjGJn+b1h0ZghCndfnbQolrYTwA=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
-github.com/kata-containers/govmm v0.0.0-20210112013750-7d320e8f5dca h1:UdXFthwasAPnmv37gLJUEFsW9FaabYA+mM6FoSi8kzU=
-github.com/kata-containers/govmm v0.0.0-20210112013750-7d320e8f5dca/go.mod h1:VmAHbsL5lLfzHW/MNL96NVLF840DNEV5i683kISgFKk=
-github.com/kata-containers/govmm v0.0.0-20210329205824-7fbc685865d2 h1:oDJsQ5avmW8PFUkSJdsuaGL3SR4/YQRno51GNGl+IDU=
-github.com/kata-containers/govmm v0.0.0-20210329205824-7fbc685865d2/go.mod h1:VmAHbsL5lLfzHW/MNL96NVLF840DNEV5i683kISgFKk=
 github.com/kata-containers/govmm v0.0.0-20210428163604-f0e9a35308ee h1:M4N7AdSHgWz/ubV5AZQdeqmK+9Ztpea6oqeXgk8GCHk=
 github.com/kata-containers/govmm v0.0.0-20210428163604-f0e9a35308ee/go.mod h1:VmAHbsL5lLfzHW/MNL96NVLF840DNEV5i683kISgFKk=
+github.com/kata-containers/govmm v0.0.0-20210520142420-eb57f004d89f h1:jXMZY7GIz5kSv3/Rdiesg1WMvgXJKNOk3KxwxgNWAVk=
+github.com/kata-containers/govmm v0.0.0-20210520142420-eb57f004d89f/go.mod h1:VmAHbsL5lLfzHW/MNL96NVLF840DNEV5i683kISgFKk=
 github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
@@ -307,15 +299,12 @@ github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3Rllmb
 github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
 github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f/go.mod h1:ZdcZmHo+o7JKHSa8/e818NopupXU1YMK5fe1lsApnBw=
-github.com/onsi/ginkgo v0.0.0-20151202141238-7f8ab55aaf3b h1:Ey6yH0acn50T/v6CB75bGP4EMJqnv9WvnjN7oZaj+xE=
 github.com/onsi/ginkgo v0.0.0-20151202141238-7f8ab55aaf3b/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v0.0.0-20170829012221-11459a886d9c/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
-github.com/onsi/ginkgo v1.10.1 h1:q/mM8GF/n0shIN8SaAZ0V+jnLPzen6WIVZdiwrRlMlo=
 github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.11.0 h1:JAKSXpt1YjtLA7YpPiqO9ss6sNXEsPfSGdwN0UHqzrw=
 github.com/onsi/ginkgo v1.11.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
-github.com/onsi/gomega v0.0.0-20151007035656-2152b45fa28a h1:KfNOeFvoAssuZLT7IntKZElKwi/5LRuxY71k+t6rfaM=
 github.com/onsi/gomega v0.0.0-20151007035656-2152b45fa28a/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/onsi/gomega v1.7.0 h1:XPnZz8VVBHjVsy1vzJmRwIcSwiUO+JFfrv/xGiigmME=
@@ -335,7 +324,6 @@ github.com/pborman/uuid v1.2.0 h1:J7Q5mO4ysT1dv8hyrUGHb9+ooztCXu1D8MY8DZYsu3g=
 github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1-0.20171018195549-f15c970de5b7/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
-github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -379,7 +367,6 @@ github.com/stretchr/objx v0.1.1 h1:2vfRuCMp5sSVIDSqO8oNnWJq7mPa6KVP3iPIwFBuy8A=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
-github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
 github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
@@ -431,7 +418,6 @@ golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTk
 golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
 golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
-golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
 golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
 golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
 golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
@@ -463,7 +449,6 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL
 golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20191004110552-13f9640d40b9/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20191108221443-4ba9e2ef068c h1:SRpq/kuj/xNci/RdvEs+RSvpfxqvLAzTKuKGlzoGdZQ=
 golang.org/x/net v0.0.0-20191108221443-4ba9e2ef068c/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
@@ -477,7 +462,6 @@ golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/
 golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
 golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
-golang.org/x/net v0.0.0-20200707034311-ab3426394381 h1:VXak5I6aEWmAXeQjA+QSZzlgNrpq9mjcfDemuexIKsU=
 golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
 golang.org/x/net v0.0.0-20200822124328-c89045814202 h1:VvcQYSHwXgi7W+TpUR6A9g6Up98WAHf3f/ulnJ62IyA=
 golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
@@ -493,7 +477,6 @@ golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e h1:vcxGaoTs7kV8m5Np9uUNQin4BrLOthgV7252N8V+FwY=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208 h1:qwRHBd0NqMbJxfbotnDhm2ByMI1Shq4Y6oRJo21SGJA=
@@ -535,17 +518,14 @@ golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1 h1:ogLJMz+qpzav7lGMh10LMvAkM/fAoGlaiiHYiFYdm80=
 golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f h1:Fqb3ao1hUmOR3GkUOg/Y+BadLwykBIzs5q8Ez2SbHyc=
 golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@@ -597,7 +577,6 @@ golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc
 golang.org/x/tools v0.0.0-20200904185747-39188db58858/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -620,7 +599,6 @@ google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz513
 google.golang.org/api v0.32.0 h1:Le77IccnTqEa8ryp9wIpX5W3zYm7Gf9LhOp9PHcwFts=
 google.golang.org/api v0.32.0/go.mod h1:/XrVsuzM0rZmrsbjJutiuftIzeuTQcEeaYcSk/mQ1dg=
 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
-google.golang.org/appengine v1.4.0 h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508=
 google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
 google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
@@ -637,14 +615,12 @@ google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQ
 google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
 google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
 google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
-google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
 google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
 google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
 google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
 gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
 gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -658,10 +634,8 @@ gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3M
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
 gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.5 h1:ymVxjfMaHvXD8RqPRmzHHsB3VvucivSkIAvJFDI5O3c=
 gopkg.in/yaml.v2 v2.2.5/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
 gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

src/runtime/pkg/katautils/config-settings.go.in

@@ -54,6 +54,7 @@ const defaultDisableImageNvdimm = false
 const defaultVhostUserStorePath string = "/var/run/kata-containers/vhost-user/"
 const defaultRxRateLimiterMaxRate = uint64(0)
 const defaultTxRateLimiterMaxRate = uint64(0)
+const defaultConfidentialGuest = false
 
 var defaultSGXEPCSize = int64(0)
 

src/runtime/pkg/katautils/config.go

@@ -1,4 +1,4 @@
-// Copyright (c) 2018 Intel Corporation
+// Copyright (c) 2018-2021 Intel Corporation
 // Copyright (c) 2018 HyperHQ Inc.
 //
 // SPDX-License-Identifier: Apache-2.0
@@ -61,6 +61,12 @@ type tomlConfig struct {
 	Runtime    runtime
 	Factory    factory
 	Netmon     netmon
+	Image      image
+}
+
+type image struct {
+	ServiceOffload bool   `toml:"service_offload"`
+	Provision      string `toml:"provision"`
 }
 
 type factory struct {
@@ -130,6 +136,7 @@ type hypervisor struct {
 	HotplugVFIOOnRootBus    bool     `toml:"hotplug_vfio_on_root_bus"`
 	DisableVhostNet         bool     `toml:"disable_vhost_net"`
 	GuestMemoryDumpPaging   bool     `toml:"guest_memory_dump_paging"`
+	ConfidentialGuest       bool     `toml:"confidential_guest"`
 }
 
 type runtime struct {
@@ -707,6 +714,7 @@ func newQemuHypervisorConfig(h hypervisor) (vc.HypervisorConfig, error) {
 		EnableAnnotations:       h.EnableAnnotations,
 		GuestMemoryDumpPath:     h.GuestMemoryDumpPath,
 		GuestMemoryDumpPaging:   h.GuestMemoryDumpPaging,
+		ConfidentialGuest:       h.ConfidentialGuest,
 	}, nil
 }
 
@@ -1061,6 +1069,7 @@ func GetDefaultHypervisorConfig() vc.HypervisorConfig {
 		RxRateLimiterMaxRate:    defaultRxRateLimiterMaxRate,
 		TxRateLimiterMaxRate:    defaultTxRateLimiterMaxRate,
 		SGXEPCSize:              defaultSGXEPCSize,
+		ConfidentialGuest:       defaultConfidentialGuest,
 	}
 }
 

src/runtime/pkg/katautils/create.go

@@ -104,7 +104,7 @@ func SetEphemeralStorageType(ociSpec specs.Spec) specs.Spec {
 // CreateSandbox create a sandbox container
 func CreateSandbox(ctx context.Context, vci vc.VC, ociSpec specs.Spec, runtimeConfig oci.RuntimeConfig, rootFs vc.RootFs,
 	containerID, bundlePath, console string, disableOutput, systemdCgroup bool) (_ vc.VCSandbox, _ vc.Process, err error) {
-	span, ctx := Trace(ctx, "createSandbox")
+	span, ctx := Trace(ctx, "CreateSandbox", []label.KeyValue{label.Key("source").String("runtime"), label.Key("package").String("katautils"), label.Key("subsystem").String("sandbox"), label.Key("container_id").String(containerID)}...)
 	defer span.End()
 
 	sandboxConfig, err := oci.SandboxConfig(ociSpec, runtimeConfig, bundlePath, containerID, console, disableOutput, systemdCgroup)
@@ -159,7 +159,7 @@ func CreateSandbox(ctx context.Context, vci vc.VC, ociSpec specs.Spec, runtimeCo
 
 	sid := sandbox.ID()
 	kataUtilsLogger = kataUtilsLogger.WithField("sandbox", sid)
-	span.SetAttributes(label.Key("sandbox").String(sid))
+	span.SetAttributes(label.Key("sandbox_id").String(sid))
 
 	containers := sandbox.GetAllContainers()
 	if len(containers) != 1 {
@@ -202,7 +202,7 @@ func checkForFIPS(sandboxConfig *vc.SandboxConfig) error {
 func CreateContainer(ctx context.Context, sandbox vc.VCSandbox, ociSpec specs.Spec, rootFs vc.RootFs, containerID, bundlePath, console string, disableOutput bool) (vc.Process, error) {
 	var c vc.VCContainer
 
-	span, ctx := Trace(ctx, "createContainer")
+	span, ctx := Trace(ctx, "CreateContainer", []label.KeyValue{label.Key("source").String("runtime"), label.Key("package").String("katautils"), label.Key("subsystem").String("sandbox"), label.Key("container_id").String(containerID)}...)
 	defer span.End()
 
 	ociSpec = SetEphemeralStorageType(ociSpec)
@@ -228,7 +228,7 @@ func CreateContainer(ctx context.Context, sandbox vc.VCSandbox, ociSpec specs.Sp
 		return vc.Process{}, err
 	}
 
-	span.SetAttributes(label.Key("sandbox").String(sandboxID))
+	span.SetAttributes(label.Key("sandbox_id").String(sandboxID))
 
 	c, err = sandbox.CreateContainer(ctx, contConfig)
 	if err != nil {

src/runtime/pkg/katautils/hook.go

@@ -26,11 +26,9 @@ func hookLogger() *logrus.Entry {
 }
 
 func runHook(ctx context.Context, hook specs.Hook, cid, bundlePath string) error {
-	span, _ := Trace(ctx, "hook")
+	span, _ := Trace(ctx, "runHook", []label.KeyValue{label.Key("source").String("runtime"), label.Key("package").String("katautils"), label.Key("subsystem").String("hook")}...)
 	defer span.End()
 
-	span.SetAttributes(label.Key("subsystem").String("runHook"))
-
 	// FIXME
 	// span.LogFields(
 	// 	log.String("hook-name", hook.Path),
@@ -90,11 +88,9 @@ func runHook(ctx context.Context, hook specs.Hook, cid, bundlePath string) error
 }
 
 func runHooks(ctx context.Context, hooks []specs.Hook, cid, bundlePath, hookType string) error {
-	span, _ := Trace(ctx, "hooks")
+	span, _ := Trace(ctx, "runHooks", []label.KeyValue{label.Key("source").String("runtime"), label.Key("package").String("katautils"), label.Key("subsystem").String("hook"), label.Key("type").String(hookType)}...)
 	defer span.End()
 
-	span.SetAttributes(label.Key("subsystem").String(hookType))
-
 	for _, hook := range hooks {
 		if err := runHook(ctx, hook, cid, bundlePath); err != nil {
 			hookLogger().WithFields(logrus.Fields{

src/runtime/pkg/katautils/tracing.go

@@ -110,12 +110,11 @@ func StopTracing(ctx context.Context) {
 }
 
 // Trace creates a new tracing span based on the specified name and parent
-// context.
-func Trace(parent context.Context, name string) (otelTrace.Span, context.Context) {
+// context and an opentelemetry label.KeyValue slice for span attributes.
+func Trace(parent context.Context, name string, tags ...label.KeyValue) (otelTrace.Span, context.Context) {
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes(label.Key("source").String("runtime"))
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(tags...))
 
 	// This is slightly confusing: when tracing is disabled, trace spans
 	// are still created - but the tracer used is a NOP. Therefore, only

src/runtime/vendor/github.com/dlespiau/covertool/LICENSE

@@ -1,201 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "{}"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright {yyyy} {name of copyright owner}
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

src/runtime/vendor/github.com/dlespiau/covertool/pkg/cover/cover.go

@@ -1,82 +0,0 @@
-// Copyright (c) 2017 Intel Corporation
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package cover
-
-import (
-	"flag"
-	"io"
-	"os"
-	"strings"
-	"testing"
-)
-
-// ParseAndStriptestFlags runs flag.Parse to parse the standard flags of a test
-// binary and remove them from os.Args.
-func ParseAndStripTestFlags() {
-	// Parse the command line using the stdlib flag package so the flags defined
-	// in the testing package get populated.
-	flag.Parse()
-
-	// Strip command line arguments that were for the testing package and are now
-	// handled.
-	var runtimeArgs []string
-	for _, arg := range os.Args {
-		if strings.HasPrefix(arg, "-test.") ||
-			strings.HasPrefix(arg, "-httptest.") {
-			continue
-		}
-		runtimeArgs = append(runtimeArgs, arg)
-	}
-	os.Args = runtimeArgs
-}
-
-type dummyTestDeps func(pat, str string) (bool, error)
-
-func (d dummyTestDeps) MatchString(pat, str string) (bool, error)   { return false, nil }
-func (d dummyTestDeps) StartCPUProfile(io.Writer) error             { return nil }
-func (d dummyTestDeps) StopCPUProfile()                             {}
-func (f dummyTestDeps) StartTestLog(w io.Writer)                    {}
-func (f dummyTestDeps) StopTestLog() error                          { return nil }
-func (d dummyTestDeps) WriteHeapProfile(io.Writer) error            { return nil }
-func (d dummyTestDeps) WriteProfileTo(string, io.Writer, int) error { return nil }
-func (f dummyTestDeps) ImportPath() string                          { return "" }
-
-// FlushProfiles flushes test profiles to disk. It works by build and executing
-// a dummy list of 1 test. This is to ensure we execute the M.after() function
-// (a function internal to the testing package) where all reporting (cpu, mem,
-// cover, ... profiles) is flushed to disk.
-func FlushProfiles() {
-	// Redirect Stdout/err temporarily so the testing code doesn't output the
-	// regular:
-	//   PASS
-	//   coverage: 21.4% of statements
-	// Thanks to this, we can test the output of the instrumented binary the same
-	// way the normal binary is.
-	oldstdout := os.Stdout
-	oldstderr := os.Stderr
-	os.Stdout, _ = os.Open(os.DevNull)
-	os.Stderr, _ = os.Open(os.DevNull)
-
-	tests := []testing.InternalTest{}
-	benchmarks := []testing.InternalBenchmark{}
-	examples := []testing.InternalExample{}
-	var f dummyTestDeps
-	dummyM := testing.MainStart(f, tests, benchmarks, examples)
-	dummyM.Run()
-
-	// restore stdout/err
-	os.Stdout = oldstdout
-	os.Stderr = oldstderr
-}

src/runtime/vendor/github.com/kata-containers/govmm/qemu/qemu.go

@@ -134,6 +134,9 @@ const (
 
 	// Loader is the Loader device driver.
 	Loader DeviceDriver = "loader"
+
+	// SpaprTPMProxy is used for enabling guest to run in secure mode on ppc64le.
+	SpaprTPMProxy DeviceDriver = "spapr-tpm-proxy"
 )
 
 func isDimmSupported(config *Config) bool {
@@ -230,6 +233,14 @@ const (
 
 	// TDXGuest represents a TDX object
 	TDXGuest ObjectType = "tdx-guest"
+
+	// SEVGuest represents an SEV guest object
+	SEVGuest ObjectType = "sev-guest"
+
+	// SecExecGuest represents an s390x Secure Execution (Protected Virtualization in QEMU) object
+	SecExecGuest ObjectType = "s390-pv-guest"
+	// PEFGuest represent ppc64le PEF(Protected Execution Facility) object.
+	PEFGuest ObjectType = "pef-guest"
 )
 
 // Object is a qemu object representation.
@@ -258,37 +269,42 @@ type Object struct {
 
 	// File is the device file
 	File string
+
+	// CBitPos is the location of the C-bit in a guest page table entry
+	// This is only relevant for sev-guest objects
+	CBitPos uint32
+
+	// ReducedPhysBits is the reduction in the guest physical address space
+	// This is only relevant for sev-guest objects
+	ReducedPhysBits uint32
 }
 
 // Valid returns true if the Object structure is valid and complete.
 func (object Object) Valid() bool {
 	switch object.Type {
 	case MemoryBackendFile:
-		if object.ID == "" || object.MemPath == "" || object.Size == 0 {
-			return false
-		}
-
+		return object.ID != "" && object.MemPath != "" && object.Size != 0
 	case TDXGuest:
-		if object.ID == "" || object.File == "" || object.DeviceID == "" {
-			return false
-		}
+		return object.ID != "" && object.File != "" && object.DeviceID != ""
+	case SEVGuest:
+		return object.ID != "" && object.File != "" && object.CBitPos != 0 && object.ReducedPhysBits != 0
+	case SecExecGuest:
+		return object.ID != ""
+	case PEFGuest:
+		return object.ID != "" && object.File != ""
 
 	default:
 		return false
 	}
-
-	return true
 }
 
 // QemuParams returns the qemu parameters built out of this Object device.
 func (object Object) QemuParams(config *Config) []string {
 	var objectParams []string
 	var deviceParams []string
+	var driveParams []string
 	var qemuParams []string
 
-	deviceParams = append(deviceParams, string(object.Driver))
-	deviceParams = append(deviceParams, fmt.Sprintf(",id=%s", object.DeviceID))
-
 	switch object.Type {
 	case MemoryBackendFile:
 		objectParams = append(objectParams, string(object.Type))
@@ -296,6 +312,8 @@ func (object Object) QemuParams(config *Config) []string {
 		objectParams = append(objectParams, fmt.Sprintf(",mem-path=%s", object.MemPath))
 		objectParams = append(objectParams, fmt.Sprintf(",size=%d", object.Size))
 
+		deviceParams = append(deviceParams, string(object.Driver))
+		deviceParams = append(deviceParams, fmt.Sprintf(",id=%s", object.DeviceID))
 		deviceParams = append(deviceParams, fmt.Sprintf(",memdev=%s", object.ID))
 	case TDXGuest:
 		objectParams = append(objectParams, string(object.Type))
@@ -303,14 +321,44 @@ func (object Object) QemuParams(config *Config) []string {
 		if object.Debug {
 			objectParams = append(objectParams, ",debug=on")
 		}
+		deviceParams = append(deviceParams, string(object.Driver))
+		deviceParams = append(deviceParams, fmt.Sprintf(",id=%s", object.DeviceID))
 		deviceParams = append(deviceParams, fmt.Sprintf(",file=%s", object.File))
+	case SEVGuest:
+		objectParams = append(objectParams, string(object.Type))
+		objectParams = append(objectParams, fmt.Sprintf(",id=%s", object.ID))
+		objectParams = append(objectParams, fmt.Sprintf(",cbitpos=%d", object.CBitPos))
+		objectParams = append(objectParams, fmt.Sprintf(",reduced-phys-bits=%d", object.ReducedPhysBits))
+
+		driveParams = append(driveParams, "if=pflash,format=raw,readonly=on")
+		driveParams = append(driveParams, fmt.Sprintf(",file=%s", object.File))
+	case SecExecGuest:
+		objectParams = append(objectParams, string(object.Type))
+		objectParams = append(objectParams, fmt.Sprintf(",id=%s", object.ID))
+	case PEFGuest:
+		objectParams = append(objectParams, string(object.Type))
+		objectParams = append(objectParams, fmt.Sprintf(",id=%s", object.ID))
+
+		deviceParams = append(deviceParams, string(object.Driver))
+		deviceParams = append(deviceParams, fmt.Sprintf(",id=%s", object.DeviceID))
+		deviceParams = append(deviceParams, fmt.Sprintf(",host-path=%s", object.File))
+
 	}
 
-	qemuParams = append(qemuParams, "-device")
-	qemuParams = append(qemuParams, strings.Join(deviceParams, ""))
+	if len(deviceParams) > 0 {
+		qemuParams = append(qemuParams, "-device")
+		qemuParams = append(qemuParams, strings.Join(deviceParams, ""))
+	}
 
-	qemuParams = append(qemuParams, "-object")
-	qemuParams = append(qemuParams, strings.Join(objectParams, ""))
+	if len(objectParams) > 0 {
+		qemuParams = append(qemuParams, "-object")
+		qemuParams = append(qemuParams, strings.Join(objectParams, ""))
+	}
+
+	if len(driveParams) > 0 {
+		qemuParams = append(qemuParams, "-drive")
+		qemuParams = append(qemuParams, strings.Join(driveParams, ""))
+	}
 
 	return qemuParams
 }

src/runtime/vendor/modules.txt

@@ -145,7 +145,6 @@ github.com/cri-o/cri-o/pkg/annotations
 github.com/davecgh/go-spew/spew
 # github.com/dlespiau/covertool v0.0.0-20180314162135-b0c4c6d0583a
 ## explicit
-github.com/dlespiau/covertool/pkg/cover
 # github.com/docker/distribution v2.7.1+incompatible
 ## explicit
 github.com/docker/distribution/digestset
@@ -222,7 +221,7 @@ github.com/hashicorp/errwrap
 # github.com/hashicorp/go-multierror v1.0.0
 ## explicit
 github.com/hashicorp/go-multierror
-# github.com/kata-containers/govmm v0.0.0-20210428163604-f0e9a35308ee
+# github.com/kata-containers/govmm v0.0.0-20210520142420-eb57f004d89f
 ## explicit
 github.com/kata-containers/govmm/qemu
 # github.com/konsorten/go-windows-terminal-sequences v1.0.1

src/runtime/virtcontainers/acrn.go

@@ -214,8 +214,7 @@ func (a *Acrn) trace(parent context.Context, name string) (otelTrace.Span, conte
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes([]label.KeyValue{label.Key("subsystem").String("hypervisor"), label.Key("type").String("acrn")}...)
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(label.String("source", "runtime"), label.String("package", "virtcontainers"), label.String("subsystem", "hypervisor"), label.String("type", "acrn"), label.String("sandbox_id", a.id)))
 
 	return span, ctx
 }
@@ -679,6 +678,10 @@ func (a *Acrn) getPids() []int {
 	return []int{a.state.PID}
 }
 
+func (a *Acrn) getVirtioFsPid() *int {
+	return nil
+}
+
 func (a *Acrn) fromGrpc(ctx context.Context, hypervisorConfig *HypervisorConfig, j []byte) error {
 	return errors.New("acrn is not supported by VM cache")
 }

src/runtime/virtcontainers/api.go

@@ -30,11 +30,7 @@ var virtLog = logrus.WithField("source", "virtcontainers")
 // context.
 func trace(parent context.Context, name string) (otelTrace.Span, context.Context) {
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes([]label.KeyValue{
-		label.Key("source").String("virtcontainers"),
-		label.Key("component").String("virtcontainers"),
-		label.Key("subsystem").String("api")}...)
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(label.String("source", "runtime"), label.String("packages", "virtcontainers"), label.String("subsystem", "api")))
 
 	return span, ctx
 }
@@ -124,11 +120,6 @@ func createSandboxFromConfig(ctx context.Context, sandboxConfig SandboxConfig, f
 		return nil, err
 	}
 
-	// The sandbox is completely created now, we can store it.
-	if err = s.storeSandbox(ctx); err != nil {
-		return nil, err
-	}
-
 	return s, nil
 }
 

src/runtime/virtcontainers/bridgedmacvlan_endpoint.go

@@ -14,6 +14,8 @@ import (
 	vcTypes "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/types"
 )
 
+var macvlanTrace = getNetworkTrace(BridgedMacvlanEndpointType)
+
 // BridgedMacvlanEndpoint represents a macvlan endpoint that is bridged to the VM
 type BridgedMacvlanEndpoint struct {
 	NetPair            NetworkInterfacePair
@@ -89,6 +91,9 @@ func (endpoint *BridgedMacvlanEndpoint) NetworkPair() *NetworkInterfacePair {
 // Attach for virtual endpoint bridges the network pair and adds the
 // tap interface of the network pair to the hypervisor.
 func (endpoint *BridgedMacvlanEndpoint) Attach(ctx context.Context, s *Sandbox) error {
+	span, ctx := macvlanTrace(ctx, "Attach", endpoint)
+	defer span.End()
+
 	h := s.hypervisor
 	if err := xConnectVMNetwork(ctx, endpoint, h); err != nil {
 		networkLogger().WithError(err).Error("Error bridging virtual ep")
@@ -107,8 +112,11 @@ func (endpoint *BridgedMacvlanEndpoint) Detach(ctx context.Context, netNsCreated
 		return nil
 	}
 
+	span, ctx := macvlanTrace(ctx, "Detach", endpoint)
+	defer span.End()
+
 	return doNetNS(netNsPath, func(_ ns.NetNS) error {
-		return xDisconnectVMNetwork(endpoint)
+		return xDisconnectVMNetwork(ctx, endpoint)
 	})
 }
 

src/runtime/virtcontainers/clh.go

@@ -362,7 +362,9 @@ func (clh *cloudHypervisor) startSandbox(ctx context.Context, timeout int) error
 
 	if clh.config.SharedFS == config.VirtioFS {
 		clh.Logger().WithField("function", "startSandbox").Info("Starting virtiofsd")
-		pid, err := clh.virtiofsd.Start(ctx)
+		pid, err := clh.virtiofsd.Start(ctx, func() {
+			clh.stopSandbox(ctx, false)
+		})
 		if err != nil {
 			return err
 		}
@@ -712,6 +714,10 @@ func (clh *cloudHypervisor) getPids() []int {
 	return pids
 }
 
+func (clh *cloudHypervisor) getVirtioFsPid() *int {
+	return &clh.state.VirtiofsdPID
+}
+
 func (clh *cloudHypervisor) addDevice(ctx context.Context, devInfo interface{}, devType deviceType) error {
 	span, _ := clh.trace(ctx, "addDevice")
 	defer span.End()
@@ -764,8 +770,7 @@ func (clh *cloudHypervisor) trace(parent context.Context, name string) (otelTrac
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes([]otelLabel.KeyValue{otelLabel.Key("subsystem").String("hypervisor"), otelLabel.Key("type").String("clh")}...)
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(otelLabel.String("source", "runtime"), otelLabel.String("package", "virtcontainers"), otelLabel.String("subsystem", "hypervisor"), otelLabel.String("type", "clh"), otelLabel.String("sandbox_id", clh.id)))
 
 	return span, ctx
 }

src/runtime/virtcontainers/container.go

@@ -360,8 +360,7 @@ func (c *Container) trace(parent context.Context, name string) (otelTrace.Span,
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes(otelLabel.Key("subsystem").String("container"))
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(otelLabel.String("source", "runtime"), otelLabel.String("package", "virtcontainers"), otelLabel.String("subsystem", "container"), otelLabel.String("container_id", c.id)))
 
 	return span, ctx
 }

src/runtime/virtcontainers/doc.go

@@ -1,17 +0,0 @@
-// Copyright (c) 2016 Intel Corporation
-//
-// SPDX-License-Identifier: Apache-2.0
-//
-
-/*
-Package virtcontainers manages hardware virtualized containers.
-Each container belongs to a set of containers sharing the same networking
-namespace and storage, also known as a sandbox.
-
-Virtcontainers sandboxes are hardware virtualized, i.e. they run on virtual machines.
-Virtcontainers will create one VM per sandbox, and containers will be created as
-processes within the sandbox VM.
-
-The virtcontainers package manages both sandboxes and containers lifecycles.
-*/
-package virtcontainers

src/runtime/virtcontainers/factory/factory.go

@@ -42,8 +42,7 @@ type factory struct {
 
 func trace(parent context.Context, name string) (otelTrace.Span, context.Context) {
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes(label.Key("subsystem").String("factory"))
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(label.String("source", "runtime"), label.String("package", "factory"), label.String("subsystem", "factory")))
 
 	return span, ctx
 }

src/runtime/virtcontainers/factory/template/template.go

@@ -16,6 +16,7 @@ import (
 	pb "github.com/kata-containers/kata-containers/src/runtime/protocols/cache"
 	vc "github.com/kata-containers/kata-containers/src/runtime/virtcontainers"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/factory/base"
+	"github.com/sirupsen/logrus"
 )
 
 type template struct {
@@ -24,6 +25,7 @@ type template struct {
 }
 
 var templateWaitForAgent = 2 * time.Second
+var templateLog = logrus.WithField("source", "virtcontainers/factory/template")
 
 // Fetch finds and returns a pre-built template factory.
 // TODO: save template metadata and fetch from storage.
@@ -86,8 +88,13 @@ func (t *template) GetVMStatus() []*pb.GrpcVMStatus {
 }
 
 func (t *template) close() {
-	syscall.Unmount(t.statePath, 0)
-	os.RemoveAll(t.statePath)
+	if err := syscall.Unmount(t.statePath, syscall.MNT_DETACH); err != nil {
+		t.Logger().WithError(err).Errorf("failed to unmount %s", t.statePath)
+	}
+
+	if err := os.RemoveAll(t.statePath); err != nil {
+		t.Logger().WithError(err).Errorf("failed to remove %s", t.statePath)
+	}
 }
 
 func (t *template) prepareTemplateFiles() error {
@@ -168,3 +175,19 @@ func (t *template) checkTemplateVM() error {
 	_, err = os.Stat(t.statePath + "/state")
 	return err
 }
+
+// Logger returns a logrus logger appropriate for logging template messages
+func (t *template) Logger() *logrus.Entry {
+	return templateLog.WithFields(logrus.Fields{
+		"subsystem": "template",
+	})
+}
+
+// SetLogger sets the logger for the factory template.
+func SetLogger(ctx context.Context, logger logrus.FieldLogger) {
+	fields := logrus.Fields{
+		"source": "virtcontainers",
+	}
+
+	templateLog = logger.WithFields(fields)
+}

src/runtime/virtcontainers/factory/template/template_test.go

@@ -121,7 +121,14 @@ func TestTemplateFactory(t *testing.T) {
 	err = vm.Stop(ctx)
 	assert.Nil(err)
 
-	// CloseFactory
+	// make tt.statePath is busy
+	os.Chdir(tt.statePath)
+
+	// CloseFactory, there is no need to call tt.CloseFactory(ctx)
 	f.CloseFactory(ctx)
-	tt.CloseFactory(ctx)
+
+	// expect tt.statePath not exist, if exist, it means this case failed.
+	_, err = os.Stat(tt.statePath)
+	assert.Error(err)
+	assert.True(os.IsNotExist(err))
 }

src/runtime/virtcontainers/fc.go

@@ -175,8 +175,7 @@ func (fc *firecracker) trace(parent context.Context, name string) (otelTrace.Spa
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes([]otelLabel.KeyValue{otelLabel.Key("subsystem").String("hypervisor"), otelLabel.Key("type").String("firecracker")}...)
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(otelLabel.String("source", "runtime"), otelLabel.String("package", "virtcontainers"), otelLabel.String("subsystem", "hypervisor"), otelLabel.String("type", "firecracker"), otelLabel.String("sandbox_id", fc.id)))
 
 	return span, ctx
 }
@@ -1181,6 +1180,10 @@ func (fc *firecracker) getPids() []int {
 	return []int{fc.info.PID}
 }
 
+func (fc *firecracker) getVirtioFsPid() *int {
+	return nil
+}
+
 func (fc *firecracker) fromGrpc(ctx context.Context, hypervisorConfig *HypervisorConfig, j []byte) error {
 	return errors.New("firecracker is not supported by VM cache")
 }

src/runtime/virtcontainers/hypervisor.go

@@ -453,6 +453,11 @@ type HypervisorConfig struct {
 	// GuestMemoryDumpPaging is used to indicate if enable paging
 	// for QEMU dump-guest-memory command
 	GuestMemoryDumpPaging bool
+
+	// Enable confidential guest support.
+	// Enable or disable different hardware features, ranging
+	// from memory encryption to both memory and CPU-state encryption and integrity.
+	ConfidentialGuest bool
 }
 
 // vcpu mapping from vcpu number to thread number
@@ -717,21 +722,61 @@ func getHostMemorySizeKb(memInfoPath string) (uint64, error) {
 	return 0, fmt.Errorf("unable get MemTotal from %s", memInfoPath)
 }
 
-// RunningOnVMM checks if the system is running inside a VM.
-func RunningOnVMM(cpuInfoPath string) (bool, error) {
-	if runtime.GOARCH == "arm64" || runtime.GOARCH == "ppc64le" || runtime.GOARCH == "s390x" {
-		virtLog.Info("Unable to know if the system is running inside a VM")
-		return false, nil
-	}
-
-	flagsField := "flags"
-
-	f, err := os.Open(cpuInfoPath)
+// CheckCmdline checks whether an option or parameter is present in the kernel command line.
+// Search is case-insensitive.
+// Takes path to file that contains the kernel command line, desired option, and permitted values
+// (empty values to check for options).
+func CheckCmdline(kernelCmdlinePath, searchParam string, searchValues []string) (bool, error) {
+	f, err := os.Open(kernelCmdlinePath)
 	if err != nil {
 		return false, err
 	}
 	defer f.Close()
 
+	// Create check function -- either check for verbatim option
+	// or check for parameter and permitted values
+	var check func(string, string, []string) bool
+	if len(searchValues) == 0 {
+		check = func(option, searchParam string, _ []string) bool {
+			return strings.EqualFold(option, searchParam)
+		}
+	} else {
+		check = func(param, searchParam string, searchValues []string) bool {
+			// split parameter and value
+			split := strings.SplitN(param, "=", 2)
+			if len(split) < 2 || split[0] != searchParam {
+				return false
+			}
+			for _, value := range searchValues {
+				if strings.EqualFold(value, split[1]) {
+					return true
+				}
+			}
+			return false
+		}
+	}
+
+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		for _, field := range strings.Fields(scanner.Text()) {
+			if check(field, searchParam, searchValues) {
+				return true, nil
+			}
+		}
+	}
+	return false, err
+}
+
+func CPUFlags(cpuInfoPath string) (map[string]bool, error) {
+	flagsField := "flags"
+
+	f, err := os.Open(cpuInfoPath)
+	if err != nil {
+		return map[string]bool{}, err
+	}
+	defer f.Close()
+
+	flags := make(map[string]bool)
 	scanner := bufio.NewScanner(f)
 	for scanner.Scan() {
 		// Expected format: ["flags", ":", ...] or ["flags:", ...]
@@ -745,23 +790,31 @@ func RunningOnVMM(cpuInfoPath string) (bool, error) {
 		}
 
 		for _, field := range fields[1:] {
-			if field == "hypervisor" {
-				return true, nil
-			}
+			flags[field] = true
 		}
 
-		// As long as we have been able to analyze the fields from
-		// "flags", there is no reason to check what comes next from
-		// /proc/cpuinfo, because we already know we are not running
-		// on a VMM.
-		return false, nil
+		return flags, nil
 	}
 
 	if err := scanner.Err(); err != nil {
-		return false, err
+		return map[string]bool{}, err
 	}
 
-	return false, fmt.Errorf("Couldn't find %q from %q output", flagsField, cpuInfoPath)
+	return map[string]bool{}, fmt.Errorf("Couldn't find %q from %q output", flagsField, cpuInfoPath)
+}
+
+// RunningOnVMM checks if the system is running inside a VM.
+func RunningOnVMM(cpuInfoPath string) (bool, error) {
+	if runtime.GOARCH == "amd64" {
+		flags, err := CPUFlags(cpuInfoPath)
+		if err != nil {
+			return false, err
+		}
+		return flags["hypervisor"], nil
+	}
+
+	virtLog.WithField("arch", runtime.GOARCH).Info("Unable to know if the system is running inside a VM")
+	return false, nil
 }
 
 func getHypervisorPid(h hypervisor) int {
@@ -810,6 +863,7 @@ type hypervisor interface {
 	// getPids returns a slice of hypervisor related process ids.
 	// The hypervisor pid must be put at index 0.
 	getPids() []int
+	getVirtioFsPid() *int
 	fromGrpc(ctx context.Context, hypervisorConfig *HypervisorConfig, j []byte) error
 	toGrpc(ctx context.Context) ([]byte, error)
 	check() error

src/runtime/virtcontainers/hypervisor_amd64.go

@@ -0,0 +1,25 @@
+// Copyright (c) 2021 Intel Corporation
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+package virtcontainers
+
+import "os"
+
+// Implementation of this function is architecture specific
+func availableGuestProtection() (guestProtection, error) {
+	flags, err := CPUFlags(procCPUInfo)
+	if err != nil {
+		return noneProtection, err
+	}
+
+	// TDX is supported and properly loaded when the firmware directory exists or `tdx` is part of the CPU flags
+	if d, err := os.Stat(tdxSysFirmwareDir); (err == nil && d.IsDir()) || flags[tdxCPUFlag] {
+		return tdxProtection, nil
+	}
+
+	// TODO: Add support for other technologies: SEV
+
+	return noneProtection, nil
+}

src/runtime/virtcontainers/hypervisor_ppc64le.go

@@ -0,0 +1,17 @@
+// Copyright (c) 2021 IBM
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package virtcontainers
+
+import "os"
+
+//Returns pefProtection if the firmware directory exists
+func availableGuestProtection() (guestProtection, error) {
+
+	if d, err := os.Stat(pefSysFirmwareDir); err == nil && d.IsDir() {
+		return pefProtection, err
+	}
+
+	return noneProtection, nil
+}

src/runtime/virtcontainers/hypervisor_s390x.go

@@ -0,0 +1,99 @@
+// Copyright (c) IBM Corp. 2021
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+package virtcontainers
+
+import (
+	"bufio"
+	"fmt"
+	"os"
+	"strconv"
+	"strings"
+)
+
+const (
+	// This is valid in other architectures, but varcheck will complain
+	// when setting it in common code as it will be regarded unused
+	procKernelCmdline = "/proc/cmdline"
+
+	// Secure Execution
+	// https://www.kernel.org/doc/html/latest/virt/kvm/s390-pv.html
+	seCPUFacilityBit = 158
+	seCmdlineParam   = "prot_virt"
+)
+
+var seCmdlineValues = []string{
+	"1", "on", "y", "yes",
+}
+
+// CPUFacilities retrieves active CPU facilities according to "Facility Indications", Principles of Operation.
+// Takes cpuinfo path (such as /proc/cpuinfo), returns map of all active facility bits.
+func CPUFacilities(cpuInfoPath string) (map[int]bool, error) {
+	facilitiesField := "facilities"
+
+	f, err := os.Open(cpuInfoPath)
+	if err != nil {
+		return map[int]bool{}, err
+	}
+	defer f.Close()
+
+	facilities := make(map[int]bool)
+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		// Expected format: ["facilities", ":", ...] or ["facilities:", ...]
+		fields := strings.Fields(scanner.Text())
+		if len(fields) < 2 {
+			continue
+		}
+
+		if !strings.HasPrefix(fields[0], facilitiesField) {
+			continue
+		}
+
+		start := 1
+		if fields[1] == ":" {
+			start = 2
+		}
+		for _, field := range fields[start:] {
+			bit, err := strconv.Atoi(field)
+			if err != nil {
+				return map[int]bool{}, err
+			}
+			facilities[bit] = true
+		}
+
+		return facilities, nil
+	}
+
+	if err := scanner.Err(); err != nil {
+		return map[int]bool{}, err
+	}
+
+	return map[int]bool{}, fmt.Errorf("Couldn't find %q from %q output", facilitiesField, cpuInfoPath)
+}
+
+// availableGuestProtection returns seProtection (Secure Execution) if available.
+// Checks that Secure Execution is available (CPU facilities) and enabled (kernel command line).
+func availableGuestProtection() (guestProtection, error) {
+	facilities, err := CPUFacilities(procCPUInfo)
+	if err != nil {
+		return noneProtection, err
+	}
+	if !facilities[seCPUFacilityBit] {
+		return noneProtection, fmt.Errorf("This CPU does not support Secure Execution")
+	}
+
+	seCmdlinePresent, err := CheckCmdline(procKernelCmdline, seCmdlineParam, seCmdlineValues)
+	if err != nil {
+		return noneProtection, err
+	}
+	if !seCmdlinePresent {
+		return noneProtection, fmt.Errorf("Protected Virtualization is not enabled on kernel command line! " +
+			"Need %s=%s (or %s) to enable Secure Execution",
+			seCmdlineParam, seCmdlineValues[0], strings.Join(seCmdlineValues[1:], ", "))
+	}
+
+	return seProtection, nil
+}

src/runtime/virtcontainers/hypervisor_s390x_test.go

@@ -0,0 +1,25 @@
+// Copyright (c) IBM Corp. 2021
+//
+// SPDX-License-Identifier: Apache-2.0
+//
+
+package virtcontainers
+
+import (
+	"math"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestCPUFacilities(t *testing.T) {
+	assert := assert.New(t)
+
+	facilities, err := CPUFacilities(procCPUInfo)
+	assert.NoError(err)
+
+	// z/Architecture facility should always be active (introduced in 2000)
+	assert.Equal(facilities[1], true)
+	// facility bits should not be as high as MaxInt
+	assert.Equal(facilities[math.MaxInt64], false)
+}

src/runtime/virtcontainers/hypervisor_test.go

@@ -401,6 +401,21 @@ func TestGetHostMemorySizeKb(t *testing.T) {
 	}
 }
 
+func TestCheckCmdline(t *testing.T) {
+	assert := assert.New(t)
+
+	cmdlineFp, err := ioutil.TempFile("", "")
+	assert.NoError(err)
+	_, err = cmdlineFp.WriteString("quiet root=/dev/sda2")
+	assert.NoError(err)
+	cmdlinePath := cmdlineFp.Name()
+	defer os.Remove(cmdlinePath)
+
+	assert.True(CheckCmdline(cmdlinePath, "quiet", []string{}))
+	assert.True(CheckCmdline(cmdlinePath, "root", []string{"/dev/sda1", "/dev/sda2"}))
+	assert.False(CheckCmdline(cmdlinePath, "ro", []string{}))
+}
+
 // nolint: unused, deadcode
 type testNestedVMMData struct {
 	content     []byte

src/runtime/virtcontainers/ipvlan_endpoint.go

@@ -14,6 +14,8 @@ import (
 	vcTypes "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/types"
 )
 
+var ipvlanTrace = getNetworkTrace(IPVlanEndpointType)
+
 // IPVlanEndpoint represents a ipvlan endpoint that is bridged to the VM
 type IPVlanEndpoint struct {
 	NetPair            NetworkInterfacePair
@@ -92,6 +94,9 @@ func (endpoint *IPVlanEndpoint) NetworkPair() *NetworkInterfacePair {
 // Attach for virtual endpoint bridges the network pair and adds the
 // tap interface of the network pair to the hypervisor.
 func (endpoint *IPVlanEndpoint) Attach(ctx context.Context, s *Sandbox) error {
+	span, ctx := ipvlanTrace(ctx, "Attach", endpoint)
+	defer span.End()
+
 	h := s.hypervisor
 	if err := xConnectVMNetwork(ctx, endpoint, h); err != nil {
 		networkLogger().WithError(err).Error("Error bridging virtual ep")
@@ -110,8 +115,11 @@ func (endpoint *IPVlanEndpoint) Detach(ctx context.Context, netNsCreated bool, n
 		return nil
 	}
 
+	span, ctx := ipvlanTrace(ctx, "Detach", endpoint)
+	defer span.End()
+
 	return doNetNS(netNsPath, func(_ ns.NetNS) error {
-		return xDisconnectVMNetwork(endpoint)
+		return xDisconnectVMNetwork(ctx, endpoint)
 	})
 }
 

src/runtime/virtcontainers/kata_agent.go

@@ -251,8 +251,7 @@ func (k *kataAgent) trace(parent context.Context, name string) (otelTrace.Span,
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes([]label.KeyValue{label.Key("subsystem").String("agent"), label.Key("type").String("kata")}...)
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(label.String("source", "runtime"), label.String("package", "virtcontainers"), label.String("subsystem", "agent")))
 
 	return span, ctx
 }
@@ -383,7 +382,7 @@ func (k *kataAgent) internalConfigure(h hypervisor, id string, config KataAgentC
 	return nil
 }
 
-func setupSandboxBindMounts(sandbox *Sandbox) error {
+func (k *kataAgent) setupSandboxBindMounts(sandbox *Sandbox) (err error) {
 	if len(sandbox.config.SandboxBindMounts) == 0 {
 		return nil
 	}
@@ -394,6 +393,20 @@ func setupSandboxBindMounts(sandbox *Sandbox) error {
 	if err := os.MkdirAll(sandboxMountDir, DirMode); err != nil {
 		return fmt.Errorf("Creating sandbox shared mount directory: %v: %w", sandboxMountDir, err)
 	}
+	var mountedList []string
+	defer func() {
+		if err != nil {
+			for _, mnt := range mountedList {
+				if derr := syscall.Unmount(mnt, syscall.MNT_DETACH|UmountNoFollow); derr != nil {
+					k.Logger().WithError(derr).Errorf("cleanup: couldn't unmount %s", mnt)
+				}
+			}
+			if derr := os.RemoveAll(sandboxMountDir); derr != nil {
+				k.Logger().WithError(derr).Errorf("cleanup: failed to remove %s", sandboxMountDir)
+			}
+
+		}
+	}()
 
 	for _, m := range sandbox.config.SandboxBindMounts {
 		mountDest := filepath.Join(sandboxMountDir, filepath.Base(m))
@@ -401,6 +414,7 @@ func setupSandboxBindMounts(sandbox *Sandbox) error {
 		if err := bindMount(context.Background(), m, mountDest, true, "private"); err != nil {
 			return fmt.Errorf("Mounting sandbox directory: %v to %v: %w", m, mountDest, err)
 		}
+		mountedList = append(mountedList, mountDest)
 
 		mountDest = filepath.Join(sandboxShareDir, filepath.Base(m))
 		if err := remountRo(context.Background(), mountDest); err != nil {
@@ -412,19 +426,30 @@ func setupSandboxBindMounts(sandbox *Sandbox) error {
 	return nil
 }
 
-func cleanupSandboxBindMounts(sandbox *Sandbox) error {
+func (k *kataAgent) cleanupSandboxBindMounts(sandbox *Sandbox) error {
 	if sandbox.config == nil || len(sandbox.config.SandboxBindMounts) == 0 {
 		return nil
 	}
 
+	var retErr error
+	bindmountShareDir := filepath.Join(getMountPath(sandbox.id), sandboxMountsDir)
 	for _, m := range sandbox.config.SandboxBindMounts {
-		mountPath := filepath.Join(getMountPath(sandbox.id), sandboxMountsDir, filepath.Base(m))
+		mountPath := filepath.Join(bindmountShareDir, filepath.Base(m))
 		if err := syscall.Unmount(mountPath, syscall.MNT_DETACH|UmountNoFollow); err != nil {
-			return fmt.Errorf("Unmounting observe directory: %v: %w", mountPath, err)
+			if retErr == nil {
+				retErr = err
+			}
+			k.Logger().WithError(err).Errorf("Failed to unmount sandbox bindmount: %v", mountPath)
 		}
 	}
+	if err := os.RemoveAll(bindmountShareDir); err != nil {
+		if retErr == nil {
+			retErr = err
+		}
+		k.Logger().WithError(err).Errorf("Failed to remove sandbox bindmount directory: %s", bindmountShareDir)
+	}
 
-	return nil
+	return retErr
 }
 
 func (k *kataAgent) configure(ctx context.Context, h hypervisor, id, sharePath string, config KataAgentConfig) error {
@@ -473,7 +498,7 @@ func (k *kataAgent) configureFromGrpc(h hypervisor, id string, config KataAgentC
 	return k.internalConfigure(h, id, config)
 }
 
-func (k *kataAgent) setupSharedPath(ctx context.Context, sandbox *Sandbox) error {
+func (k *kataAgent) setupSharedPath(ctx context.Context, sandbox *Sandbox) (err error) {
 	// create shared path structure
 	sharePath := getSharePath(sandbox.id)
 	mountPath := getMountPath(sandbox.id)
@@ -488,9 +513,16 @@ func (k *kataAgent) setupSharedPath(ctx context.Context, sandbox *Sandbox) error
 	if err := bindMount(ctx, mountPath, sharePath, true, "slave"); err != nil {
 		return err
 	}
+	defer func() {
+		if err != nil {
+			if umountErr := syscall.Unmount(sharePath, syscall.MNT_DETACH|UmountNoFollow); umountErr != nil {
+				k.Logger().WithError(umountErr).Errorf("failed to unmount vm share path %s", sharePath)
+			}
+		}
+	}()
 
 	// Setup sandbox bindmounts, if specified:
-	if err := setupSandboxBindMounts(sandbox); err != nil {
+	if err = k.setupSandboxBindMounts(sandbox); err != nil {
 		return err
 	}
 
@@ -779,10 +811,8 @@ func (k *kataAgent) startSandbox(ctx context.Context, sandbox *Sandbox) error {
 		return err
 	}
 
-	//
 	// Setup network interfaces and routes
-	//
-	interfaces, routes, neighs, err := generateVCNetworkStructures(sandbox.networkNS)
+	interfaces, routes, neighs, err := generateVCNetworkStructures(ctx, sandbox.networkNS)
 	if err != nil {
 		return err
 	}
@@ -2149,8 +2179,8 @@ func (k *kataAgent) markDead(ctx context.Context) {
 }
 
 func (k *kataAgent) cleanup(ctx context.Context, s *Sandbox) {
-	if err := cleanupSandboxBindMounts(s); err != nil {
-		k.Logger().WithError(err).Errorf("failed to cleanup observability logs bindmount")
+	if err := k.cleanupSandboxBindMounts(s); err != nil {
+		k.Logger().WithError(err).Errorf("failed to cleanup sandbox bindmounts")
 	}
 
 	// Unmount shared path

src/runtime/virtcontainers/kata_agent_test.go

@@ -1216,3 +1216,116 @@ func TestKataAgentDirs(t *testing.T) {
 		assert.Equal(ephemeralPath(), defaultEphemeralPath)
 	}
 }
+
+func TestSandboxBindMount(t *testing.T) {
+	if os.Getuid() != 0 {
+		t.Skip("Test disabled as requires root user")
+	}
+
+	assert := assert.New(t)
+	// create temporary files to mount:
+	testMountPath, err := ioutil.TempDir("", "sandbox-test")
+	assert.NoError(err)
+	defer os.RemoveAll(testMountPath)
+
+	// create a new shared directory for our test:
+	kataHostSharedDirSaved := kataHostSharedDir
+	testHostDir, err := ioutil.TempDir("", "kata-cleanup")
+	assert.NoError(err)
+	kataHostSharedDir = func() string {
+		return testHostDir
+	}
+	defer func() {
+		kataHostSharedDir = kataHostSharedDirSaved
+	}()
+
+	m1Path := filepath.Join(testMountPath, "foo.txt")
+	f1, err := os.Create(m1Path)
+	assert.NoError(err)
+	defer f1.Close()
+
+	m2Path := filepath.Join(testMountPath, "bar.txt")
+	f2, err := os.Create(m2Path)
+	assert.NoError(err)
+	defer f2.Close()
+
+	// create sandbox for mounting into
+	sandbox := &Sandbox{
+		ctx: context.Background(),
+		id:  "foobar",
+		config: &SandboxConfig{
+			SandboxBindMounts: []string{m1Path, m2Path},
+		},
+	}
+	k := &kataAgent{ctx: context.Background()}
+
+	// make the shared directory for our test:
+	dir := kataHostSharedDir()
+	err = os.MkdirAll(path.Join(dir, sandbox.id), 0777)
+	assert.Nil(err)
+	defer os.RemoveAll(dir)
+
+	sharePath := getSharePath(sandbox.id)
+	mountPath := getMountPath(sandbox.id)
+
+	err = os.MkdirAll(sharePath, DirMode)
+	assert.Nil(err)
+	err = os.MkdirAll(mountPath, DirMode)
+	assert.Nil(err)
+
+	// setup the expeted slave mount:
+	err = bindMount(sandbox.ctx, mountPath, sharePath, true, "slave")
+	assert.Nil(err)
+	defer syscall.Unmount(sharePath, syscall.MNT_DETACH|UmountNoFollow)
+
+	// Test the function. We expect it to succeed and for the mount to exist
+	err = k.setupSandboxBindMounts(sandbox)
+	assert.NoError(err)
+
+	// Test the cleanup function. We expect it to succeed for the mount to be removed.
+	err = k.cleanupSandboxBindMounts(sandbox)
+	assert.NoError(err)
+
+	// After successful cleanup, verify there are not any mounts left behind.
+	stat := syscall.Stat_t{}
+	mount1CheckPath := filepath.Join(getMountPath(sandbox.id), sandboxMountsDir, filepath.Base(m1Path))
+	err = syscall.Stat(mount1CheckPath, &stat)
+	assert.Error(err)
+	assert.True(os.IsNotExist(err))
+
+	mount2CheckPath := filepath.Join(getMountPath(sandbox.id), sandboxMountsDir, filepath.Base(m2Path))
+	err = syscall.Stat(mount2CheckPath, &stat)
+	assert.Error(err)
+	assert.True(os.IsNotExist(err))
+
+	// Now, let's setup the cleanup to fail. Setup the sandbox bind mount twice, which will result in
+	// extra mounts being present that the sandbox description doesn't account for (ie, duplicate mounts).
+	// We expect cleanup to fail on the first time, since it cannot remove the sandbox-bindmount directory because
+	// there are leftover mounts.   If we run it a second time, however, it should succeed since it'll remove the
+	// second set of mounts:
+	err = k.setupSandboxBindMounts(sandbox)
+	assert.NoError(err)
+	err = k.setupSandboxBindMounts(sandbox)
+	assert.NoError(err)
+	// Test the cleanup function. We expect it to succeed for the mount to be removed.
+	err = k.cleanupSandboxBindMounts(sandbox)
+	assert.Error(err)
+	err = k.cleanupSandboxBindMounts(sandbox)
+	assert.NoError(err)
+
+	//
+	// Now, let's setup the sandbox bindmount to fail, and verify that no mounts are left behind
+	//
+	sandbox.config.SandboxBindMounts = append(sandbox.config.SandboxBindMounts, "oh-nos")
+	err = k.setupSandboxBindMounts(sandbox)
+	assert.Error(err)
+	// Verify there aren't any mounts left behind
+	stat = syscall.Stat_t{}
+	err = syscall.Stat(mount1CheckPath, &stat)
+	assert.Error(err)
+	assert.True(os.IsNotExist(err))
+	err = syscall.Stat(mount2CheckPath, &stat)
+	assert.Error(err)
+	assert.True(os.IsNotExist(err))
+
+}

src/runtime/virtcontainers/macvtap_endpoint.go

@@ -14,6 +14,8 @@ import (
 	vcTypes "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/types"
 )
 
+var macvtapTrace = getNetworkTrace(MacvtapEndpointType)
+
 // MacvtapEndpoint represents a macvtap endpoint
 type MacvtapEndpoint struct {
 	EndpointProperties NetworkInfo
@@ -62,6 +64,9 @@ func (endpoint *MacvtapEndpoint) SetProperties(properties NetworkInfo) {
 // Attach for macvtap endpoint passes macvtap device to the hypervisor.
 func (endpoint *MacvtapEndpoint) Attach(ctx context.Context, s *Sandbox) error {
 	var err error
+	span, ctx := macvtapTrace(ctx, "Attach", endpoint)
+	defer span.End()
+
 	h := s.hypervisor
 
 	endpoint.VMFds, err = createMacvtapFds(endpoint.EndpointProperties.Iface.Index, int(h.hypervisorConfig().NumVCPUs))

src/runtime/virtcontainers/mock_hypervisor.go

@@ -109,6 +109,10 @@ func (m *mockHypervisor) getPids() []int {
 	return []int{m.mockPid}
 }
 
+func (m *mockHypervisor) getVirtioFsPid() *int {
+	return nil
+}
+
 func (m *mockHypervisor) fromGrpc(ctx context.Context, hypervisorConfig *HypervisorConfig, j []byte) error {
 	return errors.New("mockHypervisor is not supported by VM cache")
 }

src/runtime/virtcontainers/network.go

@@ -19,7 +19,6 @@ import (
 	"time"
 
 	"github.com/containernetworking/plugins/pkg/ns"
-	"github.com/containernetworking/plugins/pkg/testutils"
 	"github.com/sirupsen/logrus"
 	"github.com/vishvananda/netlink"
 	"github.com/vishvananda/netns"
@@ -58,7 +57,7 @@ const (
 	NetXConnectInvalidModel
 )
 
-//IsValid checks if a model is valid
+// IsValid checks if a model is valid
 func (n NetInterworkingModel) IsValid() bool {
 	return 0 <= int(n) && int(n) < int(NetXConnectInvalidModel)
 }
@@ -73,6 +72,21 @@ const (
 	noneNetModelStr = "none"
 )
 
+//GetModel returns the string value of a NetInterworkingModel
+func (n *NetInterworkingModel) GetModel() string {
+	switch *n {
+	case DefaultNetInterworkingModel:
+		return defaultNetModelStr
+	case NetXConnectMacVtapModel:
+		return macvtapNetModelStr
+	case NetXConnectTCFilterModel:
+		return tcFilterNetModelStr
+	case NetXConnectNoneModel:
+		return noneNetModelStr
+	}
+	return "unknown"
+}
+
 //SetModel change the model string value
 func (n *NetInterworkingModel) SetModel(modelName string) error {
 	switch modelName {
@@ -407,6 +421,11 @@ func getLinkByName(netHandle *netlink.Handle, name string, expectedLink netlink.
 
 // The endpoint type should dictate how the connection needs to happen.
 func xConnectVMNetwork(ctx context.Context, endpoint Endpoint, h hypervisor) error {
+	var err error
+
+	span, ctx := networkTrace(ctx, "xConnectVMNetwork", endpoint)
+	defer closeSpan(span, err)
+
 	netPair := endpoint.NetworkPair()
 
 	queues := 0
@@ -428,16 +447,24 @@ func xConnectVMNetwork(ctx context.Context, endpoint Endpoint, h hypervisor) err
 
 	switch netPair.NetInterworkingModel {
 	case NetXConnectMacVtapModel:
-		return tapNetworkPair(endpoint, queues, disableVhostNet)
+		networkLogger().Info("connect macvtap to VM network")
+		err = tapNetworkPair(ctx, endpoint, queues, disableVhostNet)
 	case NetXConnectTCFilterModel:
-		return setupTCFiltering(endpoint, queues, disableVhostNet)
+		networkLogger().Info("connect TCFilter to VM network")
+		err = setupTCFiltering(ctx, endpoint, queues, disableVhostNet)
 	default:
-		return fmt.Errorf("Invalid internetworking model")
+		err = fmt.Errorf("Invalid internetworking model")
 	}
+	return err
 }
 
 // The endpoint type should dictate how the disconnection needs to happen.
-func xDisconnectVMNetwork(endpoint Endpoint) error {
+func xDisconnectVMNetwork(ctx context.Context, endpoint Endpoint) error {
+	var err error
+
+	span, ctx := networkTrace(ctx, "xDisconnectVMNetwork", endpoint)
+	defer closeSpan(span, err)
+
 	netPair := endpoint.NetworkPair()
 
 	if netPair.NetInterworkingModel == NetXConnectDefaultModel {
@@ -446,12 +473,13 @@ func xDisconnectVMNetwork(endpoint Endpoint) error {
 
 	switch netPair.NetInterworkingModel {
 	case NetXConnectMacVtapModel:
-		return untapNetworkPair(endpoint)
+		err = untapNetworkPair(ctx, endpoint)
 	case NetXConnectTCFilterModel:
-		return removeTCFiltering(endpoint)
+		err = removeTCFiltering(ctx, endpoint)
 	default:
-		return fmt.Errorf("Invalid internetworking model")
+		err = fmt.Errorf("Invalid internetworking model")
 	}
+	return err
 }
 
 func createMacvtapFds(linkIndex int, queues int) ([]*os.File, error) {
@@ -533,7 +561,10 @@ func setIPs(link netlink.Link, addrs []netlink.Addr) error {
 	return nil
 }
 
-func tapNetworkPair(endpoint Endpoint, queues int, disableVhostNet bool) error {
+func tapNetworkPair(ctx context.Context, endpoint Endpoint, queues int, disableVhostNet bool) error {
+	span, _ := networkTrace(ctx, "tapNetworkPair", endpoint)
+	defer span.End()
+
 	netHandle, err := netlink.NewHandle()
 	if err != nil {
 		return err
@@ -627,7 +658,10 @@ func tapNetworkPair(endpoint Endpoint, queues int, disableVhostNet bool) error {
 	return nil
 }
 
-func setupTCFiltering(endpoint Endpoint, queues int, disableVhostNet bool) error {
+func setupTCFiltering(ctx context.Context, endpoint Endpoint, queues int, disableVhostNet bool) error {
+	span, _ := networkTrace(ctx, "setupTCFiltering", endpoint)
+	defer span.End()
+
 	netHandle, err := netlink.NewHandle()
 	if err != nil {
 		return err
@@ -696,7 +730,7 @@ func setupTCFiltering(endpoint Endpoint, queues int, disableVhostNet bool) error
 	return nil
 }
 
-// addQdiscIngress creates a new qdisc for nwtwork interface with the specified network index
+// addQdiscIngress creates a new qdisc for network interface with the specified network index
 // on "ingress". qdiscs normally don't work on ingress so this is really a special qdisc
 // that you can consider an "alternate root" for inbound packets.
 // Handle for ingress qdisc defaults to "ffff:"
@@ -799,7 +833,10 @@ func removeQdiscIngress(link netlink.Link) error {
 	return nil
 }
 
-func untapNetworkPair(endpoint Endpoint) error {
+func untapNetworkPair(ctx context.Context, endpoint Endpoint) error {
+	span, _ := networkTrace(ctx, "untapNetworkPair", endpoint)
+	defer span.End()
+
 	netHandle, err := netlink.NewHandle()
 	if err != nil {
 		return err
@@ -840,7 +877,10 @@ func untapNetworkPair(endpoint Endpoint) error {
 	return err
 }
 
-func removeTCFiltering(endpoint Endpoint) error {
+func removeTCFiltering(ctx context.Context, endpoint Endpoint) error {
+	span, _ := networkTrace(ctx, "removeTCFiltering", endpoint)
+	defer span.End()
+
 	netHandle, err := netlink.NewHandle()
 	if err != nil {
 		return err
@@ -882,15 +922,6 @@ func removeTCFiltering(endpoint Endpoint) error {
 	return nil
 }
 
-func createNetNS() (string, error) {
-	n, err := testutils.NewNS()
-	if err != nil {
-		return "", err
-	}
-
-	return n.Path(), nil
-}
-
 // doNetNS is free from any call to a go routine, and it calls
 // into runtime.LockOSThread(), meaning it won't be executed in a
 // different thread than the one expected by the caller.
@@ -945,18 +976,18 @@ func deleteNetNS(netNSPath string) error {
 	return nil
 }
 
-func generateVCNetworkStructures(networkNS NetworkNamespace) ([]*pbTypes.Interface, []*pbTypes.Route, []*pbTypes.ARPNeighbor, error) {
-
+func generateVCNetworkStructures(ctx context.Context, networkNS NetworkNamespace) ([]*pbTypes.Interface, []*pbTypes.Route, []*pbTypes.ARPNeighbor, error) {
 	if networkNS.NetNsPath == "" {
 		return nil, nil, nil, nil
 	}
+	span, _ := networkTrace(ctx, "generateVCNetworkStructures", nil)
+	defer span.End()
 
 	var routes []*pbTypes.Route
 	var ifaces []*pbTypes.Interface
 	var neighs []*pbTypes.ARPNeighbor
 
 	for _, endpoint := range networkNS.Endpoints {
-
 		var ipAddresses []*pbTypes.IPAddress
 		for _, addr := range endpoint.Properties().Addrs {
 			// Skip localhost interface
@@ -1041,6 +1072,7 @@ func generateVCNetworkStructures(networkNS NetworkNamespace) ([]*pbTypes.Interfa
 			neighs = append(neighs, &n)
 		}
 	}
+
 	return ifaces, routes, neighs, nil
 }
 
@@ -1238,8 +1270,10 @@ func createEndpoint(netInfo NetworkInfo, idx int, model NetInterworkingModel, li
 				}
 			}
 		} else if netInfo.Iface.Type == "veth" {
+			networkLogger().Info("veth interface found")
 			endpoint, err = createVethNetworkEndpoint(idx, netInfo.Iface.Name, model)
 		} else if netInfo.Iface.Type == "ipvlan" {
+			networkLogger().Info("ipvlan interface found")
 			endpoint, err = createIPVlanNetworkEndpoint(idx, netInfo.Iface.Name)
 		} else {
 			return nil, fmt.Errorf("Unsupported network interface: %s", netInfo.Iface.Type)
@@ -1253,12 +1287,31 @@ func createEndpoint(netInfo NetworkInfo, idx int, model NetInterworkingModel, li
 type Network struct {
 }
 
-func (n *Network) trace(ctx context.Context, name string) (otelTrace.Span, context.Context) {
-	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(ctx, name)
-	span.SetAttributes([]otelLabel.KeyValue{otelLabel.Key("subsystem").String("network"), otelLabel.Key("type").String("default")}...)
+var networkTrace = getNetworkTrace("")
 
-	return span, ctx
+func (n *Network) trace(ctx context.Context, name string) (otelTrace.Span, context.Context) {
+	return networkTrace(ctx, name, nil)
+}
+
+func getNetworkTrace(networkType EndpointType) func(ctx context.Context, name string, endpoint interface{}) (otelTrace.Span, context.Context) {
+	return func(ctx context.Context, name string, endpoint interface{}) (otelTrace.Span, context.Context) {
+		tracer := otel.Tracer("kata")
+		ctx, span := tracer.Start(ctx, name, otelTrace.WithAttributes(otelLabel.String("source", "runtime"), otelLabel.String("package", "virtcontainers"), otelLabel.String("subsystem", "network")))
+		if networkType != "" {
+			span.SetAttributes(otelLabel.Any("type", string(networkType)))
+		}
+		if endpoint != nil {
+			span.SetAttributes(otelLabel.Any("endpoint", endpoint))
+		}
+		return span, ctx
+	}
+}
+
+func closeSpan(span otelTrace.Span, err error) {
+	if err != nil {
+		span.SetAttributes(otelLabel.Any("error", err))
+	}
+	span.End()
 }
 
 // Run runs a callback in the specified network namespace.
@@ -1274,12 +1327,15 @@ func (n *Network) Run(ctx context.Context, networkNSPath string, cb func() error
 // Add adds all needed interfaces inside the network namespace.
 func (n *Network) Add(ctx context.Context, config *NetworkConfig, s *Sandbox, hotplug bool) ([]Endpoint, error) {
 	span, ctx := n.trace(ctx, "Add")
+	span.SetAttributes(otelLabel.String("type", config.InterworkingModel.GetModel()))
 	defer span.End()
 
 	endpoints, err := createEndpointsFromScan(config.NetNSPath, config)
 	if err != nil {
 		return endpoints, err
 	}
+	span.SetAttributes(otelLabel.Any("endpoints", endpoints))
+	span.SetAttributes(otelLabel.Bool("hotplug", hotplug))
 
 	err = doNetNS(config.NetNSPath, func(_ ns.NetNS) error {
 		for _, endpoint := range endpoints {
@@ -1309,9 +1365,7 @@ func (n *Network) Add(ctx context.Context, config *NetworkConfig, s *Sandbox, ho
 						return err
 					}
 				}
-
 			}
-
 		}
 
 		return nil
@@ -1543,7 +1597,7 @@ func addIFBRedirecting(sourceIndex int, ifbIndex int) error {
 	return nil
 }
 
-// func addTxRateLmiter implements tx rate limiter to control network I/O outbound traffic
+// addTxRateLimiter implements tx rate limiter to control network I/O outbound traffic
 // on VM level for hypervisors which don't implement rate limiter in itself, like qemu, etc.
 // We adopt different actions, based on different inter-networking models.
 // For tcfilters as inter-networking model, we simply apply htb qdisc discipline to the virtual netpair.

src/runtime/virtcontainers/network_test.go

@@ -6,9 +6,9 @@
 package virtcontainers
 
 import (
+	"context"
 	"fmt"
 	"net"
-	"os"
 	"reflect"
 	"testing"
 
@@ -20,23 +20,6 @@ import (
 	"github.com/vishvananda/netlink"
 )
 
-func TestCreateDeleteNetNS(t *testing.T) {
-	assert := assert.New(t)
-	if tc.NotValid(ktu.NeedRoot()) {
-		t.Skip(testDisabledAsNonRoot)
-	}
-
-	netNSPath, err := createNetNS()
-	assert.NoError(err)
-	assert.NotEmpty(netNSPath)
-
-	_, err = os.Stat(netNSPath)
-	assert.NoError(err)
-
-	err = deleteNetNS(netNSPath)
-	assert.NoError(err)
-}
-
 func TestGenerateInterfacesAndRoutes(t *testing.T) {
 	//
 	//Create a couple of addresses
@@ -92,7 +75,7 @@ func TestGenerateInterfacesAndRoutes(t *testing.T) {
 
 	nns := NetworkNamespace{NetNsPath: "foobar", NetNsCreated: true, Endpoints: endpoints}
 
-	resInterfaces, resRoutes, resNeighs, err := generateVCNetworkStructures(nns)
+	resInterfaces, resRoutes, resNeighs, err := generateVCNetworkStructures(context.Background(), nns)
 
 	//
 	// Build expected results:
@@ -293,10 +276,10 @@ func TestTcRedirectNetwork(t *testing.T) {
 	err = netHandle.LinkSetUp(link)
 	assert.NoError(err)
 
-	err = setupTCFiltering(endpoint, 1, true)
+	err = setupTCFiltering(context.Background(), endpoint, 1, true)
 	assert.NoError(err)
 
-	err = removeTCFiltering(endpoint)
+	err = removeTCFiltering(context.Background(), endpoint)
 	assert.NoError(err)
 
 	// Remove the veth created for testing.
@@ -331,7 +314,7 @@ func TestRxRateLimiter(t *testing.T) {
 	err = netHandle.LinkSetUp(link)
 	assert.NoError(err)
 
-	err = setupTCFiltering(endpoint, 1, true)
+	err = setupTCFiltering(context.Background(), endpoint, 1, true)
 	assert.NoError(err)
 
 	// 10Mb
@@ -345,7 +328,7 @@ func TestRxRateLimiter(t *testing.T) {
 	err = removeRxRateLimiter(endpoint, currentNS.Path())
 	assert.NoError(err)
 
-	err = removeTCFiltering(endpoint)
+	err = removeTCFiltering(context.Background(), endpoint)
 	assert.NoError(err)
 
 	// Remove the veth created for testing.
@@ -380,7 +363,7 @@ func TestTxRateLimiter(t *testing.T) {
 	err = netHandle.LinkSetUp(link)
 	assert.NoError(err)
 
-	err = setupTCFiltering(endpoint, 1, true)
+	err = setupTCFiltering(context.Background(), endpoint, 1, true)
 	assert.NoError(err)
 
 	// 10Mb
@@ -394,7 +377,7 @@ func TestTxRateLimiter(t *testing.T) {
 	err = removeTxRateLimiter(endpoint, currentNS.Path())
 	assert.NoError(err)
 
-	err = removeTCFiltering(endpoint)
+	err = removeTCFiltering(context.Background(), endpoint)
 	assert.NoError(err)
 
 	// Remove the veth created for testing.

src/runtime/virtcontainers/persist.go

@@ -191,6 +191,8 @@ func (s *Sandbox) dumpConfig(ss *persistapi.SandboxState) {
 		Cgroups:             sconfig.Cgroups,
 	}
 
+	ss.Config.SandboxBindMounts = append(ss.Config.SandboxBindMounts, sconfig.SandboxBindMounts...)
+
 	for _, e := range sconfig.Experimental {
 		ss.Config.Experimental = append(ss.Config.Experimental, e.Name)
 	}
@@ -459,6 +461,7 @@ func loadSandboxConfig(id string) (*SandboxConfig, error) {
 		DisableGuestSeccomp: savedConf.DisableGuestSeccomp,
 		Cgroups:             savedConf.Cgroups,
 	}
+	sconfig.SandboxBindMounts = append(sconfig.SandboxBindMounts, savedConf.SandboxBindMounts...)
 
 	for _, name := range savedConf.Experimental {
 		sconfig.Experimental = append(sconfig.Experimental, *exp.Get(name))

src/runtime/virtcontainers/persist/api/config.go

@@ -275,6 +275,9 @@ type SandboxConfig struct {
 
 	DisableGuestSeccomp bool
 
+	// SandboxBindMounts - list of paths to mount into guest
+	SandboxBindMounts []string
+
 	// Experimental enables experimental features
 	Experimental []string
 

src/runtime/virtcontainers/persist/api/sandbox.go

@@ -34,7 +34,6 @@ type SandboxState struct {
 
 	// CgroupPath is the cgroup hierarchy where sandbox's processes
 	// including the hypervisor are placed.
-	// FIXME: sandbox can reuse "SandboxContainer"'s CgroupPath so we can remove this field.
 	CgroupPath string
 
 	// CgroupPath is the cgroup hierarchy where sandbox's processes

src/runtime/virtcontainers/physical_endpoint.go

@@ -21,6 +21,8 @@ import (
 	"github.com/safchain/ethtool"
 )
 
+var physicalTrace = getNetworkTrace(PhysicalEndpointType)
+
 // PhysicalEndpoint gathers a physical network interface and its properties
 type PhysicalEndpoint struct {
 	IfaceName          string
@@ -76,6 +78,9 @@ func (endpoint *PhysicalEndpoint) NetworkPair() *NetworkInterfacePair {
 // Attach for physical endpoint binds the physical network interface to
 // vfio-pci and adds device to the hypervisor with vfio-passthrough.
 func (endpoint *PhysicalEndpoint) Attach(ctx context.Context, s *Sandbox) error {
+	span, ctx := physicalTrace(ctx, "Attach", endpoint)
+	defer span.End()
+
 	// Unbind physical interface from host driver and bind to vfio
 	// so that it can be passed to qemu.
 	vfioPath, err := bindNICToVFIO(endpoint)
@@ -103,6 +108,9 @@ func (endpoint *PhysicalEndpoint) Attach(ctx context.Context, s *Sandbox) error
 // Detach for physical endpoint unbinds the physical network interface from vfio-pci
 // and binds it back to the saved host driver.
 func (endpoint *PhysicalEndpoint) Detach(ctx context.Context, netNsCreated bool, netNsPath string) error {
+	span, _ := physicalTrace(ctx, "Detach", endpoint)
+	defer span.End()
+
 	// Bind back the physical network interface to host.
 	// We need to do this even if a new network namespace has not
 	// been created by virtcontainers.

src/runtime/virtcontainers/qemu.go

@@ -13,9 +13,7 @@ import (
 	"fmt"
 	"io/ioutil"
 	"math"
-	"net"
 	"os"
-	"os/exec"
 	"path/filepath"
 	"strconv"
 	"strings"
@@ -106,6 +104,8 @@ type qemu struct {
 
 	// if in memory dump progress
 	memoryDumpFlag sync.Mutex
+
+	virtiofsd Virtiofsd
 }
 
 const (
@@ -219,8 +219,7 @@ func (q *qemu) trace(parent context.Context, name string) (otelTrace.Span, conte
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes([]otelLabel.KeyValue{otelLabel.Key("subsystem").String("hypervisor"), otelLabel.Key("type").String("qemu")}...)
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(otelLabel.String("source", "runtime"), otelLabel.String("package", "virtcontainers"), otelLabel.String("subsystem", "hypervisor"), otelLabel.String("type", "qemu"), otelLabel.String("sandbox_id", q.id)))
 
 	return span, ctx
 }
@@ -605,6 +604,11 @@ func (q *qemu) createSandbox(ctx context.Context, id string, networkNS NetworkNa
 		PidFile:     filepath.Join(q.store.RunVMStoragePath(), q.id, "pid"),
 	}
 
+	qemuConfig.Devices, qemuConfig.Bios, err = q.arch.appendProtectionDevice(qemuConfig.Devices, firmwarePath)
+	if err != nil {
+		return err
+	}
+
 	if ioThread != nil {
 		qemuConfig.IOThreads = []govmmQemu.IOThread{*ioThread}
 	}
@@ -627,6 +631,20 @@ func (q *qemu) createSandbox(ctx context.Context, id string, networkNS NetworkNa
 
 	q.qemuConfig = qemuConfig
 
+	virtiofsdSocketPath, err := q.vhostFSSocketPath(q.id)
+	if err != nil {
+		return err
+	}
+
+	q.virtiofsd = &virtiofsd{
+		path:       q.config.VirtioFSDaemon,
+		sourcePath: filepath.Join(getSharePath(q.id)),
+		socketPath: virtiofsdSocketPath,
+		extraArgs:  q.config.VirtioFSExtraArgs,
+		debug:      q.config.Debug,
+		cache:      q.config.VirtioFSCache,
+	}
+
 	return nil
 }
 
@@ -634,102 +652,29 @@ func (q *qemu) vhostFSSocketPath(id string) (string, error) {
 	return utils.BuildSocketPath(q.store.RunVMStoragePath(), id, vhostFSSocket)
 }
 
-func (q *qemu) virtiofsdArgs(fd uintptr) []string {
-	// The daemon will terminate when the vhost-user socket
-	// connection with QEMU closes.  Therefore we do not keep track
-	// of this child process after returning from this function.
-	sourcePath := filepath.Join(getSharePath(q.id))
-	args := []string{
-		fmt.Sprintf("--fd=%v", fd),
-		"-o", "source=" + sourcePath,
-		"-o", "cache=" + q.config.VirtioFSCache,
-		"--syslog", "-o", "no_posix_lock"}
-	if q.config.Debug {
-		args = append(args, "-d")
-	} else {
-		args = append(args, "-f")
-	}
-
-	if len(q.config.VirtioFSExtraArgs) != 0 {
-		args = append(args, q.config.VirtioFSExtraArgs...)
-	}
-	return args
-}
-
 func (q *qemu) setupVirtiofsd(ctx context.Context) (err error) {
-	var listener *net.UnixListener
-	var fd *os.File
-
-	sockPath, err := q.vhostFSSocketPath(q.id)
-	if err != nil {
-		return err
-	}
-
-	listener, err = net.ListenUnix("unix", &net.UnixAddr{
-		Name: sockPath,
-		Net:  "unix",
+	pid, err := q.virtiofsd.Start(ctx, func() {
+		q.stopSandbox(ctx, false)
 	})
 	if err != nil {
 		return err
 	}
-	listener.SetUnlinkOnClose(false)
+	q.state.VirtiofsdPid = pid
 
-	fd, err = listener.File()
-	listener.Close() // no longer needed since fd is a dup
-	listener = nil
-	if err != nil {
-		return err
-	}
-	defer fd.Close()
-
-	const sockFd = 3 // Cmd.ExtraFiles[] fds are numbered starting from 3
-	cmd := exec.Command(q.config.VirtioFSDaemon, q.virtiofsdArgs(sockFd)...)
-	cmd.ExtraFiles = append(cmd.ExtraFiles, fd)
-	stderr, err := cmd.StderrPipe()
-	if err != nil {
-		return err
-	}
-
-	err = cmd.Start()
-	if err != nil {
-		return fmt.Errorf("virtiofs daemon %v returned with error: %v", q.config.VirtioFSDaemon, err)
-	}
-	q.state.VirtiofsdPid = cmd.Process.Pid
-
-	// Monitor virtiofsd's stderr and stop sandbox if virtiofsd quits
-	go func() {
-		scanner := bufio.NewScanner(stderr)
-		for scanner.Scan() {
-			q.Logger().WithField("source", "virtiofsd").Info(scanner.Text())
-		}
-		q.Logger().Info("virtiofsd quits")
-		// Wait to release resources of virtiofsd process
-		cmd.Process.Wait()
-		q.stopSandbox(ctx, false)
-	}()
-	return err
+	return nil
 }
 
 func (q *qemu) stopVirtiofsd(ctx context.Context) (err error) {
-
-	// kill virtiofsd
 	if q.state.VirtiofsdPid == 0 {
 		return errors.New("invalid virtiofsd PID(0)")
 	}
 
-	err = syscall.Kill(q.state.VirtiofsdPid, syscall.SIGKILL)
+	err = q.virtiofsd.Stop(ctx)
 	if err != nil {
 		return err
 	}
 	q.state.VirtiofsdPid = 0
-
-	// remove virtiofsd socket
-	sockPath, err := q.vhostFSSocketPath(q.id)
-	if err != nil {
-		return err
-	}
-
-	return os.Remove(sockPath)
+	return nil
 }
 
 func (q *qemu) getMemArgs() (bool, string, string, error) {
@@ -2251,7 +2196,6 @@ func calcHotplugMemMiBSize(mem uint32, memorySectionSizeMB uint32) (uint32, erro
 		return mem, nil
 	}
 
-	// TODO: hot add memory aligned to memory section should be more properly. See https://github.com/kata-containers/runtime/pull/624#issuecomment-419656853
 	return uint32(math.Ceil(float64(mem)/float64(memorySectionSizeMB))) * memorySectionSizeMB, nil
 }
 
@@ -2324,6 +2268,10 @@ func (q *qemu) getPids() []int {
 	return pids
 }
 
+func (q *qemu) getVirtioFsPid() *int {
+	return &q.state.VirtiofsdPid
+}
+
 type qemuGrpc struct {
 	ID             string
 	QmpChannelpath string

src/runtime/virtcontainers/qemu_amd64.go

@@ -11,6 +11,7 @@ import (
 	"time"
 
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/types"
+	"github.com/sirupsen/logrus"
 
 	govmmQemu "github.com/kata-containers/govmm/qemu"
 )
@@ -20,6 +21,8 @@ type qemuAmd64 struct {
 	qemuArchBase
 
 	vmFactory bool
+
+	devLoadersCount uint32
 }
 
 const (
@@ -30,6 +33,10 @@ const (
 	defaultQemuMachineOptions = "accel=kvm,kernel_irqchip"
 
 	qmpMigrationWaitTimeout = 5 * time.Second
+
+	tdxSysFirmwareDir = "/sys/firmware/tdx_seam/"
+
+	tdxCPUFlag = "tdx"
 )
 
 var qemuPaths = map[string]string{
@@ -106,17 +113,17 @@ func newQemuArch(config HypervisorConfig) (qemuArch, error) {
 		factory = true
 	}
 
-	if config.IOMMU {
-		var q35QemuIOMMUOptions = "accel=kvm,kernel_irqchip=split"
+	// IOMMU and Guest Protection require a split IRQ controller for handling interrupts
+	// otherwise QEMU won't be able to create the kernel irqchip
+	if config.IOMMU || config.ConfidentialGuest {
+		mp.Options = "accel=kvm,kernel_irqchip=split"
+	}
 
+	if config.IOMMU {
 		kernelParams = append(kernelParams,
 			Param{"intel_iommu", "on"})
 		kernelParams = append(kernelParams,
 			Param{"iommu", "pt"})
-
-		if mp.Type == QemuQ35 {
-			mp.Options = q35QemuIOMMUOptions
-		}
 	}
 
 	q := &qemuAmd64{
@@ -129,10 +136,17 @@ func newQemuArch(config HypervisorConfig) (qemuArch, error) {
 			kernelParams:         kernelParams,
 			disableNvdimm:        config.DisableImageNvdimm,
 			dax:                  true,
+			protection:           noneProtection,
 		},
 		vmFactory: factory,
 	}
 
+	if config.ConfidentialGuest {
+		if err := q.enableProtection(); err != nil {
+			return nil, err
+		}
+	}
+
 	q.handleImagePath(config)
 
 	return q, nil
@@ -191,3 +205,55 @@ func (q *qemuAmd64) appendImage(ctx context.Context, devices []govmmQemu.Device,
 func (q *qemuAmd64) appendBridges(devices []govmmQemu.Device) []govmmQemu.Device {
 	return genericAppendBridges(devices, q.Bridges, q.qemuMachine.Type)
 }
+
+// enable protection
+func (q *qemuAmd64) enableProtection() error {
+	var err error
+	q.protection, err = availableGuestProtection()
+	if err != nil {
+		return err
+	}
+
+	switch q.protection {
+	case tdxProtection:
+		if q.qemuMachine.Options != "" {
+			q.qemuMachine.Options += ","
+		}
+		q.qemuMachine.Options += "kvm-type=tdx,confidential-guest-support=tdx"
+		q.kernelParams = append(q.kernelParams, Param{"tdx_guest", ""})
+		virtLog.WithFields(logrus.Fields{
+			"subsystem":     "qemuAmd64",
+			"machine":       q.qemuMachine,
+			"kernel-params": q.kernelParameters}).
+			Info("Enabling TDX guest protection")
+		return nil
+
+	// TODO: Add support for other x86_64 technologies: SEV
+
+	default:
+		return fmt.Errorf("This system doesn't support Confidential Computing (Guest Protection)")
+	}
+}
+
+// append protection device
+func (q *qemuAmd64) appendProtectionDevice(devices []govmmQemu.Device, firmware string) ([]govmmQemu.Device, string, error) {
+	switch q.protection {
+	case tdxProtection:
+		id := q.devLoadersCount
+		q.devLoadersCount += 1
+		return append(devices,
+			govmmQemu.Object{
+				Driver:   govmmQemu.Loader,
+				Type:     govmmQemu.TDXGuest,
+				ID:       "tdx",
+				DeviceID: fmt.Sprintf("fd%d", id),
+				Debug:    false,
+				File:     firmware,
+			}), "", nil
+	case noneProtection:
+		return devices, firmware, nil
+
+	default:
+		return devices, "", fmt.Errorf("Unsupported guest protection technology: %v", q.protection)
+	}
+}

src/runtime/virtcontainers/qemu_amd64_test.go

@@ -276,3 +276,59 @@ func TestQemuAmd64Microvm(t *testing.T) {
 
 	assert.False(amd64.supportGuestMemoryHotplug())
 }
+
+func TestQemuAmd64AppendProtectionDevice(t *testing.T) {
+	var devices []govmmQemu.Device
+	assert := assert.New(t)
+
+	amd64 := newTestQemu(assert, QemuPC)
+
+	id := amd64.(*qemuAmd64).devLoadersCount
+	firmware := "tdvf.fd"
+	var bios string
+	var err error
+	devices, bios, err = amd64.appendProtectionDevice(devices, firmware)
+	assert.NoError(err)
+
+	// non-protection
+	assert.NotEmpty(bios)
+
+	// pef protection
+	amd64.(*qemuAmd64).protection = pefProtection
+	devices, bios, err = amd64.appendProtectionDevice(devices, firmware)
+	assert.Error(err)
+	assert.Empty(bios)
+
+	// Secure Execution protection
+	amd64.(*qemuAmd64).protection = seProtection
+	devices, bios, err = amd64.appendProtectionDevice(devices, firmware)
+	assert.Error(err)
+	assert.Empty(bios)
+
+	// sev protection
+	// TODO: update once it's supported
+	amd64.(*qemuAmd64).protection = sevProtection
+	devices, bios, err = amd64.appendProtectionDevice(devices, firmware)
+	assert.Error(err)
+	assert.Empty(bios)
+
+	// tdxProtection
+	amd64.(*qemuAmd64).protection = tdxProtection
+
+	devices, bios, err = amd64.appendProtectionDevice(devices, firmware)
+	assert.NoError(err)
+	assert.Empty(bios)
+
+	expectedOut := []govmmQemu.Device{
+		govmmQemu.Object{
+			Driver:   govmmQemu.Loader,
+			Type:     govmmQemu.TDXGuest,
+			ID:       "tdx",
+			DeviceID: fmt.Sprintf("fd%d", id),
+			Debug:    false,
+			File:     firmware,
+		},
+	}
+
+	assert.Equal(expectedOut, devices)
+}

src/runtime/virtcontainers/qemu_arch_base.go

@@ -11,6 +11,7 @@ import (
 	"errors"
 	"fmt"
 	"os"
+	"runtime"
 	"strconv"
 	"strings"
 
@@ -142,8 +143,37 @@ type qemuArch interface {
 
 	// append pvpanic device
 	appendPVPanicDevice(devices []govmmQemu.Device) ([]govmmQemu.Device, error)
+
+	// append protection device.
+	// This implementation is architecture specific, some archs may need
+	// a firmware, returns a string containing the path to the firmware that should
+	// be used with the -bios option, ommit -bios option if the path is empty.
+	appendProtectionDevice(devices []govmmQemu.Device, firmware string) ([]govmmQemu.Device, string, error)
 }
 
+// Kind of guest protection
+type guestProtection uint8
+
+const (
+	noneProtection guestProtection = iota
+
+	//Intel Trust Domain Extensions
+	//https://software.intel.com/content/www/us/en/develop/articles/intel-trust-domain-extensions.html
+	tdxProtection
+
+	// AMD Secure Encrypted Virtualization
+	// https://developer.amd.com/sev/
+	sevProtection
+
+	// IBM POWER 9 Protected Execution Facility
+	// https://www.kernel.org/doc/html/latest/powerpc/ultravisor.html
+	pefProtection
+
+	// IBM Secure Execution (IBM Z & LinuxONE)
+	// https://www.kernel.org/doc/html/latest/virt/kvm/s390-pv.html
+	seProtection
+)
+
 type qemuArchBase struct {
 	qemuMachine          govmmQemu.Machine
 	qemuExePath          string
@@ -158,6 +188,7 @@ type qemuArchBase struct {
 	kernelParams         []Param
 	Bridges              []types.Bridge
 	PFlash               []string
+	protection           guestProtection
 }
 
 const (
@@ -813,3 +844,9 @@ func (q *qemuArchBase) getPFlash() ([]string, error) {
 func (q *qemuArchBase) setPFlash(p []string) {
 	q.PFlash = p
 }
+
+// append protection device
+func (q *qemuArchBase) appendProtectionDevice(devices []govmmQemu.Device, firmware string) ([]govmmQemu.Device, string, error) {
+	virtLog.WithField("arch", runtime.GOARCH).Warnf("Confidential Computing has not been implemented for this architecture")
+	return devices, firmware, nil
+}

src/runtime/virtcontainers/qemu_ppc64le.go

@@ -27,6 +27,14 @@ const defaultQemuMachineOptions = "accel=kvm,usb=off"
 
 const qmpMigrationWaitTimeout = 5 * time.Second
 
+const pefSysFirmwareDir = "/sys/firmware/ultravisor/"
+
+const pefID = "pef0"
+
+const tpmID = "tpm0"
+
+const tpmHostPath = "/dev/tpmrm0"
+
 var kernelParams = []Param{
 	{"rcupdate.rcu_expedited", "1"},
 	{"reboot", "k"},
@@ -43,7 +51,7 @@ var supportedQemuMachine = govmmQemu.Machine{
 
 // Logger returns a logrus logger appropriate for logging qemu messages
 func (q *qemuPPC64le) Logger() *logrus.Entry {
-	return virtLog.WithField("subsystem", "qemu")
+	return virtLog.WithField("subsystem", "qemuPPC64le")
 }
 
 // MaxQemuVCPUs returns the maximum number of vCPUs supported
@@ -69,9 +77,16 @@ func newQemuArch(config HypervisorConfig) (qemuArch, error) {
 			kernelParamsNonDebug: kernelParamsNonDebug,
 			kernelParamsDebug:    kernelParamsDebug,
 			kernelParams:         kernelParams,
+			protection:           noneProtection,
 		},
 	}
 
+	if config.ConfidentialGuest {
+		if err := q.enableProtection(); err != nil {
+			return nil, err
+		}
+	}
+
 	q.handleImagePath(config)
 
 	q.memoryOffset = config.MemOffset
@@ -116,3 +131,49 @@ func (q *qemuPPC64le) appendBridges(devices []govmmQemu.Device) []govmmQemu.Devi
 func (q *qemuPPC64le) appendIOMMU(devices []govmmQemu.Device) ([]govmmQemu.Device, error) {
 	return devices, fmt.Errorf("PPC64le does not support appending a vIOMMU")
 }
+
+// Enables guest protection
+func (q *qemuPPC64le) enableProtection() error {
+	var err error
+	q.protection, err = availableGuestProtection()
+	if err != nil {
+		return err
+	}
+
+	switch q.protection {
+	case pefProtection:
+		if q.qemuMachine.Options != "" {
+			q.qemuMachine.Options += ","
+		}
+		q.qemuMachine.Options += fmt.Sprintf("confidential-guest-support=%s", pefID)
+		virtLog.WithFields(logrus.Fields{
+			"subsystem":     "qemuPPC64le",
+			"machine":       q.qemuMachine,
+			"kernel-params": q.kernelParams,
+		}).Info("Enabling PEF protection")
+		return nil
+
+	default:
+		return fmt.Errorf("This system doesn't support Confidential Computing (Guest Protection)")
+	}
+}
+
+// append protection device
+func (q *qemuPPC64le) appendProtectionDevice(devices []govmmQemu.Device, firmware string) ([]govmmQemu.Device, string, error) {
+	switch q.protection {
+	case pefProtection:
+		return append(devices,
+			govmmQemu.Object{
+				Driver:   govmmQemu.SpaprTPMProxy,
+				Type:     govmmQemu.PEFGuest,
+				ID:       pefID,
+				DeviceID: tpmID,
+				File:     tpmHostPath,
+			}), firmware, nil
+	case noneProtection:
+		return devices, firmware, nil
+
+	default:
+		return devices, "", fmt.Errorf("Unsupported guest protection technology: %v", q.protection)
+	}
+}

src/runtime/virtcontainers/qemu_s390x.go

@@ -14,6 +14,7 @@ import (
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/device/config"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/types"
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/utils"
+	"github.com/sirupsen/logrus"
 )
 
 type qemuS390x struct {
@@ -21,15 +22,18 @@ type qemuS390x struct {
 	qemuArchBase
 }
 
-const defaultQemuPath = "/usr/bin/qemu-system-s390x"
+const (
+	defaultQemuPath           = "/usr/bin/qemu-system-s390x"
+	defaultQemuMachineType    = QemuCCWVirtio
+	defaultQemuMachineOptions = "accel=kvm"
+	virtioSerialCCW           = "virtio-serial-ccw"
+	qmpMigrationWaitTimeout   = 5 * time.Second
+	logSubsystem              = "qemuS390x"
 
-const defaultQemuMachineType = QemuCCWVirtio
-
-const defaultQemuMachineOptions = "accel=kvm"
-
-const virtioSerialCCW = "virtio-serial-ccw"
-
-const qmpMigrationWaitTimeout = 5 * time.Second
+	// Secure Execution, also known as Protected Virtualization
+	// https://qemu.readthedocs.io/en/latest/system/s390x/protvirt.html
+	secExecID = "pv0"
+)
 
 // Verify needed parameters
 var kernelParams = []Param{
@@ -74,6 +78,12 @@ func newQemuArch(config HypervisorConfig) (qemuArch, error) {
 	// Set first bridge type to CCW
 	q.Bridges = append(q.Bridges, ccwbridge)
 
+	if config.ConfidentialGuest {
+		if err := q.enableProtection(); err != nil {
+			return nil, err
+		}
+	}
+
 	if config.ImagePath != "" {
 		q.kernelParams = append(q.kernelParams, commonVirtioblkKernelRootParams...)
 		q.kernelParamsNonDebug = append(q.kernelParamsNonDebug, kernelParamsSystemdNonDebug...)
@@ -303,3 +313,42 @@ func (q *qemuS390x) addDeviceToBridge(ctx context.Context, ID string, t types.Ty
 
 	return fmt.Sprintf("%04x", addr), b, nil
 }
+
+// enableProtection enables guest protection for QEMU's machine option.
+func (q *qemuS390x) enableProtection() error {
+	protection, err := availableGuestProtection()
+	if err != nil {
+		return err
+	}
+	if protection != seProtection {
+		return fmt.Errorf("Got unexpected protection %v, only seProtection (Secure Execution) is supported", protection)
+	}
+
+	q.protection = protection
+	if q.qemuMachine.Options != "" {
+		q.qemuMachine.Options += ","
+	}
+	q.qemuMachine.Options += fmt.Sprintf("confidential-guest-support=%s", secExecID)
+	virtLog.WithFields(logrus.Fields{
+		"subsystem": logSubsystem,
+		"machine":   q.qemuMachine}).
+		Info("Enabling guest protection with Secure Execution")
+	return nil
+}
+
+// appendProtectionDevice appends a QEMU object for Secure Execution.
+// Takes devices and returns updated version. Takes BIOS and returns it (no modification on s390x).
+func (q *qemuS390x) appendProtectionDevice(devices []govmmQemu.Device, firmware string) ([]govmmQemu.Device, string, error) {
+	switch q.protection {
+	case seProtection:
+		return append(devices,
+			govmmQemu.Object{
+				Type: govmmQemu.SecExecGuest,
+				ID:   secExecID,
+			}), firmware, nil
+	case noneProtection:
+		return devices, firmware, nil
+	default:
+		return devices, firmware, fmt.Errorf("Unsupported guest protection technology: %v", q.protection)
+	}
+}

src/runtime/virtcontainers/qemu_s390x_test.go

@@ -101,3 +101,50 @@ func TestQemuS390xAppendVhostUserDevice(t *testing.T) {
 	assert.NoError(err)
 	assert.Equal(devices, expected)
 }
+
+func TestQemuS390xAppendProtectionDevice(t *testing.T) {
+	assert := assert.New(t)
+	s390x := newTestQemu(assert, QemuCCWVirtio)
+
+	var devices []govmmQemu.Device
+	var bios, firmware string
+	var err error
+	devices, bios, err = s390x.appendProtectionDevice(devices, firmware)
+	assert.NoError(err)
+
+	// no protection
+	assert.Empty(bios)
+
+	// PEF protection
+	s390x.(*qemuS390x).protection = pefProtection
+	devices, bios, err = s390x.appendProtectionDevice(devices, firmware)
+	assert.Error(err)
+	assert.Empty(bios)
+
+	// TDX protection
+	s390x.(*qemuS390x).protection = tdxProtection
+	devices, bios, err = s390x.appendProtectionDevice(devices, firmware)
+	assert.Error(err)
+	assert.Empty(bios)
+
+	// SEV protection
+	s390x.(*qemuS390x).protection = sevProtection
+	devices, bios, err = s390x.appendProtectionDevice(devices, firmware)
+	assert.Error(err)
+	assert.Empty(bios)
+
+	// Secure Execution protection
+	s390x.(*qemuS390x).protection = seProtection
+
+	devices, bios, err = s390x.appendProtectionDevice(devices, firmware)
+	assert.NoError(err)
+	assert.Empty(bios)
+
+	expectedOut := []govmmQemu.Device{
+		govmmQemu.Object{
+			Type: govmmQemu.SecExecGuest,
+			ID:   secExecID,
+		},
+	}
+	assert.Equal(expectedOut, devices)
+}

src/runtime/virtcontainers/qemu_test.go

@@ -11,7 +11,6 @@ import (
 	"io/ioutil"
 	"os"
 	"path/filepath"
-	"strings"
 	"testing"
 
 	govmmQemu "github.com/kata-containers/govmm/qemu"
@@ -550,35 +549,6 @@ func createQemuSandboxConfig() (*Sandbox, error) {
 	return &sandbox, nil
 }
 
-func TestQemuVirtiofsdArgs(t *testing.T) {
-	assert := assert.New(t)
-
-	q := &qemu{
-		id: "foo",
-		config: HypervisorConfig{
-			VirtioFSCache: "none",
-			Debug:         true,
-		},
-	}
-
-	savedKataHostSharedDir := kataHostSharedDir
-	kataHostSharedDir = func() string {
-		return "test-share-dir"
-	}
-	defer func() {
-		kataHostSharedDir = savedKataHostSharedDir
-	}()
-
-	result := "--fd=123 -o source=test-share-dir/foo/shared -o cache=none --syslog -o no_posix_lock -d"
-	args := q.virtiofsdArgs(123)
-	assert.Equal(strings.Join(args, " "), result)
-
-	q.config.Debug = false
-	result = "--fd=123 -o source=test-share-dir/foo/shared -o cache=none --syslog -o no_posix_lock -f"
-	args = q.virtiofsdArgs(123)
-	assert.Equal(strings.Join(args, " "), result)
-}
-
 func TestQemuGetpids(t *testing.T) {
 	assert := assert.New(t)
 

src/runtime/virtcontainers/sandbox.go

@@ -134,8 +134,7 @@ func (s *Sandbox) trace(parent context.Context, name string) (otelTrace.Span, co
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes(otelLabel.Key("subsystem").String("sandbox"))
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(otelLabel.String("source", "runtime"), otelLabel.String("package", "virtcontainers"), otelLabel.String("subsystem", "sandbox"), otelLabel.String("sandbox_id", s.id)))
 
 	return span, ctx
 }
@@ -395,6 +394,7 @@ func (s *Sandbox) IOStream(containerID, processID string) (io.WriteCloser, io.Re
 
 func createAssets(ctx context.Context, sandboxConfig *SandboxConfig) error {
 	span, _ := trace(ctx, "createAssets")
+	span.SetAttributes(otelLabel.String("sandbox_id", sandboxConfig.ID), otelLabel.String("subsystem", "sandbox"))
 	defer span.End()
 
 	for _, name := range types.AssetTypes() {
@@ -445,6 +445,7 @@ func (s *Sandbox) getAndStoreGuestDetails(ctx context.Context) error {
 // be started.
 func createSandbox(ctx context.Context, sandboxConfig SandboxConfig, factory Factory) (*Sandbox, error) {
 	span, ctx := trace(ctx, "createSandbox")
+	span.SetAttributes(otelLabel.String("sandbox_id", sandboxConfig.ID), otelLabel.String("subsystem", "sandbox"))
 	defer span.End()
 
 	if err := createAssets(ctx, &sandboxConfig); err != nil {
@@ -483,6 +484,7 @@ func createSandbox(ctx context.Context, sandboxConfig SandboxConfig, factory Fac
 
 func newSandbox(ctx context.Context, sandboxConfig SandboxConfig, factory Factory) (sb *Sandbox, retErr error) {
 	span, ctx := trace(ctx, "newSandbox")
+	span.SetAttributes(otelLabel.String("sandbox_id", sandboxConfig.ID), otelLabel.String("subsystem", "sandbox"))
 	defer span.End()
 
 	if !sandboxConfig.valid() {
@@ -758,6 +760,9 @@ func (s *Sandbox) createNetwork(ctx context.Context) error {
 		NetNsCreated: s.config.NetworkConfig.NetNsCreated,
 	}
 
+	span.SetAttributes(otelLabel.Any("networkNS", s.networkNS))
+	span.SetAttributes(otelLabel.Any("NetworkConfig", s.config.NetworkConfig))
+
 	// In case there is a factory, network interfaces are hotplugged
 	// after vm is started.
 	if s.factory == nil {
@@ -1271,7 +1276,6 @@ func (s *Sandbox) DeleteContainer(ctx context.Context, containerID string) (VCCo
 }
 
 // StatusContainer gets the status of a container
-// TODO: update container status properly, see kata-containers/runtime#253
 func (s *Sandbox) StatusContainer(containerID string) (ContainerStatus, error) {
 	if containerID == "" {
 		return ContainerStatus{}, vcTypes.ErrNeedContainerID
@@ -1529,7 +1533,7 @@ func (s *Sandbox) Stop(ctx context.Context, force bool) error {
 
 	// shutdown console watcher if exists
 	if s.cw != nil {
-		s.Logger().Debug("stop the sandbox")
+		s.Logger().Debug("stop the console watcher")
 		s.cw.stop()
 	}
 

src/runtime/virtcontainers/sandbox_metrics.go

@@ -16,8 +16,10 @@ import (
 
 const namespaceHypervisor = "kata_hypervisor"
 const namespaceKatashim = "kata_shim"
+const namespaceVirtiofsd = "kata_virtiofsd"
 
 var (
+	// hypervisor
 	hypervisorThreads = prometheus.NewGauge(prometheus.GaugeOpts{
 		Namespace: namespaceHypervisor,
 		Name:      "threads",
@@ -62,6 +64,7 @@ var (
 		Help:      "Open FDs for hypervisor.",
 	})
 
+	// agent
 	agentRPCDurationsHistogram = prometheus.NewHistogramVec(prometheus.HistogramOpts{
 		Namespace: namespaceKatashim,
 		Name:      "agent_rpc_durations_histogram_milliseconds",
@@ -70,16 +73,61 @@ var (
 	},
 		[]string{"action"},
 	)
+
+	// virtiofsd
+	virtiofsdThreads = prometheus.NewGauge(prometheus.GaugeOpts{
+		Namespace: namespaceVirtiofsd,
+		Name:      "threads",
+		Help:      "Virtiofsd process threads.",
+	})
+
+	virtiofsdProcStatus = prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Namespace: namespaceVirtiofsd,
+		Name:      "proc_status",
+		Help:      "Virtiofsd process status.",
+	},
+		[]string{"item"},
+	)
+
+	virtiofsdProcStat = prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Namespace: namespaceVirtiofsd,
+		Name:      "proc_stat",
+		Help:      "Virtiofsd process statistics.",
+	},
+		[]string{"item"},
+	)
+
+	virtiofsdIOStat = prometheus.NewGaugeVec(prometheus.GaugeOpts{
+		Namespace: namespaceVirtiofsd,
+		Name:      "io_stat",
+		Help:      "Process IO statistics.",
+	},
+		[]string{"item"},
+	)
+
+	virtiofsdOpenFDs = prometheus.NewGauge(prometheus.GaugeOpts{
+		Namespace: namespaceVirtiofsd,
+		Name:      "fds",
+		Help:      "Open FDs for virtiofsd.",
+	})
 )
 
 func RegisterMetrics() {
+	// hypervisor
 	prometheus.MustRegister(hypervisorThreads)
 	prometheus.MustRegister(hypervisorProcStatus)
 	prometheus.MustRegister(hypervisorProcStat)
 	prometheus.MustRegister(hypervisorNetdev)
 	prometheus.MustRegister(hypervisorIOStat)
 	prometheus.MustRegister(hypervisorOpenFDs)
+	// agent
 	prometheus.MustRegister(agentRPCDurationsHistogram)
+	// virtiofsd
+	prometheus.MustRegister(virtiofsdThreads)
+	prometheus.MustRegister(virtiofsdProcStatus)
+	prometheus.MustRegister(virtiofsdProcStat)
+	prometheus.MustRegister(virtiofsdIOStat)
+	prometheus.MustRegister(virtiofsdOpenFDs)
 }
 
 // UpdateRuntimeMetrics update shim/hypervisor's metrics
@@ -125,6 +173,48 @@ func (s *Sandbox) UpdateRuntimeMetrics() error {
 		mutils.SetGaugeVecProcIO(hypervisorIOStat, ioStat)
 	}
 
+	// virtiofs metrics
+	err = s.UpdateVirtiofsdMetrics()
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (s *Sandbox) UpdateVirtiofsdMetrics() error {
+	vfsPid := s.hypervisor.getVirtioFsPid()
+	if vfsPid == nil {
+		// virtiofsd is not mandatory for a VMM.
+		return nil
+	}
+
+	proc, err := procfs.NewProc(*vfsPid)
+	if err != nil {
+		return err
+	}
+
+	// process FDs
+	if fds, err := proc.FileDescriptorsLen(); err == nil {
+		virtiofsdOpenFDs.Set(float64(fds))
+	}
+
+	// process statistics
+	if procStat, err := proc.Stat(); err == nil {
+		virtiofsdThreads.Set(float64(procStat.NumThreads))
+		mutils.SetGaugeVecProcStat(virtiofsdProcStat, procStat)
+	}
+
+	// process status
+	if procStatus, err := proc.NewStatus(); err == nil {
+		mutils.SetGaugeVecProcStatus(virtiofsdProcStatus, procStatus)
+	}
+
+	// process IO statistics
+	if ioStat, err := proc.IO(); err == nil {
+		mutils.SetGaugeVecProcIO(virtiofsdIOStat, ioStat)
+	}
+
 	return nil
 }
 

src/runtime/virtcontainers/tap_endpoint.go

@@ -17,6 +17,8 @@ import (
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/uuid"
 )
 
+var tapTrace = getNetworkTrace(TapEndpointType)
+
 // TapEndpoint represents just a tap endpoint
 type TapEndpoint struct {
 	TapInterface       TapInterface
@@ -78,6 +80,9 @@ func (endpoint *TapEndpoint) Detach(ctx context.Context, netNsCreated bool, netN
 		return nil
 	}
 
+	span, _ := tapTrace(ctx, "Detach", endpoint)
+	defer span.End()
+
 	networkLogger().WithField("endpoint-type", TapEndpointType).Info("Detaching endpoint")
 	return doNetNS(netNsPath, func(_ ns.NetNS) error {
 		return unTapNetwork(endpoint.TapInterface.TAPIface.Name)
@@ -87,6 +92,10 @@ func (endpoint *TapEndpoint) Detach(ctx context.Context, netNsCreated bool, netN
 // HotAttach for the tap endpoint uses hot plug device
 func (endpoint *TapEndpoint) HotAttach(ctx context.Context, h hypervisor) error {
 	networkLogger().Info("Hot attaching tap endpoint")
+
+	span, ctx := tapTrace(ctx, "HotAttach", endpoint)
+	defer span.End()
+
 	if err := tapNetwork(endpoint, h.hypervisorConfig().NumVCPUs, h.hypervisorConfig().DisableVhostNet); err != nil {
 		networkLogger().WithError(err).Error("Error bridging tap ep")
 		return err
@@ -102,6 +111,10 @@ func (endpoint *TapEndpoint) HotAttach(ctx context.Context, h hypervisor) error
 // HotDetach for the tap endpoint uses hot pull device
 func (endpoint *TapEndpoint) HotDetach(ctx context.Context, h hypervisor, netNsCreated bool, netNsPath string) error {
 	networkLogger().Info("Hot detaching tap endpoint")
+
+	span, ctx := tapTrace(ctx, "HotDetach", endpoint)
+	defer span.End()
+
 	if err := doNetNS(netNsPath, func(_ ns.NetNS) error {
 		return unTapNetwork(endpoint.TapInterface.TAPIface.Name)
 	}); err != nil {

src/runtime/virtcontainers/tuntap_endpoint.go

@@ -18,6 +18,8 @@ import (
 	vcTypes "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/types"
 )
 
+var tuntapTrace = getNetworkTrace(TuntapEndpointType)
+
 // TuntapEndpoint represents just a tap endpoint
 type TuntapEndpoint struct {
 	NetPair            NetworkInterfacePair
@@ -71,11 +73,15 @@ func (endpoint *TuntapEndpoint) SetProperties(properties NetworkInfo) {
 
 // Attach for tap endpoint adds the tap interface to the hypervisor.
 func (endpoint *TuntapEndpoint) Attach(ctx context.Context, s *Sandbox) error {
+	span, ctx := tuntapTrace(ctx, "Attach", endpoint)
+	defer span.End()
+
 	h := s.hypervisor
 	if err := xConnectVMNetwork(ctx, endpoint, h); err != nil {
 		networkLogger().WithError(err).Error("Error bridging virtual endpoint")
 		return err
 	}
+
 	return h.addDevice(ctx, endpoint, netDev)
 }
 
@@ -85,6 +91,9 @@ func (endpoint *TuntapEndpoint) Detach(ctx context.Context, netNsCreated bool, n
 		return nil
 	}
 
+	span, _ := tuntapTrace(ctx, "Detach", endpoint)
+	defer span.End()
+
 	networkLogger().WithField("endpoint-type", TuntapEndpointType).Info("Detaching endpoint")
 	return doNetNS(netNsPath, func(_ ns.NetNS) error {
 		return unTuntapNetwork(endpoint.TuntapInterface.TAPIface.Name)
@@ -94,6 +103,10 @@ func (endpoint *TuntapEndpoint) Detach(ctx context.Context, netNsCreated bool, n
 // HotAttach for the tap endpoint uses hot plug device
 func (endpoint *TuntapEndpoint) HotAttach(ctx context.Context, h hypervisor) error {
 	networkLogger().Info("Hot attaching tap endpoint")
+
+	span, ctx := tuntapTrace(ctx, "HotAttach", endpoint)
+	defer span.End()
+
 	if err := tuntapNetwork(endpoint, h.hypervisorConfig().NumVCPUs, h.hypervisorConfig().DisableVhostNet); err != nil {
 		networkLogger().WithError(err).Error("Error bridging tap ep")
 		return err
@@ -109,6 +122,10 @@ func (endpoint *TuntapEndpoint) HotAttach(ctx context.Context, h hypervisor) err
 // HotDetach for the tap endpoint uses hot pull device
 func (endpoint *TuntapEndpoint) HotDetach(ctx context.Context, h hypervisor, netNsCreated bool, netNsPath string) error {
 	networkLogger().Info("Hot detaching tap endpoint")
+
+	span, ctx := tuntapTrace(ctx, "HotDetach", endpoint)
+	defer span.End()
+
 	if err := doNetNS(netNsPath, func(_ ns.NetNS) error {
 		return unTuntapNetwork(endpoint.TuntapInterface.TAPIface.Name)
 	}); err != nil {

src/runtime/virtcontainers/veth_endpoint.go

@@ -14,6 +14,8 @@ import (
 	vcTypes "github.com/kata-containers/kata-containers/src/runtime/virtcontainers/pkg/types"
 )
 
+var vethTrace = getNetworkTrace(VethEndpointType)
+
 // VethEndpoint gathers a network pair and its properties.
 type VethEndpoint struct {
 	NetPair            NetworkInterfacePair
@@ -92,6 +94,9 @@ func (endpoint *VethEndpoint) SetProperties(properties NetworkInfo) {
 // Attach for veth endpoint bridges the network pair and adds the
 // tap interface of the network pair to the hypervisor.
 func (endpoint *VethEndpoint) Attach(ctx context.Context, s *Sandbox) error {
+	span, ctx := vethTrace(ctx, "Attach", endpoint)
+	defer span.End()
+
 	h := s.hypervisor
 	if err := xConnectVMNetwork(ctx, endpoint, h); err != nil {
 		networkLogger().WithError(err).Error("Error bridging virtual endpoint")
@@ -110,13 +115,19 @@ func (endpoint *VethEndpoint) Detach(ctx context.Context, netNsCreated bool, net
 		return nil
 	}
 
+	span, ctx := vethTrace(ctx, "Detach", endpoint)
+	defer span.End()
+
 	return doNetNS(netNsPath, func(_ ns.NetNS) error {
-		return xDisconnectVMNetwork(endpoint)
+		return xDisconnectVMNetwork(ctx, endpoint)
 	})
 }
 
 // HotAttach for the veth endpoint uses hot plug device
 func (endpoint *VethEndpoint) HotAttach(ctx context.Context, h hypervisor) error {
+	span, ctx := vethTrace(ctx, "HotAttach", endpoint)
+	defer span.End()
+
 	if err := xConnectVMNetwork(ctx, endpoint, h); err != nil {
 		networkLogger().WithError(err).Error("Error bridging virtual ep")
 		return err
@@ -135,8 +146,11 @@ func (endpoint *VethEndpoint) HotDetach(ctx context.Context, h hypervisor, netNs
 		return nil
 	}
 
+	span, ctx := vethTrace(ctx, "HotDetach", endpoint)
+	defer span.End()
+
 	if err := doNetNS(netNsPath, func(_ ns.NetNS) error {
-		return xDisconnectVMNetwork(endpoint)
+		return xDisconnectVMNetwork(ctx, endpoint)
 	}); err != nil {
 		networkLogger().WithError(err).Warn("Error un-bridging virtual ep")
 	}

src/runtime/virtcontainers/vhostuser_endpoint.go

@@ -23,6 +23,8 @@ import (
 // using this path.
 const hostSocketSearchPath = "/tmp/vhostuser_%s/vhu.sock"
 
+var vhostuserTrace = getNetworkTrace(VhostUserEndpointType)
+
 // VhostUserEndpoint represents a vhost-user socket based network interface
 type VhostUserEndpoint struct {
 	// Path to the vhost-user socket on the host system
@@ -77,6 +79,9 @@ func (endpoint *VhostUserEndpoint) NetworkPair() *NetworkInterfacePair {
 
 // Attach for vhostuser endpoint
 func (endpoint *VhostUserEndpoint) Attach(ctx context.Context, s *Sandbox) error {
+	span, ctx := vhostuserTrace(ctx, "Attach", endpoint)
+	defer span.End()
+
 	// Generate a unique ID to be used for hypervisor commandline fields
 	randBytes, err := utils.GenerateRandomBytes(8)
 	if err != nil {

src/runtime/virtcontainers/virtiofsd.go

@@ -9,14 +9,12 @@ import (
 	"bufio"
 	"context"
 	"fmt"
-	"io"
 	"net"
 	"os"
 	"os/exec"
 	"path/filepath"
 	"strings"
 	"syscall"
-	"time"
 
 	"github.com/kata-containers/kata-containers/src/runtime/virtcontainers/utils"
 	"github.com/pkg/errors"
@@ -26,20 +24,22 @@ import (
 	otelTrace "go.opentelemetry.io/otel/trace"
 )
 
-const (
-	//Timeout to wait in secounds
-	virtiofsdStartTimeout = 5
+var (
+	errVirtiofsdDaemonPathEmpty    = errors.New("virtiofsd daemon path is empty")
+	errVirtiofsdSocketPathEmpty    = errors.New("virtiofsd socket path is empty")
+	errVirtiofsdSourcePathEmpty    = errors.New("virtiofsd source path is empty")
+	errVirtiofsdSourceNotAvailable = errors.New("virtiofsd source path not available")
 )
 
 type Virtiofsd interface {
 	// Start virtiofsd, return pid of virtiofsd process
-	Start(context.Context) (pid int, err error)
+	Start(context.Context, onQuitFunc) (pid int, err error)
 	// Stop virtiofsd process
 	Stop(context.Context) error
 }
 
-// Helper function to check virtiofsd is serving
-type virtiofsdWaitFunc func(runningCmd *exec.Cmd, stderr io.ReadCloser, debug bool) error
+// Helper function to execute when virtiofsd quit
+type onQuitFunc func()
 
 type virtiofsd struct {
 	// path to virtiofsd daemon
@@ -58,8 +58,6 @@ type virtiofsd struct {
 	PID int
 	// Neded by tracing
 	ctx context.Context
-	// wait helper function to check if virtiofsd is serving
-	wait virtiofsdWaitFunc
 }
 
 // Open socket on behalf of virtiofsd
@@ -75,6 +73,8 @@ func (v *virtiofsd) getSocketFD() (*os.File, error) {
 	if err != nil {
 		return nil, err
 	}
+
+	// no longer needed since fd is a dup
 	defer listener.Close()
 
 	listener.SetUnlinkOnClose(false)
@@ -83,7 +83,7 @@ func (v *virtiofsd) getSocketFD() (*os.File, error) {
 }
 
 // Start the virtiofsd daemon
-func (v *virtiofsd) Start(ctx context.Context) (int, error) {
+func (v *virtiofsd) Start(ctx context.Context, onQuit onQuitFunc) (int, error) {
 	span, _ := v.trace(ctx, "Start")
 	defer span.End()
 	pid := 0
@@ -98,6 +98,7 @@ func (v *virtiofsd) Start(ctx context.Context) (int, error) {
 	if err != nil {
 		return 0, err
 	}
+	defer socketFD.Close()
 
 	cmd.ExtraFiles = append(cmd.ExtraFiles, socketFD)
 
@@ -113,22 +114,30 @@ func (v *virtiofsd) Start(ctx context.Context) (int, error) {
 
 	v.Logger().WithField("path", v.path).Info()
 	v.Logger().WithField("args", strings.Join(args, " ")).Info()
+	stderr, err := cmd.StderrPipe()
+	if err != nil {
+		return pid, err
+	}
 
 	if err = utils.StartCmd(cmd); err != nil {
 		return pid, err
 	}
 
-	defer func() {
-		if err != nil {
-			cmd.Process.Kill()
+	// Monitor virtiofsd's stderr and stop sandbox if virtiofsd quits
+	go func() {
+		scanner := bufio.NewScanner(stderr)
+		for scanner.Scan() {
+			v.Logger().WithField("source", "virtiofsd").Info(scanner.Text())
+		}
+		v.Logger().Info("virtiofsd quits")
+		// Wait to release resources of virtiofsd process
+		cmd.Process.Wait()
+		if onQuit != nil {
+			onQuit()
 		}
 	}()
 
-	if v.wait == nil {
-		v.wait = waitVirtiofsReady
-	}
-
-	return pid, socketFD.Close()
+	return cmd.Process.Pid, nil
 }
 
 func (v *virtiofsd) Stop(ctx context.Context) error {
@@ -136,10 +145,6 @@ func (v *virtiofsd) Stop(ctx context.Context) error {
 		return nil
 	}
 
-	if v.socketPath == "" {
-		return errors.New("vitiofsd socket path is empty")
-	}
-
 	err := os.Remove(v.socketPath)
 	if err != nil {
 		v.Logger().WithError(err).WithField("path", v.socketPath).Warn("removing virtiofsd socket failed")
@@ -148,19 +153,10 @@ func (v *virtiofsd) Stop(ctx context.Context) error {
 }
 
 func (v *virtiofsd) args(FdSocketNumber uint) ([]string, error) {
-	if v.sourcePath == "" {
-		return []string{}, errors.New("vitiofsd source path is empty")
-	}
-
-	if _, err := os.Stat(v.sourcePath); os.IsNotExist(err) {
-		return nil, err
-	}
 
 	args := []string{
 		// Send logs to syslog
 		"--syslog",
-		// foreground operation
-		"-f",
 		// cache mode for virtiofsd
 		"-o", "cache=" + v.cache,
 		// disable posix locking in daemon: bunch of basic posix locks properties are broken
@@ -173,7 +169,11 @@ func (v *virtiofsd) args(FdSocketNumber uint) ([]string, error) {
 	}
 
 	if v.debug {
-		args = append(args, "-o", "debug")
+		// enable debug output (implies -f)
+		args = append(args, "-d")
+	} else {
+		// foreground operation
+		args = append(args, "-f")
 	}
 
 	if len(v.extraArgs) != 0 {
@@ -185,18 +185,20 @@ func (v *virtiofsd) args(FdSocketNumber uint) ([]string, error) {
 
 func (v *virtiofsd) valid() error {
 	if v.path == "" {
-		errors.New("virtiofsd path is empty")
+		return errVirtiofsdDaemonPathEmpty
 	}
 
 	if v.socketPath == "" {
-		errors.New("Virtiofsd socket path is empty")
+		return errVirtiofsdSocketPathEmpty
 	}
 
 	if v.sourcePath == "" {
-		errors.New("virtiofsd source path is empty")
-
+		return errVirtiofsdSourcePathEmpty
 	}
 
+	if _, err := os.Stat(v.sourcePath); err != nil {
+		return errVirtiofsdSourceNotAvailable
+	}
 	return nil
 }
 
@@ -210,55 +212,11 @@ func (v *virtiofsd) trace(parent context.Context, name string) (otelTrace.Span,
 	}
 
 	tracer := otel.Tracer("kata")
-	ctx, span := tracer.Start(parent, name)
-	span.SetAttributes(label.Key("subsystem").String("virtiofds"))
+	ctx, span := tracer.Start(parent, name, otelTrace.WithAttributes(label.String("source", "runtime"), label.String("package", "virtcontainers"), label.String("subsystem", "virtiofsd")))
 
 	return span, ctx
 }
 
-func waitVirtiofsReady(cmd *exec.Cmd, stderr io.ReadCloser, debug bool) error {
-	if cmd == nil {
-		return errors.New("cmd is nil")
-	}
-
-	sockReady := make(chan error, 1)
-	go func() {
-		scanner := bufio.NewScanner(stderr)
-		var sent bool
-		for scanner.Scan() {
-			if debug {
-				virtLog.WithField("source", "virtiofsd").Debug(scanner.Text())
-			}
-			if !sent && strings.Contains(scanner.Text(), "Waiting for vhost-user socket connection...") {
-				sockReady <- nil
-				sent = true
-			}
-
-		}
-		if !sent {
-			if err := scanner.Err(); err != nil {
-				sockReady <- err
-
-			} else {
-				sockReady <- fmt.Errorf("virtiofsd did not announce socket connection")
-
-			}
-
-		}
-		// Wait to release resources of virtiofsd process
-		cmd.Process.Wait()
-	}()
-
-	var err error
-	select {
-	case err = <-sockReady:
-	case <-time.After(virtiofsdStartTimeout * time.Second):
-		err = fmt.Errorf("timed out waiting for vitiofsd ready mesage pid=%d", cmd.Process.Pid)
-	}
-
-	return err
-}
-
 func (v *virtiofsd) kill(ctx context.Context) (err error) {
 	span, _ := v.trace(ctx, "kill")
 	defer span.End()
@@ -280,7 +238,7 @@ type virtiofsdMock struct {
 }
 
 // Start the virtiofsd daemon
-func (v *virtiofsdMock) Start(ctx context.Context) (int, error) {
+func (v *virtiofsdMock) Start(ctx context.Context, onQuit onQuitFunc) (int, error) {
 	return 9999999, nil
 }
 

src/runtime/virtcontainers/virtiofsd_test.go

@@ -7,10 +7,9 @@ package virtcontainers
 
 import (
 	"context"
-	"io"
 	"io/ioutil"
 	"os"
-	"os/exec"
+	"strings"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -67,13 +66,9 @@ func TestVirtiofsdStart(t *testing.T) {
 				debug:      tt.fields.debug,
 				PID:        tt.fields.PID,
 				ctx:        tt.fields.ctx,
-				//Mock  wait function
-				wait: func(runningCmd *exec.Cmd, stderr io.ReadCloser, debug bool) error {
-					return nil
-				},
 			}
 			var ctx context.Context
-			_, err := v.Start(ctx)
+			_, err := v.Start(ctx, nil)
 			if (err != nil) != tt.wantErr {
 				t.Errorf("virtiofsd.Start() error = %v, wantErr %v", err, tt.wantErr)
 				return
@@ -81,3 +76,71 @@ func TestVirtiofsdStart(t *testing.T) {
 		})
 	}
 }
+
+func TestVirtiofsdArgs(t *testing.T) {
+	assert := assert.New(t)
+
+	v := &virtiofsd{
+		path:       "/usr/bin/virtiofsd",
+		sourcePath: "/run/kata-shared/foo",
+		cache:      "none",
+	}
+
+	expected := "--syslog -o cache=none -o no_posix_lock -o source=/run/kata-shared/foo --fd=123 -f"
+	args, err := v.args(123)
+	assert.NoError(err)
+	assert.Equal(expected, strings.Join(args, " "))
+
+	v.debug = false
+	expected = "--syslog -o cache=none -o no_posix_lock -o source=/run/kata-shared/foo --fd=456 -f"
+	args, err = v.args(456)
+	assert.NoError(err)
+	assert.Equal(expected, strings.Join(args, " "))
+}
+
+func TestValid(t *testing.T) {
+	assert := assert.New(t)
+
+	sourcePath, err := ioutil.TempDir("", "")
+	assert.NoError(err)
+	defer os.RemoveAll(sourcePath)
+
+	socketDir, err := ioutil.TempDir("", "")
+	assert.NoError(err)
+	defer os.RemoveAll(socketDir)
+
+	socketPath := socketDir + "socket.s"
+
+	newVirtiofsdFunc := func() *virtiofsd {
+		return &virtiofsd{
+			path:       "/usr/bin/virtiofsd",
+			sourcePath: sourcePath,
+			socketPath: socketPath,
+		}
+	}
+
+	// valid case
+	v := newVirtiofsdFunc()
+	err = v.valid()
+	assert.NoError(err)
+
+	v = newVirtiofsdFunc()
+	v.path = ""
+	err = v.valid()
+	assert.Equal(errVirtiofsdDaemonPathEmpty, err)
+
+	v = newVirtiofsdFunc()
+	v.sourcePath = ""
+	err = v.valid()
+	assert.Equal(errVirtiofsdSourcePathEmpty, err)
+
+	v = newVirtiofsdFunc()
+	v.socketPath = ""
+	err = v.valid()
+	assert.Equal(errVirtiofsdSocketPathEmpty, err)
+
+	v = newVirtiofsdFunc()
+	v.sourcePath = "/foo/bar"
+	err = v.valid()
+	assert.Equal(errVirtiofsdSourceNotAvailable, err)
+}

src/trace-forwarder/Cargo.lock

@@ -1,14 +1,5 @@
 # This file is automatically @generated by Cargo.
 # It is not intended for manual editing.
-[[package]]
-name = "aho-corasick"
-version = "0.7.13"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "043164d8ba5c4c3035fec9bbee8647c0261d788f3474306f93bb65901cae0e86"
-dependencies = [
- "memchr",
-]
-
 [[package]]
 name = "ansi_term"
 version = "0.11.0"
@@ -18,6 +9,15 @@ dependencies = [
  "winapi",
 ]
 
+[[package]]
+name = "ansi_term"
+version = "0.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d52a9bb7ec0cf484c551830a7ce27bd20d67eac647e1befb56b0be4ee39a55d2"
+dependencies = [
+ "winapi",
+]
+
 [[package]]
 name = "anyhow"
 version = "1.0.31"
@@ -30,6 +30,17 @@ version = "0.4.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4d25d88fd6b8041580a654f9d0c581a047baee2b3efee13275f2fc392fc75034"
 
+[[package]]
+name = "async-trait"
+version = "0.1.50"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b98e84bbb4cbcdd97da190ba0c58a1bb0de2c1fdf67d159e192ed766aeca722"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "atty"
 version = "0.2.14"
@@ -49,11 +60,10 @@ checksum = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"
 
 [[package]]
 name = "bincode"
-version = "1.3.1"
+version = "1.3.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f30d3a39baa26f9651f17b375061f3233dde33424a8b72b0dbe93a68a0bc896d"
+checksum = "b1f45e9417d87227c7a56d22e471c6206462cba514c7590c09aff4cf6d1ddcad"
 dependencies = [
- "byteorder",
  "serde",
 ]
 
@@ -64,10 +74,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
 
 [[package]]
-name = "byteorder"
-version = "1.3.4"
+name = "bumpalo"
+version = "3.6.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"
+checksum = "63396b8a4b9de3f4fdfb320ab6080762242f66a8ef174c49d8e19b674db4cdbe"
+
+[[package]]
+name = "byteorder"
+version = "1.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
 
 [[package]]
 name = "cc"
@@ -82,14 +98,22 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
 
 [[package]]
-name = "chrono"
-version = "0.4.11"
+name = "cfg-if"
+version = "1.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "80094f509cf8b5ae86a4966a39b3ff66cd7e2a3e594accec3743ff3fabeab5b2"
+checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+
+[[package]]
+name = "chrono"
+version = "0.4.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "670ad68c9088c2a963aaa298cb369688cf3f9465ce5e2d4ca10e6e0098a1ce73"
 dependencies = [
+ "libc",
  "num-integer",
  "num-traits",
  "time",
+ "winapi",
 ]
 
 [[package]]
@@ -98,7 +122,7 @@ version = "2.33.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bdfa80d47f954d53a35a64987ca1422f495b8d6483c0fe9f7117b36c2a792129"
 dependencies = [
- "ansi_term",
+ "ansi_term 0.11.0",
  "atty",
  "bitflags",
  "strsim",
@@ -113,10 +137,20 @@ version = "0.4.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cced8691919c02aac3cb0a1bc2e9b73d89e832bf9a06fc579d4e71b68a2da061"
 dependencies = [
- "crossbeam-utils",
+ "crossbeam-utils 0.7.2",
  "maybe-uninit",
 ]
 
+[[package]]
+name = "crossbeam-channel"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "06ed27e177f16d65f0f0c22a213e17c696ace5dd64b14258b52f9417ccb52db4"
+dependencies = [
+ "cfg-if 1.0.0",
+ "crossbeam-utils 0.8.4",
+]
+
 [[package]]
 name = "crossbeam-utils"
 version = "0.7.2"
@@ -124,21 +158,26 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c3c7c73a2d1e9fc0886a08b93e98eb643461230d5f1925e4036204d5f2e261a8"
 dependencies = [
  "autocfg",
- "cfg-if",
+ "cfg-if 0.1.10",
  "lazy_static",
 ]
 
 [[package]]
-name = "fnv"
-version = "1.0.7"
+name = "crossbeam-utils"
+version = "0.8.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
+checksum = "4feb231f0d4d6af81aed15928e58ecf5816aa62a2393e2c82f46973e92a9a278"
+dependencies = [
+ "autocfg",
+ "cfg-if 1.0.0",
+ "lazy_static",
+]
 
 [[package]]
 name = "futures"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1e05b85ec287aac0dc34db7d4a569323df697f9c55b99b15d6b4ef8cde49f613"
+checksum = "0e7e43a803dae2fa37c1f6a8fe121e1f7bf9548b4dfc0522a42f34145dadfc27"
 dependencies = [
  "futures-channel",
  "futures-core",
@@ -151,9 +190,9 @@ dependencies = [
 
 [[package]]
 name = "futures-channel"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f366ad74c28cca6ba456d95e6422883cfb4b252a83bed929c83abfdbbf2967d5"
+checksum = "e682a68b29a882df0545c143dc3646daefe80ba479bcdede94d5a703de2871e2"
 dependencies = [
  "futures-core",
  "futures-sink",
@@ -161,15 +200,15 @@ dependencies = [
 
 [[package]]
 name = "futures-core"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "59f5fff90fd5d971f936ad674802482ba441b6f09ba5e15fd8b39145582ca399"
+checksum = "0402f765d8a89a26043b889b26ce3c4679d268fa6bb22cd7c6aad98340e179d1"
 
 [[package]]
 name = "futures-executor"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "10d6bb888be1153d3abeb9006b11b02cf5e9b209fda28693c31ae1e4e012e314"
+checksum = "badaa6a909fac9e7236d0620a2f57f7664640c56575b71a7552fbd68deafab79"
 dependencies = [
  "futures-core",
  "futures-task",
@@ -178,16 +217,17 @@ dependencies = [
 
 [[package]]
 name = "futures-io"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "de27142b013a8e869c14957e6d2edeef89e97c289e69d042ee3a49acd8b51789"
+checksum = "acc499defb3b348f8d8f3f66415835a9131856ff7714bf10dadfc4ec4bdb29a1"
 
 [[package]]
 name = "futures-macro"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d0b5a30a4328ab5473878237c447333c093297bded83a4983d10f4deea240d39"
+checksum = "a4c40298486cdf52cc00cd6d6987892ba502c7656a16a4192a9992b1ccedd121"
 dependencies = [
+ "autocfg",
  "proc-macro-hack",
  "proc-macro2",
  "quote",
@@ -196,25 +236,23 @@ dependencies = [
 
 [[package]]
 name = "futures-sink"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3f2032893cb734c7a05d85ce0cc8b8c4075278e93b24b66f9de99d6eb0fa8acc"
+checksum = "a57bead0ceff0d6dde8f465ecd96c9338121bb7717d3e7b108059531870c4282"
 
 [[package]]
 name = "futures-task"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bdb66b5f09e22019b1ab0830f7785bcea8e7a42148683f99214f73f8ec21a626"
-dependencies = [
- "once_cell",
-]
+checksum = "8a16bef9fc1a4dddb5bee51c989e3fbba26569cbb0e31f5b303c184e3dd33dae"
 
 [[package]]
 name = "futures-util"
-version = "0.3.5"
+version = "0.3.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8764574ff08b701a084482c3c7031349104b07ac897393010494beaa18ce32c6"
+checksum = "feb5c238d27e2bf94ffdfd27b2c29e3df4a68c4193bb6427384259e2bf191967"
 dependencies = [
+ "autocfg",
  "futures-channel",
  "futures-core",
  "futures-io",
@@ -222,7 +260,7 @@ dependencies = [
  "futures-sink",
  "futures-task",
  "memchr",
- "pin-project",
+ "pin-project-lite",
  "pin-utils",
  "proc-macro-hack",
  "proc-macro-nested",
@@ -235,9 +273,20 @@ version = "0.1.14"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7abc8dd8451921606d809ba32e95b6111925cd2906060d2dcc29c070220503eb"
 dependencies = [
- "cfg-if",
+ "cfg-if 0.1.10",
  "libc",
- "wasi",
+ "wasi 0.9.0+wasi-snapshot-preview1",
+]
+
+[[package]]
+name = "getrandom"
+version = "0.2.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7fcd999463524c52659517fe2cea98493cfe485d10565e7b0fb07dbba7ad2753"
+dependencies = [
+ "cfg-if 1.0.0",
+ "libc",
+ "wasi 0.10.2+wasi-snapshot-preview1",
 ]
 
 [[package]]
@@ -261,6 +310,15 @@ version = "0.4.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "dc6f3ad7b9d11a0c00842ff8de1b60ee58661048eb8049ed33c73594f359d7e6"
 
+[[package]]
+name = "js-sys"
+version = "0.3.51"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "83bdfbace3a0e81a4253f73b49e960b053e396a11012cbd49b9b74d6a2b67062"
+dependencies = [
+ "wasm-bindgen",
+]
+
 [[package]]
 name = "kata-trace-forwarder"
 version = "0.0.1"
@@ -269,9 +327,10 @@ dependencies = [
  "bincode",
  "byteorder",
  "clap",
+ "futures",
  "libc",
  "logging",
- "nix",
+ "nix 0.20.0",
  "opentelemetry",
  "opentelemetry-jaeger",
  "protobuf",
@@ -293,9 +352,9 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
 
 [[package]]
 name = "libc"
-version = "0.2.71"
+version = "0.2.94"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9457b06509d27052635f90d6466700c65095fdf75409b3fbdd903e988b886f49"
+checksum = "18794a8ad5b29321f790b55d93dfba91e125cb1a9edbd4f8e3150acc771c1a5e"
 
 [[package]]
 name = "log"
@@ -303,7 +362,7 @@ version = "0.4.8"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "14b6052be84e6b71ab17edffc2eeabf5c2c3ae1fdb464aae35ac50c67a44e1f7"
 dependencies = [
- "cfg-if",
+ "cfg-if 0.1.10",
 ]
 
 [[package]]
@@ -340,15 +399,26 @@ checksum = "3728d817d99e5ac407411fa471ff9800a778d88a24685968b36824eaf4bee400"
 
 [[package]]
 name = "nix"
-version = "0.15.0"
+version = "0.19.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3b2e0b4f3320ed72aaedb9a5ac838690a8047c7b275da22711fddff4f8a14229"
+checksum = "b2ccba0cfe4fdf15982d1674c69b1fd80bad427d293849982668dfe454bd61f2"
 dependencies = [
  "bitflags",
  "cc",
- "cfg-if",
+ "cfg-if 1.0.0",
+ "libc",
+]
+
+[[package]]
+name = "nix"
+version = "0.20.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fa9b4819da1bc61c0ea48b63b7bc8604064dd43013e7cc325df098d49cd7c18a"
+dependencies = [
+ "bitflags",
+ "cc",
+ "cfg-if 1.0.0",
  "libc",
- "void",
 ]
 
 [[package]]
@@ -380,35 +450,34 @@ dependencies = [
  "libc",
 ]
 
-[[package]]
-name = "once_cell"
-version = "1.4.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0b631f7e854af39a1739f401cf34a8a013dfe09eac4fa4dba91e9768bd28168d"
-
 [[package]]
 name = "opentelemetry"
-version = "0.5.0"
+version = "0.14.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e1dbc1028f1f215373e28473238527c8799fbe88c58bae255585602c7316fa30"
+checksum = "492848ff47f11b7f9de0443b404e2c5775f695e1af6b7076ca25f999581d547a"
 dependencies = [
- "bincode",
+ "async-trait",
+ "crossbeam-channel 0.5.1",
  "futures",
+ "js-sys",
  "lazy_static",
  "percent-encoding",
  "pin-project",
- "prometheus",
- "rand",
+ "rand 0.8.3",
  "serde",
+ "thiserror",
 ]
 
 [[package]]
 name = "opentelemetry-jaeger"
-version = "0.4.0"
+version = "0.13.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4da8b4162e8ae52c894f23895477f0d0340d34d8d35c367edf53de83adb1a6df"
+checksum = "97fd9ed34f208e0394bfb17522ba0d890925685dfd883147670ed474339d4647"
 dependencies = [
+ "async-trait",
+ "lazy_static",
  "opentelemetry",
+ "thiserror",
  "thrift",
 ]
 
@@ -429,24 +498,30 @@ checksum = "d4fd5641d01c8f18a23da7b6fe29298ff4b55afcccdf78973b24cf3175fee32e"
 
 [[package]]
 name = "pin-project"
-version = "0.4.22"
+version = "1.0.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "12e3a6cdbfe94a5e4572812a0201f8c0ed98c1c452c7b8563ce2276988ef9c17"
+checksum = "c7509cc106041c40a4518d2af7a61530e1eed0e6285296a3d8c5472806ccc4a4"
 dependencies = [
  "pin-project-internal",
 ]
 
 [[package]]
 name = "pin-project-internal"
-version = "0.4.22"
+version = "1.0.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6a0ffd45cf79d88737d7cc85bfd5d2894bee1139b356e616fe85dc389c61aaf7"
+checksum = "48c950132583b500556b1efd71d45b319029f2b71518d979fcc208e16b42426f"
 dependencies = [
  "proc-macro2",
  "quote",
  "syn",
 ]
 
+[[package]]
+name = "pin-project-lite"
+version = "0.2.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dc0e1f259c92177c30a4c9d177246edd0a3568b25756a977d0632cf8fa37e905"
+
 [[package]]
 name = "pin-utils"
 version = "0.1.0"
@@ -461,9 +536,9 @@ checksum = "237a5ed80e274dbc66f86bd59c1e25edc039660be53194b5fe0a482e0f2612ea"
 
 [[package]]
 name = "proc-macro-hack"
-version = "0.5.16"
+version = "0.5.19"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7e0456befd48169b9f13ef0f0ad46d492cf9d2dbb918bcf38e01eed4ce3ec5e4"
+checksum = "dbf0c48bc1d91375ae5c3cd81e3722dff1abcf81a30960240640d223f59fe0e5"
 
 [[package]]
 name = "proc-macro-nested"
@@ -473,39 +548,19 @@ checksum = "eba180dafb9038b050a4c280019bbedf9f2467b61e5d892dcad585bb57aadc5a"
 
 [[package]]
 name = "proc-macro2"
-version = "1.0.18"
+version = "1.0.27"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "beae6331a816b1f65d04c45b078fd8e6c93e8071771f41b8163255bbd8d7c8fa"
+checksum = "f0d8caf72986c1a598726adc988bb5984792ef84f5ee5aa50209145ee8077038"
 dependencies = [
  "unicode-xid",
 ]
 
-[[package]]
-name = "prometheus"
-version = "0.7.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5567486d5778e2c6455b1b90ff1c558f29e751fc018130fa182e15828e728af1"
-dependencies = [
- "cfg-if",
- "fnv",
- "lazy_static",
- "protobuf",
- "quick-error",
- "spin",
-]
-
 [[package]]
 name = "protobuf"
 version = "2.14.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8e86d370532557ae7573551a1ec8235a0f8d6cb276c7c9e6aa490b511c447485"
 
-[[package]]
-name = "quick-error"
-version = "1.2.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
-
 [[package]]
 name = "quote"
 version = "1.0.7"
@@ -521,11 +576,23 @@ version = "0.7.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "6a6b1679d49b24bbfe0c803429aa1874472f50d9b363131f0e89fc356b544d03"
 dependencies = [
- "getrandom",
+ "getrandom 0.1.14",
  "libc",
- "rand_chacha",
- "rand_core",
- "rand_hc",
+ "rand_chacha 0.2.2",
+ "rand_core 0.5.1",
+ "rand_hc 0.2.0",
+]
+
+[[package]]
+name = "rand"
+version = "0.8.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0ef9e7e66b4468674bfcb0c81af8b7fa0bb154fa9f28eb840da5c447baeb8d7e"
+dependencies = [
+ "libc",
+ "rand_chacha 0.3.0",
+ "rand_core 0.6.2",
+ "rand_hc 0.3.0",
 ]
 
 [[package]]
@@ -535,7 +602,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f4c8ed856279c9737206bf725bf36935d8666ead7aa69b52be55af369d193402"
 dependencies = [
  "ppv-lite86",
- "rand_core",
+ "rand_core 0.5.1",
+]
+
+[[package]]
+name = "rand_chacha"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e12735cf05c9e10bf21534da50a147b924d555dc7a547c42e6bb2d5b6017ae0d"
+dependencies = [
+ "ppv-lite86",
+ "rand_core 0.6.2",
 ]
 
 [[package]]
@@ -544,7 +621,16 @@ version = "0.5.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "90bde5296fc891b0cef12a6d03ddccc162ce7b2aff54160af9338f8d40df6d19"
 dependencies = [
- "getrandom",
+ "getrandom 0.1.14",
+]
+
+[[package]]
+name = "rand_core"
+version = "0.6.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "34cf66eb183df1c5876e2dcf6b13d57340741e8dc255b48e40a26de954d06ae7"
+dependencies = [
+ "getrandom 0.2.3",
 ]
 
 [[package]]
@@ -553,7 +639,16 @@ version = "0.2.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ca3129af7b92a17112d59ad498c6f81eaf463253766b90396d39ea7a39d6613c"
 dependencies = [
- "rand_core",
+ "rand_core 0.5.1",
+]
+
+[[package]]
+name = "rand_hc"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3190ef7066a446f2e7f42e239d161e905420ccab01eb967c9eb27d21b2322a73"
+dependencies = [
+ "rand_core 0.6.2",
 ]
 
 [[package]]
@@ -568,10 +663,7 @@ version = "1.3.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "9c3780fcf44b193bc4d09f36d2a3c87b251da4a046c87795a0d35f4f927ad8e6"
 dependencies = [
- "aho-corasick",
- "memchr",
  "regex-syntax",
- "thread_local",
 ]
 
 [[package]]
@@ -607,18 +699,18 @@ checksum = "71d301d4193d031abdd79ff7e3dd721168a9572ef3fe51a1517aba235bd8f86e"
 
 [[package]]
 name = "serde"
-version = "1.0.114"
+version = "1.0.126"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5317f7588f0a5078ee60ef675ef96735a1442132dc645eb1d12c018620ed8cd3"
+checksum = "ec7505abeacaec74ae4778d9d9328fe5a5d04253220a85c4ee022239fc996d03"
 dependencies = [
  "serde_derive",
 ]
 
 [[package]]
 name = "serde_derive"
-version = "1.0.114"
+version = "1.0.126"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2a0be94b04690fbaed37cddffc5c134bf537c8e3329d53e982fe04c374978f8e"
+checksum = "963a7dbc9895aeac7ac90e74f34a5d5261828f79df35cbed41e10189d3804d43"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -638,9 +730,9 @@ dependencies = [
 
 [[package]]
 name = "sharded-slab"
-version = "0.0.9"
+version = "0.1.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "06d5a3f5166fb5b42a5439f2eee8b9de149e235961e3eb21c5808fc3ea17ff3e"
+checksum = "79c719719ee05df97490f80a45acfc99e5a30ce98a1e4fb67aee422745ae14e3"
 dependencies = [
  "lazy_static",
 ]
@@ -663,7 +755,7 @@ version = "2.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "51b3336ce47ce2f96673499fc07eb85e3472727b9a7a2959964b002c2ce8fbbb"
 dependencies = [
- "crossbeam-channel",
+ "crossbeam-channel 0.4.2",
  "slog",
  "take_mut",
  "thread_local",
@@ -698,12 +790,6 @@ version = "1.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c7cb5678e1615754284ec264d9bb5b4c27d2018577fd90ac0ceb578591ed5ee4"
 
-[[package]]
-name = "spin"
-version = "0.5.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6e63cff320ae2c57904679ba7cb63280a3dc4613885beafb148ee7bf9aa9042d"
-
 [[package]]
 name = "strsim"
 version = "0.8.0"
@@ -712,9 +798,9 @@ checksum = "8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a"
 
 [[package]]
 name = "syn"
-version = "1.0.33"
+version = "1.0.72"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e8d5d96e8cbb005d6959f119f773bfaebb5684296108fb32600c00cde305b2cd"
+checksum = "a1e8cdbefb79a9a5a65e0db8b47b723ee907b7c7f8496c76a1770b5c310bab82"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -733,9 +819,9 @@ version = "3.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "7a6e24d9338a0a5be79593e2fa15a648add6138caa803e2d5bc782c371732ca9"
 dependencies = [
- "cfg-if",
+ "cfg-if 0.1.10",
  "libc",
- "rand",
+ "rand 0.7.3",
  "redox_syscall",
  "remove_dir_all",
  "winapi",
@@ -750,6 +836,26 @@ dependencies = [
  "unicode-width",
 ]
 
+[[package]]
+name = "thiserror"
+version = "1.0.25"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fa6f76457f59514c7eeb4e59d891395fab0b2fd1d40723ae737d64153392e9c6"
+dependencies = [
+ "thiserror-impl",
+]
+
+[[package]]
+name = "thiserror-impl"
+version = "1.0.25"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8a36768c0fbf1bb15eca10defa29526bda730a2376c2ab4393ccfa16fb1a318d"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "thread_local"
 version = "1.0.1"
@@ -793,20 +899,21 @@ dependencies = [
 
 [[package]]
 name = "tracing"
-version = "0.1.15"
+version = "0.1.26"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a41f40ed0e162c911ac6fcb53ecdc8134c46905fdbbae8c50add462a538b495f"
+checksum = "09adeb8c97449311ccd28a427f96fb563e7fd31aabf994189879d9da2394b89d"
 dependencies = [
- "cfg-if",
+ "cfg-if 1.0.0",
+ "pin-project-lite",
  "tracing-attributes",
  "tracing-core",
 ]
 
 [[package]]
 name = "tracing-attributes"
-version = "0.1.8"
+version = "0.1.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "99bbad0de3fd923c9c3232ead88510b783e5a4d16a6154adffa3d53308de984c"
+checksum = "c42e6fa53307c8a17e4ccd4dc81cf5ec38db9209f59b222210375b54ee40d1e2"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -815,18 +922,18 @@ dependencies = [
 
 [[package]]
 name = "tracing-core"
-version = "0.1.10"
+version = "0.1.18"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0aa83a9a47081cd522c09c81b31aec2c9273424976f922ad61c053b58350b715"
+checksum = "a9ff14f98b1a4b289c6248a023c1c2fa1491062964e9fed67ab29c4e4da4a052"
 dependencies = [
  "lazy_static",
 ]
 
 [[package]]
 name = "tracing-log"
-version = "0.1.1"
+version = "0.1.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5e0f8c7178e13481ff6765bd169b33e8d554c5d2bbede5e32c356194be02b9b9"
+checksum = "a6923477a48e41c1951f1999ef8bb5a3023eb723ceadafe78ffb65dc366761e3"
 dependencies = [
  "lazy_static",
  "log",
@@ -835,22 +942,22 @@ dependencies = [
 
 [[package]]
 name = "tracing-opentelemetry"
-version = "0.4.0"
+version = "0.13.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d79fdf3c82e6ea14c0f54cc9418aef57252150c6da0bc5f18553f0136f0dcd31"
+checksum = "e2f4cb277b92a8ba1170b3b911056428ce2ef9993351baf5965bb0359a2e5963"
 dependencies = [
  "opentelemetry",
- "rand",
  "tracing",
  "tracing-core",
+ "tracing-log",
  "tracing-subscriber",
 ]
 
 [[package]]
 name = "tracing-serde"
-version = "0.1.1"
+version = "0.1.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b6ccba2f8f16e0ed268fc765d9b7ff22e965e7185d32f8f1ec8294fe17d86e79"
+checksum = "fb65ea441fbb84f9f6748fd496cf7f63ec9af5bca94dd86456978d055e8eb28b"
 dependencies = [
  "serde",
  "tracing-core",
@@ -858,11 +965,11 @@ dependencies = [
 
 [[package]]
 name = "tracing-subscriber"
-version = "0.2.6"
+version = "0.2.18"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "04a11b459109e38ff6e1b580bafef4142a11d44889f5d07424cbce2fd2a2a119"
+checksum = "aa5553bf0883ba7c9cbe493b085c29926bd41b66afc31ff72cf17ff4fb60dcd5"
 dependencies = [
- "ansi_term",
+ "ansi_term 0.12.1",
  "chrono",
  "lazy_static",
  "matchers",
@@ -871,6 +978,8 @@ dependencies = [
  "serde_json",
  "sharded-slab",
  "smallvec",
+ "thread_local",
+ "tracing",
  "tracing-core",
  "tracing-log",
  "tracing-serde",
@@ -894,20 +1003,14 @@ version = "0.8.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "f1bddf1187be692e79c5ffeab891132dfb0f236ed36a43c7ed39f1165ee20191"
 
-[[package]]
-name = "void"
-version = "1.0.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d"
-
 [[package]]
 name = "vsock"
-version = "0.1.5"
+version = "0.2.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "63b4354dabee252603a8b1a63e4adb3934dfbf1ff05ef4c3d653c2dfd67f0788"
+checksum = "50e2ef09834e1d203d24556512c0e58e66de203440bd9d74c30a33f7240091c6"
 dependencies = [
  "libc",
- "nix",
+ "nix 0.19.1",
 ]
 
 [[package]]
@@ -916,6 +1019,66 @@ version = "0.9.0+wasi-snapshot-preview1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cccddf32554fecc6acb585f82a32a72e28b48f8c4c1883ddfeeeaa96f7d8e519"
 
+[[package]]
+name = "wasi"
+version = "0.10.2+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fd6fbd9a79829dd1ad0cc20627bf1ed606756a7f77edff7b66b7064f9cb327c6"
+
+[[package]]
+name = "wasm-bindgen"
+version = "0.2.74"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d54ee1d4ed486f78874278e63e4069fc1ab9f6a18ca492076ffb90c5eb2997fd"
+dependencies = [
+ "cfg-if 1.0.0",
+ "wasm-bindgen-macro",
+]
+
+[[package]]
+name = "wasm-bindgen-backend"
+version = "0.2.74"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3b33f6a0694ccfea53d94db8b2ed1c3a8a4c86dd936b13b9f0a15ec4a451b900"
+dependencies = [
+ "bumpalo",
+ "lazy_static",
+ "log",
+ "proc-macro2",
+ "quote",
+ "syn",
+ "wasm-bindgen-shared",
+]
+
+[[package]]
+name = "wasm-bindgen-macro"
+version = "0.2.74"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "088169ca61430fe1e58b8096c24975251700e7b1f6fd91cc9d59b04fb9b18bd4"
+dependencies = [
+ "quote",
+ "wasm-bindgen-macro-support",
+]
+
+[[package]]
+name = "wasm-bindgen-macro-support"
+version = "0.2.74"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "be2241542ff3d9f241f5e2cb6dd09b37efe786df8851c54957683a49f0987a97"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+ "wasm-bindgen-backend",
+ "wasm-bindgen-shared",
+]
+
+[[package]]
+name = "wasm-bindgen-shared"
+version = "0.2.74"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d7cff876b8f18eed75a66cf49b65e7f967cb354a7aa16003fb55dbfd25b44b4f"
+
 [[package]]
 name = "winapi"
 version = "0.3.9"