kata-containers/src/tools/trace-forwarder

Trace Forwarder

Overview

The Kata Containers trace forwarder, kata-trace-forwarder, is a component running on the host system which is used to support tracing the agent process, which runs inside the Kata Containers virtual machine (VM).

The trace forwarder, which must be started before the container, listens over VSOCK for trace data sent by the agent running inside the VM. The trace spans are exported using the OpenTelemetry Protocol (OTLP) to an OpenTelemetry-compatible collector running by default on the host.

Notes:

• If agent tracing is enabled but the forwarder is not running, the agent will log an error (signalling that it cannot generate trace spans), but continue to work as normal.

• The trace forwarder requires an OTLP-compatible trace collector to be running before it is started. If a collector is not running, the trace forwarder will exit with an error.

Quick start

1. Start an OTLP-compatible collector (such as Jaeger v1.35+, OpenTelemetry Collector, or Grafana Tempo).
2. Start the trace forwarder with the appropriate OTLP endpoint.
3. Ensure agent tracing is enabled in the Kata configuration file.
4. Create a Kata container as usual.

OTLP Endpoint Configuration

The trace forwarder sends spans to an OTLP endpoint. By default, it uses http://localhost:4317 (gRPC).

To specify a different endpoint, use the --otlp-endpoint flag:

kata-trace-forwarder --otlp-endpoint http://my-collector:4317

Common OTLP endpoints:

• Jaeger (v1.35+): http://localhost:4317 (gRPC) or http://localhost:4318 (HTTP)
• OpenTelemetry Collector: http://localhost:4317 (gRPC) or http://localhost:4318 (HTTP)
• Grafana Tempo: Depends on configuration

Run

The way the trace forwarder is run depends on the configured hypervisor.

Determine configured hypervisor

To identify which hypervisor Kata is configured to use, either look in the configuration file, or run:

kata-runtime env --json|jq '.Hypervisor.Path'

QEMU

Since QEMU supports VSOCK sockets in the standard way, if you are using QEMU simply run the trace forwarder using the default options:

Run the forwarder

cargo run

You can now proceed to create a Kata container as normal.

Cloud Hypervisor and Firecracker

Cloud Hypervisor and Firecracker both use "hybrid VSOCK" which uses a local UNIX socket rather than the host kernel to handle communication with the guest. As such, you need to specify the path to the UNIX socket.

Since the trace forwarder needs to be run before the VM (sandbox) is started and since the socket path is sandbox-specific, you need to run the env command to determine the "template path". This path includes a {ID} tag that represents the real sandbox ID or name.

Examples

Configured hypervisor is Cloud Hypervisor

$ socket_path_template=$(sudo kata-runtime env --json | jq '.Hypervisor.SocketPath')
$ echo "$socket_path_template"
"/run/vc/vm/{ID}/clh.sock"

Configured hypervisor is Firecracker

$ socket_path_template=$(sudo kata-runtime env --json | jq '.Hypervisor.SocketPath')
$ echo "$socket_path_template"
"/run/vc/firecracker/{ID}/root/kata.hvsock"

Note:

Do not rely on the paths shown above: you should run the command yourself as these paths may change.

Once you have determined the template path, build and install the forwarder, create the sandbox directory and then run the trace forwarder.

Build and install

If you are using the QEMU hypervisor, this step is not necessary.

If you are using Cloud Hypervisor of Firecracker, using the tool is simpler if it has been installed.

Build

make

Install

cargo install --path .
sudo install -o root -g root -m 0755 ~/.cargo/bin/kata-trace-forwarder /usr/local/bin

Create sandbox directory

You will need to change the sandbox_id variable below to match the name of the container (sandbox) you plan to create after starting the trace forwarder.

sandbox_id="foo"
socket_path=$(echo "$socket_path_template" | sed "s/{ID}/${sandbox_id}/g" | tr -d '"')
sudo mkdir -p $(dirname "$socket_path")

Note: The socket_path_template variable was set in the Cloud Hypervisor and Firecracker section.

Run the forwarder specifying socket path

sudo kata-trace-forwarder --socket-path "$socket_path"

You can now proceed as normal to create the "foo" Kata container.

Note:

Since the trace forwarder needs to create the socket in the sandbox directory, and since that directory is owned by the root user, the trace forwarder must also be run as root. This requirement is unique to hypervisors that use hybrid VSOCK: QEMU does not require special privileges to run the trace forwarder. To reduce the impact of this, once the forwarder is running it drops privileges to run as user nobody.

Full details

For further information on how to run the trace forwarder, run:

cargo run -- --help