Port the Go runtime's enable_vcpus_pinning feature to runtime-rs.
The Go runtime already lets users pin each vCPU thread to a specific
host CPU when the vCPU count matches the sandbox cpuset size, using
sched_setaffinity. This is useful for latency-sensitive workloads that
benefit from eliminating cross-CPU migration of vCPU threads.
The approach mirrors the Go implementation:
After VM start and on every container add/update/delete, we fetch the
vCPU thread IDs (via QMP query-cpus-fast for QEMU), compute the union of
all containers' OCI cpusets, and if the two counts match, pin vCPU i to
cpuset[i]. If they diverge (hotplug, container removal, etc.) we reset
all threads back to the full cpuset so nothing gets stuck on a single
core.
The pinning check lives in CgroupsResourceInner::update_sandbox_cgroups,
which already runs at exactly the right points in the lifecycle. The
enable_vcpus_pinning flag flows from the TOML config through
CgroupConfig into the cgroup resource layer, and can also be overridden
per-pod via the io.katacontainers.config.runtime.enable_vcpus_pinning
annotation.
The QEMU config templates default to false. The NV GPU configs will get
their own default (true) in a follow-up once those templates are added.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Update all versions of rand that are controlled by us to remediate
GHSA-cq8v-f236-94qc.
Note: There are still some usages of rand 0.8.5 it that are from
transitive dependencies which we can't currently update:
- fail
- phf_generator
- opentelemetry
due to them being archived, or our usage being 17 versions out of date
Also update the rand API breakages e.g. :
- rand::thread_rng() → rand::rng() (function renamed)
- rand::distributions::Alphanumeric → rand::distr::Alphanumeric (module renamed)
- rng.gen_range() → rng.random_range() (function renamed)
Assisted-by: IBM Bob
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add a new extensible GetDiagnosticData RPC that retrieves diagnostic
information from the guest VM. The request carries a log_type string
field to specify what kind of data is requested, and a container_id
field to identify the target container.
The first supported log_type is "termination_log", which reads the
Kubernetes termination message file from inside the guest. This is
needed for shared_fs=none configurations where the host cannot
directly access the guest filesystem.
On the Go runtime side, the container stop() path now calls
GetDiagnosticData to copy the termination message to the host
when running with NoSharedFS and the terminationMessagePolicy
annotation is set to "File". The call is best-effort: failures
are logged as warnings rather than blocking container teardown.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Signed-off-by: Silenio Quarti <silenio_quarti@ca.ibm.com>
The hardcoded DEFAULT_LAUNCH_PROCESS_TIMEOUT of 6 seconds in the kata
agent is insufficient for environments with NVIDIA GPUs and NVSwitches,
where the attestation-agent needs significantly more time to collect
evidence during initialization (e.g. ~2 seconds per NVSwitch).
When the timeout expires, the agent (PID 1) exits with an error, causing
the guest kernel to perform an orderly shutdown before the
attestation-agent has finished starting.
Make this timeout configurable via the kernel parameter
agent.launch_process_timeout (in seconds), preserving the 6-second
default for backward compatibility. The Go runtime is wired up to pass
this value from the TOML config's [agent.kata] section through to the
kernel command line.
The NVIDIA GPU configs set the new default to 15 seconds.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Made-with: Cursor
Add two new configuration knobs that control the logical and physical
sector sizes advertised by virtio-blk devices to the guest:
block_device_logical_sector_size (config file)
block_device_physical_sector_size (config file)
io.katacontainers.config.hypervisor.blk_logical_sector_size (annotation)
io.katacontainers.config.hypervisor.blk_physical_sector_size (annotation)
The annotation names are abbreviated relative to the config file keys
because Kubernetes enforces a 63-character limit on annotation name
segments, and the full names would exceed it.
Both settings default to 0 (let QEMU decide). When set, they are passed
as logical_block_size and physical_block_size in the QMP device_add
command during block device hotplug.
Setting logical_sector_size smaller then container filesystem
block size will cause EINVAL on mount. The physical_sector_size can
always be set independently.
Values must be 0 or a power of 2 in the range [512, 65536]; other
values are rejected with an error at sandbox creation time.
Signed-off-by: PiotrProkop <pprokop@nvidia.com>
Remove all virtio-9p related code and configurations:
(1) Remove DRIVER_9P_TYPE and VIRTIO_9P.
(2) Remove 9p validation and adjustment logic from SharedFsInfo.
(3) Remove KATA_ANNO_CFG_HYPERVISOR_MSIZE_9P annotation handling.
(4) Update test configurations to remove msize_9p settings.
(5) Update documentation and proto comments.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
(1) Update annotations documentation to reflect new shared filesystem
options (virtio-fs, inline-virtio-fs, virtio-fs-nydus, none).
(2) Replace virtio-9p references with inline-virtio-fs in config doc.
(3) Update drop-in configuration tests to use 'none' instead of 'virtio-9p'
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
..where possible. Failing on unknown fields makes migration easier,
as we do not silently ignore configuration options that previously
worked in runtime-go. However, serde can't deny unknown fields
where flatten is used, so this can't be used everywhere sadly.
There were also errors in test fixtures that were unnoticed.
These are fixed here, too.
Signed-off-by: Paul Meyer <katexochen0@gmail.com>
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
--all option would trigger building and testing for everything within
our root workspace, which is not desired here. Let's specify the crates
of libs explicitly in our Makefile.
Signed-off-by: Ruoqing He <ruoqing.he@lingcage.com>
Remove libs from exclude list, and move them explicitly into root
workspace to make sure our core components are in a consistent state.
This is a follow up of #12413.
Signed-off-by: Ruoqing He <ruoqing.he@lingcage.com>
couldn't initialise QMP: Connection reset by peer (os error 104)
Caused by:
Connection reset by peer (os error 104)
qemu stderr: "qemu-system-ppc64: Maximum memory size 0x80000000 is not aligned to 256 MiB”
When the default max memory was assigned according to the
available host memory, it failed with the above error
Align the memory values with the block size of 256 MB on ppc64le.
Signed-off-by: Amulyam24 <amulmek1@in.ibm.com>
Add all 9 library crates which are missing in workspace including:
(1) kata-types with annotations, hypervisor configs, and K8s utilities.
(2) kata-sys-util with all sub-modules: cpu, device, fs, hooks, k8s,
mount, netns, numa, pcilibs, protection, spec, validate.
(3) protocols with ttrpc bindings: agent, health, remote, csi, oci,
confidential_data_hub.
(4) runtime-spec with OCI container state types and namespace constants.
(5) shim-interface with RESTful API and Unix socket path.
(6) logging with slog framework features: JSON, journal, filtering.
(7) safe-path with security-focused path resolution utilities.
(8) mem-agent with memory management: memcg, compact, psi.
(9) test-utils with privilege and KVM test macros.
And one more thing, uniformly adopt TOCTOU in place of the redundant
TOCTTOU abbreviation.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
With the new CDH version, the secure_mount API changes.
Further, the new CDH version no longer uses the luks-encrypt-storage
script but utilizes libcryptsetup as well as mkfs.ext4 and dd. Hence, adapt
some of the CDH and Kata components build steps
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
* Introduces the `emptydir_mode` config flag to allow instructing the runtime
to create a block device for emptyDir volumes.
* The block device is created in the original emptyDir folder on the host
so that Kubelet can monitors its disk usage and evict the pod if it exceeds
its sizeLimit. This matches runc and virtio-fs.
* The block device's disk image file is sparse to minimize host disk
footprint.
Fixes: #10560
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Virtio-mmio transport is not hardened for confidential computing (unlike
virtio-pci). Reject config that would use virtio-blk-mmio for rootfs/block
when confidential_guest is set, so CoCo guests only use virtio-blk-pci.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
This commit introduces three new fields to the `SecurityInfo` struct
to support SEV-SNP (Secure Nested Paging) attestation and measurement
capabilities:
(1) `snp_id_block`: A 96-byte Base64-encoded ID block for the
SNP_LAUNCH_FINISH command.
(2) `snp_id_auth`: A 4096-byte Base64-encoded authentication structure
accompanying the ID block.
(3) `snp_guest_policy`: A bitmask for the SNP guest policy, passed to
the SNP_LAUNCH_START command.
These fields enable users to provide identity information to the SNP
firmware, allowing for remote attestation and verified guest launches.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Introduce host_memory_mib() with OS-specific implementations
(Linux/Android via nix::sysinfo,
macOS via sysctl) selected at compile time. This improves
portability and allows consistent host memory sizing/validation
across different platforms.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
As the memory related information has been serialized at the sandbox
initalization specially at the moment of parsing configuration toml.
This commit aims to refactor MemoryInfo initialization logics:
(1) Remove memory sizing/host-memory adjustment logic from QEMU cmdline
Memory::new()
(2) Initialize/adjust memory values via kata-types MemoryInfo (single
source of truth)
(3) Replace sysinfo::System::new_with_specifics with
nix::sys::sysinfo::sysinfo() to get host RAM
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
- Trim trailing whitespace and ensure final newline in non-vendor files
- Add .editorconfig-checker.json excluding vendor dirs, *.patch, *.img,
*.dtb, *.drawio, *.svg, and pkg/cloud-hypervisor/client so CI only
checks project code
- Leave generated and binary assets unchanged (excluded from checker)
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The runtime-rs shim was failing to load its configuration when deployed
via kata-deploy because it couldn't correctly parse the ConfigPath passed
by containerd. The previous implementation naively skipped the first 2
bytes of the options and interpreted the rest as a UTF-8 string, which
doesn't work since containerd passes a properly serialized protobuf
message of type runtimeoptions.v1.Options.
This change adds the runtimeoptions.proto definition to the protocols
crate and updates the load_config function to correctly deserialize the
protobuf message and extract the config_path field, matching how the Go
runtime handles this via typeurl.UnmarshalAny.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
This commit introduces a new annotation for users to easily set network
queues via "io.katacontainers.config.hypervisor.network_queues".
And the annotation will be mapped into `NetworkInfo.network_queues`
within the configuration.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Remove the initramfs folder, its build steps, and use the kernel
based dm-verity enforcement for the handlers which used the
initramfs mode. Also, remove the initramfs verity mode
capability from the shims and their configs.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
Similar to the kernel_params annotation, add a
kernel_verity_params annotation and add logic to make these
parameters overwritable. For instance, this can be used in test
logic to provide bogus dm-verity hashes for negative tests.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
This change introduces the kernel_verity_parameters knob to the
rust based shim, picking up dm-verity information in a new config
field (the corresponding build variable is already produced by
the shim build). The change extends the shim to parse dm-verity
information from this parameter and to construct the kernel command
line appropriately, based on the indicated initramfs or kernelinit
build variant.
Signed-off-by: Manuel Huber <manuelh@nvidia.com>
HashMap cannot guarantee the order. The command line is always changed.
This commit change kv of get_agent_kernel_params to BTreeMap to make
sure the command line is not changed.
Fixes: #10977
Signed-off-by: Hui Zhu <teawater@antgroup.com>
Clippy is recommending that format args are inlined for
better clarity, so update our code to remove these warnings
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
There are test cases require interaction with KVM device, introduce
skip_if_kvm_unaccessable macro to skip them.
Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
We can use the new Error::other options rather than
Error:new(Error:Kind:Other and drop our own macro that did this mapping
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Fix the warning throw up:
```
warning: hiding a lifetime that's elided elsewhere is confusing
--> /root/go/src/github.com/kata-containers/kata-containers/src/libs/kata-types/src/utils/u32_set.rs:50:17
|
50 | pub fn iter(&self) -> Iter<u32> {
| ^^^^^ --------- the same lifetime is hidden here
| |
| the lifetime is elided here
|
= help: the same lifetime is referred to in inconsistent ways, making the signature confusing
= note: `#[warn(mismatched_lifetime_syntaxes)]` on by default
help: use `'_` for type paths
|
50 | pub fn iter(&self) -> Iter<'_, u32> {
| +++
```
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This commit introduces the capability to dynamically configure
`queue_size` and `num_queues` parameters via Pod annotations.
Currently, `kata-runtime` allows for static configuration of
`queue_size` and `num_queues` for block devices through its config
file. However, a critical issue arises when a Pod is allocated fewer
CPU cores than the statically configured `num_queues` value. In such
scenarios, the Pod fails to start, leading to operational instability
and limiting flexibility in resource allocation.
To address this, this feature enables users to override the default
queue_size and num_queues parameters by specifying them in Pod
annotations.This allows for fine-grained control and dynamic adjustment
of these parameters based on the specific resource allocation of a Pod.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
Refactors `LinuxContainerCpuResources` and `LinuxSandboxCpuResources`
to track calculated vCPU allocation using `f64` (fractional float)
instead of `u64` (milliseconds).
This ensures more precise resource calculation (`quota / period`) and
aggregation by avoiding rounding errors inherent in millisecond-based
integer tracking.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
In order to have a better way to set things up using a toml editor, we
should take the containerd approach and actually have everything
uncommnted. This will help us to unify how we deal with such values in
the future from the kata-deploy POV.
Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
