Compare commits

..

19 Commits

Author SHA1 Message Date
Fabiano Fidêncio
734e75b7a4 ci: Run runtime-rs tests for SNP
As we're in the process to stabilise runtime-rs for the coming 4.0.0
release, we better start running as many tests as possible with that.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-04-07 13:26:05 +02:00
RuoqingHe
feaec78ad0 Merge pull request #12776 from fidencio/topic/kata-deploy-move-into-the-root-workspace
kata-deploy: Move into the root workspace
2026-04-07 12:45:26 +08:00
Fabiano Fidêncio
461907918d kata-deploy: pin nydus-snapshotter via versions.yaml
Resolve externals.nydus-snapshotter version and url in the Docker image build
with yq from the repo-root versions.yaml instead of Dockerfile ARG defaults.

Drop the redundant workflow that only enforced parity between those two sources.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-04-07 10:07:06 +08:00
Fabiano Fidêncio
9e1f595160 kata-deploy: add Rust binary to root workspace
Add tools/packaging/kata-deploy/binary as a workspace member, inherit shared
dependency versions from the root manifest, and refresh Cargo.lock.

Build the kata-deploy image from the repository root: copy the workspace
layout into the rust-builder stage, run cargo test/build with -p kata-deploy,
and adjust artifact and static asset COPY paths. Update the payload build
script to invoke docker buildx with -f .../Dockerfile from the repo root.

Add a repo-root .dockerignore to keep the Docker build context smaller.
Document running unit tests with cargo test -p kata-deploy from the root.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-04-07 10:07:06 +08:00
Alex Lyn
46a7b9e75d Merge pull request #12775 from RuoqingHe/put-libs-to-root-workspace
libs: Move libs into root workspace
2026-04-07 09:25:26 +08:00
Dan Mihai
9b770793ba Merge pull request #12728 from manuelh-dev/mahuber/empty-dir-fsgrou-policy
genpolicy: adjust GID after passwd GID handling and set fs_group for encrypted emptyDir volumes
2026-04-06 10:22:34 -07:00
Fabiano Fidêncio
47770daa3b helm: Align values.yaml with try-kata-nvidia-gpu.values.yaml
We've switched to nydus there, but never did for the values.yaml.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-04-06 18:51:54 +02:00
Fabiano Fidêncio
1300145f7a tests: add k3s/rke2 to OCI 1.3.0 drop-in overlay condition
k3s and rke2 ship containerd 2.2.2, which requires the OCI 1.3.0
drop-in overlay. Move them from the separate OCI 1.2.1 branch into
the OCI 1.3.0 condition alongside nvidia-gpu, qemu-snp, qemu-tdx,
and custom container engine versions.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-04-06 18:50:20 +02:00
Fabiano Fidêncio
0a739b3b55 Merge pull request #12755 from katexochen/runtime-rs-config-cleanup
runtime-rs: cleanup config
2026-04-06 13:14:58 +02:00
Ruoqing He
cb7c790dc7 libs: Specify crates explicitly in Makefile
--all option would trigger building and testing for everything within
our root workspace, which is not desired here. Let's specify the crates
of libs explicitly in our Makefile.

Signed-off-by: Ruoqing He <ruoqing.he@lingcage.com>
2026-04-06 11:03:38 +02:00
Ruoqing He
2a024f55d0 libs: Move libs into root workspace
Remove libs from exclude list, and move them explicitly into root
workspace to make sure our core components are in a consistent state.

This is a follow up of #12413.

Signed-off-by: Ruoqing He <ruoqing.he@lingcage.com>
2026-04-06 11:03:38 +02:00
Fabiano Fidêncio
9a2825a429 runtime: config: Use OVMF for the qemu-nvidia-gpu
2ba0cb0d4a7 did the ground work for using OVMF even for the
qemu-nvidia-gpu, but missed actually setting the OVMF path to be used,
which we'e fixing now.

Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com>
2026-04-06 03:54:56 +02:00
Fabiano Fidêncio
e1fae11509 Merge pull request #12392 from Apokleos/enhance-tdx
runtime-rs: Enhance TDX in qemu
2026-04-05 20:54:43 +02:00
Alex Lyn
35cafe8715 runtime-rs: configure TDX machine options with kernel_irqchip=split
When TDX confidential guest support is enabled, set `kernel_irqchip=split`
for TDX CVM:
...
-machine \
   q35,accel=kvm,kernel_irqchip=split,confidential-guest-support=tdx \
...

Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
2026-04-05 10:18:47 +02:00
Manuel Huber
57e42b10f1 tests: nvidia: Do not use elevated privileges
Do not run the NIM containers with elevated privileges. Note that,
using hostPath requires proper host folder permissions, and that
using emptyDir requires a proper fsGroup ID.
Once issue 11162 is resolved, we can further refine the securityContext
fields for the TEE manifests.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
2026-04-01 10:23:26 -07:00
Manuel Huber
a762b136de tests: generate policy for pod-empty-dir-fsgroup
The logic in the k8s-empty-dirs.bats file missed to add a security
policy for the pod-empty-dir-fsgroup.yaml manifest. With this change,
we add the policy annotation.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
2026-04-01 10:23:26 -07:00
Manuel Huber
43489f6d56 genpolicy: fs_group for encrypted emptyDir volumes
The shim uses Storage.fs_group on block/scsi encrypted emptyDir while
genpolicy used fsgid= in options and null fs_group, leading to
denying CreateContainerRequest when using block-encrypted emptyDir in
combination with fsGroup. Thus, emit fs_group in that scenario and keep
fsgid= for the existing shared-fs/local emptyDir behavior.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
2026-04-01 10:23:26 -07:00
Manuel Huber
9923f251f5 genpolicy: adjust GID after passwd GID handling
After pod runAsUser triggers passwd-based GID resolution, genpolicy
clears AdditionalGids and inserts only the primary GID.
PodSecurityContext fsGroup and supplementalGroups get cleared, so
policy enforcement would deny CreateContainer when the runtime
includes those when specified.

This change applies fsGroup/supplementalGroups once in
get_container_process via apply_pod_fs_group_and_supplemental_groups.

Signed-off-by: Manuel Huber <manuelh@nvidia.com>
2026-04-01 10:23:25 -07:00
Paul Meyer
086c0ed18e runtime-rs: cleanup config
Some fields were misspelled, misplaced at an outdated path or copied
over from runtime-go but aren't supported in runtime-rs.
Cleaning them up to avoid confusion and ease migration.

Signed-off-by: Paul Meyer <katexochen0@gmail.com>
2026-03-31 10:51:25 +02:00
56 changed files with 832 additions and 794 deletions

13
.dockerignore Normal file
View File

@@ -0,0 +1,13 @@
# Context for tools/packaging/kata-deploy/Dockerfile (build from repo root: -f tools/packaging/kata-deploy/Dockerfile .)
#
# The Dockerfile only needs: Cargo.toml, Cargo.lock, src/, tools/packaging/kata-deploy/,
# and versions.yaml. Exclude heavy or irrelevant trees to keep context small.
.git
.github
target
kata-artifacts
docs
tests
utils
tools/packaging/kata-deploy/local-build
tools/packaging/kata-deploy/binary/target

View File

@@ -279,7 +279,6 @@ jobs:
matrix:
vmm:
- qemu
- qemu-runtime-rs
runs-on: ubuntu-22.04
env:
KATA_HYPERVISOR: ${{ matrix.vmm }}

View File

@@ -132,9 +132,7 @@ jobs:
# all due to a single flaky instance.
fail-fast: false
matrix:
vmm:
- qemu
- qemu-runtime-rs
vmm: ['qemu']
runs-on: s390x-large
env:
KATA_HYPERVISOR: ${{ matrix.vmm }}

View File

@@ -1,35 +0,0 @@
name: nydus-snapshotter-version-sync
on:
pull_request:
types:
- opened
- edited
- reopened
- synchronize
permissions: {}
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
nydus-snapshotter-version-check:
name: nydus-snapshotter-version-check
runs-on: ubuntu-22.04
steps:
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Ensure nydus-snapshotter-version is in sync inside our repo
run: |
dockerfile_version=$(grep "ARG NYDUS_SNAPSHOTTER_VERSION" tools/packaging/kata-deploy/Dockerfile | cut -f2 -d'=')
versions_version=$(yq ".externals.nydus-snapshotter.version | explode(.)" versions.yaml)
if [[ "${dockerfile_version}" != "${versions_version}" ]]; then
echo "nydus-snapshotter version must be the same in the following places: "
echo "- versions.yaml: ${versions_version}"
echo "- tools/packaging/kata-deploy/Dockerfile: ${dockerfile_version}"
exit 1
fi

View File

@@ -53,6 +53,8 @@ jobs:
vmm: qemu-tdx
- runner: sev-snp
vmm: qemu-snp
- runner: sev-snp
vmm: qemu-snp-runtime-rs
runs-on: ${{ matrix.runner }}
env:
DOCKER_REGISTRY: ${{ inputs.registry }}

479
Cargo.lock generated
View File

@@ -94,6 +94,12 @@ dependencies = [
"memchr",
]
[[package]]
name = "allocator-api2"
version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
[[package]]
name = "android_system_properties"
version = "0.1.5"
@@ -409,6 +415,28 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "async-stream"
version = "0.3.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476"
dependencies = [
"async-stream-impl",
"futures-core",
"pin-project-lite",
]
[[package]]
name = "async-stream-impl"
version = "0.3.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "async-task"
version = "4.7.1"
@@ -511,6 +539,17 @@ dependencies = [
"tower-service",
]
[[package]]
name = "backon"
version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef"
dependencies = [
"fastrand 2.3.0",
"gloo-timers",
"tokio",
]
[[package]]
name = "backtrace"
version = "0.3.76"
@@ -1272,6 +1311,16 @@ dependencies = [
"darling_macro 0.20.11",
]
[[package]]
name = "darling"
version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0"
dependencies = [
"darling_core 0.21.3",
"darling_macro 0.21.3",
]
[[package]]
name = "darling_core"
version = "0.14.4"
@@ -1299,6 +1348,20 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "darling_core"
version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1247195ecd7e3c85f83c8d2a366e4210d588e802133e1e355180a9870b517ea4"
dependencies = [
"fnv",
"ident_case",
"proc-macro2",
"quote",
"strsim",
"syn 2.0.117",
]
[[package]]
name = "darling_macro"
version = "0.14.4"
@@ -1321,6 +1384,17 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "darling_macro"
version = "0.21.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81"
dependencies = [
"darling_core 0.21.3",
"quote",
"syn 2.0.117",
]
[[package]]
name = "dashmap"
version = "5.5.3"
@@ -1600,6 +1674,27 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "derive_more"
version = "2.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134"
dependencies = [
"derive_more-impl",
]
[[package]]
name = "derive_more-impl"
version = "2.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb"
dependencies = [
"proc-macro2",
"quote",
"rustc_version",
"syn 2.0.117",
]
[[package]]
name = "device_tree"
version = "1.1.0"
@@ -1726,6 +1821,18 @@ version = "1.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "educe"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d7bc049e1bd8cdeb31b68bbd586a9464ecf9f3944af3958a7a9d0f8b9799417"
dependencies = [
"enum-ordinalize",
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "either"
version = "1.15.0"
@@ -1774,6 +1881,26 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "enum-ordinalize"
version = "4.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a1091a7bb1f8f2c4b28f1fe2cef4980ca2d410a3d727d67ecc3178c9b0800f0"
dependencies = [
"enum-ordinalize-derive",
]
[[package]]
name = "enum-ordinalize-derive"
version = "4.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "enumflags2"
version = "0.7.12"
@@ -2338,6 +2465,18 @@ version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280"
[[package]]
name = "gloo-timers"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbb143cf96099802033e0d4f4963b19fd2e0b728bcf076cd9cf7f6634f092994"
dependencies = [
"futures-channel",
"futures-core",
"js-sys",
"wasm-bindgen",
]
[[package]]
name = "go-flag"
version = "0.1.0"
@@ -2406,6 +2545,8 @@ version = "0.15.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"
dependencies = [
"allocator-api2",
"equivalent",
"foldhash",
]
@@ -2506,6 +2647,17 @@ dependencies = [
"windows-sys 0.52.0",
]
[[package]]
name = "hostname"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "617aaa3557aef3810a6369d0a99fac8a080891b68bd9f9812a1eeda0c0730cbd"
dependencies = [
"cfg-if 1.0.4",
"libc",
"windows-link",
]
[[package]]
name = "http"
version = "0.2.12"
@@ -2643,7 +2795,9 @@ dependencies = [
"http 1.4.0",
"hyper 1.8.1",
"hyper-util",
"log",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"tokio",
"tokio-rustls",
@@ -2662,6 +2816,19 @@ dependencies = [
"tokio-io-timeout",
]
[[package]]
name = "hyper-timeout"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0"
dependencies = [
"hyper 1.8.1",
"hyper-util",
"pin-project-lite",
"tokio",
"tower-service",
]
[[package]]
name = "hyper-tls"
version = "0.6.0"
@@ -3127,6 +3294,19 @@ dependencies = [
"thiserror 1.0.69",
]
[[package]]
name = "jsonpath-rust"
version = "0.7.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c00ae348f9f8fd2d09f82a98ca381c60df9e0820d8d79fce43e649b4dc3128b"
dependencies = [
"pest",
"pest_derive",
"regex",
"serde_json",
"thiserror 2.0.18",
]
[[package]]
name = "jsonptr"
version = "0.4.7"
@@ -3201,6 +3381,18 @@ dependencies = [
"tonic-build 0.8.4",
]
[[package]]
name = "k8s-openapi"
version = "0.26.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06d9e5e61dd037cdc51da0d7e2b2be10f497478ea7e120d85dad632adb99882b"
dependencies = [
"base64 0.22.1",
"chrono",
"serde",
"serde_json",
]
[[package]]
name = "kata-agent"
version = "0.1.0"
@@ -3285,6 +3477,28 @@ dependencies = [
"tokio",
]
[[package]]
name = "kata-deploy"
version = "0.1.0"
dependencies = [
"anyhow",
"clap",
"env_logger",
"k8s-openapi",
"kube",
"libc",
"log",
"regex",
"rstest",
"serde_json",
"serde_yaml 0.9.34+deprecated",
"serial_test 0.10.0",
"tempfile",
"tokio",
"toml_edit 0.22.27",
"walkdir",
]
[[package]]
name = "kata-sys-util"
version = "0.1.0"
@@ -3307,6 +3521,8 @@ dependencies = [
"slog",
"slog-scope",
"subprocess",
"tempfile",
"test-utils",
"thiserror 1.0.69",
]
@@ -3325,6 +3541,7 @@ dependencies = [
"num_cpus",
"oci-spec 0.8.4",
"regex",
"rstest",
"safe-path 0.1.0",
"serde",
"serde-enum-str",
@@ -3334,6 +3551,8 @@ dependencies = [
"slog-scope",
"sysctl",
"sysinfo",
"tempfile",
"test-utils",
"thiserror 1.0.69",
"toml",
]
@@ -3358,6 +3577,115 @@ dependencies = [
"libc",
]
[[package]]
name = "kube"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "48e7bb0b6a46502cc20e4575b6ff401af45cfea150b34ba272a3410b78aa014e"
dependencies = [
"k8s-openapi",
"kube-client",
"kube-core",
"kube-derive",
"kube-runtime",
]
[[package]]
name = "kube-client"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4987d57a184d2b5294fdad3d7fc7f278899469d21a4da39a8f6ca16426567a36"
dependencies = [
"base64 0.22.1",
"bytes 1.11.1",
"chrono",
"either",
"futures",
"home",
"http 1.4.0",
"http-body 1.0.1",
"http-body-util",
"hyper 1.8.1",
"hyper-rustls",
"hyper-timeout 0.5.2",
"hyper-util",
"jsonpath-rust",
"k8s-openapi",
"kube-core",
"pem",
"rustls",
"secrecy",
"serde",
"serde_json",
"serde_yaml 0.9.34+deprecated",
"thiserror 2.0.18",
"tokio",
"tokio-util",
"tower 0.5.3",
"tower-http",
"tracing",
]
[[package]]
name = "kube-core"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "914bbb770e7bb721a06e3538c0edd2babed46447d128f7c21caa68747060ee73"
dependencies = [
"chrono",
"derive_more",
"form_urlencoded",
"http 1.4.0",
"json-patch 4.1.0",
"k8s-openapi",
"schemars",
"serde",
"serde-value",
"serde_json",
"thiserror 2.0.18",
]
[[package]]
name = "kube-derive"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03dee8252be137772a6ab3508b81cd797dee62ee771112a2453bc85cbbe150d2"
dependencies = [
"darling 0.21.3",
"proc-macro2",
"quote",
"serde",
"serde_json",
"syn 2.0.117",
]
[[package]]
name = "kube-runtime"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6aea4de4b562c5cc89ab10300bb63474ae1fa57ff5a19275f2e26401a323e3fd"
dependencies = [
"ahash 0.8.12",
"async-broadcast 0.7.2",
"async-stream",
"backon",
"educe",
"futures",
"hashbrown 0.15.5",
"hostname",
"json-patch 4.1.0",
"k8s-openapi",
"kube-client",
"parking_lot",
"pin-project",
"serde",
"serde_json",
"thiserror 2.0.18",
"tokio",
"tokio-util",
"tracing",
]
[[package]]
name = "kvm-bindings"
version = "0.14.0"
@@ -3539,6 +3867,7 @@ dependencies = [
"slog-json",
"slog-scope",
"slog-term",
"tempfile",
]
[[package]]
@@ -3584,11 +3913,16 @@ version = "0.2.0"
dependencies = [
"anyhow",
"chrono",
"lazy_static",
"maplit",
"nix 0.30.1",
"once_cell",
"page_size",
"slog",
"slog-async",
"slog-scope",
"slog-term",
"test-utils",
"tokio",
]
@@ -4493,6 +4827,15 @@ dependencies = [
"num-traits",
]
[[package]]
name = "ordered-float"
version = "2.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c"
dependencies = [
"num-traits",
]
[[package]]
name = "ordered-multimap"
version = "0.4.3"
@@ -4602,6 +4945,16 @@ dependencies = [
"quote",
]
[[package]]
name = "pem"
version = "3.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be"
dependencies = [
"base64 0.22.1",
"serde_core",
]
[[package]]
name = "percent-encoding"
version = "2.3.2"
@@ -4621,6 +4974,49 @@ dependencies = [
"serde_json",
]
[[package]]
name = "pest"
version = "2.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0848c601009d37dfa3430c4666e147e49cdcf1b92ecd3e63657d8a5f19da662"
dependencies = [
"memchr",
"ucd-trie",
]
[[package]]
name = "pest_derive"
version = "2.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "11f486f1ea21e6c10ed15d5a7c77165d0ee443402f0780849d1768e7d9d6fe77"
dependencies = [
"pest",
"pest_generator",
]
[[package]]
name = "pest_generator"
version = "2.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8040c4647b13b210a963c1ed407c1ff4fdfa01c31d6d2a098218702e6664f94f"
dependencies = [
"pest",
"pest_meta",
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "pest_meta"
version = "2.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "89815c69d36021a140146f26659a81d6c2afa33d216d736dd4be5381a7362220"
dependencies = [
"pest",
"sha2 0.10.9",
]
[[package]]
name = "petgraph"
version = "0.5.1"
@@ -5824,7 +6220,6 @@ dependencies = [
"protobuf",
"protocols",
"resource",
"rstest",
"runtime-spec",
"serde_json",
"shim-interface",
@@ -5975,7 +6370,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4"
dependencies = [
"aws-lc-rs",
"log",
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki",
"subtle",
@@ -6074,6 +6471,7 @@ name = "safe-path"
version = "0.1.0"
dependencies = [
"libc",
"tempfile",
]
[[package]]
@@ -6120,10 +6518,23 @@ checksum = "a2b42f36aa1cd011945615b92222f6bf73c599a102a300334cd7f8dbeec726cc"
dependencies = [
"dyn-clone",
"ref-cast",
"schemars_derive",
"serde",
"serde_json",
]
[[package]]
name = "schemars_derive"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d115b50f4aaeea07e79c1912f645c7513d81715d0420f8bc77a18c6260b307f"
dependencies = [
"proc-macro2",
"quote",
"serde_derive_internals",
"syn 2.0.117",
]
[[package]]
name = "scientific"
version = "0.5.3"
@@ -6165,6 +6576,15 @@ dependencies = [
"libc",
]
[[package]]
name = "secrecy"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e891af845473308773346dc847b2c23ee78fe442e0472ac50e22a18a93d3ae5a"
dependencies = [
"zeroize",
]
[[package]]
name = "security-framework"
version = "3.7.0"
@@ -6244,6 +6664,16 @@ version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8a059d895f1a31dd928f40abbea4e7177e3d8ff3aa4152fdb7a396ae1ef63a3"
[[package]]
name = "serde-value"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3a1a3341211875ef120e117ea7fd5228530ae7e7036a779fdc9117be6b3282c"
dependencies = [
"ordered-float 2.10.1",
"serde",
]
[[package]]
name = "serde_core"
version = "1.0.228"
@@ -6264,6 +6694,17 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "serde_derive_internals"
version = "0.29.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "serde_ignored"
version = "0.1.14"
@@ -6497,6 +6938,8 @@ dependencies = [
"kata-sys-util",
"kata-types",
"nix 0.26.4",
"tempfile",
"test-utils",
"tokio",
]
@@ -7016,7 +7459,7 @@ dependencies = [
"byteorder",
"integer-encoding",
"log",
"ordered-float",
"ordered-float 1.1.1",
"threadpool",
]
@@ -7164,6 +7607,7 @@ dependencies = [
"futures-core",
"futures-sink",
"pin-project-lite",
"slab",
"tokio",
]
@@ -7228,6 +7672,18 @@ dependencies = [
"winnow 0.5.40",
]
[[package]]
name = "toml_edit"
version = "0.22.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [
"indexmap 2.13.0",
"toml_datetime 0.6.11",
"toml_write",
"winnow 0.7.15",
]
[[package]]
name = "toml_edit"
version = "0.25.4+spec-1.1.0"
@@ -7249,6 +7705,12 @@ dependencies = [
"winnow 0.7.15",
]
[[package]]
name = "toml_write"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
[[package]]
name = "tonic"
version = "0.9.2"
@@ -7265,7 +7727,7 @@ dependencies = [
"http 0.2.12",
"http-body 0.4.6",
"hyper 0.14.32",
"hyper-timeout",
"hyper-timeout 0.4.1",
"percent-encoding",
"pin-project",
"prost 0.11.9",
@@ -7334,8 +7796,10 @@ dependencies = [
"pin-project-lite",
"sync_wrapper 1.0.2",
"tokio",
"tokio-util",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
@@ -7344,16 +7808,19 @@ version = "0.6.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8"
dependencies = [
"base64 0.22.1",
"bitflags 2.11.0",
"bytes 1.11.1",
"futures-util",
"http 1.4.0",
"http-body 1.0.1",
"iri-string",
"mime",
"pin-project-lite",
"tower 0.5.3",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
@@ -7526,6 +7993,12 @@ version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb"
[[package]]
name = "ucd-trie"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971"
[[package]]
name = "uds_windows"
version = "1.2.0"

View File

@@ -6,6 +6,17 @@ rust-version = "1.88"
[workspace]
members = [
# libs
"src/libs/kata-sys-util",
"src/libs/kata-types",
"src/libs/logging",
"src/libs/mem-agent",
"src/libs/protocols",
"src/libs/runtime-spec",
"src/libs/safe-path",
"src/libs/shim-interface",
"src/libs/test-utils",
# kata-agent
"src/agent",
"src/agent/rustjail",
@@ -31,6 +42,9 @@ members = [
# genpolicy
"src/tools/genpolicy",
# kata-deploy (Kubernetes installer binary)
"tools/packaging/kata-deploy/binary",
# runtime-rs
"src/runtime-rs",
"src/runtime-rs/crates/agent",
@@ -48,10 +62,6 @@ resolver = "2"
# TODO: Add all excluded crates to root workspace
exclude = [
"src/tools",
"src/libs",
# kata-deploy binary is standalone and has its own Cargo.toml for now
"tools/packaging/kata-deploy/binary",
# We are cloning and building rust packages under
# "tools/packaging/kata-deploy/local-build/build" folder, which may mislead

View File

@@ -1,13 +0,0 @@
[workspace]
members = [
"kata-sys-util",
"kata-types",
"logging",
"mem-agent",
"protocols",
"runtime-spec",
"safe-path",
"shim-interface",
"test-utils",
]
resolver = "2"

View File

@@ -11,6 +11,17 @@ ifeq ($(USERID), 0)
override EXTRA_TEST_FLAGS = --ignored
endif
LIBS := \
-p kata-sys-util \
-p kata-types \
-p logging \
-p mem-agent \
-p protocols \
-p runtime-spec \
-p safe-path \
-p shim-interface \
-p test-utils
default: build
build:
@@ -23,13 +34,13 @@ check: clippy format
clippy:
@echo "INFO: cargo clippy..."
cargo clippy --all-targets --all-features --release \
cargo clippy $(LIBS) --all-features --release \
-- \
-D warnings
format:
@echo "INFO: cargo fmt..."
cargo fmt -- --check
cargo fmt $(LIBS) -- --check
clean:
cargo clean
@@ -38,8 +49,8 @@ clean:
# See the `test_logger_levels()` test for further information.
test:
@echo "INFO: testing libraries for development build"
cargo test --all $(EXTRA_RUSTFEATURES) -- --nocapture $(EXTRA_TEST_FLAGS)
cargo test $(LIBS) $(EXTRA_RUSTFEATURES) -- --nocapture $(EXTRA_TEST_FLAGS)
@echo "INFO: testing libraries for release build"
cargo test --release --all $(EXTRA_RUSTFEATURES) -- --nocapture $(EXTRA_TEST_FLAGS)
cargo test --release $(LIBS) $(EXTRA_RUSTFEATURES) -- --nocapture $(EXTRA_TEST_FLAGS)
.PHONY: install vendor

View File

@@ -137,16 +137,12 @@ ifeq ($(ARCH), aarch64)
EDK2_NAME := aavmf
endif
# Set firmware paths from QEMUFW/QEMUFWVOL if defined
# Set firmware path from QEMUFW if defined
FIRMWAREPATH :=
FIRMWAREVOLUMEPATH :=
ifneq (,$(QEMUCMD))
ifneq (,$(QEMUFW))
FIRMWAREPATH := $(PREFIXDEPS)/share/$(EDK2_NAME)/$(QEMUFW)
endif
ifneq (,$(QEMUFWVOL))
FIRMWAREVOLUMEPATH := $(PREFIXDEPS)/share/$(EDK2_NAME)/$(QEMUFWVOL)
endif
endif
KERNELVERITYPARAMS ?= ""
@@ -157,7 +153,6 @@ FIRMWARETDVFPATH := $(PREFIXDEPS)/share/ovmf/OVMF.inteltdx.fd
# SEV-SNP
FIRMWARE_SNP_PATH := $(PREFIXDEPS)/share/ovmf/AMDSEV.fd
FIRMWARE_VOLUME_SNP_PATH :=
##VAR DEFVCPUS=<number> Default number of vCPUs
DEFVCPUS := 1
@@ -204,7 +199,6 @@ DEFVIRTIOFSQUEUESIZE ?= 1024
# Make sure you quote args.
DEFVIRTIOFSEXTRAARGS ?= [\"--thread-pool-size=1\", \"-o\", \"announce_submounts\"]
DEFENABLEIOTHREADS := false
DEFINDEPIOTHREADS := 0
DEFENABLEVHOSTUSERSTORE := false
DEFVHOSTUSERSTOREPATH := $(PKGRUNDIR)/vhost-user
DEFVALIDVHOSTUSERSTOREPATHS := [\"$(DEFVHOSTUSERSTOREPATH)\"]
@@ -222,7 +216,6 @@ DEFCREATECONTAINERTIMEOUT ?= 30
DEFCREATECONTAINERTIMEOUT_COCO ?= 60
DEFSTATICRESOURCEMGMT_COCO = true
DEFDISABLEIMAGENVDIMM ?= false
DEFPODRESOURCEAPISOCK := ""
SED = sed
CLI_DIR = cmd
@@ -412,9 +405,6 @@ endif
# Most users will want to set this to "on,obsolete=deny,spawn=deny,resourcecontrol=deny"
# for better security. Note: "elevateprivileges=deny" doesn't work with daemonize option.
DEFSECCOMPSANDBOXPARAM := ""
# Default is empty string "" to match Rust default None (when commented out in config).
# Most users will want to set this to "system_u:system_r:container_t" for SELinux support.
DEFGUESTSELINUXLABEL := ""
endif
ifneq (,$(FCCMD))
@@ -524,7 +514,6 @@ USER_VARS += KERNELPATH_COCO
USER_VARS += KERNELPATH
USER_VARS += KERNELVIRTIOFSPATH
USER_VARS += FIRMWAREPATH
USER_VARS += FIRMWAREVOLUMEPATH
USER_VARS += MACHINEACCELERATORS
USER_VARS += CPUFEATURES
USER_VARS += DEFMACHINETYPE_CLH
@@ -584,9 +573,7 @@ USER_VARS += DEFVIRTIOFSEXTRAARGS
USER_VARS += DEFENABLEANNOTATIONS
USER_VARS += DEFENABLEANNOTATIONS_COCO
USER_VARS += DEFENABLEIOTHREADS
USER_VARS += DEFINDEPIOTHREADS
USER_VARS += DEFSECCOMPSANDBOXPARAM
USER_VARS += DEFGUESTSELINUXLABEL
USER_VARS += DEFENABLEVHOSTUSERSTORE
USER_VARS += DEFVHOSTUSERSTOREPATH
USER_VARS += DEFVALIDVHOSTUSERSTOREPATHS
@@ -628,11 +615,9 @@ USER_VARS += DEFCREATECONTAINERTIMEOUT
USER_VARS += DEFCREATECONTAINERTIMEOUT_COCO
USER_VARS += QEMUTDXEXPERIMENTALCMD
USER_VARS += FIRMWARE_SNP_PATH
USER_VARS += FIRMWARE_VOLUME_SNP_PATH
USER_VARS += KERNELTDXPARAMS
USER_VARS += DEFSHAREDFS_QEMU_TDX_VIRTIOFS
USER_VARS += FIRMWARETDVFPATH
USER_VARS += DEFPODRESOURCEAPISOCK
SOURCES := \
$(shell find . 2>&1 | grep -E '.*\.rs$$') \

View File

@@ -13,7 +13,6 @@ CPUFEATURES := pmu=off
QEMUCMD := qemu-system-aarch64
QEMUFW := AAVMF_CODE.fd
QEMUFWVOL := AAVMF_VARS.fd
# dragonball binary name
DBCMD := dragonball

View File

@@ -76,12 +76,6 @@ kernel_params = "@KERNELPARAMS@"
# If you want that qemu uses the default firmware leave this option empty
firmware = "@FIRMWAREPATH@"
# Path to the firmware volume.
# firmware TDVF or OVMF can be split into FIRMWARE_VARS.fd (UEFI variables
# as configuration) and FIRMWARE_CODE.fd (UEFI program image). UEFI variables
# can be customized per each user while UEFI code is kept same.
firmware_volume = "@FIRMWAREVOLUMEPATH@"
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
@@ -89,12 +83,12 @@ machine_accelerators = "@MACHINEACCELERATORS@"
# Qemu seccomp sandbox feature
# comma-separated list of seccomp sandbox features to control the syscall access.
# For example, `seccompsandbox= "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# For example, `seccomp_sandbox = "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# Note: "elevateprivileges=deny" doesn't work with daemonize option, so it's removed from the seccomp sandbox
# Another note: enabling this feature may reduce performance, you may enable
# /proc/sys/net/core/bpf_jit_enable to reduce the impact. see https://man7.org/linux/man-pages/man8/bpfc.8.html
# Recommended value when enabling: "on,obsolete=deny,spawn=deny,resourcecontrol=deny"
seccompsandbox = "@DEFSECCOMPSANDBOXPARAM@"
seccomp_sandbox = "@DEFSECCOMPSANDBOXPARAM@"
# CPU features
# comma-separated list of cpu features to pass to the cpu
@@ -311,11 +305,6 @@ enable_iommu_platform = false
# Your distribution recommends: @DEFVALIDVHOSTUSERSTOREPATHS@
valid_vhost_user_store_paths = @DEFVALIDVHOSTUSERSTOREPATHS@
# The timeout for reconnecting on non-server spdk sockets when the remote end goes away.
# qemu will delay this many seconds and then attempt to reconnect.
# Zero disables reconnecting, and the default is zero.
vhost_user_reconnect_timeout_sec = 0
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
@@ -350,7 +339,7 @@ enable_debug = false
#
# If set to the empty string "", no extra monitor socket is added. This is
# the default.
extra_monitor_socket = ""
dbg_monitor_socket = ""
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
@@ -378,18 +367,6 @@ disable_image_nvdimm = false
# Default false
hotplug_vfio_on_root_bus = false
# Enable hot-plugging of VFIO devices to a bridge-port,
# root-port or switch-port.
# The default setting is "no-port"
hot_plug_vfio = "no-port"
# In a confidential compute environment hot-plugging can compromise
# security.
# Enable cold-plugging of VFIO devices to a bridge-port,
# root-port or switch-port.
# The default setting is "no-port", which means disabled.
cold_plug_vfio = "no-port"
# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
@@ -483,9 +460,6 @@ guest_memory_dump_path = ""
# See: https://www.qemu.org/docs/master/qemu-qmp-ref.html#Dump-guest-memory for details
#guest_memory_dump_paging=false
# use legacy serial for guest console if available and implemented for architecture. Default false
use_legacy_serial = false
# disable applying SELinux on the VMM process (default false)
disable_selinux = @DEFDISABLESELINUX@
@@ -497,7 +471,7 @@ disable_selinux = @DEFDISABLESELINUX@
disable_guest_selinux = @DEFDISABLEGUESTSELINUX@
[factory]
[hypervisor.qemu.factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
@@ -726,20 +700,6 @@ agent_name = "@PROJECT_TYPE@"
# (default: true)
disable_guest_seccomp = @DEFDISABLEGUESTSECCOMP@
# vCPUs pinning settings
# if enabled, each vCPU thread will be scheduled to a fixed CPU
# qualified condition: num(vCPU threads) == num(CPUs in sandbox's CPUSet)
enable_vcpus_pinning = false
# Apply a custom SELinux security policy to the container process inside the VM.
# This is used when you want to apply a type other than the default `container_t`,
# so general users should not uncomment and apply it.
# (format: "user:role:type")
# Note: You cannot specify MCS policy with the label because the sensitivity levels and
# categories are determined automatically by high-level container runtimes such as containerd.
# Example value when enabling: "system_u:system_r:container_t"
guest_selinux_label = "@DEFGUESTSELINUXLABEL@"
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)

View File

@@ -60,12 +60,6 @@ kernel_params = "@KERNELPARAMS@"
# If you want that qemu uses the default firmware leave this option empty
firmware = "@FIRMWAREPATH@"
# Path to the firmware volume.
# firmware TDVF or OVMF can be split into FIRMWARE_VARS.fd (UEFI variables
# as configuration) and FIRMWARE_CODE.fd (UEFI program image). UEFI variables
# can be customized per each user while UEFI code is kept same.
firmware_volume = "@FIRMWAREVOLUMEPATH@"
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
@@ -73,12 +67,12 @@ machine_accelerators = "@MACHINEACCELERATORS@"
# Qemu seccomp sandbox feature
# comma-separated list of seccomp sandbox features to control the syscall access.
# For example, `seccompsandbox= "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# For example, `seccomp_sandbox = "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# Note: "elevateprivileges=deny" doesn't work with daemonize option, so it's removed from the seccomp sandbox
# Another note: enabling this feature may reduce performance, you may enable
# /proc/sys/net/core/bpf_jit_enable to reduce the impact. see https://man7.org/linux/man-pages/man8/bpfc.8.html
# Recommended value when enabling: "on,obsolete=deny,spawn=deny,resourcecontrol=deny"
seccompsandbox = "@DEFSECCOMPSANDBOXPARAM@"
seccomp_sandbox = "@DEFSECCOMPSANDBOXPARAM@"
# CPU features
# comma-separated list of cpu features to pass to the cpu
@@ -307,11 +301,6 @@ enable_iommu_platform = false
# Your distribution recommends: @DEFVALIDVHOSTUSERSTOREPATHS@
valid_vhost_user_store_paths = @DEFVALIDVHOSTUSERSTOREPATHS@
# The timeout for reconnecting on non-server spdk sockets when the remote end goes away.
# qemu will delay this many seconds and then attempt to reconnect.
# Zero disables reconnecting, and the default is zero.
vhost_user_reconnect_timeout_sec = 0
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
@@ -346,7 +335,7 @@ enable_debug = false
#
# If set to the empty string "", no extra monitor socket is added. This is
# the default.
extra_monitor_socket = ""
dbg_monitor_socket = ""
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
@@ -373,18 +362,6 @@ disable_image_nvdimm = false
# Default false
hotplug_vfio_on_root_bus = false
# Enable hot-plugging of VFIO devices to a bridge-port,
# root-port or switch-port.
# The default setting is "no-port"
hot_plug_vfio = "no-port"
# In a confidential compute environment hot-plugging can compromise
# security.
# Enable cold-plugging of VFIO devices to a bridge-port,
# root-port or switch-port.
# The default setting is "no-port", which means disabled.
cold_plug_vfio = "no-port"
# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
@@ -489,9 +466,6 @@ guest_memory_dump_path = ""
# See: https://www.qemu.org/docs/master/qemu-qmp-ref.html#Dump-guest-memory for details
guest_memory_dump_paging = false
# use legacy serial for guest console if available and implemented for architecture. Default false
use_legacy_serial = false
# disable applying SELinux on the VMM process (default false)
disable_selinux = @DEFDISABLESELINUX@
@@ -720,20 +694,6 @@ agent_name = "@PROJECT_TYPE@"
# (default: true)
disable_guest_seccomp = @DEFDISABLEGUESTSECCOMP@
# vCPUs pinning settings
# if enabled, each vCPU thread will be scheduled to a fixed CPU
# qualified condition: num(vCPU threads) == num(CPUs in sandbox's CPUSet)
enable_vcpus_pinning = false
# Apply a custom SELinux security policy to the container process inside the VM.
# This is used when you want to apply a type other than the default `container_t`,
# so general users should not uncomment and apply it.
# (format: "user:role:type")
# Note: You cannot specify MCS policy with the label because the sensitivity levels and
# categories are determined automatically by high-level container runtimes such as containerd.
# Example value when enabling: "system_u:system_r:container_t"
guest_selinux_label = "@DEFGUESTSELINUXLABEL@"
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)

View File

@@ -69,12 +69,6 @@ kernel_params = "@KERNELPARAMS@"
# If you want that qemu uses the default firmware leave this option empty
firmware = "@FIRMWAREPATH@"
# Path to the firmware volume.
# firmware TDVF or OVMF can be split into FIRMWARE_VARS.fd (UEFI variables
# as configuration) and FIRMWARE_CODE.fd (UEFI program image). UEFI variables
# can be customized per each user while UEFI code is kept same.
firmware_volume = "@FIRMWAREVOLUMEPATH@"
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
@@ -82,12 +76,12 @@ machine_accelerators = "@MACHINEACCELERATORS@"
# Qemu seccomp sandbox feature
# comma-separated list of seccomp sandbox features to control the syscall access.
# For example, `seccompsandbox= "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# For example, `seccomp_sandbox = "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# Note: "elevateprivileges=deny" doesn't work with daemonize option, so it's removed from the seccomp sandbox
# Another note: enabling this feature may reduce performance, you may enable
# /proc/sys/net/core/bpf_jit_enable to reduce the impact. see https://man7.org/linux/man-pages/man8/bpfc.8.html
# Recommended value when enabling: "on,obsolete=deny,spawn=deny,resourcecontrol=deny"
seccompsandbox = "@DEFSECCOMPSANDBOXPARAM@"
seccomp_sandbox = "@DEFSECCOMPSANDBOXPARAM@"
# CPU features
# comma-separated list of cpu features to pass to the cpu
@@ -345,18 +339,6 @@ msize_9p = @DEFMSIZE9P@
# Default is false
disable_image_nvdimm = true
# Enable hot-plugging of VFIO devices to a bridge-port,
# root-port or switch-port.
# The default setting is "no-port"
hot_plug_vfio = "no-port"
# In a confidential compute environment hot-plugging can compromise
# security.
# Enable cold-plugging of VFIO devices to a bridge-port,
# root-port or switch-port.
# The default setting is "no-port", which means disabled.
cold_plug_vfio = "no-port"
# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
@@ -460,9 +442,6 @@ guest_memory_dump_paging = false
# be default_memory.
enable_guest_swap = false
# use legacy serial for guest console if available and implemented for architecture. Default false
use_legacy_serial = false
# disable applying SELinux on the VMM process (default false)
disable_selinux = @DEFDISABLESELINUX@
@@ -474,7 +453,7 @@ disable_selinux = @DEFDISABLESELINUX@
disable_guest_selinux = @DEFDISABLEGUESTSELINUX@
[factory]
[hypervisor.qemu.factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
@@ -593,20 +572,6 @@ agent_name = "@PROJECT_TYPE@"
# (default: true)
disable_guest_seccomp = @DEFDISABLEGUESTSECCOMP@
# vCPUs pinning settings
# if enabled, each vCPU thread will be scheduled to a fixed CPU
# qualified condition: num(vCPU threads) == num(CPUs in sandbox's CPUSet)
enable_vcpus_pinning = false
# Apply a custom SELinux security policy to the container process inside the VM.
# This is used when you want to apply a type other than the default `container_t`,
# so general users should not uncomment and apply it.
# (format: "user:role:type")
# Note: You cannot specify MCS policy with the label because the sensitivity levels and
# categories are determined automatically by high-level container runtimes such as containerd.
# Example value when enabling: "system_u:system_r:container_t"
guest_selinux_label = "@DEFGUESTSELINUXLABEL@"
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)

View File

@@ -103,12 +103,6 @@ kernel_params = "@KERNELPARAMS@"
# If you want that qemu uses the default firmware leave this option empty
firmware = "@FIRMWARE_SNP_PATH@"
# Path to the firmware volume.
# firmware TDVF or OVMF can be split into FIRMWARE_VARS.fd (UEFI variables
# as configuration) and FIRMWARE_CODE.fd (UEFI program image). UEFI variables
# can be customized per each user while UEFI code is kept same.
firmware_volume = "@FIRMWARE_VOLUME_SNP_PATH@"
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
@@ -116,12 +110,12 @@ machine_accelerators = "@MACHINEACCELERATORS@"
# Qemu seccomp sandbox feature
# comma-separated list of seccomp sandbox features to control the syscall access.
# For example, `seccompsandbox= "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# For example, `seccomp_sandbox = "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# Note: "elevateprivileges=deny" doesn't work with daemonize option, so it's removed from the seccomp sandbox
# Another note: enabling this feature may reduce performance, you may enable
# /proc/sys/net/core/bpf_jit_enable to reduce the impact. see https://man7.org/linux/man-pages/man8/bpfc.8.html
# Recommended value when enabling: "on,obsolete=deny,spawn=deny,resourcecontrol=deny"
seccompsandbox = "@DEFSECCOMPSANDBOXPARAM@"
seccomp_sandbox = "@DEFSECCOMPSANDBOXPARAM@"
# CPU features
# comma-separated list of cpu features to pass to the cpu
@@ -289,10 +283,6 @@ block_device_cache_noflush = false
#
enable_iothreads = @DEFENABLEIOTHREADS@
# Independent IOThreads enables IO to be processed in a separate thread, it is
# for QEMU hotplug device attach to iothread, like virtio-blk.
indep_iothreads = @DEFINDEPIOTHREADS@
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
@@ -346,11 +336,6 @@ enable_iommu_platform = false
# Your distribution recommends: @DEFVALIDVHOSTUSERSTOREPATHS@
valid_vhost_user_store_paths = @DEFVALIDVHOSTUSERSTOREPATHS@
# The timeout for reconnecting on non-server spdk sockets when the remote end goes away.
# qemu will delay this many seconds and then attempt to reconnect.
# Zero disables reconnecting, and the default is zero.
vhost_user_reconnect_timeout_sec = 0
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
@@ -407,7 +392,7 @@ disable_vhost_net = false
#
# If set to the empty string "", no extra monitor socket is added. This is
# the default.
#extra_monitor_socket = "hmp"
#dbg_monitor_socket = "hmp"
#
# Default entropy source.
@@ -495,9 +480,6 @@ guest_memory_dump_paging = false
# be default_memory.
enable_guest_swap = false
# use legacy serial for guest console if available and implemented for architecture. Default false
use_legacy_serial = false
# disable applying SELinux on the VMM process (default false)
disable_selinux = @DEFDISABLESELINUX@
@@ -509,7 +491,7 @@ disable_selinux = @DEFDISABLESELINUX@
disable_guest_selinux = @DEFDISABLEGUESTSELINUX@
[factory]
[hypervisor.qemu.factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
@@ -528,30 +510,6 @@ enable_template = false
# Default "/run/vc/vm/template"
template_path = "/run/vc/vm/template"
# The number of caches of VMCache:
# unspecified or == 0 --> VMCache is disabled
# > 0 --> will be set to the specified number
#
# VMCache is a function that creates VMs as caches before using it.
# It helps speed up new container creation.
# The function consists of a server and some clients communicating
# through Unix socket. The protocol is gRPC in protocols/cache/cache.proto.
# The VMCache server will create some VMs and cache them by factory cache.
# It will convert the VM to gRPC format and transport it when gets
# requestion from clients.
# Factory grpccache is the VMCache client. It will request gRPC format
# VM and convert it back to a VM. If VMCache function is enabled,
# kata-runtime will request VM from factory grpccache when it creates
# a new sandbox.
#
# Default 0
vm_cache_number = 0
# Specify the address of the Unix socket that is used by VMCache.
#
# Default /var/run/kata-containers/cache.sock
vm_cache_endpoint = "/var/run/kata-containers/cache.sock"
[agent.@PROJECT_TYPE@]
# If enabled, make the agent display debug-level messages.
# (default: disabled)
@@ -651,19 +609,6 @@ agent_name="@PROJECT_TYPE@"
# (default: true)
disable_guest_seccomp = @DEFDISABLEGUESTSECCOMP@
# vCPUs pinning settings
# if enabled, each vCPU thread will be scheduled to a fixed CPU
# qualified condition: num(vCPU threads) == num(CPUs in sandbox's CPUSet)
enable_vcpus_pinning = false
# Apply a custom SELinux security policy to the container process inside the VM.
# This is used when you want to apply a type other than the default `container_t`,
# so general users should not uncomment and apply it.
# (format: "user:role:type")
# Note: You cannot specify MCS policy with the label because the sensitivity levels and
# categories are determined automatically by high-level container runtimes such as containerd.
guest_selinux_label = "@DEFGUESTSELINUXLABEL@"
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
@@ -757,22 +702,3 @@ enable_pprof = false
# to the hypervisor.
# (default: /run/kata-containers/dans)
dan_conf = "@DEFDANCONF@"
# pod_resource_api_sock specifies the unix socket for the Kubelet's
# PodResource API endpoint. If empty, kubernetes based cold plug
# will not be attempted. In order for this feature to work, the
# KubeletPodResourcesGet featureGate must be enabled in Kubelet,
# if using Kubelet older than 1.34.
#
# The pod resource API's socket is relative to the Kubelet's root-dir,
# which is defined by the cluster admin, and its location is:
# ${KubeletRootDir}/pod-resources/kubelet.sock
#
# cold_plug_vfio(see hypervisor config) acts as a feature gate:
# cold_plug_vfio = no_port (default) => no cold plug
# cold_plug_vfio != no_port AND pod_resource_api_sock = "" => need
# explicit CDI annotation for cold plug (applies mainly
# to non-k8s cases)
# cold_plug_vfio != no_port AND pod_resource_api_sock != "" => kubelet
# based cold plug.
pod_resource_api_sock = "@DEFPODRESOURCEAPISOCK@"

View File

@@ -83,12 +83,6 @@ kernel_verity_params = "@KERNELVERITYPARAMS@"
# If you want that qemu uses the default firmware leave this option empty
firmware = "@FIRMWARETDVFPATH@"
# Path to the firmware volume.
# firmware TDVF or OVMF can be split into FIRMWARE_VARS.fd (UEFI variables
# as configuration) and FIRMWARE_CODE.fd (UEFI program image). UEFI variables
# can be customized per each user while UEFI code is kept same.
firmware_volume = "@FIRMWAREVOLUMEPATH@"
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
@@ -96,12 +90,12 @@ machine_accelerators = "@MACHINEACCELERATORS@"
# Qemu seccomp sandbox feature
# comma-separated list of seccomp sandbox features to control the syscall access.
# For example, `seccompsandbox= "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# For example, `seccomp_sandbox = "on,obsolete=deny,spawn=deny,resourcecontrol=deny"`
# Note: "elevateprivileges=deny" doesn't work with daemonize option, so it's removed from the seccomp sandbox
# Another note: enabling this feature may reduce performance, you may enable
# /proc/sys/net/core/bpf_jit_enable to reduce the impact. see https://man7.org/linux/man-pages/man8/bpfc.8.html
# Recommended value when enabling: "on,obsolete=deny,spawn=deny,resourcecontrol=deny"
seccompsandbox = "@DEFSECCOMPSANDBOXPARAM@"
seccomp_sandbox = "@DEFSECCOMPSANDBOXPARAM@"
# CPU features
# comma-separated list of cpu features to pass to the cpu
@@ -268,10 +262,6 @@ block_device_cache_noflush = false
#
enable_iothreads = @DEFENABLEIOTHREADS@
# Independent IOThreads enables IO to be processed in a separate thread, it is
# for QEMU hotplug device attach to iothread, like virtio-blk.
indep_iothreads = @DEFINDEPIOTHREADS@
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
@@ -325,11 +315,6 @@ enable_iommu_platform = false
# Your distribution recommends: @DEFVALIDVHOSTUSERSTOREPATHS@
valid_vhost_user_store_paths = @DEFVALIDVHOSTUSERSTOREPATHS@
# The timeout for reconnecting on non-server spdk sockets when the remote end goes away.
# qemu will delay this many seconds and then attempt to reconnect.
# Zero disables reconnecting, and the default is zero.
vhost_user_reconnect_timeout_sec = 0
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
@@ -364,7 +349,7 @@ enable_debug = false
#
# If set to the empty string "", no extra monitor socket is added. This is
# the default.
extra_monitor_socket = ""
dbg_monitor_socket = ""
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
@@ -474,9 +459,6 @@ guest_memory_dump_paging = false
# be default_memory.
enable_guest_swap = false
# use legacy serial for guest console if available and implemented for architecture. Default false
use_legacy_serial = false
# disable applying SELinux on the VMM process (default false)
disable_selinux = @DEFDISABLESELINUX@
@@ -488,7 +470,7 @@ disable_selinux = @DEFDISABLESELINUX@
disable_guest_selinux = @DEFDISABLEGUESTSELINUX@
[factory]
[hypervisor.qemu.factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
@@ -507,30 +489,6 @@ enable_template = false
# Default "/run/vc/vm/template"
template_path = "/run/vc/vm/template"
# The number of caches of VMCache:
# unspecified or == 0 --> VMCache is disabled
# > 0 --> will be set to the specified number
#
# VMCache is a function that creates VMs as caches before using it.
# It helps speed up new container creation.
# The function consists of a server and some clients communicating
# through Unix socket. The protocol is gRPC in protocols/cache/cache.proto.
# The VMCache server will create some VMs and cache them by factory cache.
# It will convert the VM to gRPC format and transport it when gets
# requestion from clients.
# Factory grpccache is the VMCache client. It will request gRPC format
# VM and convert it back to a VM. If VMCache function is enabled,
# kata-runtime will request VM from factory grpccache when it creates
# a new sandbox.
#
# Default 0
vm_cache_number = 0
# Specify the address of the Unix socket that is used by VMCache.
#
# Default /var/run/kata-containers/cache.sock
vm_cache_endpoint = "/var/run/kata-containers/cache.sock"
[agent.@PROJECT_TYPE@]
# If enabled, make the agent display debug-level messages.
# (default: disabled)
@@ -631,20 +589,6 @@ agent_name="@PROJECT_TYPE@"
# (default: true)
disable_guest_seccomp = @DEFDISABLEGUESTSECCOMP@
# vCPUs pinning settings
# if enabled, each vCPU thread will be scheduled to a fixed CPU
# qualified condition: num(vCPU threads) == num(CPUs in sandbox's CPUSet)
enable_vcpus_pinning = false
# Apply a custom SELinux security policy to the container process inside the VM.
# This is used when you want to apply a type other than the default `container_t`,
# so general users should not uncomment and apply it.
# (format: "user:role:type")
# Note: You cannot specify MCS policy with the label because the sensitivity levels and
# categories are determined automatically by high-level container runtimes such as containerd.
# Example value when enabling: "system_u:system_r:container_t"
guest_selinux_label = "@DEFGUESTSELINUXLABEL@"
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
@@ -739,21 +683,3 @@ enable_pprof = false
# (default: /run/kata-containers/dans)
dan_conf = "@DEFDANCONF@"
# pod_resource_api_sock specifies the unix socket for the Kubelet's
# PodResource API endpoint. If empty, kubernetes based cold plug
# will not be attempted. In order for this feature to work, the
# KubeletPodResourcesGet featureGate must be enabled in Kubelet,
# if using Kubelet older than 1.34.
#
# The pod resource API's socket is relative to the Kubelet's root-dir,
# which is defined by the cluster admin, and its location is:
# ${KubeletRootDir}/pod-resources/kubelet.sock
#
# cold_plug_vfio(see hypervisor config) acts as a feature gate:
# cold_plug_vfio = no_port (default) => no cold plug
# cold_plug_vfio != no_port AND pod_resource_api_sock = "" => need
# explicit CDI annotation for cold plug (applies mainly
# to non-k8s cases)
# cold_plug_vfio != no_port AND pod_resource_api_sock != "" => kubelet
# based cold plug.
pod_resource_api_sock = "@DEFPODRESOURCEAPISOCK@"

View File

@@ -205,15 +205,6 @@ agent_name = "kata"
disable_guest_seccomp = true
# Apply a custom SELinux security policy to the container process inside the VM.
# This is used when you want to apply a type other than the default `container_t`,
# so general users should not uncomment and apply it.
# (format: "user:role:type")
# Note: You cannot specify MCS policy with the label because the sensitivity levels and
# categories are determined automatically by high-level container runtimes such as containerd.
# Example value when enabling: "system_u:system_r:container_t"
guest_selinux_label = "@DEFGUESTSELINUXLABEL@"
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)

View File

@@ -2610,6 +2610,7 @@ impl<'a> QemuCmdLine<'a> {
self.devices.push(Box::new(Bios::new(firmware.to_owned())));
self.machine
.set_kernel_irqchip("split")
.set_confidential_guest_support("tdx")
.set_nvdimm(false);
}

View File

@@ -84,16 +84,6 @@ impl ResourceManager {
inner.handle_network(network_config).await
}
pub async fn has_network_endpoints(&self) -> bool {
let inner = self.inner.read().await;
inner.has_network_endpoints().await
}
pub async fn setup_network_in_guest(&self) -> Result<()> {
let inner = self.inner.read().await;
inner.setup_network_in_guest().await
}
#[instrument]
pub async fn setup_after_start_vm(&self) -> Result<()> {
let mut inner = self.inner.write().await;

View File

@@ -296,33 +296,6 @@ impl ResourceManagerInner {
Ok(())
}
pub async fn has_network_endpoints(&self) -> bool {
if let Some(network) = &self.network {
match network.interfaces().await {
std::result::Result::Ok(interfaces) => !interfaces.is_empty(),
Err(_) => false,
}
} else {
false
}
}
pub async fn setup_network_in_guest(&self) -> Result<()> {
if let Some(network) = self.network.as_ref() {
let network = network.as_ref();
self.handle_interfaces(network)
.await
.context("handle interfaces during network rescan")?;
self.handle_neighbours(network)
.await
.context("handle neighbours during network rescan")?;
self.handle_routes(network)
.await
.context("handle routes during network rescan")?;
}
Ok(())
}
pub async fn setup_after_start_vm(&mut self) -> Result<()> {
self.cgroups_resource
.setup_after_start_vm(self.hypervisor.as_ref())

View File

@@ -53,9 +53,6 @@ linux_container = { workspace = true, optional = true }
virt_container = { workspace = true, optional = true }
wasm_container = { workspace = true, optional = true }
[dev-dependencies]
rstest = { workspace = true }
[features]
default = ["virt"]
linux = ["linux_container"]

View File

@@ -51,13 +51,6 @@ pub trait Sandbox: Send + Sync {
shim_pid: u32,
) -> Result<()>;
/// Re-scan the network namespace for late-discovered endpoints.
/// This handles runtimes like Docker 26+ that configure networking
/// after the Start response. The default implementation is a no-op.
async fn rescan_network(&self) -> Result<()> {
Ok(())
}
// metrics function
async fn agent_metrics(&self) -> Result<String>;
async fn hypervisor_metrics(&self) -> Result<String>;

View File

@@ -69,53 +69,6 @@ use crate::{
tracer::{KataTracer, ROOTSPAN},
};
const DOCKER_LIBNETWORK_SETKEY: &str = "libnetwork-setkey";
const DOCKER_NETNS_PREFIXES: &[&str] = &["/var/run/docker/netns/", "/run/docker/netns/"];
fn is_valid_docker_sandbox_id(id: &str) -> bool {
id.len() == 64 && id.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f'))
}
/// Discover Docker's pre-created network namespace path from OCI spec hooks.
///
/// Docker's libnetwork-setkey hook contains the sandbox ID as its
/// argument following "libnetwork-setkey", which maps to a netns file
/// under /var/run/docker/netns/<sandbox_id> or /run/docker/netns/<sandbox_id>.
fn docker_netns_path(spec: &oci::Spec) -> Option<String> {
let hooks = spec.hooks().as_ref()?;
let hook_sets: [&[oci::Hook]; 2] = [
hooks.prestart().as_deref().unwrap_or_default(),
hooks.create_runtime().as_deref().unwrap_or_default(),
];
for hooks in &hook_sets {
for hook in *hooks {
if let Some(args) = hook.args() {
for (i, arg) in args.iter().enumerate() {
if arg == DOCKER_LIBNETWORK_SETKEY && i + 1 < args.len() {
let sandbox_id = &args[i + 1];
if !is_valid_docker_sandbox_id(sandbox_id) {
continue;
}
for prefix in DOCKER_NETNS_PREFIXES {
let ns_path = format!("{}{}", prefix, sandbox_id);
if let Ok(metadata) = std::fs::symlink_metadata(&ns_path) {
if metadata.is_file() {
return Some(ns_path);
}
}
}
}
}
}
}
}
None
}
fn convert_string_to_slog_level(string_level: &str) -> slog::Level {
match string_level {
"trace" => slog::Level::Trace,
@@ -424,17 +377,8 @@ impl RuntimeHandlerManager {
if ns.path().is_some() {
netns = ns.path().clone().map(|p| p.display().to_string());
}
// Docker 26+ may configure networking outside of the OCI
// spec namespace path. Try to discover the netns from hook
// args before falling back to creating a placeholder.
else if let Some(docker_ns) = docker_netns_path(spec) {
info!(
sl!(),
"discovered Docker network namespace from hook args";
"netns" => &docker_ns
);
netns = Some(docker_ns);
} else {
// if we get empty netns from oci spec, we need to create netns for the VM
else {
let ns_name = generate_netns_name();
let raw_netns = NetNs::new(ns_name)?;
let path = Some(PathBuf::from(raw_netns.path()).display().to_string());
@@ -695,7 +639,6 @@ impl RuntimeHandlerManager {
Ok(TaskResponse::WaitProcess(exit_status))
}
TaskRequest::StartProcess(process_id) => {
let is_sandbox_container = cm.is_sandbox_container(&process_id).await;
let shim_pid = cm
.start_process(&process_id)
.await
@@ -704,25 +647,6 @@ impl RuntimeHandlerManager {
let pid = shim_pid.pid;
let process_type = process_id.process_type;
let container_id = process_id.container_id().to_string();
// Schedule an async network rescan for sandbox containers.
// This handles runtimes that configure networking after the
// Start response (e.g. Docker 26+). rescan_network is
// idempotent — it returns immediately if endpoints already
// exist.
if is_sandbox_container {
let sandbox_rescan = sandbox.clone();
tokio::spawn(async move {
if let Err(e) = sandbox_rescan.rescan_network().await {
error!(
sl!(),
"async network rescan failed — container may lack networking: {:?}",
e
);
}
});
}
tokio::spawn(async move {
let result = sandbox.wait_process(cm, process_id, pid).await;
if let Err(e) = result {
@@ -996,85 +920,3 @@ fn configure_non_root_hypervisor(config: &mut Hypervisor) -> Result<()> {
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use oci_spec::runtime::{HookBuilder, HooksBuilder, SpecBuilder};
use rstest::rstest;
const VALID_SANDBOX_ID: &str =
"a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2";
#[rstest]
#[case::all_lowercase_hex(VALID_SANDBOX_ID, true)]
#[case::all_zeros("0000000000000000000000000000000000000000000000000000000000000000", true)]
#[case::uppercase_hex("A1B2C3D4E5F6A1B2C3D4E5F6A1B2C3D4E5F6A1B2C3D4E5F6A1B2C3D4E5F6A1B2", false)]
#[case::too_short("a1b2c3d4", false)]
#[case::non_hex("zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz", false)]
#[case::path_traversal("../../../etc/passwd", false)]
#[case::empty("", false)]
fn test_is_valid_docker_sandbox_id(#[case] id: &str, #[case] expected: bool) {
assert_eq!(is_valid_docker_sandbox_id(id), expected);
}
fn make_hook_with_args(args: Vec<&str>) -> oci::Hook {
HookBuilder::default()
.path("/usr/bin/test")
.args(args.into_iter().map(String::from).collect::<Vec<_>>())
.build()
.unwrap()
}
#[rstest]
#[case::no_hooks(None, None)]
#[case::unrelated_hooks(
Some(HooksBuilder::default()
.prestart(vec![make_hook_with_args(vec!["some-hook", "arg1"])])
.build().unwrap()),
None
)]
#[case::invalid_sandbox_id(
Some(HooksBuilder::default()
.prestart(vec![make_hook_with_args(vec![
"/usr/bin/dockerd", "libnetwork-setkey", "not-a-valid-id",
])])
.build().unwrap()),
None
)]
#[case::setkey_at_end_of_args(
Some(HooksBuilder::default()
.prestart(vec![make_hook_with_args(vec![
"/usr/bin/dockerd", "libnetwork-setkey",
])])
.build().unwrap()),
None
)]
#[case::valid_prestart_but_no_file(
Some(HooksBuilder::default()
.prestart(vec![make_hook_with_args(vec![
"/usr/bin/dockerd", "libnetwork-setkey", VALID_SANDBOX_ID,
])])
.build().unwrap()),
None
)]
#[case::valid_create_runtime_but_no_file(
Some(HooksBuilder::default()
.create_runtime(vec![make_hook_with_args(vec![
"/usr/bin/dockerd", "libnetwork-setkey", VALID_SANDBOX_ID,
])])
.build().unwrap()),
None
)]
fn test_docker_netns_path(
#[case] hooks: Option<oci::Hooks>,
#[case] expected: Option<String>,
) {
let mut builder = SpecBuilder::default();
if let Some(h) = hooks {
builder = builder.hooks(h);
}
let spec = builder.build().unwrap();
assert_eq!(docker_netns_path(&spec), expected);
}
}

View File

@@ -58,7 +58,6 @@ use resource::{ResourceConfig, ResourceManager};
use runtime_spec as spec;
use std::path::Path;
use std::sync::Arc;
use std::time::Duration;
use strum::Display;
use tokio::sync::{mpsc::Sender, Mutex, RwLock};
use tracing::instrument;
@@ -974,71 +973,6 @@ impl Sandbox for VirtSandbox {
self.hypervisor.get_hypervisor_metrics().await
}
async fn rescan_network(&self) -> Result<()> {
let config = self.resource_manager.config().await;
if config.runtime.disable_new_netns {
return Ok(());
}
if dan_config_path(&config, &self.sid).exists() {
return Ok(());
}
if self.resource_manager.has_network_endpoints().await {
return Ok(());
}
let sandbox_config = match &self.sandbox_config {
Some(c) => c,
None => return Ok(()),
};
let netns_path = match &sandbox_config.network_env.netns {
Some(p) => p.clone(),
None => return Ok(()),
};
const MAX_WAIT: Duration = Duration::from_secs(5);
const POLL_INTERVAL: Duration = Duration::from_millis(50);
let deadline = tokio::time::Instant::now() + MAX_WAIT;
info!(sl!(), "waiting for network interfaces in namespace");
loop {
let network_config = NetworkConfig::NetNs(NetworkWithNetNsConfig {
network_model: config.runtime.internetworking_model.clone(),
netns_path: netns_path.clone(),
queues: self
.hypervisor
.hypervisor_config()
.await
.network_info
.network_queues as usize,
network_created: sandbox_config.network_env.network_created,
});
if let Err(e) = self.resource_manager.handle_network(network_config).await {
warn!(sl!(), "network rescan attempt failed: {:?}", e);
}
if self.resource_manager.has_network_endpoints().await {
info!(sl!(), "network interfaces discovered during rescan");
return self
.resource_manager
.setup_network_in_guest()
.await
.context("setup network in guest after rescan");
}
if tokio::time::Instant::now() >= deadline {
warn!(
sl!(),
"no network interfaces found after timeout — networking may be configured later"
);
return Ok(());
}
tokio::time::sleep(POLL_INTERVAL).await;
}
}
async fn set_policy(&self, policy: &str) -> Result<()> {
if policy.is_empty() {
debug!(sl!(), "sb: set_policy skipped without policy");

View File

@@ -143,7 +143,13 @@ DEFROOTFSTYPE := $(ROOTFSTYPE_EXT4)
FIRMWAREPATH :=
FIRMWAREVOLUMEPATH :=
FIRMWAREPATH_NV = $(FIRMWAREPATH)
FIRMWAREPATH_NV :=
ifeq ($(ARCH),amd64)
FIRMWAREPATH_NV := $(PREFIXDEPS)/share/$(EDK2_NAME)/OVMF.fd
endif
ifeq ($(ARCH),arm64)
FIRMWAREPATH_NV := $(PREFIXDEPS)/share/$(EDK2_NAME)/AAVMF_CODE.fd
endif
FIRMWARETDVFPATH := $(PREFIXDEPS)/share/ovmf/OVMF.inteltdx.fd
FIRMWARETDVFPATH_NV := $(FIRMWARETDVFPATH)

View File

@@ -166,4 +166,14 @@ impl yaml::K8sResource for CronJob {
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
yaml::get_sysctls(&self.spec.jobTemplate.spec.template.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
self.spec
.jobTemplate
.spec
.template
.spec
.securityContext
.as_ref()
}
}

View File

@@ -167,4 +167,8 @@ impl yaml::K8sResource for DaemonSet {
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
yaml::get_sysctls(&self.spec.template.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
self.spec.template.spec.securityContext.as_ref()
}
}

View File

@@ -178,4 +178,8 @@ impl yaml::K8sResource for Deployment {
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
yaml::get_sysctls(&self.spec.template.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
self.spec.template.spec.securityContext.as_ref()
}
}

View File

@@ -167,6 +167,10 @@ impl yaml::K8sResource for Job {
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
yaml::get_sysctls(&self.spec.template.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
self.spec.template.spec.securityContext.as_ref()
}
}
pub fn pod_name_regex(job_name: String) -> String {

View File

@@ -114,10 +114,12 @@ pub fn get_mount_and_storage(
if let Some(emptyDir) = &yaml_volume.emptyDir {
let settings_volumes = &settings.volumes;
let volume = match emptyDir.medium.as_deref() {
Some("Memory") => &settings_volumes.emptyDir_memory,
_ if settings.cluster_config.encrypted_emptydir => &settings_volumes.emptyDir_encrypted,
_ => &settings_volumes.emptyDir,
let (volume, block_encrypted_emptydir) = match emptyDir.medium.as_deref() {
Some("Memory") => (&settings_volumes.emptyDir_memory, false),
_ if settings.cluster_config.encrypted_emptydir => {
(&settings_volumes.emptyDir_encrypted, true)
}
_ => (&settings_volumes.emptyDir, false),
};
get_empty_dir_mount_and_storage(
@@ -127,6 +129,7 @@ pub fn get_mount_and_storage(
yaml_mount,
volume,
pod_security_context,
block_encrypted_emptydir,
);
} else if yaml_volume.persistentVolumeClaim.is_some() || yaml_volume.azureFile.is_some() {
get_shared_bind_mount(yaml_mount, p_mounts, "rprivate", "rw");
@@ -150,18 +153,42 @@ fn get_empty_dir_mount_and_storage(
yaml_mount: &pod::VolumeMount,
settings_empty_dir: &settings::EmptyDirVolume,
pod_security_context: &Option<pod::PodSecurityContext>,
block_encrypted_emptydir: bool,
) {
debug!("Settings emptyDir: {:?}", settings_empty_dir);
if yaml_mount.subPathExpr.is_none() {
let mut options = settings_empty_dir.options.clone();
if let Some(gid) = pod_security_context.as_ref().and_then(|sc| sc.fsGroup) {
// This matches the runtime behavior of only setting the fsgid if the mountpoint GID is not 0.
// https://github.com/kata-containers/kata-containers/blob/b69da5f3ba8385c5833b31db41a846a203812675/src/runtime/virtcontainers/kata_agent.go#L1602-L1607
if gid != 0 {
options.push(format!("fsgid={gid}"));
// Pod fsGroup in policy must mirror how the shim encodes it on Storage:
// - block-encrypted host emptyDirs become virtio-blk/scsi volumes; the runtime sets
// Storage.fs_group from mount metadata (handleDeviceBlockVolume in kata_agent.go).
// - shared-fs / guest-local emptyDirs use Storage.options: the runtime appends
// fsgid=<host GID> when the volume is not root-owned (handleEphemeralStorage and
// handleLocalStorage in kata_agent.go). Genpolicy uses pod fsGroup when non-zero as
// the usual kubelet-applied GID for that stat.
let pod_gid = pod_security_context.as_ref().and_then(|sc| sc.fsGroup);
let fs_group = if block_encrypted_emptydir {
match pod_gid {
Some(gid) if gid > 0 => protobuf::MessageField::some(agent::FSGroup {
group_id: u32::try_from(gid).unwrap_or_else(|_| {
panic!(
"get_empty_dir_mount_and_storage: securityContext.fsGroup {gid} \
must be <= {}",
u32::MAX
)
}),
..Default::default()
}),
_ => protobuf::MessageField::none(),
}
}
} else {
if let Some(gid) = pod_gid {
if gid != 0 {
options.push(format!("fsgid={gid}"));
}
}
protobuf::MessageField::none()
};
storages.push(agent::Storage {
driver: settings_empty_dir.driver.clone(),
driver_options: settings_empty_dir.driver_options.clone(),
@@ -173,7 +200,7 @@ fn get_empty_dir_mount_and_storage(
} else {
settings_empty_dir.mount_point.clone()
},
fs_group: protobuf::MessageField::none(),
fs_group,
shared: settings_empty_dir.shared,
special_fields: ::protobuf::SpecialFields::new(),
});

View File

@@ -937,6 +937,10 @@ impl yaml::K8sResource for Pod {
fn get_sysctls(&self) -> Vec<Sysctl> {
yaml::get_sysctls(&self.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&PodSecurityContext> {
self.spec.securityContext.as_ref()
}
}
impl Container {

View File

@@ -971,6 +971,16 @@ impl AgentPolicy {
);
}
yaml::apply_pod_fs_group_and_supplemental_groups(
&mut process,
resource.get_pod_security_context(),
is_pause_container,
);
debug!(
"get_container_process: after apply_pod_fs_group_and_supplemental_groups: User = {:?}",
&process.User
);
///////////////////////////////////////////////////////////////////////////////////////
// Container-level settings from user's YAML.
yaml_container.get_process_fields(&mut process);

View File

@@ -128,4 +128,8 @@ impl yaml::K8sResource for ReplicaSet {
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
yaml::get_sysctls(&self.spec.template.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
self.spec.template.spec.securityContext.as_ref()
}
}

View File

@@ -131,4 +131,8 @@ impl yaml::K8sResource for ReplicationController {
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
yaml::get_sysctls(&self.spec.template.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
self.spec.template.spec.securityContext.as_ref()
}
}

View File

@@ -211,6 +211,10 @@ impl yaml::K8sResource for StatefulSet {
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
yaml::get_sysctls(&self.spec.template.spec.securityContext)
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
self.spec.template.spec.securityContext.as_ref()
}
}
impl StatefulSet {

View File

@@ -107,6 +107,10 @@ pub trait K8sResource {
// for some of the K8s resource types.
}
fn get_pod_security_context(&self) -> Option<&pod::PodSecurityContext> {
None
}
fn get_sysctls(&self) -> Vec<pod::Sysctl> {
vec![]
}
@@ -388,6 +392,39 @@ fn handle_unused_field(path: &str, silent_unsupported_fields: bool) {
}
}
/// Applies pod `fsGroup` and `supplementalGroups` to `AdditionalGids`.
pub fn apply_pod_fs_group_and_supplemental_groups(
process: &mut policy::KataProcess,
security_context: Option<&pod::PodSecurityContext>,
is_pause_container: bool,
) {
if is_pause_container {
return;
}
let Some(context) = security_context else {
return;
};
if let Some(fs_group) = context.fsGroup {
let gid: u32 = fs_group.try_into().unwrap();
process.User.AdditionalGids.insert(gid);
debug!(
"apply_pod_fs_group_and_supplemental_groups: inserted fs_group = {gid} into AdditionalGids, User = {:?}",
&process.User
);
}
if let Some(supplemental_groups) = &context.supplementalGroups {
supplemental_groups.iter().for_each(|g| {
process.User.AdditionalGids.insert(*g);
});
debug!(
"apply_pod_fs_group_and_supplemental_groups: inserted supplementalGroups = {:?} into AdditionalGids, User = {:?}",
&supplemental_groups, &process.User
);
}
}
pub fn get_process_fields(
process: &mut policy::KataProcess,
must_check_passwd: &mut bool,
@@ -447,27 +484,6 @@ pub fn get_process_fields(
*must_check_passwd = false;
}
if !is_pause_container {
if let Some(fs_group) = context.fsGroup {
let gid = fs_group.try_into().unwrap();
process.User.AdditionalGids.insert(gid);
debug!(
"get_process_fields: inserted fs_group = {gid} into AdditionalGids, User = {:?}",
&process.User
);
}
if let Some(supplemental_groups) = &context.supplementalGroups {
supplemental_groups.iter().for_each(|g| {
process.User.AdditionalGids.insert(*g);
});
debug!(
"get_process_fields: inserted supplementalGroups = {:?} into AdditionalGids, User = {:?}",
&supplemental_groups, &process.User
);
}
}
if let Some(allow) = context.allowPrivilegeEscalation {
process.NoNewPrivileges = !allow
}

View File

@@ -345,12 +345,12 @@
"driver_options": [
"encryption_key=ephemeral"
],
"fs_group": null,
"fs_group": {
"group_id": 1000
},
"fstype": "ext4",
"mount_point": "/run/kata-containers/sandbox/storage/MDAvMDA=",
"options": [
"fsgid=1000"
],
"options": [],
"source": "00/00",
"shared": true
}

View File

@@ -635,7 +635,7 @@ function helm_helper() {
base_values_file="${helm_chart_dir}/try-kata-nvidia-gpu.values.yaml"
fi
;;
qemu-snp|qemu-tdx|qemu-se|qemu-se-runtime-rs|qemu-cca|qemu-coco-dev|qemu-coco-dev-runtime-rs)
qemu-snp|qemu-snp-runtime-rs|qemu-tdx|qemu-se|qemu-se-runtime-rs|qemu-cca|qemu-coco-dev|qemu-coco-dev-runtime-rs)
# Use TEE example file
if [[ -f "${helm_chart_dir}/try-kata-tee.values.yaml" ]]; then
base_values_file="${helm_chart_dir}/try-kata-tee.values.yaml"

View File

@@ -11,7 +11,7 @@ source "${BATS_TEST_DIRNAME}/../../common.bash"
load "${BATS_TEST_DIRNAME}/confidential_kbs.sh"
SUPPORTED_GPU_TEE_HYPERVISORS=("qemu-nvidia-gpu-snp" "qemu-nvidia-gpu-tdx")
SUPPORTED_TEE_HYPERVISORS=("qemu-snp" "qemu-tdx" "qemu-se" "qemu-se-runtime-rs" "${SUPPORTED_GPU_TEE_HYPERVISORS[@]}")
SUPPORTED_TEE_HYPERVISORS=("qemu-snp" "qemu-snp-runtime-rs" "qemu-tdx" "qemu-se" "qemu-se-runtime-rs" "${SUPPORTED_GPU_TEE_HYPERVISORS[@]}")
SUPPORTED_NON_TEE_HYPERVISORS=("qemu-coco-dev" "qemu-coco-dev-runtime-rs")
function setup_unencrypted_confidential_pod() {
@@ -36,7 +36,7 @@ function get_remote_command_per_hypervisor() {
qemu-se*)
echo "cd /sys/firmware/uv; cat prot_virt_guest | grep 1"
;;
qemu-snp)
qemu-snp|qemu-snp-runtime-rs)
echo "dmesg | grep \"Memory Encryption Features active:.*SEV-SNP\""
;;
qemu-tdx)

View File

@@ -187,7 +187,7 @@ function deploy_kata() {
# Workaround to avoid modifying the workflow yaml files
case "${KATA_HYPERVISOR}" in
qemu-tdx|qemu-snp|qemu-nvidia-gpu-*)
qemu-tdx|qemu-snp|qemu-snp-runtime-rs|qemu-nvidia-gpu-*)
USE_EXPERIMENTAL_SETUP_SNAPSHOTTER=true
SNAPSHOTTER="nydus"
EXPERIMENTAL_FORCE_GUEST_PULL=false
@@ -447,7 +447,7 @@ function cleanup() {
}
function deploy_snapshotter() {
if [[ "${KATA_HYPERVISOR}" == "qemu-tdx" || "${KATA_HYPERVISOR}" == "qemu-snp" ]]; then
if [[ "${KATA_HYPERVISOR}" == "qemu-tdx" || "${KATA_HYPERVISOR}" == "qemu-snp" || "${KATA_HYPERVISOR}" == "qemu-snp-runtime-rs" ]]; then
echo "[Skip] ${SNAPSHOTTER} is pre-installed in the TEE machine"
return
fi
@@ -461,7 +461,7 @@ function deploy_snapshotter() {
}
function cleanup_snapshotter() {
if [[ "${KATA_HYPERVISOR}" == "qemu-tdx" || "${KATA_HYPERVISOR}" == "qemu-snp" ]]; then
if [[ "${KATA_HYPERVISOR}" == "qemu-tdx" || "${KATA_HYPERVISOR}" == "qemu-snp" || "${KATA_HYPERVISOR}" == "qemu-snp-runtime-rs" ]]; then
echo "[Skip] ${SNAPSHOTTER} is pre-installed in the TEE machine"
return
fi

View File

@@ -15,7 +15,7 @@ setup() {
[ "${KATA_HYPERVISOR}" == "qemu-se-runtime-rs" ] && skip "Requires CPU hotplug which isn't supported on ${KATA_HYPERVISOR} yet"
[[ "${KATA_HYPERVISOR}" == qemu-coco-dev* ]] && skip "Requires CPU hotplug which disabled by static_sandbox_resource_mgmt"
( [ "${KATA_HYPERVISOR}" == "qemu-tdx" ] || [ "${KATA_HYPERVISOR}" == "qemu-snp" ] || \
[ "${KATA_HYPERVISOR}" == "qemu-se" ] ) \
[ "${KATA_HYPERVISOR}" == "qemu-snp-runtime-rs" ] || [ "${KATA_HYPERVISOR}" == "qemu-se" ] ) \
&& skip "TEEs do not support memory / CPU hotplug"
pod_name="constraints-cpu-test"
@@ -121,7 +121,7 @@ teardown() {
[ "${KATA_HYPERVISOR}" == "qemu-se-runtime-rs" ] && skip "Requires CPU hotplug which isn't supported on ${KATA_HYPERVISOR} yet"
[[ "${KATA_HYPERVISOR}" == qemu-coco-dev* ]] && skip "Requires CPU hotplug which disabled by static_sandbox_resource_mgmt"
( [ "${KATA_HYPERVISOR}" == "qemu-tdx" ] || [ "${KATA_HYPERVISOR}" == "qemu-snp" ] || \
[ "${KATA_HYPERVISOR}" == "qemu-se" ] ) \
[ "${KATA_HYPERVISOR}" == "qemu-snp-runtime-rs" ] || [ "${KATA_HYPERVISOR}" == "qemu-se" ] ) \
&& skip "TEEs do not support memory / CPU hotplug"
# Debugging information

View File

@@ -22,10 +22,17 @@ setup() {
pod_name="sharevol-kata"
pod_logs_file=""
setup_common || die "setup_common failed"
yaml_file="${pod_config_dir}/pod-empty-dir.yaml"
# Add policy to yaml
policy_settings_dir="$(create_tmp_policy_settings_dir "${pod_config_dir}")"
add_requests_to_policy_settings "${policy_settings_dir}" "ReadStreamRequest"
}
@test "Empty dir volumes" {
local yaml_file
local mount_command
local dd_command
yaml_file="${pod_config_dir}/pod-empty-dir.yaml"
mount_command=(sh -c "mount | grep cache")
add_exec_to_policy_settings "${policy_settings_dir}" "${mount_command[@]}"
@@ -33,11 +40,9 @@ setup() {
dd_command=(sh -c "dd if=/dev/zero of=/tmp/cache/file1 bs=1M count=50; echo $?")
add_exec_to_policy_settings "${policy_settings_dir}" "${dd_command[@]}"
add_requests_to_policy_settings "${policy_settings_dir}" "ReadStreamRequest"
# Add policy to yaml
auto_generate_policy "${policy_settings_dir}" "${yaml_file}"
}
@test "Empty dir volumes" {
# Create the pod
kubectl create -f "${yaml_file}"
@@ -55,20 +60,25 @@ setup() {
local agnhost_name
local agnhost_version
local gid
local image
local logs
local pod_file
local pod_yaml
local pod_yaml_in
local uid
# This is a reproducer of k8s e2e "[sig-storage] EmptyDir volumes when FSGroup is specified [LinuxOnly] [NodeFeature:FSGroup] new files should be created with FSGroup ownership when container is non-root" test
pod_file="${pod_config_dir}/pod-empty-dir-fsgroup.yaml"
pod_yaml_in="${pod_config_dir}/pod-empty-dir-fsgroup.yaml.in"
pod_yaml="${pod_config_dir}/pod-empty-dir-fsgroup.yaml"
agnhost_name="${container_images_agnhost_name}"
agnhost_version="${container_images_agnhost_version}"
image="${agnhost_name}:${agnhost_version}"
export AGNHOST_IMAGE="${agnhost_name}:${agnhost_version}"
envsubst '${AGNHOST_IMAGE}' <"${pod_yaml_in}" >"${pod_yaml}"
# Add policy to yaml
auto_generate_policy "${policy_settings_dir}" "${pod_yaml}"
# Try to avoid timeout by prefetching the image.
sed -e "s#\${agnhost_image}#${image}#" "$pod_file" |\
kubectl create -f -
kubectl create -f "${pod_yaml}"
cmd="kubectl get pods ${pod_name} | grep Completed"
waitForProcess "${wait_time}" "${sleep_time}" "${cmd}"
@@ -90,6 +100,7 @@ setup() {
teardown() {
[ ! -f "$pod_logs_file" ] || rm -f "$pod_logs_file"
[[ -n "${pod_config_dir:-}" ]] && rm -f "${pod_config_dir}/pod-empty-dir-fsgroup.yaml"
delete_tmp_policy_settings_dir "${policy_settings_dir}"
teardown_common "${node}" "${node_start_time:-}"

View File

@@ -10,6 +10,7 @@ load "${BATS_TEST_DIRNAME}/confidential_common.sh"
export KATA_HYPERVISOR="${KATA_HYPERVISOR:-qemu-nvidia-gpu}"
# when using hostPath, ensure directory is writable by container user
export LOCAL_NIM_CACHE="/opt/nim/.cache"
SKIP_MULTI_GPU_TESTS=${SKIP_MULTI_GPU_TESTS:-false}

View File

@@ -16,14 +16,18 @@ metadata:
# cc_init_data annotation will be added by genpolicy with CDH configuration
# from the custom default-initdata.toml created by create_nim_initdata_file()
spec:
# Explicit user/group/supplementary groups to support nydus guest-pull.
# See issue https://github.com/kata-containers/kata-containers/issues/11162 and
# other references to this issue in the genpolicy source folder.
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
supplementalGroups: [4, 20, 24, 25, 27, 29, 30, 44, 46]
restartPolicy: Never
runtimeClassName: kata
imagePullSecrets:
- name: ngc-secret-instruct
securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0
containers:
- name: ${POD_NAME_INSTRUCT}
image: nvcr.io/nim/meta/llama-3.1-8b-instruct:1.13.1

View File

@@ -14,10 +14,6 @@ spec:
runtimeClassName: kata
imagePullSecrets:
- name: ngc-secret-instruct
securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0
containers:
- name: ${POD_NAME_INSTRUCT}
image: nvcr.io/nim/meta/llama-3.1-8b-instruct:1.13.1

View File

@@ -16,15 +16,18 @@ metadata:
# cc_init_data annotation will be added by genpolicy with CDH configuration
# from the custom default-initdata.toml created by create_nim_initdata_file()
spec:
# Explicit user/group/supplementary groups to support nydus guest-pull.
# See issue https://github.com/kata-containers/kata-containers/issues/11162 and
# other references to this issue in the genpolicy source folder.
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
restartPolicy: Always
runtimeClassName: kata
serviceAccountName: default
imagePullSecrets:
- name: ngc-secret-embedqa
securityContext:
fsGroup: 0
runAsGroup: 0
runAsUser: 0
containers:
- name: ${POD_NAME_EMBEDQA}
image: nvcr.io/nim/nvidia/llama-3.2-nv-embedqa-1b-v2:1.10.1

View File

@@ -10,15 +10,16 @@ metadata:
labels:
app: ${POD_NAME_EMBEDQA}
spec:
# unlike the instruct manifest, this image needs securityContext to
# avoid NVML/GPU permission failures
securityContext:
runAsUser: 1000
runAsGroup: 1000
restartPolicy: Always
runtimeClassName: kata
serviceAccountName: default
imagePullSecrets:
- name: ngc-secret-embedqa
securityContext:
fsGroup: 0
runAsGroup: 0
runAsUser: 0
containers:
- name: ${POD_NAME_EMBEDQA}
image: nvcr.io/nim/nvidia/llama-3.2-nv-embedqa-1b-v2:1.10.1

View File

@@ -15,7 +15,7 @@ spec:
fsGroup: 123
containers:
- name: mounttest-container
image: ${agnhost_image}
image: ${AGNHOST_IMAGE}
command:
- /agnhost
args:
@@ -28,7 +28,7 @@ spec:
- name: emptydir-volume
mountPath: /test-volume
- name: mounttest-container-2
image: ${agnhost_image}
image: ${AGNHOST_IMAGE}
command:
- /agnhost
args:

View File

@@ -138,7 +138,7 @@ add_runtime_handler_annotations() {
fi
case "${KATA_HYPERVISOR}" in
qemu-coco-dev | qemu-snp | qemu-tdx | qemu-coco-dev-runtime-rs)
qemu-coco-dev | qemu-snp | qemu-snp-runtime-rs | qemu-tdx | qemu-coco-dev-runtime-rs)
info "Add runtime handler annotations for ${KATA_HYPERVISOR}"
local handler_value="kata-${KATA_HYPERVISOR}"
for K8S_TEST_YAML in runtimeclass_workloads_work/*.yaml

View File

@@ -82,7 +82,7 @@ auto_generate_policy_enabled() {
is_coco_platform() {
case "${KATA_HYPERVISOR}" in
"qemu-tdx"|"qemu-snp"|"qemu-coco-dev"|"qemu-coco-dev-runtime-rs"|"qemu-nvidia-gpu-tdx"|"qemu-nvidia-gpu-snp")
"qemu-tdx"|"qemu-snp"|"qemu-snp-runtime-rs"|"qemu-coco-dev"|"qemu-coco-dev-runtime-rs"|"qemu-nvidia-gpu-tdx"|"qemu-nvidia-gpu-snp")
return 0
;;
*)
@@ -148,9 +148,7 @@ install_genpolicy_drop_ins() {
# 20-* OCI version overlay
if [[ "${KATA_HOST_OS:-}" == "cbl-mariner" ]]; then
cp "${examples_dir}/20-oci-1.2.0-drop-in.json" "${settings_d}/"
elif is_k3s_or_rke2; then
cp "${examples_dir}/20-oci-1.2.1-drop-in.json" "${settings_d}/"
elif is_nvidia_gpu_platform || [[ "${KATA_HYPERVISOR}" == "qemu-snp" ]] || [[ "${KATA_HYPERVISOR}" == "qemu-tdx" ]] || [[ -n "${CONTAINER_ENGINE_VERSION:-}" ]]; then
elif is_k3s_or_rke2 || is_nvidia_gpu_platform || [[ "${KATA_HYPERVISOR}" == "qemu-snp" ]] || [[ "${KATA_HYPERVISOR}" == "qemu-snp-runtime-rs" ]] || [[ "${KATA_HYPERVISOR}" == "qemu-tdx" ]] || [[ -n "${CONTAINER_ENGINE_VERSION:-}" ]]; then
cp "${examples_dir}/20-oci-1.3.0-drop-in.json" "${settings_d}/"
fi
@@ -342,7 +340,7 @@ hard_coded_policy_tests_enabled() {
# CI is testing hard-coded policies just on a the platforms listed here. Outside of CI,
# users can enable testing of the same policies (plus the auto-generated policies) by
# specifying AUTO_GENERATE_POLICY=yes.
local -r enabled_hypervisors=("qemu-coco-dev" "qemu-snp" "qemu-tdx" "qemu-coco-dev-runtime-rs")
local -r enabled_hypervisors=("qemu-coco-dev" "qemu-snp" "qemu-snp-runtime-rs" "qemu-tdx" "qemu-coco-dev-runtime-rs")
for enabled_hypervisor in "${enabled_hypervisors[@]}"
do
if [[ "${enabled_hypervisor}" == "${KATA_HYPERVISOR}" ]]; then

View File

@@ -7,17 +7,17 @@
FROM golang:1.24-alpine AS nydus-binary-downloader
# Keep the version here aligned with "ndyus-snapshotter.version"
# in versions.yaml
ARG NYDUS_SNAPSHOTTER_VERSION=v0.15.13
ARG NYDUS_SNAPSHOTTER_REPO=https://github.com/containerd/nydus-snapshotter
COPY versions.yaml /tmp/versions.yaml
RUN \
set -e && \
apk add --no-cache curl yq-go && \
NYDUS_SNAPSHOTTER_VERSION="$(yq eval -e '.externals.nydus-snapshotter.version | explode(.)' /tmp/versions.yaml)" && \
NYDUS_SNAPSHOTTER_REPO="$(yq eval -e '.externals.nydus-snapshotter.url | explode(.)' /tmp/versions.yaml)" && \
mkdir -p /opt/nydus-snapshotter && \
ARCH="$(uname -m)" && \
if [ "${ARCH}" = "x86_64" ]; then ARCH=amd64 ; fi && \
if [ "${ARCH}" = "aarch64" ]; then ARCH=arm64; fi && \
apk add --no-cache curl && \
curl -fOL --progress-bar "${NYDUS_SNAPSHOTTER_REPO}/releases/download/${NYDUS_SNAPSHOTTER_VERSION}/nydus-snapshotter-${NYDUS_SNAPSHOTTER_VERSION}-linux-${ARCH}.tar.gz" && \
tar xvzpf "nydus-snapshotter-${NYDUS_SNAPSHOTTER_VERSION}-linux-${ARCH}.tar.gz" -C /opt/nydus-snapshotter && \
rm "nydus-snapshotter-${NYDUS_SNAPSHOTTER_VERSION}-linux-${ARCH}.tar.gz"
@@ -49,10 +49,13 @@ RUN \
apt-get clean && rm -rf /var/lib/apt/lists/ && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain ${RUST_TOOLCHAIN}
WORKDIR /kata-deploy
# Build from the repository root so kata-deploy uses the root Cargo workspace:
# docker build -f tools/packaging/kata-deploy/Dockerfile .
WORKDIR /kata
# Copy standalone binary project
COPY binary /kata-deploy
COPY Cargo.toml Cargo.lock ./
COPY src ./src
COPY tools/packaging/kata-deploy/binary ./tools/packaging/kata-deploy/binary
# Install target and run tests based on architecture
# - AMD64/arm64: use musl for fully static binaries
@@ -93,23 +96,23 @@ RUN \
RUN \
rust_target="$(cat /tmp/rust_target)"; \
echo "Running binary tests with target ${rust_target}..." && \
RUSTFLAGS="-D warnings" cargo test --target "${rust_target}" -- --test-threads=1 && \
RUSTFLAGS="-D warnings" cargo test -p kata-deploy --target "${rust_target}" -- --test-threads=1 && \
echo "All tests passed!"
RUN \
rust_target="$(cat /tmp/rust_target)"; \
echo "Building kata-deploy binary for ${rust_target}..." && \
RUSTFLAGS="-D warnings" cargo build --release --target "${rust_target}" && \
RUSTFLAGS="-D warnings" cargo build --release -p kata-deploy --target "${rust_target}" && \
mkdir -p /kata-deploy/bin && \
cp "/kata-deploy/target/${rust_target}/release/kata-deploy" /kata-deploy/bin/kata-deploy && \
cp "/kata/target/${rust_target}/release/kata-deploy" /kata-deploy/bin/kata-deploy && \
echo "Cleaning up build artifacts to save disk space..." && \
rm -rf /kata-deploy/target && \
rm -rf /kata/target && \
cargo clean
#### Extract kata artifacts
FROM alpine:3.22 AS artifact-extractor
ARG KATA_ARTIFACTS=kata-static.tar.zst
ARG KATA_ARTIFACTS=tools/packaging/kata-deploy/kata-static.tar.zst
ARG DESTINATION=/opt/kata-artifacts
COPY ${KATA_ARTIFACTS} /tmp/
@@ -222,11 +225,11 @@ COPY --from=runtime-assembler /output/lib/ /lib/
COPY --from=runtime-assembler /output/lib64/ /lib64/
# Copy nydus snapshotter
COPY nydus-snapshotter ${DESTINATION}/nydus-snapshotter
COPY tools/packaging/kata-deploy/nydus-snapshotter ${DESTINATION}/nydus-snapshotter
COPY --from=nydus-binary-downloader /opt/nydus-snapshotter/bin/containerd-nydus-grpc ${DESTINATION}/nydus-snapshotter/
COPY --from=nydus-binary-downloader /opt/nydus-snapshotter/bin/nydus-overlayfs ${DESTINATION}/nydus-snapshotter/
# Copy runtimeclasses and node-feature-rules
COPY node-feature-rules ${DESTINATION}/node-feature-rules
COPY tools/packaging/kata-deploy/node-feature-rules ${DESTINATION}/node-feature-rules
ENTRYPOINT ["/usr/bin/kata-deploy"]

View File

@@ -1,58 +1,38 @@
[package]
name = "kata-deploy"
version = "0.1.0"
authors.workspace = true
edition = "2021"
license.workspace = true
rust-version = "1.90.0"
authors = ["The Kata Containers community <kata-dev@lists.katacontainers.io>"]
license = "Apache-2.0"
[[bin]]
name = "kata-deploy"
path = "src/main.rs"
[dependencies]
# Error handling
anyhow = "1.0"
# Logging
log = "0.4"
anyhow.workspace = true
clap.workspace = true
env_logger = "0.10"
# Command line parsing
clap = { version = "4.5", features = ["derive"] }
# TOML parsing and manipulation
toml_edit = "0.22"
# YAML parsing and manipulation
serde_yaml = "0.9"
# Kubernetes API client
kube = { version = "2.0", features = ["runtime", "derive"] }
k8s-openapi = { version = "0.26", default-features = false, features = [
"v1_33",
] }
# System operations (using nsenter command instead of syscalls)
libc = "0.2"
# JSON serialization
serde_json = "1.0"
# File operations
walkdir = "2"
# String manipulation
regex = "1.10"
# Async runtime (required by kube-rs and for async main)
tokio = { version = "1.38", features = [
kube = { version = "2.0", features = ["runtime", "derive"] }
libc.workspace = true
log.workspace = true
regex.workspace = true
serde_json.workspace = true
serde_yaml = "0.9"
tokio = { workspace = true, features = [
"rt-multi-thread",
"macros",
"signal",
"time",
] }
toml_edit = "0.22"
walkdir = "2"
[dev-dependencies]
tempfile = "3.8"
rstest = "0.18"
rstest.workspace = true
serial_test.workspace = true
tempfile.workspace = true

View File

@@ -757,18 +757,15 @@ fn get_arch_var_or_base(base_name: &str, arch: &str) -> Option<String> {
mod tests {
//! Tests for configuration parsing and validation.
//!
//! IMPORTANT: All tests in this crate MUST be run serially (--test-threads=1)
//! because they manipulate shared environment variables. Running tests in parallel
//! will cause race conditions and test failures.
//!
//! Use: cargo test --bin kata-deploy -- --test-threads=1
//! Tests that touch environment variables use `serial_test::serial` so they do not run
//! in parallel within this process. For extra isolation you can still use
//! `cargo test -p kata-deploy config::tests -- --test-threads=1`.
use super::*;
use rstest::rstest;
use serial_test::serial;
// NOTE: These tests modify environment variables which are process-global.
// Run with: cargo test config::tests -- --test-threads=1
// to ensure proper test isolation.
// NOTE: Env-var tests use #[serial] (see above) for safe parallel execution with other modules.
/// Helper to clean up common environment variables used in tests
fn cleanup_env_vars() {
@@ -867,6 +864,7 @@ mod tests {
);
}
#[serial]
#[test]
fn test_get_arch() {
let arch = get_arch().unwrap();
@@ -874,6 +872,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_get_arch_var() {
std::env::set_var("SHIMS_X86_64", "test1 test2");
@@ -887,10 +886,12 @@ mod tests {
#[rstest]
#[case(false, "config.toml.d")]
#[case(true, "config-v3.toml.d")]
#[serial]
fn test_k3s_rke2_drop_in_dir_name(#[case] use_v3: bool, #[case] expected: &str) {
assert_eq!(k3s_rke2_drop_in_dir_name(use_v3), expected);
}
#[serial]
#[test]
fn test_k3s_rke2_rendered_config_path() {
assert_eq!(k3s_rke2_rendered_config_path(), "/etc/containerd/config.toml");
@@ -905,6 +906,7 @@ mod tests {
#[case("version = 2\n", false, false)]
#[case("imports = [\"/path/config-v3.toml.d/*.toml\"]", true, true)]
#[case("imports = [\"/path/config.toml.d/*.toml\"]", true, false)]
#[serial]
fn test_k3s_rke2_rendered_has_import(
#[case] content: &str,
#[case] use_v3: bool,
@@ -913,6 +915,7 @@ mod tests {
assert_eq!(k3s_rke2_rendered_has_import(content, use_v3), expected);
}
#[serial]
#[test]
fn test_multi_install_suffix_not_set() {
setup_minimal_env();
@@ -929,6 +932,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_multi_install_suffix_with_value() {
setup_minimal_env();
@@ -950,6 +954,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_multi_install_suffix_different_values() {
let suffixes = ["staging", "prod", "v2", "test123"];
@@ -970,6 +975,7 @@ mod tests {
}
}
#[serial]
#[test]
fn test_multi_install_prefix_and_suffix() {
setup_minimal_env();
@@ -988,6 +994,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_empty_shims_no_custom_runtimes() {
setup_minimal_env();
@@ -1013,6 +1020,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_default_shim_not_in_shims() {
setup_minimal_env();
@@ -1025,6 +1033,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_hypervisor_annotation_invalid_shim() {
setup_minimal_env();
@@ -1041,6 +1050,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_agent_https_proxy_invalid_shim() {
setup_minimal_env();
@@ -1057,6 +1067,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_snapshotter_mapping_invalid_shim() {
setup_minimal_env();
@@ -1067,6 +1078,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_pull_type_mapping_invalid_shim() {
setup_minimal_env();
@@ -1077,6 +1089,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_force_guest_pull_invalid_shim() {
setup_minimal_env();
@@ -1087,6 +1100,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_validate_success() {
setup_minimal_env();
@@ -1106,6 +1120,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_missing_node_name_fails() {
cleanup_env_vars();
@@ -1116,6 +1131,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_empty_node_name_fails() {
setup_minimal_env();
@@ -1125,6 +1141,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_empty_default_shim_fails() {
setup_minimal_env();
@@ -1137,6 +1154,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_whitespace_only_default_shim_fails() {
setup_minimal_env();
@@ -1147,6 +1165,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_whitespace_only_shims_fails() {
setup_minimal_env();
@@ -1156,6 +1175,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_agent_no_proxy_invalid_shim() {
setup_minimal_env();
@@ -1166,6 +1186,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_multi_install_suffix_empty_treated_as_none() {
setup_minimal_env();
@@ -1177,6 +1198,7 @@ mod tests {
cleanup_env_vars();
}
#[serial]
#[test]
fn test_arch_specific_all_variables() {
// Test ALL architecture-specific variables work without base variables

View File

@@ -41,7 +41,7 @@ updateStrategy:
debug: false
snapshotter:
setup: [] # ["nydus", "erofs"] or []
setup: ["nydus"] # ["nydus", "erofs"] or []
# Shim configuration
# By default (disableAll: false), all shims with enabled: ~ (null) are enabled.
@@ -153,8 +153,8 @@ shims:
- amd64
allowedHypervisorAnnotations: []
containerd:
snapshotter: ""
forceGuestPull: true
snapshotter: "nydus"
forceGuestPull: false
crio:
guestPull: true
agent:
@@ -176,8 +176,8 @@ shims:
- amd64
allowedHypervisorAnnotations: []
containerd:
snapshotter: ""
forceGuestPull: true
snapshotter: "nydus"
forceGuestPull: false
crio:
guestPull: true
agent:

View File

@@ -11,25 +11,48 @@ set -o nounset
set -o pipefail
set -o errtrace
KATA_DEPLOY_DIR="`dirname ${0}`/../../kata-deploy"
SCRIPT_DIR="$(cd "$(dirname "${0}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/../../../.." && pwd)"
KATA_DEPLOY_DIR="${REPO_ROOT}/tools/packaging/kata-deploy"
STAGED_ARTIFACT="${KATA_DEPLOY_DIR}/kata-static.tar.zst"
KATA_DEPLOY_ARTIFACT="${1:-"kata-static.tar.zst"}"
REGISTRY="${2:-"quay.io/kata-containers/kata-deploy"}"
TAG="${3:-}"
echo "Copying ${KATA_DEPLOY_ARTIFACT} to ${KATA_DEPLOY_DIR}"
cp ${KATA_DEPLOY_ARTIFACT} ${KATA_DEPLOY_DIR}
# Only remove a staged copy we created (skip when source is already the staged path).
REMOVE_STAGED_ON_EXIT=false
cleanup() {
if [ "${REMOVE_STAGED_ON_EXIT}" = true ]; then
rm -f "${STAGED_ARTIFACT}"
fi
}
trap cleanup EXIT
pushd ${KATA_DEPLOY_DIR}
src_rp="$(realpath -e "${KATA_DEPLOY_ARTIFACT}" 2>/dev/null || true)"
dest_rp="$(realpath -e "${STAGED_ARTIFACT}" 2>/dev/null || true)"
if [ -n "${src_rp}" ] && [ -n "${dest_rp}" ] && [ "${src_rp}" = "${dest_rp}" ]; then
echo "Artifact already at staged path ${STAGED_ARTIFACT}; skipping copy"
else
echo "Copying ${KATA_DEPLOY_ARTIFACT} to ${STAGED_ARTIFACT}"
cp "${KATA_DEPLOY_ARTIFACT}" "${STAGED_ARTIFACT}"
REMOVE_STAGED_ON_EXIT=true
fi
pushd "${REPO_ROOT}"
arch=$(uname -m)
[ "$arch" = "x86_64" ] && arch="amd64"
[ "$arch" = "aarch64" ] && arch="arm64"
# Disable provenance and SBOM so each tag is a single image manifest. quay.io rejects
# pushing multi-arch manifest lists that include attestation manifests ("manifest invalid").
PLATFORM="linux/${arch}"
IMAGE_TAG="${REGISTRY}:kata-containers-$(git rev-parse HEAD)-${arch}"
IMAGE_TAG="${REGISTRY}:kata-containers-$(git -C "${REPO_ROOT}" rev-parse HEAD)-${arch}"
DOCKERFILE="${REPO_ROOT}/tools/packaging/kata-deploy/Dockerfile"
echo "Building the image"
docker buildx build --platform "${PLATFORM}" --provenance false --sbom false \
-f "${DOCKERFILE}" \
--tag "${IMAGE_TAG}" --push .
if [ -n "${TAG}" ]; then
@@ -37,6 +60,7 @@ if [ -n "${TAG}" ]; then
echo "Building the ${ADDITIONAL_TAG} image"
docker buildx build --platform "${PLATFORM}" --provenance false --sbom false \
-f "${DOCKERFILE}" \
--tag "${ADDITIONAL_TAG}" --push .
fi

View File

@@ -383,8 +383,6 @@ externals:
url: "https://github.com/dragonflyoss/image-service"
version: "v2.2.3"
# Keep the version here aligned with the NYDUS_SNAPSHOTTER_VERSION
# on tools/packaging/kata-deploy/Dockerfile
nydus-snapshotter:
description: "Snapshotter for Nydus image acceleration service"
url: "https://github.com/containerd/nydus-snapshotter"