So attestation-agent and others have a version including the ttrpc bump
to v0.8.4, allowing us to use the latest LTS kernel.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
We've been appending to the wrong variable for quite some time, it
seems, leading to not actually regenerating the rootfs when needed.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Set CONFIG_BLK_DEV_WRITE_MOUNTED=y to restore previous kernel behaviour.
Kernel v6.8+ will by default block buffer writes to block devices mounted by filesystems.
This unfortunately is what we need to use mounted loop devices needed by some teams
to build OSIs and as an overlay backing store.
More info on this config item [here](https://cateee.net/lkddb/web-lkddb/BLK_DEV_WRITE_MOUNTED.html)
Fixes: #10808
Signed-off-by: Simon Kaegi <simon.kaegi@gmail.com>
Run:
```
cargo update -p cookie-store
cargo update -p publicsuffix
```
to update the version of idna and resolve CVE-2024-12224
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Removed a rogue printf and updated the logging to say
that we're waiting for CDI spec(s) to be generated rather
than saying there is an error, it's not we have a timeout
after that it is an error.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
With the create_container_timeout the dial_timeout is lest important.
Add the custom timeout for GPUs in create_container_timeout
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
The tags created automatically for published Github releases
are probably not annotated, so by simply running `git describe` we are
not getting the correct tag. Use a `git describe --tags` to allow git
to look at all tags, not just annotated ones.
Signed-off-by: Anastassios Nanos <ananos@nubificus.co.uk>
AgentConfig now has the cdi_timeout from the kernel
cmdline, update the proper function signature and use
it in the for loop.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Some systems like a DGX where we have 8 H100 or 8 H800 GPUs
need some extended time to be initialized. We need to make
sure we can configure CDI timeout, to enable even systems with 16 GPUs.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Phase 1 of Issue #10840
AMD has deprecated SEV support on
Kata Containers, and going forward,
SNP will be the only AMD feature
supported. As a first step in this
deprecation process, we are removing
the SEV CI workflow from the test suite
to unblock the CI.
Will be adding future commits to
remove redundant SEV code paths.
Signed-Off-By: Adithya Krishnan Kannan <AdithyaKrishnan.Kannan@amd.com>
The block volume test has failed on 10/10 nightlies
and all the PRs I've seen, so skip it until it can be assessed.
See #10873
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Based on the guidance from @Xynnn007 in #10851
> The new version of image-rs will do attestation once
ClientBuilder.build().await() is called, while the old version
will do so lazily the first image pull request comes.
Looks like it's called in rpc::start() in kata-agent, when
I'm afraid the network hasn't been initialized yet.
> I am not sure if the guest network is prepared after
the DNS is configured (in create_sandbox),
if so we can move (the init_image_service) right after that.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
As this brings in the commit bumping ttrpc to 0.8.4, which fixes
connection issues with kernel 6.12.9+.
As image-rs has a new builder pattern and several of the values in the
image client config have been renamed, let's change the agent to account
for this.
Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
cgroups v2 enforces stricter delegation rules, preventing operations on
cgroups outside our ownership boundary. When running Docker-in-Docker (DinD),
processes must be attached to an "init" subcgroup within the systemd unit.
This fix detects and uses the init subcgroup when proxying process attachment.
Fixes#10733
Signed-off-by: Antoine Gaillard <antoine.gaillard@datadoghq.com>
When trying to deploy nydus on kcli locally we get the
following failure:
```
root@sh-kata-ci1:~# kubectl get pods -n nydus-system
NAMESPACE NAME READY STATUS RESTARTS AGE
nydus-system nydus-snapshotter-5kdqs 0/1 CrashLoopBackOff 4 (84s ago) 7m29s
```
Digging into this I found that the nydus-snapshotter service
is failing with:
```
ubuntu@kata-k8s-worker-0:~$ journalctl -u nydus-snapshotter.service
-- Logs begin at Wed 2025-02-12 15:06:08 UTC, end at Wed 2025-02-12 15:20:27 UTC. --
Feb 12 15:10:39 kata-k8s-worker-0 systemd[1]: Started nydus snapshotter.
Feb 12 15:10:39 kata-k8s-worker-0 containerd-nydus-grpc[6349]: /usr/local/bin/containerd-nydus-grpc:
/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required b>
Feb 12 15:10:39 kata-k8s-worker-0 containerd-nydus-grpc[6349]: /usr/local/bin/containerd-nydus-grpc:
/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required b>
Feb 12 15:10:39 kata-k8s-worker-0 systemd[1]: nydus-snapshotter.service: Main process exited, code=exited, status=1/FAILURE
```
I think this is because 20.04 has version:
```
ubuntu@kata-k8s-worker-0:~$ ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.16) 2.31
```
so it's too old for the nydus snapshotter.
Also 20.04 is EoL soon, so bumping is better.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Some problem hidden in `dbs` crates are revealed after making these
crates workspace components, fix according to `cargo clippy` suggests.
Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
Peer pods have a linux namespace of type network. We want to make sure that all
container in the same pod use the same namespace. Therefore, we add the first
namespace path to the state and check all other requests against that.
This commit also adds the corresponding integration test in the policy crate
showcasing the benefit of having rust integration tests for the policy.
Signed-off-by: Leonard Cohnen <lc@edgeless.systems>
The generated rego policies for `CreateContainerRequest` are stateful and that
state is handled in the policy crate. We use this policy crate in the
genpolicy integration test to be able to test if those state changes are
handled correctly without spinning up an agent or even a cluster.
This also allows to easily test on a e.g., CreateContainerRequest level
instead of relying on changing the yaml that is applied to a cluster.
Signed-off-by: Leonard Cohnen <lc@edgeless.systems>
This commit allows to programmatically invoke genpolicy. This allows for other
rust tools that don't want to consume genpolicy as binary to generate policies.
One such use-case is the policy integration test implemented in the following
commits.
Signed-off-by: Leonard Cohnen <lc@edgeless.systems>
The policy module augments the policy generated with genpolicy by keeping and
providing state to each invocation.
Therefore, it is not sufficient anymore to test the passing of requests in
the genpolicy crate.
Since in Rust, integration tests cannot call functions that are not exposed
publicly, this commit factors out the policy module of the agent into its
own crate and exposes the necessary functions to be consumed by the agent
and an integration tests. The integration test itself is implemented in the
following commits.
Signed-off-by: Leonard Cohnen <lc@edgeless.systems>
the latest containerd had an issue for its e2e test, thus we should do
the following fix to workaround this issue. For much info about this issue,
please see:
https://github.com/containerd/containerd/pull/11240
Once this pr was merged and release new version, we can remove
this workaround.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
A test case is added based on the intergrated cri-containerd case.
The difference between cri containerd integrated testcase and sandbox
api testcase is the "sandboxer" setting in the sandbox runtime handler.
If the "sandboxer" is set to "" or "podsandbox", then containerd will
use the legacy shimv2 api, and if the "sandboxer" is set to "shim", then
it will use the sandbox api to launch the pod.
In addition, add a containerd v2.0.0 version. Because containerd officially
supports the sandbox api from version 2.0.0.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
add and resiger the sandbox api service, thus runtime-rs
can deal with the sandbox api rpc call from the containerd.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
For Kata-Containers, we add SandboxService for these new calls alongside
the existing
TaskService, including processing requests and replies, and properly
calling
VirtSandbox's interfaces. By splitting the start logic of the sandbox,
virt_container
is compatible with calls from the SandboxService and TaskService. In
addition, we modify
the processing of resource configuration to solve the problem that
SandboxService does not
have a spec file when creating a pod.
Sandbox api can be supported from containerd 1.7. But there's a
difference from container 2.0.
To enbale it from 2.0, you can support the sandbox api for a specific
runtime by adding:
sandboxer = "shim", take kata runtime as an example:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
sandboxer = "shim"
privileged_without_host_devices = true
pod_annotations = ["io.katacontainers.*"]
For container version 1.7, you can enable it by:
1: add env ENABLE_CRI_SANDBOXES=true
2: add sandbox_mode = "shim" to runtime config.
Acknowledgement
This work was based on @wllenyj's POC code:
(f5b62a2d7c)
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
Signed-off-by: wllenyj <wllenyj@linux.alibaba.com>
For the processing of init sandbox, the init of task
api has some more special processing procedures than
the init of sandbox api, so these two types of init
are separated here.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
When start the sandbox, the sandbox id would be passed from the
shim command line, and it only need to get the containerd id from
oci spec when starting the pod container instead of the pod sandbox.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
When the sandbox api was enabled, the pause container would
be removed and sandbox start api only pass an empty bundle
directory, which means there's no oci spec file under it, thus
the cgroup config couldn't get the cgroup path from pause container's
oci spec. So we should set a default cgroup path for sandbox api
case.
In the future, we can promote containerd to pass the cgroup path during
the sandbox start phase.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
Remove block_device_cache_direct from config of fc in runtime-rs because
fc doesn't support this config.
Fixes: #10849
Signed-off-by: Hui Zhu <teawater@antgroup.com>
Add block_device_cache_direct to config of ch and dragonball in
runtime-rs because they support this config.
Fixes: #10849
Signed-off-by: Hui Zhu <teawater@antgroup.com>
This commit change config in CloudHypervisorInner to normal
HypervisorConfig to decrease the change of its type.
Fixes: #10849
Signed-off-by: Hui Zhu <teawater@antgroup.com>
Otherwise we may end up simply unpacking kata-containers specific
binaries into the same location that system ones are needed, leading to
a broken system (most likely what happened with the metrics CI, and also
what's happening with the GHA runners).
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
We've been hitting issues with the CentOS 9 Stream machine, which Intel
doesn't have cycles to debug.
After raising this up in the Confidential Containers community meeting
we got the green light from Red Hat (Ariel Adam) to just disable the CI
based on CentOS 9 Stream for now.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
IBM SE ensures to make initrd measured by genprotimg and verified by ultravisor.
Let's not build the measured rootf on s390x.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This is needed in order to avoid agent build issues, such as:
```
error[E0658]: use of unstable library feature 'lazy_cell'
--> /home/ansible/.cargo/git/checkouts/guest-components-1e54b222ad8d9630/514c561/ocicrypt-rs/src/lib.rs:10:5
|
10 | use std::sync::LazyLock;
| ^^^^^^^^^^^^^^^^^^^
|
= note: see issue #109736 <https://github.com/rust-lang/rust/issues/109736> for more information
```
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
As we want to make sure a new builder image is generated if the rust
version is bumped.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Due to the way that multi-arch support is done, on various platforms
we will get a clippy error:
```
error: this expression always evaluates to false
```
which might not be true on those other platforms, so
allow this code pattern to suppress the clippy error
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
k8s-policy-job is modeled after the older k8s-job, and it appears
that both of them fail occasionally on coco-dev.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Preparing to install nydus permanently on the AMD node,
so disabling deploy and delete command for SNP and SEV.
Signed-off-by: Arvind Kumar <arvinkum@amd.com>
We introduced extratarballs with a make target. The CI
currently only uploads tarballs that are listed in the matrix.
The NV kernel builds a headers package which needs to be uploaded
as well.
The get-artifacts has a glob to download all artifacts hence we
should be good.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
So it avoids us hitting
```
error[E0282]: type annotations needed for `Box<_>`
--> /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/time-0.3.31/src/format_description/parse/mod.rs:83:9
|
83 | let items = format_items
| ^^^^^
...
86 | Ok(items.into())
| ---- type must be known at this point
|
help: consider giving `items` an explicit type, where the placeholders `_` are specified
|
83 | let items: Box<_> = format_items
| ++++++++
```
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
On powerpc64le platform the ip neigh command has
a trailing space after the state, so the test is failing e.g.
```
assertion `left == right` failed
left: "169.254.1.1 lladdr 6a:92:3a:59:70:aa PERMANENT \n"
right: "169.254.1.1 lladdr 6a:92:3a:59:70:aa PERMANENT\n"
```
Trim the whitespace to make the test pass on all platforms
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
`host_is_vmcontainer_capable` is required, but wasn't
implemented for powerpc64, so copy the aarch64 approach
@Amulyam24
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
In #7236 the guest protection code was moved to kata-sys-utils,
but some of it was left behind, and the adjustment to the new
location wasn't completed, so the powerpc64 code doesn't
build now we've fixed the cfg to test it.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Some of the Kernel structs have `#[allow(dead_code)]`
but not all and this results in the clippy error:
```
error: fields `name` and `value` are never read
```
so complete the job started before to remove the error.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Clippy errors with:
```
error: field `driver` is never read
--> crates/resource/src/network/utils/link/driver_info.rs:77:9
|
76 | pub struct DriverInfo {
| ---------- field in this struct
77 | pub driver: String,
| ^^^^^^
```
We set this, but never read it, so clippy is correct,
but I'm not sure if it's useful for logging, or other purposes,
so I'll allow it for now.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Clippy fails with:
```
warning: unexpected `cfg` condition value: `test-mock`
--> /root/go/src/github.com/kata-containers/kata-containers/src/dragonball/src/dbs_pci/src/vfio.rs:1929:17
|
1929 | #[cfg(all(test, feature = "test-mock"))]
| ^^^^^^^^^^^^^^^^^^^^^ help: remove the condition
|
= note: no expected values for `feature`
= help: consider adding `test-mock` as a feature in `Cargo.toml`
```
So add it as an expected cfg in the linter to skip this
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Clippy fails with:
```
error: unexpected `cfg` condition value: `enable-vendor`
--> crates/hypervisor/src/device/driver/vfio.rs:180:11
|
180 | #[cfg(feature = "enable-vendor")]
| ^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: expected values for `feature` are: `ch-config`, `cloud-hypervisor`, `default`, and `dragonball`
= help: consider adding `enable-vendor` as a feature in `Cargo.toml`
```
So add it as an expected cfg in the linter to skip this
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Clippy complains about:
```
error: for loop over a `&Result`. This is more readably written as an `if let` statement
--> crates/hypervisor/src/firecracker/fc_api.rs:99:22
|
99 | for param in &kernel_params.to_string() {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
```
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Fix clippy error:
```
direct implementation of `ToString`
```
by switching to implement Display instead
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Clippy errors with:
```
error: field `0` is never read
--> crates/hypervisor/src/qemu/cmdline_generator.rs:375:25
|
375 | DeviceAlreadyExists(String), // Error when trying to add an existing device
| ------------------- ^^^^^^
```
but this is used when creating the error later, so add an allow
to ignore this warning
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Fix clippy error
```
error: usage of a legacy numeric constant
```
by swapping `std::u8::MAX` for `u8::MAX`
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Clippy errors with:
```
error: field `0` is never read
```
but the field is required for the `map_err`, so ignore this
error for now to avoid too much disruption
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
There were references to `config_manager::DeviceInfoGroup`
which doesn't exist, so I guess it means `DeviceConfigInfo`
instead, so update them
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Clippy errors with:
```
error: doc list item missing indentation
```
which I think is because the Return is between two list
items, so add a blank line to separate this into a separate
paragraph
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
clippy errors with:
```
error: initializer for `thread_local` value can be made `const`
```
so update as suggested
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Fix clippy error:
```
direct implementation of `ToString`
```
by switching to implement Display instead
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Fix clippy error
```
error: usage of a legacy numeric constant
```
by swapping `std::i32::<MIN/MAX>` for `i32::<MIN/MAX>`
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
```
error: file opened with `create`, but `truncate` behavior not defined
```
`truncate(true)` ensures the file is entirely overwritten with new data
which I believe is the behaviour we want
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
```
error: bound is defined in more than one place
```
Move Sized into the later definition of `R` & `W`
rather than defining them in two places
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
```
error: file opened with `create`, but `truncate` behavior not defined
```
`truncate(true)` ensures the file is entirely overwritten with new data
which I believe is the behaviour we want
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
```
error: field `image` is never read
--> src/registry.rs:35:9
|
34 | pub struct Container {
| --------- field in this struct
35 | pub image: String,
| ^^^^^
|
= note: `Container` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
= note: `-D dead-code` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(dead_code)]`
error: field `use_cache` is never read
--> src/utils.rs:106:9
|
105 | pub struct Config {
| ------ field in this struct
106 | pub use_cache: bool,
| ^^^^^^^^^
|
= note: `Config` has derived impls for the traits `Debug` and `Clone`, but these are intentionally ignored during dead code analysis
error: could not compile `genpolicy` (bin "genpolicy") due to 2 previous errors
```
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Based on comments from @Amulyam24 we need to use
the `target_endian = "little"` as well as target_arch = "powerpc64"
to ensure we are working on powerpc64le.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Starting with version 1.80, the Rust linter does not accept an invalid
value for `target_arch` in configuration checks:
```
Compiling kata-sys-util v0.1.0 (/home/ddd/Work/kata/kata-containers/src/libs/kata-sys-util)
error: unexpected `cfg` condition value: `powerpc64le`
--> /home/ddd/Work/kata/kata-containers/src/libs/kata-sys-util/src/protection.rs:17:34
|
17 | #[cfg(any(target_arch = "s390x", target_arch = "powerpc64le"))]
| ^^^^^^^^^^^^^^-------------
| |
| help: there is a expected value with a similar name: `"powerpc64"`
|
= note: expected values for `target_arch` are: `aarch64`, `arm`, `arm64ec`, `avr`, `bpf`, `csky`, `hexagon`, `loongarch64`, `m68k`, `mips`, `mips32r6`, `mips64`, `mips64r6`, `msp430`, `nvptx64`, `powerpc`, `powerpc64`, `riscv32`, `riscv64`, `s390x`, `sparc`, `sparc64`, `wasm32`, `wasm64`, `x86`, and `x86_64`
= note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration
= note: `-D unexpected-cfgs` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(unexpected_cfgs)]`
```
According [to GitHub user @Urgau][explain], this is a new warning
introduced in Rust 1.80, but the problem exists before. The correct
architecture name should be `powerpc64`, and the differentiation
between `powerpc64le` and `powerpc64` should use the `target_endian =
"little"` check.
[explain]: #10072 (comment)
Fixes: #10067
Signed-off-by: Christophe de Dinechin <dinechin@redhat.com>
[emlima: fix some more occurences and typos]
Signed-off-by: Emanuel Lima <emlima@redhat.com>
[stevenhorsman: fix some more occurences and typos]
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add aarch64 and x86_64 handling. Especially build the Rust
dependency with the correct rust musl target.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
In v4.44.5 of `yq`, artifacts for riscv64 are released. Update the
version used for `yq` and enable `install_yq.sh` to work on riscv64.
Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
The static_checks_versions test uses yamllint which fails with:
```
[comments] too few spaces before comment
```
many times and so makes code reviews more annoying with
all these extra messages. Other it's probably not the worse issues,
I checked the
[yaml spec](https://yaml.org/spec/1.2.2/#66-comments)
and it does say
> Comments must be separated from other tokens by white space character*s*
so it's easiest to fix it and move on.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
I've also seen cases (the qemu, crio, k0s tests) where Delete kata-deploy is still
running for this test after 2 hours, and had to be manually
cancelled, so let's try adding a 5m timeout to the kata-deploy delete to stop CI jobs hanging.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
#10714 added support for building a specific commit,
but due to the clone only having `--depth=1`, we can only
reset to a commit if it's the latest on the `main` branch,
otherwise we will get:
```
+ git clone --depth 1 --branch main https://gitlab.com/virtio-fs/virtiofsd virtiofsd
Cloning into 'virtiofsd'...
warning: redirecting to https://gitlab.com/virtio-fs/virtiofsd.git/
+ pushd virtiofsd
+ git reset --hard cecc61bca981ab42aae6ec490dfd59965e79025e
...
fatal: Could not parse object 'cecc61bca981ab42aae6ec490dfd59965e79025e'.
```
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Referenced AMD developer page for latest SEV firmware.
Instructions to point to upstream 6.11 kernel or later.
Referenced sev-utils and AMDESE fork for kernel setup.
Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
use upstream qemu in snp and nvidia snp configs.
load ovmf with bios flag on qemu cmdline instead of file.
Fixes: #10750
Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
snp standard attestation with the upstream kernel and qemu do not support extended attestation with certs.
Fixes: #10750
Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
Previously, the test for VFIO-AP coldplug only checked whether a
passthrough device was attached to the VM guest. This commit expands
the test to include a full set of zcrypttest to verify that the device
functions properly within a container.
Additionally, since containerd has been upgraded to v1.7.25 on the
test machine, it is no longer necessary to run the test via crictl.
The commit removes all related codes/files.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This commit updates the device handler to call check_ap_device()
instead of wait_for_ap_device() for VFIO-AP coldplug.
The handler now returns a SpecUpdate for passthrough devices if
the device is online (e.g., `/sys/devices/ap/card05/05.001f/online`
is set to 1).
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This commit introduces a new gRPC device type, `vfio-ap-cold`, to support
VFIO-AP coldplug. This enables the VM guest to handle passthrough devices
differently from VFIO-AP hotplug.
With this new type, the guest no longer needs to wait for events (e.g., device
addition) because the device already exists at the time the device type is checked.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Even though ociSpec.Linux.Devices is preserved when vfio_mode is VFIO,
it has not been updated correctly for coldplug scenarios. This happens
because the device info passed to the agent via CreateContainerRequest
is dropped by the Kata runtime.
This commit ensures that the device info is added to the sandbox's
device manager when vfio_mode is VFIO and coldPlugVFIO is true
(e.g., vfio-ap-cold), allowing ociSpec.Linux.Devices to be properly
updated with the device information before the container is created on
the guest.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Since we're importing some build script for nvidia and we're
setting set -u we have some unbound variables in rootfs.sh
add initialization for those.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
stack-only types are handled properly with the
parse_cmdline_param macro advancted types like
String couldn't be guarded by a guard function since
it passed the variable by value rather than reference.
Now we can have guard functions for the String type
parse_cmdline_param!(
param,
CGROUP_NO_V1,
config.cgroup_no_v1,
get_string_value,
| no_v1 | no_v1 == "all"
);
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
For AGENT_INIT=yes we do not run systemd and hence
systemd.unified_... does not mean anything to other init
systems. Providing cgroup_no_v1=all is enough to signal
other init systemd to use cgroupV2.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Since we're defaulting to AGENT_INIT=no for all the initrd/images
adapt the NV build to properly get kata-agent installed.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
While I wish we could be bumping to the very same version everywhere,
it's not possible and it's been quite a ride to get a combination of
things that work.
Let me try to describe my approach here:
* Do *NOT* stay on 20.04
* This version will be EOL'ed by April
* This version has a very old version of systemd that causes a bug
when trying to online the cpusets for guests using systemd as
init, causing then a breakage on the qemu-coco-non-tee and TDX
non-attestation set of tests
* Bump to 22.04 when possible
* This was possible for the majority of the cases, but for the
confidential initrd & confidential images for x86_64, the reason
being failures on AMD SEV CI (which I didn't debug), and a kernel
panic on the CentOS 9 Stream TDX machine
* 22.04 is being used instead of 24.04 as multistrap is simply broken
on Ubuntu 24.04, and I'd prefer to stay on an LTS release whenever
it's possible
* Bump to 24.10 for x86_64 image confidential
* This was done as we got everything working with 24.10 in the CI.
* This requires using libtdx-attest from noble (Ubuntu 24.04), as
Intel only releases their sgx stuff for LTS releases.
* Stick to 20.04 for x86_64 initrd confidential
* 24.10 caused a panic on their CI
* This is only being used by AMD so far, so they can decide when to
bump, after doing the proper testing & debug that the bump will work
as expected for them
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
We first try without passing the `--break-system-packages` argument, as
that's not supported on Ubuntu 22.04 or older, but that's required on
Ubuntu 24.04 or newer.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Otherwise a bump in the os name and / or os version would lead to the CI
using a cached artefact.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
We have gotten Ubuntu 20.04 working pretty much "by luck", as multistrap
fails the deployment, and then a hacky function was introduced to add
the proper dbus links. However, this does not scale at all, and we
should:
* Fail if multistrap fails
* I won't do this for Ubuntu 20.04 as it's working for now and soon
enough it'll be EOL
* Add better logging to ensure someone can know when multistrap fails
Below you can find the failure that we're hitting on Ubuntu 20.04:
```sh
Errors were encountered while processing:
dbus
ERR: dpkg configure reported an error.
Native mode configuration reported an error!
I: Tidying up apt cache and list data.
Multistrap system reported 1 error in /rootfs/.
I: Tidying up apt cache and list data.
```
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Right now we're hitting an interesting situation with osbuilder, where
regardless of what's being passed Ubuntu 20.04 (focal) is being used
when building the rootfs-image, as shown in the snippets of the logs
below:
```
ffidenci@tatu:~/src/upstream/kata-containers/kata-containers$ make rootfs-image-confidential-tarball
/home/ffidenci/src/upstream/kata-containers/kata-containers/tools/packaging/kata-deploy/local-build//kata-deploy-copy-libseccomp-installer.sh "agent"
make agent-tarball-build
...
make pause-image-tarball-build
...
make coco-guest-components-tarball-build
...
make kernel-confidential-tarball-build
...
make rootfs-image-confidential-tarball-build
make[1]: Entering directory '/home/ffidenci/src/upstream/kata-containers/kata-containers'
/home/ffidenci/src/upstream/kata-containers/kata-containers/tools/packaging/kata-deploy/local-build//kata-deploy-binaries-in-docker.sh --build=rootfs-image-confidential
sha256:f16c57890b0e85f6e1bbe1957926822495063bc6082a83e6ab7f7f13cabeeb93
Build kata version 3.13.0: rootfs-image-confidential
INFO: DESTDIR /home/ffidenci/src/upstream/kata-containers/kata-containers/tools/packaging/kata-deploy/local-build/build/rootfs-image-confidential/destdir
INFO: Create image
build image
~/src/upstream/kata-containers/kata-containers/tools/osbuilder ~/src/upstream/kata-containers/kata-containers/tools/packaging/kata-deploy/local-build/build/rootfs-image-confidential/builddir
INFO: Build image
INFO: image os: ubuntu
INFO: image os version: latest
Creating rootfs for ubuntu
/home/ffidenci/src/upstream/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/rootfs.sh -o 3.13.0-13f0807e9f5687d8e5e9a0f4a0a8bb57ca50d00c-dirty -r /home/ffidenci/src/upstream/kata-containers/kata-containers/tools/packaging/kata-deploy/local-build/build/rootfs-image-confidential/builddir/rootfs-image/ubuntu_rootfs ubuntu
INFO: rootfs_lib.sh file found. Loading content
~/src/upstream/kata-containers/kata-containers/tools/osbuilder/rootfs-builder/ubuntu ~/src/upstream/kata-containers/kata-containers/tools/osbuilder
~/src/upstream/kata-containers/kata-containers/tools/osbuilder
INFO: rootfs_lib.sh file found. Loading content
INFO: build directly
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [4276 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1297 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [30.9 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [4187 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4663 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1589 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [34.6 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [4463 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [55.2 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [28.6 kB]
Fetched 34.1 MB in 5s (6284 kB/s)
...
```
The reason this is happening is due to a few issues in different places:
1. IMG_OS_VERSION, passed to osbuilder, is not used anywhere and
OS_VERSION should be used instead. And we should break if OS_VERSION
is not properly passed down
2. Using UBUNTU_CODENAME is simply wrong, as it'll use whatever comes as
the base container from kata-deploy's local-build scripts, and it has
just been working by luck
Note that at the same time this commit fixes the wrong behaviour, it
would break the rootfses build as they are, this we need to set the
versions.yaml to use 20.04 were it was already using 20.04 even without
us knowing.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
As this is required as part of the osbuilder tool to be able to properly
set the repositories used when building the rootfs.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
While having variables are nice, those are more extensive to write down,
and actually confusing for tired developer eyes to read, plus we're
mixing the use of the yaml variables here and there together with not
using them for some architectures.
With the best "all or nothing" spirit, let's just make it easier for our
developers to read the versions.yaml and easily understand what's being
used.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
As the devices controller works in a different way in cgroupsv2, the
"/sys/fs/cgroup/devices/devices.list" file simply doesn't exist.
For now, let's skip the test till the test maintainer decides to
re-enable it for cgroupsv2.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
The changes done are:
* cpu/cpu.shares was replaced by cpu.weight
* The weight, according to our reference[0], is calculated by:
weight = (1 + ((request - 2) * 9999) / 262142)
* cpu/cpu.cfs_quota_us & cpu/cpu.cfs_period_us were replaced by cpu.max,
where quota and period are written together (in this order)
[0]: https://github.com/containers/crun/blob/main/crun.1.md#cgroup-v2
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This reverts commit 091ad2a1b2, in order
to ensure tests would be running with cgroupsv2 on the guest.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
In the last couple of days I've seen the blogbench
metrics write latency test on clh fail a few times because
the latency was too low, so adjust the minimum range
to tolerate quicker finishes.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The static-checks targets are `pull_request`, so
they can run the PR workflow version, so we want to
update the required-tests.yaml so that static-check
workflow changes do trigger static checks in order
to test them properly.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Now we have the build-assets running on the gh-hosted
runners, try the same approach for the static-checks
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
I've noticed the following error when running the tests with SEV:
```
2025-01-21T17:10:28.7999896Z # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2025-01-21T17:10:28.8000614Z # @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
2025-01-21T17:10:28.8001217Z # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
2025-01-21T17:10:28.8001857Z # IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
2025-01-21T17:10:28.8003009Z # Someone could be eavesdropping on you right now (man-in-the-middle attack)!
2025-01-21T17:10:28.8003348Z # It is also possible that a host key has just been changed.
2025-01-21T17:10:28.8004422Z # The fingerprint for the ED25519 key sent by the remote host is
2025-01-21T17:10:28.8005019Z # SHA256:x7wF8zI+LLyiwphzmUhqY12lrGY4gs5qNCD81f1Cn1E.
2025-01-21T17:10:28.8005459Z # Please contact your system administrator.
2025-01-21T17:10:28.8006734Z # Add correct host key in /home/kata/.ssh/known_hosts to get rid of this message.
2025-01-21T17:10:28.8007031Z # Offending ED25519 key in /home/kata/.ssh/known_hosts:178
2025-01-21T17:10:28.8007254Z # remove with:
2025-01-21T17:10:28.8008172Z # ssh-keygen -f "/home/kata/.ssh/known_hosts" -R "10.244.0.71"
```
And this was causing a failure to ssh into the confidential pod.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Relying on dmesg is really not ideal, as we may lose important info,
mainly those which happen very early in the boot, depending on the size
of kernel ring buffer.
So, for this specific test, let's increase the kernel ring buffer, by
default, to 4M.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Let's make sure that we don't use Kata Containers' agent as init for the
Confidential related rootfses*, as we don't want to increase the agent's
complexity for no reason ... mainly when we can rely on a proper init
system.
*:
- images already used systemd as init
- initrds are now using systemd as init
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Bumps the go_modules group with 1 update in the /src/runtime directory: [golang.org/x/net](https://github.com/golang/net).
Bumps the go_modules group with 1 update in the /src/tools/csi-kata-directvolume directory: [golang.org/x/net](https://github.com/golang/net).
Bumps the go_modules group with 1 update in the /tools/testing/kata-webhook directory: [golang.org/x/net](https://github.com/golang/net).
Updates `golang.org/x/net` from 0.25.0 to 0.33.0
- [Commits](https://github.com/golang/net/compare/v0.25.0...v0.33.0)
Updates `golang.org/x/net` from 0.23.0 to 0.33.0
- [Commits](https://github.com/golang/net/compare/v0.25.0...v0.33.0)
Updates `golang.org/x/net` from 0.23.0 to 0.33.0
- [Commits](https://github.com/golang/net/compare/v0.25.0...v0.33.0)
---
updated-dependencies:
- dependency-name: golang.org/x/net
dependency-type: indirect
dependency-group: go_modules
- dependency-name: golang.org/x/net
dependency-type: direct:production
dependency-group: go_modules
- dependency-name: golang.org/x/net
dependency-type: indirect
dependency-group: go_modules
...
Signed-off-by: dependabot[bot] <support@github.com>
When the agent is run as the init process cgroupfs is being
setup. In the case of cgroupsV1 we needed to enable the memory hiearchy
this is now per default enabled in cgroupsV2. Additionally the file
/sys/fs/cgroup/memory/memory.use_hierarchy isn't even available with V2.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
The `Create AKS cluster` step in `run-k8s-tests-on-aks.yaml` is likely
to fail fail since we are trying to issue `PUT` to `aks` in a relatively
high frequency, while the `aks` end has it's limit on `bucket-size` and
`refill-rate`, documented here [1].
Use `nick-fields/retry@v3` to retry in 10 seconds after request fail,
based on observations that AKS were request 7, or 8 second delays
before retry as part of their 429 response
[1] https://learn.microsoft.com/en-us/azure/aks/quotas-skus-regions#throttling-limits-on-aks-resource-provider-apisFixes: #10772
Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
With this change, `virtiofsd` (gnu target) could be built and then to be
used with other components.
Depends: #10741Fixes: #10739
Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
While working on #10559, I realized that some parts of the codebase use
$GH_PR_NUMBER, while other parts use $PR_NUMBER.
Notably, in that PR, since I used $GH_PR_NUMBER for CoCo non-TEE tests
without realizing that TEE tests use $PR_NUMBER, the tests on that PR
fail on TEEs:
https://github.com/kata-containers/kata-containers/actions/runs/12818127344/job/35744760351?pr=10559#step:10:45
...
44 error: error parsing STDIN: error converting YAML to JSON: yaml: line 90: mapping values are not allowed in this context
...
135 image: ghcr.io/kata-containers/csi-kata-directvolume:
...
So let's unify on $GH_PR_NUMBER so that this issue doesn't repro in the
future: I replaced all instances of PR_NUMBER with GH_PR_NUMBER.
Note that since some test scripts also refer to that variable, the CI
for this PR will fail (would have also happened with the converse
substitution), hence I'm not adding the ok-to-test label and we should
force-merge this after review.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
With ubuntu 20.04 image, virtiofsd gnu target couldn't be built due to
"unsupported ISA subset z" reported by "cc".
Updating to ubuntu 22.04 image addresses this problem.
Relates: #10739
Signed-off-by: Ruoqing He <heruoqing@iscas.ac.cn>
The pyinstaller is located per default under /usr/local/bin
some prior versions were installing it to ${HOME}.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Skip logging empty lines of text from the Guest console output, if
there are any such lines.
Without this change, the Guest console log from CLH + /dev/pts/0 has
twice as many lines of text. Half of these lines are empty.
Fixes: #10737
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
On s390x, some tests for trusted storage occasionally failed due to:
```bash
etcdserver: request timed out
```
or
```bash
Internal error occurred: resource quota evaluation timed out
```
These timeouts were not observed previously on k3s but occur
sporadically on kubeadm. Importantly, they appear to be temporary
and transient, which means they can be ignored in most cases.
To address this, we introduced a new wrapper function, `retry_kubectl_apply()`,
for `kubectl create`. This function retries applying a given manifest up to 5
times if it fails due to a timeout. However, it will still catch and handle
any other errors during pod creation.
Fixes: #10651
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Move the deletion of unnecessary systemd units and files from
image_builder.sh into rootfs.sh.
The files being deleted can be applicable to other image file formats
too, not just to the rootfs-image format created by image_builder.sh.
Also, image_builder.sh was deleting these files *after* it calculated
the size of the rootfs files, thus missing out on the opportunity to
possibly create a smaller image file.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
This update addresses an issue with token verification for SE and SNP
introduced in the last update by #10541.
Bumping the project to the latest commit resolves the issue.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
The earlier implementation relied on using a specific mount-path prefix - `/sealed`
to determine that the referenced secret is a sealed secret.
However that was restrictive for certain use cases as it forced
the user to always use a specific mountpath naming convention.
This commit introduces an alternative implementation to relax the
restriction. A sealed secret can be mounted in any mount-path.
However it comes with a potential performance penality. The
implementation loops through all volume mounts and reads the file
to determine if it's a sealed secret or not.
Fixes: #10398
Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
We hit a failure with:
```
time="2025-01-09T09:55:58Z" level=warning msg="Failed Minval (0.017600 > 0.015000) for [network-iperf3]"
```
The range is very big, but in the last 3 test runs I reviewed we have got a minimum value of 0.015s
and a max value of 0.052, so there is a ~350% difference possible
so I think we need to have a wide range to make this stable.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Because az client restricts the name to be less than 64 characters. In
some cases (e.g. KATA_HYPERVISOR=qemu-runtime-rs) the generated name
will exceed the limit. This changed the function to shorten the name:
* SHA1 is computed from metadata then compound the cluster's name
* metadata as plain-text are passed as --tags
Fixes: #9850
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The bump to kernel 6.12 seems to have reduced the latency in
the metrics test, so increase the ranges for the minimal value,
to account for this.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Together with the bump, let's also bump the rust version needed to build
the package, with the caveat that virtiofsd doesn't actually use a
pinned version as part of their CI, so we're bumping to whatever is the
version on `alpine:rust` (which is used in their CI).
It's important to note that we're using a version which brings in one
extra patch apart from the release, as the next virtiofsd release will
happen at the end of February, 2025.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Right now we've been only building releases from virtiofsd, but we'll
need to pin a specific commit till v1.14.0 is out, thus let's add the
needed machinery to do so.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
The guest-components directory has been re-arranged slightly. Adjust the
installation path of the LUKS helper script to account for this.
Signed-off-by: Tobin Feldman-Fitzthum <tobin@ibm.com>
Trustee has some new features including a plugin backend, support for
PKCS11 resources, improvements to token verification, and adjustments to
logging, and more.
Also update guest-components to pickup improvements and keep the KBS
protocol in sync.
Signed-off-by: Tobin Feldman-Fitzthum <tobin@ibm.com>
Since we bumped to the 6.12.x LTS kernel, we've also adjusted the
aggressivity of the OOM test, which may be enough to allow us to
re-enable it for mariner.
Fixes: #8821
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Since from https://github.com/containerd/containerd/pull/9096
containerd removed cri-containerd-*.tar.gz release bundles,
thus we'd better change the tarball name to "containerd".
BTW, the containerd tarball containerd the follow files:
bin/
bin/containerd-shim
bin/ctr
bin/containerd-shim-runc-v1
bin/containerd-stress
bin/containerd
bin/containerd-shim-runc-v2
thus we should untar containerd into /usr/local directory instead of "/"
to keep align with the cri-containerd.
In addition, there's no containerd.service file,runc binary and cni-plugin
included, thus we should add a specific containerd.service file and
install install the runc binary and cni-pluginspecifically.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
Got following issue:
kata-ctl direct-volume add /kubelet/kata-direct-vol-002/directvol002
"{\"device\": \"/home/t4/teawater/coco/t.img\", \"volume-type\":
\"directvol\", \"fstype\": \"\", \"metadata\":"{}", \"options\": []}"
subsystem: kata-ctl_main
Dec 30 09:43:41.150 ERRO Os {
code: 2,
kind: NotFound,
message: "No such file or directory",
}
The reason is KATA_DIRECT_VOLUME_ROOT_PATH is not exist.
This commit create_dir_all KATA_DIRECT_VOLUME_ROOT_PATH before join_path
to handle this issue.
Fixes: #10695
Signed-off-by: Hui Zhu <teawater@antgroup.com>
Unsurprisingly now we've got passed the containerd test
hangs on the ppc64le, we are hitting others in the "Prepare the
self-hosted runner" stage, so add timeouts to all of them
to avoid CI blockages.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Add information about what required jobs are and
our initial guidelines for how jobs are eligible for being
made required, or non-required
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
`set_hypervisor_config()` and `set_passfd_listener_port()` acquire inner
lock, so that `mut` for `hypervisor` is unneeded.
Fixes: #10682
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
Fix the logic to make the test skipped on qemu-coco-dev,
rather than the opposite and update the syntax to make it
clearer as it incorrectly got written and reviewed by three
different people in it's prior form.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
In order to check that we don't accidentally overwrite
release artifacts, we should add a check if the release
name already exists and bail if it does.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Building static binaries for aarch64 requires disabling PIE
We get an GOT overflow and the OS libraries are only build with fpic
and not with fPIC which enables unlimited sized GOT tables.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
The current script logic assigns an empty string to APID and APQI
when APQN consists entirely of zeros (e.g., "00.0000").
However, this behavior is incorrect, as "00" and "0000" are valid
values and should be represented as "0".
This commit ensures that the script assigns the default string “0”
to APID and APQI if their computed values are empty.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Due to the agent-api tests requiring the agent to be deployed in the
CI by the tarball, so in the short-term lets only do this on the release
stage, so that both kata-manager works with the release and the
agent-api tests work with the other CI builds.
In the longer term we need to re-evaluate what is in our tarballs
(issue #10619), but want to unblock the tests in the short-term.
Fixes: #10630
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
With the code I originally did I think there is potentially
a case where we can get a failure due to timing of steps.
Before this change the `build-asset-shim-v2`
job could start the `get-artifacts` step and concurrently
`remove-rootfs-binary-artifacts` could run and delete the artifact
during the download and result in the error. In this commit, I
try to resolve this by making sure that the shim build waits
for the artifact deletes to complete before starting.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
We want to have the latest QEMU version available
which is as of this writing v9.1.2
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
qemu: Add new options for 9.1.2
We need to fence specific options depending on the version
and disable ones that are not needed anymore
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
qemu: Add no_patches.txt
Since we do not have any patches for this version
let's create the appropriate files.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
The new QEMU build needs python-tomli, now that we bumped Ubuntu
we can include the needed tomli package
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
We're disabling pmem support, it is heavilly broken with
Ubuntu's static build of QEMU and not needed
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
In jammy we have the liburing package available, hence
remove the source build and include the package.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
This path fixes the logic of getting the type of volume: when the type of
OCI mount is Some("none") and the options have "bind" or "rbind", the
type will be considered as "bind".
Fixes: #10642
Signed-off-by: Xuewei Niu <niuxuewei.nxw@antgroup.com>
This commit sets memory config `shared` to false in cloud hypervisor
when creating vm with shared_fs=None && hugePages = false.
Currently in runtime/virtcontainers/clh.go,the memory config shared is by default set to true.
As per the CLH memory document,
(a) shared=true is needed in case like when using virtio_fs since virtiofs daemon runs as separate process than clh.
(b) for shared_fs=none + hugespages=false, shared=false can be set to use private anonymous memory for guest (with no file backing).
(c) Another memory config thp (use transparent huge pages) is always enabled by default.
As per documentation, (b) + (c) can be used in combination.
However, with the current CLH implementation, the above combination cannot be used since shared=true is always set.
Fixes#10547
Signed-off-by: Sumedh Alok Sharma <sumsharma@microsoft.com>
> A && B || C is not if-then-else. C may run when A is true
Refactor the echo so that we can't get into a situation where
the retry of workspace delete happens if the original one was
successful
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Bump some actions that are significantly out-of-date
and out of sync with the versions used in other workflows
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The tdx runners got split into two different
runners, so we need to update the known self-hosted
runner labels
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
On PRs that update anything in the workflows directory,
add an actionlint run to validate our workflow files for errors
and hopefully catch issues earlier.
Fixes: #9646
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
So users can simply download the chart and use it accordingly without
the need to download the full repo.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Add sub command MemAgentMemcgSet and MemAgentCompactSet to agent-ctl to
configate the mem-agent inside the running kata-containers.
Fixes: #10625
Signed-off-by: Hui Zhu <teawater@antgroup.com>
Add MemAgentMemcgSet and MemAgentCompactSet to agent API to set the config of
mem-agent memcg and compact.
Fixes: #10625
Signed-off-by: Hui Zhu <teawater@antgroup.com>
mem-agent is a component designed for managing memory in Linux
environments.
Sub-feature memcg: Utilizes the MgLRU feature to monitor each cgroup's
memory usage and periodically reclaim cold memory.
Sub-feature compact: Periodically compacts memory to facilitate the
kernel's free page reporting feature, enabling the release of more idle
memory from guests.
During memory reclamation and compaction, mem-agent monitors system
pressure using Pressure Stall Information (PSI). If the system pressure
becomes too high, memory reclamation or compaction will automatically
stop.
Fixes: #10625
Signed-off-by: Hui Zhu <teawater@antgroup.com>
Add mglru, debugfs and psi to dragonball-experimental/mem_agent.conf to
support mem_agent function.
Fixes: #10625
Signed-off-by: Hui Zhu <teawater@antgroup.com>
This is super useful for development / debugging scenarios, mainly when
dealing with limited hardware availability, as this change allows
multiple people to develop into one single machine, while still using
kata-deploy.
Fixes: #10546
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
At the same time that INSTALLATION_PREFIX was added, I was working on
the helm changes to properly do the cleanup / deletion when it's
removed. However, I missed adding the INSTALLATION_PREFIX env var
there. which I'm doing now.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
- Remove default_namespace from settings
- Ensure container namespaces in a pod match each other in case no namespace is specified in the YAML
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
The tests is unstable on this platform, so skip it for now to prevent
the regular known failures covering up other issues. See #10616
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
- Fix copy-paste errors in artifact filters for arm64 and ppc64le
- Remove the trailing wildcard filter that falsely ends up removing agent-ctl
and replace with the tarball-suffix, which should exactly match the artifacts
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
With the building/publishing step for the CSI driver validated, we can
set that as a requirement for the CoCo tests.
Depends on: #10561
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The driver build recipe has a script to check the current Go version against
the go.mod version. However, the script is broken ($expected is unbound) and I
don't believe we do this for other components. On top of this, Go should be
backward-compatible. Let's keep things simple for now and we can evaluate
restoring this script in the future if need be.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This fully implements the compilation step for csi-kata-directvolume.
This component can now be built by the CI running:
$ cd tools/packaging/kata-deploy/local-build
$ make csi-kata-directvolume-tarball
A couple notes:
* When installing the binary, we rename it from directvolplugin to
csi-kata-directvolume on the fly to make it more readable.
* We add go to the tools builder Dockerfile to support building this
tool.
* I've noticed the file install_libseccomp.sh gets created by the build
process so I've added it to a .gitignore.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The devno assignment logic is repeated in 5 different places
during device addition.
To improve code maintainability and readability, this commit
introduces a standalone function, `get_devno_ccw()`,
to handle the deduplication.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Before this patch there was a mismatch between the JSON path under which
the state of the rule evaluation is set in comparison to under which
it is retrieved.
This resulted in the behavior that each time the policy was evaluated,
it thought it was the _first_ time the policy was evaluated.
This also means that the consistency check for the `sandbox_name`
was ineffective.
Signed-off-by: Leonard Cohnen <lc@edgeless.systems>
With full debug logging enabled there might be around 1,500 redials
so log just ~15 of these redials to avoid flooding the log.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
The default initrd confidential target will have a
variant=confidential we need to accomodate this
and make sure we also accomodate aaa-xxx-confidential targets.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
We've also seen the qemu metrics tests are failing due to the results
being slightly outside the max range for network-iperf3 parallel and minimum for network-iperf3 jitter tests on PRs that have no code changes,
so we've increase the bounds to not see false negatives.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
We've seen a couple of instances recently where the metrics
tests are failing due to the results being below the minimum
value by ~2%.
For tests like latency I'm not sure why values being too low would
be an issue, but I've updated the minpercent range of the failing tests
to try and get them passing.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
As the following CI job has been marked as required:
- kata-containers-ci-on-push / run-k8s-tests-on-zvsi / run-k8s-tests (devmapper, qemu, kubeadm)
we need to add it to the gatekeeper's required job list.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This reverts commit 559018554b.
As we've noticed that this is causing issues with initrd builds in the
CI.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This will help us to simply allow a new dummy build whenever a new
component is added.
As long as the format `$(call DUMMY,$@)` is followed, we should be good
to go without taking the risk of breaking the CI.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This will make our lives considerably easier when it comes to cleaning
up content added, while it's also a groundwork needed for having
multiple installations running in parallel.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
We need the publish certain artefacts for the rootfs,
like the agent, guest-components, pause bundle etc
as they are consumed in the `build-asset-rootfs` step.
However after this point they aren't needed and probably
shouldn't be included in the overall kata tarball, so delete
them once they aren't needed any more to avoid them
being included.
Fixes: #10575
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Let's actually mount the whole /etc/k0s as /etc/containerd, so we can
easily access the containerd configuration file which has the version in
it, allowing us to parse it instead of just making a guess based on
kubernetes distro being used.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
On Ubuntu 24.04, with the distro default containerd, we're already
getting:
```
$ containerd config default | grep "version = "
version = 3
```
With that in mind, let's make sure that we're ready to support this from
the next release.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
A new attribute named `devno` is added to DeviceVirtioScsi.
It will be used to specify a device number for a CCW bus type.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
A new attribute named `devno` is added to DeviceVhostUserFs.
It will be used to specify a device number for a CCW bus type.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
A new attribute named `devno` is added to VhostVsock.
It will be used to specify a device number for a CCW bus type.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
A new attribute named `devno` is added to DeviceVirtioSerial.
It will be used to specify a device number for a CCW bus type.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
A new attribute named `devno` is added to DeviceVirtioBlk.
It will be used to specify a device number for a CCW bus type.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
To explicitly specify a device number on the QEMU command line
for the following devices using the CCW transport on s390x:
- SerialDevice
- BlockDevice
- VhostUserDevice
- SCSIController
- VSOCKDevice
this commit introduces a new structure CcwSubChannel and implements
the following methods:
- add_device()
- remove_device()
- address_format_ccw()
- set_addr()
You can see the detailed explanation for each method in the comment.
This resolves the 1st part of #10573.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
let's print the also the existing result's id when printing the
information about ignoring older result id to simplify debugging.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
tha matching run_id means we're dealing with the same job but with
updated results and not with an older job. Update the results in such
case.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
This adds a new job to build and publish the CSI driver Docker image.
Of course this job will fail after we merge this PR because the CSI driver
compilation job hasn't been implemented yet. However that will be implemented
directly after in #10561.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This adds a no-op build step to compile the CSI driver. The actual compilation
will be implemented in an ulterior PR, so as to ensure we don't break the CI.
Addresses: #10560
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Build ubuntu rootfs with Docker failed with error:
`Unable to find libclang`
Fix this error by adding libclang-dev to the dependency.
Signed-off-by: Jitang Lei <leijitang@outlook.com>
We need to clean-up any created files/dirs otherwise
we cause problems on self-hosted runners. Using tempdir which
will be removed automatically.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Now we are downloading artifacts to create the rootfs
we need to ensure they are uploaded always,
even on releases
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
KinD checks for the presence of this (and other) kernel configuration
via scripts like
https://blog.hypriot.com/post/verify-kernel-container-compatibility/ or
attempts to directly use /proc/sys/kernel/keys/ without checking to see
if it exists, causing an exit when it does not see it.
Docker/it's consumers apparently expect to be able to use the kernel
keyring and it's associated syscalls from/for containers.
There aren't any known downsides to enabling this except that it would
by definition enable additional syscalls defined in
https://man7.org/linux/man-pages/man7/keyrings.7.html which are
reachable from userspace. This minimally increases the attack surface of
the Kata Kernel, but this attack surface is minimal (especially since
the kernel is most likely being executed by some kind of hypervisor) and
highly restricted compared to the utility of enabling this feature to
get further containerization compatibility.
Signed-off-by: Crypt0s <BryanHalf@gmail.com>
This will be helpful in order to increase the OS coverage (we'll be
using both Ubuntu 24.04 and CentOS 9 Stream), while also reducing the
amount spent on the tests (as one machine will only run attestation
related tests, and the other the tests that do *not* require
attestation).
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
We're doing this as, at Intel, we have two different kind of machines we
can plug into our CI. Without going much into details, only one of
those two kinds of machines will work for the attestation tests we
perform with ITA, thus in order to speed up the CI and improve test
coverage (OS wise), we're going to run different tests in different
machines.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
As discussed in the CI working group,
we are temporarily skipping the SNP CI
to unblock the remaining workflow.
Will revert after fixing the SNP runner.
Signed-Off-By: Adithya Krishnan Kannan <AdithyaKrishnan.Kannan@amd.com>
Use regorous engine's add_data method to add state to the policy.
This data can later be accessed inside rego context through the data namespace.
Support state modifications (json-patches) that may be returned as a result from policy evaluation.
Also initialize a policy engine data slice "pstate" dedicated for storing state.
Fixes#10087
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
The Clear Linux rootfs is not being tested anywhere, and it seems Intel
doesn't have the capacity to review the PRs related to this (combined
with the lack of interested from the rest of the community on reviewing
PRs that are specific to this untested rootfs).
With this in mind, I'm suggesting we drop Clear Linux support and focus
on what we can actually maintain.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This reverts commit f15e16b692, as we
don't have to do this since we're relying on the
`static_sandbox_resource_mgmt` feature, which gives us the correct
amount of memory and CPUs to be allocated.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Trustee's deployment must set the correct https_proxy as env var on the
container that will talk to the ITA / ITTS server, otherwise the kbs
service won't be able to start, causing then issues in our CI.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Signed-off-by: Krzysztof Sandowicz <krzysztof.sandowicz@intel.com>
Fedora F40 removed python3 from the base container, to avoid such issues
let's rely on the latest and greates official python container.
Fixes: #10497
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
This PR adds the get artifacts which are needed when installing kata
tools in stability workflow to avoid failures saying that artifacts
are missing.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
As discussed in the AC meeting, we don't have a maintainer,
(or users?) of runk, and the CI is unstable, so giving we can't
support it, we shouldn't waste CI cycles on it.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
As discussed on the AC call, we are lacking maintainers for the
metrics tests. As a starting point for potentially phasing them
out, we discussed starting with removing the test for stratovirt
as a non-core hypervisor and a job that is problematic in leaving
behind resources that need cleaning up.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This PR adds the install kata tools step as part of the k8s stability workflow.
To avoid the failures saying that certain kata components are not installed it.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Currently the error we are checking for is
`CreateContainerRequest timed out`, but this message
doesn't always seem to be printed to our pod log.
Try using a more general message that should be present
more reliably.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
We have an error with service name resolution with this test when using crio.
This error could not be reproduced outside of the CI for now.
Skipping it to keep the CI job running until we find a solution.
See: #10414
Signed-off-by: Julien Ropé <jrope@redhat.com>
By the moment we're testing it also with qemu-coco-dev, it becomes
easier for a developer without access to TEE to also test it locally.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
The qemu-coco-dev runtime class should be as close as possible to what
the TEEs runtime classes are doing, and this was one of the options that
ended up overlooked till now.
Shout out to Dan Mihai for noticing that!
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Most likely this was overlooked during the development / review, but
we're actually interested on the size rather than on the pagesize of the
hugepages.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
So far we were not prepared to deal with release candidates as those:
* Do not have a sha256sum in the sha256sums provided by the kernel cdn
* Come from a different URL (directly from Linus)
* Have a different suffix (.tar.gz, instead of .tar.xz)
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This doesn't change much on how we're doing things Today, but it
simplifies a lot cases that may be added later on (and will be) like
building -rc kernels.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
By doing this we can ensure this can be re-used, if needed (and it'll be
needed), for also getting the URL.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
-f forces the (re)generaton of the config when doing the setup, which
helps a lot on local development whilst not causing any harm in the CI
builds.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Let's just do a simple `sed` and **not** use the repo that became
private.
This is not a backport of https://github.com/tianocore/edk2/pull/6402,
but it's a similar approach that allows us to proceed without the need
to pick up a newer version of edk2.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This PR fixes the source to avoid duplication specially in the common.sh
script and avoid failures saying that certain script is not in the directory.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This change is motivated by a problem in peerpod's podvms. In this setup
the lifecycle of guest components is managed by systemd. The current code
skips over init steps like setting the ocicrypt-rs env and initialization
of a CDH client in this case.
To address this the launch of the processes has been isolated into its
own fn.
Signed-off-by: Magnus Kulke <magnuskulke@microsoft.com>
This reverts commit 973b8a1d8f.
As @danmihai1 points out https://github.com/bats-core/bats-core/issues/364
states that using traps in bats is error prone, so this could be the cause
of the confidential test instability we've been seeing, like it was
in the static checks, so let's try and revert this.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
At @danmihai1's suggestion add a die message in case
the call to setup_common fails, so we can see if in the test
output.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
We've seen some issues with tests not being run in
some of the Coco CI jobs (Issue #10451) and in the
envrionments that are more stable we noticed that
they had a newer version of bats installed.
Try updating the version to 1.10+ and print out
the version for debug purposes
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
We need to know if we're building a nvidia initrd or image
Additionally if we build a regular or confidential VARIANT
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
Fixes: #9478
We want to keep track of the driver versions build during initrd/image build so update the artifact_name after the fact.
Signed-off-by: Zvonko Kaiser <zkaiser@nvidia.com>
This commit introduces changes to enable testing kata-agent's container
APIs of CreateContainer/StartContainer/RemoveContainer. The changeset
include:
- using confidential-containers image-rs crate to pull/unpack/mount a
container image. Currently supports only un-authenicated registry pull
- re-factor api handlers to reduce cmdline complexity and handle
request generation logic in tool
- introduce an OCI config template for container creation
- add test case
Fixes#9707
Signed-off-by: Sumedh Alok Sharma <sumsharma@microsoft.com>
This PR updates the virtualization document by removing a url link
which is not longer valid.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
The current version of the oci-spec crate compiles RLimit structs only
for Linux and Solaris. Until this is fixed upstream, add compilation
conditions to the type converters for the affected structs.
Fixes: #10071
Signed-off-by: Markus Rudy <mr@edgeless.systems>
The parse_json_string function is specific to parsing capability strings
out of ttRPC proto definitions and does not benefit from being available
to other crates. Moving it into the protocols crate allows removing
kata-sys-util as a dependency, which in turn enables compiling the
library on darwin.
Fixes: #10071
Signed-off-by: Markus Rudy <mr@edgeless.systems>
The cgroups.rs source file was removed in
234d7bca04. With cgroups support handled
in runtime-rs, the cgroups dependency on kata-sys-util can be removed.
Signed-off-by: Markus Rudy <mr@edgeless.systems>
This commit introduces changes to use ubuntu for statically
building kata tools. In the existing CI setup, the tools
currently build only for x86_64 architecture.
It also fixes the build error seen for agent-ctl PR#10395.
Fixes#10441
Signed-off-by: Sumedh Alok Sharma <sumsharma@microsoft.com>
This PR adds missing steps in the gha run script for the kata stability
workflow.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This PR removes the kernel config repo variable at the build kernel
script as it is not used.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
The cloud-api-adaptor needs to support different types of pod VM
instance.
We needs to pass some annotations like machine_type, default_vcpus and
default_memory to prepare the VMs.
Signed-off-by: Chasing1020 <643601464@qq.com>
This patch adds the support of the remote hypervisor type for runtime-rs.
The cloud-api-adaptor needs the annotations and network namespace path
to create the VMs.
The remote hypervisor opens a UNIX domain socket specified in the config
file, and sends ttrpc requests to a external process to control sandbox
VMs.
Fixes: #10350
Signed-off-by: Chasing1020 <643601464@qq.com>
Add GPU annotations for remote hypervisor to help
with the right instance selection based on number of GPUs
and model
Signed-off-by: Pradipta Banerjee <pradipta.banerjee@gmail.com>
Last November, SUSE discontinued support for s390x, leaving k3s
on this platform stuck at k8s version 1.28, while upstream k8s
has since reached 1.31. Fortunately, kubeadm allows us to create
a 1.30 Kubernetes cluster on s390x.
This commit switches the KUBERNETES option from k3s to kubeadm
for s390x and removes a dedicated cluster creation step.
Now, cluster setup and teardown occur in ACTIONS_RUNNER_HOOK_JOB_{STARTED,COMPLETED}.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
When creating a k8s cluster via kubeadm, the devmapper setup
for containerd requires a different configuration.
This commit introduces a new `kubeadm` option for the KUBERNETES
variable and adjusts the path to the containerd config file for
devmapper setup.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
All the oras push logic happens from inside `${workdir}`, while the
root_hash.txt extraction and renaming was not taking this into
consideration.
This was not caught during the manually triggered runs as those do not
perform the oras push.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
As we're now building everything needed to test TDX with measured rootfs
support, let's bring this test back in (for TDX only, at least for now).
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
The approach taken for now is to export MEASURED_ROOTFS=yes on the
workflow files for the architectures using confidential stuff, and leave
the "normal" build without having it set (to avoid any change of
expectation on the current bevahiour).
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Let's make sure we take the root_hashes into consideration to decide
whether the shim-v2 should or should not be used from the cached
artefacts.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Let's cache the root_hash.txt from the confidential image so we can use
them later on to decide whether there was a rootfs change that would
require shim-v2 to be rebuilt.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Let's ensure we remove the component and any extra tarball provided by
ORAS in case the cached component cannot be used.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
We may decide to add this later on, but for now this is only targetting
TEEs and the confidential image / initrd.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Let's ensure that we get the already built rootfs tarball from previous
steps of the action at the time we're building the shim-v2.
The reason we do that is because the rootfs binary tarballs has a
root_hash.txt file that contains the information needed the shim-v2
build scripts to add the measured rootfs arguments to the shim-v2
configuration files.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
As the rootfs will have what we need to add as part of the shim-v2
configuration files for measured rootfs, we **must** ensure this is
built **before** shim-v2.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
By doing this we can just re-use the dependencies already built, saving
us a reasonable amount of time.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This is a helper script that does basically what's already being done by
the s390x CI, which is:
* Move a folder with the components that we were stored / downloaded
during the GHA execution to the expected `build` location
* Get rid of the dependencies for a specific asset, as the dependencies
are already pulled in from previous GHA steps
For now this script is only being added but not yet executed anywhere,
and that will come as the next step in this series.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
So far we haven't been storing the rootfs dependencies as part of our
workflows, but we better do it to re-use them as part of the rootfs
build.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Return of proper error to the initiator is not guaranteed.
Method StopVM could kill shim process together with VM pieces.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
The only reason we had this one passing for amd64 is because the check
was done using the wrong variable (`matrix.stage`, while in the other
workflows the variable used is `inputs.stage`).
The commit that broke the release process is 67a8665f51, which
blindly copy & pasted the logic from the matrix assets.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Corrects device filemode permissions typo/regression in rustjail to `666` instead of `066`.
`666` is the standard and expected value for these devices in containers.
Fixes: #10454
Signed-off-by: Simon Kaegi <simon.kaegi@gmail.com>
This PR increase the time to run the stressng k8s tests for the
CoCo stability CI.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
More than one developer can and should be able to run this workflow at
the same time, without cancelling the job started by another developer.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Let's use "dev" instead of "manually-triggered" as it avoids the name
being too long, which results in failures to create AKS clusters.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This PR adds the trap statement into the kata doc
script to clean up properly the temporary files.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
As a new workflow was added for the cases where developers want to test
their changes in the workflow itself, let's make sure we stop allowing
manual triggers on this workflow, which can lead to a polluted /
misleading weather of the CI.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This workflow is intended to replace the `workflow_dispatch` trigger
currently present as part of the `ci-nightly.yaml`.
The reasoning behind having this done in this way is because of our good
and old GHA behaviour for `pull_request_target`, which requires a PR to
be merged in order to check the changes in the workflow itself, which
leads to:
* when a change in a workflow is done, developers (should) do:
* push their branch to the kata-containers repo
* manually trigger the "nightly" CI in order to ensure the changes
don't break anything
* this can result in the "nightly" CI weather being polluted
* we don't have the guarantee / assurance about the last n nightly
runs anymore
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Change the PACKAGES variable for the cbl-mariner rootfs-builder
to use the kata-packages-uvm meta package from
packages.microsoft.com to define the set of packages to be
contained in the UVM.
This aligns the UVM build for the Azure Linux distribution
with the UVM build done for the Kata Containers offering on
Azure Kubernetes Services (AKS).
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
By doing this we can ensure that whenever the rootfs changes, we'll be
able to get the new root_hash.txt and use it.
This is the very first step to bring the measured rootfs tests back.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
When QEMU is terminated by signal 15, it deletes the PidFile.
Upon detecting that QEMU has exited, the shim executes the stopVM function.
If the PidFile is not found, the PID is set to 0.
Subsequently, the shim executes `kill -9 0`, which terminates the current process group.
This prevents any further logic from being executed, resulting in resources not being cleaned up.
Signed-off-by: wangyaqi54 <wangyaqi54@jd.com>
Semantics are lifted straight out of the go runtime for compatibility.
We introduce DeviceVirtioScsi to represent a virtio-scsi device and
instantiate it if block device driver in the configuration file is set
to virtio-scsi. We also introduce ObjectIoThread which is instantiated
if the configuration file additionally enables iothreads.
Signed-off-by: Pavel Mores <pmores@redhat.com>
When do update_container_namespaces updating namespaces, setting
all UTS(and IPC) namespace paths to None resulted in hostnames
set prior to the update becoming ineffective. This was primarily
due to an error made while aligning with the oci spec: in an attempt
to match empty strings with None values in oci-spec-rs, all paths
were incorrectly set to None.
Fixes#10325
Signed-off-by: alex.lyn <alex.lyn@antgroup.com>
Moved the parsing of the oci image marker into its own step, since we
only need to perform that for attestation purposes and some cached
images might not have that file in the tarball.
Signed-off-by: Magnus Kulke <magnuskulke@microsoft.com>
This adds provenance attestation logic for agent binaries that are
published to an oci registry via ORAS.
As a downstream consumer of the kata-agent binary the Peerpod project
needs to verify that the artifact has been built on kata's CI.
To create an attestation we need to know the exact digest of the oci
artifact, at the point when the artifact was pushed.
Therefore we record the full oci image as returned by oras push.
The pushing and tagging logic has been slightly reworked to make this
task less repetetive.
The oras cli accepts multiple tags separated by comma on pushes, so a
push can be performed atomically instead of iterating through tags and
pushing each individually. This removes the risk of partially successful
push operations (think: rate limits on the oci registry).
So far the provenance creation has been only enabled for agent builds on
amd64 and xs390x.
Signed-off-by: Magnus Kulke <magnuskulke@microsoft.com>
This PR updates the CI documentation referring to the several tests and
in which kind of instances is running them.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Use rstest for unit test rather than TestData arrays where
possible to make the code more compact, easier to read
and open the possibility to enhance test cases with a
description more easily.
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This PR replaces the arch uname -m to use the arch_to_golang
variable in the script to have a better uniformity across the script.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
inotify-configmap-pod.yaml is using: "inotifywait --timeout 120",
so wait for up to 180 seconds for the pod termination to be
reported.
Hopefully, some of the sporadic errors from #10413 will be avoided
this way:
not ok 1 configmap update works, and preserves symlinks
waitForProcess "${wait_time}" "$sleep_time" "${command}" failed
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
While Kubernetes defines `binaryData` as `[]byte`,
when defined in a YAML file the raw bytes are
base64 encoded. Therefore, we need to read the YAML
value as `String` and not as `Vec<u8>`.
Fixes: #10410
Signed-off-by: Leonard Cohnen <lc@edgeless.systems>
Adds dependencies of 'clang' & 'protobuf' to be installed in runners
when building agent-ctl sources having image pull support.
Fixes#10400
Signed-off-by: Sumedh Alok Sharma <sumsharma@microsoft.com>
This PR removes egrep command as it has been deprecated and it replaces by
grep in the test images script.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Use the configuration used by AKS (static_sandbox_resource_mgmt=true)
for CI testing on Mariner hosts.
Hopefully pod startup will become more predictable on these hosts -
e.g., by avoiding the occasional hotplug timeouts described by #10413.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
When dealing with a specific release, it was easier to just do some
adjustments on the image that has to be used for ITA without actually
adding a new entry in the versions.yaml.
However, it's been proven to be more complicated than that when it comes
to dealing with staged images, and we better explicitly add (and
update) those versions altogether to avoid CI issues.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
The root cause is that the CDH client is a global variable, and unit tests `test_unseal_env` and `test_unseal_file`
share this lock-free global variable, leading to resource contention and destruction.
Merging the two unit tests into one test_sealed_secret will resolve this issue.
Fixes: #10403
Signed-off-by: ChengyuZhu6 <zhucy0405@gmail.com>
As mariner has switched to using an image instead of an initrd, let's
just drop the abiliy to build the initrd and avoid keeping something in
the tree that won't be used.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
As the mariner image is already in place, and the tests were modified to
use them (as part of this series), let's just stop building it as part
of the CI.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
The reason we're doing this is because mariner image uses, by default,
cgroups default-hierarchy as `unified` (aka, cgroupsv2).
In order to keep the same initrd behaviour for mariner, let's enforce
that `SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1
systemd.legacy_systemd_cgroup_controller=yes
systemd.unified_cgroup_hierarchy=0` is passed to the kernel cmdline, at
least for now.
Other tests that are setting `kernel_params` are not running on mariner,
then we're safe taking this path as it's done as part of this PR.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
As an image has been added for mariner as part of the commit 63c1f81c2,
let's start using it in the CI, instead of using the initrd.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
This reverts commit 31e09058af, as it's
breaking the agent unit tests CI.
This is a stop gap till Chengyu Zhu finds the time to properly address
the issue, avoiding the CI to be blocked for now.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
In https://github.com/confidential-containers/trustee/pull/521
the overlays logic was modified to add non-SE
s390x support and simplify non-ibm-se platforms.
We need to update the logic in `kbs_k8s_deploy`
to match and can remove the dummying of `IBM_SE_CREDS_DIR`
for non-SE now
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This PR adds the trap statement in the test images script to clean up
tmp files.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Kata CI will start testing the new rootfs-image-mariner instead of the
older rootfs-initrd-mariner image.
The "official" AKS images are moving from a rootfs-initrd-mariner
format to the rootfs-image-mariner format. Making the same change in
Kata CI is useful to keep this testing in sync with the AKS settings.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Refactor CDH-related operations into the cdh_handler function to make the `create_container` code clearer.
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
Users must set the mount path to `/sealed/<path>` for kata agent to detect the sealed secret mount
and handle it in createcontainer stage.
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
Signed-off-by: Linda Yu <linda.yu@intel.com>
Introduced `unseal_file` function to unseal secret as files:
- Implemented logic to handle symlinks and regular files within the sealed secret directory.
- For each entry, call CDH to unseal secrets and the unsealed contents are written to a new file, and a symlink is created to replace the sealed symlink.
Fixes: #8123
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
Signed-off-by: Linda Yu <linda.yu@intel.com>
By checking for AGENT_POLICY we ensure we only try to read
allow-all.rego if AGENT_POLICY is set to "yes"
Signed-off-by: Emanuel Lima <emlima@redhat.com>
The behavior of Kata CI doesn't change.
For local testing using kubernetes/gha-run.sh and AUTO_GENERATE_POLICY=yes:
1. Before these changes users were forced to use:
- SEV, SNP, or TDX guests, or
- KATA_HOST_OS=cbl-mariner
2. After these changes users can also use other platforms that are
configured with "shared_fs = virtio-fs" - e.g.,
- KATA_HOST_OS=ubuntu + KATA_HYPERVISOR=qemu
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
This PR removes duplicated arch variable definition in the rootfs script
as this variable and its value is already defined at the top of the
script.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
add_device() now checks if QEMU is running already by checking if we have
a QMP connection. If we do a new function hotplug_device() is called
which hotplugs the device if it's a network one.
Signed-off-by: Pavel Mores <pmores@redhat.com>
With the helpers from previous commit, the actual hotplugging
implementation, though lengthy, is mostly just assembling a QMP command
to hotplug the network device backend and then doing the same for the
corresponding frontend.
Note that hotplug_network_device() takes cmdline_generator types Netdev
and DeviceVirtioNet. This is intentional and aims to take advantage of
the similarity between parameter sets needed to coldplug and hotplug
devices reuse and simplify our code. To enable using the types from qmp,
accessors were added as needed.
Signed-off-by: Pavel Mores <pmores@redhat.com>
Before adding network device hotplugging functionality itself we add
a couple of helpers in a separate commit since their functionality is
non-trivial.
To hotplug a device we need a free PCI slot. We add find_free_slot()
which can be called to obtain one. It looks for PCI bridges connected
to the root bridge and looks for an unoccupied slot on each of them. The
first found is returned to the caller. The algorithm explicitly doesn't
support any more complex bridge hierarchies since those are never produced
when coldplugging PCI bridges.
Sending netdev queue and vhost file descriptors to QEMU is slightly
involved and implemented in pass_fd(). The actual socket has to be passed
in an SCM_RIGHTS socket control message (also called ancillary data, see
man 3 cmsg) so we have to use the msghdr structure and sendmsg() call
(see man 2 sendmsg) to send the message. Since qapi-rs doesn't support
sending messages with ancillary data we have to do the sending sort of
"under it", manually, by retrieving qapi-rs's socket and using it directly.
Signed-off-by: Pavel Mores <pmores@redhat.com>
NetworkConfig::index has been used to generate an id for a network device
backend. However, it turns out that it's not unique (it's always zero
as confirmed by a comment at its definition) so it's not suitable to
generate an id that needs to be unique.
Use the host device name instead.
Signed-off-by: Pavel Mores <pmores@redhat.com>
Network device hotplugging will use the same infrastructure (Netdev,
DeviceVirtioNet) as coldplugging, i.e. QemuCmdLine. To make the code
of network device setup visible outside of QemuCmdLine we factor it out
to a non-member function `get_network_device()` and make QemuCmdLine just
delegate to it.
Signed-off-by: Pavel Mores <pmores@redhat.com>
The function takes a whole QemuCmdLine but only actually uses
HypervisorConfig. We increase callability of the function by limiting
its interface to what it needs. This will come handy shortly.
Signed-off-by: Pavel Mores <pmores@redhat.com>
At least one PCI bridge is necessary to hotplug PCI devices. We only
support PCI (at this point at least) since that's what the go runtime
does (note that looking at the code in virtcontainers it might seem that
other bus types are supported, however when the bridge objects are passed
to govmm, all but PCI bridges are actually ignored). The entire logic of
bridge setup is lifted from runtime-go for compatibility's sake.
Signed-off-by: Pavel Mores <pmores@redhat.com>
with multiple iterations/reruns we need to use the latest run of each
workflow. For that we can use the "run_id" and only update results of
the same or newer run_ids.
To do that we need to store the "run_id". To avoid adding individual
attributes this commit stores the full job object that contains the
status, conclussion as well as other attributes of the individual jobs,
which might come handy in the future in exchange for slightly bigger
memory overhead (still we only store the latest run of required jobs
only).
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
some tests require certain labels before they are executed. When our PR
is not labeled appropriately the gatekeeper detects skipped required
tests and reports a failure. With this change we add "required-labeles"
to the tests mapping and check the expected labels first informing the
user about the missing labeles before even checking the test statuses.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
the test names are using `;` and regexps were designed to use `,` but
during development simply joined the expressions by `|`. This should
work but might be confusing so let's go with the semi-colon separator
everywhere.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
The Github SHA of triggering PR should be exported in the environment
so that gatekeeper can fetch the right workflows/jobs.
Note: by default github will export GITHUB_SHA in the job's environment
but that value cannot be used if the gatekeeper was triggered from a
pull_request_target event, because the SHA correspond to the push
branch.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This will allow to cancel-in-progress the gatekeeper jobs.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
to allow selective testing as well as selective list of required tests
let's add a mapping of required jobs/tests in "skips.py" and a
"gatekeaper" workflow that will ensure the expected required jobs were
successful. Then we can only mark the "gatekeaper" as the required job
and modify the logic to suit our needs.
Fixes: #9237
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
The behavior of Kata CI doesn't change.
For local testing using kubernetes/gha-run.sh:
1. Before these changes:
- AUTO_GENERATE_POLICY=yes was always used by the users of SEV, SNP,
TDX, or KATA_HOST_OS=cbl-mariner.
2. After these changes:
- Users of SEV, SNP, TDX, or KATA_HOST_OS=cbl-mariner must specify
AUTO_GENERATE_POLICY=yes if they want to auto-generate policy.
- These users have the option to test just using hard-coded policies
(e.g., using the default policy built into the Guest rootfs) by
using AUTO_GENERATE_POLICY=no. AUTO_GENERATE_POLICY=no is the default
value of this env variable.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
This PR adds the trap statement in the confidential kbs script
to clean up temporary files and ensure we are leaving them.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
The tests is disabled for qemu-coco-dev / qemu-tdx, but it doesn't seen
to actually be failing on those. Plus, it's passing on SEV / SNP, which
means that we most likely missed re-enabling this one in the past.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
The quay.io registry returns the tags sorted alphabetically and doesn't
seem to provide a way to sort it by age. Let's use "git log" to get all
changes between the commits and print all tags that were actually
pushed.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
Currently, `qemu-runtime-rs` does not support `virtio-scsi`,
which causes the `k8s-block-volume.bats` test to fail.
We should skip this test until `virtio-scsi` is supported by the runtime.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
The nginx container seems to error out when using UID=123.
Depending on the timing between container initialization and "kubectl
wait", the test might have gotten lucky and found the pod briefly in
Ready state before nginx errored out. But on some of the nodes, the pod
never got reported as Ready.
Also, don't block in "kubectl wait --for=condition=Ready" when wrapping
that command in a waitForProcess call, because waitForProcess is
designed for short-lived commands.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
This PR updates the fast footprint script to remove the use
of egrep as this command has been deprecated and change it
to use grep command.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This imports the k8s-block-volume test from the tests repo and modifies
it slightly to set up the host volume on the AKS host.
This is a follow-up to #7132.
Fixes: #7164
Signed-off-by: Salvador Fuentes <salvador.fuentes@intel.com>
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
The ita kustomization for Trustee, as well as previously used one
(DCAP), doesn't have a $(uname -m) directory after the deployment
directory name.
Let's follow the same logic used for the deploy-kbs script and clean
those up accordingly.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Intel Tiber Trust Services (formerly known as Intel Trust Authority) is
Intel's own attestation service, and we want to take advantage of the
TDX CI in order to ensure ITTS works as expected.
In order to do so, let's replace the former method used (DCAP) to use
ITTS instead.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Inorder to support sandbox api, intorduce the sandbox_config
struct and split the sandbox start stage from init process.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
There are many similar or duplicated code patterns in `teardown()`.
This commit consolidates them into a new function, `teardown_common()`,
which is now called within `teardown()`.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
The current `exec_host()` accepts a given node name and
creates a node debugger pod, even if the name is invalid.
This could result in the creation of an unnecessary pending
pod (since we are using nodeAffinity; if the given name
does not match any actual node names, the pod won’t be scheduled),
which wastes resources.
This commit introduces validation for the node name to
prevent this situation.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This commit enables basic amd64 tests of docker for runtime-rs by adding
vmm types "dragonball" and "cloud-hypervisor".
Signed-off-by: Sicheng Liu <lsc2001@outlook.com>
Docker cannot exit normally after the container process exits when
used with runtime-rs since it doesn't receive the exit event. This
commit enable runtime-rs to send TaskExit to containerd after process
exits.
Also, it moves "system_time_into" and "option_system_time_into" from
crates/runtimes/common/src/types/trans_into_shim.rs to a new utility
mod.
Signed-off-by: Sicheng Liu <lsc2001@outlook.com>
When the sandbox api was enabled, the pasue container
wouldn't be created, thus the shared sandbox pidns
should be fallbacked to the first container's init process,
instead of return any error here.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
It was observed that the custom node debugger pod is not
cleaned up when a test times out.
This commit ensures the pod is cleaned up by triggering
the cleanup on EXIT, preventing any debugger pods from
being left behind.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
When using network adapters that support SR-IOV, a VFIO device can be
plugged into a guest VM and claimed as a network interface. This can
significantly enhance network performance.
Fixes: #9758
Signed-off-by: Lei Huang <leih@nvidia.com>
With #10232 merged, we now have a persistent node debugger pod throughout the test.
As a result, there’s no need to spawn another debugger pod using `kubectl debug`,
which could lead to false negatives due to premature pod termination, as reported
in #10081.
This commit removes the `print_node_journal()` call that uses `kubectl debug` and
instead uses `exec_host()` to capture the host journal. The `exec_host()` function
is relocated to `tests/integration/kubernetes/lib.sh` to prevent cyclical dependencies
between `tests_common.sh` and `lib.sh`.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
in b9d88f74ed the `runtime_class` CM was
added which overrides the one we previously set. Let's reorder our logic
to first deploy webhook and then override the default CM in order to use
the one we really want.
Since we need to change dirs we also have to use realpath to ensure the
files are located well.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
`assert_pod_fail()` currently calls `k8s_create_pod()` to ensure that a pod
does not become ready within the default 120s. However, this delays the test's
completion even if an error message is detected earlier in the journal.
This commit removes the use of `k8s_create_pod()` and modifies `assert_pod_fail()`
to fail as soon as the pod enters a failed state.
All failing pods end up in one of the following states:
- CrashLoopBackOff
- ImagePullBackOff
The function now polls the pod's state every 5 seconds to check for these conditions.
If the pod enters a failed state, the function immediately returns 0. If the pod
does not reach a failed state within 120 seconds, it returns 1.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This PR removes some qemu information which is not longer valid as
this is referring to the tests repository and to kata 1.x.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
rename the task_service to service, in order to
incopperate with the following added sandbox
services.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
In order to make different from sandbox request/response, this commit
changed the task request/response to TaskRequest/TaskResponse.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
Since the wait_vm would be called before calling stop_vm,
which would take the reader lock, thus blocking the stop_vm
getting the writer lock, which would trigge the dead lock.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
Since the block_on would block on the current thread
which would prevent other async tasks to be run on this
worker thread, thus change it to use the async task for
this task.
Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
- Reflect the need to update the versions in the Helm Chart
- Add the lock branch instruction
- Add clarity about the permissions needed to complete tasks
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
* Clarifies instructions for k0s.
* Adds kata-deploy step for each cluster type.
* Removes the old kata-deploy-stable step for vanilla k8s.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This PR introduces support for selectively compiling Dragonball in
runtime-rs. By default, Dragonball will continue to be compiled into
the containerd-shim-kata-v2 executable, but users now have the option
to disable Dragonball compilation.
Fixes#10310
Signed-off-by: sidney chang <2190206983@qq.com>
Now that the issue with handling loop devices has been resolved,
this commit re-enables the guest-pull-image tests for `qemu-coco-dev`.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Timeouts occur (e.g. `create_container_timeout` and `wait_time`)
when using qemu-coco-dev.
This commit increases these timeouts for the trusted image storage
test cases
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
If the host running the tests is different from the host where the cluster is running,
the *_loop_device() functions do not work as expected because the device is created
on the test host, while the cluster expects the device to be local.
This commit ensures that all commands for the relevant functions are executed via exec_host()
so that a device should be handled on a cluster node.
Additionally, it modifies exec_host() to return the exit code of the last executed command
because the existing logic with `kubectl debug` sometimes includes unexpected characters
that are difficult to handle. `kubectl exec` appears to properly return the exit code for
a given command to it.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Creating and deleting a node debugger pod for every `exec_host()`
call is inefficient.
This commit changes the test suite to create and delete the pod
only once, globally.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This commit addresses an issue with handling loop devices
via a node debugger due to restricted privileges.
It runs a pod with full privileges, allowing it to mount
the host root to `/host`, similar to the node debugger.
This change enables us to run tests for trusted image storage
using the `qemu-coco-dev` runtime class.
Fixes: #10133
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add cdi devices including ContainerDevice definition and
annotation_container_device method to annotate vfio device
in OCI Spec annotations which is inserted into Guest with
its mapping of vendor-class and guest pci path.
Fixes#10145
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
We need vfio device's properties device, vendor and
class, but we can only get property device and vendor.
just extend it with class is ok.
Fixes#10145
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The kata webhook requires a configmap to define what runtime class it
should set for the newly created pods. Additionally, the configmap
allows others to modify the default runtime class name we wish to set
(in case the handler is kata but the name of the runtimeclass is
different).
Finally, this PR changes the webhook-check to compare the runtime of the
newly created pod against the specific runtime class in the configmap,
if said confimap doesn't exist, then it will default to "kata".
Signed-off-by: Martin <mheberling@microsoft.com>
This PR increases the timeout to run k8s tests for Kata CoCo TDX
to avoid the random failures of timeout.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
As we don't have any CI, nor maintainer to keep ACRN code around, we
better have it removed than give users the expectation that it should or
would work at some point.
Signed-off-by: Fabiano Fidêncio <fabiano@fidencio.org>
Yq installation shouldn't force to use sudo in case yq is already installed in correct version.
Signed-off-by: Pawel Proskurnicki <pawel.proskurnicki@intel.com>
It's a prerequisite PR to make built-in vmm dragonball compilation
options configurable.
Extract TAP device-related code from dragonball's dbs_utils into a
separate library within the runtime-rs hypervisor module.
To enhance functionality and reduce dependencies, the extracted code
has been reimplemented using the libc crate and the ifreq structure.
Fixes#10182
Signed-off-by: sidney chang <2190206983@qq.com>
2024-09-13 07:32:14 -07:00
600 changed files with 28509 additions and 8437 deletions
@@ -499,19 +499,6 @@ If you do not want to install the respective QEMU version, the configuration fil
See the [static-build script for QEMU](../tools/packaging/static-build/qemu/build-static-qemu.sh) for a reference on how to get, setup, configure and build QEMU for Kata.
### Build a custom QEMU for aarch64/arm64 - REQUIRED
> **Note:**
>
> - You should only do this step if you are on aarch64/arm64.
> - You should include [Eric Auger's latest PCDIMM/NVDIMM patches](https://patchwork.kernel.org/cover/10647305/) which are
> under upstream review for supporting NVDIMM on aarch64.
>
You could build the custom `qemu-system-aarch64` as required with the following command:
| [image](background.md#root-filesystem-image) | [Ubuntu](https://ubuntu.com) (for x86_64 systems)| systemd | Fully tested in our CI | systemd offers flexibility |
| [initrd](#initrd-image) | [Alpine Linux](https://alpinelinux.org) | Kata [agent](README.md#agent) (as no systemd support) | Security hardened and tiny C library |
@@ -50,7 +50,7 @@ We provide `Dragonball` Sandbox to enable built-in VMM by integrating VMM's func
#### How To Support Async
The kata-runtime is controlled by TOKIO_RUNTIME_WORKER_THREADS to run the OS thread, which is 2 threads by default. For TTRPC and container-related threads run in the `tokio` thread in a unified manner, and related dependencies need to be switched to Async, such as Timer, File, Netlink, etc. With the help of Async, we can easily support no-block I/O and timer. Currently, we only utilize Async for kata-runtime. The built-in VMM keeps the OS thread because it can ensure that the threads are controllable.
**For N tokio worker threads and M containers**
**For N `tokio` worker threads and M containers**
- Sync runtime(both OS thread and `tokio` task are OS thread but without `tokio` worker thread) OS thread number: 4 + 12*M
- Async runtime(only OS thread is OS thread) OS thread number: 2 + N
@@ -103,7 +103,6 @@ In our case, there will be a variety of resources, and every resource has severa
| `Cgroup V2` | | Stage 2 | 🚧 |
| Hypervisor | `Dragonball` | Stage 1 | 🚧 |
| | QEMU | Stage 2 | 🚫 |
| | ACRN | Stage 3 | 🚫 |
| | Cloud Hypervisor | Stage 3 | 🚫 |
| | Firecracker | Stage 3 | 🚫 |
@@ -166,4 +165,4 @@ In our case, there will be a variety of resources, and every resource has severa
- What is the security boundary for the monolithic / "Built-in VMM" case?
It has the security boundary of virtualization. More details will be provided in next stage.
It has the security boundary of virtualization. More details will be provided in next stage.
@@ -10,7 +10,19 @@ To run Kata Containers in SNP-VMs, the following software stack is used.
The host BIOS and kernel must be capable of supporting AMD SEV-SNP and configured accordingly. For Kata Containers, the host kernel with branch [`sev-snp-iommu-avic_5.19-rc6_v3`](https://github.com/AMDESE/linux/tree/sev-snp-iommu-avic_5.19-rc6_v3) and commit [`3a88547`](https://github.com/AMDESE/linux/commit/3a885471cf89156ea555341f3b737ad2a8d9d3d0) is known to work in conjunction with SEV Firmware version 1.51.3 (0xh\_1.33.03) available on AMD's [SEV developer website](https://developer.amd.com/sev/). See [AMD's guide](https://github.com/AMDESE/AMDSEV/tree/sev-snp-devel) to configure the host accordingly. Verify that you are able to run SEV-SNP encrypted VMs first. The guest components required for Kata Containers are built as described below.
The host BIOS and kernel must be capable of supporting AMD SEV-SNP and the host must be configured accordingly.
The latest SEV Firmware version is available on AMD's [SEV Developer Webpage](https://www.amd.com/en/developer/sev.html). It can also be updated via a platform OEM BIOS update.
The host kernel must be equal to or later than upstream version [6.11](https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.11.tar.xz).
[`sev-utils`](https://github.com/amd/sev-utils/blob/coco-202501150000/docs/snp.md) is an easy way to install the required host kernel with the `setup-host` command. However, it will also build compatible guest kernel, OVMF, and QEMU components which are not necessary as these components are packaged with kata. The `sev-utils` script utility can be used with these additional components to test the memory encrypted launch and attestation of a base QEMU SNP guest.
For a simplified way to build just the upstream compatible host kernel, use the Confidential Containers fork of [AMDESE AMDSEV](https://github.com/confidential-containers/amdese-amdsev/tree/amd-snp-202501150000). Individual components can be built by running the following command:
```
./build.sh kernel host --install
```
**Tip**: It is easiest to first have Kata Containers running on your system and then modify it to run containers in SNP-VMs. Follow the [Developer guide](../Developer-Guide.md#warning) and then follow the below steps. Nonetheless, you can just follow this guide from the start.
@@ -29,16 +41,16 @@ __SNP-specific steps:__
- Build the SNP-specific kernel as shown below (see this [guide](../../tools/packaging/kernel/README.md#build-kata-containers-kernel) for more information)
```bash
$ pushd kata-containers/tools/packaging/
$ ./kernel/build-kernel.sh -a x86_64 -x snp setup
$ ./kernel/build-kernel.sh -a x86_64 -x snp build
$ sudo -E PATH="${PATH}" ./kernel/build-kernel.sh -x snp install
@@ -46,7 +46,6 @@ There are several kinds of Kata configurations and they are listed below.
| `io.katacontainers.config.hypervisor.block_device_cache_set` | `boolean` | cache-related options will be set to block devices or not |
| `io.katacontainers.config.hypervisor.block_device_driver` | string | the driver to be used for block device, valid values are `virtio-blk`, `virtio-scsi`, `nvdimm`|
| `io.katacontainers.config.hypervisor.cpu_features` | `string` | Comma-separated list of CPU features to pass to the CPU (QEMU) |
| `io.katacontainers.config.hypervisor.ctlpath` (R) | `string` | Path to the `acrnctl` binary for the ACRN hypervisor |
| `io.katacontainers.config.hypervisor.default_max_vcpus` | uint32| the maximum number of vCPUs allocated for the VM by the hypervisor |
| `io.katacontainers.config.hypervisor.default_memory` | uint32| the memory assigned for a VM by the hypervisor in `MiB` |
| `io.katacontainers.config.hypervisor.default_vcpus` | float32| the default vCPUs assigned for a VM by the hypervisor |
@@ -95,6 +94,8 @@ There are several kinds of Kata configurations and they are listed below.
| `io.katacontainers.config.hypervisor.virtio_fs_extra_args` | string | extra options passed to `virtiofs` daemon |
| `io.katacontainers.config.hypervisor.enable_guest_swap` | `boolean` | enable swap in the guest |
| `io.katacontainers.config.hypervisor.use_legacy_serial` | `boolean` | uses legacy serial device for guest's console (QEMU) |
| `io.katacontainers.config.hypervisor.default_gpus` | uint32 | the minimum number of GPUs required for the VM. Only used by remote hypervisor to help with instance selection |
| `io.katacontainers.config.hypervisor.default_gpu_model` | string | the GPU model required for the VM. Only used by remote hypervisor to help with instance selection |
This document provides an overview on how to run Kata containers with ACRN hypervisor and device model.
## Introduction
ACRN is a flexible, lightweight Type-1 reference hypervisor built with real-time and safety-criticality in mind. ACRN uses an open source platform making it optimized to streamline embedded development.
Some of the key features being:
- Small footprint - Approx. 25K lines of code (LOC).
- Real Time - Low latency, faster boot time, improves overall responsiveness with hardware.
- Adaptability - Multi-OS support for guest operating systems like Linux, Android, RTOSes.
- Rich I/O mediators - Allows sharing of various I/O devices across VMs.
- Optimized for a variety of IoT (Internet of Things) and embedded device solutions.
Please refer to ACRN [documentation](https://projectacrn.github.io/latest/index.html) for more details on ACRN hypervisor and device model.
## Pre-requisites
This document requires the presence of the ACRN hypervisor and Kata Containers on your system. Install using the instructions available through the following links:
- For networking, ACRN supports either MACVTAP or TAP. If MACVTAP is not enabled in the Service OS, please follow the below steps to update the kernel:
$ sudo sed -i "s/$kernel_img/bzImage/g" /mnt/loader/entries/$conf_file
$ sync && sudo umount /mnt && sudo reboot
```
- Kata Containers installation: Automated installation does not seem to be supported for Clear Linux, so please use [manual installation](../Developer-Guide.md) steps.
> **Note:** Create rootfs image and not initrd image.
In order to run Kata with ACRN, your container stack must provide block-based storage, such as device-mapper.
> **Note:** Currently, by design you can only launch one VM from Kata Containers using ACRN hypervisor (SDC scenario). Based on feedback from community we can increase number of VMs.
## Configure Docker
To configure Docker for device-mapper and Kata,
1. Stop Docker daemon if it is already running.
```bash
$ sudo systemctl stop docker
```
2. Set `/etc/docker/daemon.json` with the following contents.
```
{
"storage-driver": "devicemapper"
}
```
3. Restart docker.
```bash
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
```
4. Configure [Docker](../Developer-Guide.md#update-the-docker-systemd-unit-file) to use `kata-runtime`.
## Configure Kata Containers with ACRN
To configure Kata Containers with ACRN, copy the generated `configuration-acrn.toml` file when building the `kata-runtime` to either `/etc/kata-containers/configuration.toml` or `/usr/share/defaults/kata-containers/configuration.toml`.
The following command shows full paths to the `configuration.toml` files that the runtime loads. It will use the first path that exists. (Please make sure the kernel and image paths are set correctly in the `configuration.toml` file)
```bash
$ sudo kata-runtime --show-default-config-paths
```
>**Warning:** Please offline CPUs using [this](offline_cpu.sh) script, else VM launches will fail.
```bash
$ sudo ./offline_cpu.sh
```
Start an ACRN based Kata Container,
```bash
$ sudo docker run -ti --runtime=kata-runtime busybox sh
```
You will see ACRN(`acrn-dm`) is now running on your system, as well as a `kata-shim`. You should obtain an interactive shell prompt. Verify that all the Kata processes terminate once you exit the container.
```bash
$ ps -ef | grep -E "kata|acrn"
```
Validate ACRN hypervisor by using `kata-runtime kata-env`,
After running the command above, the default config file `configuration.toml` will be installed under `/usr/share/defaults/kata-containers/`, the binary file `containerd-shim-kata-v2` will be installed under `/usr/local/bin/` .
### Install Shim Without Builtin Dragonball VMM
By default, runtime-rs includes the `Dragonball` VMM. To build without the built-in `Dragonball` hypervisor, use `make USE_BUILDIN_DB=false`:
```bash
$ cd kata-containers/src/runtime-rs
$ make USE_BUILDIN_DB=false
```
After building, specify the desired hypervisor during installation using `HYPERVISOR`. For example, to use `qemu` or `cloud-hypervisor`:
```
sudo make install HYPERVISOR=qemu
```
or
```
sudo make install HYPERVISOR=cloud-hypervisor
```
### Build Kata Containers Kernel
Follow the [Kernel installation guide](/tools/packaging/kernel/README.md).
The host could be a bare metal instance or a virtual machine. If using a
virtual machine, make sure that KVM nesting is enabled. The following
instructions reference an Intel® C62X chipset. Some of the instructions must be
The host could be a bare metal instance or a virtual machine. If using a
virtual machine, make sure that KVM nesting is enabled. The following
instructions reference an Intel® C62X chipset. Some of the instructions must be
modified if using a different Intel® QAT device. The Intel® QAT chipset can be
identified by executing the following.
@@ -74,7 +74,7 @@ $ for i in 0434 0435 37c8 1f18 1f19; do lspci -d 8086:$i; done
These packages are necessary to compile the Kata kernel, Intel® QAT driver, and to
prepare the rootfs for Kata. [Docker](https://docs.docker.com/engine/install/ubuntu/)
also needs to be installed to be able to build the rootfs. To test that
also needs to be installed to be able to build the rootfs. To test that
everything works a Kubernetes pod is started requesting Intel® QAT resources. For the
pass through of the virtual functions the kernel boot parameter needs to have
`INTEL_IOMMU=on`.
@@ -89,7 +89,7 @@ $ sudo reboot
### Download Intel® QAT drivers
This will download the [Intel® QAT drivers](https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html).
This will download the [Intel® QAT drivers](https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html).
Make sure to check the website for the latest version.
bail!("sealed secret file {:?} does not exist",path);
}
// Iterate over all entries to handle the sealed secret file.
// For example, the directory is as follows:
// The secret directory in the guest: /run/kata-containers/shared/containers/21bbf0d932b70263d65d7052ecfd72ee46de03f766650cb378e93852ddb30a54-5063be11b6800f96-sealed-secret-target/:
# This is kinda dirty step here simply because cargo fmt --all will apply fmt to all dependencies of dragonball which will include /src/libs/protocols with some file generated during compilation time and could not be formatted when you use cargo fmt --all before building the whole project. In order to avoid this problem, we do fmt check in this following way.
# This is kinda dirty step here simply because cargo fmt --all will apply fmt to all dependencies of dragonball which will include /src/libs/protocols with some file generated during compilation time and could not be formatted when you use cargo fmt --all before building the whole project. In order to avoid this problem, we do fmt check in this following way.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
