Let's stop skipping the CDH tests for TDX, as know we should have an
environmemnt where it can run and should pass. :-)
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
We must ensure we use the host ip to connect to the PCCS running on the
host side, instead of using localhost (which has a different meaning
from inside the KBS pod).
The reason we're using `hostname -i` isntead of the helper functions, is
because the helper functions need the coco-kbs deployed for them to
work, and what we do is before the deployment.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This is generally useful debug output on test failures,
and specifically this has been useful for nydus-related
issues recently.
Signed-off-by: Chris Porter <porter@ibm.com>
A new PULL_TYPE environment variable is recognized by the kata-deploy's
install script to allow it to configure CRIO-O for guest-pull image pulling
type.
The tests/integration/kubernetes/gha-run.sh change allows for testing it:
```
export PULL_TYPE=guest-pull
cd tests/integration/kubernetes
./gha-run.sh deploy-k8s
```
Fixes#9474
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This PR replaces the name to use a variable that is already defined
to have a better uniformity across the general script.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
It was observed that the `node_start_time` value is sometimes empty,
leading to a test failure.
This commit retries fetching the value up to 3 times.
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This test fails when using `shared_fs=none` with the nydus snapshotter.
Issue tracked here: #9666
Skipping for now.
Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
This test is failing due to the initContainers not being properly
handled with the guest image pulling.
Issue tracked here: #9668
Skipping for now.
Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
This test fails when using `shared_fs=none` with the nydus napshotter,
Issue tracked here: #9664
Skipping for now.
Signed-Off-By: Ryan Savino <ryan.savino@amd.com>
To prevent download failures caused by high traffic to the Docker image,
opt for quay.io/prometheus/busybox:latest over docker.io/library/busybox:latest .
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
CreateContainerRequest objects can specify devices to be created inside
the guest VM. This change ensures that requested devices have a
corresponding entry in the PodSpec.
Devices that are added to the pod dynamically, for example via the
Device Plugin architecture, can be allowlisted globally by adding their
definition to the settings file.
Fixes: #9651
Signed-off-by: Markus Rudy <mr@edgeless.systems>
End of file should not end with --- mark. This will confuse tools like
yq and kubectl that might think this is another object.
Signed-off-by: Beraldo Leal <bleal@redhat.com>
Since yq frequently updates, let's upgrade to a version from February to
bypass potential issues with versions 4.41-4.43 for now. We can always
upgrade to the newest version if necessary.
Fixes#9354
Depends-on:github.com/kata-containers/tests#5818
Signed-off-by: Beraldo Leal <bleal@redhat.com>
Wainer noticed this is failing for the coco-qemu-dev case, and decided
to skip it, notifying me that he didn't fully understand why it was not
failing on TDX.
Turns out, though, this is also failing on TDX, and we need to skip it
there as well.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Add the CLI flag --runtime-class-names, which is used during
policy generation. For resources that can define a
runtimeClassName (e.g., Pods, Deployments, ReplicaSets,...)
the value must have any of the --runtime-class-names as
prefix, otherwise the resource is ignored.
This allows to run genpolicy on larger yaml
files defining many different resources and only generating
a policy for resources which will be deployed in a
confidential context.
Signed-off-by: Leonard Cohnen <lc@edgeless.systems>
Use the variable BATS_TEST_COMPLETED which is defined by the bats framework
when the test finishes. `BATS_TEST_COMPLETED=` (empty) means the test failed,
so the node syslogs will be printed only at that condition.
Fixes: #9750
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test fails with qemu-coco-dev configuration and guest-pull image pull.
Issue: #9667
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
It's cloning the nydus-snapshotter repo from the version specified in
versions.yaml, however, the deployment files are set to pull in the
latest version of the snapshotter image. With this version we are
pinning the image version too.
This is a temporary fix as it should be better worked out at nydus-snapshotter
project side.
Fixes: #9742
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test fails with qemu-coco-dev configuration and guest-pull image pull.
Issue: #9668
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test fails with qemu-coco-dev configuration and guest-pull image pull.
Issue: #9666
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test fails with qemu-coco-dev configuration and guest-pull image pull.
Issue: #9664
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test fails with qemu-coco-dev configuration and guest-pull image pull.
Unlike other tests that I've seen failing on this scenario, k8s-seccomp.bats
fails after a couple of consecutive executions, so it's that kind of failure
that happens once in a while.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This will enable the k8s tests to leverage guest pulling when
PULL_TYPE=guest-pull for qemu-coco-dev runtimeclass.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The runtime handler annotation is required for Kubernetes <= 1.28 and
guest-pull pull type. So leverage $PULL_TYPE (which is exported by CI jobs)
to conditionally apply the annotation.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This takes a few minutes that could be saved, so let's avoid doing this
on all the platforms, but simply do this when it's needed (the baremetal
use case).
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Currently only "baremetal" runs all the tests, but we could easily run
"all" locally or using the github provided runners, even when not using
a "baremetal" system.
The reason I'd like to have a differentiation between "all" and
"baremetal" is because "baremetal" may require some cleanup, which "all"
can simply skip if testing against a fresh created VM.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
For now we've only exposed the option to deploy kata-deploy for k3s and
vanilla kubernetes when using containerd.
However, I do need to also deploy k0s and rke2 for an internal CI, and
having those exposed here do not hurt, and allow us to easily expand the
CI at any time in the future.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
The k8s test suite halts on the first failure, i.e., failing-fast. This
isn't the behavior that we used to see when running tests on Jenkins and it
seems that running the entire test suite is still the most productive way. So
this disable fail-fast by default.
However, if you still wish to run on fail-fast mode then just export
K8S_TEST_FAIL_FAST=yes in your environment.
Fixes: #9697
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test has failed in confidential runtime jobs. Skip it
until we don't have a fix.
Fixes: #9663
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
One of our machines is running CentOS 9 Stream, and we could easily
verify that we can build and install the kbs client there, thus we're
expanding the installation script to also support CentOS 9 Stream.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
`externals.coco-kbs.toolchain` is not defined, get the rust_version from
`externals.coco-trustee.toolchain` instead.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This PR enables the installation and unistallation of the kbs client
as well as general coco components needed for the TDX GHA CI.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This PR adds the use of custom Intel DCAP configuration when
deploying the KBS.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This test doesn't fail with the guest image pulling, but it for sure
should. :-)
We can see in the bats logs, something like:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 31s default-scheduler Successfully assigned kata-containers-k8s-tests/liveness-exec to 984fee00bd70.jf.intel.com
Normal Pulled 23s kubelet Successfully pulled image "quay.io/prometheus/busybox:latest" in 345ms (345ms including waiting)
Normal Started 21s kubelet Started container liveness
Warning Unhealthy 7s (x3 over 13s) kubelet Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
Normal Killing 7s kubelet Container liveness failed liveness probe, will be restarted
Normal Pulled 7s kubelet Successfully pulled image "quay.io/prometheus/busybox:latest" in 389ms (389ms including waiting)
Warning Failed 5s kubelet Error: failed to create containerd task: failed to create shim task: the file /bin/sh was not found: unknown
Normal Pulling 5s (x3 over 23s) kubelet Pulling image "quay.io/prometheus/busybox:latest"
Normal Pulled 4s kubelet Successfully pulled image "quay.io/prometheus/busybox:latest" in 342ms (342ms including waiting)
Normal Created 4s (x3 over 23s) kubelet Created container liveness
Warning Failed 3s kubelet Error: failed to create containerd task: failed to create shim task: failed to mount /run/kata-containers/f0ec86fb156a578964007f7773a3ccbdaf60023106634fe030f039e2e154cd11/rootfs to /run/kata-containers/liveness/rootfs, with error: ENOENT: No such file or directory: unknown
Warning BackOff 1s (x3 over 3s) kubelet Back-off restarting failed container liveness in pod liveness-exec_kata-containers-k8s-tests(b1a980bf-a5b3-479d-97c2-ebdb45773eff)
```
Let's skip it for now as we have an issue opened to track it down:
https://github.com/kata-containers/kata-containers/issues/9665
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This is another one that is related to initContainers not being properly
handled with the guest image pulling.
Let's skip it for now as we have
https://github.com/kata-containers/kata-containers/issues/9668 to track
it down.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Similarly to firecracker, which doesn't have support for virtio-fs /
virtio-9p, TDX used with `shared_fs=none` will face the very same
limitations.
The tests affected are:
* k8s-credentials-secrets.bats
* k8s-file-volume.bats
* k8s-inotify.bats
* k8s-nested-configmap-secret.bats
* k8s-projected-volume.bats
* k8s-volume.bats
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This test fails when using `shared_fs=none` with the nydus snapshotter,
and we're tracking the issue here:
https://github.com/kata-containers/kata-containers/issues/9664
For now, let's have it skipped.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
The test has been failing on TDX for a while, and an issue has been
created to track it down, see:
https://github.com/kata-containers/kata-containers/issues/9663
For now, let's have it skipped.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's make sure the logs will print the correct annotation and its
value, instead of always mentioning "kernel" and "initrd".
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
We're doing that as all tests are going to be running with
`shared_fs=none`, meaning that we don't need any specific test for this
case anymore.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This function will set the needed annotation for enforcing that the
image pull will be handled by the snapshotter set for the runtime
handler, instead of using the default one.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Given that we think the containerd -> snapshotter image cache
problems have been resolved by bumping to nydus-snapshotter v0.3.13
we can try removing the skips to test this out
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
- We previously have an expectation for the pause rootfs
to be pull on the host when we did a guest pull. We weren't
really clear why, but it is plausible related to the issues we had
with containerd and nydus caching. Now that is fixed we can begin
to address this with setting shared_fs=none, but let's start with
updating the rootfs host check to be not higher than expected
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
This test is called from `tests/integration/run_kuberentes_tests.sh`,
which already ensures that yq is installed.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
On 1423420, I've mistakenly disabled the tests entirely, for both
non-TEEs and TEEs.
This happened as I didn't realise that `confidential_setup` would take
non-TEEs into consideration. :-/
Now, let me follow-up on that and make sure that the tests will be
running on non-TEEs.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This function is a helper to check whether the KATA_HYPERVISOR being
used is a confidential hardware (TEE) or not, and we can use it to
skip or only run tests on those platforms when needed.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's rename it to `is_confidential_runtime_class`, and adapt all the
places where it's called.
The new name provides a better description, leading to a better
understanding of what the function really does.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's skip the policy addition for now, in order to get the TDX CI back
up and running, and then we can re-enable it as soon as we get
https://github.com/kata-containers/kata-containers/issues/9612 fixed.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's skip those tests on TEEs as we've been facing a reasonable amount
of issues, most likely on the containerd side, related to pulling the
image on the guest.
Once we're able to fix the issues on containerd, we can get back and
re-enable those by reverting this commit.
The decision of disabling the tests for TEEs is because the machines may
end up in a state where human intervention is necessary to get them back
to a functional state, and that's really not optimal for our CI.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This PR adds a k8s negative policy test to the confidential attestation
bats test.
Fixes#9437
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
The CH v39 upgrade in #9575 is currently blocked because of a bug in the
Mariner host kernel. To address this, we temporarily tweak the Mariner
CI to use an Ubuntu host and the Kata guest kernel, while retaining the
Mariner initrd. This is tracked in #9594.
Importantly, this allows us to preserve CI for genpolicy. We had to
tweak the default rules.rego however, as the OCI version is now
different in the Ubuntu host. This is tracked in #9593.
This change has been tested together with CH v39 in #9588.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
With the addition of the 'qemu-coco-dev' runtimeClass we no longer need
to run CoCo tests on non-TEE environments with 'qemu'. As a result the
tests also no longer need to set the "io.katacontainers.config.hypervisor.image"
annotation to pods.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Enable the k8s tests for cloud hypervisor with devicemapper.
Fixes: #9221.
Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
Co-authored-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Avoid auto-generating Policy on platforms that haven't been tested
yet with auto-generated Policy.
Support for auto-generated Policy on these additional platforms is
coming up in future PRs, so the tests being fixed here were
prematurely enabled.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Remove k8s-policy-set-keys.bats in preparation for using the regorus
crate instead of the OPA daemon for evaluating the Agent Policy. This
test depended on sending HTTP requests to OPA.
Fixes: #9388
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Auto-generate the policy and then simulate attacks from the K8s
control plane by modifying the test yaml files. The policy then
detects and blocks those changes.
These test cases are using K8s Pods. Additional policy failures
are injected during CI using other types of K8s resources - e.g.,
using Jobs and Replication Controllers - from separate PRs.
Fixes: #9491
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Auto-generate the policy and then simulate attacks from the K8s
control plane by modifying the test yaml files. The policy then
detects and blocks those changes.
These test cases are using K8s Replication Controllers. Additional
policy failures will be injected using other types of K8s resources
- e.g., using Pods and/or Jobs - in separate PRs.
Fixes: #9463
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
When running on non-TEE environments (e.g. KATA_HYPERVISOR=qemu) the tests should
be stressing the CoCo image (/opt/kata/share/kata-containers/kata-containers-confidential.img)
although currently the default image/initrd is built to be able to do guest-pull as well.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Enabled guest-pull tests on non-TEE environment. It know requires the SNAPSHOTTER environment
variable to avoid it running on jobs where nydus-snapshotter is not installed
Fixes: #9410
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This PR adds the function to uninstall kbs client command function
specially when we are running with baremetal devices.
Fixes#9460
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This PR defines the PULL_TYPE variable to avoid failures of unbound
variable when this is being test it locally.
Fixes#9453
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Auto-generate the policy and then simulate attacks from the K8s
control plane by modifying the test yaml files. The policy then
detects and blocks those changes.
These test cases are using K8s Jobs. Additional policy failures
will be injected using other types of K8s resources - e.g., using
Pods and/or Replication Controllers - in future PRs.
Fixes: #9406
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
- Add v1 image test case
- Install protobuf-compiler in build check
- Reset containerd config to default in kubernetes test if we are testing genpolicy
- Update docker_credential crate
- Add test that uses default pull method
- Use GENPOLICY_PULL_METHOD in test
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
This PR improves the kbs_k8s_delete function to verify that the
resources were properly deleted for baremetal environments.
Fixes#9379
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
- At the moment we aren't supporting ppc64le or
aarch64 for
CoCo, so filter out these tests from running
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
- Add basic test case to check that a ruuning
pod can use the api-server-rest (and attestation-agent
and confidential-data-hub indirectly) to get a resource
from a remote KBS
Fixes#9057
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Co-authored-by: Linda Yu <linda.yu@intel.com>
Co-authored-by: stevenhorsman <steven@uk.ibm.com>
Co-authored-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Use the "allow all" policy for k8s-sandbox-vcpus-allocation.bats,
instead of relying on the Kata Guest image to use the same policy
as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-nginx-connectivity.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-volume.bats, instead of relying
on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-sysctls.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-security-context.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-seccomp.bats, instead of relying
on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-projected-volume.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-pod-quota.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-optional-empty-secret.bats,
instead of relying on the Kata Guest image to use the same policy as
its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-measured-rootfs.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-liveness-probes.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-inotify.bats, instead of relying
on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-guest-pull-image.bats, instead of
relying on the Kata Guest image to use the same policy as its default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-footloose.bats, instead of
relying on the Kata Guest image to use the same policy as its
default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Use the "allow all" policy for k8s-empty-dirs.bats, instead of
relying on the Kata Guest image to use the same policy as its
default.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Check from:
- k8s-exec-rejected.bats
- k8s-policy-set-keys.bats
if policy testing is enabled or not, to reduce the complexity of
run_kubernetes_tests.sh. After these changes, there are no policy
specific commands left in run_kubernetes_tests.sh.
add_allow_all_policy_to_yaml() is moving out of run_kubernetes_tests.sh
too, but it not used yet. It will be used in future commits.
Fixes: #9395
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Don't add the "allow all" policy to all the test YAML files anymore.
After this change, the k8s tests assume that all the Kata CI Guest
rootfs image files either:
- Don't support Agent Policy at all, or
- Include an "allow all" default policy.
This relience/assumption will be addressed in a future commit.
Fixes: #9395
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
the service might not listen on the default port, use the full service
address to ensure we are talking to the right resource.
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
this can be used on kcli or other systems where cluster nodes are
accessible from all places where the tests are running.
Fixes: #9272
Signed-off-by: Lukáš Doktor <ldoktor@redhat.com>
Since v2.2.6 it can detect TDX guests on Azure, so let's bump it even if
Azure peer-pods are not currently used as part of our CI.
Fixes: #9348
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This PR fixes the unbound variables error when trying to run
the setup script locally in order to avoid errors.
Fixes#9328
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This PR updates the journal log names for kubernetes artifacts
in order to make sure that we have different names when we are
running parallel GHA jobs.
Fixes#9308
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This PR terminates the nydus namespace to avoid the error of
that the flag needs an argument.
Fixes#9264
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Those two architectures are not TEE capable, thus we can just skip
running those tests there.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This is needed as the TDX machine is hosted inside Intel and relies on
proxies in order to connect to the external world. Not having those set
causes issues when pulling the image inside the guest.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Add a test case of pulling image inside the guest for confidential
containers.
Signed-off-by: Da Li Liu <liudali@cn.ibm.com>
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
Co-authored-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Co-authored-by: stevenhorsman <steven@uk.ibm.com>
Co-authored-by: Georgina Kinge <georgina.kinge@ibm.com>
Co-authored-by: Megan Wright <Megan.Wright@ibm.com>
This commit updates `filter_k8s_test.sh` to handle skipped tests that
include comments. In addition to the existing parameter expansion,
the following expansions have been added:
- Removal of a comment
- Stripping of trailing spaces
Fixes: #9304
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Initialize $GITHUB_ENV to avoid nounset error when running the scripts locally
out of Github Actions.
Fixed commit 9ba5e3d2a8Fixes#9217
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Recently confidential-containers/kbs repository was renamed to
confidential-containers/trustee. Github will automatically resolve the
old URL but we better adjust it in code.
The trustee repository will be cloned to $COCO_TRUSTEE_DIR. Adjusted
file paths and pushd/popd's to use $COCO_KBS_DIR
($COCO_TRUSTEE_DIR/kbs).
On versions.yaml changed from `coco-kbs` to `coco-trustee` as in the
future we might need other trustee components, so keeping it generic.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Added the kbs_set_resources_policy() function to set the KBS policy. Also the
kbs_set_allow_all_resources() and kbs_set_deny_all_resources to set the
"allow all" and "deny all" policy, respectively.
Fixes#9056
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Added utility functions to manage resources in KBS:
- kbs_set_resource(), where the resource data is passed via argument
- kbs_set_resource_from_file(), where the resource data is found in a
file
Fixes#9056
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Added kbs_install_cli function to build and install the kbs-client
executable if not present into the system.
Removed the stub from gha-run.sh; now the install kbs-client in the
.github/workflows/run-kata-deploy-tests-on-aks.yaml will effectively
install the executable.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Added functions to return the service host, port or full-qualified
HTTP address, respectively, kbs_k8s_svc_host(), kbs_k8s_svc_port(),
and kbs_k8s_svc_http_addr().
Fixes#9056
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The k3s distribution of k8s uses an embedded version of containerd and
configures it to log to a file, not the journal. Hence, although we
collect the journal as a test artifact, we also need to collect the
actual log files for containerd.
Also collect the k3s containerd config files to help with debugging.
Fixes: #9104.
Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>
The jobs running on garm will collect journal information. The data gathered
is based on the time the tests started running. The $start_time is
exported on run_tests() and used in collect_artifacts(). It happens that
run_tests() and collect_artifacts() are called on different steps of the
workflow and the environment variables aren't preserved between them,
i.e, $start_time exported on the first step is not available on the
subsequents.
To solve that issue, let's save $start_time in the file pointed out by
$GITHUB_ENV that Github actions uses to export variables. In case $GITHUB_ENV is
empty then probably it is running locally outside of Github, so it won't
save the start time value.
Fixes#9217
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Until this point the deployed KBS service is only reachable from within
the cluster. This introduces a generic mechanism to apply an Ingress
configuration to expose the service externally.
The first implemened ingress is for AKS. In case the HTTP application
routing isn't enabled in the cluster (this is required for ingress), an
add-on is applied.
It was added the get_cluster_specific_dns_zone() and
enable_cluster_http_application_routing() helper functions
to gha-run-k8s-common.sh.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Introduce the tests/integration/kubernetes/confidential_kbs.sh library
that contains functions to manage the KBS on CI. Initially implemented
the kbs_k8s_deploy() and kbs_k8s_delete() functions to, respectively,
deploy and delete KBS on Kubernetes. Also hooked those functions in the
tests/integration/kubernetes/gha-run.sh script to follow the convention
of running commands from Github Workflows:
$ .tests/integration/kubernetes/gha-run.sh deploy-coco-kbs
$ .tests/integration/kubernetes/gha-run.sh delete-coco-kbs
Fixes#9058
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This PR adds the collect artifacts function in gha-run script for
the kubernetes tests.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Currently, the checking for kata-deploy is running assume that the
daemonset scheduled at least one pod, however it might not had and the
kubectl wait command fails due to "error: no matching resources found".
On CI I've observed that fail intermittently. I suspect the service
account kata-deploy-sa take a while to show up then no kata-deploy is
scheduled in meanwhile.
Changed the checker logic to use waitForProcess() to keep testing if it is
already running, or hit the timeout (still 10m).
Fixes#9183
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Kubernetes v1.29 introduced a new `PodReadyToStartContainers` condition
that gets inserted at index 0 in the conditions array. This means that
the expected `PodCompleted` reason can now be either at index 0 with
kubernetes v1.28 and older or at index 1 starting with kubernetes v1.29.
This is fragile at best since the `kubectl wait` doesn't allow to combine
multiple checks. Also, checking the reason is dubious as it doesn't really
tell if the pods have actually completed or not.
Check the pod phase to be `Succeeded` instead, this guarantees that :
> All containers in the Pod have terminated in success, and will not
> be restarted.
Fixes#9178
Signed-off-by: Greg Kurz <groug@kaod.org>
Changed the "run k8s tests on AKS" workflows to get the CoCo KBS
installed so that we can run attestation tests.
The plan is to run attestation tests only on a subset of non-TEE jobs
initially, so this commit restricts to install KBS only on kata-qemu
configuration. Actually at this point it is added only stubs commands
to tests/integration/kubernetes/gha-run.sh that should be implemented
in a future commit.
Fixes#9058
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
CI failed to deploy nydus snapshotter because it was not cleaned up last time.
So we can try to cleanup nydus snapshotter before deploying it.
Fixes: #9121
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
Co-authored-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
teardown() gets executed after each test case, so there is no need to
clean-up before teardown.
Fixes: #9072
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Add set_namespace_to_policy_settings() for changing the pod namespace
in genpolicy settings.
Fixes: #9072
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
This has been introduced by 53bc4a432b,
where the condition was changed.
The correct condition is:
* If the list of supported tees does not contain the kata hypervisor
and the list of supported non tees does not contain the kata
hypervisor.
The error is that we were checking whether kata-hypervisor would contain
the list of supported tees, and that would almost always be false
(unless in the case where the list had an one and only one element).
Fixes: #9055 -- part II
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
A test `vcpu allocation k8s test` exhibits different behavior on s390x
For more details, please refer to issue #9093.
This commit is to make the test skipped until the issue is resolved on
the platform.
Fixes: #9093
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
This PR adds the ability to run k8s confidential tests in a
non-TEE environment.
Fixes#9055
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This will ensure no leftovers are in the node, which has been cause the
TDX CI to fail every now and then.
Fixes: #9081
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Delete the debugger pod created during the test, rather than already
existing debugger pods.
Also, send the output of "kubectl delete" to stderr, just in case it's
useful for debugging.
Fixes: #9069
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
We can use daemonset to deploy nydus snapshotter, which will decrease
one manual step both for Kata Containers and Confidential Containers CI.
Fixes: #8584
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
This PR replaces the add_kernel_initrd_annotations_to_yaml function
more generic so later can be used for other components.
Fixes#9054
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
1. add_requests_to_policy_settings allows one or more ttrpc requests
from the Host to the Guest. Example:
add_requests_to_policy_settings "${policy_settings_dir}" \
"ReadStreamRequest" "WriteStreamRequest"
2. add_copy_from_host_to_policy_settings allows executing on the Guest
the commands initiated behind the scenes by "kubectl cp" from the
Host to the Guest. Example:
add_copy_from_host_to_policy_settings "${policy_settings_dir}"
3. add_copy_from_guest_to_policy_settings allows executing on the Guest
the commands initiated behind the scenes by "kubectl cp" from the
Guest to the Host. Example:
add_copy_from_guest_to_policy_settings "${policy_settings_dir}" \
"/tmp/file.txt"
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
1. Rename install_kata_common to install_kata_core.
2. Add TODO for better way to install the Kata tools.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Automatically generate the test policy for k8s-attach-handlers.bats,
if AUTO_GENERATE_POLICY is enabled.
Steps:
- Create a temporary directory for the current test and copy the
common genpolicy settings into this new directory.
- Change genpolicy settings in the temp directory to allow the
"kubectl exec" command that this test needs. (For CoCo, exec is
blocked by the default policy settings)
- Auto-generate the policy for the test YAML file.
- Test as usual, using the YAML file.
- Clean-up the temporary settings described above.
Fixes: #8921
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Enable AUTO_GENERATE_POLICY for one of the Kata CI K8s test platforms.
Additional platforms will be enabled after testing them.
When AUTO_GENERATE_POLICY is enabled, create genpolicy settings that
are common for all tests. Some of the tests will make temporary copies
of these common settings and customize them as needed.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
1. Avoid repeating "kata-containers-k8s-tests".
2. Allow users to specify a different test namespace.
3. Introduce the TEST_CLUSTER_NAMESPACE variable, that will also be
useful when auto-generating the Agent Policy for these tests.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
This PR skips the k8s tests that are not working with cloud hypervisor
runtime-rs with its proper issue.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This PR is a split of #8585.
make the changes on the Github workflows, and the skeleton to deploy_snapshotter()
and cleanup_snapshotter() in tests/integration/kubernetes/gha-run.sh in this commit.
After initially merging this patch to trigger CI jobs for CoCo, which will begin executing
the dummy functions deploy_snapshotter() and cleanup_snapshotter(), the implementation details for these functions
remain in #8585. Our subsequent step involves transferring this logic to the PR #8484, enabling the PR to undergo CI testing prior to its merge.
Fixes: #8997
Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
This PR adds workflow for running kubernetes test suite on ppc64le.
It uses scripts to create and delete the cluster using kubeadm as none of the current cluster creation tools are supported on Power.
Fixes: #7950
Signed-off-by: Amulyam24 <amulmek1@in.ibm.com>
The existing confidential basic test titled `Test unencrypted
confidential container launch success and verify that we are
running in a secure enclave` has been updated to incorporate
IBM Secure Execution (`qemu-se`).
Previously, a secure image was absent from kata-deploy, hindering
the inclusion of IBM SE in the test.
Thanks to the #6755 update, it is now possible to test the TEE.
This modification extends the existing test by introducing
`qemu-se`. The specific changes are outlined below:
- Add an additional test `cc-se-e2e-tests` to s390x nightly
- Expansion of `REMOTE_COMMAND_PER_HYPERVISOR` for `qemu-se`
- Temporary exclusion of two test cases currently incompatible with IBM SE
(`cpu-ns` is a common issue across all TEEs, while `inotify`
will be addressed in a subsequent pull request).
Fixes: #8913
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
Add --show-output-of-passing-tests to the k8s integration tests. The
output of a passing test can be helpful when investigating a failure
of the same test.
Fixes: #8885
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
The changes to install and test genpolicy must come later, after CI
picks up these gha changes.
Fixes: #8856
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Log the list of the current pods between tests because these pods
might be related to cluster nodes occasionally running out of memory.
Fixes: #8769
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
To become more resilient against these kinds of errors:
deployment.apps/confidential-unencrypted created
pod/confidential-unencrypted-c5fdd6964-rrb6q condition met
ssh: connect to host 10.42.0.109 port 22: Connection refused
Fixes: #8687
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
This PR fixes the indentation of the confidential common
script for kubernetes tests.
Fixes#8698
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
This will make the life easier for dragonball developers to properly
enable the tests once the tests are ready.
Fixes: #8569
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Print more information useful for debugging. Also, use a separate YAML
file for this test, instead of reusing someone else's file.
Fixes: #8270
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Implements the following test case:
Scenario: Check incorrect hash fails
**Given** I have a version of kata installed that has a kernel with the
initramfs built and config with rootfs_verity.scheme=dm-verity
rootfs_verity.hash=<incorrect hash of rootfs> set in the kernel_params
**When** I try and create a container a basic pod
**Then** The pod is doesn't run
**And** Ideally we'd get a helpful message to indicate why
Currently on CI only qemu-tdx is built with measured
rootfs support in the kernel, so the test is restriced to that
runtimeclass.
Fixes#7415
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Bring the setup_common() from CCv0 branch test's
integration/kubernetes/confidential/tests_common.sh. It should be used
to reduce boilerplates on the setup() of the tests.
Unlike the original code, this won't export the `test_start_time` variable
as it wouldn't be accurate to grab logs from the worker nodes due
date/time mismatch between the running tests machine and the worker
node. The function export the `node` variable which holds the name of
a random node which has kata installed. Apart from that, it exports the
`node_start_time` which capture the date/time when the test started,
relative to the `node`.
Tests that should inspect the logs can schedule pods/resources to the `node`
and use `node_start_time` as the value reference to grep the logs.
Fixes#7590
Co-authored-by: stevenhorsman <steven@uk.ibm.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Bring the assert_logs_contain() from CCv0 branch tests'
integration/kubernetes/confidential/lib.sh.
Introduced the print_node_journal() which uses `kubectl debug` to print
the systemd's journal of a k8s's node.
Fixes#7590
Co-authored-by: stevenhorsman <steven@uk.ibm.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This new function allow to the annotations to metadata section in a yaml
configuration file.
Co-authored-by: Ryan Savino <ryan.savino@amd.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Copied the new_pod_config() and pod-config.yaml.in from CCv0 branch
tests' integration/kubernetes/confidential/tests_common.sh and fixtures.
Unlike the original version, new_pod_config() now gets the runtimeclass
by parameter as the RUNTIMECLASS environment variable seems not broadly
used on main branch's CI.
The pod-config.yaml.in was changed as the diff shows below. In
particular the imagePullSecrets was removed to avoid it throwing a
warning on the pod's log.
```
--- a/tests/integration/kubernetes/runtimeclass_workloads/pod-config.yaml.in
+++ b/tests/integration/kubernetes/runtimeclass_workloads/pod-config.yaml.in
@@ -5,12 +5,10 @@
apiVersion: v1
kind: Pod
metadata:
- name: busybox-cc
+ name: test-e2e
spec:
runtimeClassName: $RUNTIMECLASS
containers:
- - name: nginx
+ - name: test_container
image: $IMAGE
- imagePullPolicy: Always
- imagePullSecrets:
- - name: cococred
\ No newline at end of file
+ imagePullPolicy: Always
\ No newline at end of file
```
Co-authored-by: Georgina Kinge <georgina.kinge@ibm.com>
Co-authored-by: Megan Wright <Megan.Wright@ibm.com>
Co-authored-by: stevenhorsman <steven@uk.ibm.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The following functions were copied from CCv0's branch test's
integration/kubernetes/confidential/lib.sh. I did just smalls
refactorings (shortened their names and delinted shellcheck warnings):
- k8s_delete_all_pods_if_any_exists()
- k8s_wait_pod_be_ready()
- k8s_create_pod()
- assert_pod_fail()
Co-authored-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Co-authored-by: Georgina Kinge <georgina.kinge@ibm.com>
Co-authored-by: Jordan Jackson <jordan.jackson@ibm.com>
Co-authored-by: Megan Wright <Megan.Wright@ibm.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Co-authored-by: Wang, Arron <arron.wang@intel.com>
As part of the CI migration, this PR is to add workflows for containerd and k8s for s390x.
Fixes: #7930
Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
As we've done some changes in the VMM vcpu allocation, let's introduce
basic tests to make sure that we're getting the expected behaviour.
The test consists in checking 3 scenarios:
* default_vcpus = 0 | no limits set
* this should allocate 1 vcpu
* default_vcpus = 0.75 | limits set to 0.25
* this should allocate 1 vcpu
* default_vcpus = 0.75 | limits set to 1.2
* this should allocate 2 vcpus
The tests are very basic, but they do ensure we're rounding things up to
what the new logic is supposed to do.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
The current kata-deploy code has been doing a `sed` to add allowed
hypervisor annotations, so CBL mariner can be tested with their own
kernel and initrd.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Encode policy file during test - easier to understand than hard-coding
the encoded file contents.
Fixes: #8214
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
The KUBERNETES variable is mostly used by kata-deploy whether to apply
k3s specific deployments or not. It is used to select the type of
kubernetes to be installed (k3s, k0s, rancher...etc) and it is always
set on CI. Running the script locally we want to set a value by default
to avoid `KUBERNETES: unbound variable` errors.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test can give false-positive on a multi-node cluster. Changed it to
use the new get_one_kata_node() and the modified exec_host() to run the
setup commands on a given node (that has kata installed) and ensure the
test pod is scheduled at that same node.
Fixes#7619
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
This test can give false-positive on a multi-node cluster. Changed it to
use the new get_one_kata_node() and the modified exec_host() to run the
setup commands on a given node (that has kata installed) and ensure the
test pod is scheduled at that same node.
Fixes#7619
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The exec_host() simply fails on cluster with multi-nodes because
`kubectl get node -o name" will return a list o names. Moreover, it will
return control nodes names which usually don't have kata installed.
Fixes#7619
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The introduced get_one_kata_node() returns the first node that
has the kata-runtime=true label, i.e., supposedly a node with
kata installed.
This is useful for tests that should run on a determined worker
node on a multi-nodes cluster.
Fixes#7619
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Let KATA_HYPERVISOR be qemu by default in gh-run.sh as this variable
is required to tweak some configurations of kata-deploy.
Fixes#7620
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The deploy-kata() of gha-run.sh will wait for 10 minutes for the kata
deploy installation finish. This allow users of the script to overwrite
that value by exporting the KATA_DEPLOY_WAIT_TIMEOUT environment
variable.
Fixes#7620
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Fixed a couple of warns shellcheck emitted and disabled others:
* SC2154 (var is referenced but not assigned)
* SC2086 (Double quote to prevent globbing and word splitting)
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The only difference to the other platforms is that it needs to
export KUBECONFIG.
Fixes#7620
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The cleanup-kcli() behaves like other deploy kata for
bare-metal (e.g. sev, tdx...etc) except that KUBECONFIG
should be exported.
Fixes#7620
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The cleanup-kcli() behaves like other clean up for bare-metal (e.g. sev,
tdx...etc) except that KUBECONFIG should be exported.
Fixes#7620
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
On CI workflows the variables DOCKER_REGISTRY, DOCKER_REPO and
DOCKER_TAG are exported to match the built image. However, when running
the script outside of CI context, a developer might just use the latest
image which in this case will be
`quay.io/kata-containers/kata-deploy-ci:kata-containers-latest`.
Fixes#7620
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Adapted the gha-run.sh script to create a Kubernetes cluster locally
using the kcli tool.
Use `./gha-run.sh create-cluster-kcli` to create it, and
`./gha-run.sh delete-cluster-kcli` to delete.
Fixes#7620
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
The k8s.gcr.io is deprecated for a while now and has been redirected to
registry.k8s.io. However on some bare-metal machines in our testing
pools that redirection is not working, so let's just replace the
registries.
Fixes#8098
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
(cherry picked from commit b2c3bca558c38deff2117d5909d9071c23c05590)
We've faced this as part of the CI, only happening with the CRI-O tests:
```
not ok 1 Test readonly volume for pods
# (from function `exec_host' in file tests_common.sh, line 51,
# in test file k8s-file-volume.bats, line 25)
# `exec_host "echo "$file_body" > $tmp_file"' failed with status 127
# [bats-exec-test:38] INFO: k8s configured to use runtimeclass
# bash: line 1: $'\r': command not found
#
# Error from server (NotFound): pods "test-file-volume" not found
```
I must say I didn't dig into figuring out why this is happening, but we
may be safe enough to just trail the '\r', as long as all the tests keep
passing on containerd.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This is based on official CRI-O documentations[0] and right now we're
making this specific to Ubuntu as that's what we have as runners.
We may want to expand this in the future, but we're good for now.
[0]:
https://github.com/cri-o/cri-o/blob/main/install.md#apt-based-operating-systems
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Otherwise only the first test will be executed
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let me start with a fair warning that this commit is hard to split into
different parts that could be easily tested (or not tested, just
ignored) without breaking pieces.
Now, about the commit itself, as we're on the run to reduce costs
related to our sponsorship on Azure, we can split the k8s tests we run
in 2 simple groups:
* Tests that can be run in the smaller Azure instance (D2s_v5)
* Tests that required the normal Azure instance (D4s_v5)
With this in mind, we're now passing to the tests which type of host
we're using, which allows us to select to run either one of the two
types of tests, or even both in case of running the tests on a baremetal
system.
Fixes: #7972
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
The list of tests which require a bigger VM instance is:
* k8s-number-cpus.bats -- failing on all CIs
* k8s-parallel.bats -- only failing on the cbl-mariner CI
* k8s-scale-nginx.bats -- only failing on the cbl-mariner CI
We'll keep those disabled while we re-work the logic to **only run
those** in a bigger (and more expensive) VM instance.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Without setting the cpu limit / request to 1, we can make this test run
in a smaller VM instance without any issue.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Fix kernel and initrd annotations in the k8s tests on Mariner. These
annotations must be applied to the spec.template for Deployment, Job
and ReplicationController resources.
Fixes: #7764
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
There's absolutely no need to have the skip check as part of the test
itself when it's already done as part of the setup function.
We're only touching the files here that were touched in the previous
commit.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's keep both checks for now, but in the future we'll be able to
remove the check for "firecracker", as the hypervisor name used as part
of the GitHub Actions has to match what's used as part of the
kata-deploy stuff, which is `fc` (as in `kata-fc for the runtime class)
instead of `firecracker`.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
We've been using the `kata-deploy-tdx` target as that also uses k3s as
base, but it's better to just have a specific garm target.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
So we have a better control on which flavour of kubernetes kata-deploy
is expected to be targetting.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
As we were using `tee` without the `-a` (or `--apend`) aptton, the
containerd config would be overwritten, leading to a NotReady state of
the Node.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's download the vanilla kubectl binary into `/usr/bin/`, as we need
to avoid hitting issues like:
```sh
error: open /etc/rancher/k3s/k3s.yaml.lock: permission denied
```
The issue basically happens because k3s links `/usr/local/bin/kubectl`
to `/usr/local/bin/k3s`, and that does extra stuff that vanilla
`kubectl` doesn't do.
Also, in order to properly use the k3s.yaml config with the vanilla
kubectl, we're copying it to ~/.kube/config.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Otherwise the /etc/rancher/k3s/k3s.yaml is not readable by other users
than root.
As --write-config-mode is being passed, and that's an option that has to
be passed to the `server`, -s is also added to the command line.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
`wait` waits for a job to complete, not a number of seconds. Not sure
how I got that wrong in the first place, but it's what it's.
Fixes: #6542
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
This function right now is completely based on what's part of the tests
repo[0], and that's the reason I'm keeping the `Signed-off-by` of all
the contributors to that file.
This is not perfect, though, as it changes the default snapshotter to
devmapper, instead of only doing so for the Kata Containers specific
runtime handlers. OTOH, this is exactly what we've always been doing as
part of the tests.
We'll improve it, soon enough, when we get to also add a way for
kata-deploy to set up different snapshotters for different handlers.
But, for now, this is as good (or as bad) as it's always been.
It's important to note that the devmapper setup doesn't take into
consideration a BM machine, and this is not suitable for that. We're
really only targetting GHA runners which will be thrown away after the
run is over.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Signed-off-by: Shiming Zhang <wzshiming@foxmail.com>
Signed-off-by: Marcel Apfelbaum <marcel@redhat.com>
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
One can use different kubernetes flavours for getting a kubernetes
cluster up and running.
As part of our CI, though, I really would like to avoid contributors
spending time maintaining and updating kubernetes dependencies, as done
with the tests repo, and which has been proven to be really good on
getting things rotten.
With this in mind, I'm taking the bullet and using "k3s" as the way to
deploy kubernetes for the devmapper related tests, and that's the reason
I'm adding a function to do so, and this will be used later on as part
of this series.
It's important to note that the k3s setup doesn't take into
consideration a BM machine, and this is not suitable for that. We're
really only targetting GHA runners which will be thrown away after the
run is over.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Use AGENT_POLICY=yes when building the Guest images, and add a
permissive test policy to the k8s tests for:
- CBL-Mariner
- SEV
- SNP
- TDX
Also, add an example of policy rejecting ExecProcessRequest.
Fixes: #7667
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Let's expand the confidential test to also support TDX.
The main difference on the test, though, is that we're not grepping for
a string in the `dmesg` output, but rather relying on `cpuid` to detect
a TDX guest.
Fixes: #7184
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Add a test case for the launch of unencrypted confidential
container, verifying that we are running inside a TEE.
Right now the test only works with SEV, but it'll be expanded in the
coming commits, as part of this very same series.
Fixes: #7184
Signed-Off-By: Unmesh Deodhar <udeodhar@amd.com>
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
k8s-pid-ns.bats was already using the test name from
k8s-kill-all-process-in-container.bats - probably a copy/paste bug.
Fixes: #7753
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
By doing this we can make sure there won't be any clash on the cluster
name created for either the k8s or the kata-deploy tests.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
In a follow-up series, we'll add a whole suite for the kata-deploy
tests. With this in mind, let's already get rid of this one and avoid
more kata-deploy tests to land here.
Fixes: #7642
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
The default `kata` runtime class would get created with the `kata`
handler instead of `kata-$KATA_HYPERVISOR`. This made Kata use the wrong
hypervisor and broke CI.
Fixes: #7663
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Let's split a good portion of `tests/integration/kuberentes/gha-run.sh`
out, and put them in a place where they can be used to the soon-to-come
kata-deploy specific tests.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
With these 2 simple checks we can ensure that we do not regress on the
behaviour of allowing the runtime classes / default runtime class to be
created by the kata-deploy payload.
Fixes: #7491
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Let's add here the image we'll be using for unencrypted confidential
tests. Later on, we'll make sure to build and use this image as part of
our CI.
The image can easily be built as a multi-arch image, and has `cpuid`
installed in case of `x86_64` build, so it can be used to detect whether
we're running on a TEE guest without having to rely on `dmesg | grep
...`.
Fixes: #7595
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
We don't have to do any sed to replace the runtimeclass being used by
the moment we start taking advantage of the `DEFAULT_SHIM` environment
variable exposed merged in the previous commits.
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
Instead of using package manager to install bats, building
this from source. This gives us the updated version of bats
which supports functions such as setup_file and
teardown_file.
We can use these functions into our current tests.
Fixes: #7597
Signed-off-by: Unmesh Deodhar <udeodhar@amd.com>
Otherwise the VM deletion may not delete, leaving us with several
machines behind.
Fixes: #7509
Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
