713 Commits

Author SHA1 Message Date
Dan Mihai
0a4fc071ac tests: k8s: k8s-custom-dns auto-generated policy
Auto-generate policy for k8s-custom-dns.bats.

Fixes: #9072

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-21 18:08:07 +00:00
Dan Mihai
f693f49e92 tests: k8s: k8s-credentials-secrets policy
Auto-generate policy for k8s-credentials-secrets.bats.

Fixes: #9072

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-21 18:08:07 +00:00
Dan Mihai
d3d27bbb5b tests: k8s: k8s-configmap auto-generated policy
Auto-generate policy for k8s-configmap.bats.

Fixes: #9072

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-21 18:08:07 +00:00
Dan Mihai
b318535536 tests: k8s: auto-generate k8s-caps.bats policy
Auto-generated policy for k8s-caps.bats.

Fixes: #9072

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-21 18:08:07 +00:00
Fabiano Fidêncio
3468ac3b6e ci: k8s: Fix checks used to skip confidential tests
This has been introduced by 53bc4a432b,
where the condition was changed.

The correct condition is:
* If the list of supported tees does not contain the kata hypervisor
  and the list of supported non tees does not contain the kata
  hypervisor.

The error is that we were checking whether kata-hypervisor would contain
the list of supported tees, and that would almost always be false
(unless in the case where the list had an one and only one element).

Fixes: #9055 -- part II

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
2024-02-18 10:10:45 +01:00
Hyounggyu Choi
8b3f7f353d CI|k8s: Skip vcpu allocation test for s390x
A test `vcpu allocation k8s test` exhibits different behavior on s390x
For more details, please refer to issue #9093.
This commit is to make the test skipped until the issue is resolved on
the platform.

Fixes: #9093

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
2024-02-15 12:26:35 +01:00
GabyCT
9cf343779f Merge pull request #9062 from GabyCT/topic/nonteet
tests: Add ability to run non-TEE environments
2024-02-13 14:28:07 -06:00
Gabriela Cervantes
53bc4a432b tests: Add ability to run non-TEE environments
This PR adds the ability to run k8s confidential tests in a
non-TEE environment.

Fixes #9055

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2024-02-13 17:27:55 +00:00
Fabiano Fidêncio
3877a9f49a ci: Clean up kata-deploy ds before starting the tests
This will ensure no leftovers are in the node, which has been cause the
TDX CI to fail every now and then.

Fixes: #9081

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
2024-02-13 14:10:44 +01:00
GabyCT
00be9ae872 Merge pull request #9070 from microsoft/danmihai1/debug-containers
tests: k8s: avoid deleting unrelated pods
2024-02-12 15:24:15 -06:00
Dan Mihai
a21ca9b7c9 tests: k8s: avoid deleting unrelated pods
Delete the debugger pod created during the test, rather than already
existing debugger pods.

Also, send the output of "kubectl delete" to stderr, just in case it's
useful for debugging.

Fixes: #9069

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-09 22:48:41 +00:00
Dan Mihai
a054462eb7 Merge pull request #9051 from microsoft/danmihai1/k8s-copy-file
tests: k8s: k8s-copy-file auto-generated policy
2024-02-09 12:30:49 -08:00
ChengyuZhu6
97fbf360cc gha: Cleanup nydus snapshotter by the daemonset
Cleanup nydus snapshotter by the daemonset.

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
2024-02-09 14:47:13 +01:00
ChengyuZhu6
43b04fd0c0 gha: Deploy nydus snapshotter by the daemonset
We can use daemonset to deploy nydus snapshotter, which will decrease
one manual step both for Kata Containers and Confidential Containers CI.

Fixes: #8584

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
2024-02-09 14:47:09 +01:00
Gabriela Cervantes
0b508f301b tests:k8s: make add_kernel_initrd_anotations function generic
This PR replaces the add_kernel_initrd_annotations_to_yaml function
more generic so later can be used for other components.

Fixes #9054

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2024-02-08 19:30:43 +00:00
Dan Mihai
f139c7dc60 tests: k8s: k8s-copy-file auto-generated policy
Auto-generate policy for k8s-copy-file.bats.

Fixes: #9050

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-08 13:26:05 +00:00
Dan Mihai
1179306afa tests: k8s: additional policy testing utilities
1. add_requests_to_policy_settings allows one or more ttrpc requests
   from the Host to the Guest. Example:

add_requests_to_policy_settings "${policy_settings_dir}" \
   "ReadStreamRequest" "WriteStreamRequest"

2. add_copy_from_host_to_policy_settings allows executing on the Guest
   the commands initiated behind the scenes by "kubectl cp" from the
   Host to the Guest. Example:

add_copy_from_host_to_policy_settings "${policy_settings_dir}"

3. add_copy_from_guest_to_policy_settings allows executing on the Guest
   the commands initiated behind the scenes by "kubectl cp" from the
   Guest to the Host. Example:

add_copy_from_guest_to_policy_settings "${policy_settings_dir}" \
   "/tmp/file.txt"

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-08 13:25:41 +00:00
Dan Mihai
6b5e57f7c7 tests: k8s: address PR review feedback
1. Rename install_kata_common to install_kata_core.

2. Add TODO for better way to install the Kata tools.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 18:51:56 +00:00
Dan Mihai
dd16bc393f tests: k8s: k8s-attach-handlers generated policy
Automatically generate the test policy for k8s-attach-handlers.bats,
if AUTO_GENERATE_POLICY is enabled.

Steps:

- Create a temporary directory for the current test and copy the
  common genpolicy settings into this new directory.

- Change genpolicy settings in the temp directory to allow the
  "kubectl exec" command that this test needs. (For CoCo, exec is
  blocked by the default policy settings)

- Auto-generate the policy for the test YAML file.

- Test as usual, using the YAML file.

- Clean-up the temporary settings described above.

Fixes: #8921

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:26:03 +00:00
Dan Mihai
0de407f8b7 tests: k8s: enable AUTO_GENERATE_POLICY
Enable AUTO_GENERATE_POLICY for one of the Kata CI K8s test platforms.
Additional platforms will be enabled after testing them.

When AUTO_GENERATE_POLICY is enabled, create genpolicy settings that
are common for all tests. Some of the tests will make temporary copies
of these common settings and customize them as needed.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:25:54 +00:00
Dan Mihai
05b2e4f606 tests: k8s: install genpolicy
Install the genpolicy app before starting test execution.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:25:42 +00:00
Dan Mihai
8aa8b70573 tests: k8s: add policy test utilities
Add script functions useful for auto-generating and testing policy.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:24:06 +00:00
Dan Mihai
24a17a2e1b tests: k8s: output the names of test files
Output the names of test files, for easier search through logs.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:23:54 +00:00
Dan Mihai
bf533de31a tests: k8s: add DEBUG support for test scripts
Make these scripts easier to debug.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:23:46 +00:00
Dan Mihai
1b4ef672ef tests: k8s: reduce namespace name duplication
1. Avoid repeating "kata-containers-k8s-tests".
2. Allow users to specify a different test namespace.
3. Introduce the TEST_CLUSTER_NAMESPACE variable, that will also be
   useful when auto-generating the Agent Policy for these tests.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:23:38 +00:00
Dan Mihai
8a5ba5fb34 tests: k8s: allow run_kubernetes_tests.sh exec
Allow everyone to directly execute run_kubernetes_tests.sh, for easier
local testing.

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-02-07 02:23:30 +00:00
GabyCT
6337f300a8 Merge pull request #8628 from GabyCT/topic/enablek8stclh
tests: k8s: Enable tests for cloud hypervisor runtime-rs without devicemapper
2024-02-06 14:28:35 -06:00
Gabriela Cervantes
cf049fc718 k8s: Skip k8s tests that are not working
This PR skips the k8s tests that are not working with cloud hypervisor
runtime-rs with its proper issue.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2024-02-06 16:52:02 +00:00
ChengyuZhu6
a214bd8d13 gha: Enable nydus snapshotter in CoCo ci tests
This PR is a split of #8585.
make the changes on the Github workflows, and the skeleton to deploy_snapshotter()
and cleanup_snapshotter() in tests/integration/kubernetes/gha-run.sh in this commit.

After initially merging this patch to trigger CI jobs for CoCo, which will begin executing
the dummy functions deploy_snapshotter() and cleanup_snapshotter(), the implementation details for these functions
remain in #8585. Our subsequent step involves transferring this logic to the PR #8484, enabling the PR to undergo CI testing prior to its merge.

Fixes: #8997

Signed-off-by: ChengyuZhu6 <chengyu.zhu@intel.com>
2024-02-05 18:51:59 +08:00
Gabriela Cervantes
eb5b7d3bf8 tests: k8s: Enable tests for cloud hypervisor runtime-rs
This PR enable the k8s tests for cloud hypervisor runtime-rs.

Fixes #8627

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2024-02-02 17:58:58 +00:00
Amulyam24
f8585db8d9 gha: add kubernetes tests workflow for ppc64le
This PR adds workflow for running kubernetes test suite on ppc64le.

It uses scripts to create and delete the cluster using kubeadm as none of the current cluster creation tools are supported on Power.

Fixes: #7950

Signed-off-by: Amulyam24 <amulmek1@in.ibm.com>
2024-02-01 12:23:11 +05:30
Hyounggyu Choi
ab462a4b89 tests: Add IBM SE to the basic confidential test
The existing confidential basic test titled `Test unencrypted
confidential container launch success and verify that we are
running in a secure enclave` has been updated to incorporate
IBM Secure Execution (`qemu-se`).
Previously, a secure image was absent from kata-deploy, hindering
the inclusion of IBM SE in the test.
Thanks to the #6755 update, it is now possible to test the TEE.

This modification extends the existing test by introducing
`qemu-se`. The specific changes are outlined below:

- Add an additional test `cc-se-e2e-tests` to s390x nightly
- Expansion of `REMOTE_COMMAND_PER_HYPERVISOR` for `qemu-se`
- Temporary exclusion of two test cases currently incompatible with IBM SE
(`cpu-ns` is a common issue across all TEEs, while `inotify`
will be addressed in a subsequent pull request).

Fixes: #8913

Signed-off-by: Hyounggyu Choi <Hyounggyu.Choi@ibm.com>
2024-01-26 06:04:39 +01:00
Dan Mihai
66c012d052 tests: k8s: bats --show-output-of-passing-tests
Add --show-output-of-passing-tests to the k8s integration tests. The
output of a passing test can be helpful when investigating a failure
of the same test.

Fixes: #8885

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-01-24 03:04:28 +00:00
Dan Mihai
ea9c659d36 gha: get ready to install genpolicy
The changes to install and test genpolicy must come later, after CI
picks up these gha changes.

Fixes: #8856

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-01-19 23:37:49 +00:00
Dan Mihai
b7c31e3b98 tests: cbl-mariner: disable k8s-oom.bats
Disable k8s-oom.bats on cbl-mariner until it passes more often.

Fixes: #8824

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-01-14 17:39:25 +00:00
Dan Mihai
90c782f928 tests: list the current k8s pods
Log the list of the current pods between tests because these pods
might be related to cluster nodes occasionally running out of memory.

Fixes: #8769

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2024-01-05 16:41:43 +00:00
Dan Mihai
d916da15dd Merge pull request #8688 from microsoft/danmihai1/k8s-confidential
tests: retry connection to pod SSH server
2023-12-20 15:01:26 -08:00
Dan Mihai
8aa390279e tests: retry connection to pod SSH server
To become more resilient against these kinds of errors:

deployment.apps/confidential-unencrypted created
pod/confidential-unencrypted-c5fdd6964-rrb6q condition met
ssh: connect to host 10.42.0.109 port 22: Connection refused

Fixes: #8687

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2023-12-20 02:48:05 +00:00
Gabriela Cervantes
1469a5efca tests: k8s: Fix indentation in confidential common script
This PR fixes the indentation of the confidential common
script for kubernetes tests.

Fixes #8698

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2023-12-18 20:25:06 +00:00
Gabriela Cervantes
c92b14da97 tests: k8s: Fix indentation in setup script
This PR fixes the indentation of the kubernetes setup script.

Fixes #8675

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2023-12-14 16:26:22 +00:00
Gabriela Cervantes
ce694b905b tests: Fix indentation of gha-run script
This PR fixes the indentation of gha run script.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2023-12-07 16:56:19 +00:00
Gabriela Cervantes
33b300431e tests: Enable but do not run k8s tests for cloud hypervisor
This PR enables but do not run k8s tests for cloud hypervisor
for runtime-rs.

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
2023-12-07 16:39:15 +00:00
Fabiano Fidêncio
d149b9f9ca Merge pull request #7231 from wainersm/measured_rootfs-improvements
Build for measured rootfs improvements
2023-12-05 22:20:33 +01:00
Fabiano Fidêncio
05ce52d746 devmapper: dragonball: Enable, but do not run, the tests
This will make the life easier for dragonball developers to properly
enable the tests once the tests are ready.

Fixes: #8569

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>
2023-12-05 15:29:23 +01:00
Dan Mihai
96deea52f2 tests: more k8s-exec-rejected debug output
Print more information useful for debugging. Also, use a separate YAML
file for this test, instead of reusing someone else's file.

Fixes: #8270

Signed-off-by: Dan Mihai <dmihai@microsoft.com>
2023-11-29 18:05:15 +00:00
Wainer dos Santos Moschetta
48bdca4c49 tests/k8s: add k8s-measured-rootfs.bats
Implements the following test case:

  Scenario: Check incorrect hash fails
  **Given** I have a version of kata installed that has a kernel with the
  initramfs built and config with rootfs_verity.scheme=dm-verity
  rootfs_verity.hash=<incorrect hash of rootfs> set in the kernel_params
  **When** I try and create a container a basic pod
  **Then** The pod is doesn't run
  **And**  Ideally we'd get a helpful message to indicate why

Currently on CI only qemu-tdx is built with measured
rootfs support in the kernel, so the test is restriced to that
runtimeclass.

Fixes #7415
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
2023-11-28 11:21:54 -03:00
Wainer dos Santos Moschetta
1eae657b91 tests/k8s: add set_node() to lib.sh
Use this new function to set the node where the pod should be scheduled
to.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
2023-11-28 11:21:53 -03:00
Wainer dos Santos Moschetta
c6075c8627 tests/k8s: add setup common
Bring the setup_common() from CCv0 branch test's
integration/kubernetes/confidential/tests_common.sh. It should be used
to reduce boilerplates on the setup() of the tests.

Unlike the original code, this won't export the `test_start_time` variable
as it wouldn't be accurate to grab logs from the worker nodes due
date/time mismatch between the running tests machine and the worker
node. The function export the `node` variable which holds the name of
a random node which has kata installed. Apart from that, it exports the
`node_start_time` which capture the date/time when the test started,
relative to the `node`.

Tests that should inspect the logs can schedule pods/resources to the `node`
and use `node_start_time` as the value reference to grep the logs.

Fixes #7590
Co-authored-by: stevenhorsman <steven@uk.ibm.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
2023-11-28 11:21:53 -03:00
Wainer dos Santos Moschetta
220a2d9a15 tests/k8s: add assert_logs_contain() to lib.sh
Bring the assert_logs_contain() from CCv0 branch tests'
integration/kubernetes/confidential/lib.sh.

Introduced the print_node_journal() which uses `kubectl debug` to print
the systemd's journal of a k8s's node.

Fixes #7590
Co-authored-by: stevenhorsman <steven@uk.ibm.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
2023-11-28 11:21:53 -03:00
Wainer dos Santos Moschetta
9a9c7a5c6f tests/k8s: add set_metadata_annotation() to lib.sh
This new function allow to the annotations to metadata section in a yaml
configuration file.

Co-authored-by: Ryan Savino <ryan.savino@amd.com>
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
2023-11-28 11:21:53 -03:00