gha: k8s: Make the tests more reliable

We like it or not, every now and then we'll have to deal with flaky
tests, and our tests using GHA are not exempt from that fact.

With this simple commit, we're trying to improve the reliability of the
tests in a few different fronts:

* Giving enough time for the script used by kata-deploy to be executed
  * We've hit issues as the kata-deploy pod is considered "Ready" at the
    moment it starts running, not when it finishes the needed setup. We
    should also be looking on how to solve this on the kata-deploy side
    but, for now, let's ensure our tests do not break with the current
    kata-deploy behavior.

* Merging the "Deploy kata-deploy" and "Run tests" steps
  * We've hit issues re-running tests and seeing even more failures than
    the ones we're trying to debug, as a step will simply be taken as
    succeeded as part of the re-run, in case it was successful executed
    as part of the first run.  This causes issues with the kata-deploy
    deployment, as the tests would start running before even having the
    node set up for running Kata Containers.

Fixes: #6865 #6649

Signed-off-by: Fabiano Fidêncio <fabiano.fidencio@intel.com>

.github/workflows/run-k8s-tests-on-aks.yaml

@@ -61,7 +61,8 @@ jobs:
         run: |
           az aks get-credentials -g "kataCI" -n ${{ github.event.pull_request.number }}-${{ github.event.pull_request.head.sha }}-${{ matrix.vmm }}-amd64
 
-      - name: Deploy kata-deploy
+      - name: Run tests
+        timeout-minutes: 30
         run: |
           sed -i -e "s|quay.io/kata-containers/kata-deploy:latest|${{ inputs.registry }}/${{ inputs.repo }}:${{ inputs.tag }}|g" tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
           cat tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
@@ -72,9 +73,11 @@ jobs:
           kubectl -n kube-system wait --timeout=10m --for=condition=Ready -l name=kata-deploy pod
           kubectl apply -f tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
 
-      - name: Run tests
-        timeout-minutes: 30
-        run: |
+          # This is needed as the kata-deploy pod will be set to "Ready" when it starts running,
+          # which may cause issues like not having the node properly labeled or the artefacts
+          # properly deployed when the tests actually start running.
+          sleep 60s
+
           pushd tests/integration/kubernetes
           sed -i -e 's|runtimeClassName: kata|runtimeClassName: kata-${{ matrix.vmm }}|' runtimeclass_workloads/*.yaml
           bash run_kubernetes_tests.sh

.github/workflows/run-k8s-tests-on-sev.yaml

@@ -27,7 +27,8 @@ jobs:
         with:
           ref: ${{ github.event.pull_request.head.sha }}
 
-      - name: Deploy kata-deploy
+      - name: Run tests
+        timeout-minutes: 30
         run: |
           sed -i -e "s|quay.io/kata-containers/kata-deploy:latest|${{ inputs.registry }}/${{ inputs.repo }}:${{ inputs.tag }}|g" tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
           cat tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
@@ -38,9 +39,11 @@ jobs:
           kubectl -n kube-system wait --timeout=10m --for=condition=Ready -l name=kata-deploy pod
           kubectl apply -f tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
 
-      - name: Run tests
-        timeout-minutes: 30
-        run: |
+          # This is needed as the kata-deploy pod will be set to "Ready" when it starts running,
+          # which may cause issues like not having the node properly labeled or the artefacts
+          # properly deployed when the tests actually start running.
+          sleep 60s
+
           pushd tests/integration/kubernetes
           sed -i -e 's|runtimeClassName: kata|runtimeClassName: kata-${{ matrix.vmm }}|' runtimeclass_workloads/*.yaml
           bash run_kubernetes_tests.sh

.github/workflows/run-k8s-tests-on-snp.yaml

@@ -27,7 +27,8 @@ jobs:
         with:
           ref: ${{ github.event.pull_request.head.sha }}
 
-      - name: Deploy kata-deploy
+      - name: Run tests
+        timeout-minutes: 30
         run: |
           sed -i -e "s|quay.io/kata-containers/kata-deploy:latest|${{ inputs.registry }}/${{ inputs.repo }}:${{ inputs.tag }}|g" tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
           cat tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
@@ -38,9 +39,11 @@ jobs:
           kubectl -n kube-system wait --timeout=10m --for=condition=Ready -l name=kata-deploy pod
           kubectl apply -f tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
 
-      - name: Run tests
-        timeout-minutes: 30
-        run: |
+          # This is needed as the kata-deploy pod will be set to "Ready" when it starts running,
+          # which may cause issues like not having the node properly labeled or the artefacts
+          # properly deployed when the tests actually start running.
+          sleep 60s
+
           pushd tests/integration/kubernetes
           sed -i -e 's|runtimeClassName: kata|runtimeClassName: kata-${{ matrix.vmm }}|' runtimeclass_workloads/*.yaml
           bash run_kubernetes_tests.sh

.github/workflows/run-k8s-tests-on-tdx.yaml

@@ -27,7 +27,8 @@ jobs:
         with:
           ref: ${{ github.event.pull_request.head.sha }}
 
-      - name: Deploy kata-deploy
+      - name: Run tests
+        timeout-minutes: 30
         run: |
           sed -i -e "s|quay.io/kata-containers/kata-deploy:latest|${{ inputs.registry }}/${{ inputs.repo }}:${{ inputs.tag }}|g" tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
           cat tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
@@ -38,9 +39,11 @@ jobs:
           kubectl -n kube-system wait --timeout=10m --for=condition=Ready -l name=kata-deploy pod
           kubectl apply -f tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml
 
-      - name: Run tests
-        timeout-minutes: 30
-        run: |
+          # This is needed as the kata-deploy pod will be set to "Ready" when it starts running,
+          # which may cause issues like not having the node properly labeled or the artefacts
+          # properly deployed when the tests actually start running.
+          sleep 60s
+
           pushd tests/integration/kubernetes
           sed -i -e 's|runtimeClassName: kata|runtimeClassName: kata-${{ matrix.vmm }}|' runtimeclass_workloads/*.yaml
           bash run_kubernetes_tests.sh