With OrderedNamespaceDeletion enabled by default, the namespace
controller now deletes resources before deleting the namespace itself.
At the same time, some existing tests create many resources. When
those tests finish around the same time, the namespace controller can
become overloaded. As a result, it may take longer to start deleting
resources created by this test.
Initially, this patch increased the test timeout to account for that
delay. However, the test still flaked in 1 out of 10 jobs, which
suggests increasing the timeouts isn't enough.
Once the test was made serial, no flakes were observed.
If preparing for a pod failed because the driver was not registered, it failed
after already having added the pod and claim to the claim info cache. The
effect was that deletion of the pod was blocked until the driver got
installed. Other errors had a similar effect. Now as many errors as possible
are checked in a read-only loop before proceeding to changing the claim info
cache.
The error message that was surfaced for the problem was not very readable:
Failed to prepare dynamic resources: failed to get gRPC client for driver dra-1411.k8s.io: plugin name dra-1411.k8s.io not found in the list of registered DRA plugins
To address this, error messages and wrapping get updated according to these
guidelines:
// Most errors returned by the manager show up in the context of a pod.
// They try to adher to the following convention:
// - Don't include the pod.
// - Use terms that are familiar to users.
// - Don't include the namespace, it can be inferred from the context.
// - Avoid repeated "failed to ...: failed to ..." when wrapping errors.
// - Avoid wrapping when it does not provide relevant additional information to keep the user-visible error short.
* Move the autoscaling e2e tests to be ran in parallel to have coverage for concurrent hpa processing
* Move the autoscaling e2e tests to be ran in parallel to have coverage for concurrent hpa processing
* Update autoscaling_utils.go
This can happen when a test is run in a cluster where a required feature gate
is turned off. Letting the test continue then fails less clearly. For example,
a test for partitionable devices runs with ResourceSlices which don't have
the additional fields for partitionable devices.
* Fix test flakiness caused by shadowed error in reboot test and stale pods affecting eviction test
- Fixed a bug in `ExecCommandInContainerWithFullOutput` usage in `hybrid_network.go` where the `err` variable was shadowed, causing test failures to be silently ignored.
- Added cleanup logic in `eviction.go` to explicitly delete pods created by the reboot node test (e.g., `img-puller`, `reboot-host-test-windows`) before starting the eviction test.
- This improves reliability of the eviction test when the reboot test runs beforehand and leaves behind memory-consuming pods.
* Add cleanup containers in reboot node test
* Workaround for Calico HNS issue by avoiding eviction of undeletable pods
Add a workaround for a known Calico issue (https://github.com/projectcalico/calico/issues/6974) where pods on Windows nodes may become undeletable after a reboot. This causes the eviction manager to attempt evicting these pods but fail due to HNS namespace deletion errors.
The workaround avoids scheduling critical test pods on rebooted nodes to prevent interference.
TODO: Remove this workaround once the Calico issue is resolved.
* Fix the lint issue.
* Address comments
When expectation is that something does *not* happen, the test has to check for
a certain time. Checking just once is not sufficient because it might not have
happened immediately.
As can be seen in the output of a test failure in the CI, the test waits for
"quota-not-terminating" to be updated, but then fails because creating the pod
exceeds the "quota-terminating" quota which has not been updated yet. As both
ResourceQuotas apply, it needs to wait for both of them to be updated.
STEP: Deleting the pod - k8s.io/kubernetes/test/e2e/apimachinery/resource_quota.go:1597 @ 05/18/25 07:50:46.531
STEP: Ensuring resource quota status released the pod usage - k8s.io/kubernetes/test/e2e/apimachinery/resource_quota.go:1601 @ 05/18/25 07:50:46.547
I0518 07:50:46.551520 75078 resource_quota.go:2508] Got expected ResourceQuota:
...
name: quota-not-terminating
namespace: scope-selectors-1836
resourceVersion: "15914"
uid: db72668e-2fe0-4543-ada2-a3ff0ac77dd5
spec:
hard:
limits.cpu: "2"
limits.memory: 1Gi
pods: "1"
requests.cpu: "1"
requests.memory: 500Mi
scopeSelector:
matchExpressions:
- operator: Exists
scopeName: NotTerminating
status:
hard:
limits.cpu: "2"
limits.memory: 1Gi
pods: "1"
requests.cpu: "1"
requests.memory: 500Mi
used:
limits.cpu: "0"
limits.memory: "0"
pods: "0"
requests.cpu: "0"
requests.memory: "0"
STEP: Creating a terminating pod - k8s.io/kubernetes/test/e2e/apimachinery/resource_quota.go:1610 @ 05/18/25 07:50:46.551
I0518 07:50:46.560944 75078 resource_quota.go:1612] Unexpected error:
<*errors.StatusError | 0xc002dba500>:
pods "terminating-pod" is forbidden: exceeded quota: quota-terminating, requested: pods=1, used: pods=1, limited: pods=1
{
ErrStatus:
code: 403
details:
kind: pods
name: terminating-pod
message: 'pods "terminating-pod" is forbidden: exceeded quota: quota-terminating,
requested: pods=1, used: pods=1, limited: pods=1'
metadata: {}
reason: Forbidden
status: Failure,
}
[FAILED] pods "terminating-pod" is forbidden: exceeded quota: quota-terminating, requested: pods=1, used: pods=1, limited: pods=1
This is meant to help debug this flake:
[It] [sig-api-machinery] ResourceQuota should verify ResourceQuota with terminating scopes through scope selectors.
[FAILED] pods "test-pod" is forbidden: exceeded quota: quota-terminating, requested: pods=1, used: pods=1, limited: pods=1
In [It] at: k8s.io/kubernetes/test/e2e/apimachinery/resource_quota.go:1572 @ 05/14/25 08:49:07.456
Dumping the ResourceQuota object each time the test progresses will
help with understanding the test and the actual behavior.
In at least one case, the controller only got as far as creating 93 out of 100
slices in the allotted one minute:
[FAILED] Timed out after 60.001s.
Value for field 'Items' failed to satisfy matcher.
Expected
<[]v1beta2.ResourceSlice | len:93, cap:146>:
...
to have length 100
In [It] at: k8s.io/kubernetes/test/e2e/dra/dra.go:1520 @ 05/09/25 17:49:42.592
[sig-scheduling] SchedulerPreemption [Serial] validates various priority Pods preempt expectedly with the async preemption: replace finalizers with preStop hook and TerminationGracePeriodSeconds
there are many cases of an e2e feature being the main filtering mechanism for a test.
However, this makes it tricky for suite authors to filter tests based on available features.
This mechanism is still useful for e.g. making sure a runtime on a suite is of the correct level,
but in general we should both label tests with the Feature set they require (whether runtime is correct) as well
as the feature gates they test.
Signed-off-by: Peter Hunt <pehunt@redhat.com>
