properly support the resource metrics endpoint when `PodAndContainerStatsFromCRI` is enabled and fix the related e2e tests.
Stats Provider:
- add container-level CPU and memory stats to `ListPodCPUAndMemoryStats` so the resource metrics endpoint has complete data
- add `aggregatePodSwapStats` to compute pod-level swap from container stats (CRI doesn't provide pod-level swap directly)
- add missing memory stats fields: `AvailableBytes`, `PageFaults`, and `MajorPageFaults`
- add platform-specific implementations for Linux and Windows
Tests:
- skip cAdvisor metrics test when `PodAndContainerStatsFromCRI` is enabled (cAdvisor metrics aren't available in that mode)
- fix expected metrics in `ResourceMetricsAPI` test
- `node_swap_usage_bytes` is only available with cAdvisor (need to verify!)
- Add `dumpResourceMetricsForPods` helper to log actual metric values when tests fail, making debugging easier
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
For debugging double allocation of the same
device (https://github.com/kubernetes/kubernetes/issues/133602) it is necessary
to have information about pools, devices and in-flight claims. Log calls get
extended and the config for DRA CI jobs updated to enable higher verbosity for
relevant source files.
Log output in such a cluster at verbosity 6 looks like this:
I1215 10:28:54.166872 1 allocator_incubating.go:130] "Gathered pool information" logger="FilterWithNominatedPods.Filter.DynamicResources" pod="dra-8841/tester-3" node="kind-worker2" pools={"count":1,"devices":["dra-8841.k8s.io/kind-worker2/device-00"],"meta":[{"InvalidReason":"","id":"dra-8841.k8s.io/kind-worker2","isIncomplete":false,"isInvalid":false}]}
I1215 10:28:54.166941 1 allocator_incubating.go:254] "Gathered information about devices" logger="FilterWithNominatedPods.Filter.DynamicResources" pod="dra-8841/tester-3" node="kind-worker2" allocatedDevices={"count":2,"devices":["dra-8841.k8s.io/kind-worker/device-00","dra-8841.k8s.io/kind-worker3/device-00"]} minDevicesToBeAllocated=1
The pod resize e2e tests use memory limits as low as 20Mi for Guaranteed
QoS pods. On OpenShift/CRI-O, the container runtime (runc) runs inside
the pod's cgroup and requires ~20-22MB of memory during container
creation and restart operations. This causes intermittent OOM kills
when the pod's memory limit is at or below runc's memory footprint.
This issue does not occur on containerd-based clusters because
containerd's shim runs outside the pod's cgroup by default (ShimCgroup=""),
so runc's memory is not charged against the pod's limit.
Increase memory limits to provide sufficient headroom for runc:
- originalMem: 20Mi -> 35Mi
- reducedMem: 15Mi -> 30Mi
- increasedMem: 25Mi -> 40Mi
The test validates resize behavior, not minimal memory limits, so
larger values do not reduce test coverage.
Signed-off-by: Damien Grisonnet <dgrisonn@redhat.com>
It needs to run as [Serial], so it accidentally does not evict other Pods.
Consider this scenario on a busy clusters, with all nodes at their attachment limit.
1. pod1 of the preemption test runs, pod2 is created.
2. The scheduler evicts pod1. That frees the RWOP volume and it also frees the last attachment slot on the node.
3. Some other e2e tests creates a Pod and scheduler puts it on a node, taking the last attachment slot.
4. The scheduler schedules pod2 agaian and it sees there is no node with a free attachment slot -> new round of eviction, now evicting a pod of unrelated e2e tests. The unrelated test will fail.
All version up to and including x.y.z-alpha.0 are treated as if we were
still the previous minor version x.(y-1). There are two reason for this:
- During code freeze around (at?) -rc.0, the master branch already
identfies itself as the next release with -alpha.0. Without this
special case, we would change the version skew testing from what
has been tested and been known to work to something else, which
can and at least once did break.
- Early in the next cycle the differences compared to the previous
release are small, so it's more interesting to go back further.
The ipallocator was blindly assuming that all errors are retryable, that
causes that the allocator tries to exhaust all the possibilities to
allocate an IP address.
If the error is not retryable this means the allocator will generate as
many API calls as existing available IPs are in the allocator, causing
CPU exhaustion since this requests are coming from inside the apiserver.
In addition to handle the error correctly, this patch also interpret the
error to return the right status code depending on the error type.
Co-authored-by: carlory <baofa.fan@daocloud.io>
