properly support the resource metrics endpoint when `PodAndContainerStatsFromCRI` is enabled and fix the related e2e tests.
Stats Provider:
- add container-level CPU and memory stats to `ListPodCPUAndMemoryStats` so the resource metrics endpoint has complete data
- add `aggregatePodSwapStats` to compute pod-level swap from container stats (CRI doesn't provide pod-level swap directly)
- add missing memory stats fields: `AvailableBytes`, `PageFaults`, and `MajorPageFaults`
- add platform-specific implementations for Linux and Windows
Tests:
- skip cAdvisor metrics test when `PodAndContainerStatsFromCRI` is enabled (cAdvisor metrics aren't available in that mode)
- fix expected metrics in `ResourceMetricsAPI` test
- `node_swap_usage_bytes` is only available with cAdvisor (need to verify!)
- Add `dumpResourceMetricsForPods` helper to log actual metric values when tests fail, making debugging easier
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Add ownersLock to protect concurrent access to node.owners between
GraphBuilder.processGraphChanges() (writer) and GC worker goroutines
reading in blockingDependents() and unblockOwnerReferences() methods.
Also fix concurrent reads in the HTTP debug handler (/graph endpoint)
for owners, dependents, beingDeleted, deletingDependents, and virtual
fields by using their respective thread-safe accessor methods.
The PatchNodeOnce function has historically exited early
in scanarious when we Get a Node object, but the next Patch
API call on the same Node object fails. This can happen
in setups that are under a lot of resource pressure
or different network timeout scenarious.
Instead of exiting early and allow listing certain errors,
always retry on any Patch error. This aligns with the
general idea that kubeadm retries *all* API calls.
For debugging double allocation of the same
device (https://github.com/kubernetes/kubernetes/issues/133602) it is necessary
to have information about pools, devices and in-flight claims. Log calls get
extended and the config for DRA CI jobs updated to enable higher verbosity for
relevant source files.
Log output in such a cluster at verbosity 6 looks like this:
I1215 10:28:54.166872 1 allocator_incubating.go:130] "Gathered pool information" logger="FilterWithNominatedPods.Filter.DynamicResources" pod="dra-8841/tester-3" node="kind-worker2" pools={"count":1,"devices":["dra-8841.k8s.io/kind-worker2/device-00"],"meta":[{"InvalidReason":"","id":"dra-8841.k8s.io/kind-worker2","isIncomplete":false,"isInvalid":false}]}
I1215 10:28:54.166941 1 allocator_incubating.go:254] "Gathered information about devices" logger="FilterWithNominatedPods.Filter.DynamicResources" pod="dra-8841/tester-3" node="kind-worker2" allocatedDevices={"count":2,"devices":["dra-8841.k8s.io/kind-worker/device-00","dra-8841.k8s.io/kind-worker3/device-00"]} minDevicesToBeAllocated=1
This change makes `defaulter-gen` consistent with the rest of the code-generator
tools like `deepcopy-gen`, `validation-gen` and others.
In order for defaults to be generated, the `defaulter-gen` tool will expect to
find the respective `+k8s:defaulter-gen` marker.
Additional details can be found in the following issue:
https://github.com/kubernetes/kubernetes/issues/135417
Previously, when a Pod residing in the 'unschedulablePods' queue was updated and subsequently rejected by PreEnqueue plugins (returning 'Wait'), the logic in 'moveToActiveQ' would return early because the Pod was already present in the queue.
This caused the 'scheduler_gated_pods_total' metric to fail to increment, leading to metric inconsistencies (and potentially negative values upon Pod deletion).
This change adds a check to detect the transition from Ungated to Gated. If detected, the Pod is removed and re-added to the queue to ensure metrics are correctly swapped (Unschedulable-- and Gated++).
Added regression test 'TestSchedulingQueueMetrics_UngatedToGated' to verify the fix.
Signed-off-by: Vlad Shkrabkov <vshkrabkov@google.com>
- Replaced time.Sleep with gomega.Consistently to ensure reliable failure detection if a restart occurs during the wait
- Refactored createPodWith*Probes functions into a single helper to remove code duplication
Signed-off-by: KasimVali2207 <kasimvali2207@gmail.com>
This commit adds e2e stress tests to verify that liveness probes do not cause
unexpected container restarts under load. The tests create many containers
(50 per test) with liveness probes configured to run every 1 second.
Three test cases are included:
- HTTP liveness probe stress test
- TCP liveness probe stress test
- gRPC liveness probe stress test
Each test waits for all containers to be running, observes probe behavior for
2 minutes, and validates that no containers have restarted unexpectedly.
These tests address the bug fix from issue kubernetes#89898 and serve as a
replacement for the skipped unit test from PR kubernetes#115329.
