When a container restarts before kubelet restarts, containerMap has
multiple entries (old exited + new running). GetContainerID() may
return the exited container, causing the running check to fail. Fixed
by checking if ANY container for the pod/name is running.
Also filter terminal pods from podresources since they no longer
consume resources, and fix test error handling to avoid exiting
Eventually immediately on transient errors.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Bumping to 5 is useful in unit tests. Those tend to not produce less output and
ideally use per-test output, so we end up keeping only the output of failed
tests where increased verbosity also in CI runs is useful.
But ktesting now also gets imported into e2e test binaries through the
framework. There the increased verbosity is apparently causing OOM killing in
some jobs which previously worked fine.
Long term we need a better solution than simply disabling the verbosity
change. We could modify each unit test to call SetDefaultVerbosity, but that's
tedious. Perhaps an env variable? It cannot be a command line flag because not
all unit tests accept `-v`.
This has been replaced by `//build:...` for a long time now.
Removal of the old build tag was automated with:
for i in $(git grep -l '^// +build' | grep -v -e '^vendor/'); do if ! grep -q '^// Code generated' "$i"; then sed -i -e '/^\/\/ +build/d' "$i"; fi; done
The Kubelet's DRA manager was failing to report device health status in a pod's status for certain types of resource claims. The logic incorrectly assumed that the claim name used within the container's spec (`container.resources.claims[*].name`) was the same as the metadata name of the actual ResourceClaim object.
This assumption is false in two key scenarios:
1. When a claim is generated from a `ResourceClaimTemplate`, Kubernetes creates a `ResourceClaim` object with a randomized suffix in its name.
2. When a user defines a pre-existing claim in `pod.spec.resourceClaims`, they can provide a local name that differs from the actual `ResourceClaim` object's name.
In both cases, the code would fail to find the claim's information in its internal cache, resulting in the health status not being populated, as reported in issue #134482.
This fix corrects the logic by using `pod.Status.ResourceClaimStatuses` as the authoritative map to look up the actual, generated name of the `ResourceClaim` object. This ensures that both templated and renamed claims are resolved correctly before their health status is retrieved from the cache.
Additionally, this change introduces a new node e2e test that specifically covers templated and renamed claims to prevent future regressions.
Correctly removed the resource from both healthy and unhealthy device
maps when resource endpoint expires.
This makes resource endpoint and devices maps consistent.
This commit migrates the container manager package to use
contextual logging.
This follows the contextual logging migration pattern where:
- Logger is passed from the boundary down to implementations
- klog.TODO() is used at call sites where context is not yet available
- Functions with context.Context extract logger via klog.FromContext(ctx)
- klog.InfoS/ErrorS/V().InfoS calls are replaced with logger.Info/Error/V().Info
- Sub-managers receive proper logger context from their callers
Some call sites still use klog.TODO() as placeholders, with TODO
comments indicating these will be replaced with proper contextual loggers
as the migration continues upward through the call stack.
Test/fake implementations use klog.Background() which is the appropriate
choice for test code where no real context is available.
Mock file updates (pkg/kubelet/cm/testing/mocks.go) were done manually
due to mockery tool incompatibility with Go 1.25 and will be committed
separately with an explanation.
Key changes:
1. PodContainerManager and ContainerManager interface updates:
- PodContainerManager.EnsureExists: add logger parameter
- PodContainerManager.Destroy: add logger parameter
- PodContainerManager.ReduceCPULimits: add logger parameter
- PodContainerManager.SetPodCgroupConfig: add logger parameter
- ContainerManager.UpdateQOSCgroups: add logger parameter
- Update all implementations (Linux, Windows, stub, noop, fake)
2. Helper function updates:
- GetKubeletContainer: add logger parameter (Linux and unsupported platforms)
- Update cmd/kubelet/app/server.go to pass logger to GetKubeletContainer
- Comment out type assertion in helpers.go due to signature change
3. Cgroup manager contextual logging:
- CgroupManager interface methods updated to accept logger:
* Destroy: add logger parameter
* ReduceCPULimits: add logger parameter
* SetCgroupConfig: add logger parameter
- Update cgroupCommon and unsupportedCgroupManager implementations
- Migrate klog.InfoS/V().InfoS calls to logger.Info/V().Info
4. Container manager implementation updates:
- Extract logger from context in NewContainerManager() and Start()
- Pass logger to sub-managers (deviceManager, topologyManager)
- Update DRA manager initialization to use logger from context
- Migrate klog.InfoS/ErrorS to logger.Info/Error throughout
5. Call sites updated with TODO comments:
- pkg/kubelet/kubelet.go: UpdateQOSCgroups, EnsureExists
- pkg/kubelet/kubelet_pods.go: UpdateQOSCgroups, ReduceCPULimits, Destroy
- pkg/kubelet/kuberuntime/kuberuntime_manager.go: SetPodCgroupConfig
- pkg/kubelet/kuberuntime/kuberuntime_manager_test.go: test expectations
Most call sites currently use klog.TODO() as placeholders, with TODO
comments indicating these will be replaced with proper contextual loggers
as the migration continues upward through the call stack.
Test/fake implementations use klog.Background() which is the appropriate
choice for test code where no real context is available.
Mock file updates (pkg/kubelet/cm/testing/mocks.go) were done manually
due to mockery tool incompatibility with Go 1.25 and will be committed
separately with an explanation.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
This change completes the contextual logging migration for the memory
manager by updating the Manager interface and all implementations to
accept logr.Logger parameters instead of context.Context.
Key changes:
- Update Manager interface methods to accept logr.Logger:
* AddContainer, RemoveContainer, GetMemoryNUMANodes
* GetAllocatableMemory, GetMemory (removed context entirely)
- Update Policy interface methods to accept logr.Logger instead of context.Context
- Pass logger to NewManager() and policy constructors (NewPolicyStatic, NewPolicyNone, NewPolicyBestEffort)
- Update internal_container_lifecycle to use klog.TODO() when calling memory manager methods
- Update fake manager to accept and use logger parameter
- Update all test code to pass logger instead of context
This aligns with the contextual logging migration pattern where:
- Functions that need a logger accept logr.Logger parameter directly
- Logger is passed from the boundary (e.g., Start()) down to implementation
- klog.TODO() is used temporarily in call sites where logger is not yet available
- Context is only used where truly needed (e.g., Start() method)
This follows the same pattern as recent migrations in:
- pkg/kubelet/cm/topologymanager (#134174)
- pkg/kubelet/cm/devicemanager (#134293)
- pkg/kubelet/cm/cpumanager (#125912)
Related to the initial memory manager contextual logging work in #130727.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
Updated allocateContainerResources to skip resources referring
extended resource claims as they are managed by DRA drivers.
These resources shouldn't be processed by device plugins.
migrate the cpumanager code to contextual logging
design musings:
- golang contexts "[...] carries deadlines, cancellation signals, and
other request-scoped values across API boundaries and between
processes." (see: https://pkg.go.dev/context#pkg-overview).
Thus, it generally makes sense to add contexts to most of the APIs
and functions as the resource management is done in the admission flow, when
ultimately we process a request and we have time bounds;
- however, receiving a context from the outside and depending on it
kind also conflicts with the goal of having uniform logging and "
- attach key/value pairs that get included in all log messages
- add names that describe which component or operation triggered a log messages"
(see:
https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3077-contextual-logging#summary)
Some of this conflict is due to the transitional nature of the PRs
which we do when updating the codelet codebase. We can envision a
final state on which the top level caller adds identifying information
like pod UID and pod namespace/name pair; but up until all the calling
chain is updated and wired, the results are gonna be inconsistent.
- continuing from the point above, the most realistic transition plan
for the kubelet is outlined in
https://github.com/kubernetes/kubernetes/issues/130069 .
Since we do want to move in parallel and migrate on a
subsystem-by-subsystem basis, each subsystem will be in charge to add
the data they want to their contexts. I think this is the only
reasonnable way forward, and the cost will be overhead and redundancy.
We should do sweeping changes to uniform the flows once we completed
the full transition to ensure uniformity.
- there are some well known packages which want to operate by design
only in in-memory data structure, which arguably can't block.
Shoehorning a context feels not great. Instead, we intentionally pass
only the logger, which still is the core idea of contextual logging
(see:
https://github.com/kubernetes/kubernetes/pull/125912#discussion_r1935919144)
examples are the topology subpackage and the cpu_assignment logic
(takeByTopology and friends)
Signed-off-by: Francesco Romani <fromani@redhat.com>
unit tests for the policy options add test options to the
global state without proper cleanup; this is done only in tests
so it has limited room for doing damage, but still weakens
the test signal. Let's avoid that adding proper cleanup.
Signed-off-by: Francesco Romani <fromani@redhat.com>
