The approach copied from node taint eviction was to fire off one goroutine per
pod the intended time. This leads to the "thundering herd" problem: when a
single taint causes eviction of several pods and those all have no or the same
toleration grace period, then they all get deleted concurrently at the same
time.
For node taint eviction that is limited by the number of pods per node, which
is typically ~100. In an integration test, that already led to problems with
watchers:
cacher.go:855] cacher (pods): 100 objects queued in incoming channel.
cache_watcher.go:203] Forcing pods watcher close due to unresponsiveness: key: "/pods/", labels: "", fields: "". len(c.input) = 10, len(c.result) = 10, graceful = false
It also causes spikes in memory consumption (mostly the 2KB stack per goroutine
plus closure) with no upper limit.
Using a workqueue makes concurrency more deterministic because there is an
upper limit. In the integration test, 10 workers kept the watch active.
Another advantage is that failures to evict the pod get retried with
exponential backoff per affected pod forever. Previously, evicting was tried a
few times with a fixed rate and then the controller gave up. If the apiserver
was down long enough, pods didn't get evicted.
When the ResourceSlice no longer exists, the ResourceSlice tracker didn't and
couldn't report the tainted devices even if they are allocated and in use. The
controller must keep track of DeviceTaintRules itself and handle this scenario.
In this scenario it is impossible to evaluation CEL expressions because the
necessary device attributes aren't available. We could:
- Copy them in the allocation result: too large, big change.
- Limit usage of CEL expressions to rules with no eviction: inconsistent.
- Remove the fields which cannot be supported well.
The last option is chosen.
The tracker is now no longer needed by the eviction controller. Reading
directly from the informer means that we cannot assume that pointers are
consistent. We have to track ResourceSlices by their name, not their pointer.
This raises the number of allowed taints per device to 16 by lowering
the number of allowed devices to 64 per ResourceSlice if (and only if!)
taints are used.
"effect: None" and DeviceTaintRule status with conditions get added
to support giving feedback to admins.
Instead of merely adding the new effect value, this also changes validation of
the enum so that unknown values are valid if they were already stored. This
will simplify adding new effects in the future because validation won't fail
for them after a downgrade. Consumers must treat them like this new None
effect, i.e. ignore them.
We recently received this bug report: #132966. The issue seems to be
that the new range configured doesn't work for existing pods on the
node, and therefore the kubelet fails to start.
While I'll also improve the documentation to mention you need to drain
the node, let's also improve the error message saying which pod this
error comes from.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
This is a collection of updating several unversioned modules. Those updates are
eventually going to reach Kubernetes, doing them together now makes PRs for
future major updates smaller.
This commit migrates the container manager package to use
contextual logging.
This follows the contextual logging migration pattern where:
- Logger is passed from the boundary down to implementations
- klog.TODO() is used at call sites where context is not yet available
- Functions with context.Context extract logger via klog.FromContext(ctx)
- klog.InfoS/ErrorS/V().InfoS calls are replaced with logger.Info/Error/V().Info
- Sub-managers receive proper logger context from their callers
Some call sites still use klog.TODO() as placeholders, with TODO
comments indicating these will be replaced with proper contextual loggers
as the migration continues upward through the call stack.
Test/fake implementations use klog.Background() which is the appropriate
choice for test code where no real context is available.
Mock file updates (pkg/kubelet/cm/testing/mocks.go) were done manually
due to mockery tool incompatibility with Go 1.25 and will be committed
separately with an explanation.
Key changes:
1. PodContainerManager and ContainerManager interface updates:
- PodContainerManager.EnsureExists: add logger parameter
- PodContainerManager.Destroy: add logger parameter
- PodContainerManager.ReduceCPULimits: add logger parameter
- PodContainerManager.SetPodCgroupConfig: add logger parameter
- ContainerManager.UpdateQOSCgroups: add logger parameter
- Update all implementations (Linux, Windows, stub, noop, fake)
2. Helper function updates:
- GetKubeletContainer: add logger parameter (Linux and unsupported platforms)
- Update cmd/kubelet/app/server.go to pass logger to GetKubeletContainer
- Comment out type assertion in helpers.go due to signature change
3. Cgroup manager contextual logging:
- CgroupManager interface methods updated to accept logger:
* Destroy: add logger parameter
* ReduceCPULimits: add logger parameter
* SetCgroupConfig: add logger parameter
- Update cgroupCommon and unsupportedCgroupManager implementations
- Migrate klog.InfoS/V().InfoS calls to logger.Info/V().Info
4. Container manager implementation updates:
- Extract logger from context in NewContainerManager() and Start()
- Pass logger to sub-managers (deviceManager, topologyManager)
- Update DRA manager initialization to use logger from context
- Migrate klog.InfoS/ErrorS to logger.Info/Error throughout
5. Call sites updated with TODO comments:
- pkg/kubelet/kubelet.go: UpdateQOSCgroups, EnsureExists
- pkg/kubelet/kubelet_pods.go: UpdateQOSCgroups, ReduceCPULimits, Destroy
- pkg/kubelet/kuberuntime/kuberuntime_manager.go: SetPodCgroupConfig
- pkg/kubelet/kuberuntime/kuberuntime_manager_test.go: test expectations
Most call sites currently use klog.TODO() as placeholders, with TODO
comments indicating these will be replaced with proper contextual loggers
as the migration continues upward through the call stack.
Test/fake implementations use klog.Background() which is the appropriate
choice for test code where no real context is available.
Mock file updates (pkg/kubelet/cm/testing/mocks.go) were done manually
due to mockery tool incompatibility with Go 1.25 and will be committed
separately with an explanation.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
