The Kubelet's DRA manager was failing to report device health status in a pod's status for certain types of resource claims. The logic incorrectly assumed that the claim name used within the container's spec (`container.resources.claims[*].name`) was the same as the metadata name of the actual ResourceClaim object.
This assumption is false in two key scenarios:
1. When a claim is generated from a `ResourceClaimTemplate`, Kubernetes creates a `ResourceClaim` object with a randomized suffix in its name.
2. When a user defines a pre-existing claim in `pod.spec.resourceClaims`, they can provide a local name that differs from the actual `ResourceClaim` object's name.
In both cases, the code would fail to find the claim's information in its internal cache, resulting in the health status not being populated, as reported in issue #134482.
This fix corrects the logic by using `pod.Status.ResourceClaimStatuses` as the authoritative map to look up the actual, generated name of the `ResourceClaim` object. This ensures that both templated and renamed claims are resolved correctly before their health status is retrieved from the cache.
Additionally, this change introduces a new node e2e test that specifically covers templated and renamed claims to prevent future regressions.
This commit migrates the container manager package to use
contextual logging.
This follows the contextual logging migration pattern where:
- Logger is passed from the boundary down to implementations
- klog.TODO() is used at call sites where context is not yet available
- Functions with context.Context extract logger via klog.FromContext(ctx)
- klog.InfoS/ErrorS/V().InfoS calls are replaced with logger.Info/Error/V().Info
- Sub-managers receive proper logger context from their callers
Some call sites still use klog.TODO() as placeholders, with TODO
comments indicating these will be replaced with proper contextual loggers
as the migration continues upward through the call stack.
Test/fake implementations use klog.Background() which is the appropriate
choice for test code where no real context is available.
Mock file updates (pkg/kubelet/cm/testing/mocks.go) were done manually
due to mockery tool incompatibility with Go 1.25 and will be committed
separately with an explanation.
Key changes:
1. PodContainerManager and ContainerManager interface updates:
- PodContainerManager.EnsureExists: add logger parameter
- PodContainerManager.Destroy: add logger parameter
- PodContainerManager.ReduceCPULimits: add logger parameter
- PodContainerManager.SetPodCgroupConfig: add logger parameter
- ContainerManager.UpdateQOSCgroups: add logger parameter
- Update all implementations (Linux, Windows, stub, noop, fake)
2. Helper function updates:
- GetKubeletContainer: add logger parameter (Linux and unsupported platforms)
- Update cmd/kubelet/app/server.go to pass logger to GetKubeletContainer
- Comment out type assertion in helpers.go due to signature change
3. Cgroup manager contextual logging:
- CgroupManager interface methods updated to accept logger:
* Destroy: add logger parameter
* ReduceCPULimits: add logger parameter
* SetCgroupConfig: add logger parameter
- Update cgroupCommon and unsupportedCgroupManager implementations
- Migrate klog.InfoS/V().InfoS calls to logger.Info/V().Info
4. Container manager implementation updates:
- Extract logger from context in NewContainerManager() and Start()
- Pass logger to sub-managers (deviceManager, topologyManager)
- Update DRA manager initialization to use logger from context
- Migrate klog.InfoS/ErrorS to logger.Info/Error throughout
5. Call sites updated with TODO comments:
- pkg/kubelet/kubelet.go: UpdateQOSCgroups, EnsureExists
- pkg/kubelet/kubelet_pods.go: UpdateQOSCgroups, ReduceCPULimits, Destroy
- pkg/kubelet/kuberuntime/kuberuntime_manager.go: SetPodCgroupConfig
- pkg/kubelet/kuberuntime/kuberuntime_manager_test.go: test expectations
Most call sites currently use klog.TODO() as placeholders, with TODO
comments indicating these will be replaced with proper contextual loggers
as the migration continues upward through the call stack.
Test/fake implementations use klog.Background() which is the appropriate
choice for test code where no real context is available.
Mock file updates (pkg/kubelet/cm/testing/mocks.go) were done manually
due to mockery tool incompatibility with Go 1.25 and will be committed
separately with an explanation.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
With kubernetes/kubernetes#132028 merged, pods in terminal states are no longer
reported by the podresources API. The previous test logic accounted for the old
behavior where even failed pods appeared in the API response (tracked under k/k
issue #119423). As a result, we used to expect the failed test pod to be present in the
response but with an empty device set.
This change updates the test to reflect the new, correct behavior:
1. The failed test pod should no longer appear in the podresources API response.
2. The test now asserts absence of the failed pod rather than checking for an empty device assignment.
This simplifies the test logic and aligns expectations with the current upstream
behavior of the podresources API.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
As discovered in ticket 134737 `hostname` is buggy on busybox
due to the musl backend that it uses. something in the /etc/hosts
that k8s generates trips its parser and it doesn't work properly
in the ipv6 tests.
To workaround that use an image that has a glibc backend,
so that the hostname command works.
In the device plugin node reboot e2e test, the test previously waited a
short period for the resources exported by the sample device plugin to
appear on the local node. On slower test nodes, the plugin may take
longer to register, causing flakes where the expected devices are not
yet available.
This change increases the polling duration to 2 minutes, ensuring the test
waits long enough for the expected device capacity and allocatable resources
to appear, improving test stability.
This commit also updates the assertion message to be more explicit making
failures clearer and improving test reliability.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
In the device plugin node reboot e2e test, the registration trigger
(control file deletion) was being executed immediately after pod creation.
This could create a race condition: the device plugin container might not
be fully running, causing the test to flake when devices were not reported
as available on the node.
This change explicitly waits for the sample device plugin pod to reach the
Running/Ready state before deleting the registration control file. This
ensures that the device plugin is ready to register its devices with the
kubelet, eliminating a possible source of test flakiness.
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
- Remove feature.KubeletPSI label as it may be redundant for categorization
- Add framework.WithFeatureGate(features.KubeletPSI) for proper feature gate checking
- Remove redundant utilfeature.DefaultFeatureGate.Enabled check in BeforeEach
This addresses the cleanup goal in issue #134172 to replace feature labels
with feature gates where appropriate.
This reports and fixes for test/e2e:
ERROR: E2E suite initialization was faulty, these errors must be fixed:
ERROR: apimachinery/mutatingadmissionpolicy.go:184: full test name is not unique: "[sig-api-machinery] MutatingAdmissionPolicy [Privileged:ClusterAdmin] [Feature:MutatingAdmissionPolicy] [FeatureGate:MutatingAdmissionPolicy] [Beta] [Feature:OffByDefault] should support MutatingAdmissionPolicy API operations" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e/apimachinery/mutatingadmissionpolicy.go:184, /nvme/gopath/src/k8s.io/kubernetes/test/e2e/apimachinery/mutatingadmissionpolicy.go:606)
ERROR: apimachinery/mutatingadmissionpolicy.go:412: full test name is not unique: "[sig-api-machinery] MutatingAdmissionPolicy [Privileged:ClusterAdmin] [Feature:MutatingAdmissionPolicy] [FeatureGate:MutatingAdmissionPolicy] [Beta] [Feature:OffByDefault] should support MutatingAdmissionPolicyBinding API operations" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e/apimachinery/mutatingadmissionpolicy.go:412, /nvme/gopath/src/k8s.io/kubernetes/test/e2e/apimachinery/mutatingadmissionpolicy.go:834)
ERROR: common/node/pod_level_resources.go:250: full test name is not unique: "[sig-node] Pod Level Resources [Serial] [Feature:PodLevelResources] [FeatureGate:PodLevelResources] [Beta] Guaranteed QoS pod with container resources" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e/common/node/pod_level_resources.go:250 (2x))
ERROR: dra/dra.go:1899: full test name is not unique: "[sig-node] [DRA] kubelet [Feature:DynamicResourceAllocation] [FeatureGate:DRAConsumableCapacity] [Alpha] [Feature:OffByDefault] [FeatureGate:DynamicResourceAllocation] must allow multiple allocations and consume capacity [KubeletMinVersion:1.34]" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e/dra/dra.go:1899 (2x))
ERROR: storage/testsuites/volume_group_snapshottable.go:173: full test name is not unique: "[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: (delete policy)] volumegroupsnapshottable [Feature:volumegroupsnapshot] VolumeGroupSnapshottable should create snapshots for multiple volumes in a pod" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_group_snapshottable.go:173 (2x))
ERROR: storage/testsuites/volume_group_snapshottable.go:173: full test name is not unique: "[sig-storage] CSI Volumes [Driver: pd.csi.storage.gke.io] [Serial] [Testpattern: (delete policy)] volumegroupsnapshottable [Feature:volumegroupsnapshot] VolumeGroupSnapshottable should create snapshots for multiple volumes in a pod" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_group_snapshottable.go:173 (2x))
And for test/e2e_node:
ERROR: cpu_manager_test.go:1622: full test name is not unique: "[sig-node] CPU Manager [Serial] [Feature:CPUManager] when checking the CFS quota management should disable for guaranteed pod with exclusive CPUs assigned" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e_node/cpu_manager_test.go:1622, /nvme/gopath/src/k8s.io/kubernetes/test/e2e_node/cpu_manager_test.go:1642)
ERROR: eviction_test.go:800: full test name is not unique: "[sig-node] LocalStorageCapacityIsolationFSQuotaMonitoring [Slow] [Serial] [Disruptive] [Feature:LocalStorageCapacityIsolationQuota] [Feature:LSCIQuotaMonitoring] [Feature:UserNamespacesSupport] when we run containers that should cause use quotas for LSCI monitoring (quotas enabled: true) should eventually evict all of the correct pods" (/nvme/gopath/src/k8s.io/kubernetes/test/e2e_node/eviction_test.go:800 (2x))
8fed05c5b7 fixed the cleanup of tests which start
registrar and service in a single call. But tests which first started the
registrar and then the service separately still had the problem:
- registrar is started with test context
- pods remain running at end of test
- registrar stops because of test context cancellation
- pods remain pending despite deletion because the driver gets
unregistered (timing dependent, so this may have flaked)
The fix is to also clean up the registrar after the test, in reverse startup
order.
Added e2e_node test to verify that the Kubelet establishes only
a single gRPC connection with the DRA plugin for all service calls
during the plugin lifecycle.
The test uses a custom listener to count accepted connections and
asserts that only one connection is used for NodePrepareResources,
NodeUnprepareResources, and NodeWatchResources calls.
dd9917ddce fixed one test which did not wait for
pods to be deleted and then, depending on the timing, left ResourceClaims
prepared because the driver stopped before kubelet could call
NodeUnprepareResources.
But this is a more systematic issue also with other tests, so now the any test
which starts a DRA plugin automatically uses the same common cleanup code:
- delete pods in the test names
- wait for the driver to not have any active ResourceClaims
- stop the driver
