Commit Graph

309 Commits

Author SHA1 Message Date
Kubernetes Prow Robot
210881f0f0 Merge pull request #135485 from saschagrunert/fix-device-plugin-termination-grace-period
Fix device plugin admission failure after container restart
2026-01-12 22:24:22 +05:30
Sascha Grunert
172a65c71d Fix device plugin admission failure after container restart
When a container restarts before kubelet restarts, containerMap has
multiple entries (old exited + new running). GetContainerID() may
return the exited container, causing the running check to fail. Fixed
by checking if ANY container for the pod/name is running.

Also filter terminal pods from podresources since they no longer
consume resources, and fix test error handling to avoid exiting
Eventually immediately on transient errors.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2026-01-12 11:55:25 +01:00
Davanum Srinivas
1b33c52ef1 Fix data race in devicemanager PluginDisconnected logging
Don't log the endpoint struct in PluginDisconnected() - it contains
a gRPC client and logging it via reflection races with Close().

The logger walks the struct with reflection, reading gRPC internals,
while Close() is modifying mutex state in the background. Removing
the endpoint from the log avoids the race. The resourceName is already
logged which is enough to identify which endpoint disconnected.

This race becomes visible with gRPC v1.76.0+ due to timing changes
in buffer handling and goroutine lifecycle during Close(). The
ci-kubernetes-unit-dependencies job was hitting this ~10% of the time
when testing with updated dependencies.

CI failures:
- https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-unit-dependencies/2009065501396832256/build-log.txt
- https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-unit-dependencies/2008641956699181056/build-log.txt
- https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-unit-dependencies/2007491912692404224/build-log.txt
- https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-unit-dependencies/2007431281842851840/build-log.txt

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2026-01-09 08:19:33 -05:00
Sergey Kanzhelev
2d3e063923 mark device manager as haelthy before it started for the first time 2025-11-06 20:25:01 +00:00
Ed Bartosh
36510ccf56 devicemanager: fix cleanup of healthy and unhealthy devices
Correctly removed the resource from both healthy and unhealthy device
maps when resource endpoint expires.

This makes resource endpoint and devices maps consistent.
2025-11-03 10:57:58 +02:00
Ed Bartosh
7b2acb74c0 DRA: fix admit logic for extended resources
Updated allocateContainerResources to skip resources referring
extended resource claims as they are managed by DRA drivers.
These resources shouldn't be processed by device plugins.
2025-10-27 16:24:35 +02:00
Ed Bartosh
a20b34aa82 devicemanager: fix golangci-lint errors 2025-09-28 21:13:46 +03:00
Ed Bartosh
d45a4557c1 Migrate device manager to contextual logging 2025-09-26 16:50:34 +03:00
Kubernetes Prow Robot
a3bfeca891 Merge pull request #133728 from Frank-svg-dev/master
Fix the spelling error of grpc in the log
2025-09-16 20:00:09 -07:00
Kubernetes Prow Robot
8826364a57 Merge pull request #131483 from tallclair/kubeletconfig
Clean up pkg/kubelet/config
2025-09-15 15:54:07 -07:00
Kubernetes Prow Robot
7a1e1b6fb1 Merge pull request #130376 from phuhung273/pluginmanager-context-logging
kubelet: Migrate pkg/kubelet/pluginmanager to contextual logging
2025-09-03 12:55:13 -07:00
pamforever
731e6def04 Fix the spelling error of grpc in the log 2025-08-27 19:14:22 +08:00
Tim Allclair
ff6f4e1f63 Move kubelet config code to kubeletconfig 2025-08-06 12:08:19 -07:00
Sascha Grunert
c889ee17a2 Convert kubelet plugin manager from gogo to protoc
Use standard protoc for the kubelet plugin manager instead of gogo.

Part of https://github.com/kubernetes/kubernetes/issues/96564

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2025-07-23 16:55:05 +02:00
Kubernetes Prow Robot
47d9d86326 Merge pull request #133028 from saschagrunert/deviceplugin-proto
Convert `k8s.io/kubelet/pkg/apis/deviceplugin` from gogo to protoc
2025-07-21 14:14:55 -07:00
Sascha Grunert
3026020b44 Convert k8s.io/kubelet/pkg/apis/deviceplugin from gogo to protoc
Use standard protoc for the device plugin API instead of gogo.

Part of kubernetes#96564

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2025-07-21 10:04:01 +02:00
carlory
bd30b0adef remove general avaliable feature-gate DevicePluginCDIDevices
Signed-off-by: carlory <baofa.fan@daocloud.io>
2025-07-15 16:55:12 +08:00
phuhung273
5d20dc55bf chore(kubelet): migrate pluginmanager to contextual logging 2025-07-11 20:20:13 +07:00
Patrick Ohly
b471c2c11f DRA kubelet: support rolling upgrades
The key difference is that the kubelet must remember all plugin instances
because it could always happen that the new instance dies and leaves only the
old one running.

The endpoints of each instance must be different. Registering a plugin with the
same endpoint as some other instance is not supported and triggers an error,
which should get reported as "not registered" to the plugin. This should only
happen when the kubelet missed some unregistration event and re-registers the
same instance again. The recovery in this case is for the plugin to shut down,
remove its socket, which should get observed by kubelet, and then try again
after a restart.
2025-03-18 12:32:35 +01:00
mansikulkarni96
b9f6013c12 fix: handle socket file detection on Windows
Cherry-picked 4060ee60c1
Update socket file detection logic to use os.Stat as per upstream
Go fix for golang/go#33357. This resolves
the issue where socket files could not be properly identified on
Windows systems.
2025-02-21 18:02:28 -05:00
Kubernetes Prow Robot
39f1c90ac0 Merge pull request #129735 from swatisehgal/device-mgr-logs-improvements
Device Manager logging improvements
2025-02-07 07:11:56 -08:00
Swati Sehgal
f449697457 node: device-mgr: Adhere to the message style guidelines
Ensure that the log messages adhere to the message style guildelines
as captured [here](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines).

Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-02-06 18:13:10 +00:00
Swati Sehgal
40c86d8e06 node: device-mgr: Ensure consistent use of named arguments
Throughout the devicemanager codebase the named argument to represent
resource for logging pupose is `resourceName` as opposed to `resource`.
The latter can only be seen in topology_hints.go files. To ensure consistency
with the rest of the codebase and also because we want to adhere to the
recommendations in the Kubernetes documentation about named arguments:
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/migration-to-structured-logging.md#name-arguments
we update the key from `resource` to `resourceName`.

Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-02-06 17:22:24 +00:00
Swati Sehgal
19e406a357 node: device-mgr: Update klog.Infof(..., err) to klog.ErrorS(err,...)
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-02-06 17:21:53 +00:00
Swati Sehgal
9cd041156f node: device-mgr: Add logs in the happy path
We have reasonable amount of logs when things go wrong.
While debugging, it can be useful to have logs to indicate that
things have gone as expected.

Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-02-06 17:21:53 +00:00
Swati Sehgal
a585bd74c4 node: device-mgr: Keep log level consistent across all gRPC calls
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-02-06 17:21:53 +00:00
Swati Sehgal
4b613ae96f node: device-mgr: Bump log level for expected skips
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-02-06 17:21:53 +00:00
Swati Sehgal
584e224b43 node: device-mgr: Add metadata to logs
Ensure that if possible, we provide sufficient metadata
inclusing pod name and UID to allow filtering by pod name or its
UID.

Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-02-06 17:21:49 +00:00
Francesco Romani
e766b04ade node: cm: use maps.Clone instead of reinvent it
As pointed out in https://github.com/kubernetes/kubernetes/pull/128657#discussion_r1832973928

Signed-off-by: Francesco Romani <fromani@redhat.com>
2025-01-30 14:45:04 +01:00
Swati Sehgal
c56426bd9f node: device-mgr: Update klog.Infof(..., err) to klog.ErrorS(err,...)
Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-01-21 16:21:16 +00:00
Swati Sehgal
f8596d6d28 node: device-mgr: Change ErrorS(nil, ...) to InfoS
Ensure consistency across resource managers and update
ErrorS(nil, ...) to InfoS. Similar changes have been
proposed in CPU Manager and Memory Manager.

Signed-off-by: Swati Sehgal <swsehgal@redhat.com>
2025-01-21 16:21:09 +00:00
vivzbansal
763e810fb5 refactor code to add sidecar container support in IPPR 2024-11-07 21:20:48 +00:00
zhangzhifei16
1381e41f28 feat: Integrate device plugin registration gRPC server health checks. 2024-11-05 19:59:56 +08:00
Sergey Kanzhelev
1297d0cdd1 converge DRA and Device Plugin plugins registration 2024-10-30 16:58:13 +00:00
Kubernetes Prow Robot
a339a36a36 Merge pull request #127506 from ffromani/cpu-pool-size-metrics
node: metrics: add metrics about cpu pool sizes
2024-10-30 00:17:24 +00:00
AxeZhan
2ffb568540 rename functions 2024-10-25 12:53:24 +08:00
likakuli
212c4c4851 feat: revert #103979 for it's duplicated
Signed-off-by: likakuli <1154584512@qq.com>
2024-10-23 11:18:42 +02:00
Kubernetes Prow Robot
ded7ad554e Merge pull request #125513 from mauri870/hotfix/grpc-handle-err
kubelet/cm/devicemanager: log grpc Serve error
2024-10-18 02:49:03 +01:00
Matthieu MOREL
f736cca0e5 fix: enable expected-actual rule from testifylint in module k8s.io/kubernetes
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-09-27 07:56:31 +02:00
Matthieu MOREL
27b98be303 fix: enable nil-compare and error-nil rules from testifylint in module k8s.io/kubernetes
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-09-25 06:02:47 +02:00
rongfu.leng
ead64fb8f0 add resourceupdates.Update chan buffer
Signed-off-by: rongfu.leng <lenronfu@gmail.com>
2024-09-24 16:48:32 +00:00
Matthieu MOREL
fa0e38981c fix: enable compares rule from testifylint in module k8s.io/kubernetes
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2024-09-22 11:20:05 +02:00
rongfu.leng
d04a54c50b optimize code, filter podUID is empty string
Signed-off-by: rongfu.leng <lenronfu@gmail.com>
2024-09-13 01:48:14 +00:00
Kubernetes Prow Robot
14f2cab4de Merge pull request #126976 from jsturtevant/socket-file-revert
Revert "fix: handle socket file detection on Windows"
2024-09-03 18:31:16 +01:00
Kubernetes Prow Robot
a4ec0c039a Merge pull request #126435 from bart0sh/PR151-Kubelet-devicemanager-stop-using-CDI-annotations
Kubelet: stop using CDI annotations
2024-08-29 16:49:30 +01:00
James Sturtevant
3ca610757e Revert "fix: handle socket file detection on Windows"
This reverts commit 4060ee60c1.
2024-08-28 10:31:58 -07:00
Ed Bartosh
ea3c6628b7 Kubelet: stop using CDI annotations
Removing setting CDI annotations by the device manager as CRI field
CDIDevices is mature enough to be used instead.
2024-07-29 18:26:27 +03:00
Paco Xu
78d3830d97 ignore order of containers status allocated resources 2024-07-29 16:48:00 +08:00
Sergey Kanzhelev
62f96d2748 set AllocatedResourcesStatus in the Pod Status 2024-07-24 00:29:35 +00:00
Mauri de Souza Meneguzzo
694d813c55 kubelet/cm/devicemanager: handle grpc Serve error 2024-07-20 14:42:05 -03:00