Fixes TestConnectionHandling/no-wipe-on-reconnect which was failing with
"Only one usage of each socket address is normally permitted" because the
teardown function wasn't waiting for the gRPC server to fully shut down
before returning, causing socket cleanup issues in subsequent test runs.
The fix ensures proper synchronization during server shutdown so that the
Unix socket is fully released before the teardown completes. This prevents
socket binding conflicts when tests run in sequence.
When gRPC notifies the kubelet that a connection ended, the kubelet tries to
reconnect because it needs to know when a DRA driver comes back. The same code
gets called when a connection goes idle, by default after 30 minutes. In that
and only that case the conn.Connect call deadlocks while calling into the gRPC
idle manager.
This can be reproduced with a new unit test which artificially shortens the
idle timeout. This fix is to move the Connect call into a goroutine because
then both HandleConn and Connect can proceed. It's sufficient that Connect
finishes at some point, it doesn't need to be immediately.
This change introduces the ability for the Kubelet to monitor and report
the health of devices allocated via Dynamic Resource Allocation (DRA).
This addresses a key part of KEP-4680 by providing visibility into
device failures, which helps users and controllers diagnose pod failures.
The implementation includes:
- A new `v1alpha1.NodeHealth` gRPC service with a `WatchResources`
stream that DRA plugins can optionally implement.
- A health information cache within the Kubelet's DRA manager to track
the last known health of each device and handle plugin disconnections.
- An asynchronous update mechanism that triggers a pod sync when a
device's health changes.
- A new `allocatedResourcesStatus` field in `v1.ContainerStatus` to
expose the device health information to users via the Pod API.
Update vendor
KEP-4680: Fix lint, boilerplate, and codegen issues
Add another e2e test, add TODO for KEP4680 & update test infra helpers
Add Feature Gate e2e test
Fixing presubmits
Fix var names, feature gating, and nits
Fix DRA Health gRPC API according to review feedback
As before when adding v1beta2, DRA drivers built using the
k8s.io/dynamic-resource-allocation helper packages remain compatible with all
Kubernetes release >= 1.32. The helper code picks whatever API version is
enabled from v1beta1/v1beta2/v1.
However, the control plane now depends on v1, so a cluster configuration where
only v1beta1 or v1beta2 are enabled without the v1 won't work.
Call p.conn.Close() in DRAPluginManager.remove to ensure the gRPC
connection is properly closed when a plugin is unregistered. This
prevents reconnecting attempts, resource leaks and ensures clean
shutdown of plugin connections during plugin removal.
Update setupFakeGRPCServer to ignore grpc.ErrServerStopped when serving
the gRPC test server. This prevents unnecessary panics during server
shutdown and ensures clean test teardown. The change improves test
stability by allowing expected server stop errors to be handled gracefully.
The API is a direct copy of v1beta1, with v1beta1 replaced by v1.
The interoperability support is the same that was used for v1alpha4.
Adding it "reverts" the removal of the v1alpha4 support in
a57f15e081, except that now v1beta1 is the legacy
API which needs conversion. If kubelet and the plugin both support v1, no
conversion is needed.
v1alpha4 was added in 1.31 and superseded by v1beta1 in 1.32. Since that
release, plugins are also required to advertise the supported gRPC services
during registration. In practice, all known DRA drivers use the helper code
from 1.32 or newer and thus don't need the legacy support.
Using the same socket path for different test cases caused
test failure on windows:
listen unix C:\Users\azureuser\AppData\Local\Temp\TestRegistrationHandler3881105518\001/dra-plugin-a.sock:
bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
Creating unique socket path for every test case should fix it.
Unix socket path has to be <= 108 characters in length on Windows.
Shortened DRA socket path for TestConnectionHandling and
TestRegistrationHandler tests should fix the test run on Windows.
Conceptually TimedWorkersQueue is similar to the current code: it spawns
goroutines and cancels them. Using it makes the code a bit shorter, even though
the TimedWorkersQueue API could be a bit nicer and more consistent (key string
vs. WorkerArgs as parameters).
Depending on the tainteviction package is a bit odd, which is the reason why
TimedWorkersQueue wasn't already used earlier. But there don't seem to be other
implementations of this common
problem. https://pkg.go.dev/k8s.io/client-go/util/workqueue#TypedDelayingInterface
doesn't work because queue entries cannot be removed.
This doesn't really solve the problem of tracking goroutines for wiping because
TimedWorkersQueue doesn't support that. But not tracking is arguably better
than doing it wrong and this only affects unit tests, so it should be okay.
This ensures that ResourceSlices get removed also when a plugin becomes
unresponsive without removing the registration socket.
Tests are from https://github.com/kubernetes/kubernetes/pull/131073 by Ed
with some modifications, the implementation is new.
Instead of creating the gRPC connection on demand and forcing gRPC to connect,
we establish it immediately and rely on gRPC to handle the underlying
connection automatically like it usually does.
It's not clear what benefit the one second connection timeout had. The way it
is now, gRPC calls still fail when the underlying connection cannot be
established. Having to have a separate context for establishing that connection
just made the code more complex.
The DRAPluginManager is the central component which manages plugins. Making it responsible
for creating them reduces the number of places where a DRAPlugin struct needs to
be initialized. Doing this in the DRAPluginManager instead of a stand-alone function
simplifies the implementation of connection monitoring, because that will be
something that is tied to the DRAPluginManager state.
After merging with RegistrationHandler, the store abstraction is more than a
dump map. It implements additional logic, so renaming it seems warranted.
To avoid confusion with other "plugin managers", the DRA prefix is used.
The plugin type gets updated accordingly.
This is done in a separate commit for ease of review. The "store" field is
kept because that really is just a dump lookup structure.
This moves wiping into the plugins store. The advantages are:
- Only a single mutex is needed in the plugin store.
- The code which decides about queuing and canceling wiping
has access to all relevant information while that information
is protected against concurrent modifications.
- It prepares for making that code more complex ("connection
monitoring").
In retrospect it is not clear whether the RegistrationHandler did the right
thing in all cases when endpoints got registered and removed concurrently.
The RegistrationHandler became a thin, stateless layer on top of the store.
Because it's not really needed anymore, the required methods for the plugin
manager now get provided directly by the store.
The disadvantage is the slightly more complex initialization, but that was also
a problem before which just hadn't been solved: wiping ran without the context
of the manager. Now it does.
The rest of the system logs information using "driverName" as key in structured
logging. The kubelet should do the same.
This also gets clarified in the code, together with using consistent a
consistent name for a Plugin pointer: "plugin" instead of "client" or
"instance".
The New in NewDRAPluginClient made no sense because it's not constructing
anything, and it returns a plugin, not a client -> GetDRAPlugin.
Replaced the Manager interface with a simple struct. There was no need for the
interface.
Replaced the global instance of the DRA plugins store with a fresh instanced
owned by the manager. This makes unit testing a bit easier (no need to restore
state, would enable parallelizing long-running tests).
Simplified the plugin.PluginsStore type to just "plugin.Store" because it
stuttered. The Plugin type is kept because having one struct named after its
package is one common exception from the "don't stutter" guideline.
For the sake of clarify, the package gets imported as "draplugin" (=
<parent dir>/<package>).
Removed unused NewManager "node" parameter and replaced direct construction of
cache and manager with calls to NewManager because that is how the manager
should get constructed (less code, too).
Fixed incorrect description of Manager: the plugin store is the entity which
manages drivers, not the manager. The manager is focused on the DRA logic
around ResourceClaims.
If preparing for a pod failed because the driver was not registered, it failed
after already having added the pod and claim to the claim info cache. The
effect was that deletion of the pod was blocked until the driver got
installed. Other errors had a similar effect. Now as many errors as possible
are checked in a read-only loop before proceeding to changing the claim info
cache.
The error message that was surfaced for the problem was not very readable:
Failed to prepare dynamic resources: failed to get gRPC client for driver dra-1411.k8s.io: plugin name dra-1411.k8s.io not found in the list of registered DRA plugins
To address this, error messages and wrapping get updated according to these
guidelines:
// Most errors returned by the manager show up in the context of a pod.
// They try to adher to the following convention:
// - Don't include the pod.
// - Use terms that are familiar to users.
// - Don't include the namespace, it can be inferred from the context.
// - Avoid repeated "failed to ...: failed to ..." when wrapping errors.
// - Avoid wrapping when it does not provide relevant additional information to keep the user-visible error short.
If the test binary ran long enough after test completion to reach the
ResourceSlice removal grace period, that background activity started and failed
because it was using out-dated state and an invalid testing.T pointer, causing
a panic.
The root cause was to leave those background activities running. They need to
be stopped before a test returns.
The key difference is that the kubelet must remember all plugin instances
because it could always happen that the new instance dies and leaves only the
old one running.
The endpoints of each instance must be different. Registering a plugin with the
same endpoint as some other instance is not supported and triggers an error,
which should get reported as "not registered" to the plugin. This should only
happen when the kubelet missed some unregistration event and re-registers the
same instance again. The recovery in this case is for the plugin to shut down,
remove its socket, which should get observed by kubelet, and then try again
after a restart.
When doing an update of a DaemonSet, first the old pod gets stopped and
then the new one is started. This causes the kubelet to remove all
ResourceSlices directly after removal and forces the new pod to recreate all of
them.
Now the kubelet waits 30 seconds before it deletes ResourceSlices. If a new
driver registers during that period, nothing is done at all. The new driver
finds the existing ResourceSlices and only needs to update them if something
changed.
The downside is that if the driver gets removed permanently, this creates a
delay where pods might still get scheduled to the node although the driver is
not going to run there anymore and thus the pods will be stuck.
Set expected slice fields in the reactor function instead of
test cleanup instead of doing it in the test cleanup.
This should fix the test failure caused by kubelet calling reactor function
before the test cleanup sets the deleteCollectionForDriver variable.
Reusing types from the alpha in the beta made it possible to provide and use
both versions without conversion. The downside was that removal of the alpha
would have been harder, if not impossible. DRA drivers could continue to
use the alpha types and provided the beta interface automatically.
Now the two versions are completely separate gRPC APIs, although in practice
there are no differences besides the name. Support for the alpha API in kubelet
is provided via automatically generated conversion and manually written
interface wrappers.
Those are provided as part of the v1alpha4 package. The advantage of having all
of that in a central place is that it'll be easier to remove when no longer
needed.
Listing supported gRPC services (e.g. drav1alpha3.Node, drav1beta1.DRAPlugin)
during registration enables the kubelet to determine in advance which methods
it can call.
Versioning by Kubernetes release makes less sense because it doesn't say
anything about which gRPC service is supported. New ones might get added and
obsolete ones removed. Some services might be optional.
In the past, this versioning support wasn't really used. At least one version
had to be provided and kubelet tried to use the plugin with the highest
version. This version comparison gets dropped. In the unlikely situation
that different plugins register under the same name, the most recent one is
used.
Because advertising gRPC services is a new convention, plugins only reporting
some version are treated as providing the old alpha gRPC service.
The v1beta1 API is identical to the previous v1alpha4, which erroneously was
still called "v1alpha3" in a few places, including the gRPC interface
definition itself.
The only reason for v1beta1 is to document the increased maturity of this API.
To simplify the transition, kubelet supports both v1alpha4 and v1beta1, picking
the more recent one automatically. All that DRA driver authors need to do to
implement v1beta1 is to update to the latest
k8s.io/dynamic-resource-allocation/kubeletplugin: it will automatically
register both API versions unless explicitly configured otherwise, which is
mostly just for testing.
DRA driver authors may replace their package import of v1alpha4 with v1beta1,
but they don't have to because the types in both packages are the same.
Added error assertion for NodePrepareResources call unveiled
"rpc error: code = DeadlineExceeded desc = context deadline exceeded"
failure in the TestGRPCConnIsReused test.
Setting clientCallTimeout field when creating plugin should fix it.
Some of the E2E node tests were flaky. Their timeout apparently was chosen
under the assumption that kubelet would retry immediately after a failed gRPC
call, with a factor of 2 as safety margin. But according to
0449cef8fd,
kubelet has a different, higher retry period of 90 seconds, which was exactly
the test timeout. The test timeout has to be higher than that.
As the tests don't use the gRPC call timeout anymore, it can be made
private. While at it, the name and documentation gets updated.
This adds the ability to select specific requests inside a claim for a
container.
NodePrepareResources is always called, even if the claim is not used by any
container. This could be useful for drivers where that call has some effect
other than injecting CDI device IDs into containers. It also ensures that
drivers can validate configs.
The pod resource API can no longer report a class for each claim because there
is no such 1:1 relationship anymore. Instead, that API reports claim,
API devices (with driver/pool/device as ID) and CDI device IDs. The kubelet
itself doesn't extract that information from the claim. Instead, it relies on
drivers to report this information when the claim gets prepared. This isolates
the kubelet from API changes.
Because of a faulty E2E test, kubelet was told to contact the wrong driver for
a claim. This was not visible in the kubelet log output. Now changes to the
claim info cache are getting logged. While at it, naming of variables and some
existing log output gets harmonized.
Co-authored-by: Oksana Baranova <oksana.baranova@intel.com>
Co-authored-by: Ed Bartosh <eduard.bartosh@intel.com>
This is in preparation for revamping the resource.k8s.io completely. Because
there will be no support for transitioning from v1alpha2 to v1alpha3, the
roundtrip test data for that API in 1.29 and 1.30 gets removed.
Repeating the version in the import name of the API packages is not really
required. It was done for a while to support simpler grepping for usage of
alpha APIs, but there are better ways for that now. So during this transition,
"resourceapi" gets used instead of "resourcev1alpha3" and the version gets
dropped from informer and lister imports. The advantage is that the next bump
to v1beta1 will affect fewer source code lines.
Only source code where the version really matters (like API registration)
retains the versioned import.
