As usual, consumers of an allocated claim react to the information stored in
the status. In this case, the scheduler did not copy the tolerations into the
status and as a result a pod with a toleration for NoExecute got scheduled and
then immediately evicted.
Some additional logging gets added to make the handling easier to track in the
eviction controller. Example YAMLs allow reproducing the use case manually.
This change introduces the ability for the Kubelet to monitor and report
the health of devices allocated via Dynamic Resource Allocation (DRA).
This addresses a key part of KEP-4680 by providing visibility into
device failures, which helps users and controllers diagnose pod failures.
The implementation includes:
- A new `v1alpha1.NodeHealth` gRPC service with a `WatchResources`
stream that DRA plugins can optionally implement.
- A health information cache within the Kubelet's DRA manager to track
the last known health of each device and handle plugin disconnections.
- An asynchronous update mechanism that triggers a pod sync when a
device's health changes.
- A new `allocatedResourcesStatus` field in `v1.ContainerStatus` to
expose the device health information to users via the Pod API.
Update vendor
KEP-4680: Fix lint, boilerplate, and codegen issues
Add another e2e test, add TODO for KEP4680 & update test infra helpers
Add Feature Gate e2e test
Fixing presubmits
Fix var names, feature gating, and nits
Fix DRA Health gRPC API according to review feedback
As before when adding v1beta2, DRA drivers built using the
k8s.io/dynamic-resource-allocation helper packages remain compatible with all
Kubernetes release >= 1.32. The helper code picks whatever API version is
enabled from v1beta1/v1beta2/v1.
However, the control plane now depends on v1, so a cluster configuration where
only v1beta1 or v1beta2 are enabled without the v1 won't work.
Add a cancelMainContext field to ExamplePlugin and options. This
function allows the plugin to cancel an upper-level context when a
background error occurs. The HandleError method now calls this
function if it is set.
This enables DRA drivers to perform a graceful shutdown when a fatal
background error triggers context cancellation.
Refactor StartPlugin and related test helpers to accept a variadic
list of options of any type, allowing both public and test-specific
options to be passed.
Added an errHandler callback to startGRPCServer for handling errors.
Replaced direct logging of server failures with calls to the
provided error handler. This change ensures that errors are surfaced
and handled consistently.
Co-authored-by: Ed Bartosh <eduard.bartosh@intel.com>
The helper code is useful for a separate Ginkgo suite for upgrade/downgrade
testing. We don't want to import test/e2e/dra there because that would also
define additional tests.
DRA drivers must provide ResourceSlices using the v1beta2 API types.
The controller then converts under the hood to v1beta1 if needed, i.e.
drivers are compatible with Kubernetes 1.32 and Kubernetes 1.33, as
long as at least one beta API group is enabled.
Testing pivots from using v1beta1 as the main API to v1beta2, with only one
test case exercising v1beta1.
A user of the controller can register an error handler via the controller
options. For a kubelet plugin, the error handler is method in the interface
which must be implemented. This is a conscious choice to make DRA driver
developer aware that they should react intelligently to errors.
The controller will invoke that handler with all errors that it encounters
while syncing the desired set of slices. This includes validation errors from
the apiserver if the driver's slices are invalid. Dropped fields get reported
with a special DroppedFieldsError.
YAML files were patched with:
sed -i -e 's;registry.k8s.io/e2e-test-images/agnhost:2...;registry.k8s.io/e2e-test-images/agnhost:2.54;' $(git grep -l agnhost:2 test/e2e/testing-manifests/ test/fixtures/)
The test/images/kitten and test/images/nautilus base images are still on an
older agnhost because updating those is better left to the owners.
`terminationGracePeriodSeconds: 0` was a mistake, it bypasses the normal
pod shutdown in the kubelet.
The right way to shut down a pod quickly is to have it react to SIGTERM.
The busybox implementation of "sleep" doesn't. `agnhost pause` does,
so let's use that instead.
For E2E tests, the InfiniteSleepCommand was already change about a year ago to
react to SIGTERM, so the `terminationGracePeriodSeconds: 1` workaround is no
longer needed.
Some more out-dated reference to resource class. Keeping the pod running is
better for demonstrating the lifecycle of claims because it is actually
possible to see a claim in the allocated state.
This adds the "DeviceTaint" top-level type to v1alpha3 and related fields to
ResourceSlice and ResourceClaim. It's complete enough bring up an API server
and generate files.
When the new RollingUpdate option is used, the DRA driver gets deployed such
that it uses unique socket paths and uses file locking to serialize gRPC
calls. This enables the kubelet to pick arbitrarily between two concurrently
instances. The handover is seamless (no downtime, no removal of ResourceSlices
by the kubelet).
For file locking, the fileutils package from etcd is used because that was
already a Kubernetes dependency. Unfortunately that package brings in some
additional indirect dependency for DRA drivers (zap, multierr), but those
seem acceptable.
When supporting rolling updates, we cannot use the same fixed socket paths for
old and new pod. With the revised API, the caller no longer specifies the full
socket paths, only directories. The logic about how to name sockets then can be
in the helper.
While at it, avoid passing a context to the gRPC helper code when
all that the helper code needs is a logger. That leads to confusion
about whether cancellation has an effect.
The goal is to simplify writing DRA drivers. This is also a first step towards
supporting seamless upgrades.
DRA drivers no longer need to implement the kubelet plugin API
directly. Instead, the helper wraps an implementation of an interface. The
helper then provides common functionality:
- retrieve and validate ResourceClaims
- serialize gRPC calls (enabled by default, can be opted out)
- gRPC logging
The definition of that interface is meant to be comprehensive enough that a
correct DRA driver can be implemented by following the documentation of the
package, without having to cross-reference KEPs.
The DRAPlugin interface used to be the abstract API of the helper. Now it's
what the DRA driver kubelet plugin needs to implement. The helper is a concrete
Server struct with no exported fields. It only exports the methods that
drivers need when using the helper.
While at it, support for the v1alpha4 API gets removed from the helper, which
implies removing the corresponding E2E tests. The kubelet implementation will
be dropped separately.
Reusing types from the alpha in the beta made it possible to provide and use
both versions without conversion. The downside was that removal of the alpha
would have been harder, if not impossible. DRA drivers could continue to
use the alpha types and provided the beta interface automatically.
Now the two versions are completely separate gRPC APIs, although in practice
there are no differences besides the name. Support for the alpha API in kubelet
is provided via automatically generated conversion and manually written
interface wrappers.
Those are provided as part of the v1alpha4 package. The advantage of having all
of that in a central place is that it'll be easier to remove when no longer
needed.
The v1beta1 API is identical to the previous v1alpha4, which erroneously was
still called "v1alpha3" in a few places, including the gRPC interface
definition itself.
The only reason for v1beta1 is to document the increased maturity of this API.
To simplify the transition, kubelet supports both v1alpha4 and v1beta1, picking
the more recent one automatically. All that DRA driver authors need to do to
implement v1beta1 is to update to the latest
k8s.io/dynamic-resource-allocation/kubeletplugin: it will automatically
register both API versions unless explicitly configured otherwise, which is
mostly just for testing.
DRA driver authors may replace their package import of v1alpha4 with v1beta1,
but they don't have to because the types in both packages are the same.
This removes the DRAControlPlaneController feature gate, the fields controlled
by it (claim.spec.controller, claim.status.deallocationRequested,
claim.status.allocation.controller, class.spec.suitableNodes), the
PodSchedulingContext type, and all code related to the feature.
The feature gets removed because there is no path towards beta and GA and DRA
with "structured parameters" should be able to replace it.
When draplugin was instanciated without nodeName or kubeclient
options, an error is returned when PublishResources is called.
E2E: Update DRA test app to support error from PublishResources
Signed-off-by: Alexey Fomenko <alexey.fomenko@intel.com>
This fixes the message (node name and "cluster-scoped" were switched) and
simplifies the VAP:
- a single matchCondition short circuits completely unless they're a user
we care about
- variables to extract the userNodeName and objectNodeName once
(using optionals to gracefully turn missing claims and fields into empty strings)
- leaves very tiny concise validations
Co-authored-by: Jordan Liggitt <liggitt@google.com>
The advantages of using a validation admission policy (VAP) are that no changes
are needed in Kubernetes and that admins have full flexibility if and how they
want to control which users are allowed to use "admin access" in their
requests.
The downside is that without admins taking actions, the feature is enabled
out-of-the-box in a cluster. Documentation for DRA will have to make it very
clear that something needs to be done in multi-tenant clusters.
The test/e2e/testing-manifests/dra/admin-access-policy.yaml shows how to do
this. The corresponding E2E tests ensures that it actually works as intended.
For some reason, adding the namespace to the message expression leads to a
type check errors, so it's currently commented out.
As agreed in https://github.com/kubernetes/enhancements/pull/4709, immediate
allocation is one of those features which can be removed because it makes no
sense for structured parameters and the justification for classic DRA is weak.
This is in preparation for revamping the resource.k8s.io completely. Because
there will be no support for transitioning from v1alpha2 to v1alpha3, the
roundtrip test data for that API in 1.29 and 1.30 gets removed.
Repeating the version in the import name of the API packages is not really
required. It was done for a while to support simpler grepping for usage of
alpha APIs, but there are better ways for that now. So during this transition,
"resourceapi" gets used instead of "resourcev1alpha3" and the version gets
dropped from informer and lister imports. The advantage is that the next bump
to v1beta1 will affect fewer source code lines.
Only source code where the version really matters (like API registration)
retains the versioned import.
In reality, the kubelet plugin of a DRA driver is meant to be deployed as a
daemonset with a service account that limits its
permissions. https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#additional-metadata-in-pod-bound-tokens
ensures that the node name is bound to the pod, which then can be used
in a validating admission policy (VAP) to ensure that the operations are
limited to the node.
In E2E testing, we emulate that via impersonation. This ensures that the plugin
does not accidentally depend on additional permissions.
