As usual, consumers of an allocated claim react to the information stored in
the status. In this case, the scheduler did not copy the tolerations into the
status and as a result a pod with a toleration for NoExecute got scheduled and
then immediately evicted.
Some additional logging gets added to make the handling easier to track in the
eviction controller. Example YAMLs allow reproducing the use case manually.
Promoting real tests turned out to be harder than expected (should be rewritten
to be self-contained, additional reviews, etc.).
They would not achieve 100% endpoint+operation coverage because real tests only
use some of the operations. Therefore each API type has to be covered with
CRUD-style tests which only exercise the apiserver, then maybe additional
functional tests can be added later (depending on time and motivation).
The machinery for testing different API types is meant to be reusable, so it
gets added in the new e2e/framework/conformance helper package.
That was the original intent, but the implementation then ended up checking
ResourceClaims in all namespaces. Depending on timing this was merely
misleading (showing ResourceClaim changes from a different test running in
parallel), but with upcoming CRUD tests which intentionally set an allocation
result without a finalizer it breaks the non-CRUD tests when they check the
those CRUD ResourceClaims.
The initial testing of DRAExtendedResource changed the common setup for all
tests so that each test runs with seven DeviceClasses. This is unnecessary and
confusing, for example when looking at the endpoint stats for the simple
conformance test ("Are the stats correct? The test should only create one
class.").
This update changes it so that each test which needs more than the default
class explicitly creates those itself. For clarity, the special -1 index gets a
name and naming of the DeviceClasses gets updated to keep the default class
simple (no additional suffix).
Using the same "example.com/resource0" in tests which run in parallel might
explain https://github.com/kubernetes/kubernetes/issues/133653. This gets
replaced with extended resource names based on the driver name, which is unique
in each parallel test.
The test needs to schedule 256 pods at once, which only works with three
nodes (default limit is 100, but could also be lower). It's also a stress test
which flaked recently.
For now it gets removed without a replacement. A similar integration test is in
development, but too big (needs some infrastructure changes in
test/integration/dra) to add during code freeze.
This change introduces the ability for the Kubelet to monitor and report
the health of devices allocated via Dynamic Resource Allocation (DRA).
This addresses a key part of KEP-4680 by providing visibility into
device failures, which helps users and controllers diagnose pod failures.
The implementation includes:
- A new `v1alpha1.NodeHealth` gRPC service with a `WatchResources`
stream that DRA plugins can optionally implement.
- A health information cache within the Kubelet's DRA manager to track
the last known health of each device and handle plugin disconnections.
- An asynchronous update mechanism that triggers a pod sync when a
device's health changes.
- A new `allocatedResourcesStatus` field in `v1.ContainerStatus` to
expose the device health information to users via the Pod API.
Update vendor
KEP-4680: Fix lint, boilerplate, and codegen issues
Add another e2e test, add TODO for KEP4680 & update test infra helpers
Add Feature Gate e2e test
Fixing presubmits
Fix var names, feature gating, and nits
Fix DRA Health gRPC API according to review feedback
It hasn't been on-by-default before, therefore it does not get locked to the
new default on yet. This has some impact on the scheduler configuration
because the plugin is now enabled by default.
Because the feature is now GA, it doesn't need to be a label on E2E tests,
which wouldn't be possible anyway once it gets removed entirely.
As before when adding v1beta2, DRA drivers built using the
k8s.io/dynamic-resource-allocation helper packages remain compatible with all
Kubernetes release >= 1.32. The helper code picks whatever API version is
enabled from v1beta1/v1beta2/v1.
However, the control plane now depends on v1, so a cluster configuration where
only v1beta1 or v1beta2 are enabled without the v1 won't work.
There's no need to clarify how many nodes are used in the test because the
overall test names are still unique without that (verified with go test -v
./test/e2e -args -list-tests | grep -w DRA | wc -l).
"must be possible for the driver to update the ResourceClaim.Status.Devices
once allocated" was also run as kubelet test although it only checks the
control plane.
Before:
[sig-node] [DRA] [FeatureGate:DynamicResourceAllocation] [Beta] [Feature:OffByDefault] control plane with single node [ConformanceCandidate] must be possible for the driver to update the ResourceClaim.Status.Devices once allocated [FeatureGate:DRAResourceClaimDeviceStatus] [Beta]
[sig-node] [DRA] [FeatureGate:DynamicResourceAllocation] [Beta] [Feature:OffByDefault] kubelet [Feature:DynamicResourceAllocation] on single node must be possible for the driver to update the ResourceClaim.Status.Devices once allocated [FeatureGate:DRAResourceClaimDeviceStatus] [Beta]
After:
[sig-node] [DRA] [FeatureGate:DynamicResourceAllocation] [Beta] [Feature:OffByDefault] control plane with single node [ConformanceCandidate] must be possible for the driver to update the ResourceClaim.Status.Devices once allocated [FeatureGate:DRAResourceClaimDeviceStatus] [Beta]
All of these tests already ran successfully in
https://testgrid.k8s.io/sig-release-master-informing#kind-master-beta&include-filter-by-regex=DRA
and thus can get promoted to a conformance tests once DRA is GA.
The "control plane truncates the name of a generated resource claim" was marked
as depending on a recent kubelet incorrectly.
Removal of the feature label fixes b84271a779.
Removal of KubeletMinVersion for control plane tests depends on running only
kubelet tests in the kubelet version skew jobs.
Upgrade/downgrade testing after the DRA GA graduation gets stuck because the
ResourceSlice controller's informer doesn't pick up that the ResourceSlice got
deleted.
This additional log output shows a List+Watch after a watch got closed due to a
disconnect from the apiserver, so right now everything is working as expected.
The API is a direct copy of v1beta1, with v1beta1 replaced by v1.
The interoperability support is the same that was used for v1alpha4.
Adding it "reverts" the removal of the v1alpha4 support in
a57f15e081, except that now v1beta1 is the legacy
API which needs conversion. If kubelet and the plugin both support v1, no
conversion is needed.
Add a cancelMainContext field to ExamplePlugin and options. This
function allows the plugin to cancel an upper-level context when a
background error occurs. The HandleError method now calls this
function if it is set.
This enables DRA drivers to perform a graceful shutdown when a fatal
background error triggers context cancellation.
Refactor StartPlugin and related test helpers to accept a variadic
list of options of any type, allowing both public and test-specific
options to be passed.
Added an errHandler callback to startGRPCServer for handling errors.
Replaced direct logging of server failures with calls to the
provided error handler. This change ensures that errors are surfaced
and handled consistently.
Co-authored-by: Ed Bartosh <eduard.bartosh@intel.com>
The test brings up the cluster and uses that power to run through
an upgrade/downgrade scenario. Version skew testing (running tests while the cluster
is partially up- or downgraded) could be added.
The new helper code for managing the cluster is written so that it could be
used both in an integration test and an E2E
test. https://github.com/kubernetes/kubernetes/pull/122481 could make that a
bit easier in an E2E test, but is not absolutely required.
In contrast to running on a normal cluster, pods need no privileges.
Instead, the caller has to make sure that the test itself can write
into system directories used by the cluster.
The helper code is useful for a separate Ginkgo suite for upgrade/downgrade
testing. We don't want to import test/e2e/dra there because that would also
define additional tests.
If we know that the test binary shares the filesystem with the cluster (for
example, when using local-up-cluster.sh), then we can avoid the whole
complicated portproxy solution and work directly with the paths on the
host.
Only works with suitable permissions! /var/lib/kubelet/plugins,
/var/lib/kubelet/plugin_registry, and /var/run/cdi must be writable.
portproxy remains the default because it automatically gains sufficient
permissions also when combined with local-up-cluster.sh.
This is a DRA-specific stop-gap solution for using the E2E framework together
with ktesting. Long-term this should better land in the E2E framework itself.
There was one flake:
expected pod to be pod is scheduled, got instead:
...
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2025-07-06T12:08:54Z"
message: 'running "DynamicResources" filter plugin: pool network-attached is invalid:
duplicate device name dev-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0000'
The root cause was that the test produced device names which were unique within
their slice, but not within the pool. This didn't matter for the test itself,
but for tests running concurrently of after it, because those slices were also
not cleaned up properly.
Timing in pull-kubernetes-e2e-gce also was different.
After enabling more tests in 00dd610c93,
periodic ci-kind-dra-all became unstable, with random flakes at the time when
the "supports sharing a claim sequentially". Probably the cluster became
overwhelmed by the number of pods which wanted to run in parallel. Running this
tests in the serial phase of a Ginkgo run should fix this.
Tests which only exercise the control plane don't need DRA drivers on the nodes
and thus can run in any cluster where the API and feature gate is
enabled. Eventually they can become conformance tests.
The actual test cases follow the same pattern and in some cases are run twice,
once for "control plane" testing and once for "kubelet" testing. The difference
is that in "control plane" mode, the driver's don't get deployed and pods are
only expected to get scheduled instead of starting to run.
Commit 71b2f32d70 missed one test which needs the
"kubelet >= 1.33" label because that test didn't run in the canary
presubmit. It's now failing in the ci-kind-dra-n-2 job.
