The "create 100 slices" E2E sometimes flaked with timeouts (e.g. 95 out of 100
slices created). It created too much load for an E2E test.
The same test now uses ktesting as API, which makes it possible to run it as
integration test with the original 100 slices and with more moderate 10 slices
as E2E test.
(cherry picked from commit c47ad64820)
Refactoring the DRA upgrade/downgrade testing such that it runs as Go test
depended on supporting ktesting in the E2E framework. That change worked during
presubmit testing, but broke some periodic jobs. Therefore the relevant commits
from https://github.com/kubernetes/kubernetes/pull/135664/commits get reverted:
c47ad64820 DRA e2e+integration: test ResourceSlice controller
047682908d ktesting: replace Begin/End with TContext.Step
de47714879 DRA upgrade/downgrade: rewrite as Go unit test
7c7b1e1018 DRA e2e: make driver deployment possible in Go unit tests
65ef31973c DRA upgrade/downgrade: split out individual test steps
47b613eded e2e framework: support creating TContext
The last one is what must have caused the problem, but the other commits depend
on it.
The "create 100 slices" E2E sometimes flaked with timeouts (e.g. 95 out of 100
slices created). It created too much load for an E2E test.
The same test now uses ktesting as API, which makes it possible to run it as
integration test with the original 100 slices and with more moderate 10 slices
as E2E test.
The original implementation was inspired by how context.Context is handled via
wrapping a parent context. That approach had several issues:
- It is useful to let users call methods (e.g. tCtx.ExpectNoError)
instead of ktesting functions with a tCtx parameters, but that only
worked if all implementations of the interface implemented that
set of methods. This made extending those methods cumbersome (see
the commit which added Require+Assert) and could potentially break
implementations of the interface elsewhere, defeating part of the
motivation for having the interface in the first place.
- It was hard to see how the different TContext wrappers cooperated
with each other.
- Layering injection of "ERROR" and "FATAL ERROR" on top of prefixing
with the klog header caused post-processing of a failed unit test to
remove that line because it looked like log output. Other log output
lines where kept because they were not indented.
- In Go <=1.25, the `go vet sprintf` check only works for functions and
methods if they get called directly and themselves directly pass their
parameters on to fmt.Sprint. The check does not work when calling
methods through an interface. Support for that is coming in Go 1.26,
but will depend on bumping the Go version also in go.mod and thus
may not be immediately possible in Kubernetes.
- Interface documentation in
https://pkg.go.dev/k8s.io/kubernetes@v1.34.2/test/utils/ktesting#TContext
is a monolithic text block. Documentation for methods is more readable and allows
referencing those methods with [] (e.g. [TC.Errorf] works, [TContext.Errorf]
didn't).
The revised implementation is a single struct with (almost) no exported
fields. The two exceptions (embedded context.Context and TB) are useful because
it avoids having to write wrappers for several functions resp. necessary
because Helper cannot be wrapped. Like a logr.LogSink, With* methods can make a
shallow copy and then change some fields in the cloned instance.
The former `ktesting.TContext` interface is now a type alias for
`*ktesting.TC`. This ensures that existing code using ktesting doesn't need to
be updated and because that code is a bit more compact (`tCtx
ktesting.TContext` instead of `tCtx *ktesting.TContext` when not using such an
alias). Hiding that it is a pointer might discourage accessing the exported
fields because it looks like an interface.
Output gets fixed and improved such that:
- "FATAL ERROR" and "ERROR" are at the start of the line, followed by the klog header.
- The failure message follows in the next line.
- Continuation lines are always indented.
The set of methods exposed via TB is now a bit more complete (Attr, Chdir).
All former stand-alone With* functions are now also available as methods and
should be used instead of the functions. Those will be removed.
Linting of log calls now works and found some issues.
- Rescheduling on binding failure using taints or device removal
- Triggering binding timeout when BindingConditions remain unmet
- Recovery to a device without BindingConditions after binding timeout
Signed-off-by: Tsubasa Watanabe <w.tsubasa@fujitsu.com>
To improve maintainability and reduce the size of dra_test.go,
this commit extracts the function "testDeviceBindingConditions"
into a new file: binding_conditions_test.go.
No functional changes are introduced.
Signed-off-by: Tsubasa Watanabe <w.tsubasa@fujitsu.com>
Support for DeviceTaintRules depends on a significant amount of
additional code:
- ResourceSlice tracker is a NOP without it.
- Additional informers and corresponding permissions in scheduler and controller.
- Controller code for handling status.
Not all users necessarily need DeviceTaintRules, so adding a second feature
gate for that code makes it possible to limit the blast radius of bugs in that
code without having to turn off device taints and tolerations entirely.
This raises the number of allowed taints per device to 16 by lowering
the number of allowed devices to 64 per ResourceSlice if (and only if!)
taints are used.
"effect: None" and DeviceTaintRule status with conditions get added
to support giving feedback to admins.
Instead of merely adding the new effect value, this also changes validation of
the enum so that unknown values are valid if they were already stored. This
will simplify adding new effects in the future because validation won't fail
for them after a downgrade. Consumers must treat them like this new None
effect, i.e. ignore them.
They now live in a separate file, use ktesting+Gomega consistently, and
follow (mostly) best practices:
- strict validation during create
- wait for completion of delete
The latter is not complete yet because it was found that some binding condition
test keeps a ResourceCLaim in an undeletable state. That's okayish (namespace
stays around until apiserver gets restarted), but is worth looking into later.
The core functionality was enabled by default in 1.34 without any issues that
would have suggested turning it off, so now we can lock it to on-by-default.
Tests which cover disabling the feature must use version emulation.
It hasn't been on-by-default before, therefore it does not get locked to the
new default on yet. This has some impact on the scheduler configuration
because the plugin is now enabled by default.
Because the feature is now GA, it doesn't need to be a label on E2E tests,
which wouldn't be possible anyway once it gets removed entirely.
As before when adding v1beta2, DRA drivers built using the
k8s.io/dynamic-resource-allocation helper packages remain compatible with all
Kubernetes release >= 1.32. The helper code picks whatever API version is
enabled from v1beta1/v1beta2/v1.
However, the control plane now depends on v1, so a cluster configuration where
only v1beta1 or v1beta2 are enabled without the v1 won't work.
Calling klog.FlushAndExit causes the `go test` binary to quit
without properly recording which test failed. Both callers of
StartScheduler already have a ktesting.TContext, so switching
to that is easy and also reduces the number of parameters.
This covers disabling the feature via the configuration, failing to schedule
because of timeouts for all nodes, and retrying after ResourceSlice changes with
partial success (timeout for one node, success for the other).
While at it, some helper code gets improved.
It's unclear why k8s.io/kubernetes/pkg/apis/resource/install needs
to be imported explicitly. Having the apiserver and scheduler ready
to be started ensures that all APIs are available.
If a ResourceSlice got published by the ResourceSlice controller in a DRA
driver and then that ResourceSlice got deleted quickly (within one minute, the
mutation cache TTL) by someone (for example, the kubelet because of a restart),
then the controller did not react properly to the deletion unless some other
event triggered the syncing of the pool.
Found while adding upgrade/downgrade tests with a driver which keeps running
across the upgrade/downgrade.
The exact sequence leading to this were:
- controller adds ResourceSlice, schedules a sync for one minute in the future (the TTL)
- someone else deletes the ResourceSlice
- add and delete events schedule another sync 30 seconds in the future (the delay),
*overwriting* the other scheduled sync
- sync runs once, finds deleted slices in the mutation cache,
does not re-create them, and also does not run again
One possible fix would be to set a resync period. But then work is done
periodically, whether it's necessary or not.
Another fix is to ensure that the TTL is shorter than the delay. Then when a
sync occurs, all locally stored additional slices are expired. But that renders
the whole storing of recently created slices in the cache pointless.
So the fix used here is to keep track of when another sync has to run because
of added slices. At the end of each sync, the next sync gets scheduled if (and
only if) needed, until eventually syncing can stop.
The v1alpha3 version is still needed for DeviceTaintRule, but the rest of the
types and most structs became obsolete in v1.32 when we introduced v1beta1 and
bumped the storage version to v1beta1.
Removing them now simplifies adding new features because new fields don't need
to be added to these obsolete types. This could have been done already in 1.33,
but wasn't to minimize disrupting on-going work.
This moves the enabled/disabled test into the common test/integration/dra which
simplifies the code a bit and amortizes the cost of starting the apiserver
because several different tests can use the same instance, running in parallel.
While at it, setting the status via SSA also gets tested.
As soon as we have more than one test using the scheduler, we need some
coordination between tests. This is handled by a singleton which starts the
scheduler for the first user and stops it after the last one is gone.
To avoid having to pass around an additional parameter, the context is used to
access the singleton under the hood.
This enables proper scheduling tests. Most of them are probably better done in
scheduler_perf where the same test then can also be used for benchmarking and
creating objects is a bit better supported (from YAML, for example), but some
special cases (in particular, anything involving error injection) are better
done here.
DRA drivers must provide ResourceSlices using the v1beta2 API types.
The controller then converts under the hood to v1beta1 if needed, i.e.
drivers are compatible with Kubernetes 1.32 and Kubernetes 1.33, as
long as at least one beta API group is enabled.
Testing pivots from using v1beta1 as the main API to v1beta2, with only one
test case exercising v1beta1.
A user of the controller can register an error handler via the controller
options. For a kubelet plugin, the error handler is method in the interface
which must be implemented. This is a conscious choice to make DRA driver
developer aware that they should react intelligently to errors.
The controller will invoke that handler with all errors that it encounters
while syncing the desired set of slices. This includes validation errors from
the apiserver if the driver's slices are invalid. Dropped fields get reported
with a special DroppedFieldsError.
Both the new DeviceTaint.TimeAdded and dropped fields when
the DRADeviceTaints feature is disabled confused the ResourceSlice
controller because what is stored and sent back can be different
from what the controller wants to store.
It's now more lenient regarding TimeAdded (doesn't need to be exact because of
rounding during serialization, only having a value on the server is okay)
and dropped fields (doesn't try to store them again). It also preserves
a server-side TimeAdded when updating slices.
