Update scheduler performance test examples to use `-benchtime=1x`
instead of `-benchtime=1ns` for explicitly running each benchmark
exactly once. This makes the intent clearer and aligns the examples
with recommended Go benchmark usage.
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
The original implementation was inspired by how context.Context is handled via
wrapping a parent context. That approach had several issues:
- It is useful to let users call methods (e.g. tCtx.ExpectNoError)
instead of ktesting functions with a tCtx parameters, but that only
worked if all implementations of the interface implemented that
set of methods. This made extending those methods cumbersome (see
the commit which added Require+Assert) and could potentially break
implementations of the interface elsewhere, defeating part of the
motivation for having the interface in the first place.
- It was hard to see how the different TContext wrappers cooperated
with each other.
- Layering injection of "ERROR" and "FATAL ERROR" on top of prefixing
with the klog header caused post-processing of a failed unit test to
remove that line because it looked like log output. Other log output
lines where kept because they were not indented.
- In Go <=1.25, the `go vet sprintf` check only works for functions and
methods if they get called directly and themselves directly pass their
parameters on to fmt.Sprint. The check does not work when calling
methods through an interface. Support for that is coming in Go 1.26,
but will depend on bumping the Go version also in go.mod and thus
may not be immediately possible in Kubernetes.
- Interface documentation in
https://pkg.go.dev/k8s.io/kubernetes@v1.34.2/test/utils/ktesting#TContext
is a monolithic text block. Documentation for methods is more readable and allows
referencing those methods with [] (e.g. [TC.Errorf] works, [TContext.Errorf]
didn't).
The revised implementation is a single struct with (almost) no exported
fields. The two exceptions (embedded context.Context and TB) are useful because
it avoids having to write wrappers for several functions resp. necessary
because Helper cannot be wrapped. Like a logr.LogSink, With* methods can make a
shallow copy and then change some fields in the cloned instance.
The former `ktesting.TContext` interface is now a type alias for
`*ktesting.TC`. This ensures that existing code using ktesting doesn't need to
be updated and because that code is a bit more compact (`tCtx
ktesting.TContext` instead of `tCtx *ktesting.TContext` when not using such an
alias). Hiding that it is a pointer might discourage accessing the exported
fields because it looks like an interface.
Output gets fixed and improved such that:
- "FATAL ERROR" and "ERROR" are at the start of the line, followed by the klog header.
- The failure message follows in the next line.
- Continuation lines are always indented.
The set of methods exposed via TB is now a bit more complete (Attr, Chdir).
All former stand-alone With* functions are now also available as methods and
should be used instead of the functions. Those will be removed.
Linting of log calls now works and found some issues.
* First version of batching w/out signatures.
* First version of pod signatures.
* Integrate batching with signatures.
* Fix merge conflicts.
* Fixes from self-review.
* Test fixes.
* Fix a bug that limited batches to size 2
Also add some new high-level logging and
simplify the pod affinity signature.
* Re-enable batching on perf tests for now.
* fwk.NewStatus(fwk.Success)
* Review feedback.
* Review feedback.
* Comment fix.
* Two plugin specific unit tests.:
* Add cycle state to the sign call, apply to topo spread.
Also add unit tests for several plugi signature
calls.
* Review feedback.
* Switch to distinct stats for hint and store calls.
* Switch signature from string to []byte
* Revert cyclestate in signs. Update node affinity.
Node affinity now sorts all of the various
nested arrays in the structure. CycleState no
longer in signature; revert to signing fewer
cases for pod spread.
* hack/update-vendor.sh
* Disable signatures when extenders are configured.
* Update pkg/scheduler/framework/runtime/batch.go
Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com>
* Update staging/src/k8s.io/kube-scheduler/framework/interface.go
Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com>
* Review feedback.
* Disable node resource signatures when extended DRA enabled.
* Review feedback.
* Update pkg/scheduler/framework/plugins/imagelocality/image_locality.go
Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com>
* Update pkg/scheduler/framework/interface.go
Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com>
* Update pkg/scheduler/framework/plugins/nodedeclaredfeatures/nodedeclaredfeatures.go
Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com>
* Update pkg/scheduler/framework/runtime/batch.go
Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com>
* Review feedback.
* Fixes for review suggestions.
* Add integration tests.
* Linter fixes, test fix.
* Whitespace fix.
* Remove broken test.
* Unschedulable test.
* Remove go.mod changes.
---------
Co-authored-by: Maciej Skoczeń <87243939+macsko@users.noreply.github.com>
Extend Fit and BalancedAllocation PreScore state with the the
allocated state, the list of ResourceSlices and the device class
mapping. Gather these once during PreScore and pass them through
the scoring path instead of re-fetching for every scoring call.
This should speed up scoring of DRA extended resources, lowering
scheduling overhead.
Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
Co-authored-by: Maciej Skoczeń <mskoczen@google.com>
Co-authored-by: Dominik Marciński <gmidon@gmail.com>
BenchmarkPerfSchedulingExperimental_benchmark_dra_experimental_2025-11-05T07:10:33Z.json
is ignored by perf-dash. It probably needs to be called
BenchmarkPerfScheduling_benchmark_*.
Support for DeviceTaintRules depends on a significant amount of
additional code:
- ResourceSlice tracker is a NOP without it.
- Additional informers and corresponding permissions in scheduler and controller.
- Controller code for handling status.
Not all users necessarily need DeviceTaintRules, so adding a second feature
gate for that code makes it possible to limit the blast radius of bugs in that
code without having to turn off device taints and tolerations entirely.
When adding new features to the "experimental" allocator we are not directly
breaking "normal" usage of DRA because that uses the "incubating"
allocator. But we still want to know about potential performance problems also
for those test cases. If there is one, we might not be able to implement an
experimental feature in a way that allows promoting it later.
This is achieved by running the benchmarks twice as sub-tests, with different
topic names. The topic names then can be used to select different thresholds,
using a new extension of the scheduler_pref JSON config format.
For some packages (in particular the code DRA), all allocator implementations
can handle the testcases. Some other packages are for less stable features and
work with fewer implementations. Now all unit tests are run with all suitable
implementations, to increase code coverage.
Benchmarks are fixed to the most mature implementation because they would be
costly to run in more than one. When promoting an allocation implementation we
can do before/after comparisons to detect potential performance regressions.
The downside of this approach is that we need to remember to extend the list
of supported implementations when promoting features, otherwise testing will miss
some new supported implementation.
Without this, the effect of the following feature gate config would be random:
featureGates:
AllBeta: false
SomeBetaFeature: true
That's random because the order of iterating of the map is randomized by Go and
`AllBeta: false` would disable `SomeBetaFeature` if (and only if) applied last.
Now by sorting alphabetically, AllAlpha/Beta come first in practice. It's not a
complete solution, some future feature gate name might come before it.
The startup phase may have allocated memory that can be garbage-collected.
Forcing GC to run before measurements avoids noise if the garbage collection
kicks in during the measurement and potentially reduces the heap size reported
by metrics.
The exact effect has not been measured, it just seems useful.
When looking at a CPU profile, the cache mutation detection stood out. "make
test-integration" enables it by default. We try to benchmark "real" production
setups, therefore we have to prevent that by setting it to false ourselves.
The steady-state pod scheduling is less suitable for integration tests because
the duration is either short (making the test potentially flaky if nothing gets
scheduled yet due to the time constraint) or long (making the test run too
long). It is more useful for benchmark testcase because of the bounded runtime.
Now a single workload definition can be used in both modes with a configuration
parameter for "steadyState".
Workload definitions get updated accordingly. While at it, their names get
simplified and some (in the case of the main DRA config) redundant testcases
get removed.
Some of the DRA testcases schedule pods in a steady state for a certain
duration. They pass even if no pods got scheduled at all because in contrast to
the non-steady-state variants they don't wait for fixed number of pods to be
scheduled. This made them unsuitable for integration testing because a real
problem is not flagged as test failure. Now "zero pods scheduled" is detected
for them.
However, they are still not good integration tests (either run quickly and then
risk being flaky or run for a longer time period and then are slow). Revisiting
how they are used in configurations will be done separately.
The time required for pulling ResourceSlices into the scheduler is relevant in
two cases:
- The scheduler was (re)started and waits for informers to sync.
- A driver got deployed and needs to inform the scheduler about its devices.
The new workload measures the second scenario. It's indirectly relevant for
the first one because it allows drawing conclusion about the code which is also
involved in the first one.
After creating ResourceSlices, the workload was allowed to proceed even while
the scheduler was still busy receiving those new ResourceSlices. This blurred
the line between "setup" and "measurement" phase of DRA workloads. It's not
immediately clear how much that affected results, but it is cleaner to block.
This is done by returning the scheduler instance to the main scheduler_perf
loop and then pass the SharedDRAManager into the driver setup operation. There
it can be used to poll until that manager has processed all ResourceSlices.
Before, metrics gathered by testing.B (runtime_seconds,
-benchmem's B/op and allocs/op) covered the entire test case, including
starting the apiserver and the initialization steps of a workload. Now those
metrics are also limited to the period where the workload is configured to
collect metrics.
A mutex lock around decrementing runningPods was missing, leading to a data
race report in ci-kubernetes-integration-race-master:
WARNING: DATA RACE
Read at 0x00c001bd20a8 by goroutine 95696:
k8s.io/kubernetes/test/integration/scheduler_perf.createPodsSteadily.func7()
/home/prow/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:2133 +0x238
...
Previous write at 0x00c001bd20a8 by goroutine 101407:
k8s.io/kubernetes/test/integration/scheduler_perf.createPodsSteadily.func5()
/home/prow/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:2064 +0x1a4
...
Explicitly shutting down background flushing addresses the following data race:
WARNING: DATA RACE
Write at 0x0000091173b8 by goroutine 166:
k8s.io/component-base/logs/api/v1.apply.FlushLogger.func2()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/contextual.go:95 +0x35
k8s.io/klog/v2.SetLoggerWithOptions()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/contextual.go:75 +0xb4
k8s.io/component-base/logs/api/v1.apply()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:277 +0x10fb
k8s.io/component-base/logs/api/v1.validateAndApply()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:132 +0x90
k8s.io/component-base/logs/api/v1.ValidateAndApply()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:94 +0xd4
k8s.io/component-base/logs/api/v1.ResetForTest()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:314 +0xd5
k8s.io/kubernetes/test/integration/scheduler_perf.setupTestCase.func1()
/nvme/gopath/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:1062 +0x95
testing.(*common).Cleanup.func1()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1211 +0x16f
testing.(*common).runCleanup()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1445 +0x2b3
testing.(*common).runCleanup()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1445 +0x2b3
testing.tRunner.func2()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1786 +0x4c
runtime.deferreturn()
/nvme/gopath/go-1.24.0/src/runtime/panic.go:605 +0x5d
k8s.io/apiserver/pkg/endpoints.(*APIInstaller).Install()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go:208 +0x3ad
...
Previous read at 0x0000091173b8 by goroutine 169:
k8s.io/klog/v2.(*loggingT).flushAll()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1215 +0x166
k8s.io/klog/v2.(*loggingT).lockAndFlushAll()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1193 +0x58
k8s.io/klog/v2.(*loggingT).lockAndFlushAll-fm()
<autogenerated>:1 +0x33
k8s.io/klog/v2.(*flushDaemon).run.func1()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1143 +0x179
That the cleanup code is shown as being called by runtime/panic.go and
k8s.io/apiserver/pkg/endpoints/installer.go seems to be an oddity of `go test
-race`. Under a debugger it gets called normally, without a panic, and the test
finishes normally either way.
It's debatable whether StopFlushDaemon should be called by the caller of
ResetForTest or by ResetForTest itself. It's explicitly called out as "not
thread safe", so making the caller responsible seems appropriate.
