Any error result from PreBind was treated as a pod scheduling failure. This was
overlooked when moving blocking API calls in the DRA plugin into a PreBind
implementation, leading to:
E0604 15:45:50.980929 306340 schedule_one.go:1048] "Error scheduling pod; retrying" err="waiting for resource driver" pod="test/test-draqld28"
That's because DRA's PreBind does some updates in the apiserver, then returns
Pending to wait for the outcome.
The fix is to allow PreBind to return the same special status codes as other
extension points.
* Don't evaluate extra nodes if there's no score plugin defined
* Fix existing unit test (add no op scoring plugin)
* Add unit tests for no score plugin scenario
* address review comments
* add a test with non-filter, non-scoring extender
* Deleted from the cache in the handling of scheduling failures due to missing Node
Signed-off-by: utam0k <k0ma@utam0k.jp>
* Support only `nodes`
* Remove unnecessary error catch
Signed-off-by: utam0k <k0ma@utam0k.jp>
* Fix a build error
Signed-off-by: utam0k <k0ma@utam0k.jp>
* Fix a build error
Signed-off-by: utam0k <k0ma@utam0k.jp>
---------
Signed-off-by: utam0k <k0ma@utam0k.jp>
The logging instrumentation for contextual logging that was added for 1.29
slowed down the scheduler (i.e. logging verbosity <= 3) by a significant
percentage (-28.66% for SchedulingBasic/5000Nodes at -v3) if (and only if!)
contextual logging was enabled.
Retrieving the logger from the context causes no measurable slowdown, it's only
the various WithName/WithValues calls which cause this.
By being more careful about when to use those, the performance impact can be
avoided:
- At -v3 or lower, only `WithValues("pod")` is used once per scheduling cycle.
This has the intended effect that all log messages for the cycle include the
pod information. Once contextual logging is GA, "pod" key/value pairs can
be removed from all log calls.
- At -v4 or higher, richer log entries get produced where `WithValues` is also
used for the node (when applicable) and `WithName` is used for the current
operation and plugin.
With these changes, enabling contextual logging causes no measurable slowdown
at -v3 or lower. At -v4, the slowdown depends on the test case (-30.51%
throughput for SchedulingBasic/5000Nodes, no change for
SchedulingCSIPVs/5000Nodes). For some unknown reason (measuring bias?),
SchedulingCSIPVs/500Nodes has a ~3& *higher* throughput with contextual
logging.
When binding has failed, `Done` gets called by
`handleBindingCycleError`. Calling it again is at best redundant and worse,
suffers from a data race:
- the `assumedPodInfo` is placed in the backoff queue
- an event causes the `Pod` pointer to get updated in it
- reading `assumedPodInfo.Pod.UID` races with that write
This race was found with`go test -race`.
* feature(sscheduling_queue): track events per Pods
* fix typos
* record events in one slice and make each in-flight Pod to refer it
* fix: use Pop() in test before AddUnschedulableIfNotPresent to register in-flight Pods
* eliminate MakeNextPodFuncs
* call Done inside the scheduling queue
* fix comment
* implement done() not to require lock in it
* fix UTs
* improve the receivedEvents implementation based on suggestions
* call DonePod when we don't call AddUnschedulableIfNotPresent
* fix UT
* use queuehint to filter out events for in-flight Pods
* fix based on suggestion from aldo
* fix based on suggestion from Wei
* rename lastEventBefore → previousEvent
* fix based on suggestion
* address comments from aldo
* fix based on the suggestion from Abdullah
* gate in-flight Pods logic by the SchedulingQueueHints feature gate
We only added failed plulgins, but actually this will not work unless
we make the status with a fitError because we only copy the failured plugins
to podInfo if it is a fitError
Signed-off-by: kerthcet <kerthcet@gmail.com>
