Merge pull request #61445 from resouer/extender-priority

Automatic merge from submit-queue (batch tested with PRs 62063, 62169, 62155, 62139, 61445). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Schedule even if extender is not available when using extender 

**What this PR does / why we need it**:
When using scheduler extender, if the extender is not available scheduling of all pods fail.
We should let the scheduling happen but display error message that extender is failing.

`IsIgnorable()`  is added to extender to indicate: if scheduling of all pods should fail when it's unavailable

**Backward compabtiility:**

We use `IsIgnorable` instead of `IsCritical` so that when this flag is not set, the default value will be `false`, i.e. not ignorable, which consistent with the current behavior in existing extenders.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes: #60616

**Special notes for your reviewer**:
kindly cc @ravisantoshgudimetla to see if this meets your expectation

TODO: update the examples in kubernetes/examples, but the strategy there is not clear to me for now

**Release note**:

```release-note
Schedule even if extender is not available when using extender 
```

pkg/scheduler/algorithm/scheduler_interface.go

@@ -64,6 +64,10 @@ type SchedulerExtender interface {
 
 	// SupportsPreemption returns if the scheduler extender support preemption or not.
 	SupportsPreemption() bool
+
+	// IsIgnorable returns true indicates scheduling should not fail when this extender
+	// is unavailable. This gives scheduler ability to fail fast and tolerate non-critical extenders as well.
+	IsIgnorable() bool
 }
 
 // ScheduleAlgorithm is an interface implemented by things that know how to schedule pods