Merge pull request #62243 from resouer/fix-62068

Automatic merge from submit-queue (batch tested with PRs 59592, 62308, 62523, 62635, 62243). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Separate pod priority from preemption

**What this PR does / why we need it**:
Users request to split priority and preemption feature gate so they can use priority separately.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #62068 

**Special notes for your reviewer**:

~~I kept use `ENABLE_POD_PRIORITY` as ENV name for gce cluster scripts for backward compatibility reason. Please let me know if other approach is preffered.~~

~~This is a potential **break change** as existing clusters will be affected, we may need to include this in 1.11 maybe?~~

TODO: update this doc https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

[Update] Usage: in config file for scheduler:
```yaml
apiVersion: componentconfig/v1alpha1
kind: KubeSchedulerConfiguration
...
disablePreemption: true
```

**Release note**:

```release-note
Split PodPriority and PodPreemption feature gate
```

cmd/kube-scheduler/app/server.go

@@ -419,6 +419,8 @@ type SchedulerServer struct {
 	HealthzServer *http.Server
 	// MetricsServer is optional.
 	MetricsServer *http.Server
+	// Disable pod preemption or not.
+	DisablePreemption bool
 }
 
 // NewSchedulerServer creates a runnable SchedulerServer from configuration.
@@ -483,6 +485,7 @@ func NewSchedulerServer(config *componentconfig.KubeSchedulerConfiguration, mast
 		LeaderElection:                 leaderElectionConfig,
 		HealthzServer:                  healthzServer,
 		MetricsServer:                  metricsServer,
+		DisablePreemption:              config.DisablePreemption,
 	}, nil
 }
 
@@ -697,6 +700,7 @@ func (s *SchedulerServer) SchedulerConfig() (*scheduler.Config, error) {
 		storageClassInformer,
 		s.HardPodAffinitySymmetricWeight,
 		utilfeature.DefaultFeatureGate.Enabled(features.EnableEquivalenceClassCache),
+		s.DisablePreemption,
 	)
 
 	source := s.AlgorithmSource
@@ -754,5 +758,7 @@ func (s *SchedulerServer) SchedulerConfig() (*scheduler.Config, error) {
 	}
 	// Additional tweaks to the config produced by the configurator.
 	config.Recorder = s.Recorder
+
+	config.DisablePreemption = s.DisablePreemption
 	return config, nil
 }