From 05dcf743acb276633f8b511c87f8e4c4e43428af Mon Sep 17 00:00:00 2001 From: David Oppenheimer Date: Sat, 5 Dec 2015 14:10:25 -0800 Subject: [PATCH] MetadataPolicy design doc. --- docs/design/metadata-policy.md | 160 +++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 docs/design/metadata-policy.md diff --git a/docs/design/metadata-policy.md b/docs/design/metadata-policy.md new file mode 100644 index 00000000000..1d02fcf4215 --- /dev/null +++ b/docs/design/metadata-policy.md @@ -0,0 +1,160 @@ + + + + +WARNING +WARNING +WARNING +WARNING +WARNING + +

PLEASE NOTE: This document applies to the HEAD of the source tree

+ +If you are using a released version of Kubernetes, you should +refer to the docs that go with that version. + +Documentation for other releases can be found at +[releases.k8s.io](http://releases.k8s.io). + +-- + + + + + +# MetadataPolicy and its use in choosing the scheduler in a multi-scheduler system + +## Introduction + +This document describes a new API resource, `MetadataPolicy`, that configures an +admission controller to take one or more actions based on an object's metadata. +Initially the metadata fields that the predicates can examine are labels and annotations, +and the actions are to add one or more labels and/or annotations, or to reject creation/update +of the object. In the future other actions might be supported, such as applying an initializer. + +The first use of `MetadataPolicy` will be to decide which scheduler should schedule a pod +in a [multi-scheduler](../proposals/multiple-schedulers.md) Kubernetes system. In particular, the +policy will add the scheduler name annotation to a pod based on an annotation that +is already on the pod that indicates the QoS of the pod. +(That annotation was presumably set by a simpler admission controller that +uses code, rather than configuration, to map the resource requests and limits of a pod +to QoS, and attaches the corresponding annotation.) + +We anticipate a number of other uses for `MetadataPolicy`, such as defaulting for +labels and annotations, prohibiting/requiring particular labels or annotations, or +choosing a scheduling policy within a scheduler. We do not discuss them in this doc. + + +## API + +```go +// MetadataPolicySpec defines the configuration of the MetadataPolicy API resource. +// Every rule is applied, in an unspecified order, but if the action for any rule +// that matches is to reject the object, then the object is rejected without being mutated. +type MetadataPolicySpec struct { + Rules []MetadataPolicyRule `json:"rules,omitempty"` +} + +// If the PolicyPredicate is met, then the PolicyAction is applied. +// Example rules: +// reject object if label with key X is present (i.e. require X) +// reject object if label with key X is not present (i.e. forbid X) +// add label X=Y if label with key X is not present (i.e. default X) +// add annotation A=B if object has annotation C=D or E=F +type MetadataPolicyRule struct { + PolicyPredicate PolicyPredicate `json:"policyPredicate"` + PolicyAction PolicyAction `json:policyAction"` +} + +// All criteria must be met for the PolicyPredicate to be considered met. +type PolicyPredicate struct { + // Note that Namespace is not listed here because MetadataPolicy is per-Namespace. + LabelSelector *LabelSelector `json:"labelSelector,omitempty"` + AnnotationSelector *LabelSelector `json:"annotationSelector,omitempty"` +} + +// Apply the indicated Labels and/or Annotations (if present), unless Reject is set +// to true, in which case reject the object without mutating it. +type PolicyAction struct { + // If true, the object will be rejected and not mutated. + Reject bool `json:"reject"` + // The labels to add or update, if any. + UpdatedLabels *map[string]string `json:"updatedLabels,omitempty"` + // The annotations to add or update, if any. + UpdatedAnnotations *map[string]string `json:"updatedAnnotations,omitempty"` +} + +// MetadataPolicy describes the MetadataPolicy API resource, which is used for specifying +// policies that should be applied to objects based on the objects' metadata. All MetadataPolicy's +// are applied to all objects in the namespace; the order of evaluation is not guaranteed, +// but if any of the matching policies have an action of rejecting the object, then the object +// will be rejected without being mutated. +type MetadataPolicy struct { + unversioned.TypeMeta `json:",inline"` + // Standard object's metadata. + // More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata + ObjectMeta `json:"metadata,omitempty"` + + // Spec defines the metadata policy that should be enforced. + // http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#spec-and-status + Spec MetadataPolicySpec `json:"spec,omitempty"` +} + +// MetadataPolicyList is a list of MetadataPolicy items. +type MetadataPolicyList struct { + unversioned.TypeMeta `json:",inline"` + // Standard list metadata. + // More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#types-kinds + unversioned.ListMeta `json:"metadata,omitempty"` + + // Items is a list of MetadataPolicy objects. + // More info: http://releases.k8s.io/HEAD/docs/design/admission_control_resource_quota.md#admissioncontrol-plugin-resourcequota + Items []MetadataPolicy `json:"items"` +} +``` + +## Implementation plan + +1. Create `MetadataPolicy` API resource +1. Create admission controller that implements policies defined in `MetadataPolicy` +1. Create admission controller that sets annotation +`scheduler.alpha.kubernetes.io/qos: ` +(where `QOS` is one of `Guaranteed, Burstable, BestEffort`) +based on pod's resource request and limit. + +## Future work + +Longer-term we will have QoS be set on create and update by the registry, similar to `Pending` phase today, +instead of having an admission controller (that runs before the one that takes `MetadataPolicy` as input) +do it. + +We plan to eventually move from having an admission controller +set the scheduler name as a pod annotation, to using the initializer concept. In particular, the +scheduler will be an initializer, and the admission controller that decides which scheduler to use +will add the scheduler's name to the list of initializers for the pod (presumably the scheduler +will be the last initializer to run on each pod). +The admission controller would still be configured using the `MetadataPolicy` described here, only the +mechanism the admission controller uses to record its decision of which scheduler to use would change. + +## Related issues + +The main issue for multiple schedulers is #11793. There was also a lot of discussion +in PRs #17197 and #17865. + +We could use the approach described here to choose a scheduling +policy within a single scheduler, as opposed to choosing a scheduler, a desire mentioned in #9920. +Issue #17097 describes a scenario unrelated to scheduler-choosing where `MetadataPolicy` could be used. +Issue #17324 proposes to create a generalized API for matching +"claims" to "service classes"; matching a pod to a scheduler would be one use for such an API. + + + + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/metadata-policy.md?pixel)]() +