From cea5cf42b68205c28bf164cdfb40e0fa6d9d8adc Mon Sep 17 00:00:00 2001 From: David Oppenheimer Date: Fri, 4 Dec 2015 23:59:50 -0800 Subject: [PATCH] Inter-pod topological affinity/anti-affinity design doc. --- docs/design/podaffinity.md | 615 +++++++++++++++++++++++++++++++++++++ 1 file changed, 615 insertions(+) create mode 100644 docs/design/podaffinity.md diff --git a/docs/design/podaffinity.md b/docs/design/podaffinity.md new file mode 100644 index 00000000000..108697c6960 --- /dev/null +++ b/docs/design/podaffinity.md @@ -0,0 +1,615 @@ + + + + +WARNING +WARNING +WARNING +WARNING +WARNING + +

PLEASE NOTE: This document applies to the HEAD of the source tree

+ +If you are using a released version of Kubernetes, you should +refer to the docs that go with that version. + +Documentation for other releases can be found at +[releases.k8s.io](http://releases.k8s.io). + +-- + + + + + +# Inter-pod topological affinity and anti-affinity + +## Introduction + +NOTE: It is useful to read about [node affinity](https://github.com/kubernetes/kubernetes/pull/18261) first. + +This document describes a proposal for specifying and implementing inter-pod topological affinity and +anti-affinity. By that we mean: rules that specify that certain pods should be placed +in the same topological domain (e.g. same node, same rack, same zone, same +power domain, etc.) as some other pods, or, conversely, should *not* be placed in the +same topological domain as some other pods. + +Here are a few example rules; we explain how to express them using the API described +in this doc later, in the section "Examples." +* Affinity + * Co-locate the pods from a particular service or Job in the same availability zone, + without specifying which zone that should be. + * Co-locate the pods from service S1 with pods from service S2 because S1 uses S2 + and thus it is useful to minimize the network latency between them. Co-location + might mean same nodes and/or same availability zone. +* Anti-affinity + * Spread the pods of a service across nodes and/or availability zones, + e.g. to reduce correlated failures + * Give a pod "exclusive" access to a node to guarantee resource isolation -- it must never share the node with other pods + * Don't schedule the pods of a particular service on the same nodes as pods of + another service that are known to interfere with the performance of the pods of the first service. + +For both affinity and anti-affinity, there are three variants. Two variants have the +property of requiring the affinity/anti-affinity to be satisfied for the pod to be allowed +to schedule onto a node; the difference between them is that if the condition ceases to +be met later on at runtime, for one of them the system will try to eventually evict the pod, +while for the other the system may not try to do so. The third variant +simply provides scheduling-time *hints* that the scheduler will try +to satisfy but may not be able to. These three variants are directly analogous to the three +variants of [node affinity](https://github.com/kubernetes/kubernetes/pull/18261). + +Note that this proposal is only about *inter-pod* topological affinity and anti-affinity. +There are other forms of topological affinity and anti-affinity. For example, +you can use [node affinity](https://github.com/kubernetes/kubernetes/pull/18261) to require (prefer) +that a set of pods all be scheduled in some specific zone Z. Node affinity is not +capable of expressing inter-pod dependencies, and conversely the API +we descibe in this document is not capable of expressing node affinity rules. +For simplicity, we will use the terms "affinity" and "anti-affinity" to mean +"inter-pod topological affinity" and "inter-pod topological anti-affinity," respectively, +in the remainder of this document. + +## API + +We will add one field to `PodSpec` + +```go +Affinity *Affinity `json:"affinity,omitempty"` +``` + +The `Affinity` type is defined as follows + +```go +type Affinity struct { + PodAffinity *PodAffinity `json:"podAffinity,omitempty"` + PodAntiAffinity *PodAntiAffinity `json:"podAntiAffinity,omitempty"` +} + +type PodAffinity struct { + // If the affinity requirements specified by this field are not met at + // scheduling time, the pod will not be scheduled onto the node. + // If the affinity requirements specified by this field cease to be met + // at some point during pod execution (e.g. due to a pod label update), the + // system will try to eventually evict the pod from its node. + // When there are multiple elements, the lists of nodes corresponding to each + // PodAffinityTerm are intersected, i.e. all terms must be satisfied. + RequiredDuringSchedulingRequiredDuringExecution []PodAffinityTerm `json:"requiredDuringSchedulingRequiredDuringExecution,omitempty"` + // If the affinity requirements specified by this field are not met at + // scheduling time, the pod will not be scheduled onto the node. + // If the affinity requirements specified by this field cease to be met + // at some point during pod execution (e.g. due to a pod label update), the + // system may or may not try to eventually evict the pod from its node. + // When there are multiple elements, the lists of nodes corresponding to each + // PodAffinityTerm are intersected, i.e. all terms must be satisfied. + RequiredDuringSchedulingIgnoredDuringExecution []PodAffinityTerm `json:"requiredDuringSchedulingIgnoredDuringExecution,omitempty"` + // The scheduler will prefer to schedule pods to nodes that satisfy + // the affinity expressions specified by this field, but it may choose + // a node that violates one or more of the expressions. The node that is + // most preferred is the one with the greatest sum of weights, i.e. + // for each node that meets all of the scheduling requirements (resource + // request, RequiredDuringScheduling affinity expressions, etc.), + // compute a sum by iterating through the elements of this field and adding + // "weight" to the sum if the node matches the corresponding MatchExpressions; the + // node(s) with the highest sum are the most preferred. + PreferredDuringSchedulingIgnoredDuringExecution []WeightedPodAffinityTerm `json:"preferredDuringSchedulingIgnoredDuringExecution,omitempty"` +} + +type PodAntiAffinity struct { + // If the anti-affinity requirements specified by this field are not met at + // scheduling time, the pod will not be scheduled onto the node. + // If the anti-affinity requirements specified by this field cease to be met + // at some point during pod execution (e.g. due to a pod label update), the + // system will try to eventually evict the pod from its node. + // When there are multiple elements, the lists of nodes corresponding to each + // PodAffinityTerm are intersected, i.e. all terms must be satisfied. + RequiredDuringSchedulingRequiredDuringExecution []PodAffinityTerm `json:"requiredDuringSchedulingRequiredDuringExecution,omitempty"` + // If the anti-affinity requirements specified by this field are not met at + // scheduling time, the pod will not be scheduled onto the node. + // If the anti-affinity requirements specified by this field cease to be met + // at some point during pod execution (e.g. due to a pod label update), the + // system may or may not try to eventually evict the pod from its node. + // When there are multiple elements, the lists of nodes corresponding to each + // PodAffinityTerm are intersected, i.e. all terms must be satisfied. + RequiredDuringSchedulingIgnoredDuringExecution []PodAffinityTerm `json:"requiredDuringSchedulingIgnoredDuringExecution,omitempty"` + // The scheduler will prefer to schedule pods to nodes that satisfy + // the anti-affinity expressions specified by this field, but it may choose + // a node that violates one or more of the expressions. The node that is + // most preferred is the one with the greatest sum of weights, i.e. + // for each node that meets all of the scheduling requirements (resource + // request, RequiredDuringScheduling anti-affinity expressions, etc.), + // compute a sum by iterating through the elements of this field and adding + // "weight" to the sum if the node matches the corresponding MatchExpressions; the + // node(s) with the highest sum are the most preferred. + PreferredDuringSchedulingIgnoredDuringExecution []WeightedPodAffinityTerm `json:"preferredDuringSchedulingIgnoredDuringExecution,omitempty"` +} + +type WeightedPodAffinityTerm struct { + // weight is in the range 1-100 + Weight int `json:"weight"` + PodAffinityTerm PodAffinityTerm `json:"podAffinityTerm"` +} + +type PodAffinityTerm struct { + LabelSelector *LabelSelector `json:"labelSelector,omitempty"` + // namespaces specifies which namespaces the LabelSelector applies to (matches against); + // nil list means "this pod's namespace," empty list means "all namespaces" + // The json tag here is not "omitempty" since we need to distinguish nil and empty. + // See https://golang.org/pkg/encoding/json/#Marshal for more details. + Namespaces []api.Namespace `json:"namespaces"` + // empty topology key is interpreted by the scheduler as "all topologies" + TopologyKey string `json:"topologyKey,omitempty"` +} +``` + +Note that the `Namespaces` field is necessary because normal `LabelSelector` is scoped +to the pod's namespace, but we need to be able to match against all pods globally. + +To explain how this API works, let's say that the `PodSpec` of a pod `P` has an `Affinity` +that is configured as follows (note that we've omitted and collapsed some fields for +simplicity, but this should sufficiently convey the intent of the design): + +```go +PodAffinity { + RequiredDuringScheduling: {{LabelSelector: P1, TopologyKey: "node"}}, + PreferredDuringScheduling: {{LabelSelector: P2, TopologyKey: "zone"}}, +} +PodAntiAffinity { + RequiredDuringScheduling: {{LabelSelector: P3, TopologyKey: "rack"}}, + PreferredDuringScheduling: {{LabelSelector: P4, TopologyKey: "power"}} +} +``` + +Then when scheduling pod P, the scheduler +* Can only schedule P onto nodes that are running pods that satisfy `P1`. (Assumes all nodes have a label with key "node" and value specifying their node name.) +* Should try to schedule P onto zones that are running pods that satisfy `P3`. (Assumes all nodes have a label with key "zone" and value specifying their zone.) +* Cannot schedule P onto any racks that are running pods that satisfy `P2`. (Assumes all nodes have a label with key "rack" and value specifying their rack name.) +* Should try not to schedule P onto any power domains that are running pods that satisfy `P4`. (Assumes all nodes have a label with key "power" and value specifying their power domain.) + +When `RequiredDuringScheduling` has multiple elements, the requirements are ANDed. +For `PreferredDuringScheduling` the weights are added for the terms that are satisfied for each node, and +the node(s) with the highest weight(s) are the most preferred. + +In reality there are two variants of `RequiredDuringScheduling`: one suffixed with +`RequiredDuringEecution` and one suffixed with `IgnoredDuringExecution`. For the +first variant, if the affinity/anti-affinity ceases to be met at some point during +pod execution (e.g. due to a pod label update), the system will try to eventually evict the pod +from its node. In the second variant, the system may or may not try to eventually +evict the pod from its node. + +## A comment on symmetry + +One thing that makes affinity and anti-affinity tricky is symmetry. + +Imagine a cluster that is running pods from two services, S1 and S2. Imagine that the pods of S1 have a RequiredDuringScheduling anti-affinity rule +"do not run me on nodes that are running pods from S2." It is not sufficient just to check that there are no S2 pods on a node when +you are scheduling a S1 pod. You also need to ensure that there are no S1 pods on a node when you are scheduling a S2 pod, +*even though the S2 pod does not have any anti-affinity rules*. Otherwise if an S1 pod schedules before an S2 pod, the S1 +pod's RequiredDuringScheduling anti-affinity rule can be violated by a later-arriving S2 pod. More specifically, if S1 has the aforementioned +RequiredDuringScheduling anti-affinity rule, then +* if a node is empty, you can schedule S1 or S2 onto the node +* if a node is running S1 (S2), you cannot schedule S2 (S1) onto the node + +Note that while RequiredDuringScheduling anti-affinity is symmetric, +RequiredDuringScheduling affinity is *not* symmetric. That is, if the pods of S1 have a RequiredDuringScheduling affinity rule "run me on nodes that are running +pods from S2," it is not required that there be S1 pods on a node in order to schedule a S2 pod onto that node. More +specifically, if S1 has the aforementioned RequiredDuringScheduling affinity rule, then +* if a node is empty, you can schedule S2 onto the node +* if a node is empty, you cannot schedule S1 onto the node +* if a node is running S2, you can schedule S1 onto the node +* if a node is running S1+S2 and S1 terminates, S2 continues running +* if a node is running S1+S2 and S2 terminates, the system terminates S1 (eventually) + +However, although RequiredDuringScheduling affinity is not symmetric, there is an implicit PreferredDuringScheduling affinity rule corresponding to every +RequiredDuringScheduling affinity rule: if the pods of S1 have a RequiredDuringScheduling affinity rule "run me on nodes that are running +pods from S2" then it is not required that there be S1 pods on a node in order to schedule a S2 pod onto that node, +but it would be better if there are. + +PreferredDuringScheduling is symmetric. +If the pods of S1 had a PreferredDuringScheduling anti-affinity rule "try not to run me on nodes that are running pods from S2" +then we would prefer to keep a S1 pod that we are scheduling off of nodes that are running S2 pods, and also +to keep a S2 pod that we are scheduling off of nodes that are running S1 pods. Likewise if the pods of +S1 had a PreferredDuringScheduling affinity rule "try to run me on nodes that are running pods from S2" then we would prefer +to place a S1 pod that we are scheduling onto a node that is running a S2 pod, and also to place +a S2 pod that we are scheduling onto a node that is running a S1 pod. + +## Examples + +Here are some examples of how you would express various affinity and anti-affinity rules using the API we described. + +### Affinity + +In the examples below, the word "put" is intentionally ambiguous; the rules are the same +whether "put" means "must put" (RequiredDuringScheduling) or "try to put" +(PreferredDuringScheduling)--all that changes is which field the rule goes into. +Also, we only discuss scheduling-time, and ignore the execution-time. +Finally, some of the examples +use "zone" and some use "node," just to make the examples more interesting; any of the examples +with "zone" will also work for "node" if you change the `TopologyKey`, and vice-versa. + +* **Put the pod in zone Z**: +Tricked you! It is not possible express this using the API described here. For this you should use node affinity. + +* **Put the pod in a zone that is running at least one pod from service S**: +`{LabelSelector: , TopologyKey: "zone"}` + +* **Put the pod on a node that is already running a pod that requires a license for software package P**: +Assuming pods that require a license for software package P have a label `{key=license, value=P}`: +`{LabelSelector: "license" In "P", TopologyKey: "node"}` + +* **Put this pod in the same zone as other pods from its same service**: +Assuming pods from this pod's service have some label `{key=service, value=S}`: +`{LabelSelector: "service" In "S", TopologyKey: "zone"}` + +This last example illustrates a small issue with this API when it is used +with a scheduler that processes the pending queue one pod at a time, like the current +Kubernetes scheduler. The RequiredDuringScheduling rule +`{LabelSelector: "service" In "S", TopologyKey: "zone"}` +only "works" once one pod from service S has been scheduled. But if all pods in service +S have this RequiredDuringScheduling rule in their PodSpec, then the RequiredDuringScheduling rule +will block the first +pod of the service from ever scheduling, since it is only allowed to run in a zone with another pod from +the same service. And of course that means none of the pods of the service will be able +to schedule. This problem *only* applies to RequiredDuringScheduling affinity, not +PreferredDuringScheduling affinity or any variant of anti-affinity. +There are at least three ways to solve this problem +* **short-term**: have the scheduler use a rule that if the RequiredDuringScheduling affinity requirement +matches a pod's own labels, and there are no other such pods anywhere, then disregard the requirement. +This approach has a corner case when running parallel schedulers that are allowed to +schedule pods from the same replicated set (e.g. a single PodTemplate): both schedulers may try to +schedule pods from the set +at the same time and think there are no other pods from that set scheduled yet (e.g. they are +trying to schedule the first two pods from the set), but by the time +the second binding is committed, the first one has already been committed, leaving you with +two pods running that do not respect their RequiredDuringScheduling affinity. There is no +simple way to detect this "conflict" at scheduling time given the current system implementation. +* **longer-term**: when a controller creates pods from a PodTemplate, for exactly *one* of those +pods, it should omit any RequiredDuringScheduling affinity rules that select the pods of that PodTemplate. +* **very long-term/speculative**: controllers could present the scheduler with a group of pods from +the same PodTemplate as a single unit. This is similar to the first approach described above but +avoids the corner case. No special logic is needed in the controllers. Moreover, this would allow +the scheduler to do proper [gang scheduling](https://github.com/kubernetes/kubernetes/issues/16845) +since it could receive an entire gang simultaneously as a single unit. + +### Anti-affinity + +As with the affinity examples, the examples here can be RequiredDuringScheduling or +PreferredDuringScheduling anti-affinity, i.e. +"don't" can be interpreted as "must not" or as "try not to" depending on whether the rule appears +in `RequiredDuringScheduling` or `PreferredDuringScheduling`. + +* **Spread the pods of this service S across nodes and zones**: +`{{LabelSelector: , TopologyKey: "node"}, {LabelSelector: , TopologyKey: "zone"}}` +(note that if this is specified as a RequiredDuringScheduling anti-affinity, then the first clause is redundant, since the second +clause will force the scheduler to not put more than one pod from S in the same zone, and thus by +definition it will not put more than one pod from S on the same node, assuming each node is in one zone. +This rule is more useful as PreferredDuringScheduling anti-affinity, e.g. one might expect it to be common in +[Ubernetes](../../docs/proposals/federation.md) clusters.) + +* **Don't co-locate pods of this service with pods from service "evilService"**: +`{LabelSelector: selector that matches evilService's pods, TopologyKey: "node"}` + +* **Don't co-locate pods of this service with any other pods including pods of this service**: +`{LabelSelector: empty, TopologyKey: "node"}` + +* **Don't co-locate pods of this service with any other pods except other pods of this service**: +Assuming pods from the service have some label `{key=service, value=S}`: +`{LabelSelector: "service" NotIn "S", TopologyKey: "node"}` +Note that this works because `"service" NotIn "S"` matches pods with no key "service" +as well as pods with key "service" and a corresponding value that is not "S." + +## Algorithm + +An example algorithm a scheduler might use to implement affinity and anti-affinity rules is as follows. +There are certainly more efficient ways to do it; this is just intended to demonstrate that the API's +semantics are implementable. + +Terminology definition: We say a pod P is "feasible" on a node N if P meets all of the scheduler +predicates for scheduling P onto N. Note that this algorithm is only concerned about scheduling +time, thus it makes no distinction between RequiredDuringExecution and IgnoredDuringExecution. + +To make the algorithm slightly more readable, we use the term "HardPodAffinity" as shorthand +for "RequiredDuringSchedulingScheduling pod affinity" and "SoftPodAffinity" as shorthand for +"PreferredDuringScheduling pod affinity." Analogously for "HardPodAntiAffinity" and "SoftPodAntiAffinity." + +** TODO: Update this algorithm to take weight for SoftPod{Affinity,AntiAffinity} into account; +currently it assumes all terms have weight 1. ** + +``` +Z = the pod you are scheduling +{N} = the set of all nodes in the system // this algorithm will reduce it to the set of all nodes feasible for Z +// Step 1a: Reduce {N} to the set of nodes satisfying Z's HardPodAffinity in the "forward" direction +X = {Z's PodSpec's HardPodAffinity} +foreach element H of {X} + P = {all pods in the system that match H.LabelSelector} + M map[string]int // topology value -> number of pods running on nodes with that topology value + foreach pod Q of {P} + L = {labels of the node on which Q is running, represented as a map from label key to label value} + M[L[H.TopologyKey]]++ + {N} = {N} intersect {all nodes of N with label [key=H.TopologyKey, value=any K such that M[K]>0]} +// Step 1b: Further reduce {N} to the set of nodes also satisfying Z's HardPodAntiAffinity +// This step is identical to Step 1a except the M[K] > 0 comparison becomes M[K] == 0 +X = {Z's PodSpec's HardPodAntiAffinity} +foreach element H of {X} + P = {all pods in the system that match H.LabelSelector} + M map[string]int // topology value -> number of pods running on nodes with that topology value + foreach pod Q of {P} + L = {labels of the node on which Q is running, represented as a map from label key to label value} + M[L[H.TopologyKey]]++ + {N} = {N} intersect {all nodes of N with label [key=H.TopologyKey, value=any K such that M[K]==0]} +// Step 2: Further reduce {N} by enforcing symmetry requirement for other pods' HardPodAntiAffinity +foreach node A of {N} + foreach pod B that is bound to A + if any of B's HardPodAntiAffinity are currently satisfied but would be violated if Z runs on A, then remove A from {N} +// At this point, all node in {N} are feasible for Z. +// Step 3a: Soft version of Step 1a +Y map[string]int // node -> number of Z's soft affinity/anti-affinity preferences satisfied by that node +Initialize the keys of Y to all of the nodes in {N}, and the values to 0 +X = {Z's PodSpec's SoftPodAffinity} +Repeat Step 1a except replace the last line with "foreach node W of {N} having label [key=H.TopologyKey, value=any K such that M[K]>0], Y[W]++" +// Step 3b: Soft version of Step 1b +X = {Z's PodSpec's SoftPodAntiAffinity} +Repeat Step 1b except replace the last line with "foreach node W of {N} not having label [key=H.TopologyKey, value=any K such that M[K]>0], Y[W]++" +// Step 4: Symmetric soft, plus treat forward direction of hard affinity as a soft +foreach node A of {N} + foreach pod B that is bound to A + increment Y[A] by the number of B's SoftPodAffinity, SoftPodAntiAffinity, and HardPodAffinity that are satisfied if Z runs on A but are not satisfied if Z does not run on A +// We're done. {N} contains all of the nodes that satisfy the affinity/anti-affinity rules, and Y is +// a map whose keys are the elements of {N} and whose values are how "good" of a choice N is for Z with +// respect to the explicit and implicit affinity/anti-affinity rules (larger number is better). +``` + +## Special considerations for RequiredDuringScheduling anti-affinity + +In this section we discuss three issues with RequiredDuringScheduling anti-affinity: +Denial of Service (DoS), co-existing with daemons, and determining which pod(s) to kill. +See issue #18265 for additional discussion of these topics. + +### Denial of Service + +Without proper safeguards, a pod using RequiredDuringScheduling anti-affinity can intentionally +or unintentionally cause various problems for other pods, due to the symmetry property of anti-affinity. + +The most notable danger is the ability for a +pod that arrives first to some topology domain, to block all other pods from +scheduling there by stating a conflict with all other pods. +The standard approach +to preventing resource hogging is quota, but simple resource quota cannot prevent +this scenario because the pod may request very little resources. Addressing this +using quota requires a quota scheme that charges based on "opportunity cost" rather +than based simply on requested resources. For example, when handling a pod that expresses +RequiredDuringScheduling anti-affinity for all pods using a "node" `TopologyKey` +(i.e. exclusive access to a node), it could charge for the resources of the +average or largest node in the cluster. Likewise if a pod expresses RequiredDuringScheduling +anti-affinity for all pods using a "cluster" `TopologyKey`, it could charge for the resources of the +entire cluster. If a cluster administrator wants to overcommit quota, for +example to allow more than N pods across all users to request exclusive node +access in a cluster with N nodes, then a priority/preemption scheme should be added +so that the most important pods run when resource demand exceeds supply. + +Our initial implementation will use quota that charges based on opportunity cost. + +A weaker variant of the problem described in the previous paragraph is a pod's ability to use anti-affinity to degrade +the scheduling quality of another pod, but not completely block it from scheduling. +For example, a set of pods S1 could use node affinity to request to schedule onto a set +of nodes that some other set of pods S2 prefers to schedule onto. If the pods in S1 +have RequiredDuringScheduling or even PreferredDuringScheduling pod anti-affinity for S2, +then due to the symmetry property of anti-affinity, they can prevent the pods in S2 from +scheduling onto their preferred nodes if they arrive first (for sure in the RequiredDuringScheduling case, and +with some probability that depends on the weighting scheme for the PreferredDuringScheduling case). +A very sophisticated priority and/or quota scheme could mitigate this, or alternatively +we could eliminate the symmetry property of the implementation of PreferredDuringScheduling anti-affinity. +Then only RequiredDuringScheduling anti-affinity could affect scheduling quality +of another pod, and as we described in the previous paragraph, such pods could be charged +quota for the full topology domain, thereby reducing the potential for abuse. + +We won't try to address this issue in our initial implementation; we can consider one +of the approaches mentioned above if it turns out to be a problem in practice. + +### Co-existing with daemons + +A cluster administrator +may wish to allow pods that express anti-affinity against all pods, to nonetheless co-exist with +system daemon pods, such as those run by DaemonSet. In principle, we would like the specification +for RequiredDuringScheduling inter-pod anti-affinity to allow "toleration" of one or more +other pods (see #18263 for a more detailed explanation of the toleration concept). There are +at least two ways to accomplish this: + +* Scheduler special-cases the namespace(s) where daemons live, in the + sense that it ignores pods in those namespaces when it is + determining feasibility for pods with anti-affinity. The name(s) of + the special namespace(s) could be a scheduler configuration + parameter, and default to `kube-system`. We could allow + multiple namespaces to be specified if we want cluster admins to be + able to give their own daemons this special power (they would add + their namespace to the list in the scheduler configuration). And of + course this would be symmetric, so daemons could schedule onto a node + that is already running a pod with anti-affinity. + +* We could add an explicit "toleration" concept/field to allow the + user to specify namespaces that are excluded when they use + RequiredDuringScheduling anti-affinity, and use an admission + controller/defaulter to ensure these namespaces are always listed. + +Our initial implementation will use the first approach. + +### Determining which pod(s) to kill (for RequiredDuringSchedulingRequiredDuringExecution) + +Because anti-affinity is symmetric, in the case of RequiredDuringSchedulingRequiredDuringExecution +anti-affinity, the system must determine which pod(s) to kill when a pod's labels are updated in +such as way as to cause them to conflict with one or more other pods' RequiredDuringSchedulingRequiredDuringExecution +anti-affinity rules. In the absence of a priority/preemption scheme, our rule will be that the pod +with the anti-affinity rule that becomes violated should be the one killed. +A pod should only specify constraints that apply to +namespaces it trusts to not do malicious things. Once we have priority/preemption, we can +change the rule to say that the lowest-priority pod(s) are killed until all +RequiredDuringSchedulingRequiredDuringExecution anti-affinity is satisfied. + +## Special considerations for RequiredDuringScheduling affinity + +The DoS potential of RequiredDuringScheduling *anti-affinity* stemmed from its symmetry: +if a pod P requests anti-affinity, P cannot schedule onto a node with conflicting pods, +and pods that conflict with P cannot schedule onto the node one P has been scheduled there. +The design we have described says that the symmetry property for RequiredDuringScheduling *affinity* +is weaker: if a pod P says it can only schedule onto nodes running pod Q, this +does not mean Q can only run on a node that is running P, but the scheduler will try +to schedule Q onto a node that is running P (i.e. treats the reverse direction as +preferred). This raises the same scheduling quality concern as we menioned at the +end of the Denial of Service section above, and can be addressed in similar ways. + +The nature of affinity (as opposed to anti-affinity) means that there is no issue of +determining which pod(s) to kill +when a pod's labels change: it is obviously the pod with the affinity rule that becomes +violated that must be killed. (Killing a pod never "fixes" violation of an affinity rule; +it can only "fix" violation an anti-affinity rule.) However, affinity does have a +different question related to killing: how long should the system wait before declaring +that RequiredDuringSchedulingRequiredDuringExecution affinity is no longer met at runtime? +For example, if a pod P has such an affinity for a pod Q and pod Q is temporarily killed +so that it can be updated to a new binary version, should that trigger killing of P? More +generally, how long should the system wait before declaring that P's affinity is +violated? (Of course affinity is expressed in terms of label selectors, not for a specific +pod, but the scenario is easier to describe using a concrete pod.) This is closely related to +the concept of forgiveness (see issue #1574). In theory we could make this time duration be +configurable by the user on a per-pod basis, but for the first version of this feature we will +make it a configurable property of whichever component does the killing and that applies across +all pods using the feature. Making it configurable by the user would require a nontrivial change +to the API syntax (since the field would only apply to RequiredDuringSchedulingRequiredDuringExecution +affinity). + +## Implementation plan + +1. Add the `Affinity` field to PodSpec and the `PodAffinity` and `PodAntiAffinity` types to the API along with all of their descendant types. +2. Implement a scheduler predicate that takes `RequiredDuringSchedulingIgnoredDuringExecution` +affinity and anti-affinity into account. Include a workaround for the issue described at the end of the Affinity section of the Examples section (can't schedule first pod). +3. Implement a scheduler priority function that takes `PreferredDuringSchedulingIgnoredDuringExecution` affinity and anti-affinity into account +4. Implement a quota mechanism that charges for the entire topology domain when `RequiredDuringScheduling` anti-affinity is used. Later +this should be refined to only apply when it is used to request exclusive access, not when it is used to express conflict with specific pods. +5. Implement the recommended solution to the "co-existing with daemons" issue +6. At this point, the feature can be deployed. +7. Add the `RequiredDuringSchedulingRequiredDuringExecution` field to affinity and anti-affinity, and make sure +the pieces of the system already implemented for `RequiredDuringSchedulingIgnoredDuringExecution` also take +`RequiredDuringSchedulingRequiredDuringExecution` into account (e.g. the scheduler predicate, the quota mechanism, +the "co-existing with daemons" solution). +8. Add `RequiredDuringSchedulingRequiredDuringExecution` for "node" `TopologyKey` to Kubelet's admission decision +9. Implement code in Kubelet *or* the controllers that evicts a pod that no longer satisfies +`RequiredDuringSchedulingRequiredDuringExecution`. If Kubelet then only for "node" `TopologyKey`; +if controller then potentially for all `TopologyKeys`'s. +(see [this comment](https://github.com/kubernetes/kubernetes/issues/12744#issuecomment-164372008)). +Do so in a way that addresses the "determining which pod(s) to kill" issue. + +We assume Kubelet publishes labels describing the node's membership in all of the relevant scheduling +domains (e.g. node name, rack name, availability zone name, etc.). See #9044. + +## Backward compatiblity + +Old versions of the scheduler will ignore `Affinity`. + +Users should not start using `Affinity` until the full implementation has +been in Kubelet and the master for enough binary versions that we feel +comfortable that we will not need to roll back either Kubelet or +master to a version that does not support them. Longer-term we will +use a programatic approach to enforcing this (#4855). + +## Extensibility + +The design described here is the result of careful analysis of use cases, a decade of experience +with Borg at Google, and a review of similar features in other open-source container orchestration +systems. We believe that it properly balances the goal of expressiveness against the goals of +simplicity and efficiency of implementation. However, we recognize that +use cases may arise in the future that cannot be expressed using the syntax described here. +Although we are not implementing an affinity-specific extensibility mechanism for a variety +of reasons (simplicity of the codebase, simplicity of cluster deployment, desire for Kubernetes +users to get a consistent experience, etc.), the regular Kubernetes +annotation mechanism can be used to add or replace affinity rules. The way this work would is +1. Define one or more annotations to describe the new affinity rule(s) +1. User (or an admission controller) attaches the annotation(s) to pods to request the desired scheduling behavior. +If the new rule(s) *replace* one or more fields of `Affinity` then the user would omit those fields +from `Affinity`; if they are *additional rules*, then the user would fill in `Affinity` as well as the +annotation(s). +1. Scheduler takes the annotation(s) into account when scheduling. + +If some particular new syntax becomes popular, we would consider upstreaming it by integrating +it into the standard `Affinity`. + +## Future work and non-work + +One can imagine that in the anti-affinity RequiredDuringScheduling case +one might want to associate a number with the rule, +for example "do not allow this pod to share a rack with more than three other +pods (in total, or from the same service as the pod)." We could allow this to be +specified by adding an integer `Limit` to `PodAffinityTerm` just for the +`RequiredDuringScheduling` case. However, this flexibility complicates the +system and we do not intend to implement it. + +It is likely that the specification and implementation of pod anti-affinity +can be unified with [taints and tolerations](https://github.com/kubernetes/kubernetes/pull/18263), +and likewise that the specification and implementation of pod affinity +can be unified with [node affinity](https://github.com/kubernetes/kubernetes/pull/18261). +The basic idea is that pod labels would be "inherited" by the node, and pods +would only be able to specify affinity and anti-affinity for a node's labels. +Our main motivation for not unifying taints and tolerations with +pod anti-affinity is that we foresee taints and tolerations as being a concept that +only cluster administrators need to understand (and indeed in some setups taints and +tolerations wouldn't even be directly manipulated by a cluster administrator, +instead they would only be set by an admission controller that is implementing the administrator's +high-level policy about different classes of special machines and the users who belong to the groups +allowed to access them). Moreover, the concept of nodes "inheriting" labels +from pods seems complicated; it seems conceptually simpler to separate rules involving +relatively static properties of nodes from rules involving which other pods are running +on the same node or larger topology domain. + +Data/storage affinity is related to pod affinity, and is likely to draw on some of the +ideas we have used for pod affinity. Today, data/storage affinity is expressed using +node affinity, on the assumption that the pod knows which node(s) store(s) the data +it wants. But a more flexible approach would allow the pod to name the data rather than +the node. + +## Related issues + +The review for this proposal is in #18265. + +The topic of affinity/anti-affinity has generated a lot of discussion. The main issue +is #367 but #14484/#14485, #9560, #11369, #14543, #11707, #3945, #341, #1965, and #2906 +all have additional discussion and use cases. + +As the examples in this document have demonstrated, topological affinity is very useful +in clusters that are spread across availability zones, e.g. to co-locate pods of a service +in the same zone to avoid a wide-area network hop, or to spread pods across zones for +failure tolerance. #17059, #13056, #13063, and #4235 are relevant. + +Issue #15675 describes connection affinity, which is vaguely related. + +This proposal is to satisfy #14816. + +## Related work + +** TODO: cite references ** + + + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/podaffinity.md?pixel)]() +