diff --git a/docs/design/admission_control_limit_range.md b/docs/design/admission_control_limit_range.md new file mode 100644 index 00000000000..69fe144b248 --- /dev/null +++ b/docs/design/admission_control_limit_range.md @@ -0,0 +1,122 @@ +# Admission control plugin: LimitRanger + +## Background + +This document proposes a system for enforcing min/max limits per resource as part of admission control. + +## Model Changes + +A new resource, **LimitRange**, is introduced to enumerate min/max limits for a resource type scoped to a +Kubernetes namespace. + +``` +const ( + // Limit that applies to all pods in a namespace + LimitTypePod string = "Pod" + // Limit that applies to all containers in a namespace + LimitTypeContainer string = "Container" +) + +// LimitRangeItem defines a min/max usage limit for any resource that matches on kind +type LimitRangeItem struct { + // Type of resource that this limit applies to + Type string `json:"type,omitempty"` + // Max usage constraints on this kind by resource name + Max ResourceList `json:"max,omitempty"` + // Min usage constraints on this kind by resource name + Min ResourceList `json:"min,omitempty"` +} + +// LimitRangeSpec defines a min/max usage limit for resources that match on kind +type LimitRangeSpec struct { + // Limits is the list of LimitRangeItem objects that are enforced + Limits []LimitRangeItem `json:"limits"` +} + +// LimitRange sets resource usage limits for each kind of resource in a Namespace +type LimitRange struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + + // Spec defines the limits enforced + Spec LimitRangeSpec `json:"spec,omitempty"` +} + +// LimitRangeList is a list of LimitRange items. +type LimitRangeList struct { + TypeMeta `json:",inline"` + ListMeta `json:"metadata,omitempty"` + + // Items is a list of LimitRange objects + Items []LimitRange `json:"items"` +} +``` + +## AdmissionControl plugin: LimitRanger + +The **LimitRanger** plug-in introspects all incoming admission requests. + +It makes decisions by evaluating the incoming object against all defined **LimitRange** objects in the request context namespace. + +The following min/max limits are imposed: + +**Type: Container** + +| ResourceName | Description | +| ------------ | ----------- | +| cpu | Min/Max amount of cpu per container | +| memory | Min/Max amount of memory per container | + +**Type: Pod** + +| ResourceName | Description | +| ------------ | ----------- | +| cpu | Min/Max amount of cpu per pod | +| memory | Min/Max amount of memory per pod | + +If the incoming object would cause a violation of the enumerated constraints, the request is denied with a set of +messages explaining what constraints were the source of the denial. + +If a constraint is not enumerated by a **LimitRange** it is not tracked. + +## kube-apiserver + +The server is updated to be aware of **LimitRange** objects. + +The constraints are only enforced if the kube-apiserver is started as follows: + +``` +$ kube-apiserver -admission_control=LimitRanger +``` + +## kubectl + +kubectl is modified to support the **LimitRange** resource. + +```kubectl describe``` provides a human-readable output of limits. + +For example, + +``` +$ kubectl namespace myspace +$ kubectl create -f examples/limitrange/limit-range.json +$ kubectl get limits +NAME +limits +$ kubectl describe limits limits +Name: limits +Type Resource Min Max +---- -------- --- --- +Pod memory 1Mi 1Gi +Pod cpu 250m 2 +Container cpu 250m 2 +Container memory 1Mi 1Gi +``` + +## Future Enhancements: Define limits for a particular pod or container. + +In the current proposal, the **LimitRangeItem** matches purely on **LimitRangeItem.Type** + +It is expected we will want to define limits for particular pods or containers by name/uid and label/field selector. + +To make a **LimitRangeItem** more restrictive, we will intend to add these additional restrictions at a future point in time. diff --git a/docs/design/admission_control_resource_quota.md b/docs/design/admission_control_resource_quota.md new file mode 100644 index 00000000000..08bc6bec315 --- /dev/null +++ b/docs/design/admission_control_resource_quota.md @@ -0,0 +1,152 @@ +# Admission control plugin: ResourceQuota + +## Background + +This document proposes a system for enforcing hard resource usage limits per namespace as part of admission control. + +## Model Changes + +A new resource, **ResourceQuota**, is introduced to enumerate hard resource limits in a Kubernetes namespace. + +A new resource, **ResourceQuotaUsage**, is introduced to support atomic updates of a **ResourceQuota** status. + +``` +// The following identify resource constants for Kubernetes object types +const ( + // Pods, number + ResourcePods ResourceName = "pods" + // Services, number + ResourceServices ResourceName = "services" + // ReplicationControllers, number + ResourceReplicationControllers ResourceName = "replicationcontrollers" + // ResourceQuotas, number + ResourceQuotas ResourceName = "resourcequotas" +) + +// ResourceQuotaSpec defines the desired hard limits to enforce for Quota +type ResourceQuotaSpec struct { + // Hard is the set of desired hard limits for each named resource + Hard ResourceList `json:"hard,omitempty"` +} + +// ResourceQuotaStatus defines the enforced hard limits and observed use +type ResourceQuotaStatus struct { + // Hard is the set of enforced hard limits for each named resource + Hard ResourceList `json:"hard,omitempty"` + // Used is the current observed total usage of the resource in the namespace + Used ResourceList `json:"used,omitempty"` +} + +// ResourceQuota sets aggregate quota restrictions enforced per namespace +type ResourceQuota struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + + // Spec defines the desired quota + Spec ResourceQuotaSpec `json:"spec,omitempty"` + + // Status defines the actual enforced quota and its current usage + Status ResourceQuotaStatus `json:"status,omitempty"` +} + +// ResourceQuotaUsage captures system observed quota status per namespace +// It is used to enforce atomic updates of a backing ResourceQuota.Status field in storage +type ResourceQuotaUsage struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + + // Status defines the actual enforced quota and its current usage + Status ResourceQuotaStatus `json:"status,omitempty"` +} + +// ResourceQuotaList is a list of ResourceQuota items +type ResourceQuotaList struct { + TypeMeta `json:",inline"` + ListMeta `json:"metadata,omitempty"` + + // Items is a list of ResourceQuota objects + Items []ResourceQuota `json:"items"` +} + +``` + +## AdmissionControl plugin: ResourceQuota + +The **ResourceQuota** plug-in introspects all incoming admission requests. + +It makes decisions by evaluating the incoming object against all defined **ResourceQuota.Status.Hard** resource limits in the request +namespace. If acceptance of the resource would cause the total usage of a named resource to exceed its hard limit, the request is denied. + +The following resource limits are imposed as part of core Kubernetes at the namespace level: + +| ResourceName | Description | +| ------------ | ----------- | +| cpu | Total cpu usage | +| memory | Total memory usage | +| pods | Total number of pods | +| services | Total number of services | +| replicationcontrollers | Total number of replication controllers | +| resourcequotas | Total number of resource quotas | + +Any resource that is not part of core Kubernetes must follow the resource naming convention prescribed by Kubernetes. + +This means the resource must have a fully-qualified name (i.e. mycompany.org/shinynewresource) + +If the incoming request does not cause the total usage to exceed any of the enumerated hard resource limits, the plug-in will post a +**ResourceQuotaUsage** document to the server to atomically update the observed usage based on the previously read +**ResourceQuota.ResourceVersion**. This keeps incremental usage atomically consistent, but does introduce a bottleneck (intentionally) +into the system. + +To optimize system performance, it is encouraged that all resource quotas are tracked on the same **ResourceQuota** document. As a result, +its encouraged to actually impose a cap on the total number of individual quotas that are tracked in the **Namespace** to 1 by explicitly +capping it in **ResourceQuota** document. + +## kube-apiserver + +The server is updated to be aware of **ResourceQuota** objects. + +The quota is only enforced if the kube-apiserver is started as follows: + +``` +$ kube-apiserver -admission_control=ResourceQuota +``` + +## kube-controller-manager + +A new controller is defined that runs a synch loop to calculate quota usage across the namespace. + +**ResourceQuota** usage is only calculated if a namespace has a **ResourceQuota** object. + +If the observed usage is different than the recorded usage, the controller sends a **ResourceQuotaUsage** resource +to the server to atomically update. + +The synchronization loop frequency will control how quickly DELETE actions are recorded in the system and usage is ticked down. + +To optimize the synchronization loop, this controller will WATCH on Pod resources to track DELETE events, and in response, recalculate +usage. This is because a Pod deletion will have the most impact on observed cpu and memory usage in the system, and we anticipate +this being the resource most closely running at the prescribed quota limits. + +## kubectl + +kubectl is modified to support the **ResourceQuota** resource. + +```kubectl describe``` provides a human-readable output of quota. + +For example, + +``` +$ kubectl namespace myspace +$ kubectl create -f examples/resourcequota/resource-quota.json +$ kubectl get quota +NAME +myquota +$ kubectl describe quota myquota +Name: myquota +Resource Used Hard +-------- ---- ---- +cpu 100m 20 +memory 0 1.5Gb +pods 1 10 +replicationControllers 1 10 +services 2 3 +``` \ No newline at end of file