From ec568d7d9a5983175764c230b287df14b3a458a0 Mon Sep 17 00:00:00 2001 From: derekwaynecarr Date: Thu, 18 Dec 2014 13:58:23 -0500 Subject: [PATCH 1/5] Resource controller proposal --- docs/design/resource_controller.md | 231 +++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 docs/design/resource_controller.md diff --git a/docs/design/resource_controller.md b/docs/design/resource_controller.md new file mode 100644 index 00000000000..2150f6dc796 --- /dev/null +++ b/docs/design/resource_controller.md @@ -0,0 +1,231 @@ +# Kubernetes Proposal: ResourceController + +**Related PR:** + +| Topic | Link | +| ----- | ---- | +| Admission Control Proposal | https://github.com/GoogleCloudPlatform/kubernetes/pull/2501 | +| Separate validation from RESTStorage | https://github.com/GoogleCloudPlatform/kubernetes/issues/2977 | + +## Background + +This document proposes a system for enforcing resource limits as part of admission control. + +## Model Changes + +A new resource, **ResourceController**, is introduced to enumerate resource usage constraints scoped to a Kubernetes namespace. + +Authorized users are able to set the **ResourceController.Spec** fields to enumerate desired constraints. + +``` +// ResourceController is an enumerated set of resources constraints enforced as part admission control plug-in +type ResourceController struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + // Spec represents the imposed constraints for allowed resources + Spec ResourceControllerSpec `json:"spec,omitempty"` + // Status represents the observed allocated resources to inform constraints + Status ResourceControllerStatus `json:"status,omitempty"` +} + +type ResourceControllerSpec struct { + // Allowed represents the available resources allowed in a quota + Allowed ResourceList `json:"allowed,omitempty"` +} + +type ResourceControllerStatus struct { + // Allowed represents the available resources allowed in a quota + Allowed ResourceList `json:"allowed,omitempty"` + // Allocated represents the allocated resources leveraged against your quota + Allocated ResourceList `json:"allocated,omitempty"` +} + +// ResourceControllerList is a collection of resource controllers. +type ResourceControllerList struct { + TypeMeta `json:",inline"` + ListMeta `json:"metadata,omitempty"` + Items []ResourceController `json:"items"` +} +``` + +Authorized users are able to provide a **ResourceObservation** to control a **ResourceController.Status**. + +``` +// ResourceObservation is written by a resource-controller to update ResourceController.Status +type ResourceObservation struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + + // Status represents the observed allocated resources to inform constraints + Status ResourceControllerStatus `json:"status,omitempty"` +} +``` + +## AdmissionControl plugin: ResourceLimits + +The **ResourceLimits** plug-in introspects all incoming admission requests. + +It makes decisions by introspecting the incoming object, current status, and enumerated constraints on **ResourceController**. + +The following constraints are proposed as enforceable: + +| Key | Type | Description | +| ------ | -------- | -------- | +| kubernetes.io/namespace/pods | int | Maximum number of pods per namespace | +| kubernetes.io/namespace/replicationControllers | int | Maximum number of replicationControllers per namespace | +| kubernetes.io/namespace/services | int | Maximum number of services per namespace | +| kubernetes.io/pods/containers | int | Maximum number of containers per pod | +| kubernetes.io/pods/containers/memory/max | int | Maximum amount of memory per container in a pod | +| kubernetes.io/pods/containers/memory/min | int | Minimum amount of memory per container in a pod | +| kubernetes.io/pods/containers/cpu/max | int | Maximum amount of CPU per container in a pod | +| kubernetes.io/pods/containers/cpu/min | int | Minimum amount of CPU per container in a pod | +| kubernetes.io/pods/cpu/max | int | Maximum CPU usage across all containers per pod | +| kubernetes.io/pods/cpu/min | int | Minimum CPU usage across all containers per pod | +| kubernetes.io/pods/memory/max | int | Maximum memory usage across all containers in pod | +| kubernetes.io/pods/memory/min | int | Minimum memory usage across all containers in pod | +| kubernetes.io/replicationController/replicas | int | Maximum number of replicas per replication controller | + +If the incoming resource would cause a violation of the enumerated constraints, the request is denied with a set of +messages explaining what constraints were the source of the denial. + +If a constraint is not enumerated by a **ResourceController** it is not tracked. + +If a constraint spans resources, for example, it tracks the total number of some **kind** in a **namespace**, +the plug-in will post a **ResourceObservation** with the new incremented **Allocated*** usage for that constraint +using a compare-and-swap to ensure transactional integrity. It is possible that the allocated usage will be persisted +on a create operation, but the create can fail later in the request flow for some other unknown reason. For this scenario, +the allocated usage will appear greater than the actual usage, the **kube-resource-controller** is responsible for +synchronizing the observed allocated usage with actual usage. For delete requests, we will not decrement usage right away, +and will always rely on the **kube-resource-controller** to bring the observed value in line. This is needed until +etcd supports atomic transactions across multiple resources. + +## kube-apiserver + +The server is updated to be aware of **ResourceController** and **ResourceObservation** objects. + +The constraints are only enforced if the kube-apiserver is started as follows: + +``` +$ kube-apiserver -admission_control=ResourceLimits +``` + +## kube-resource-controller + +This is a new daemon that observes **ResourceController** objects in the cluster, and updates their status with current cluster state. + +The daemon runs a synchronization loop to do the following: + +For each resource controller, perform the following steps: + + 1. Reconcile **ResourceController.Status.Allowed** with **ResourceController.Spec.Allowed** + 2. Reconcile **ResourceController.Status.Allocated** with constraints enumerated in **ResourceController.Status.Allowed** + 3. If there was a change, atomically update **ResourceObservation** to force an update to **ResourceController.Status** + +At step 1, allow the **kube-resource-controller** to support an administrator supplied override to enforce that what the +set of constraints desired to not conflict with any configured global constraints. For example, do not let +a **kubernetes.io/pods/memory/max** for any pod in any namespace exceed 8GB. These global constraints could be supplied +via an alternate location in **etcd**, for example, a **ResourceController** in an **infra** namespace that is populated on +bootstrap. + +At step 2, for fields that track total number of {kind} in a namespace, we query the cluster to ensure that the observed status +is in-line with the actual tracked status. This is a stop-gap to etcd supporting transactions across resource updates to ensure +that when a resource is deleted, we can update the observed status. + +## kubectl + +kubectl is modified to support the **ResourceController** resource. + +```kubectl describe``` provides a human-readable output of current constraints and usage in the namespace. + +For example, + +``` +$ kubectl namespace myspace +$ kubectl create -f examples/resource-controller/resource-controller.json +$ kubectl get resourceControllers +NAME LABELS +limits +$ kubectl describe resourceController limits +Name: limits +Key Enforced Allocated +---- ----- ---- +Max pods 15 13 +Max replication controllers 2 2 +Max services 5 0 +Max containers per pod 2 0 +Max replica size 10 0 +... +``` + +## Scenario: How this works in practice + +Admin user wants to impose resource constraints in namespace ```dev``` to enforce the following: + +1. A pod cannot use more than 8GB of RAM +2. The namespace cannot run more than 100 pods at a time. + +To enforce this constraint, the Admin does the following: + +``` +$ cat resource-controller.json +{ + "id": "limits", + "kind": "ResourceController", + "apiVersion": "v1beta1", + "spec": { + "allowed": { + "kubernetes.io/namespace/pods": 100, + "kubernetes.io/pods/memory/max": 8000, + } + }, + "labels": {} +} +$ kubectl namespace dev +$ kubectl create -f resource-controller.json +``` + +The **kube-resource-controller** sees that a new **ResourceController** resource was created, and updates its +status with the current observations in the namespace. + +The Admin describes the resource controller to see the current status: + +``` +$ kubectl describe resourceController limits +Name: limits +Key Enforced Allocated +---- ----- ---- +Max pods 100 50 +Max memory per pod 8000 4000 +```` + +The Admin sees that the current ```dev``` namespace is using 50 pods, and the largest pod consumes 4GB of RAM. + +The Developer that uses this namespace uses the system until he discovers he has exceeded his limits: + +``` +$ kubectl namespace dev +$ kubectl create -f pod.json +Unable to create pod. You have exceeded your max pods in the namespace of 100. +``` + +or + +``` +$ kubectl namespace dev +$ kubectl create -f pod.json +Unable to create pod. It exceeds the max memory usage per pod of 8000 MB. +``` + +The Developer can observe his constraints as appropriate: +``` +$ kubectl describe resourceController limits +Name: limits +Key Enforced Allocated +---- ----- ---- +Max pods 100 100 +Max memory per pod 8000 4000 +```` + +And as a consequence reduce his current number of running pods, or memory requirements of the pod to proceed. +Or he could contact the Admin for his namespace to allocate him more resources. + From 2803782af6fa9e6f9b66e432d077e1ab2c82648b Mon Sep 17 00:00:00 2001 From: derekwaynecarr Date: Thu, 22 Jan 2015 22:31:28 -0500 Subject: [PATCH 2/5] Design document for LimitRange --- docs/design/admission_control_limit_range.md | 122 +++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 docs/design/admission_control_limit_range.md diff --git a/docs/design/admission_control_limit_range.md b/docs/design/admission_control_limit_range.md new file mode 100644 index 00000000000..69fe144b248 --- /dev/null +++ b/docs/design/admission_control_limit_range.md @@ -0,0 +1,122 @@ +# Admission control plugin: LimitRanger + +## Background + +This document proposes a system for enforcing min/max limits per resource as part of admission control. + +## Model Changes + +A new resource, **LimitRange**, is introduced to enumerate min/max limits for a resource type scoped to a +Kubernetes namespace. + +``` +const ( + // Limit that applies to all pods in a namespace + LimitTypePod string = "Pod" + // Limit that applies to all containers in a namespace + LimitTypeContainer string = "Container" +) + +// LimitRangeItem defines a min/max usage limit for any resource that matches on kind +type LimitRangeItem struct { + // Type of resource that this limit applies to + Type string `json:"type,omitempty"` + // Max usage constraints on this kind by resource name + Max ResourceList `json:"max,omitempty"` + // Min usage constraints on this kind by resource name + Min ResourceList `json:"min,omitempty"` +} + +// LimitRangeSpec defines a min/max usage limit for resources that match on kind +type LimitRangeSpec struct { + // Limits is the list of LimitRangeItem objects that are enforced + Limits []LimitRangeItem `json:"limits"` +} + +// LimitRange sets resource usage limits for each kind of resource in a Namespace +type LimitRange struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + + // Spec defines the limits enforced + Spec LimitRangeSpec `json:"spec,omitempty"` +} + +// LimitRangeList is a list of LimitRange items. +type LimitRangeList struct { + TypeMeta `json:",inline"` + ListMeta `json:"metadata,omitempty"` + + // Items is a list of LimitRange objects + Items []LimitRange `json:"items"` +} +``` + +## AdmissionControl plugin: LimitRanger + +The **LimitRanger** plug-in introspects all incoming admission requests. + +It makes decisions by evaluating the incoming object against all defined **LimitRange** objects in the request context namespace. + +The following min/max limits are imposed: + +**Type: Container** + +| ResourceName | Description | +| ------------ | ----------- | +| cpu | Min/Max amount of cpu per container | +| memory | Min/Max amount of memory per container | + +**Type: Pod** + +| ResourceName | Description | +| ------------ | ----------- | +| cpu | Min/Max amount of cpu per pod | +| memory | Min/Max amount of memory per pod | + +If the incoming object would cause a violation of the enumerated constraints, the request is denied with a set of +messages explaining what constraints were the source of the denial. + +If a constraint is not enumerated by a **LimitRange** it is not tracked. + +## kube-apiserver + +The server is updated to be aware of **LimitRange** objects. + +The constraints are only enforced if the kube-apiserver is started as follows: + +``` +$ kube-apiserver -admission_control=LimitRanger +``` + +## kubectl + +kubectl is modified to support the **LimitRange** resource. + +```kubectl describe``` provides a human-readable output of limits. + +For example, + +``` +$ kubectl namespace myspace +$ kubectl create -f examples/limitrange/limit-range.json +$ kubectl get limits +NAME +limits +$ kubectl describe limits limits +Name: limits +Type Resource Min Max +---- -------- --- --- +Pod memory 1Mi 1Gi +Pod cpu 250m 2 +Container cpu 250m 2 +Container memory 1Mi 1Gi +``` + +## Future Enhancements: Define limits for a particular pod or container. + +In the current proposal, the **LimitRangeItem** matches purely on **LimitRangeItem.Type** + +It is expected we will want to define limits for particular pods or containers by name/uid and label/field selector. + +To make a **LimitRangeItem** more restrictive, we will intend to add these additional restrictions at a future point in time. From c772de762f7edf69a5b066b2b617452ad57a6041 Mon Sep 17 00:00:00 2001 From: derekwaynecarr Date: Fri, 23 Jan 2015 12:38:59 -0500 Subject: [PATCH 3/5] ResourceQuota proposal --- .../admission_control_resource_quota.md | 146 ++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 docs/design/admission_control_resource_quota.md diff --git a/docs/design/admission_control_resource_quota.md b/docs/design/admission_control_resource_quota.md new file mode 100644 index 00000000000..c5cc60c4cd0 --- /dev/null +++ b/docs/design/admission_control_resource_quota.md @@ -0,0 +1,146 @@ +# Admission control plugin: ResourceQuota + +## Background + +This document proposes a system for enforcing hard resource usage limits per namespace as part of admission control. + +## Model Changes + +A new resource, **ResourceQuota**, is introduced to enumerate hard resource limits in a Kubernetes namespace. + +A new resource, **ResourceQuotaUsage**, is introduced to support atomic updates of a **ResourceQuota** status. + +``` +// The following identify resource constants for Kubernetes object types +const ( + // Pods, number + ResourcePods ResourceName = "pods" + // Services, number + ResourceServices ResourceName = "services" + // ReplicationControllers, number + ResourceReplicationControllers ResourceName = "replicationcontrollers" + // ResourceQuotas, number + ResourceQuotas ResourceName = "resourcequotas" +) + +// ResourceQuotaSpec defines the desired hard limits to enforce for Quota +type ResourceQuotaSpec struct { + // Hard is the set of desired hard limits for each named resource + Hard ResourceList `json:"hard,omitempty"` +} + +// ResourceQuotaStatus defines the enforced hard limits and observed use +type ResourceQuotaStatus struct { + // Hard is the set of enforced hard limits for each named resource + Hard ResourceList `json:"hard,omitempty"` + // Used is the current observed total usage of the resource in the namespace + Used ResourceList `json:"used,omitempty"` +} + +// ResourceQuota sets aggregate quota restrictions enforced per namespace +type ResourceQuota struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + + // Spec defines the desired quota + Spec ResourceQuotaSpec `json:"spec,omitempty"` + + // Status defines the actual enforced quota and its current usage + Status ResourceQuotaStatus `json:"status,omitempty"` +} + +// ResourceQuotaUsage captures system observed quota status per namespace +// It is used to enforce atomic updates of a backing ResourceQuota.Status field in storage +type ResourceQuotaUsage struct { + TypeMeta `json:",inline"` + ObjectMeta `json:"metadata,omitempty"` + + // Status defines the actual enforced quota and its current usage + Status ResourceQuotaStatus `json:"status,omitempty"` +} + +// ResourceQuotaList is a list of ResourceQuota items +type ResourceQuotaList struct { + TypeMeta `json:",inline"` + ListMeta `json:"metadata,omitempty"` + + // Items is a list of ResourceQuota objects + Items []ResourceQuota `json:"items"` +} + +``` + +## AdmissionControl plugin: ResourceQuota + +The **ResourceQuota** plug-in introspects all incoming admission requests. + +It makes decisions by evaluating the incoming object against all defined **ResourceQuota.Status.Hard** resource limits in the request +namespace. If acceptance of the resource would cause the total usage of a named resource to exceed its hard limit, the request is denied. + +The following resource limits are imposed as part of core Kubernetes: + +| ResourceName | Description | +| ------------ | ----------- | +| cpu | Total cpu usage | +| memory | Total memory usage | +| pods | Total number of pods | +| services | Total number of services | +| replicationcontrollers | Total number of replication controllers | +| resourcequotas | Total number of resource quotas | + +Any resource that is not part of core Kubernetes must follow the resource naming convention prescribed by Kubernetes. + +This means the resource must have a fully-qualified name (i.e. mycompany.org/shinynewresource) + +If the incoming request does not cause the total usage to exceed any of the enumerated hard resource limits, the plug-in will post a +**ResourceQuotaUsage** document to the server to atomically update the observed usage based on the previously read +**ResourceQuota.ResourceVersion**. This keeps incremental usage atomically consistent, but does introduce a bottleneck (intentionally) +into the system. + +## kube-apiserver + +The server is updated to be aware of **ResourceQuota** objects. + +The quota is only enforced if the kube-apiserver is started as follows: + +``` +$ kube-apiserver -admission_control=ResourceQuota +``` + +## kube-controller-manager + +A new controller is defined that runs a synch loop to run usage stats across the namespace. + +If the observed usage is different than the recorded usage, the controller sends a **ResourceQuotaUsage** resource +to the server to atomically update. + +The synchronization loop frequency will control how quickly DELETE actions are recorded in the system and usage is ticked down. + +To optimize the synchronization loop, this controller will WATCH on Pod resources to track DELETE events, and in response, recalculate +usage. This is because a Pod deletion will have the most impact on observed cpu and memory usage in the system, and we anticipate +this being the resource most closely running at the prescribed quota limits. + +## kubectl + +kubectl is modified to support the **ResourceQuota** resource. + +```kubectl describe``` provides a human-readable output of quota. + +For example, + +``` +$ kubectl namespace myspace +$ kubectl create -f examples/resourcequota/resource-quota.json +$ kubectl get quota +NAME +myquota +$ kubectl describe quota myquota +Name: myquota +Resource Used Hard +-------- ---- ---- +cpu 100m 20 +memory 0 1.5Gb +pods 1 10 +replicationControllers 1 10 +services 2 3 +``` \ No newline at end of file From 6c17bfa5264a83b5be167a6189a1dc0b668b2fc1 Mon Sep 17 00:00:00 2001 From: derekwaynecarr Date: Fri, 23 Jan 2015 12:39:53 -0500 Subject: [PATCH 4/5] Remove resource_controller proposal --- docs/design/resource_controller.md | 231 ----------------------------- 1 file changed, 231 deletions(-) delete mode 100644 docs/design/resource_controller.md diff --git a/docs/design/resource_controller.md b/docs/design/resource_controller.md deleted file mode 100644 index 2150f6dc796..00000000000 --- a/docs/design/resource_controller.md +++ /dev/null @@ -1,231 +0,0 @@ -# Kubernetes Proposal: ResourceController - -**Related PR:** - -| Topic | Link | -| ----- | ---- | -| Admission Control Proposal | https://github.com/GoogleCloudPlatform/kubernetes/pull/2501 | -| Separate validation from RESTStorage | https://github.com/GoogleCloudPlatform/kubernetes/issues/2977 | - -## Background - -This document proposes a system for enforcing resource limits as part of admission control. - -## Model Changes - -A new resource, **ResourceController**, is introduced to enumerate resource usage constraints scoped to a Kubernetes namespace. - -Authorized users are able to set the **ResourceController.Spec** fields to enumerate desired constraints. - -``` -// ResourceController is an enumerated set of resources constraints enforced as part admission control plug-in -type ResourceController struct { - TypeMeta `json:",inline"` - ObjectMeta `json:"metadata,omitempty"` - // Spec represents the imposed constraints for allowed resources - Spec ResourceControllerSpec `json:"spec,omitempty"` - // Status represents the observed allocated resources to inform constraints - Status ResourceControllerStatus `json:"status,omitempty"` -} - -type ResourceControllerSpec struct { - // Allowed represents the available resources allowed in a quota - Allowed ResourceList `json:"allowed,omitempty"` -} - -type ResourceControllerStatus struct { - // Allowed represents the available resources allowed in a quota - Allowed ResourceList `json:"allowed,omitempty"` - // Allocated represents the allocated resources leveraged against your quota - Allocated ResourceList `json:"allocated,omitempty"` -} - -// ResourceControllerList is a collection of resource controllers. -type ResourceControllerList struct { - TypeMeta `json:",inline"` - ListMeta `json:"metadata,omitempty"` - Items []ResourceController `json:"items"` -} -``` - -Authorized users are able to provide a **ResourceObservation** to control a **ResourceController.Status**. - -``` -// ResourceObservation is written by a resource-controller to update ResourceController.Status -type ResourceObservation struct { - TypeMeta `json:",inline"` - ObjectMeta `json:"metadata,omitempty"` - - // Status represents the observed allocated resources to inform constraints - Status ResourceControllerStatus `json:"status,omitempty"` -} -``` - -## AdmissionControl plugin: ResourceLimits - -The **ResourceLimits** plug-in introspects all incoming admission requests. - -It makes decisions by introspecting the incoming object, current status, and enumerated constraints on **ResourceController**. - -The following constraints are proposed as enforceable: - -| Key | Type | Description | -| ------ | -------- | -------- | -| kubernetes.io/namespace/pods | int | Maximum number of pods per namespace | -| kubernetes.io/namespace/replicationControllers | int | Maximum number of replicationControllers per namespace | -| kubernetes.io/namespace/services | int | Maximum number of services per namespace | -| kubernetes.io/pods/containers | int | Maximum number of containers per pod | -| kubernetes.io/pods/containers/memory/max | int | Maximum amount of memory per container in a pod | -| kubernetes.io/pods/containers/memory/min | int | Minimum amount of memory per container in a pod | -| kubernetes.io/pods/containers/cpu/max | int | Maximum amount of CPU per container in a pod | -| kubernetes.io/pods/containers/cpu/min | int | Minimum amount of CPU per container in a pod | -| kubernetes.io/pods/cpu/max | int | Maximum CPU usage across all containers per pod | -| kubernetes.io/pods/cpu/min | int | Minimum CPU usage across all containers per pod | -| kubernetes.io/pods/memory/max | int | Maximum memory usage across all containers in pod | -| kubernetes.io/pods/memory/min | int | Minimum memory usage across all containers in pod | -| kubernetes.io/replicationController/replicas | int | Maximum number of replicas per replication controller | - -If the incoming resource would cause a violation of the enumerated constraints, the request is denied with a set of -messages explaining what constraints were the source of the denial. - -If a constraint is not enumerated by a **ResourceController** it is not tracked. - -If a constraint spans resources, for example, it tracks the total number of some **kind** in a **namespace**, -the plug-in will post a **ResourceObservation** with the new incremented **Allocated*** usage for that constraint -using a compare-and-swap to ensure transactional integrity. It is possible that the allocated usage will be persisted -on a create operation, but the create can fail later in the request flow for some other unknown reason. For this scenario, -the allocated usage will appear greater than the actual usage, the **kube-resource-controller** is responsible for -synchronizing the observed allocated usage with actual usage. For delete requests, we will not decrement usage right away, -and will always rely on the **kube-resource-controller** to bring the observed value in line. This is needed until -etcd supports atomic transactions across multiple resources. - -## kube-apiserver - -The server is updated to be aware of **ResourceController** and **ResourceObservation** objects. - -The constraints are only enforced if the kube-apiserver is started as follows: - -``` -$ kube-apiserver -admission_control=ResourceLimits -``` - -## kube-resource-controller - -This is a new daemon that observes **ResourceController** objects in the cluster, and updates their status with current cluster state. - -The daemon runs a synchronization loop to do the following: - -For each resource controller, perform the following steps: - - 1. Reconcile **ResourceController.Status.Allowed** with **ResourceController.Spec.Allowed** - 2. Reconcile **ResourceController.Status.Allocated** with constraints enumerated in **ResourceController.Status.Allowed** - 3. If there was a change, atomically update **ResourceObservation** to force an update to **ResourceController.Status** - -At step 1, allow the **kube-resource-controller** to support an administrator supplied override to enforce that what the -set of constraints desired to not conflict with any configured global constraints. For example, do not let -a **kubernetes.io/pods/memory/max** for any pod in any namespace exceed 8GB. These global constraints could be supplied -via an alternate location in **etcd**, for example, a **ResourceController** in an **infra** namespace that is populated on -bootstrap. - -At step 2, for fields that track total number of {kind} in a namespace, we query the cluster to ensure that the observed status -is in-line with the actual tracked status. This is a stop-gap to etcd supporting transactions across resource updates to ensure -that when a resource is deleted, we can update the observed status. - -## kubectl - -kubectl is modified to support the **ResourceController** resource. - -```kubectl describe``` provides a human-readable output of current constraints and usage in the namespace. - -For example, - -``` -$ kubectl namespace myspace -$ kubectl create -f examples/resource-controller/resource-controller.json -$ kubectl get resourceControllers -NAME LABELS -limits -$ kubectl describe resourceController limits -Name: limits -Key Enforced Allocated ----- ----- ---- -Max pods 15 13 -Max replication controllers 2 2 -Max services 5 0 -Max containers per pod 2 0 -Max replica size 10 0 -... -``` - -## Scenario: How this works in practice - -Admin user wants to impose resource constraints in namespace ```dev``` to enforce the following: - -1. A pod cannot use more than 8GB of RAM -2. The namespace cannot run more than 100 pods at a time. - -To enforce this constraint, the Admin does the following: - -``` -$ cat resource-controller.json -{ - "id": "limits", - "kind": "ResourceController", - "apiVersion": "v1beta1", - "spec": { - "allowed": { - "kubernetes.io/namespace/pods": 100, - "kubernetes.io/pods/memory/max": 8000, - } - }, - "labels": {} -} -$ kubectl namespace dev -$ kubectl create -f resource-controller.json -``` - -The **kube-resource-controller** sees that a new **ResourceController** resource was created, and updates its -status with the current observations in the namespace. - -The Admin describes the resource controller to see the current status: - -``` -$ kubectl describe resourceController limits -Name: limits -Key Enforced Allocated ----- ----- ---- -Max pods 100 50 -Max memory per pod 8000 4000 -```` - -The Admin sees that the current ```dev``` namespace is using 50 pods, and the largest pod consumes 4GB of RAM. - -The Developer that uses this namespace uses the system until he discovers he has exceeded his limits: - -``` -$ kubectl namespace dev -$ kubectl create -f pod.json -Unable to create pod. You have exceeded your max pods in the namespace of 100. -``` - -or - -``` -$ kubectl namespace dev -$ kubectl create -f pod.json -Unable to create pod. It exceeds the max memory usage per pod of 8000 MB. -``` - -The Developer can observe his constraints as appropriate: -``` -$ kubectl describe resourceController limits -Name: limits -Key Enforced Allocated ----- ----- ---- -Max pods 100 100 -Max memory per pod 8000 4000 -```` - -And as a consequence reduce his current number of running pods, or memory requirements of the pod to proceed. -Or he could contact the Admin for his namespace to allocate him more resources. - From badf5560a462850428d31088ca8e9141f1e778aa Mon Sep 17 00:00:00 2001 From: derekwaynecarr Date: Fri, 23 Jan 2015 12:41:44 -0500 Subject: [PATCH 5/5] Doc tweaks --- docs/design/admission_control_resource_quota.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/design/admission_control_resource_quota.md b/docs/design/admission_control_resource_quota.md index c5cc60c4cd0..08bc6bec315 100644 --- a/docs/design/admission_control_resource_quota.md +++ b/docs/design/admission_control_resource_quota.md @@ -77,7 +77,7 @@ The **ResourceQuota** plug-in introspects all incoming admission requests. It makes decisions by evaluating the incoming object against all defined **ResourceQuota.Status.Hard** resource limits in the request namespace. If acceptance of the resource would cause the total usage of a named resource to exceed its hard limit, the request is denied. -The following resource limits are imposed as part of core Kubernetes: +The following resource limits are imposed as part of core Kubernetes at the namespace level: | ResourceName | Description | | ------------ | ----------- | @@ -97,6 +97,10 @@ If the incoming request does not cause the total usage to exceed any of the enum **ResourceQuota.ResourceVersion**. This keeps incremental usage atomically consistent, but does introduce a bottleneck (intentionally) into the system. +To optimize system performance, it is encouraged that all resource quotas are tracked on the same **ResourceQuota** document. As a result, +its encouraged to actually impose a cap on the total number of individual quotas that are tracked in the **Namespace** to 1 by explicitly +capping it in **ResourceQuota** document. + ## kube-apiserver The server is updated to be aware of **ResourceQuota** objects. @@ -109,7 +113,9 @@ $ kube-apiserver -admission_control=ResourceQuota ## kube-controller-manager -A new controller is defined that runs a synch loop to run usage stats across the namespace. +A new controller is defined that runs a synch loop to calculate quota usage across the namespace. + +**ResourceQuota** usage is only calculated if a namespace has a **ResourceQuota** object. If the observed usage is different than the recorded usage, the controller sends a **ResourceQuotaUsage** resource to the server to atomically update.