kubernetes/docs/proposals/controller-ref.md

ControllerRef proposal

Author: gmarek@ Last edit: 2016-05-11 Status: raw

Approvers:

• briangrant
• dbsmith

Table of Contents

• Goal of ControllerReference
• Non goals
• API and semantic changes
• Upgrade/downgrade procedure
• Orphaning/adoption
• Implementation plan (sketch)
• Considered alternatives

Goal of ControllerReference

Main goal of ControllerReference effort is to solve a problem of overlapping controllers that fight over some resources (e.g. ReplicaSets fighting with ReplicationControllers over Pods), which cause serious problems such as exploding memory of Controller Manager.

We don’t want to have (just) an in-memory solution, as we don’t want a Controller Manager crash to cause massive changes in object ownership in the system. I.e. we need to persist the information about "owning controller".

Secondary goal of this effort is to improve performance of various controllers and schedulers, by removing the need for expensive lookup for all matching "controllers".

Non goals

Cascading deletion is not a goal of this effort. Cascading deletion will use ownerReferences, which is a separate effort.

ControllerRef will extend OwnerReference and reuse machinery written for it (GarbageCollector, adoption/orphaning logic).

API and semantic changes

There will be a new API field in the OwnerReference in which we will store an information if given owner is a managing controller:

OwnerReference {
    …
    Controller bool
    …
}

From now on by ControllerRef we mean an OwnerReference with Controller=true.

Most controllers (all that manage collections of things defined by label selector) will have slightly changed semantics: currently controller owns an object if its selector matches object’s labels and if it doesn't notice an older controller of the same kind that also matches the object's labels, but after introduction of ControllerReference a controller will own an object iff selector matches labels and the OwnerReference with Controller=truepoints to it.

If the owner's selector or owned object's labels change, the owning controller will be responsible for orphaning (clearing Controller field in the OwnerReference and/or deleting OwnerReference altogether) objects, after which adoption procedure (setting Controller field in one of OwnerReferencec and/or adding new OwnerReferences) might occur, if another controller has a selector matching.

For debugging purposes we want to add an adoptionTime annotation prefixed with kubernetes.io/ which will keep the time of last controller ownership transfer.

Upgrade/downgrade procedure

Because ControllerRef will be a part of OwnerReference effort it will have the same upgrade/downgrade procedures.

Orphaning/adoption

Because ControllerRef will be a part of OwnerReference effort it will have the same orphaning/adoption procedures.

Controllers will orphan objects they own in two cases:

• Change of label/selector causing selector to stop matching labels (executed by the controller)
• Deletion of a controller with Orphaning=true (executed by the GarbageCollector)

We will need a secondary orphaning mechanism in case of unclean controller deletion:

• GarbageCollector will remove ControllerRef from objects that no longer points to existing controllers

Controller will adopt (set Controller field in the OwnerReference that points to it) an object whose labels match its selector iff:

• there are no OwnerReferences with Controller set to true in OwnerReferences array
• DeletionTimestamp is not set and
• Controller is the first controller that will manage to adopt the Pod from all Controllers that have matching label selector and don't have DeletionTimestamp set.

By design there are possible races during adoption if multiple controllers can own a given object.

To prevent re-adoption of an object during deletion the DeletionTimestamp will be set when deletion is starting. When a controller has a non-nil DeletionTimestamp it won’t take any actions except updating its Status (in particular it won’t adopt any objects).

Implementation plan (sketch):

• Add API field for Controller,
• Extend OwnerReference adoption procedure to set a Controller field in one of the owners,
• Update all affected controllers to respect ControllerRef.

Necessary related work:

• OwnerReferences are correctly added/deleted,
• GarbageCollector removes dangling references,
• Controllers don't take any meaningful actions when DeletionTimestamps is set.

Considered alternatives

• Generic "ReferenceController": centralized component that managed adoption/orphaning
  • Dropped because: hard to write something that will work for all imaginable 3rd party objects, adding hooks to framework makes it possible for users to write their own logic
• Separate API field for ControllerRef in the ObjectMeta.
  • Dropped because: nontrivial relationship between ControllerRef and OwnerReferences when it comes to deletion/adoption.