From 87f2590f373306ea34e5f147d8beb5172256fdf5 Mon Sep 17 00:00:00 2001 From: David Oppenheimer Date: Thu, 17 Dec 2015 23:00:11 -0800 Subject: [PATCH] How to build Mesos/Omega-style frameworks on Kubernetes. --- docs/devel/mesos-style.md | 169 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 docs/devel/mesos-style.md diff --git a/docs/devel/mesos-style.md b/docs/devel/mesos-style.md new file mode 100644 index 00000000000..c8d096be14e --- /dev/null +++ b/docs/devel/mesos-style.md @@ -0,0 +1,169 @@ + + + + +WARNING +WARNING +WARNING +WARNING +WARNING + +

PLEASE NOTE: This document applies to the HEAD of the source tree

+ +If you are using a released version of Kubernetes, you should +refer to the docs that go with that version. + +Documentation for other releases can be found at +[releases.k8s.io](http://releases.k8s.io). + +-- + + + + + +# Building Mesos/Omega-style frameworks on Kubernetes + +## Introduction + +We have observed two different cluster management architectures, which can be categorized as "Borg-style" and "Mesos/Omega-style." +(In the remainder of this document, we will abbreviate the latter as "Mesos-style.") +Although out-of-the box Kubernetes uses a Borg-style architecture, it can also be configured in a Mesos-style architecture, +and in fact can support both styles at the same time. This document describes the two approaches and describes how +to deploy a Mesos-style architecture on Kubernetes. + +(As an aside, the converse is also true: one can deploy a Borg/Kubernetes-style architecture on Mesos.) + +This document is NOT intended to provide a comprehensive comparison of Borg and Mesos. For example, we omit discussion +of the tradeoffs between scheduling with full knowledge of cluster state vs. scheduling using the "offer" model. +(That issue is discussed in some detail in the Omega paper (see references section at the end of this doc).) + + +## What is a Borg-style architecture? + +A Borg-style architecture is characterized by: +* a single logical API endpoint for clients, where some amount of processing is done on requests, such as admission control and applying defaults +* generic (non-application-specific) collection abstractions described declaratively, +* generic controllers/state machines that manage the lifecycle of the collection abstractions and the containers spawned from them +* a generic scheduler + +For example, Borg's primary collection abstraction is a Job, and every application that runs on Borg--whether it's a user-facing +service like the GMail front-end, a batch job like a MapReduce, or an infrastructure service like GFS--must represent itself as +a Job. Borg has corresponding state machine logic for managing Jobs and their instances, and a scheduler that's responsible +for assigning the instances to machines. + +The flow of a request in Borg is: + +1. Client submits a collection object to the Borgmaster API endpoint +1. Admission control, quota, applying defaults, etc. run on the collection +1. If the collection is admitted, it is persisted, and the collection state machine creates the underlying instances +1. The scheduler assigns a hostname to the instance, and tells the Borglet to start the instance's container(s) +1. Borglet starts the container(s) +1. The instance state machine manages the instances and the collection state machine manages the collection during their lifetimes + +Out-of-the-box Kubernetes has *workload-specific* abstractions (ReplicaSet, Job, DaemonSet, etc.) and corresponding controllers, +and in the future may have [workload-specific schedulers](../../docs/proposals/multiple-schedulers.md), +e.g. different schedulers for long-running services vs. short-running batch. But these abstractions, controllers, and +schedulers are not *application-specific*. + +The usual request flow in Kubernetes is very similar, namely + +1. Client submits a collection object (e.g. ReplicaSet, Job, ...) to the API server +1. Admission control, quota, applying defaults, etc. run on the collection +1. If the collection is admitted, it is persisted, and the corresponding collection controller creates the underlying pods +1. Admission control, quota, applying defaults, etc. runs on each pod; if there are multiple schedulers, one of the admission +controllers will write the scheduler name as an annotation based on a policy +1. If a pod is admitted, it is persisted +1. The appropriate scheduler assigns a nodeName to the instance, which triggers the Kubelet to start the pod's container(s) +1. Kubelet starts the container(s) +1. The controller corresponding to the collection manages the pod and the collection during their lifetime + +In the Borg model, application-level scheduling and cluster-level scheduling are handled by separate +components. For example, a MapReduce master might request Borg to create a job with a certain number of instances +with a particular resource shape, where each instance corresponds to a MapReduce worker; the MapReduce master would +then schedule individual units of work onto those workers. + +## What is a Mesos-style architecture? + +Mesos is fundamentally designed to support multiple application-specific "frameworks." A framework is +composed of a "framework scheduler" and a "framework executor." We will abbreviate "framework scheduler" +as "framework" since "scheduler" means something very different in Kubernetes (something that just +assigns pods to nodes). + +Unlike Borg and Kubernetes, where there is a single logical endpoint that receives all API requests (the Borgmaster and API server, +respectively), in Mesos every framework is a separate API endpoint. Mesos does not have any standard set of +collection abstractions, controllers/state machines, or schedulers; the logic for all of these things is contained +in each [application-specific framework](http://mesos.apache.org/documentation/latest/frameworks/) individually. +(Note that the notion of application-specific does sometimes blur into the realm of workload-specific, +for example [Chronos](https://github.com/mesos/chronos) is a generic framework for batch jobs. +However, regardless of what set of Mesos frameworks you are using, the key properties remain: each +framework is its own API endpoint with its own client-facing and internal abstractions, state machines, and scheduler). + +A Mesos framework can integrate application-level scheduling and cluster-level scheduling into a single component. + +Note: Although Mesos frameworks expose their own API endpoints to clients, they consume a common +infrastructure via a common API endpoint for controlling tasks (launching, detecting failure, etc.) and learning about available +cluster resources. More details [here](http://mesos.apache.org/documentation/latest/scheduler-http-api/). + +## Building a Mesos-style framework on Kubernetes + +Implementing the Mesos model on Kubernetes boils down to enabling application-specific collection abstractions, +controllers/state machines, and scheduling. There are just three steps: +* Use API plugins to create API resources for your new application-specific collection abstraction(s) +* Implement controllers for the new abstractions (and for managing the lifecycle of the pods the controllers generate) +* Implement a scheduler with the application-specific scheduling logic + +Note that the last two can be combined: a Kubernetes controller can do the scheduling for the pods it creates, +by writing node name to the pods when it creates them. + +Once you've done this, you end up with an architecture that is extremely similar to the Mesos-style--the +Kubernetes controller is effectively a Mesos framework. The remaining differences are +* In Kubernetes, all API operations go through a single logical endpoint, the API server (we say logical because the API server can be replicated). +In contrast, in Mesos, API operations go to a particular framework. However, the Kubernetes API plugin model makes this difference fairly small. +* In Kubernetes, application-specific admission control, quota, defaulting, etc. rules can be implemented +in the API server rather than in the controller. Of course you can choose to make these operations be no-ops for +your application-specific collection abstractions, and handle them in your controller. +* On the node level, Mesos allows application-specific executors, whereas Kubernetes only has +executors for Docker and Rocket containers. + +The end-to-end flow is + +1. Client submits an application-specific collection object to the API server +2. The API server plugin for that collection object forwards the request to the API server that handles that collection type +3. Admission control, quota, applying defaults, etc. runs on the collection object +4. If the collection is admitted, it is persisted +5. The collection controller sees the collection object and in response creates the underlying pods and chooses which nodes they will run on by setting node name +6. Kubelet sees the pods with node name set and starts the container(s) +7. The collection controller manages the pods and the collection during their lifetimes + +(note that if the controller and scheduler are separated, then step 5 breaks down into multiple steps: +(5a) collection controller creates pods with empty node name. (5b) API server admission control, quota, defaulting, +etc. runs on the pods; one of the admission controller steps writes the scheduler name as an annotation on each pods +(see #18262 for more details). +(5c) The corresponding application-specific scheduler chooses a node and writes node name, which triggers the Kubelet to start the pod's container(s).) + +As a final note, the Kubernetes model allows multiple levels of iterative refinement of runtime abstractions, +as long as the lowest level is the pod. For example, clients of application Foo might create a `FooSet` +which is picked up by the FooController which in turn creates `BatchFooSet` and `ServiceFooSet` objects, +which are picked up by the BatchFoo controller and ServiceFoo controller respectively, which in turn +create pods. In between each of these steps there is an opportunity for object-specific admission control, +quota, and defaulting to run in the API server, though these can instead be handled by the controllers. + + +## References + +Mesos is described [here](https://www.usenix.org/legacy/event/nsdi11/tech/full_papers/Hindman_new.pdf). +Omega is described [here](http://research.google.com/pubs/pub41684.html). +Borg is described [here](http://research.google.com/pubs/pub43438.html). + + + + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/mesos-style.md?pixel)]() +