From 7d824ba47f779ea9924f6a90da0f160f58a911bd Mon Sep 17 00:00:00 2001 From: Yifan Gu Date: Wed, 28 Sep 2016 17:01:52 -0700 Subject: [PATCH 1/3] Proposal: Add a rktlet proposal in upstream. Mostly copied from docs/proposals/kubelet-rkt-runtime.md, plus roadmaps. --- docs/proposals/kubelet-rkt-runtime.md | 127 ++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 docs/proposals/kubelet-rkt-runtime.md diff --git a/docs/proposals/kubelet-rkt-runtime.md b/docs/proposals/kubelet-rkt-runtime.md new file mode 100644 index 00000000000..2779e4f519f --- /dev/null +++ b/docs/proposals/kubelet-rkt-runtime.md @@ -0,0 +1,127 @@ + + + + +WARNING +WARNING +WARNING +WARNING +WARNING + +

PLEASE NOTE: This document applies to the HEAD of the source tree

+ +If you are using a released version of Kubernetes, you should +refer to the docs that go with that version. + +Documentation for other releases can be found at +[releases.k8s.io](http://releases.k8s.io). + +-- + + + + + +Next generation rkt runtime integration +======================================= + +Authors: Euan Kemp (@euank), Yifan Gu (@yifan-gu) + +## Abstract + +This proposal describes the design and road path for integrating rkt with kubelet with the new container runtime interface. + +## Background + +Currently, the Kubernetes project supports rkt as a container runtime via an implementation under [pkg/kubelet/rkt package](https://github.com/kubernetes/kubernetes/tree/v1.5.0-alpha.0/pkg/kubelet/rkt). + +This implementation, for historical reasons, has required implementing a large amount of logic shared by the original Docker implementation. + +In order to make additional container runtime integrations easier, more clearly defined, and more consistent, a new [Container Runtime Interface](https://github.com/kubernetes/kubernetes/blob/v1.5.0-alpha.0/pkg/kubelet/api/v1alpha1/runtime/api.proto) (CRI) is being designed. +The existing runtimes, in order to both prove the correctness of the interface and reduce maintenance burden, are incentivized to move to this interface. + +This document proposes how the rkt runtime integration will transition to using the CRI. + +## Goals + +### Full-featured + +The CRI integration must work as well as the existing integration in terms of features. + +Until that's the case, the existing integration will continue to be maintained. + +### Easy to Deploy + +The new integration should not be any more difficult to deploy and configure than the existing integration. + +### Easy to Develop + +This iteration should be as easy to work and iterate on as the original one. + +It will be available in an initial usable form quickly in order to validate the CRI. + +## Design + +In order to fulfill the above goals, the rkt CRI integration will make the following choices: + +### Remain in-process with Kubelet + +The current rkt container runtime integration is able to be deployed simply by deploying the kubelet binary. + +Similarly, the Docker integration (as visible in the [dockershim](https://github.com/kubernetes/kubernetes/blob/v1.5.0-alpha.0/pkg/kubelet/api/v1alpha1/runtime/api.proto) package) is making the choice to remain there. + +This is, in no small part, to make it *Easy to Deploy*. + +Remaining in-process also helps this integration not regress on performance, one axis of being *Full-Featured*. + +### Developed as a Separate Repository + +Brian Grant's discussion on splitting the Kubernetes project into [separate repos](https://github.com/kubernetes/kubernetes/issues/24343) is a compelling argument for why it makes sense to split this work into a separate repo. + +In order to be *Easy to Develop*, this iteration will be maintained as a separate repository, and re-vendored back in. + +This choice will also allow better long-term growth in terms of better issue-management, testing pipelines, and so on. + +Unfortunately, in the short term, it's possible that some aspects of this will also cause pain and it's very difficult to weight each side correctly. + +### Exec the rkt binary (initially) + +While significant work on the rkt [api-service](https://coreos.com/rkt/docs/latest/subcommands/api-service.html) has been made, +it has also been a source of problems and additional complexity, +and was never transitioned to entirely. + +In addition, the rkt cli has historically been the primary interface to the rkt runtime. + +The initial integration will execute the rkt binary directly for app creation/start/stop/removal, as well as image pulling/removal. + +The creation of pod sanbox is also done via rkt command line, but it will run under `systemd-run` so it's monitored by the init process. + +In the future, some of these decisions are expected to be changed such that rkt is vendored as a library dependency for all operations, and other init systems will be supported as well. + + +## Roadmap and Milestones + +1. rktlet integrate with kubelet to support basic pod/container lifecycle (pod creation, container creation/start/stop, pod stop/removal) [[Done]](https://github.com/kubernetes-incubator/rktlet/issues/9) +2. rktlet integrate with kubelet to support more advanced features: + - Support kubelet networking, host network + - Support mount / volumes [[#33526]](https://github.com/kubernetes/kubernetes/issues/33526) + - Support exposing ports + - Support privileged containers + - Support selinux options [[#33139]](https://github.com/kubernetes/kubernetes/issues/33139) + - Support attach [[#29579]](https://github.com/kubernetes/kubernetes/issues/29579) + - Support exec [[#29579]](https://github.com/kubernetes/kubernetes/issues/29579) + - Support logging [[#33111]](https://github.com/kubernetes/kubernetes/pull/33111) + +3. rktlet integrate with kubelet, pass 100% e2e and node e2e tests, with nspawn stage1. +4. rktlet integrate with kubelet, pass 100% e2e and node e2e tests, with kvm stage1. +5. Revendor rktlet into `pkg/kubelet/rkt` to replace the current rkt. + + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/kubelet-rkt-runtime.md?pixel)]() + From 2d76b9f240c7fbbcc8fbd8bac494351bab1e8dc8 Mon Sep 17 00:00:00 2001 From: Yifan Gu Date: Thu, 29 Sep 2016 19:50:20 -0700 Subject: [PATCH 2/3] Remove reference to dockershim. --- docs/proposals/kubelet-rkt-runtime.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/proposals/kubelet-rkt-runtime.md b/docs/proposals/kubelet-rkt-runtime.md index 2779e4f519f..e5a7bdc98c8 100644 --- a/docs/proposals/kubelet-rkt-runtime.md +++ b/docs/proposals/kubelet-rkt-runtime.md @@ -73,8 +73,6 @@ In order to fulfill the above goals, the rkt CRI integration will make the follo The current rkt container runtime integration is able to be deployed simply by deploying the kubelet binary. -Similarly, the Docker integration (as visible in the [dockershim](https://github.com/kubernetes/kubernetes/blob/v1.5.0-alpha.0/pkg/kubelet/api/v1alpha1/runtime/api.proto) package) is making the choice to remain there. - This is, in no small part, to make it *Easy to Deploy*. Remaining in-process also helps this integration not regress on performance, one axis of being *Full-Featured*. From f8682ac1bb9302775f24a1bb1929e02b0ee33cd7 Mon Sep 17 00:00:00 2001 From: Yifan Gu Date: Mon, 10 Oct 2016 19:07:54 -0700 Subject: [PATCH 3/3] More clarification on the in-process integration. --- docs/proposals/kubelet-rkt-runtime.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/proposals/kubelet-rkt-runtime.md b/docs/proposals/kubelet-rkt-runtime.md index e5a7bdc98c8..850b5a18e5c 100644 --- a/docs/proposals/kubelet-rkt-runtime.md +++ b/docs/proposals/kubelet-rkt-runtime.md @@ -77,6 +77,12 @@ This is, in no small part, to make it *Easy to Deploy*. Remaining in-process also helps this integration not regress on performance, one axis of being *Full-Featured*. +### Communicate through gRPC + +Although the kubelet and rktlet will be compiled together, the runtime and kubelet will still communicate through gRPC interface for better API abstraction. + +For the near short term, they will still talk through a unix socket before we implement a custom gRPC connection that skips the network stack. + ### Developed as a Separate Repository Brian Grant's discussion on splitting the Kubernetes project into [separate repos](https://github.com/kubernetes/kubernetes/issues/24343) is a compelling argument for why it makes sense to split this work into a separate repo. @@ -117,7 +123,8 @@ In the future, some of these decisions are expected to be changed such that rkt 3. rktlet integrate with kubelet, pass 100% e2e and node e2e tests, with nspawn stage1. 4. rktlet integrate with kubelet, pass 100% e2e and node e2e tests, with kvm stage1. -5. Revendor rktlet into `pkg/kubelet/rkt` to replace the current rkt. +5. Revendor rktlet into `pkg/kubelet/rktshim`, and start deprecating the `pkg/kubelet/rkt` package. +6. Eventually replace the current `pkg/kubelet/rkt` package.