From e6c4234bd2a622b62965e2d87fede158496c2722 Mon Sep 17 00:00:00 2001 From: Yifan Gu Date: Fri, 15 Jan 2016 17:02:35 -0800 Subject: [PATCH 1/6] docs: Add more info about the networking setup for rkt. --- docs/getting-started-guides/rkt/README.md | 57 +++++++++++++++++++++-- 1 file changed, 53 insertions(+), 4 deletions(-) diff --git a/docs/getting-started-guides/rkt/README.md b/docs/getting-started-guides/rkt/README.md index 0b508a8846c..2ff2e166294 100644 --- a/docs/getting-started-guides/rkt/README.md +++ b/docs/getting-started-guides/rkt/README.md @@ -47,8 +47,57 @@ We still have [a bunch of work](http://issue.k8s.io/8262) to do to make the expe - Note that for rkt version later than v0.7.0, `metadata service` is not required for running pods in private networks. So now rkt pods will not register the metadata service be default. -- Since release [v1.2.0-alpha.5](https://github.com/kubernetes/kubernetes/releases/tag/v1.2.0-alpha.5), [rkt API service](https://github.com/coreos/rkt/blob/master/api/v1alpha/README.md) - is required to be running on the machine. +- Since release [v1.2.0-alpha.5](https://github.com/kubernetes/kubernetes/releases/tag/v1.2.0-alpha.5), +[rkt API service](https://github.com/coreos/rkt/blob/master/api/v1alpha/README.md) is required to be running on the machine. + +### Network Setup + +rkt uses [CNI(Container Network Interface)](https://github.com/appc/cni) to manage the container networks. +Currently, all launched pods will try to join a network called `rkt.kubernetes.io` by default, which is defined [here](https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.6/pkg/kubelet/rkt/rkt.go#L91). +So in order for the pod to get the correct IP address, we need to setup the CNI config file for this `rkt.kubernetes.io` network correctly: + +- Using Flannel + +If [flannel](https://github.com/coreos/flannel) is used, then we need to create a flannel CNI config, for example: + +```console +$ cat </etc/rkt/net.d/k8s_cluster.conf +{ + "name": "rkt.kubernetes.io", + "type": "flannel" +} +EOF +``` + +Where `k8s_cluster.conf` is an arbitary name for the config file. +`name` in the config file must be `rkt.kubernetes.io`, and `type` should be `flannel`. +More details about flannel CNI plugin can be found [here](https://github.com/appc/cni/blob/master/Documentation/flannel.md) + +- On GCE + +As each VM on GCE can have extra 256 IP addresses that get routed to it, we don't have to use flannel for a Kubernetes cluster on GCE. +But we still to create a CNI config, for example: +```console +$ cat </etc/rkt/net.d/k8s_cluster.conf +{ + "name": "rkt.kubernetes.io", + "type": "bridge", + "bridge": "cbr0", + "isGateway": true, + "ipam": { + "type": "host-local", + "subnet": "10.255.228.1/24", + "gateway": "10.255.228.1" + } +} +EOF +``` +Here we created a `bridge` plugin config for the network, which specifies the bridge name `cbr0`. It also specifies the CIDR in the `ipam` field. + +As you can imagine, creating those files for a multi-node cluster is impractical. +Currently, we are working on Kuberenetes to use the CNI by default (see [#18795](https://github.com/kubernetes/kubernetes/pull/18795/files)). +After those work is done, we won't need to manually created such config files for rkt. +Besides, if you really want to try this out, [here](https://gist.github.com/yifan-gu/fbb911db83d785915543) is an example patch how you can create such files automatically. ### Local cluster @@ -86,13 +135,13 @@ $ export KUBE_CONTAINER_RUNTIME=rkt You can optionally choose the version of rkt used by setting `KUBE_RKT_VERSION`: ```console -$ export KUBE_RKT_VERSION=0.8.0 +$ export KUBE_RKT_VERSION=0.15.0 ``` Then you can launch the cluster by: ```console -$ kube-up.sh +$ cluster/kube-up.sh ``` Note that we are still working on making all containerized the master components run smoothly in rkt. Before that we are not able to run the master node with rkt yet. From bb6bb9f18eac20f33ebc754ac3d3169d31ee694f Mon Sep 17 00:00:00 2001 From: Josh Wood Date: Sat, 16 Jan 2016 15:00:20 -0800 Subject: [PATCH 2/6] docs/getting-started/rkt/: Add and tune rkt network setup info Add info about rkt CNI networking setup to README. Supersedes https://github.com/kubernetes/kubernetes/pull/19745. --- docs/getting-started-guides/rkt/README.md | 50 +++++++++++++++-------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/docs/getting-started-guides/rkt/README.md b/docs/getting-started-guides/rkt/README.md index 2ff2e166294..663897fd11d 100644 --- a/docs/getting-started-guides/rkt/README.md +++ b/docs/getting-started-guides/rkt/README.md @@ -48,17 +48,22 @@ We still have [a bunch of work](http://issue.k8s.io/8262) to do to make the expe - Note that for rkt version later than v0.7.0, `metadata service` is not required for running pods in private networks. So now rkt pods will not register the metadata service be default. - Since release [v1.2.0-alpha.5](https://github.com/kubernetes/kubernetes/releases/tag/v1.2.0-alpha.5), -[rkt API service](https://github.com/coreos/rkt/blob/master/api/v1alpha/README.md) is required to be running on the machine. +the [rkt API service](https://github.com/coreos/rkt/blob/master/api/v1alpha/README.md) +must be running on the node. ### Network Setup -rkt uses [CNI(Container Network Interface)](https://github.com/appc/cni) to manage the container networks. -Currently, all launched pods will try to join a network called `rkt.kubernetes.io` by default, which is defined [here](https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.6/pkg/kubelet/rkt/rkt.go#L91). -So in order for the pod to get the correct IP address, we need to setup the CNI config file for this `rkt.kubernetes.io` network correctly: +rkt uses the [Container Network Interface (CNI)](https://github.com/appc/cni) +to manage container networking. By default, all pods attempt to join a network +called `rkt.kubernetes.io`, which is currently defined [in `rkt.go`] +(https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.6/pkg/kubelet/rkt/rkt.go#L91). +In order for pods to get correct IP addresses, the CNI config file must be +edited to add this `rkt.kubernetes.io` network: -- Using Flannel +#### Using flannel -If [flannel](https://github.com/coreos/flannel) is used, then we need to create a flannel CNI config, for example: +[flannel](https://github.com/coreos/flannel) can be configured with a +CNI config like: ```console $ cat </etc/rkt/net.d/k8s_cluster.conf @@ -69,14 +74,18 @@ $ cat </etc/rkt/net.d/k8s_cluster.conf EOF ``` -Where `k8s_cluster.conf` is an arbitary name for the config file. -`name` in the config file must be `rkt.kubernetes.io`, and `type` should be `flannel`. -More details about flannel CNI plugin can be found [here](https://github.com/appc/cni/blob/master/Documentation/flannel.md) +While `k8s_cluster.conf` is a rather arbitrary name for the config file itself, +and can be adjusted to suit local conventions, the keys and values should be exactly +as shown above. `name` must be `rkt.kubernetes.io` and `type` should be `flannel`. +More details about the flannel CNI plugin can be found +[in the CNI documentation](https://github.com/appc/cni/blob/master/Documentation/flannel.md). -- On GCE +#### On GCE + +Each VM on GCE can have up to 256 IP addresses routed to it, so flannel isn't called +for in most smaller Kubernetes clusters on GCE. This makes the necessary CNI config +file a bit more verbose: -As each VM on GCE can have extra 256 IP addresses that get routed to it, we don't have to use flannel for a Kubernetes cluster on GCE. -But we still to create a CNI config, for example: ```console $ cat </etc/rkt/net.d/k8s_cluster.conf { @@ -92,12 +101,19 @@ $ cat </etc/rkt/net.d/k8s_cluster.conf } EOF ``` -Here we created a `bridge` plugin config for the network, which specifies the bridge name `cbr0`. It also specifies the CIDR in the `ipam` field. -As you can imagine, creating those files for a multi-node cluster is impractical. -Currently, we are working on Kuberenetes to use the CNI by default (see [#18795](https://github.com/kubernetes/kubernetes/pull/18795/files)). -After those work is done, we won't need to manually created such config files for rkt. -Besides, if you really want to try this out, [here](https://gist.github.com/yifan-gu/fbb911db83d785915543) is an example patch how you can create such files automatically. +This example creates a `bridge` plugin configuration for the CNI network, specifying +the bridge name `cbr0`. It also specifies the CIDR, in the `ipam` field. + +Creating these files for any moderately-sized cluster is at best inconvenient. +Work is in progress to +[enable Kubernetes to use the CNI by default] +(https://github.com/kubernetes/kubernetes/pull/18795/files). +As that work matures, such manual CNI config munging will become unneccessary for +primary use cases. For early adopters, an initial example shows one way to +[automatically generate these CNI configurations] +(https://gist.github.com/yifan-gu/fbb911db83d785915543) +for rkt. ### Local cluster From 75fcbbf2c6b29bb76ccbfa43247f91e05185bb51 Mon Sep 17 00:00:00 2001 From: Josh Wood Date: Wed, 20 Jan 2016 02:06:54 -0800 Subject: [PATCH 3/6] docs/getting-started/rkt: Improve sans flannel example --- docs/getting-started-guides/rkt/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting-started-guides/rkt/README.md b/docs/getting-started-guides/rkt/README.md index 663897fd11d..e3dbbff368b 100644 --- a/docs/getting-started-guides/rkt/README.md +++ b/docs/getting-started-guides/rkt/README.md @@ -82,9 +82,9 @@ More details about the flannel CNI plugin can be found #### On GCE -Each VM on GCE can have up to 256 IP addresses routed to it, so flannel isn't called -for in most smaller Kubernetes clusters on GCE. This makes the necessary CNI config -file a bit more verbose: +Each VM on GCE has an additional 256 IP addresses routed to it, so +it is possible to forego flannel in smaller clusters. This makes the +necessary CNI config file a bit more verbose: ```console $ cat </etc/rkt/net.d/k8s_cluster.conf From d6d2f6f203525a4c4a0919cffc71415b83878d93 Mon Sep 17 00:00:00 2001 From: Josh Wood Date: Wed, 20 Jan 2016 02:10:40 -0800 Subject: [PATCH 4/6] docs/getting-started/rkt: Correct spelling error --- docs/getting-started-guides/rkt/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/getting-started-guides/rkt/README.md b/docs/getting-started-guides/rkt/README.md index e3dbbff368b..2a0f499b305 100644 --- a/docs/getting-started-guides/rkt/README.md +++ b/docs/getting-started-guides/rkt/README.md @@ -109,8 +109,8 @@ Creating these files for any moderately-sized cluster is at best inconvenient. Work is in progress to [enable Kubernetes to use the CNI by default] (https://github.com/kubernetes/kubernetes/pull/18795/files). -As that work matures, such manual CNI config munging will become unneccessary for -primary use cases. For early adopters, an initial example shows one way to +As that work matures, such manual CNI config munging will become unnecessary +for primary use cases. For early adopters, an initial example shows one way to [automatically generate these CNI configurations] (https://gist.github.com/yifan-gu/fbb911db83d785915543) for rkt. From 54a416fcc1b9c28d337ef0d00680947f0c99bc6a Mon Sep 17 00:00:00 2001 From: Josh Wood Date: Wed, 20 Jan 2016 17:25:50 -0800 Subject: [PATCH 5/6] docs/getting-started/rkt: Add default route to k8s_cluster.conf example --- docs/getting-started-guides/rkt/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/getting-started-guides/rkt/README.md b/docs/getting-started-guides/rkt/README.md index 2a0f499b305..c96316d582c 100644 --- a/docs/getting-started-guides/rkt/README.md +++ b/docs/getting-started-guides/rkt/README.md @@ -97,7 +97,10 @@ $ cat </etc/rkt/net.d/k8s_cluster.conf "type": "host-local", "subnet": "10.255.228.1/24", "gateway": "10.255.228.1" - } + }, + "routes": [ + { "dst": "0.0.0.0/0" } + ] } EOF ``` From 3d836868570adb7cd608efe689e35aa5a60c27f0 Mon Sep 17 00:00:00 2001 From: Josh Wood Date: Wed, 20 Jan 2016 23:00:06 -0800 Subject: [PATCH 6/6] docs/getting-started/rkt: Clarify flannel prerequisites: need flannel + another etcd --- docs/getting-started-guides/rkt/README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/getting-started-guides/rkt/README.md b/docs/getting-started-guides/rkt/README.md index c96316d582c..197e74b414b 100644 --- a/docs/getting-started-guides/rkt/README.md +++ b/docs/getting-started-guides/rkt/README.md @@ -62,8 +62,12 @@ edited to add this `rkt.kubernetes.io` network: #### Using flannel -[flannel](https://github.com/coreos/flannel) can be configured with a -CNI config like: +In addition to the basic prerequisites above, each node must be running +a [flannel](https://github.com/coreos/flannel) daemon. This implies +that a flannel-supporting etcd service must be available to the cluster +as well, apart from the Kubernetes etcd, which will not yet be +available at flannel configuration time. Once it's running, flannel can +be set up with a CNI config like: ```console $ cat </etc/rkt/net.d/k8s_cluster.conf