diff --git a/examples/examples_test.go b/examples/examples_test.go index 79131a8fe32..6dfed34ca82 100644 --- a/examples/examples_test.go +++ b/examples/examples_test.go @@ -241,7 +241,7 @@ func TestExampleObjectSchemas(t *testing.T) { "pod": &api.Pod{}, }, "../examples/storage/hazelcast": { - "hazelcast-controller": &api.ReplicationController{}, + "hazelcast-deployment": &extensions.Deployment{}, "hazelcast-service": &api.Service{}, }, "../examples/meteor": { diff --git a/examples/storage/hazelcast/README.md b/examples/storage/hazelcast/README.md index d92f2124537..7f9325512f5 100644 --- a/examples/storage/hazelcast/README.md +++ b/examples/storage/hazelcast/README.md @@ -4,7 +4,7 @@ The following document describes the development of a _cloud native_ [Hazelcast] Any topology changes are communicated and handled by Hazelcast nodes themselves. -This document also attempts to describe the core components of Kubernetes: _Pods_, _Services_, and _Replication Controllers_. +This document also attempts to describe the core components of Kubernetes: _Pods_, _Services_, and _Deployments_. ### Prerequisites @@ -30,7 +30,7 @@ In this case, we shall not run a single Hazelcast pod, because the discovery mec ### Adding a Hazelcast Service -In Kubernetes a _[Service](../../../docs/user-guide/services.md)_ describes a set of Pods that perform the same task. For example, the set of nodes in a Hazelcast cluster. An important use for a Service is to create a load balancer which distributes traffic across members of the set. But a _Service_ can also be used as a standing query which makes a dynamically changing set of Pods available via the Kubernetes API. This is actually how our discovery mechanism works, by relying on the service to discover other Hazelcast pods. +In Kubernetes a _[Service](../../../docs/user-guide/services.md)_ describes a set of Pods that perform the same task. For example, the set of nodes in a Hazelcast cluster. An important use for a Service is to create a load balancer which distributes traffic across members of the set. But a _Service_ can also be used as a standing query which makes a dynamically changing set of Pods available via the Kubernetes API. This is actually how our discovery mechanism works, by relying on the service to discover other Hazelcast pods. Here is the service description: @@ -65,52 +65,41 @@ $ kubectl create -f examples/storage/hazelcast/hazelcast-service.yaml The real power of Kubernetes and Hazelcast lies in easily building a replicated, resizable Hazelcast cluster. -In Kubernetes a _[Replication Controller](../../../docs/user-guide/replication-controller.md)_ is responsible for replicating sets of identical pods. Like a _Service_ it has a selector query which identifies the members of it's set. Unlike a _Service_ it also has a desired number of replicas, and it will create or delete _Pods_ to ensure that the number of _Pods_ matches up with it's desired state. +In Kubernetes a _[_Deployment_](../../../docs/user-guide/deployments.md)_ is responsible for replicating sets of identical pods. Like a _Service_ it has a selector query which identifies the members of its set. Unlike a _Service_ it also has a desired number of replicas, and it will create or delete _Pods_ to ensure that the number of _Pods_ matches up with its desired state. -Replication Controllers will "adopt" existing pods that match their selector query, so let's create a Replication Controller with a single replica to adopt our existing Hazelcast Pod. +Deployments will "adopt" existing pods that match their selector query, so let's create a Deployment with a single replica to adopt our existing Hazelcast Pod. ```yaml -apiVersion: v1 -kind: ReplicationController +apiVersion: extensions/v1beta1 +kind: Deployment metadata: + name: hazelcast labels: name: hazelcast - name: hazelcast spec: - replicas: 1 - selector: - name: hazelcast template: metadata: labels: name: hazelcast spec: containers: - - resources: - limits: - cpu: 0.1 - image: quay.io/pires/hazelcast-kubernetes:0.6.1 - name: hazelcast - env: - - name: "DNS_DOMAIN" - value: "cluster.local" - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - ports: - - containerPort: 5701 - name: hazelcast + - name: hazelcast + image: quay.io/pires/hazelcast-kubernetes:0.8.0 + imagePullPolicy: Always + env: + - name: "DNS_DOMAIN" + value: "cluster.local" + ports: + - name: hazelcast + containerPort: 5701 ``` -[Download example](hazelcast-controller.yaml?raw=true) +[Download example](hazelcast-deployment.yaml?raw=true) -There are a few things to note in this description. First is that we are running the `quay.io/pires/hazelcast-kubernetes` image, tag `0.5`. This is a `busybox` installation with JRE 8 Update 45. However it also adds a custom [`application`](https://github.com/pires/hazelcast-kubernetes-bootstrapper) that finds any Hazelcast nodes in the cluster and bootstraps an Hazelcast instance accordingly. The `HazelcastDiscoveryController` discovers the Kubernetes API Server using the built in Kubernetes discovery service, and then uses the Kubernetes API to find new nodes (more on this later). - -You may also note that we tell Kubernetes that the container exposes the `hazelcast` port. Finally, we tell the cluster manager that we need 1 cpu core. +You may note that we tell Kubernetes that the container exposes the `hazelcast` port. The bulk of the replication controller config is actually identical to the Hazelcast pod declaration above, it simply gives the controller a recipe to use when creating new pods. The other parts are the `selector` which contains the controller's selector query, and the `replicas` parameter which specifies the desired number of replicas, in this case 1. @@ -119,127 +108,124 @@ Last but not least, we set `DNS_DOMAIN` environment variable according to your K Create this controller: ```sh -$ kubectl create -f examples/storage/hazelcast/hazelcast-controller.yaml +$ kubectl create -f examples/storage/hazelcast/hazelcast-deployment.yaml ``` After the controller provisions successfully the pod, you can query the service endpoints: - ```sh -$ kubectl get endpoints hazelcast -o json -{ - "kind": "Endpoints", - "apiVersion": "v1", - "metadata": { - "name": "hazelcast", - "namespace": "default", - "selfLink": "/api/v1/namespaces/default/endpoints/hazelcast", - "uid": "094e507a-2700-11e5-abbc-080027eae546", - "resourceVersion": "4094", - "creationTimestamp": "2015-07-10T12:34:41Z", - "labels": { - "name": "hazelcast" - } - }, - "subsets": [ - { - "addresses": [ - { - "ip": "10.244.37.3", - "targetRef": { - "kind": "Pod", - "namespace": "default", - "name": "hazelcast-nsyzn", - "uid": "f57eb6b0-2706-11e5-abbc-080027eae546", - "resourceVersion": "4093" - } - } - ], - "ports": [ - { - "port": 5701, - "protocol": "TCP" - } - ] - } - ] -} +$ kubectl get endpoints hazelcast -o yaml +apiVersion: v1 +kind: Endpoints +metadata: + creationTimestamp: 2016-12-16T08:57:27Z + labels: + name: hazelcast + name: hazelcast + namespace: default + resourceVersion: "11360" + selfLink: /api/v1/namespaces/default/endpoints/hazelcast + uid: 46447198-70eb-11e6-940c-0800278ab84d +subsets: +- addresses: + - ip: 10.244.37.2 + targetRef: + kind: Pod + name: hazelcast-1790698550-3heau + namespace: default + resourceVersion: "11359" + uid: c9c3febd-70eb-11e6-940c-0800278ab84d + ports: + - port: 5701 + protocol: TCP ``` You can see that the _Service_ has found the pod created by the replication controller. -Now it gets even more interesting. - -Let's scale our cluster to 2 pods: - +Now it gets even more interesting. Let's scale our cluster to 2 pods: ```sh -$ kubectl scale rc hazelcast --replicas=2 +$ kubectl scale deployment hazelcast --replicas 2 ``` Now if you list the pods in your cluster, you should see two hazelcast pods: ```sh -$ kubectl get pods -NAME READY STATUS RESTARTS AGE -hazelcast-nanfb 1/1 Running 0 40s -hazelcast-nsyzn 1/1 Running 0 2m -kube-dns-xudrp 3/3 Running 0 1h +$ kubectl get deployment,pods +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE +deploy/hazelcast 2 2 2 2 1m + +NAME READY STATUS RESTARTS AGE +po/hazelcast-3980717115-k1xsk 1/1 Running 0 1m +po/hazelcast-3980717115-pbhbq 1/1 Running 0 22s ``` To prove that this all works, you can use the `log` command to examine the logs of one pod, for example: ```sh -$ kubectl log hazelcast-nanfb hazelcast -2015-07-10 13:26:34.443 INFO 5 --- [ main] com.github.pires.hazelcast.Application : Starting Application on hazelcast-nanfb with PID 5 (/bootstrapper.jar started by root in /) -2015-07-10 13:26:34.535 INFO 5 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@42cfcf1: startup date [Fri Jul 10 13:26:34 GMT 2015]; root of context hierarchy -2015-07-10 13:26:35.888 INFO 5 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup -2015-07-10 13:26:35.924 INFO 5 --- [ main] c.g.p.h.HazelcastDiscoveryController : Asking k8s registry at https://kubernetes.default.svc.cluster.local.. -2015-07-10 13:26:37.259 INFO 5 --- [ main] c.g.p.h.HazelcastDiscoveryController : Found 2 pods running Hazelcast. -2015-07-10 13:26:37.404 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.5] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.244.77.3, 10.244.37.3] -2015-07-10 13:26:37.405 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.5] Prefer IPv4 stack is true. -2015-07-10 13:26:37.415 INFO 5 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.5] Picked Address[10.244.77.3]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true -2015-07-10 13:26:37.852 INFO 5 --- [ main] com.hazelcast.spi.OperationService : [10.244.77.3]:5701 [someGroup] [3.5] Backpressure is disabled -2015-07-10 13:26:37.879 INFO 5 --- [ main] c.h.s.i.o.c.ClassicOperationExecutor : [10.244.77.3]:5701 [someGroup] [3.5] Starting with 2 generic operation threads and 2 partition operation threads. -2015-07-10 13:26:38.531 INFO 5 --- [ main] com.hazelcast.system : [10.244.77.3]:5701 [someGroup] [3.5] Hazelcast 3.5 (20150617 - 4270dc6) starting at Address[10.244.77.3]:5701 -2015-07-10 13:26:38.532 INFO 5 --- [ main] com.hazelcast.system : [10.244.77.3]:5701 [someGroup] [3.5] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved. -2015-07-10 13:26:38.533 INFO 5 --- [ main] com.hazelcast.instance.Node : [10.244.77.3]:5701 [someGroup] [3.5] Creating TcpIpJoiner -2015-07-10 13:26:38.534 INFO 5 --- [ main] com.hazelcast.core.LifecycleService : [10.244.77.3]:5701 [someGroup] [3.5] Address[10.244.77.3]:5701 is STARTING -2015-07-10 13:26:38.672 INFO 5 --- [ cached1] com.hazelcast.nio.tcp.SocketConnector : [10.244.77.3]:5701 [someGroup] [3.5] Connecting to /10.244.37.3:5701, timeout: 0, bind-any: true -2015-07-10 13:26:38.683 INFO 5 --- [ cached1] c.h.nio.tcp.TcpIpConnectionManager : [10.244.77.3]:5701 [someGroup] [3.5] Established socket connection between /10.244.77.3:59951 -2015-07-10 13:26:45.699 INFO 5 --- [ration.thread-1] com.hazelcast.cluster.ClusterService : [10.244.77.3]:5701 [someGroup] [3.5] +kubectl logs -f hazelcast-39807171 +15-k1xsk +2017-01-30 12:42:50.774 INFO 6 --- [ main] com.github.pires.hazelcast.Application : Starting Application on hazelcast-3980717115-k1xsk with PID 6 (/bootstrapper.jar started by root in /) +2017-01-30 12:42:50.781 INFO 6 --- [ main] com.github.pires.hazelcast.Application : No active profile set, falling back to default profiles: default +2017-01-30 12:42:50.852 INFO 6 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@14514713: startup date [Mon Jan 30 12:42:50 GMT 2017]; root of context hierarchy +2017-01-30 12:42:52.304 INFO 6 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup +2017-01-30 12:42:52.323 INFO 6 --- [ main] c.g.p.h.HazelcastDiscoveryController : Asking k8s registry at https://kubernetes.default.svc.cluster.local.. +2017-01-30 12:42:52.857 INFO 6 --- [ main] c.g.p.h.HazelcastDiscoveryController : Found 1 pods running Hazelcast. +2017-01-30 12:42:52.990 INFO 6 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.7.5] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.244.9.2] +2017-01-30 12:42:52.990 INFO 6 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.7.5] Prefer IPv4 stack is true. +2017-01-30 12:42:53.002 INFO 6 --- [ main] c.h.instance.DefaultAddressPicker : [LOCAL] [someGroup] [3.7.5] Picked [10.244.9.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true +2017-01-30 12:42:53.032 INFO 6 --- [ main] com.hazelcast.system : [10.244.9.2]:5701 [someGroup] [3.7.5] Hazelcast 3.7.5 (20170124 - 111f332) starting at [10.244.9.2]:5701 +2017-01-30 12:42:53.032 INFO 6 --- [ main] com.hazelcast.system : [10.244.9.2]:5701 [someGroup] [3.7.5] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. +2017-01-30 12:42:53.032 INFO 6 --- [ main] com.hazelcast.system : [10.244.9.2]:5701 [someGroup] [3.7.5] Configured Hazelcast Serialization version : 1 +2017-01-30 12:42:53.343 INFO 6 --- [ main] c.h.s.i.o.impl.BackpressureRegulator : [10.244.9.2]:5701 [someGroup] [3.7.5] Backpressure is disabled +2017-01-30 12:42:54.273 INFO 6 --- [ main] com.hazelcast.instance.Node : [10.244.9.2]:5701 [someGroup] [3.7.5] Creating TcpIpJoiner +2017-01-30 12:42:54.507 INFO 6 --- [ main] c.h.s.i.o.impl.OperationExecutorImpl : [10.244.9.2]:5701 [someGroup] [3.7.5] Starting 2 partition threads +2017-01-30 12:42:54.508 INFO 6 --- [ main] c.h.s.i.o.impl.OperationExecutorImpl : [10.244.9.2]:5701 [someGroup] [3.7.5] Starting 3 generic threads (1 dedicated for priority tasks) +2017-01-30 12:42:54.525 INFO 6 --- [ main] com.hazelcast.core.LifecycleService : [10.244.9.2]:5701 [someGroup] [3.7.5] [10.244.9.2]:5701 is STARTING +2017-01-30 12:42:54.529 INFO 6 --- [ main] c.h.n.t.n.NonBlockingIOThreadingModel : [10.244.9.2]:5701 [someGroup] [3.7.5] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads +2017-01-30 12:42:54.578 INFO 6 --- [ main] com.hazelcast.cluster.impl.TcpIpJoiner : [10.244.9.2]:5701 [someGroup] [3.7.5] -Members [2] { - Member [10.244.37.3]:5701 - Member [10.244.77.3]:5701 this + +Members [1] { + Member [10.244.9.2]:5701 - f9cae801-59da-49d9-b8de-7719abb53844 this } -2015-07-10 13:26:47.722 INFO 5 --- [ main] com.hazelcast.core.LifecycleService : [10.244.77.3]:5701 [someGroup] [3.5] Address[10.244.77.3]:5701 is STARTED -2015-07-10 13:26:47.723 INFO 5 --- [ main] com.github.pires.hazelcast.Application : Started Application in 13.792 seconds (JVM running for 14.542) +2017-01-30 12:42:54.660 INFO 6 --- [ main] com.hazelcast.core.LifecycleService : [10.244.9.2]:5701 [someGroup] [3.7.5] [10.244.9.2]:5701 is STARTED +2017-01-30 12:42:54.662 INFO 6 --- [ main] com.github.pires.hazelcast.Application : Started Application in 5.078 seconds (JVM running for 5.771) +2017-01-30 12:44:08.780 INFO 6 --- [thread-Acceptor] c.h.nio.tcp.SocketAcceptorThread : [10.244.9.2]:5701 [someGroup] [3.7.5] Accepting socket connection from /10.244.93.3:45945 +2017-01-30 12:44:08.814 INFO 6 --- [cached.thread-1] c.h.nio.tcp.TcpIpConnectionManager : [10.244.9.2]:5701 [someGroup] [3.7.5] Established socket connection between /10.244.9.2:5701 and /10.244.93.3:45945 +2017-01-30 12:44:15.785 INFO 6 --- [ration.thread-0] c.h.internal.cluster.ClusterService : [10.244.9.2]:5701 [someGroup] [3.7.5] + +Members [2] { + Member [10.244.9.2]:5701 - f9cae801-59da-49d9-b8de-7719abb53844 this + Member [10.244.93.3]:5701 - 4e15667b-ce17-40c2-b045-abe3fb25d48b +} ``` Now let's scale our cluster to 4 nodes: - ```sh -$ kubectl scale rc hazelcast --replicas=4 +$ kubectl scale deployment hazelcast --replicas 4 ``` -Examine the status again by checking the logs and you should see the 4 members connected. +Examine the status again by checking a node's logs and you should see the 4 members connected. Something like: +``` +(...) + +Members [4] { + Member [10.244.9.2]:5701 - f9cae801-59da-49d9-b8de-7719abb53844 this + Member [10.244.93.3]:5701 - 4e15667b-ce17-40c2-b045-abe3fb25d48b + Member [10.244.9.3]:5701 - e0f36fa4-16bf-4009-a034-d4e7a4105003 + Member [10.244.93.4]:5701 - 7ac96b48-aa47-4410-885f-1ad0fc3690f0 +} +``` ### tl; dr; For those of you who are impatient, here is the summary of the commands we ran in this tutorial. ```sh -# create a service to track all hazelcast nodes -kubectl create -f examples/storage/hazelcast/hazelcast-service.yaml - -# create a replication controller to replicate hazelcast nodes -kubectl create -f examples/storage/hazelcast/hazelcast-controller.yaml - -# scale up to 2 nodes -kubectl scale rc hazelcast --replicas=2 - -# scale up to 4 nodes -kubectl scale rc hazelcast --replicas=4 +kubectl create -f service.yaml +kubectl create -f deployment.yaml +kubectl scale deployment hazelcast --replicas 2 +kubectl scale deployment hazelcast --replicas 4 ``` ### Hazelcast Discovery Source diff --git a/examples/storage/hazelcast/hazelcast-controller.yaml b/examples/storage/hazelcast/hazelcast-controller.yaml deleted file mode 100644 index 199a6619881..00000000000 --- a/examples/storage/hazelcast/hazelcast-controller.yaml +++ /dev/null @@ -1,31 +0,0 @@ -apiVersion: v1 -kind: ReplicationController -metadata: - labels: - name: hazelcast - name: hazelcast -spec: - replicas: 1 - selector: - name: hazelcast - template: - metadata: - labels: - name: hazelcast - spec: - containers: - - resources: - limits: - cpu: 0.1 - image: quay.io/pires/hazelcast-kubernetes:0.6.1 - name: hazelcast - env: - - name: "DNS_DOMAIN" - value: "cluster.local" - - name: POD_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - ports: - - containerPort: 5701 - name: hazelcast diff --git a/examples/storage/hazelcast/hazelcast-deployment.yaml b/examples/storage/hazelcast/hazelcast-deployment.yaml new file mode 100644 index 00000000000..e14cbbe2c6c --- /dev/null +++ b/examples/storage/hazelcast/hazelcast-deployment.yaml @@ -0,0 +1,22 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: hazelcast + labels: + name: hazelcast +spec: + template: + metadata: + labels: + name: hazelcast + spec: + containers: + - name: hazelcast + image: quay.io/pires/hazelcast-kubernetes:0.8.0 + imagePullPolicy: Always + env: + - name: "DNS_DOMAIN" + value: "cluster.local" + ports: + - name: hazelcast + containerPort: 5701 diff --git a/examples/storage/hazelcast/image/Dockerfile b/examples/storage/hazelcast/image/Dockerfile deleted file mode 100644 index 05798ee734b..00000000000 --- a/examples/storage/hazelcast/image/Dockerfile +++ /dev/null @@ -1,24 +0,0 @@ -# Copyright 2016 The Kubernetes Authors. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -FROM quay.io/pires/docker-jre:8u45-2 - - -EXPOSE 5701 - -RUN \ - curl -Lskj https://github.com/pires/hazelcast-kubernetes-bootstrapper/releases/download/0.5/hazelcast-kubernetes-bootstrapper-0.5.jar \ - -o /bootstrapper.jar - -CMD java -jar /bootstrapper.jar