mirror of
https://github.com/k3s-io/kubernetes.git
synced 2025-07-30 15:05:27 +00:00
commit
494ac9ebc0
164
examples/storm/README.md
Normal file
164
examples/storm/README.md
Normal file
@ -0,0 +1,164 @@
|
||||
# Storm example
|
||||
|
||||
Following this example, you will create a functional [Apache
|
||||
Storm](http://storm.apache.org/) cluster using Kubernetes and
|
||||
[Docker](http://docker.io).
|
||||
|
||||
You will setup an [Apache ZooKeeper](http://zookeeper.apache.org/)
|
||||
service, a Storm master service (a.k.a. Nimbus server), and a set of
|
||||
Storm workers (a.k.a. supervisors).
|
||||
|
||||
For the impatient expert, jump straight to the [tl;dr](#tl-dr)
|
||||
section.
|
||||
|
||||
## Step Zero: Prerequisites
|
||||
|
||||
This example assumes you have a Kubernetes cluster installed and
|
||||
running, and that you have installed the ```kubectl``` command line
|
||||
tool somewhere in your path. Please see the [getting
|
||||
started](../../docs/getting-started-guides) for installation
|
||||
instructions for your platform.
|
||||
|
||||
## Step One: Start your ZooKeeper service
|
||||
|
||||
ZooKeeper is a distributed coordination service that Storm uses as a
|
||||
bootstrap and for state storage.
|
||||
|
||||
Use the `examples/storm/zookeeper.json` file to create a pod running
|
||||
the ZooKeeper service.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/storm/zookeeper.json
|
||||
```
|
||||
|
||||
Then, use the `examples/storm/zookeeper-service.json` file to create a
|
||||
logical service endpoint that Storm can use to access the ZooKeeper
|
||||
pod.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/storm/zookeeper-service.json
|
||||
```
|
||||
|
||||
You should make sure the ZooKeeper pod is Running and accessible
|
||||
before proceeding.
|
||||
|
||||
### Check to see if ZooKeeper is running
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS
|
||||
zookeeper 192.168.86.4 zookeeper mattf/zookeeper 172.18.145.8/172.18.145.8 name=zookeeper Running
|
||||
```
|
||||
|
||||
### Check to see if ZooKeeper is accessible
|
||||
|
||||
```shell
|
||||
$ kubectl get services
|
||||
NAME LABELS SELECTOR IP PORT
|
||||
kubernetes component=apiserver,provider=kubernetes <none> 10.254.0.2 443
|
||||
kubernetes-ro component=apiserver,provider=kubernetes <none> 10.254.0.1 80
|
||||
zookeeper name=zookeeper name=zookeeper 10.254.139.141 2181
|
||||
|
||||
$ echo ruok | nc 10.254.139.141 2181; echo
|
||||
imok
|
||||
```
|
||||
|
||||
## Step Two: Start your Nimbus service
|
||||
|
||||
The Nimbus service is the master (or head) service for a Storm
|
||||
cluster. It depends on a functional ZooKeeper service.
|
||||
|
||||
Use the `examples/storm/storm-nimbus.json` file to create a pod running
|
||||
the Nimbus service.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/storm/storm-nimbus.json
|
||||
```
|
||||
|
||||
Then, use the `examples/storm/storm-nimbus-service.json` file to
|
||||
create a logical service endpoint that Storm workers can use to access
|
||||
the Nimbus pod.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/storm/storm-nimbus-service.json
|
||||
```
|
||||
|
||||
Ensure that the Nimbus service is running and functional.
|
||||
|
||||
### Check to see if Nimbus is running and accessible
|
||||
|
||||
```shell
|
||||
$ kubectl get services
|
||||
NAME LABELS SELECTOR IP PORT
|
||||
kubernetes component=apiserver,provider=kubernetes <none> 10.254.0.2 443
|
||||
kubernetes-ro component=apiserver,provider=kubernetes <none> 10.254.0.1 80
|
||||
zookeeper name=zookeeper name=zookeeper 10.254.139.141 2181
|
||||
nimbus name=nimbus name=nimbus 10.254.115.208 6627
|
||||
|
||||
$ sudo docker run -it -w /opt/apache-storm mattf/storm-base sh -c '/configure.sh 10.254.139.141 10.254.115.208; ./bin/storm list'
|
||||
...
|
||||
No topologies running.
|
||||
```
|
||||
|
||||
## Step Three: Start your Storm workers
|
||||
|
||||
The Storm workers (or supervisors) do the heavy lifting in a Storm
|
||||
cluster. They run your stream processing topologies and are managed by
|
||||
the Nimbus service.
|
||||
|
||||
The Storm workers need both the ZooKeeper and Nimbus services to be
|
||||
running.
|
||||
|
||||
Use the `examples/storm/storm-worker-controller.json` file to create a
|
||||
ReplicationController that manages the worker pods.
|
||||
|
||||
```shell
|
||||
$ kubectl create -f examples/storm/storm-worker-controller.json
|
||||
```
|
||||
|
||||
### Check to see if the workers are running
|
||||
|
||||
One way to check on the workers is to get information from the
|
||||
ZooKeeper service about how many clients it has.
|
||||
|
||||
```shell
|
||||
$ echo stat | nc 10.254.139.141 2181; echo
|
||||
Zookeeper version: 3.4.6--1, built on 10/23/2014 14:18 GMT
|
||||
Clients:
|
||||
/192.168.48.0:44187[0](queued=0,recved=1,sent=0)
|
||||
/192.168.45.0:39568[1](queued=0,recved=14072,sent=14072)
|
||||
/192.168.86.1:57591[1](queued=0,recved=34,sent=34)
|
||||
/192.168.8.0:50375[1](queued=0,recved=34,sent=34)
|
||||
/192.168.45.0:39576[1](queued=0,recved=34,sent=34)
|
||||
|
||||
Latency min/avg/max: 0/2/2570
|
||||
Received: 23199
|
||||
Sent: 23198
|
||||
Connections: 5
|
||||
Outstanding: 0
|
||||
Zxid: 0xa39
|
||||
Mode: standalone
|
||||
Node count: 13
|
||||
```
|
||||
|
||||
There should be one client from the Nimbus service and one per
|
||||
worker. Ideally, you should get ```stat``` output from ZooKeeper
|
||||
before and after creating the ReplicationController.
|
||||
|
||||
(Pull requests welcome for alternative ways to validate the workers)
|
||||
|
||||
## tl;dr
|
||||
|
||||
```kubectl create -f zookeeper.json```
|
||||
|
||||
```kubectl create -f zookeeper-service.json```
|
||||
|
||||
Make sure the ZooKeeper Pod is running (use: ```kubectl get pods```).
|
||||
|
||||
```kubectl create -f storm-nimbus.json```
|
||||
|
||||
```kubectl create -f storm-nimbus-service.json```
|
||||
|
||||
Make sure the Nimbus Pod is running.
|
||||
|
||||
```kubectl create -f storm-worker-controller.json```
|
9
examples/storm/storm-nimbus-service.json
Normal file
9
examples/storm/storm-nimbus-service.json
Normal file
@ -0,0 +1,9 @@
|
||||
{
|
||||
"id": "nimbus",
|
||||
"kind": "Service",
|
||||
"apiVersion": "v1beta1",
|
||||
"port": 6627,
|
||||
"containerPort": 6627,
|
||||
"selector": { "name": "nimbus" },
|
||||
"labels": { "name": "nimbus" }
|
||||
}
|
20
examples/storm/storm-nimbus.json
Normal file
20
examples/storm/storm-nimbus.json
Normal file
@ -0,0 +1,20 @@
|
||||
{
|
||||
"id": "nimbus",
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1beta1",
|
||||
"desiredState": {
|
||||
"manifest": {
|
||||
"version": "v1beta1",
|
||||
"id": "nimbus",
|
||||
"containers": [{
|
||||
"name": "nimbus",
|
||||
"image": "mattf/storm-nimbus",
|
||||
"cpu": 100,
|
||||
"ports": [{ "containerPort": 6627 }]
|
||||
}]
|
||||
}
|
||||
},
|
||||
"labels": {
|
||||
"name": "nimbus"
|
||||
}
|
||||
}
|
31
examples/storm/storm-worker-controller.json
Normal file
31
examples/storm/storm-worker-controller.json
Normal file
@ -0,0 +1,31 @@
|
||||
{
|
||||
"id": "storm-worker-controller",
|
||||
"kind": "ReplicationController",
|
||||
"apiVersion": "v1beta1",
|
||||
"desiredState": {
|
||||
"replicas": 3,
|
||||
"replicaSelector": {"name": "storm-worker"},
|
||||
"podTemplate": {
|
||||
"desiredState": {
|
||||
"manifest": {
|
||||
"version": "v1beta1",
|
||||
"id": "storm-worker-controller",
|
||||
"containers": [{
|
||||
"name": "storm-worker",
|
||||
"image": "mattf/storm-worker",
|
||||
"cpu": 200,
|
||||
"ports": [{"containerPort": 6700, "hostPort": 6700},
|
||||
{"containerPort": 6701, "hostPort": 6701},
|
||||
{"containerPort": 6702, "hostPort": 6702},
|
||||
{"containerPort": 6703, "hostPort": 6703}]
|
||||
}]
|
||||
}
|
||||
},
|
||||
"labels": {
|
||||
"name": "storm-worker",
|
||||
"uses": "nimbus"
|
||||
}
|
||||
}
|
||||
},
|
||||
"labels": {"name": "storm-worker"}
|
||||
}
|
9
examples/storm/zookeeper-service.json
Normal file
9
examples/storm/zookeeper-service.json
Normal file
@ -0,0 +1,9 @@
|
||||
{
|
||||
"id": "zookeeper",
|
||||
"kind": "Service",
|
||||
"apiVersion": "v1beta1",
|
||||
"port": 2181,
|
||||
"containerPort": 2181,
|
||||
"selector": { "name": "zookeeper" },
|
||||
"labels": { "name": "zookeeper" }
|
||||
}
|
20
examples/storm/zookeeper.json
Normal file
20
examples/storm/zookeeper.json
Normal file
@ -0,0 +1,20 @@
|
||||
{
|
||||
"id": "zookeeper",
|
||||
"kind": "Pod",
|
||||
"apiVersion": "v1beta1",
|
||||
"desiredState": {
|
||||
"manifest": {
|
||||
"version": "v1beta1",
|
||||
"id": "zookeeper",
|
||||
"containers": [{
|
||||
"name": "zookeeper",
|
||||
"image": "mattf/zookeeper",
|
||||
"cpu": 100,
|
||||
"ports": [{ "containerPort": 2181 }]
|
||||
}]
|
||||
}
|
||||
},
|
||||
"labels": {
|
||||
"name": "zookeeper"
|
||||
}
|
||||
}
|
Loading…
Reference in New Issue
Block a user