Merge pull request #31125 from a-robinson/example-petset-cockroachdb

Automatic merge from submit-queue

Productionize the cockroachdb example a little more

Includes:
* A service for clients to use
* Readiness/liveness probes
* An extended graceful termination period
* Automatic prometheus monitoring (when prometheus is configured to watch for annotations on services, as in [CoreOS's recent blog post](https://coreos.com/blog/prometheus-and-kubernetes-up-and-running.html), for example)

I'm leaving the management of certs to future work, but if anyone that sees this needs help with them in the meantime, don't hesitate to reach out.

Successor to #28446

@bprashanth - if you're still interested in / open to an e2e test (as mentioned in https://github.com/cockroachdb/cockroach/issues/5967#issuecomment-230188807), let me know and I'll put one together. If so, I assume you'd want it as part of the `petset` test group rather than the `examples` tests?

cc @tschottdorf 

**Release note**:
```release-note
NONE
```

examples/cockroachdb/README.md

@@ -72,6 +72,25 @@ steps.
 
 Follow the steps in [minikube.sh](minikube.sh) (or simply run that file).
 
+## Accessing the database
+
+Along with our PetSet configuration, we expose a standard Kubernetes service
+that offers a load-balanced virtual IP for clients to access the database
+with. In our example, we've called this service `cockroachdb-public`.
+
+Start up a client pod and open up an interactive, (mostly) Postgres-flavor
+SQL shell using:
+
+```console
+$ kubectl run -it cockroach-client --image=cockroachdb/cockroach --restart=Never --command -- bash
+root@cockroach-client # ./cockroach sql --host cockroachdb-public
+```
+
+You can see example SQL statements for inserting and querying data in the
+included [demo script](demo.sh), but can use almost any Postgres-style SQL
+commands. Some more basic examples can be found within
+[CockroachDB's documentation](https://www.cockroachlabs.com/docs/learn-cockroachdb-sql.html).
+
 ## Simulating failures
 
 When all (or enough) nodes are up, simulate a failure like this:
@@ -80,14 +99,15 @@ When all (or enough) nodes are up, simulate a failure like this:
 kubectl exec cockroachdb-0 -- /bin/bash -c "while true; do kill 1; done"
 ```
 
-On one of the other pods, run `./cockroach sql --host $(hostname)` and use
-(mostly) Postgres-flavor SQL. The example runs with three-fold replication,
-so it can tolerate one failure of any given node at a time.
-Note also that there is a brief period of time immediately after the creation
-of the cluster during which the three-fold replication is established, and
-during which killing a node may lead to unavailability.
+You can then reconnect to the database as demonstrated above and verify
+that no data was lost. The example runs with three-fold replication, so
+it can tolerate one failure of any given node at a time. Note also that
+there is a brief period of time immediately after the creation of the
+cluster during which the three-fold replication is established, and during
+which killing a node may lead to unavailability.
 
-There is also a [demo script](demo.sh).
+The [demo script](demo.sh) gives an example of killing one instance of the
+database and ensuring the other replicas have all data that was written.
 
 ## Scaling up or down
 
@@ -96,6 +116,15 @@ volume claim first). If you ran `minikube.sh`, there's a spare volume so you
 can immediately scale up by one. Convince yourself that the new node
 immediately serves reads and writes.
 
+## Cleaning up when you're done
+
+Because all of the resources in this example have been tagged with the label `app=cockroachdb`,
+we can clean up everything that we created in one quick command using a selector on that label:
+
+```shell
+kubectl delete petsets,pods,persistentvolumes,persistentvolumeclaims,services -l app=cockroachdb
+```
+
 
 <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
 [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/examples/cockroachdb/README.md?pixel)]()

examples/cockroachdb/cockroachdb-petset.yaml

@@ -1,9 +1,9 @@
 apiVersion: v1
 kind: Service
 metadata:
-  annotations:
-    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
-  name: cockroachdb
+  # This service is meant to be used by clients of the database. It exposes a ClusterIP that will
+  # automatically load balance connections to the different database pods.
+  name: cockroachdb-public
   labels:
     app: cockroachdb
 spec:
@@ -14,6 +14,33 @@ spec:
     targetPort: 26257
     name: grpc
   # The secondary port serves the UI as well as health and debug endpoints.
+  - port: 8080
+    targetPort: 8080
+    name: http
+  selector:
+    app: cockroachdb
+---
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+    # Make sure DNS is resolvable during initialization.
+    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
+    # Enable automatic monitoring of all instances when Prometheus is running in the cluster.
+    prometheus.io/scrape: "true"
+    prometheus.io/path: "_status/vars"
+    prometheus.io/port: "8080"
+  # This service only exists to create DNS entries for each pet in the petset such that they can resolve
+  # each other's IP addresses. It does not create a load-balanced ClusterIP and should not be used
+  # directly by clients in most circumstances.
+  name: cockroachdb
+  labels:
+    app: cockroachdb
+spec:
+  ports:
+  - port: 26257
+    targetPort: 26257
+    name: grpc
   - port: 8080
     targetPort: 8080
     name: http
@@ -49,6 +76,16 @@ spec:
           name: grpc
         - containerPort: 8080
           name: http
+        livenessProbe:
+          httpGet:
+            path: /_admin/v1/health
+            port: http
+          initialDelaySeconds: 30
+        readinessProbe:
+          httpGet:
+            path: /_admin/v1/health
+            port: http
+          initialDelaySeconds: 10
         volumeMounts:
         - name: datadir
           mountPath: /cockroach/cockroach-data
@@ -79,6 +116,9 @@ spec:
               CRARGS+=("--join" "cockroachdb")
             fi
             /cockroach/cockroach ${CRARGS[*]}
+      # No pre-stop hook is required, a SIGTERM plus some time is all that's
+      # needed for graceful shutdown of a node.
+      terminationGracePeriodSeconds: 60
       volumes:
       - name: datadir
         persistentVolumeClaim:

examples/cockroachdb/minikube.sh

@@ -28,6 +28,9 @@
 
 set -exuo pipefail
 
+# Clean up anything from a prior run:
+kubectl delete petsets,pods,persistentvolumes,persistentvolumeclaims,services -l app=cockroachdb
+
 # Make persistent volumes and (correctly named) claims. We must create the
 # claims here manually even though that sounds counter-intuitive. For details
 # see https://github.com/kubernetes/contrib/pull/1295#issuecomment-230180894.
@@ -40,6 +43,7 @@ metadata:
   name: pv${i}
   labels:
     type: local
+    app: cockroachdb
 spec:
   capacity:
     storage: 1Gi
@@ -54,6 +58,8 @@ kind: PersistentVolumeClaim
 apiVersion: v1
 metadata:
   name: datadir-cockroachdb-${i}
+  labels:
+    app: cockroachdb
 spec:
   accessModes:
     - ReadWriteOnce