Add registry action to the kubernetes-worker layer

This commit is contained in:
Jacek N 2017-03-14 15:40:09 +00:00 committed by George Kraft
parent ca4afd8773
commit ebd2f88f6b
7 changed files with 320 additions and 3 deletions

View File

@ -41,6 +41,27 @@ a unit for maintenance.
Resuming the workload will [uncordon](http://kubernetes.io/docs/user-guide/kubectl/kubectl_uncordon/) a paused unit. Workloads will automatically migrate unless otherwise directed via their application declaration.
## Private registry
With the "registry" action that is part for the kubernetes-worker charm, you can very easily create a private docker registry, with authentication, and available over TLS. Please note that the registry deployed with the action is not HA, and uses storage tied to the kubernetes node where the pod is running. So if the registry pod changes is migrated from one node to another for whatever reason, you will need to re-publish the images.
### Example usage
Create the relevant authentication files. Let's say you want user `userA` to authenticate with the password `passwordA`. Then you'll do :
echo "userA:passwordA" > htpasswd-plain
htpasswd -c -b -B htpasswd userA passwordA
(the `htpasswd` program comes with the `apache2-utils` package)
Supposing your registry will be reachable at `myregistry.company.com`, and that you already have your TLS key in the `registry.key` file, and your TLS certificate (with `myregistry.company.com` as Common Name) in the `registry.crt` file, you would then run :
juju run-action kubernetes-worker/0 registry domain=myregistry.company.com htpasswd="$(base64 -w0 htpasswd)" htpasswd-plain="$(base64 -w0 htpasswd-plain)" tlscert="$(base64 -w0 registry.crt)" tlskey="$(base64 -w0 registry.key)" ingress=true
If you then decide that you want do delete the registry, just run :
juju run-action kubernetes-worker/0 registry delete=true ingress=true
## Known Limitations
Kubernetes workers currently only support 'phaux' HA scenarios. Even when configured with an HA cluster string, they will only ever contact the first unit in the cluster map. To enable a proper HA story, kubernetes-worker units are encouraged to proxy through a [kubeapi-load-balancer](https://jujucharms.com/kubeapi-load-balancer)
@ -48,5 +69,4 @@ application. This enables a HA deployment without the need to
re-render configuration and disrupt the worker services.
External access to pods must be performed through a [Kubernetes
Ingress Resource](http://kubernetes.io/docs/user-guide/ingress/). More
information
Ingress Resource](http://kubernetes.io/docs/user-guide/ingress/).

View File

@ -14,6 +14,32 @@ microbot:
delete:
type: boolean
default: False
description: Removes a microbots deployment, service, and ingress if True.
description: Remove a microbots deployment, service, and ingress if True.
upgrade:
description: Upgrade the kubernetes snaps
registry:
description: Create a private Docker registry
params:
htpasswd:
type: string
description: base64 encoded htpasswd file used for authentication.
htpasswd-plain:
type: string
description: base64 encoded plaintext version of the htpasswd file, needed by docker daemons to authenticate to the registry.
tlscert:
type: string
description: base64 encoded TLS certificate for the registry. Common Name must match the domain name of the registry.
tlskey:
type: string
description: base64 encoded TLS key for the registry.
domain:
type: string
description: The domain name for the registry. Must match the Common Name of the certificate.
ingress:
type: boolean
default: false
description: Create an Ingress resource for the registry (or delete resource object if "delete" is True)
delete:
type: boolean
default: false
description: Remove a registry replication controller, service, and ingress if True.

View File

@ -0,0 +1,136 @@
#!/usr/bin/python3
#
# For a usage examples, see README.md
#
# TODO
#
# - make the action idempotent (i.e. if you run it multiple times, the first
# run will create/delete the registry, and the reset will be a no-op and won't
# error out)
#
# - take only a plain authentication file, and create the encrypted version in
# the action
#
# - validate the parameters (make sure tlscert is a certificate, that tlskey is a
# proper key, etc)
#
# - when https://bugs.launchpad.net/juju/+bug/1661015 is fixed, handle the
# base64 encoding the parameters in the action itself
import os
import sys
from base64 import b64encode
from charmhelpers.core.hookenv import action_get
from charmhelpers.core.hookenv import action_set
from charms.templating.jinja2 import render
from subprocess import call
os.environ['PATH'] += os.pathsep + os.path.join(os.sep, 'snap', 'bin')
deletion = action_get('delete')
context = {}
# These config options must be defined in the case of a creation
param_error = False
for param in ('tlscert', 'tlskey', 'domain', 'htpasswd', 'htpasswd-plain'):
value = action_get(param)
if not value and not deletion:
key = "registry-create-parameter-{}".format(param)
error = "failure, parameter {} is required".format(param)
action_set({key: error})
param_error = True
context[param] = value
# Create the dockercfg template variable
dockercfg = '{"%s:443": {"auth": "%s", "email": "root@localhost"}}' % \
(context['domain'], context['htpasswd-plain'])
context['dockercfg'] = b64encode(dockercfg.encode()).decode('ASCII')
if param_error:
sys.exit(0)
# This one is either true or false, no need to check if it has a "good" value.
context['ingress'] = action_get('ingress')
# Declare a kubectl template when invoking kubectl
kubectl = ['kubectl', '--kubeconfig=/root/cdk/kubeconfig']
# Remove deployment if requested
if deletion:
resources = ['svc/kube-registry', 'rc/kube-registry-v0', 'secrets/registry-tls-data',
'secrets/registry-auth-data', 'secrets/registry-access']
if action_get('ingress'):
resources.append('ing/registry-ing')
delete_command = kubectl + ['delete', '--ignore-not-found=true'] + resources
delete_response = call(delete_command)
if delete_response == 0:
action_set({'registry-delete': 'success'})
else:
action_set({'registry-delete': 'failure'})
sys.exit(0)
# Creation request
render('registry.yaml', '/root/cdk/addons/registry.yaml',
context)
create_command = kubectl + ['create', '-f',
'/root/cdk/addons/registry.yaml']
create_response = call(create_command)
if create_response == 0:
action_set({'registry-create': 'success'})
# Create a ConfigMap if it doesn't exist yet, else patch it.
# A ConfigMap is needed to change the default value for nginx' client_max_body_size.
# The default is 1MB, and this is the maximum size of images that can be
# pushed on the registry. 1MB images aren't useful, so we bump this value to 1024MB.
cm_name = 'nginx-load-balancer-conf'
check_cm_command = kubectl + ['get', 'cm', cm_name]
check_cm_response = call(check_cm_command)
if check_cm_response == 0:
# There is an existing ConfigMap, patch it
patch = '{"data":{"max-body-size":"1024m"}}'
patch_cm_command = kubectl + ['patch', 'cm', cm_name, '-p', patch]
patch_cm_response = call(patch_cm_command)
if patch_cm_response == 0:
action_set({'configmap-patch': 'success'})
else:
action_set({'configmap-patch': 'failure'})
else:
# No existing ConfigMap, create it
render('registry-configmap.yaml', '/root/cdk/addons/registry-configmap.yaml',
context)
create_cm_command = kubectl + ['create', '-f', '/root/cdk/addons/registry-configmap.yaml']
create_cm_response = call(create_cm_command)
if create_cm_response == 0:
action_set({'configmap-create': 'success'})
else:
action_set({'configmap-create': 'failure'})
# Patch the "default" serviceaccount with an imagePullSecret.
# This will allow the docker daemons to authenticate to our private
# registry automatically
patch = '{"imagePullSecrets":[{"name":"registry-access"}]}'
patch_sa_command = kubectl + ['patch', 'sa', 'default', '-p', patch]
patch_sa_response = call(patch_sa_command)
if patch_sa_response == 0:
action_set({'serviceaccount-patch': 'success'})
else:
action_set({'serviceaccount-patch': 'failure'})
else:
action_set({'registry-create': 'failure'})

View File

@ -203,6 +203,11 @@ def install_cni_plugins():
hookenv.log(install)
check_call(install)
# Used by the "registry" action. The action is run on a single worker, but
# the registry pod can end up on any worker, so we need this directory on
# all the workers.
os.makedirs('/srv/registry', exist_ok=True)
set_state('kubernetes-worker.cni-plugins.installed')

View File

@ -0,0 +1,6 @@
apiVersion: v1
data:
body-size: 1024m
kind: ConfigMap
metadata:
name: nginx-load-balancer-conf

View File

@ -1,4 +1,9 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-load-balancer-conf
---
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-ingress-controller
@ -45,3 +50,4 @@ spec:
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --nginx-configmap=$(POD_NAMESPACE)/nginx-load-balancer-conf

View File

@ -0,0 +1,118 @@
apiVersion: v1
kind: Secret
metadata:
name: registry-tls-data
type: Opaque
data:
tls.crt: {{ tlscert }}
tls.key: {{ tlskey }}
---
apiVersion: v1
kind: Secret
metadata:
name: registry-auth-data
type: Opaque
data:
htpasswd: {{ htpasswd }}
---
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-registry
version: v0
template:
metadata:
labels:
k8s-app: kube-registry
version: v0
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2
resources:
# keep request = limit to keep this container in guaranteed class
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: basic_realm
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: /auth/htpasswd
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
- name: auth-dir
mountPath: /auth
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
hostPath:
path: /srv/registry
- name: auth-dir
secret:
secretName: registry-auth-data
---
apiVersion: v1
kind: Service
metadata:
name: kube-registry
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeRegistry"
spec:
selector:
k8s-app: kube-registry
type: LoadBalancer
ports:
- name: registry
port: 5000
protocol: TCP
---
apiVersion: v1
kind: Secret
metadata:
name: registry-access
data:
.dockercfg: {{ dockercfg }}
type: kubernetes.io/dockercfg
{%- if ingress %}
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: registry-ing
spec:
tls:
- hosts:
- {{ domain }}
secretName: registry-tls-data
rules:
- host: {{ domain }}
http:
paths:
- backend:
serviceName: kube-registry
servicePort: 5000
path: /
{% endif %}