Merge pull request #7591 from derekwaynecarr/fix_origin

Fix OpenShift example
This commit is contained in:
Clayton Coleman 2015-05-06 15:43:11 -04:00
commit 52e94b1e2c
8 changed files with 190 additions and 329 deletions

View File

@ -2,109 +2,154 @@
This example shows how to run OpenShift Origin as a pod on an existing Kubernetes cluster.
This example demonstrates usage of a pod with a secret volume mount.
OpenShift Origin runs with a rich set of role based policy rules out of the box that requires authentication from users
via certificates. When run as a pod on an existing Kubernetes cluster, it proxies access to the underlying Kubernetes services
to provide security.
As a result, this example is a complex end-to-end configuration that shows how to configure certificates for a service that runs
on Kubernetes, and requires a number of configuration files to be injected dynamically via a secret volume to the pod.
### Step 0: Prerequisites
This example assumes that you have a basic understanding of Kubernetes and that you have forked the repository and [turned up a Kubernetes cluster](https://github.com/GoogleCloudPlatform/kubernetes#contents):
This example assumes that you have an understanding of Kubernetes and that you have forked the repository.
This example has been tested against the **gce** and **vagrant** based KUBERNETES_PROVIDER.
OpenShift Origin creates privileged containers when running Docker builds during the source-to-image process.
If you are using a Salt based KUBERNETES_PROVIDER (**gce**, **vagrant**, **aws**), you should enable the
ability to create privileged containers via the API.
```shell
$ cd kubernetes
$ vi cluster/saltbase/pillar/privilege.sls
# If true, allow privileged containers to be created by API
allow_privileged: true
```
Now spin up a cluster using your preferred KUBERNETES_PROVIDER
```shell
$ export KUBERNETES_PROVIDER=gce
$ hack/dev-build-and-up.sh
$ cluster/kube-up.sh
```
### Step 1: Generate resources
The demonstration will require the following resources:
1. A Kubernetes Secret that contains information needed to securely communicate to your Kubernetes master as an administrator
2. A Kubernetes Pod that contains information for how to run OpenShift Origin that consumes this Secret securely
3. A Kubernetes Service that exposes OpenShift Origin API via an external load balancer
4. A Kubernetes Service that exposes OpenShift Origin UI via an external load balancer
To generate these resources, we will run a script that introspects your configured KUBERNETES_PROVIDER:
Next, let's setup some variables, and create a local folder that will hold generated configuration files.
```shell
$ examples/openshift-origin/resource-generator.sh
$ export OPENSHIFT_EXAMPLE=$(pwd)/examples/openshift-origin
$ export OPENSHIFT_CONFIG=${OPENSHIFT_EXAMPLE}/config
$ mkdir ${OPENSHIFT_CONFIG}
```
A Kubernetes Secret was generated that contains the following data:
1. kubeconfig: a valid kubeconfig file that is used by OpenShift Origin to communicate to the master
2. kube-ca: a certificate authority for the Kubernetes master
3. kube-auth-path: a Kubernetes authorization file
4. kube-cert: a Kubernetes certificate
5. kube-key: a Kubernetes key file
### Step 1: Export your Kubernetes configuration file for use by OpenShift pod
As required by a Kubernetes secret, each piece of data is base64 encoded - with no line wraps.
OpenShift Origin uses a configuration file to know how to access your Kubernetes cluster with administrative authority.
```
$ cluster/kubectl.sh config view --output=yaml --flatten=true --minify=true > ${OPENSHIFT_CONFIG}/kubeconfig
```
The output from this command will contain a single file that has all the required information needed to connect to your
Kubernetes cluster that you previously provisioned. This file should be considered sensitive, so do not share this file with
untrusted parties.
We will later use this file to tell OpenShift how to bootstap its own configuration.
### Step 2: Create an External Load Balancer to Route Traffic to OpenShift
An external load balancer is needed to route traffic to our OpenShift master service that will run as a pod on your
Kubernetes cluster.
You can view the file by doing:
```shell
$ cat examples/openshift-origin/secret.json
$ cluster/kubectl.sh create -f $OPENSHIFT_EXAMPLE/openshift-service.yaml
```
Caution: This file contains all of the required information to operate as a Kubernetes admin on your cluster, so only share this file with trusted parties.
### Step 3: Generate configuration file for your OpenShift master pod
A Kubernetes Pod file was generated that can run OpenShift Origin on your cluster.
The OpenShift master requires a configuration file as input to know how to bootstrap the system.
The OpenShift Origin pod file has a volume mount that references the Kubernetes secret we created to know how to work with the underlying Kubernetes provider.
In order to build this configuration file, we need to know the public IP address of our external load balancer in order to
build default certificates.
You can view the file by doing:
Grab the public IP address of the service we previously created.
```shell
$ cat examples/openshift-origin/pod.json
$ export PUBLIC_IP=$(cluster/kubectl.sh get services openshift --template="{{ index .spec.publicIPs 0 }}")
$ echo $PUBLIC_IP
```
Finally, a Kubernetes service was generated for the UI and the API and available via an external load balancer:
Ensure you have a valid PUBLIC_IP address before continuing in the example.
``shell
$ cat examples/openshift-origin
### Step 2: Create the secret in Kubernetes
To provision the secret on Kubernetes:
We now need to run a command on your host to generate a proper OpenShift configuration. To do this, we will volume mount the configuration directory that holds your Kubernetes kubeconfig file from the prior step.
```shell
$ cluster/kubectl.sh create -f examples/openshift-origin/secret.json
docker run --privileged -v ${OPENSHIFT_CONFIG}:/config openshift/origin start master --write-config=/config --kubeconfig=/config/kubeconfig --master=https://localhost:8443 --public-master=https://${PUBLIC_IP}:8443
```
You should see your secret resource was created by listing:
```shell
$ cluster/kubectl.sh get secrets
You should now see a number of certificates minted in your configuration directory, as well as a master-config.yaml file that tells the OpenShift master how to execute. In the next step, we will bundle this into a Kubernetes Secret that our OpenShift master pod will consume.
### Step 4: Bundle the configuration into a Secret
We now need to bundle the contents of our configuration into a secret for use by our OpenShift master pod.
OpenShift includes an experimental command to make this easier.
First, update the ownership for the files previously generated:
```
$ sudo -E chown ${USER} -R ${OPENSHIFT_CONFIG}
```
### Step 3: Provisioning OpenShift Origin
To create the OpenShift Origin pod:
Then run the following command to collapse them into a Kubernetes secret.
```shell
$ cluster/kubectl.sh create -f examples/openshift-origin/pod.json
docker run -i -t --privileged -e="OPENSHIFTCONFIG=/config/admin.kubeconfig" -v ${OPENSHIFT_CONFIG}:/config openshift/origin ex bundle-secret openshift-config -f /config &> ${OPENSHIFT_EXAMPLE}/secret.json
```
### Step 4: Provisioning OpenShift Origin Services
To create the OpenShift Origin Services that expose the API and UI:
Now, lets create the secret in your Kubernetes cluster.
```shell
$ cluster/kubectl.sh create -f examples/openshift-origin/ui-service.json
$ cluster/kubectl.sh create -f examples/openshift-origin/api-service.json
$ cluster/kubectl.sh create -f ${OPENSHIFT_EXAMPLE}/secret.json
```
### Step 5: Open Firewall Ports
**NOTE: This secret is secret and should not be shared with untrusted parties.**
If you are running on GCE, you need to open the following ports:
### Step 5: Deploy OpenShift Master
We are now ready to deploy OpenShift.
We will deploy a pod that runs the OpenShift master. The OpenShift master will delegate to the underlying Kubernetes
system to manage Kubernetes specific resources. For the sake of simplicity, the OpenShift master will run with an embedded etcd to hold OpenShift specific content. This demonstration will evolve in the future to show how to run etcd in a pod so that content is not destroyed if the OpenShift master fails.
```shell
$ gcloud compute instances list
FIND THE MINION NAME PREFIX
$ gcloud compute firewall-rules create openshift-origin-node-8444 --allow tcp:8444 --target-tags kubernetes-minion-prq8
$ gcloud compute firewall-rules create openshift-origin-node-8443 --allow tcp:8443 --target-tags kubernetes-minion-prq8
$ cluster/kubectl.sh create -f ${OPENSHIFT_EXAMPLE}/openshift-controller.yaml
```
### Step 4: Try out OpenShift Origin
TODO add more detail here:
You should now get a pod provisioned whose name begins with openshift.
```shell
$ cluster/kubectl.sh get pods | grep openshift
$ cluster/kubectl.sh log openshift-t7147 origin
Running: cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl log openshift-t7t47 origin
2015-04-30T15:26:00.454146869Z I0430 15:26:00.454005 1 start_master.go:296] Starting an OpenShift master, reachable at 0.0.0.0:8443 (etcd: [https://10.0.27.2:4001])
2015-04-30T15:26:00.454231211Z I0430 15:26:00.454223 1 start_master.go:297] OpenShift master public address is https://104.197.73.241:8443
```
Depending upon your cloud provider, you may need to open up an external firewall rule for tcp:8443. For GCE, you can run the following:
```shell
gcloud compute --project "your-project" firewall-rules create "origin" --allow tcp:8443 --network "your-network" --source-ranges "0.0.0.0/0"
```
Consult your cloud provider's documentation for more information.
Open a browser and visit the OpenShift master public address reported in your log.
You can use the CLI commands by running the following:
```shell
$ docker run --privileged --entrypoint="/usr/bin/bash" -it -e="OPENSHIFTCONFIG=/config/admin.kubeconfig" -v ${OPENSHIFT_CONFIG}:/config openshift/origin
$ osc config use-context public-default
$ osc --help
```

View File

@ -14,21 +14,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Deletes pod, deletes secret
# Cleans up resources from the example, assumed to be run from Kubernetes repo root
set -o errexit
set -o nounset
set -o pipefail
ORIGIN=$(dirname "${BASH_SOURCE}")
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
## Delete the services
${KUBE_ROOT}/cluster/kubectl.sh delete services origin-api
${KUBE_ROOT}/cluster/kubectl.sh delete services origin-ui
## Delete the pod
${KUBE_ROOT}/cluster/kubectl.sh delete pods openshift
## Delete the secret
${KUBE_ROOT}/cluster/kubectl.sh delete secrets kubernetes-secret
export OPENSHIFT_EXAMPLE=$(pwd)/examples/openshift-origin
export OPENSHIFT_CONFIG=${OPENSHIFT_EXAMPLE}/config
rm -fr ${OPENSHIFT_CONFIG}
cluster/kubectl.sh delete secrets openshift-config
cluster/kubectl.sh stop rc openshift
cluster/kubectl.sh delete rc openshift
cluster/kubectl.sh delete services openshift

View File

@ -1,37 +0,0 @@
#!/bin/bash
# Copyright 2014 The Kubernetes Authors All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Generates secret, creates secret on kube, creates pod on kube
set -o errexit
set -o nounset
set -o pipefail
ORIGIN=$(dirname "${BASH_SOURCE}")
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
## Generate resources
${ORIGIN}/resource-generator.sh
## Create the secret
${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/secret.json
## Create the pod
${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/pod.json
## Create the services
${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/api-service.json
${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/ui-service.json

View File

@ -0,0 +1,33 @@
#!/bin/bash
# Copyright 2014 The Kubernetes Authors All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Creates resources from the example, assumed to be run from Kubernetes repo root
export OPENSHIFT_EXAMPLE=$(pwd)/examples/openshift-origin
export OPENSHIFT_CONFIG=${OPENSHIFT_EXAMPLE}/config
mkdir ${OPENSHIFT_CONFIG}
cluster/kubectl.sh config view --output=yaml --flatten=true --minify=true > ${OPENSHIFT_CONFIG}/kubeconfig
cluster/kubectl.sh create -f $OPENSHIFT_EXAMPLE/openshift-service.yaml
sleep 30
export PUBLIC_IP=$(cluster/kubectl.sh get services openshift --template="{{ index .spec.publicIPs 0 }}")
echo $PUBLIC_IP
export PORTAL_IP=$(cluster/kubectl.sh get services openshift --template="{{ .spec.portalIP }}")
echo $PORTAL_IP
docker run --privileged -v ${OPENSHIFT_CONFIG}:/config openshift/origin start master --write-config=/config --kubeconfig=/config/kubeconfig --master=https://localhost:8443 --public-master=https://${PUBLIC_IP}:8443
sudo -E chown ${USER} -R ${OPENSHIFT_CONFIG}
docker run -i -t --privileged -e="OPENSHIFTCONFIG=/config/admin.kubeconfig" -v ${OPENSHIFT_CONFIG}:/config openshift/origin ex bundle-secret openshift-config -f /config &> ${OPENSHIFT_EXAMPLE}/secret.json
cluster/kubectl.sh create -f ${OPENSHIFT_EXAMPLE}/secret.json
cluster/kubectl.sh create -f ${OPENSHIFT_EXAMPLE}/openshift-controller.yaml
cluster/kubectl.sh get pods | grep openshift

View File

@ -0,0 +1,33 @@
apiVersion: v1beta3
kind: ReplicationController
metadata:
labels:
name: openshift
name: openshift
spec:
replicas: 1
selector:
name: openshift
template:
metadata:
labels:
name: openshift
spec:
containers:
- args:
- start
- master
- --config=/config/master-config.yaml
image: "openshift/origin"
name: origin
ports:
- containerPort: 8443
name: openshift
volumeMounts:
- mountPath: /config
name: config
readOnly: true
volumes:
- name: config
secret:
secretName: openshift-config

View File

@ -0,0 +1,12 @@
apiVersion: v1beta3
kind: Service
metadata:
name: openshift
spec:
ports:
- port: 8443
name: openshift
targetPort: 8443
selector:
name: openshift
createExternalLoadBalancer: true

View File

@ -1,18 +0,0 @@
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/secret-volume/kube-ca
server: https://146.148.35.28
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
auth-path: /etc/secret-volume/kube-auth-path

View File

@ -1,198 +0,0 @@
#!/bin/bash
# Copyright 2014 The Kubernetes Authors All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Generates pod and secret to deploy origin against configured Kubernetes provider
set -o errexit
set -o nounset
set -o pipefail
ORIGIN=$(dirname "${BASH_SOURCE}")
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../..
source "${KUBE_ROOT}/cluster/kubectl.sh" > /dev/null 2>&1
# Check all prerequisites are on the path
HAVE_JQ=$(which jq)
if [[ -z ${HAVE_JQ} ]]; then
echo "Please install jq"
exit 1
fi
HAVE_BASE64=$(which base64)
if [[ -z ${HAVE_BASE64} ]]; then
echo "Please install base64"
exit 1
fi
# Capture information about your kubernetes cluster
TEMPLATE="--template=\"{{ index . \"current-context\" }}\""
CURRENT_CONTEXT=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" )
TEMPLATE="--template=\"{{range .contexts}}{{ if eq .name ${CURRENT_CONTEXT} }}{{ .context.cluster }}{{end}}{{end}}\""
CURRENT_CLUSTER=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" )
TEMPLATE="--template=\"{{range .contexts}}{{ if eq .name ${CURRENT_CONTEXT} }}{{ .context.user }}{{end}}{{end}}\""
CURRENT_USER=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" )
TEMPLATE="--template=\"{{range .clusters}}{{ if eq .name ${CURRENT_CLUSTER} }}{{ index . \"cluster\" \"certificate-authority\" }}{{end}}{{end}}\""
CERTIFICATE_AUTHORITY=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" )
TEMPLATE="--template=\"{{range .clusters}}{{ if eq .name ${CURRENT_CLUSTER} }}{{ .cluster.server }}{{end}}{{end}}\""
KUBE_MASTER=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" )
TEMPLATE="--template=\"{{range .users}}{{ if eq .name ${CURRENT_USER} }}{{ index . \"user\" \"auth-path\" }}{{end}}{{end}}\""
AUTH_PATH=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" )
# Build an auth_path file to embed as a secret
AUTH_PATH_DATA=$(cat ${AUTH_PATH} )
KUBE_USER=$( echo ${AUTH_PATH_DATA} | jq '.User' )
KUBE_PASSWORD=$( echo ${AUTH_PATH_DATA} | jq '.Password' )
KUBE_CERT_FILE=$( echo ${AUTH_PATH_DATA} | jq '.CertFile' )
KUBE_KEY_FILE=$( echo ${AUTH_PATH_DATA} | jq '.KeyFile' )
cat <<EOF >"${ORIGIN}/origin-auth-path"
{
"User": ${KUBE_USER},
"Password": ${KUBE_PASSWORD},
"CAFile": "/etc/secret-volume/kube-ca",
"CertFile": "/etc/secret-volume/kube-cert",
"KeyFile": "/etc/secret-volume/kube-key"
}
EOF
# Collect all the secrets and encode as base64
ORIGIN_KUBECONFIG_DATA=$( cat ${ORIGIN}/origin-kubeconfig.yaml | base64 --wrap=0)
ORIGIN_CERTIFICATE_AUTHORITY_DATA=$(cat ${CERTIFICATE_AUTHORITY} | base64 --wrap=0)
ORIGIN_AUTH_PATH_DATA=$(cat ${ORIGIN}/origin-auth-path | base64 --wrap=0)
ORIGIN_CERT_FILE=$( cat ${KUBE_CERT_FILE//\"/} | base64 --wrap=0)
ORIGIN_KEY_FILE=$( cat ${KUBE_KEY_FILE//\"/} | base64 --wrap=0)
cat <<EOF >"${ORIGIN}/secret.json"
{
"apiVersion": "v1beta2",
"kind": "Secret",
"id": "kubernetes-secret",
"data": {
"kubeconfig": "${ORIGIN_KUBECONFIG_DATA}",
"kube-ca": "${ORIGIN_CERTIFICATE_AUTHORITY_DATA}",
"kube-auth-path": "${ORIGIN_AUTH_PATH_DATA}",
"kube-cert": "${ORIGIN_CERT_FILE}",
"kube-key": "${ORIGIN_KEY_FILE}"
}
}
EOF
echo "Generated Kubernetes Secret file: ${ORIGIN}/secret.json"
# Generate an OpenShift Origin pod
# TODO: In future, move this to a replication controller when we are not running etcd in container
cat <<EOF >"${ORIGIN}/pod.json"
{
"apiVersion": "v1beta1",
"id": "openshift",
"kind": "Pod",
"labels": {"name": "origin"},
"desiredState": {
"manifest": {
"containers": [
{
"command": [
"start",
"master",
"--kubernetes=${KUBE_MASTER}",
"--kubeconfig=/etc/secret-volume/kubeconfig",
"--public-kubernetes=https://10.245.1.3:8443",
"--public-master=https://10.245.1.3:8443",
],
"image": "openshift/origin:latest",
"imagePullPolicy": "PullIfNotPresent",
"name": "origin",
"ports": [
{
"name": "https-api",
"containerPort": 8443,
"hostPort": 8443,
},
{
"name": "https-ui",
"containerPort": 8444,
"hostPort": 8444,
}
],
"volumeMounts": [
{
"mountPath": "/etc/secret-volume",
"name": "secret-volume",
"readOnly": true
}
]
}
],
"restartPolicy": {
"never": {}
},
"version": "v1beta2",
"volumes": [
{
"name": "secret-volume",
"source": {
"secret": {
"target": {
"kind": "Secret",
"name": "kubernetes-secret",
"namespace": "default"
}
}
}
}
]
}
}
}
EOF
echo "Generated Kubernetes Pod file: ${ORIGIN}/pod.json"
cat <<EOF >"${ORIGIN}/api-service.json"
{
"apiVersion": "v1beta1",
"kind": "Service",
"id": "origin-api",
"port": 8443,
"containerPort": 8443,
"selector": { "name": "origin" },
}
EOF
echo "Generated Kubernetes Service file: ${ORIGIN}/api-service.json"
cat <<EOF >"${ORIGIN}/ui-service.json"
{
"apiVersion": "v1beta1",
"kind": "Service",
"id": "origin-ui",
"port": 8444,
"containerPort": 8444,
"selector": { "name": "origin" },
}
EOF
echo "Generated Kubernetes Service file: ${ORIGIN}/ui-service.json"