Retry object creation with --validate in kube-addons

The better solution is some fence with Salt, but the actual logs
provided in the bug don't support any race condition here, plus the
ordering in the Salt configuration seems correct.

We haven't seen this again in a while, but given the results of the
situation (a borked cluster), I'm proposing a relatively simple
workaround.

Fixes #4357 (dubiously)
This commit is contained in:
Zach Loafman 2015-03-11 13:42:41 -07:00
parent 1231e65829
commit 018b11bc3a

View File

@ -17,10 +17,23 @@
# The business logic for whether a given object should be created
# was already enforced by salt, and /etc/kubernetes/addons is the
# managed result is of that. Start everything below that directory.
echo "== Kubernetes addon manager started at $(date -Is) =="
KUBECTL=/usr/local/bin/kubectl
function create-object() {
obj=$1
for tries in {1..5}; do
if ${KUBECTL} --server="127.0.0.1:8080" create --validate=true -f ${obj}; then
return
fi
echo "++ ${obj} failed, attempt ${try} (sleeping 5) ++"
sleep 5
done
}
echo "== Kubernetes addon manager started at $(date -Is) =="
for obj in $(find /etc/kubernetes/addons -name \*.yaml); do
${KUBECTL} --server="127.0.0.1:8080" create -f ${obj} &
create-object ${obj} &
echo "++ addon ${obj} started in pid $! ++"
done
noerrors="true"