Modify() was replacing components one at a time: stop X, start X, stop Y, start Y, ... in version-skew order (apiserver last on downgrade). This caused a crash during downgrade: KCM-1.35 started against the still- running apiserver-1.36, passed its /healthz, and then immediately lose its connection when apiserver-1.36 was killed by the localupcluster. KCM-1.35 would reconnect to the not-yet-ready apiserver-1.35, hit a 403 RBAC error during controller initialization, and exit — because that initialization phase does not retry on RBAC errors. Fix by splitting Modify() into two phases: Phase 1 — stop all components to be replaced, in reverse startup order (kube-proxy down to apiserver), so dependent components release their connections before the apiserver is stopped. Phase 2 — start all replacement components in standard startup order (apiserver first), so each component connects to a fully-ready apiserver.
This directory contains a testsuite with automatic upgrade/downgrade tests for DRA. Conceptually this is like an integration test, in the sense that it starts/stops cluster components and runs tests against them. It has its own directory because it needs to be started differently than other integration tests or unit tests, which makes it more like an E2E suite.
The difference is that it starts Kubernetes components by running the actual binaries, relying on local-up-cluster.sh for the logic and configuration steps. local-up-cluster.sh needs additional permissions and preparations on the host.
To run it:
-
Make sure that hack/local-up-cluster.sh works:
- sudo must work
- Set env variables as necessary for your environment.
-
Ensure that /var/lib/kubelet/plugins, /var/lib/kubelet/plugins_registry, and /var/run/cdi are writable.
-
Build binaries with
make. -
Export
KUBERNETES_SERVER_BIN_DIR=$(pwd)/_output/local/bin/linux/amd64(or whatever is your GOOS/GOARCH and output directory). -
Optional: export
KUBERNETES_SERVER_CACHE_DIR=$(pwd)/_output/local/bin/linx/amd64/cache-dirto reuse downloaded release binaries across test invocations. -
Optional: set ARTIFACTS to store component log files persistently. Otherwise a test tmp directory is used.
-
Invoke as a Go test (no need for the ginkgo CLI), for example:
go test -v -count=1 -timeout=1h ./test/e2e_dra dlv test ./test/e2e_dra -- -test.v make test KUBE_TIMEOUT=-timeout=1h WHAT=test/e2e_dra FULL_LOG=true KUBE_TEST_ARGS="-count=1"
make test instead of make test-integration is intentional: local-up-cluster.sh
itself wants to start etcd. -count=1 ensures that test runs each time it is invoked.
-v/-test.v/FULL_LOG=true make the test output visible while the test runs.
To simplify starting from scratch, ./test/e2e_dra/run.sh cleans
up, sets permissions, and then invokes whatever command is specified on the
command line:
./test/e2e_dra/run.sh go test ./test/e2e_dra