From 15213d0a1c056ab0ddf60b67abdfbc1801567746 Mon Sep 17 00:00:00 2001 From: Paul Morie Date: Mon, 13 Jun 2016 15:10:46 -0400 Subject: [PATCH 1/2] Improve developer docs on unit and integration testing --- docs/devel/testing.md | 114 +++++++++++++++++++++++++++++------------- 1 file changed, 80 insertions(+), 34 deletions(-) diff --git a/docs/devel/testing.md b/docs/devel/testing.md index 72f1c32823e..c0e622f381b 100644 --- a/docs/devel/testing.md +++ b/docs/devel/testing.md @@ -29,7 +29,7 @@ Documentation for other releases can be found at # Testing guide -Updated: 5/3/2016 +Updated: 5/21/2016 **Table of Contents** @@ -37,19 +37,23 @@ Updated: 5/3/2016 - [Testing guide](#testing-guide) - [Unit tests](#unit-tests) - [Run all unit tests](#run-all-unit-tests) - - [Run some unit tests](#run-some-unit-tests) + - [Set go flags during unit tests](#set-go-flags-during-unit-tests) + - [Run unit tests from certain packages](#run-unit-tests-from-certain-packages) + - [Run specific unit test cases in a package](#run-specific-unit-test-cases-in-a-package) - [Stress running unit tests](#stress-running-unit-tests) - [Unit test coverage](#unit-test-coverage) - [Benchmark unit tests](#benchmark-unit-tests) - [Integration tests](#integration-tests) - [Install etcd dependency](#install-etcd-dependency) - [Run integration tests](#run-integration-tests) + - [Run a specific integration test](#run-a-specific-integration-test) - [End-to-End tests](#end-to-end-tests) This assumes you already read the [development guide](development.md) to -install go, godeps, and configure your git client. +install go, godeps, and configure your git client. All command examples are +relative to the `kubernetes` root directory. Before sending pull requests you should at least make sure your changes have passed both unit and integration tests. @@ -62,8 +66,8 @@ passing, so it is often a good idea to make sure the e2e tests work as well. * Unit tests should be fully hermetic - Only access resources in the test binary. * All packages and any significant files require unit tests. -* The preferred method of testing multiple scenarios or inputs -is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) +* The preferred method of testing multiple scenarios or input is + [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) - Example: [TestNamespaceAuthorization](../../test/integration/auth_test.go) * Unit tests must pass on OS X and Windows platforms. - Tests using linux-specific features must be skipped or compiled out. @@ -73,32 +77,59 @@ is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) ### Run all unit tests +The `hack/test-go.sh` script is the entrypoint for running the unit tests that +ensures that `GOPATH` is set up correctly. If you have `GOPATH` set up +correctly, you can also just use `go test` directly. + ```sh cd kubernetes hack/test-go.sh # Run all unit tests. ``` -### Run some unit tests +### Set go flags during unit tests + +You can set [go flags](https://golang.org/cmd/go/) by setting the +`KUBE_GOFLAGS` environment variable. + +### Run unit tests from certain packages + +The `hack/test-go.sh` script accepts packages as arguments; the +`k8s.io/kubernetes` prefix is added automatically to these: ```sh -cd kubernetes - -# Run all tests under pkg (requires client to be in $GOPATH/src/k8s.io) -go test ./pkg/... - -# Run all tests in the pkg/api (but not subpackages) -go test ./pkg/api +hack/test-go.sh pkg/api # run tests for pkg/api +hack/test-go.sh pkg/api pkg/kubelet # run tests for pkg/api and pkg/kubelet ``` +In a shell, it's often handy to use brace expansion: + +```sh +hack/test-go.sh pkg/{api,kubelet} # run tests for pkg/api and pkg/kubelet +``` + +### Run specific unit test cases in a package + +You can set the test args using the `KUBE_TEST_ARGS` environment variable. +You can use this to pass the `-run` argument to `go test`, which accepts a +regular expression for the name of the test that should be run. + +```sh +# Runs TestValidatePod in pkg/api/validation with the verbose flag set +KUBE_GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$' hack/test-go.sh pkg/api/validation + +# Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation +KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$" hack/test-go.sh pkg/api/validation +``` + +For other supported test flags, see the [golang +documentation](https://golang.org/cmd/go/#hdr-Description_of_testing_flags). + ### Stress running unit tests Running the same tests repeatedly is one way to root out flakes. You can do this efficiently. - ```sh -cd kubernetes - # Have 2 workers run all tests 5 times each (10 total iterations). hack/test-go.sh -p 2 -i 5 ``` @@ -112,43 +143,39 @@ Currently, collecting coverage is only supported for the Go unit tests. To run all unit tests and generate an HTML coverage report, run the following: ```sh -cd kubernetes KUBE_COVER=y hack/test-go.sh ``` -At the end of the run, an the HTML report will be generated with the path printed to stdout. +At the end of the run, an the HTML report will be generated with the path +printed to stdout. -To run tests and collect coverage in only one package, pass its relative path under the `kubernetes` directory as an argument, for example: +To run tests and collect coverage in only one package, pass its relative path +under the `kubernetes` directory as an argument, for example: ```sh -cd kubernetes KUBE_COVER=y hack/test-go.sh pkg/kubectl ``` -Multiple arguments can be passed, in which case the coverage results will be combined for all tests run. - -Coverage results for the project can also be viewed on [Coveralls](https://coveralls.io/r/kubernetes/kubernetes), and are continuously updated as commits are merged. Additionally, all pull requests which spawn a Travis build will report unit test coverage results to Coveralls. Coverage reports from before the Kubernetes Github organization was created can be found [here](https://coveralls.io/r/GoogleCloudPlatform/kubernetes). +Multiple arguments can be passed, in which case the coverage results will be +combined for all tests run. ### Benchmark unit tests To run benchmark tests, you'll typically use something like: ```sh -cd kubernetes go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch ``` This will do the following: -1. `-run=XXX` will turn off regular unit tests - * Technically it will run test methods with XXX in the name. +1. `-run=XXX` is a regular expression filter on the name of test cases to run 2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name * See `grep -nr BenchmarkWatch .` for examples 3. `-benchmem` enables memory allocation stats See `go help test` and `go help testflag` for additional info. - ## Integration tests * Integration tests should only access other resources on the local machine @@ -158,26 +185,24 @@ See `go help test` and `go help testflag` for additional info. * The preferred method of testing multiple scenarios or inputs is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) - Example: [TestNamespaceAuthorization](../../test/integration/auth_test.go) -* Integration tests must run in parallel - - Each test should create its own master, httpserver and config. +* Each test should create its own master, httpserver and config. - Example: [TestPodUpdateActiveDeadlineSeconds](../../test/integration/pods_test.go) * See [coding conventions](coding-conventions.md). ### Install etcd dependency -Kubernetes integration tests require your PATH to include an [etcd](https://github.com/coreos/etcd/releases) installation. -Kubernetes includes a script to help install etcd on your machine. +Kubernetes integration tests require your `PATH` to include an +[etcd](https://github.com/coreos/etcd/releases) installation. Kubernetes +includes a script to help install etcd on your machine. ```sh # Install etcd and add to PATH # Option a) install inside kubernetes root -cd kubernetes hack/install-etcd.sh # Installs in ./third_party/etcd echo export PATH="$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH # Option b) install manually -cd kubernetes grep -E "image.*etcd" cluster/saltbase/etcd/etcd.manifest # Find version # Install that version using yum/apt-get/etc echo export PATH="$PATH:" >> ~/.profile # Add to PATH @@ -185,11 +210,32 @@ echo export PATH="$PATH:" >> ~/.profile # Add to PATH ### Run integration tests +The integration tests are run using the `hack/test-integration.sh` script. +The Kubernetes integration tests are writting using the normal golang testing +package but expect to have a running etcd instance to connect to. The `test- +integration.sh` script wraps `hack/test-go.sh` and sets up an etcd instance +for the integration tests to use. + ```sh -cd kubernetes hack/test-integration.sh # Run all integration tests. ``` +This script runs the golang tests in package +[`test/integration`](../../test/integration/) +and a special watch cache test in `cmd/integration/integration.go`. + +### Run a specific integration test + +You can use also use the `KUBE_TEST_ARGS` environment variable with the `hack +/test-integration.sh` script to run a specific integration test case: + +```sh +# Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set. +KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ^TestPodUpdateActiveDeadlineSeconds$" hack/test-integration.sh +``` + +If you set `KUBE_TEST_ARGS`, the test case will be run with only the `v1` API +version and the watch cache test is skipped. ## End-to-End tests From 25f25cbafd8aabe32bce2ab8d6b652b9c622e19b Mon Sep 17 00:00:00 2001 From: Paul Morie Date: Mon, 13 Jun 2016 15:11:02 -0400 Subject: [PATCH 2/2] Improve debugging experience for single integration test case --- docs/devel/testing.md | 2 +- hack/test-integration.sh | 22 ++++++++++++++++------ 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/devel/testing.md b/docs/devel/testing.md index c0e622f381b..13127609219 100644 --- a/docs/devel/testing.md +++ b/docs/devel/testing.md @@ -146,7 +146,7 @@ To run all unit tests and generate an HTML coverage report, run the following: KUBE_COVER=y hack/test-go.sh ``` -At the end of the run, an the HTML report will be generated with the path +At the end of the run, an HTML report will be generated with the path printed to stdout. To run tests and collect coverage in only one package, pass its relative path diff --git a/hack/test-integration.sh b/hack/test-integration.sh index 9f47d1e257a..3105122f38b 100755 --- a/hack/test-integration.sh +++ b/hack/test-integration.sh @@ -37,28 +37,34 @@ KUBE_TEST_API_VERSIONS=${KUBE_TEST_API_VERSIONS:-"v1,extensions/v1beta1;v1,autos KUBE_TIMEOUT="-timeout 600s" KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY:-"-1"} LOG_LEVEL=${LOG_LEVEL:-2} +KUBE_TEST_ARGS=${KUBE_TEST_ARGS:-} cleanup() { + kube::log::status "Cleaning up etcd" kube::etcd::cleanup kube::log::status "Integration test cleanup complete" } runTests() { + kube::log::status "Starting etcd instance" kube::etcd::start - kube::log::status "Running integration test cases" # TODO: Re-enable race detection when we switch to a thread-safe etcd client # KUBE_RACE="-race" - KUBE_GOFLAGS="-tags 'integration no-docker' " \ + KUBE_GOFLAGS="${KUBE_GOFLAGS:-} -tags 'integration no-docker'" \ KUBE_RACE="" \ KUBE_TIMEOUT="${KUBE_TIMEOUT}" \ KUBE_TEST_API_VERSIONS="$1" \ "${KUBE_ROOT}/hack/test-go.sh" test/integration - kube::log::status "Running integration test scenario with watch cache on" - KUBE_TEST_API_VERSIONS="$1" "${KUBE_OUTPUT_HOSTBIN}/integration" --v=${LOG_LEVEL} \ - --max-concurrency="${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY}" --watch-cache=true + # Run the watch cache tests + # KUBE_TEST_ARGS doesn't mean anything to the watch cache test. + if [[ -z "${KUBE_TEST_ARGS}" ]]; then + kube::log::status "Running integration test scenario with watch cache on" + KUBE_TEST_API_VERSIONS="$1" "${KUBE_OUTPUT_HOSTBIN}/integration" --v=${LOG_LEVEL} \ + --max-concurrency="${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY}" --watch-cache=true + fi cleanup } @@ -73,12 +79,16 @@ checkEtcdOnPath() { checkEtcdOnPath - "${KUBE_ROOT}/hack/build-go.sh" "$@" cmd/integration # Run cleanup to stop etcd on interrupt or other kill signal. trap cleanup EXIT +# If a test case is specified, just run once with v1 API version and exit +if [[ -n "${KUBE_TEST_ARGS}" ]]; then + runTests v1 +fi + # Convert the CSV to an array of API versions to test IFS=';' read -a apiVersions <<< "${KUBE_TEST_API_VERSIONS}" for apiVersion in "${apiVersions[@]}"; do