diff --git a/docs/devel/testing.md b/docs/devel/testing.md index 475adc12648..4ae988595c5 100644 --- a/docs/devel/testing.md +++ b/docs/devel/testing.md @@ -34,7 +34,7 @@ Documentation for other releases can be found at # Testing guide -Updated: 5/3/2016 +Updated: 5/21/2016 **Table of Contents** @@ -42,19 +42,23 @@ Updated: 5/3/2016 - [Testing guide](#testing-guide) - [Unit tests](#unit-tests) - [Run all unit tests](#run-all-unit-tests) - - [Run some unit tests](#run-some-unit-tests) + - [Set go flags during unit tests](#set-go-flags-during-unit-tests) + - [Run unit tests from certain packages](#run-unit-tests-from-certain-packages) + - [Run specific unit test cases in a package](#run-specific-unit-test-cases-in-a-package) - [Stress running unit tests](#stress-running-unit-tests) - [Unit test coverage](#unit-test-coverage) - [Benchmark unit tests](#benchmark-unit-tests) - [Integration tests](#integration-tests) - [Install etcd dependency](#install-etcd-dependency) - [Run integration tests](#run-integration-tests) + - [Run a specific integration test](#run-a-specific-integration-test) - [End-to-End tests](#end-to-end-tests) This assumes you already read the [development guide](development.md) to -install go, godeps, and configure your git client. +install go, godeps, and configure your git client. All command examples are +relative to the `kubernetes` root directory. Before sending pull requests you should at least make sure your changes have passed both unit and integration tests. @@ -67,8 +71,8 @@ passing, so it is often a good idea to make sure the e2e tests work as well. * Unit tests should be fully hermetic - Only access resources in the test binary. * All packages and any significant files require unit tests. -* The preferred method of testing multiple scenarios or inputs -is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) +* The preferred method of testing multiple scenarios or input is + [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) - Example: [TestNamespaceAuthorization](../../test/integration/auth_test.go) * Unit tests must pass on OS X and Windows platforms. - Tests using linux-specific features must be skipped or compiled out. @@ -78,32 +82,59 @@ is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) ### Run all unit tests +The `hack/test-go.sh` script is the entrypoint for running the unit tests that +ensures that `GOPATH` is set up correctly. If you have `GOPATH` set up +correctly, you can also just use `go test` directly. + ```sh cd kubernetes hack/test-go.sh # Run all unit tests. ``` -### Run some unit tests +### Set go flags during unit tests + +You can set [go flags](https://golang.org/cmd/go/) by setting the +`KUBE_GOFLAGS` environment variable. + +### Run unit tests from certain packages + +The `hack/test-go.sh` script accepts packages as arguments; the +`k8s.io/kubernetes` prefix is added automatically to these: ```sh -cd kubernetes - -# Run all tests under pkg (requires client to be in $GOPATH/src/k8s.io) -go test ./pkg/... - -# Run all tests in the pkg/api (but not subpackages) -go test ./pkg/api +hack/test-go.sh pkg/api # run tests for pkg/api +hack/test-go.sh pkg/api pkg/kubelet # run tests for pkg/api and pkg/kubelet ``` +In a shell, it's often handy to use brace expansion: + +```sh +hack/test-go.sh pkg/{api,kubelet} # run tests for pkg/api and pkg/kubelet +``` + +### Run specific unit test cases in a package + +You can set the test args using the `KUBE_TEST_ARGS` environment variable. +You can use this to pass the `-run` argument to `go test`, which accepts a +regular expression for the name of the test that should be run. + +```sh +# Runs TestValidatePod in pkg/api/validation with the verbose flag set +KUBE_GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$' hack/test-go.sh pkg/api/validation + +# Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation +KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$" hack/test-go.sh pkg/api/validation +``` + +For other supported test flags, see the [golang +documentation](https://golang.org/cmd/go/#hdr-Description_of_testing_flags). + ### Stress running unit tests Running the same tests repeatedly is one way to root out flakes. You can do this efficiently. - ```sh -cd kubernetes - # Have 2 workers run all tests 5 times each (10 total iterations). hack/test-go.sh -p 2 -i 5 ``` @@ -117,43 +148,39 @@ Currently, collecting coverage is only supported for the Go unit tests. To run all unit tests and generate an HTML coverage report, run the following: ```sh -cd kubernetes KUBE_COVER=y hack/test-go.sh ``` -At the end of the run, an the HTML report will be generated with the path printed to stdout. +At the end of the run, an HTML report will be generated with the path +printed to stdout. -To run tests and collect coverage in only one package, pass its relative path under the `kubernetes` directory as an argument, for example: +To run tests and collect coverage in only one package, pass its relative path +under the `kubernetes` directory as an argument, for example: ```sh -cd kubernetes KUBE_COVER=y hack/test-go.sh pkg/kubectl ``` -Multiple arguments can be passed, in which case the coverage results will be combined for all tests run. - -Coverage results for the project can also be viewed on [Coveralls](https://coveralls.io/r/kubernetes/kubernetes), and are continuously updated as commits are merged. Additionally, all pull requests which spawn a Travis build will report unit test coverage results to Coveralls. Coverage reports from before the Kubernetes Github organization was created can be found [here](https://coveralls.io/r/GoogleCloudPlatform/kubernetes). +Multiple arguments can be passed, in which case the coverage results will be +combined for all tests run. ### Benchmark unit tests To run benchmark tests, you'll typically use something like: ```sh -cd kubernetes go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch ``` This will do the following: -1. `-run=XXX` will turn off regular unit tests - * Technically it will run test methods with XXX in the name. +1. `-run=XXX` is a regular expression filter on the name of test cases to run 2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name * See `grep -nr BenchmarkWatch .` for examples 3. `-benchmem` enables memory allocation stats See `go help test` and `go help testflag` for additional info. - ## Integration tests * Integration tests should only access other resources on the local machine @@ -163,26 +190,24 @@ See `go help test` and `go help testflag` for additional info. * The preferred method of testing multiple scenarios or inputs is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests) - Example: [TestNamespaceAuthorization](../../test/integration/auth_test.go) -* Integration tests must run in parallel - - Each test should create its own master, httpserver and config. +* Each test should create its own master, httpserver and config. - Example: [TestPodUpdateActiveDeadlineSeconds](../../test/integration/pods_test.go) * See [coding conventions](coding-conventions.md). ### Install etcd dependency -Kubernetes integration tests require your PATH to include an [etcd](https://github.com/coreos/etcd/releases) installation. -Kubernetes includes a script to help install etcd on your machine. +Kubernetes integration tests require your `PATH` to include an +[etcd](https://github.com/coreos/etcd/releases) installation. Kubernetes +includes a script to help install etcd on your machine. ```sh # Install etcd and add to PATH # Option a) install inside kubernetes root -cd kubernetes hack/install-etcd.sh # Installs in ./third_party/etcd echo export PATH="$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH # Option b) install manually -cd kubernetes grep -E "image.*etcd" cluster/saltbase/etcd/etcd.manifest # Find version # Install that version using yum/apt-get/etc echo export PATH="$PATH:" >> ~/.profile # Add to PATH @@ -190,11 +215,32 @@ echo export PATH="$PATH:" >> ~/.profile # Add to PATH ### Run integration tests +The integration tests are run using the `hack/test-integration.sh` script. +The Kubernetes integration tests are writting using the normal golang testing +package but expect to have a running etcd instance to connect to. The `test- +integration.sh` script wraps `hack/test-go.sh` and sets up an etcd instance +for the integration tests to use. + ```sh -cd kubernetes hack/test-integration.sh # Run all integration tests. ``` +This script runs the golang tests in package +[`test/integration`](../../test/integration/) +and a special watch cache test in `cmd/integration/integration.go`. + +### Run a specific integration test + +You can use also use the `KUBE_TEST_ARGS` environment variable with the `hack +/test-integration.sh` script to run a specific integration test case: + +```sh +# Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set. +KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ^TestPodUpdateActiveDeadlineSeconds$" hack/test-integration.sh +``` + +If you set `KUBE_TEST_ARGS`, the test case will be run with only the `v1` API +version and the watch cache test is skipped. ## End-to-End tests diff --git a/hack/test-integration.sh b/hack/test-integration.sh index 9f47d1e257a..3105122f38b 100755 --- a/hack/test-integration.sh +++ b/hack/test-integration.sh @@ -37,28 +37,34 @@ KUBE_TEST_API_VERSIONS=${KUBE_TEST_API_VERSIONS:-"v1,extensions/v1beta1;v1,autos KUBE_TIMEOUT="-timeout 600s" KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY:-"-1"} LOG_LEVEL=${LOG_LEVEL:-2} +KUBE_TEST_ARGS=${KUBE_TEST_ARGS:-} cleanup() { + kube::log::status "Cleaning up etcd" kube::etcd::cleanup kube::log::status "Integration test cleanup complete" } runTests() { + kube::log::status "Starting etcd instance" kube::etcd::start - kube::log::status "Running integration test cases" # TODO: Re-enable race detection when we switch to a thread-safe etcd client # KUBE_RACE="-race" - KUBE_GOFLAGS="-tags 'integration no-docker' " \ + KUBE_GOFLAGS="${KUBE_GOFLAGS:-} -tags 'integration no-docker'" \ KUBE_RACE="" \ KUBE_TIMEOUT="${KUBE_TIMEOUT}" \ KUBE_TEST_API_VERSIONS="$1" \ "${KUBE_ROOT}/hack/test-go.sh" test/integration - kube::log::status "Running integration test scenario with watch cache on" - KUBE_TEST_API_VERSIONS="$1" "${KUBE_OUTPUT_HOSTBIN}/integration" --v=${LOG_LEVEL} \ - --max-concurrency="${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY}" --watch-cache=true + # Run the watch cache tests + # KUBE_TEST_ARGS doesn't mean anything to the watch cache test. + if [[ -z "${KUBE_TEST_ARGS}" ]]; then + kube::log::status "Running integration test scenario with watch cache on" + KUBE_TEST_API_VERSIONS="$1" "${KUBE_OUTPUT_HOSTBIN}/integration" --v=${LOG_LEVEL} \ + --max-concurrency="${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY}" --watch-cache=true + fi cleanup } @@ -73,12 +79,16 @@ checkEtcdOnPath() { checkEtcdOnPath - "${KUBE_ROOT}/hack/build-go.sh" "$@" cmd/integration # Run cleanup to stop etcd on interrupt or other kill signal. trap cleanup EXIT +# If a test case is specified, just run once with v1 API version and exit +if [[ -n "${KUBE_TEST_ARGS}" ]]; then + runTests v1 +fi + # Convert the CSV to an array of API versions to test IFS=';' read -a apiVersions <<< "${KUBE_TEST_API_VERSIONS}" for apiVersion in "${apiVersions[@]}"; do