From fd4fee788bf2dc76c154ccc94574bba726f7f087 Mon Sep 17 00:00:00 2001 From: Erick Fejta Date: Sun, 6 Mar 2016 19:07:34 -0800 Subject: [PATCH] Add simplified testing instructions and etcd installation check. --- docs/devel/development.md | 79 ++------------- docs/devel/testing.md | 198 ++++++++++++++++++++++++++++++++++++++ hack/test-integration.sh | 11 +++ 3 files changed, 216 insertions(+), 72 deletions(-) create mode 100644 docs/devel/testing.md diff --git a/docs/devel/development.md b/docs/devel/development.md index e1a01015d13..5b8dbdad73a 100644 --- a/docs/devel/development.md +++ b/docs/devel/development.md @@ -168,7 +168,7 @@ export PATH=$PATH:$GOPATH/bin ### Using godep -Here's a quick walkthrough of one way to use godeps to add or update a Kubernetes dependency into Godeps/_workspace. For more details, please see the instructions in [godep's documentation](https://github.com/tools/godep). +Here's a quick walkthrough of one way to use godeps to add or update a Kubernetes dependency into Godeps/\_workspace. For more details, please see the instructions in [godep's documentation](https://github.com/tools/godep). 1) Devote a directory to this endeavor: @@ -230,83 +230,18 @@ Please send dependency updates in separate commits within your PR, for easier re 6) If you updated the Godeps, please also update `Godeps/LICENSES` by running `hack/update-godep-licenses.sh`. +## Testing -## Unit tests +Three basic commands let you run unit, integration and/or unit tests: ```sh cd kubernetes -hack/test-go.sh +hack/test-go.sh # Run unit tests +hack/test-integration.sh # Run integration tests, requires etcd +go run hack/e2e.go -v --build --up --test --down # Run e2e tests ``` -Alternatively, you could also run: - -```sh -cd kubernetes -godep go test ./... -``` - -If you only want to run unit tests in one package, you could run ``godep go test`` under the package directory. For example, the following commands will run all unit tests in package kubelet: - -```console -$ cd kubernetes # step into the kubernetes directory. -$ cd pkg/kubelet -$ godep go test -# some output from unit tests -PASS -ok k8s.io/kubernetes/pkg/kubelet 0.317s -``` - -## Coverage - -Currently, collecting coverage is only supported for the Go unit tests. - -To run all unit tests and generate an HTML coverage report, run the following: - -```sh -cd kubernetes -KUBE_COVER=y hack/test-go.sh -``` - -At the end of the run, an the HTML report will be generated with the path printed to stdout. - -To run tests and collect coverage in only one package, pass its relative path under the `kubernetes` directory as an argument, for example: - -```sh -cd kubernetes -KUBE_COVER=y hack/test-go.sh pkg/kubectl -``` - -Multiple arguments can be passed, in which case the coverage results will be combined for all tests run. - -Coverage results for the project can also be viewed on [Coveralls](https://coveralls.io/r/kubernetes/kubernetes), and are continuously updated as commits are merged. Additionally, all pull requests which spawn a Travis build will report unit test coverage results to Coveralls. Coverage reports from before the Kubernetes Github organization was created can be found [here](https://coveralls.io/r/GoogleCloudPlatform/kubernetes). - -## Integration tests - -You need an [etcd](https://github.com/coreos/etcd/releases) in your path. To download a copy of the latest version used by Kubernetes, either - * run `hack/install-etcd.sh`, which will download etcd to `third_party/etcd`, and then set your `PATH` to include `third_party/etcd`. - * inspect `cluster/saltbase/salt/etcd/etcd.manifest` for the correct version, and then manually download and install it to some place in your `PATH`. - -```sh -cd kubernetes -hack/test-integration.sh -``` - -## End-to-End tests - -See [End-to-End Testing in Kubernetes](e2e-tests.md). - -## Testing out flaky tests - -[Instructions here](flaky-tests.md) - -## Benchmarking - -To run benchmark tests, you'll typically use something like: - - $ godep go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch - -The `-run=XXX` prevents normal unit tests for running, while `-bench` is a regexp for selecting which benchmarks to run. -See `go test -h` for more instructions on generating profiles from benchmarks. +See the [testing guide](testing.md) for additional information and scenarios. ## Regenerating the CLI documentation diff --git a/docs/devel/testing.md b/docs/devel/testing.md new file mode 100644 index 00000000000..03eddeb48e1 --- /dev/null +++ b/docs/devel/testing.md @@ -0,0 +1,198 @@ + + + + +WARNING +WARNING +WARNING +WARNING +WARNING + +

PLEASE NOTE: This document applies to the HEAD of the source tree

+ +If you are using a released version of Kubernetes, you should +refer to the docs that go with that version. + +Documentation for other releases can be found at +[releases.k8s.io](http://releases.k8s.io). + +-- + + + + + +# Testing guide + +This assumes you already read the [development guide](development.md) to +install go, godeps and configure your git client. + +In order to send pull requests you need to make sure you changes pass +unit, integration tests. + +Kubernetes only merges pull requests when e2e tests are passing, so it is often +a good idea to make sure these work as well. + +## Unit tests + +Unit tests should be fully hermetic and +access no resources outside the test binary. + +### Run all unit tests + +```sh +cd kubernetes +hack/test-go.sh # Run all unit tests. +``` + +### Run some unit tests + +```sh +cd kubernetes + +# Run all tests under pkg (requires client to be in $GOPATH/src/k8s.io) +godep go test ./pkg/... + +# Run all tests in the pkg/api (but not subpackages) +godep go test ./pkg/api +``` + +### Stress running unit tests + +Running the same tests repeatedly is one way to root out flakes. +You can do this efficiently. + + +```sh +cd kubernetes + +# Have 2 workers run all tests 5 times each (10 total iterations). +hack/test-go.sh -p 2 -i 5 +``` + +For more advanced ideas please see [flaky-tests.md](flaky-tests.md). + +### Unit test coverage + +Currently, collecting coverage is only supported for the Go unit tests. + +To run all unit tests and generate an HTML coverage report, run the following: + +```sh +cd kubernetes +KUBE_COVER=y hack/test-go.sh +``` + +At the end of the run, an the HTML report will be generated with the path printed to stdout. + +To run tests and collect coverage in only one package, pass its relative path under the `kubernetes` directory as an argument, for example: + +```sh +cd kubernetes +KUBE_COVER=y hack/test-go.sh pkg/kubectl +``` + +Multiple arguments can be passed, in which case the coverage results will be combined for all tests run. + +Coverage results for the project can also be viewed on [Coveralls](https://coveralls.io/r/kubernetes/kubernetes), and are continuously updated as commits are merged. Additionally, all pull requests which spawn a Travis build will report unit test coverage results to Coveralls. Coverage reports from before the Kubernetes Github organization was created can be found [here](https://coveralls.io/r/GoogleCloudPlatform/kubernetes). + +### Benchmark unit tests + +To run benchmark tests, you'll typically use something like: + +```sh +cd kubernetes +godep go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch +``` + +This will do the following: + +1. `-run=XXX` will turn off regular unit tests + * Technically it will run test methods with XXX in the name. +2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name + * See `grep -nr BenchmarkWatch .` for examples +3. `-benchmem` enables memory allocation stats + +See `go help test` and `go help testflag` for additional info. + + +## Integration tests + +Integration tests should only access other resources on the local machine, +most commonly etcd or a kubernetes/docker binary. + +### Install etcd dependency + +Kubernetes integration tests require your PATH to include an [etcd](https://github.com/coreos/etcd/releases) installation. +Kubernetes includes a script to help install etcd on your machine. + +```sh +# Install etcd and add to PATH + +# Option a) install inside kubernetes root +cd kubernetes +hack/install-etcd.sh # Installs in ./third_party/etcd +echo export PATH="$PATH:$(pwd)/third_party/etcd" >> .profile # Add to PATH + +# Option b) install manually +cd kubernetes +grep -E "image.*etcd" cluster/saltbase/etcd/etcd.manifest # Find version +# Install that version using yum/apt-get/etc +echo export PATH="$PATH:" >> .profile # Add to PATH +``` + +### Run integration tests + +```sh +cd kubernetes +hack/test-integration.sh # Run all integration tests. +``` + + +## End-to-End tests + +### e2e test philosophy + +In general passing unit and integration tests should provide sufficient +confidence +to allow code to merge. If that is not the case, please *invest more time adding +unit and integration test coverage*. These tests run faster and a smaller failure domain. + +However, end-to-end (e2e) tests provide maximum confidence that +the system is working in exchange for reduced performance and a +higher debugging cost. + +e2e tests deploy a real kubernetes cluster of real nodes on a concrete provider such as GCE. The tests then manipulate the cluster in certain ways and assert the expected results. + +For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md). + +### Running e2e tests + +```sh +cd kubernetes +go run hack/e2e.go -v --build --up --test --down + +# Change code, run unit and integration tests +# Push to an existing cluster, or bring up a cluster if it's down. +go run hack/e2e.go -v --pushup + +# Run all tests on an already up cluster +go run hack/e2e.go -v --test + +# Run only conformance tests +go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]" + +# Run tests on a specific provider +KUBERNETES_PROVIDER=aws go run hack/e2e.go --build --pushup --test --down +``` + +For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md). + + +[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/testing.md?pixel)]() + diff --git a/hack/test-integration.sh b/hack/test-integration.sh index d97980287e9..a13b8eb9d64 100755 --- a/hack/test-integration.sh +++ b/hack/test-integration.sh @@ -66,6 +66,17 @@ runTests() { cleanup } +checkEtcdOnPath() { + kube::log::status "Checking etcd is on PATH" + which etcd && return + kube::log::status "Cannot find etcd, cannot run integration tests." + kube::log::status "Please see docs/devel/testing.md for instructions." + return 1 +} + +checkEtcdOnPath + + KUBE_API_VERSIONS="v1,autoscaling/v1,batch/v1,extensions/v1beta1" "${KUBE_ROOT}/hack/build-go.sh" "$@" cmd/integration # Run cleanup to stop etcd on interrupt or other kill signal.