github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-07-24 12:15:52 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #26032 from pmorie/improve-tests

Browse Source 
Testing improvements
This commit is contained in:
Daniel Smith 2016-06-15 15:44:33 -07:00 committed by GitHub
commit 4a040d11ca
2 changed files with 96 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

114
docs/devel/testing.md
View File

@ -34,7 +34,7 @@ Documentation for other releases can be found at
# Testing guide
Updated: 5/3/2016
Updated: 5/21/2016
**Table of Contents**
<!-- BEGIN MUNGE: GENERATED_TOC -->
@ -42,19 +42,23 @@ Updated: 5/3/2016
- [Testing guide](#testing-guide)
- [Unit tests](#unit-tests)
- [Run all unit tests](#run-all-unit-tests)
- [Run some unit tests](#run-some-unit-tests)
- [Set go flags during unit tests](#set-go-flags-during-unit-tests)
- [Run unit tests from certain packages](#run-unit-tests-from-certain-packages)
- [Run specific unit test cases in a package](#run-specific-unit-test-cases-in-a-package)
- [Stress running unit tests](#stress-running-unit-tests)
- [Unit test coverage](#unit-test-coverage)
- [Benchmark unit tests](#benchmark-unit-tests)
- [Integration tests](#integration-tests)
- [Install etcd dependency](#install-etcd-dependency)
- [Run integration tests](#run-integration-tests)
- [Run a specific integration test](#run-a-specific-integration-test)
- [End-to-End tests](#end-to-end-tests)
<!-- END MUNGE: GENERATED_TOC -->
This assumes you already read the [development guide](development.md) to
install go, godeps, and configure your git client.
install go, godeps, and configure your git client. All command examples are
relative to the `kubernetes` root directory.
Before sending pull requests you should at least make sure your changes have
passed both unit and integration tests.
@ -67,8 +71,8 @@ passing, so it is often a good idea to make sure the e2e tests work as well.
* Unit tests should be fully hermetic
- Only access resources in the test binary.
* All packages and any significant files require unit tests.
* The preferred method of testing multiple scenarios or inputs
is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
* The preferred method of testing multiple scenarios or input is
[table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](../../test/integration/auth_test.go)
* Unit tests must pass on OS X and Windows platforms.
- Tests using linux-specific features must be skipped or compiled out.
@ -78,32 +82,59 @@ is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
### Run all unit tests
The `hack/test-go.sh` script is the entrypoint for running the unit tests that
ensures that `GOPATH` is set up correctly. If you have `GOPATH` set up
correctly, you can also just use `go test` directly.
```sh
cd kubernetes
hack/test-go.sh # Run all unit tests.
```
### Run some unit tests
### Set go flags during unit tests
You can set [go flags](https://golang.org/cmd/go/) by setting the
`KUBE_GOFLAGS` environment variable.
### Run unit tests from certain packages
The `hack/test-go.sh` script accepts packages as arguments; the
`k8s.io/kubernetes` prefix is added automatically to these:
```sh
cd kubernetes
# Run all tests under pkg (requires client to be in $GOPATH/src/k8s.io)
go test ./pkg/...
# Run all tests in the pkg/api (but not subpackages)
go test ./pkg/api
hack/test-go.sh pkg/api # run tests for pkg/api
hack/test-go.sh pkg/api pkg/kubelet # run tests for pkg/api and pkg/kubelet
```
In a shell, it's often handy to use brace expansion:
```sh
hack/test-go.sh pkg/{api,kubelet} # run tests for pkg/api and pkg/kubelet
```
### Run specific unit test cases in a package
You can set the test args using the `KUBE_TEST_ARGS` environment variable.
You can use this to pass the `-run` argument to `go test`, which accepts a
regular expression for the name of the test that should be run.
```sh
# Runs TestValidatePod in pkg/api/validation with the verbose flag set
KUBE_GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$' hack/test-go.sh pkg/api/validation
# Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation
KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$" hack/test-go.sh pkg/api/validation
```
For other supported test flags, see the [golang
documentation](https://golang.org/cmd/go/#hdr-Description_of_testing_flags).
### Stress running unit tests
Running the same tests repeatedly is one way to root out flakes.
You can do this efficiently.
```sh
cd kubernetes
# Have 2 workers run all tests 5 times each (10 total iterations).
hack/test-go.sh -p 2 -i 5
```
@ -117,43 +148,39 @@ Currently, collecting coverage is only supported for the Go unit tests.
To run all unit tests and generate an HTML coverage report, run the following:
```sh
cd kubernetes
KUBE_COVER=y hack/test-go.sh
```
At the end of the run, an the HTML report will be generated with the path printed to stdout.
At the end of the run, an HTML report will be generated with the path
printed to stdout.
To run tests and collect coverage in only one package, pass its relative path under the `kubernetes` directory as an argument, for example:
To run tests and collect coverage in only one package, pass its relative path
under the `kubernetes` directory as an argument, for example:
```sh
cd kubernetes
KUBE_COVER=y hack/test-go.sh pkg/kubectl
```
Multiple arguments can be passed, in which case the coverage results will be combined for all tests run.
Coverage results for the project can also be viewed on [Coveralls](https://coveralls.io/r/kubernetes/kubernetes), and are continuously updated as commits are merged. Additionally, all pull requests which spawn a Travis build will report unit test coverage results to Coveralls. Coverage reports from before the Kubernetes Github organization was created can be found [here](https://coveralls.io/r/GoogleCloudPlatform/kubernetes).
Multiple arguments can be passed, in which case the coverage results will be
combined for all tests run.
### Benchmark unit tests
To run benchmark tests, you'll typically use something like:
```sh
cd kubernetes
go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch
```
This will do the following:
1. `-run=XXX` will turn off regular unit tests
* Technically it will run test methods with XXX in the name.
1. `-run=XXX` is a regular expression filter on the name of test cases to run
2. `-bench=BenchmarkWatch` will run test methods with BenchmarkWatch in the name
* See `grep -nr BenchmarkWatch .` for examples
3. `-benchmem` enables memory allocation stats
See `go help test` and `go help testflag` for additional info.
## Integration tests
* Integration tests should only access other resources on the local machine
@ -163,26 +190,24 @@ See `go help test` and `go help testflag` for additional info.
* The preferred method of testing multiple scenarios or inputs
is [table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Example: [TestNamespaceAuthorization](../../test/integration/auth_test.go)
* Integration tests must run in parallel
- Each test should create its own master, httpserver and config.
* Each test should create its own master, httpserver and config.
- Example: [TestPodUpdateActiveDeadlineSeconds](../../test/integration/pods_test.go)
* See [coding conventions](coding-conventions.md).
### Install etcd dependency
Kubernetes integration tests require your PATH to include an [etcd](https://github.com/coreos/etcd/releases) installation.
Kubernetes includes a script to help install etcd on your machine.
Kubernetes integration tests require your `PATH` to include an
[etcd](https://github.com/coreos/etcd/releases) installation. Kubernetes
includes a script to help install etcd on your machine.
```sh
# Install etcd and add to PATH
# Option a) install inside kubernetes root
cd kubernetes
hack/install-etcd.sh # Installs in ./third_party/etcd
echo export PATH="$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH
# Option b) install manually
cd kubernetes
grep -E "image.*etcd" cluster/saltbase/etcd/etcd.manifest # Find version
# Install that version using yum/apt-get/etc
echo export PATH="$PATH:<LOCATION>" >> ~/.profile # Add to PATH
@ -190,11 +215,32 @@ echo export PATH="$PATH:<LOCATION>" >> ~/.profile # Add to PATH
### Run integration tests
The integration tests are run using the `hack/test-integration.sh` script.
The Kubernetes integration tests are writting using the normal golang testing
package but expect to have a running etcd instance to connect to. The `test-
integration.sh` script wraps `hack/test-go.sh` and sets up an etcd instance
for the integration tests to use.
```sh
cd kubernetes
hack/test-integration.sh # Run all integration tests.
```
This script runs the golang tests in package
[`test/integration`](../../test/integration/)
and a special watch cache test in `cmd/integration/integration.go`.
### Run a specific integration test
You can use also use the `KUBE_TEST_ARGS` environment variable with the `hack
/test-integration.sh` script to run a specific integration test case:
```sh
# Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set.
KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ^TestPodUpdateActiveDeadlineSeconds$" hack/test-integration.sh
```
If you set `KUBE_TEST_ARGS`, the test case will be run with only the `v1` API
version and the watch cache test is skipped.
## End-to-End tests

22
hack/test-integration.sh
View File

@ -37,28 +37,34 @@ KUBE_TEST_API_VERSIONS=${KUBE_TEST_API_VERSIONS:-"v1,extensions/v1beta1;v1,autos
KUBE_TIMEOUT="-timeout 600s"
KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY:-"-1"}
LOG_LEVEL=${LOG_LEVEL:-2}
KUBE_TEST_ARGS=${KUBE_TEST_ARGS:-}
cleanup() {
kube::log::status "Cleaning up etcd"
kube::etcd::cleanup
kube::log::status "Integration test cleanup complete"
}
runTests() {
kube::log::status "Starting etcd instance"
kube::etcd::start
kube::log::status "Running integration test cases"
# TODO: Re-enable race detection when we switch to a thread-safe etcd client
# KUBE_RACE="-race"
KUBE_GOFLAGS="-tags 'integration no-docker' " \
KUBE_GOFLAGS="${KUBE_GOFLAGS:-} -tags 'integration no-docker'" \
KUBE_RACE="" \
KUBE_TIMEOUT="${KUBE_TIMEOUT}" \
KUBE_TEST_API_VERSIONS="$1" \
"${KUBE_ROOT}/hack/test-go.sh" test/integration
kube::log::status "Running integration test scenario with watch cache on"
KUBE_TEST_API_VERSIONS="$1" "${KUBE_OUTPUT_HOSTBIN}/integration" --v=${LOG_LEVEL} \
--max-concurrency="${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY}" --watch-cache=true
# Run the watch cache tests
# KUBE_TEST_ARGS doesn't mean anything to the watch cache test.
if [[ -z "${KUBE_TEST_ARGS}" ]]; then
kube::log::status "Running integration test scenario with watch cache on"
KUBE_TEST_API_VERSIONS="$1" "${KUBE_OUTPUT_HOSTBIN}/integration" --v=${LOG_LEVEL} \
--max-concurrency="${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY}" --watch-cache=true
fi
cleanup
}
@ -73,12 +79,16 @@ checkEtcdOnPath() {
checkEtcdOnPath
"${KUBE_ROOT}/hack/build-go.sh" "$@" cmd/integration
# Run cleanup to stop etcd on interrupt or other kill signal.
trap cleanup EXIT
# If a test case is specified, just run once with v1 API version and exit
if [[ -n "${KUBE_TEST_ARGS}" ]]; then
runTests v1
fi
# Convert the CSV to an array of API versions to test
IFS=';' read -a apiVersions <<< "${KUBE_TEST_API_VERSIONS}"
for apiVersion in "${apiVersions[@]}"; do