From 0dc1f5da0416feb21a253976cd33af59076681cd Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Tue, 22 Mar 2016 13:09:31 -0700 Subject: [PATCH 1/6] Update conformance test policy Mostly doc updates and cruft removal - describe conformance test policy and howto in e2e-tests.md - rm e2e test info from testing.md in the name of DRY - rm cluster/test-conformance.sh; unusable in release tar, not e2e.go - update e2e test link in write-a-getting-started-guide.md --- cluster/test-conformance.sh | 30 ----------- docs/devel/e2e-tests.md | 36 +++++++++---- docs/devel/testing.md | 50 +------------------ docs/devel/writing-a-getting-started-guide.md | 2 +- 4 files changed, 29 insertions(+), 89 deletions(-) delete mode 100755 cluster/test-conformance.sh diff --git a/cluster/test-conformance.sh b/cluster/test-conformance.sh deleted file mode 100755 index 291a05b5b51..00000000000 --- a/cluster/test-conformance.sh +++ /dev/null @@ -1,30 +0,0 @@ -#!/bin/bash - -# Copyright 2015 The Kubernetes Authors All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Conformance Tests a running Kubernetes cluster. -# Validates that the cluster was deployed, is accessible, and at least -# satisfies end-to-end tests marked as being a part of the conformance suite. -# Emphasis on broad coverage and being non-destructive over thoroughness. - -set -o errexit -set -o nounset -set -o pipefail - -KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. - -TEST_ARGS="$@" - -exec "${KUBE_ROOT}/hack/conformance-test.sh" ${TEST_ARGS} diff --git a/docs/devel/e2e-tests.md b/docs/devel/e2e-tests.md index 913f60f3aa6..afa1337d969 100644 --- a/docs/devel/e2e-tests.md +++ b/docs/devel/e2e-tests.md @@ -77,12 +77,15 @@ go run hack/e2e.go -v --pushup # Run all tests go run hack/e2e.go -v --test -# Run tests matching the regex "\[Conformance\]" (the conformance tests) -go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]" +# Run tests matching the regex "\[Feature:Performance\]" +go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Feature:Performance\]" # Conversely, exclude tests that match the regex "Pods.*env" go run hack/e2e.go -v -test --test_args="--ginkgo.focus=Pods.*env" +# Run tests in parallel, skip any that must be run serially +GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.skip=\[Serial\]" + # Flags can be combined, and their actions will take place in this order: # --build, --push|--up|--pushup, --test|--tests=..., --down # @@ -96,9 +99,6 @@ KUBERNETES_PROVIDER=aws go run hack/e2e.go -v --build --pushup --test --down # kubectl output. go run hack/e2e.go -v -ctl='get events' go run hack/e2e.go -v -ctl='delete pod foobar' - -# Alternately, if you have the e2e cluster up and no desire to see the event stream, you can run ginkgo-e2e.sh directly: -hack/ginkgo-e2e.sh --ginkgo.focus=\[Conformance\] ``` The tests are built into a single binary which can be run used to deploy a Kubernetes system or run tests against an already-deployed Kubernetes system. See `go run hack/e2e.go --help` (or the flag definitions in `hack/e2e.go`) for more options, such as reusing an existing cluster. @@ -208,13 +208,31 @@ We are working on implementing clearer partitioning of our e2e tests to make run ### Conformance tests -Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. `[Conformance]` test policies are a work-in-progress (see #18162). +Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. -End-to-end testing, as described above, is for [development distributions](writing-a-getting-started-guide.md). A conformance test is used on a [versioned distro](writing-a-getting-started-guide.md). (Links WIP) +As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned with an eye towards backwards compatibility. Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg: + + - A v1.3 development cluster should pass v1.0, v1.1, v1.2 conformance tests + - A v1.2 cluster should pass v1.0, v1.1, v1.2 conformance tests + - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 conformance tests -The conformance test runs a subset of the e2e-tests against a manually-created cluster. It does not require support for up/push/down and other operations. To run a conformance test, you need to know the IP of the master for your cluster and the authorization arguments to use. The conformance test is intended to run against a cluster at a specific binary release of Kubernetes. See [conformance-test.sh](http://releases.k8s.io/HEAD/hack/conformance-test.sh). +Conformance tests are designed to be run with no cloud provider configured. Conformance tests can be run against clusters that have not been created with `hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and credentials. -### Defining what Conformance means +```sh +# setup for conformance tests +export KUBECONFIG=/path/to/kubeconfig +export KUBE_CONFORMANCE_TEST=y + +# run all conformance tests +go run hack/e2e.go -v --test_args="--ginkgo.focus=\[Conformance\]" + +# run all parallel-safe conformance tests in parallel +GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]" +# ... and finish up with remaining tests in serial +go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]" +``` + +### Defining Conformance Subset It is impossible to define the entire space of Conformance tests without knowing the future, so instead, we define the compliment of conformance tests, below. diff --git a/docs/devel/testing.md b/docs/devel/testing.md index b918f044cdc..3bc1c141543 100644 --- a/docs/devel/testing.md +++ b/docs/devel/testing.md @@ -174,55 +174,7 @@ hack/test-integration.sh # Run all integration tests. ## End-to-End tests -* e2e tests build kubernetes and deploy a cluster of nodes. - - Generally on a specific cloud provider. -* Access gcr.io images -* Access a specific, non-latest image tag (unless testing pulling). -* Tests may not flake due to intermittent issues. -* Use ginko to desribe steps. - - See [should run a job to completion when tasks succeed](../../test/e2e/job.go) -* Use [NewDefaultFramework](../../test/e2e/framework.go) - - Contains clients, namespace and auto resource cleanup -* See [coding conventions](coding-conventions.md). - -### e2e test philosophy - -In general passing unit and integration tests should provide sufficient -confidence to allow code to merge. If that is not the case, -please *invest more time adding unit and integration test coverage*. -These tests run faster and have a smaller failure domain. - -However, end-to-end (e2e) tests provide maximum confidence that -the system is working in exchange for reduced performance and a -higher debugging cost. - -e2e tests deploy a real kubernetes cluster of real nodes on a concrete provider -such as GCE. The tests then manipulate the cluster in certain ways and -assert the expected results. - -For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md). - -### Running e2e tests - -```sh -cd kubernetes -go run hack/e2e.go -v --build --up --test --down - -# Change code, run unit and integration tests -# Push to an existing cluster, or bring up a cluster if it's down. -go run hack/e2e.go -v --pushup - -# Run all tests on an already up cluster -go run hack/e2e.go -v --test - -# Run only conformance tests -go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]" - -# Run tests on a specific provider -KUBERNETES_PROVIDER=aws go run hack/e2e.go --build --pushup --test --down -``` - -For a more in depth discussion please read [End-to-End Testing in Kubernetes](e2e-tests.md). +Please refer to [End-to-End Testing in Kubernetes](e2e-tests.md). [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/testing.md?pixel)]() diff --git a/docs/devel/writing-a-getting-started-guide.md b/docs/devel/writing-a-getting-started-guide.md index 729977b0b3b..fbe5aa1ba18 100644 --- a/docs/devel/writing-a-getting-started-guide.md +++ b/docs/devel/writing-a-getting-started-guide.md @@ -68,7 +68,7 @@ These guidelines say *what* to do. See the Rationale section for *why*. own repo. - Add or update a row in [The Matrix](../../docs/getting-started-guides/README.md). - State the binary version of Kubernetes that you tested clearly in your Guide doc. - - Setup a cluster and run the [conformance test](development.md#conformance-testing) against it, and report the + - Setup a cluster and run the [conformance tests](e2e-tests.md#conformance-tests) against it, and report the results in your PR. - Versioned distros should typically not modify or add code in `cluster/`. That is just scripts for developer distros. From f306193386d510bacd3a22812573cc87bc7770fc Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Tue, 22 Mar 2016 13:16:14 -0700 Subject: [PATCH 2/6] Use e2e.go for conformance test instead of script Totally guessing at the escaping in that string --- contrib/mesos/ci/test-conformance.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/mesos/ci/test-conformance.sh b/contrib/mesos/ci/test-conformance.sh index ff030c22853..8ce2ccb965e 100755 --- a/contrib/mesos/ci/test-conformance.sh +++ b/contrib/mesos/ci/test-conformance.sh @@ -31,7 +31,7 @@ TEST_ARGS="$@" KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE}")/../../.." && pwd) -TEST_CMD="KUBECONFIG=~/.kube/config hack/conformance-test.sh" +TEST_CMD="KUBE_CONFORMANCE_TEST=y KUBECONFIG=~/.kube/config go run hack/e2e.go --test --test_args=\"--ginkgo.focus=\\[Conformance\\]\"" if [ -n "${CONFORMANCE_BRANCH}" ]; then # create a CONFORMANCE_BRANCH clone in a subdirectory TEST_CMD=" From 3f501f61accce247f182ab31a8188475642baa08 Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Tue, 22 Mar 2016 13:28:13 -0700 Subject: [PATCH 3/6] Remove hack/conformance-test.sh --- hack/conformance-test.sh | 87 ---------------------------------------- 1 file changed, 87 deletions(-) delete mode 100755 hack/conformance-test.sh diff --git a/hack/conformance-test.sh b/hack/conformance-test.sh deleted file mode 100755 index 70b8a5cab96..00000000000 --- a/hack/conformance-test.sh +++ /dev/null @@ -1,87 +0,0 @@ -#!/bin/bash - -# Copyright 2015 The Kubernetes Authors All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# When run as described below, the conformance test tests whether a cluster -# supports key features for Kubernetes version 1.0. - -# Instructions: -# - Setup a Kubernetes cluster with $NUM_NODES nodes (defined below). -# - Provide a Kubeconfig file whose current context is set to the -# cluster to be tested, and with suitable auth setting. -# - Specify the location of that kubeconfig with, e.g.: -# declare -x KUBECONFIG="$HOME/.kube/config" -# - Make sure only essential pods are running and there are no failed/pending pods. -# - Go to a git tree that contains the kubernetes source. -# - git clone git://github.com/kubernetes/kubernetes.git -# - Checkout the upstream/conformance-test-v1 branch -# - git checkout upstream/conformance-test-v1 -# - The working tree will be in a "detached HEAD" state. -# - Make binaries needed by e2e -# make clean -# make quick-release -# - Run the test and capture output: -# hack/conformance-test.sh 2>&1 | tee conformance.$(date +%FT%T%z).log -# - -# About the conformance test: -# The conformance test checks whether a kubernetes cluster supports -# a minimum set of features to be called "Kubernetes". It is similar -# to `hack/e2e.go` but it differs in that: -# - hack/e2e.go is intended to test a cluster with binaries built at HEAD, -# while this conformance test does not care what version the binaries are. -# - this means the user needs to setup a cluster first. -# - this means the user does not need to write any cluster/... scripts. Custom -# clusters can be tested. -# - hack/e2e.go is intended to run e2e tests built at HEAD, while -# this conformance test is intended to be run e2e tests built at a particular -# version. This ensures that all conformance testees run the same set of tests, -# regardless of when they test for conformance. -# - it excludes certain e2e tests: -# - tests that are specific to certain cloud providers -# - tests of optional features, such as volume types. -# - tests of performance, scale, or reliability -# - known flaky tests. -# TODO: when preparing to release a new major or minor version of Kubernetes, -# create a new conformance-test-vX.Y branch, update mentions of that branch in this file, -# reevaluate the set of e2e tests, -# update documentation at docs/getting-started-guides/README.md to have -# a new column for conformance at that new version, and notify -# community. - -TEST_ARGS="$@" - -: ${KUBECONFIG:?"Must set KUBECONFIG before running conformance test."} -echo "Conformance test using current-context of ${KUBECONFIG}" -echo -n "Conformance test run date:" -date -echo -n "Conformance test SHA:" -HEAD_SHA=$(git rev-parse HEAD) -echo $HEAD_SHA -echo "Conformance test version tag(s):" -git tag --points-at $HEAD_SHA -echo -echo "Conformance test checking conformance with Kubernetes version 1.0" - -# It runs a whitelist of tests: all tests which are flagged with [Conformance] -# somewhere in the description (i.e. either in the Describe part or the It part). -# The list of tagged conformance tests can be retrieved by: -# -# NUM_NODES=4 KUBERNETES_CONFORMANCE_TEST="y" \ -# hack/ginkgo-e2e.sh -ginkgo.focus='\[Conformance\]' -ginkgo.dryRun=true - -declare -x KUBERNETES_CONFORMANCE_TEST="y" -declare -x NUM_NODES=4 -exec hack/ginkgo-e2e.sh -ginkgo.focus='\[Conformance\]' ${TEST_ARGS} From 9182fbf8f2b878330cefd3d51f1a07a53a552a4f Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Tue, 22 Mar 2016 13:48:31 -0700 Subject: [PATCH 4/6] Run hack/update-generated-docs.sh --- docs/devel/e2e-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/devel/e2e-tests.md b/docs/devel/e2e-tests.md index afa1337d969..23d73e67c05 100644 --- a/docs/devel/e2e-tests.md +++ b/docs/devel/e2e-tests.md @@ -211,7 +211,7 @@ We are working on implementing clearer partitioning of our e2e tests to make run Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned with an eye towards backwards compatibility. Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg: - + - A v1.3 development cluster should pass v1.0, v1.1, v1.2 conformance tests - A v1.2 cluster should pass v1.0, v1.1, v1.2 conformance tests - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 conformance tests From d8d785f244a594fd7bac501b253bad2c00babf7c Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Tue, 22 Mar 2016 15:36:27 -0700 Subject: [PATCH 5/6] Update versioning per supported releases policy --- docs/devel/e2e-tests.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/devel/e2e-tests.md b/docs/devel/e2e-tests.md index 23d73e67c05..5d23bc9abbc 100644 --- a/docs/devel/e2e-tests.md +++ b/docs/devel/e2e-tests.md @@ -210,10 +210,10 @@ We are working on implementing clearer partitioning of our e2e tests to make run Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. -As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned with an eye towards backwards compatibility. Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg: +As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned, with the same backwards compatibility guarantees as laid out in [our versioning policy](../design/versioning.md#supported-releases). Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg: - - A v1.3 development cluster should pass v1.0, v1.1, v1.2 conformance tests - - A v1.2 cluster should pass v1.0, v1.1, v1.2 conformance tests + - A v1.3 development cluster should pass v1.1, v1.2 conformance tests + - A v1.2 cluster should pass v1.1, v1.2 conformance tests - A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 conformance tests Conformance tests are designed to be run with no cloud provider configured. Conformance tests can be run against clusters that have not been created with `hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and credentials. From d794b86d3435a4adeb9e34353508cd6da66079cb Mon Sep 17 00:00:00 2001 From: Aaron Crickenberger Date: Wed, 23 Mar 2016 12:14:54 -0700 Subject: [PATCH 6/6] Fix typo --- contrib/mesos/ci/test-conformance.sh | 2 +- docs/devel/e2e-tests.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/mesos/ci/test-conformance.sh b/contrib/mesos/ci/test-conformance.sh index 8ce2ccb965e..5f9bbb55087 100755 --- a/contrib/mesos/ci/test-conformance.sh +++ b/contrib/mesos/ci/test-conformance.sh @@ -31,7 +31,7 @@ TEST_ARGS="$@" KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE}")/../../.." && pwd) -TEST_CMD="KUBE_CONFORMANCE_TEST=y KUBECONFIG=~/.kube/config go run hack/e2e.go --test --test_args=\"--ginkgo.focus=\\[Conformance\\]\"" +TEST_CMD="KUBERNETES_CONFORMANCE_TEST=y KUBECONFIG=~/.kube/config go run hack/e2e.go --test --test_args=\"--ginkgo.focus=\\[Conformance\\]\"" if [ -n "${CONFORMANCE_BRANCH}" ]; then # create a CONFORMANCE_BRANCH clone in a subdirectory TEST_CMD=" diff --git a/docs/devel/e2e-tests.md b/docs/devel/e2e-tests.md index 5d23bc9abbc..175c323bf04 100644 --- a/docs/devel/e2e-tests.md +++ b/docs/devel/e2e-tests.md @@ -221,7 +221,7 @@ Conformance tests are designed to be run with no cloud provider configured. Con ```sh # setup for conformance tests export KUBECONFIG=/path/to/kubeconfig -export KUBE_CONFORMANCE_TEST=y +export KUBERNETES_CONFORMANCE_TEST=y # run all conformance tests go run hack/e2e.go -v --test_args="--ginkgo.focus=\[Conformance\]"