From 2d09e07ae6de579e159b897eb58acc77b9153549 Mon Sep 17 00:00:00 2001 From: Isaac Hollander McCreery Date: Fri, 8 Jan 2016 11:35:30 -0800 Subject: [PATCH] Add documentation for test labels --- docs/devel/e2e-tests.md | 14 ++++++++++++++ hack/jenkins/e2e.sh | 1 + test/e2e/e2e_test.go | 3 ++- 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/devel/e2e-tests.md b/docs/devel/e2e-tests.md index 902ba1c15ff..6eae78cfd7a 100644 --- a/docs/devel/e2e-tests.md +++ b/docs/devel/e2e-tests.md @@ -102,6 +102,20 @@ As mentioned earlier there are a host of other options that are available, but a - `rm -rf /var/run/kubernetes`, clear kube generated credentials, sometimes stale permissions can cause problems. - `sudo iptables -F`, clear ip tables rules left by the kube-proxy. +## Kinds of tests + +We are working on implementing clearer partitioning of our e2e tests to make running a known set of tests easier (#10548). Tests can be labeled with any of the following labels, in order of increasing precedence (that is, each label listed below supersedes the previous ones): + +- If a test has no labels, it is expected to run fast (under five minutes), be able to be run in parallel, and be consistent. +- `[Slow]`: If a test takes more than five minutes to run (by itself or in parallel with many other tests), it is labeled `[Slow]`. This partition allows us to run almost all of our tests quickly in parallel, without waiting for the stragglers to finish. +- `[Serial]`: If a test cannot be run in parallel with other tests (e.g. it takes too many resources or restarts nodes), it is labeled `[Serial]`, and should be run in serial as part of a separate suite. +- `[Disruptive]`: If a test restarts components that might cause other tests to fail or break the cluster completely, it is labeled `[Disruptive]`. Any `[Disruptive]` test is also assumed to qualify for the `[Serial]` label, but need not be labeled as both. These tests are not run against soak clusters to avoid restarting components. +- `[Flaky]`: If a test is found to be flaky, it receives the `[Flaky]` label until it is fixed. A `[Flaky]` label should be accompanied with a reference to the issue for de-flaking the test, because while a test remains labeled `[Flaky]`, it is not monitored closely in CI. `[Flaky]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given. +- `[Skipped]`: `[Skipped]` is a legacy label that we're phasing out. If a test is marked `[Skipped]`, there should be an issue open to label it properly. `[Skipped]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given. +- `[Feature:...]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:...]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`. `[Feature:...]` tests are not run in our core suites, instead running in custom suites. + +Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. `[Conformance]` test policies are a work-in-progress; see #18162. + ## Adding a New Test As mentioned above, prior to adding a new test, it is a good idea to perform a `-ginkgo.dryRun=true` on the system, in order to see if a behavior is already being tested, or to determine if it may be possible to augment an existing set of tests for a specific use case. diff --git a/hack/jenkins/e2e.sh b/hack/jenkins/e2e.sh index 0bee84fcd12..3444745e92f 100755 --- a/hack/jenkins/e2e.sh +++ b/hack/jenkins/e2e.sh @@ -269,6 +269,7 @@ CURRENT_RELEASE_PUBLISHED_VERSION="ci/latest-1.1" GCE_DEFAULT_SKIP_TESTS=( "\[Example\]" # previously in REBOOT_SKIP_TESTS..., dates back before version control (#10078) "\[Skipped\]" + "\[Feature:.+\]" ) # PROVIDER SKIPS -------------------------------------- diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index 21b29771dec..802b01097b4 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -147,7 +147,8 @@ func TestE2E(t *testing.T) { // Disable skipped tests unless they are explicitly requested. if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { - config.GinkgoConfig.SkipString = `\[Skipped\]` + // TODO(ihmccreery) Remove [Skipped] once all [Skipped] labels have been reclassified. + config.GinkgoConfig.SkipString = `\[Flaky\]|\[Skipped\]|\[Feature\]` } gomega.RegisterFailHandler(ginkgo.Fail)