github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-09-09 21:21:14 +00:00
Code Issues Projects Releases Wiki Activity

Update conformance test policy

Browse Source 
Mostly doc updates and cruft removal

- describe conformance test policy and howto in e2e-tests.md
- rm e2e test info from testing.md in the name of DRY
- rm cluster/test-conformance.sh; unusable in release tar, not e2e.go
- update e2e test link in write-a-getting-started-guide.md
This commit is contained in:
Aaron Crickenberger
2016-03-22 13:09:31 -07:00
parent 2bb6f74bf9
commit 0dc1f5da04
4 changed files with 29 additions and 89 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
docs/devel/e2e-tests.md
View File

@@ -77,12 +77,15 @@ go run hack/e2e.go -v --pushup
# Run all tests
go run hack/e2e.go -v --test
# Run tests matching the regex "\[Conformance\]" (the conformance tests)
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Conformance\]"
# Run tests matching the regex "\[Feature:Performance\]"
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=\[Feature:Performance\]"
# Conversely, exclude tests that match the regex "Pods.*env"
go run hack/e2e.go -v -test --test_args="--ginkgo.focus=Pods.*env"
# Run tests in parallel, skip any that must be run serially
GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.skip=\[Serial\]"
# Flags can be combined, and their actions will take place in this order:
# --build, --push|--up|--pushup, --test|--tests=..., --down
#
@@ -96,9 +99,6 @@ KUBERNETES_PROVIDER=aws go run hack/e2e.go -v --build --pushup --test --down
# kubectl output.
go run hack/e2e.go -v -ctl='get events'
go run hack/e2e.go -v -ctl='delete pod foobar'
# Alternately, if you have the e2e cluster up and no desire to see the event stream, you can run ginkgo-e2e.sh directly:
hack/ginkgo-e2e.sh --ginkgo.focus=\[Conformance\]
```
The tests are built into a single binary which can be run used to deploy a Kubernetes system or run tests against an already-deployed Kubernetes system. See `go run hack/e2e.go --help` (or the flag definitions in `hack/e2e.go`) for more options, such as reusing an existing cluster.
@@ -208,13 +208,31 @@ We are working on implementing clearer partitioning of our e2e tests to make run
### Conformance tests
Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels. `[Conformance]` test policies are a work-in-progress (see #18162).
Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels.
End-to-end testing, as described above, is for [development distributions](writing-a-getting-started-guide.md). A conformance test is used on a [versioned distro](writing-a-getting-started-guide.md). (Links WIP)
As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned with an eye towards backwards compatibility. Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg:
- A v1.3 development cluster should pass v1.0, v1.1, v1.2 conformance tests
- A v1.2 cluster should pass v1.0, v1.1, v1.2 conformance tests
- A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 conformance tests
The conformance test runs a subset of the e2e-tests against a manually-created cluster. It does not require support for up/push/down and other operations. To run a conformance test, you need to know the IP of the master for your cluster and the authorization arguments to use. The conformance test is intended to run against a cluster at a specific binary release of Kubernetes. See [conformance-test.sh](http://releases.k8s.io/HEAD/hack/conformance-test.sh).
Conformance tests are designed to be run with no cloud provider configured. Conformance tests can be run against clusters that have not been created with `hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and credentials.
### Defining what Conformance means
```sh
# setup for conformance tests
export KUBECONFIG=/path/to/kubeconfig
export KUBE_CONFORMANCE_TEST=y
# run all conformance tests
go run hack/e2e.go -v --test_args="--ginkgo.focus=\[Conformance\]"
# run all parallel-safe conformance tests in parallel
GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
# ... and finish up with remaining tests in serial
go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]"
```
### Defining Conformance Subset
It is impossible to define the entire space of Conformance tests without knowing the future, so instead, we define the compliment of conformance tests, below.