mirror of
https://github.com/k3s-io/kubernetes.git
synced 2025-07-25 04:33:26 +00:00
Merge pull request #25099 from mikebrow/devel-tree-80col-updates
Automatic merge from submit-queue devel/ tree minor edits Address line wrap issue #1488. Also cleans up other minor editing issues in the docs/devel/* tree such as spelling errors, links,... Signed-off-by: Mike Brown <brownwm@us.ibm.com>
This commit is contained in:
k8s-merge-robot
commit
27f418848d
@ -50,65 +50,129 @@ Updated: 5/3/2016
|
||||
## Code conventions
|
||||
|
||||
- Bash
|
||||
|
||||
- https://google-styleguide.googlecode.com/svn/trunk/shell.xml
|
||||
- Ensure that build, release, test, and cluster-management scripts run on OS X
|
||||
|
||||
- Ensure that build, release, test, and cluster-management scripts run on
|
||||
OS X
|
||||
|
||||
- Go
|
||||
|
||||
- Ensure your code passes the [presubmit checks](development.md#hooks)
|
||||
- [Go Code Review Comments](https://github.com/golang/go/wiki/CodeReviewComments)
|
||||
|
||||
- [Go Code Review
|
||||
Comments](https://github.com/golang/go/wiki/CodeReviewComments)
|
||||
|
||||
- [Effective Go](https://golang.org/doc/effective_go.html)
|
||||
|
||||
- Comment your code.
|
||||
- [Go's commenting conventions](http://blog.golang.org/godoc-documenting-go-code)
|
||||
- If reviewers ask questions about why the code is the way it is, that's a sign that comments might be helpful.
|
||||
- [Go's commenting
|
||||
conventions](http://blog.golang.org/godoc-documenting-go-code)
|
||||
- If reviewers ask questions about why the code is the way it is, that's a
|
||||
sign that comments might be helpful.
|
||||
|
||||
|
||||
- Command-line flags should use dashes, not underscores
|
||||
|
||||
|
||||
- Naming
|
||||
- Please consider package name when selecting an interface name, and avoid redundancy.
|
||||
- e.g.: `storage.Interface` is better than `storage.StorageInterface`.
|
||||
- Do not use uppercase characters, underscores, or dashes in package names.
|
||||
- Please consider package name when selecting an interface name, and avoid
|
||||
redundancy.
|
||||
|
||||
- e.g.: `storage.Interface` is better than `storage.StorageInterface`.
|
||||
|
||||
- Do not use uppercase characters, underscores, or dashes in package
|
||||
names.
|
||||
- Please consider parent directory name when choosing a package name.
|
||||
- so pkg/controllers/autoscaler/foo.go should say `package autoscaler` not `package autoscalercontroller`.
|
||||
- Unless there's a good reason, the `package foo` line should match the name of the directory in which the .go file exists.
|
||||
- Importers can use a different name if they need to disambiguate.
|
||||
- Locks should be called `lock` and should never be embedded (always `lock sync.Mutex`). When multiple locks are present, give each lock a distinct name following Go conventions - `stateLock`, `mapLock` etc.
|
||||
- API conventions
|
||||
- [API changes](api_changes.md)
|
||||
- [API conventions](api-conventions.md)
|
||||
|
||||
- so pkg/controllers/autoscaler/foo.go should say `package autoscaler`
|
||||
not `package autoscalercontroller`.
|
||||
- Unless there's a good reason, the `package foo` line should match
|
||||
the name of the directory in which the .go file exists.
|
||||
- Importers can use a different name if they need to disambiguate.
|
||||
|
||||
- Locks should be called `lock` and should never be embedded (always `lock
|
||||
sync.Mutex`). When multiple locks are present, give each lock a distinct name
|
||||
following Go conventions - `stateLock`, `mapLock` etc.
|
||||
|
||||
- [API changes](api_changes.md)
|
||||
|
||||
- [API conventions](api-conventions.md)
|
||||
|
||||
- [Kubectl conventions](kubectl-conventions.md)
|
||||
|
||||
- [Logging conventions](logging.md)
|
||||
|
||||
## Testing conventions
|
||||
|
||||
- All new packages and most new significant functionality must come with unit tests
|
||||
- Table-driven tests are preferred for testing multiple scenarios/inputs; for example, see [TestNamespaceAuthorization](../../test/integration/auth_test.go)
|
||||
- Significant features should come with integration (test/integration) and/or [end-to-end (test/e2e) tests](e2e-tests.md)
|
||||
- All new packages and most new significant functionality must come with unit
|
||||
tests
|
||||
|
||||
- Table-driven tests are preferred for testing multiple scenarios/inputs; for
|
||||
example, see [TestNamespaceAuthorization](../../test/integration/auth_test.go)
|
||||
|
||||
- Significant features should come with integration (test/integration) and/or
|
||||
[end-to-end (test/e2e) tests](e2e-tests.md)
|
||||
- Including new kubectl commands and major features of existing commands
|
||||
- Unit tests must pass on OS X and Windows platforms - if you use Linux specific features, your test case must either be skipped on windows or compiled out (skipped is better when running Linux specific commands, compiled out is required when your code does not compile on Windows).
|
||||
|
||||
- Unit tests must pass on OS X and Windows platforms - if you use Linux
|
||||
specific features, your test case must either be skipped on windows or compiled
|
||||
out (skipped is better when running Linux specific commands, compiled out is
|
||||
required when your code does not compile on Windows).
|
||||
|
||||
- Avoid relying on Docker hub (e.g. pull from Docker hub). Use gcr.io instead.
|
||||
- Avoid waiting for a short amount of time (or without waiting) and expect an asynchronous thing to happen (e.g. wait for 1 seconds and expect a Pod to be running). Wait and retry instead.
|
||||
|
||||
- Avoid waiting for a short amount of time (or without waiting) and expect an
|
||||
asynchronous thing to happen (e.g. wait for 1 seconds and expect a Pod to be
|
||||
running). Wait and retry instead.
|
||||
|
||||
- See the [testing guide](testing.md) for additional testing advice.
|
||||
|
||||
## Directory and file conventions
|
||||
|
||||
- Avoid package sprawl. Find an appropriate subdirectory for new packages. (See [#4851](http://issues.k8s.io/4851) for discussion.)
|
||||
- Libraries with no more appropriate home belong in new package subdirectories of pkg/util
|
||||
- Avoid general utility packages. Packages called "util" are suspect. Instead, derive a name that describes your desired function. For example, the utility functions dealing with waiting for operations are in the "wait" package and include functionality like Poll. So the full name is wait.Poll
|
||||
- Avoid package sprawl. Find an appropriate subdirectory for new packages.
|
||||
(See [#4851](http://issues.k8s.io/4851) for discussion.)
|
||||
- Libraries with no more appropriate home belong in new package
|
||||
subdirectories of pkg/util
|
||||
|
||||
- Avoid general utility packages. Packages called "util" are suspect. Instead,
|
||||
derive a name that describes your desired function. For example, the utility
|
||||
functions dealing with waiting for operations are in the "wait" package and
|
||||
include functionality like Poll. So the full name is wait.Poll
|
||||
|
||||
- All filenames should be lowercase
|
||||
|
||||
- Go source files and directories use underscores, not dashes
|
||||
- Package directories should generally avoid using separators as much as possible (when packages are multiple words, they usually should be in nested subdirectories).
|
||||
- Package directories should generally avoid using separators as much as
|
||||
possible (when packages are multiple words, they usually should be in nested
|
||||
subdirectories).
|
||||
|
||||
- Document directories and filenames should use dashes rather than underscores
|
||||
- Contrived examples that illustrate system features belong in /docs/user-guide or /docs/admin, depending on whether it is a feature primarily intended for users that deploy applications or cluster administrators, respectively. Actual application examples belong in /examples.
|
||||
- Examples should also illustrate
|
||||
[best practices for configuration and using the system](../user-guide/config-best-practices.md)
|
||||
|
||||
- Contrived examples that illustrate system features belong in
|
||||
/docs/user-guide or /docs/admin, depending on whether it is a feature primarily
|
||||
intended for users that deploy applications or cluster administrators,
|
||||
respectively. Actual application examples belong in /examples.
|
||||
- Examples should also illustrate [best practices for configuration and
|
||||
using the system](../user-guide/config-best-practices.md)
|
||||
|
||||
- Third-party code
|
||||
- Go code for normal third-party dependencies is managed using [Godeps](https://github.com/tools/godep)
|
||||
|
||||
- Go code for normal third-party dependencies is managed using
|
||||
[Godeps](https://github.com/tools/godep)
|
||||
|
||||
- Other third-party code belongs in `/third_party`
|
||||
- forked third party Go code goes in `/third_party/forked`
|
||||
- forked _golang stdlib_ code goes in `/third_party/golang`
|
||||
|
||||
- Third-party code must include licenses
|
||||
|
||||
- This includes modified third-party code and excerpts, as well
|
||||
|
||||
## Coding advice
|
||||
|
||||
- Go
|
||||
|
||||
- [Go landmines](https://gist.github.com/lavalamp/4bd23295a9f32706a48f)
|
||||
|
||||
|
||||
|
||||
@ -34,44 +34,86 @@ Documentation for other releases can be found at
|
||||
|
||||
# On Collaborative Development
|
||||
|
||||
Kubernetes is open source, but many of the people working on it do so as their day job. In order to avoid forcing people to be "at work" effectively 24/7, we want to establish some semi-formal protocols around development. Hopefully these rules make things go more smoothly. If you find that this is not the case, please complain loudly.
|
||||
Kubernetes is open source, but many of the people working on it do so as their
|
||||
day job. In order to avoid forcing people to be "at work" effectively 24/7, we
|
||||
want to establish some semi-formal protocols around development. Hopefully these
|
||||
rules make things go more smoothly. If you find that this is not the case,
|
||||
please complain loudly.
|
||||
|
||||
## Patches welcome
|
||||
|
||||
First and foremost: as a potential contributor, your changes and ideas are welcome at any hour of the day or night, weekdays, weekends, and holidays. Please do not ever hesitate to ask a question or send a PR.
|
||||
First and foremost: as a potential contributor, your changes and ideas are
|
||||
welcome at any hour of the day or night, weekdays, weekends, and holidays.
|
||||
Please do not ever hesitate to ask a question or send a PR.
|
||||
|
||||
## Code reviews
|
||||
|
||||
All changes must be code reviewed. For non-maintainers this is obvious, since you can't commit anyway. But even for maintainers, we want all changes to get at least one review, preferably (for non-trivial changes obligatorily) from someone who knows the areas the change touches. For non-trivial changes we may want two reviewers. The primary reviewer will make this decision and nominate a second reviewer, if needed. Except for trivial changes, PRs should not be committed until relevant parties (e.g. owners of the subsystem affected by the PR) have had a reasonable chance to look at PR in their local business hours.
|
||||
All changes must be code reviewed. For non-maintainers this is obvious, since
|
||||
you can't commit anyway. But even for maintainers, we want all changes to get at
|
||||
least one review, preferably (for non-trivial changes obligatorily) from someone
|
||||
who knows the areas the change touches. For non-trivial changes we may want two
|
||||
reviewers. The primary reviewer will make this decision and nominate a second
|
||||
reviewer, if needed. Except for trivial changes, PRs should not be committed
|
||||
until relevant parties (e.g. owners of the subsystem affected by the PR) have
|
||||
had a reasonable chance to look at PR in their local business hours.
|
||||
|
||||
Most PRs will find reviewers organically. If a maintainer intends to be the primary reviewer of a PR they should set themselves as the assignee on GitHub and say so in a reply to the PR. Only the primary reviewer of a change should actually do the merge, except in rare cases (e.g. they are unavailable in a reasonable timeframe).
|
||||
Most PRs will find reviewers organically. If a maintainer intends to be the
|
||||
primary reviewer of a PR they should set themselves as the assignee on GitHub
|
||||
and say so in a reply to the PR. Only the primary reviewer of a change should
|
||||
actually do the merge, except in rare cases (e.g. they are unavailable in a
|
||||
reasonable timeframe).
|
||||
|
||||
If a PR has gone 2 work days without an owner emerging, please poke the PR thread and ask for a reviewer to be assigned.
|
||||
If a PR has gone 2 work days without an owner emerging, please poke the PR
|
||||
thread and ask for a reviewer to be assigned.
|
||||
|
||||
Except for rare cases, such as trivial changes (e.g. typos, comments) or emergencies (e.g. broken builds), maintainers should not merge their own changes.
|
||||
Except for rare cases, such as trivial changes (e.g. typos, comments) or
|
||||
emergencies (e.g. broken builds), maintainers should not merge their own
|
||||
changes.
|
||||
|
||||
Expect reviewers to request that you avoid [common go style mistakes](https://github.com/golang/go/wiki/CodeReviewComments) in your PRs.
|
||||
Expect reviewers to request that you avoid [common go style
|
||||
mistakes](https://github.com/golang/go/wiki/CodeReviewComments) in your PRs.
|
||||
|
||||
## Assigned reviews
|
||||
|
||||
Maintainers can assign reviews to other maintainers, when appropriate. The assignee becomes the shepherd for that PR and is responsible for merging the PR once they are satisfied with it or else closing it. The assignee might request reviews from non-maintainers.
|
||||
Maintainers can assign reviews to other maintainers, when appropriate. The
|
||||
assignee becomes the shepherd for that PR and is responsible for merging the PR
|
||||
once they are satisfied with it or else closing it. The assignee might request
|
||||
reviews from non-maintainers.
|
||||
|
||||
## Merge hours
|
||||
|
||||
Maintainers will do merges of appropriately reviewed-and-approved changes during their local "business hours" (typically 7:00 am Monday to 5:00 pm (17:00h) Friday). PRs that arrive over the weekend or on holidays will only be merged if there is a very good reason for it and if the code review requirements have been met. Concretely this means that nobody should merge changes immediately before going to bed for the night.
|
||||
Maintainers will do merges of appropriately reviewed-and-approved changes during
|
||||
their local "business hours" (typically 7:00 am Monday to 5:00 pm (17:00h)
|
||||
Friday). PRs that arrive over the weekend or on holidays will only be merged if
|
||||
there is a very good reason for it and if the code review requirements have been
|
||||
met. Concretely this means that nobody should merge changes immediately before
|
||||
going to bed for the night.
|
||||
|
||||
There may be discussion an even approvals granted outside of the above hours, but merges will generally be deferred.
|
||||
There may be discussion an even approvals granted outside of the above hours,
|
||||
but merges will generally be deferred.
|
||||
|
||||
If a PR is considered complex or controversial, the merge of that PR should be delayed to give all interested parties in all timezones the opportunity to provide feedback. Concretely, this means that such PRs should be held for 24
|
||||
hours before merging. Of course "complex" and "controversial" are left to the judgment of the people involved, but we trust that part of being a committer is the judgment required to evaluate such things honestly, and not be
|
||||
motivated by your desire (or your cube-mate's desire) to get their code merged. Also see "Holds" below, any reviewer can issue a "hold" to indicate that the PR is in fact complicated or complex and deserves further review.
|
||||
If a PR is considered complex or controversial, the merge of that PR should be
|
||||
delayed to give all interested parties in all timezones the opportunity to
|
||||
provide feedback. Concretely, this means that such PRs should be held for 24
|
||||
hours before merging. Of course "complex" and "controversial" are left to the
|
||||
judgment of the people involved, but we trust that part of being a committer is
|
||||
the judgment required to evaluate such things honestly, and not be motivated by
|
||||
your desire (or your cube-mate's desire) to get their code merged. Also see
|
||||
"Holds" below, any reviewer can issue a "hold" to indicate that the PR is in
|
||||
fact complicated or complex and deserves further review.
|
||||
|
||||
PRs that are incorrectly judged to be merge-able, may be reverted and subject to re-review, if subsequent reviewers believe that they in fact are controversial or complex.
|
||||
PRs that are incorrectly judged to be merge-able, may be reverted and subject to
|
||||
re-review, if subsequent reviewers believe that they in fact are controversial
|
||||
or complex.
|
||||
|
||||
|
||||
## Holds
|
||||
|
||||
Any maintainer or core contributor who wants to review a PR but does not have time immediately may put a hold on a PR simply by saying so on the PR discussion and offering an ETA measured in single-digit days at most. Any PR that has a hold shall not be merged until the person who requested the hold acks the review, withdraws their hold, or is overruled by a preponderance of maintainers.
|
||||
Any maintainer or core contributor who wants to review a PR but does not have
|
||||
time immediately may put a hold on a PR simply by saying so on the PR discussion
|
||||
and offering an ETA measured in single-digit days at most. Any PR that has a
|
||||
hold shall not be merged until the person who requested the hold acks the
|
||||
review, withdraws their hold, or is overruled by a preponderance of maintainers.
|
||||
|
||||
|
||||
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
|
||||
|
||||
@ -47,7 +47,8 @@ branch, but release branches of Kubernetes should not change.
|
||||
## Building Kubernetes
|
||||
|
||||
Official releases are built using Docker containers. To build Kubernetes using
|
||||
Docker please follow [these instructions](http://releases.k8s.io/HEAD/build/README.md).
|
||||
Docker please follow [these
|
||||
instructions](http://releases.k8s.io/HEAD/build/README.md).
|
||||
|
||||
### Go development environment
|
||||
|
||||
@ -55,14 +56,16 @@ Kubernetes is written in the [Go](http://golang.org) programming language.
|
||||
To build Kubernetes without using Docker containers, you'll need a Go
|
||||
development environment. Builds for Kubernetes 1.0 - 1.2 require Go version
|
||||
1.4.2. Builds for Kubernetes 1.3 and higher require Go version 1.6.0. If you
|
||||
haven't set up a Go development environment, please follow [these instructions](http://golang.org/doc/code.html)
|
||||
to install the go tools and set up a GOPATH.
|
||||
haven't set up a Go development environment, please follow [these
|
||||
instructions](http://golang.org/doc/code.html) to install the go tools and set
|
||||
up a GOPATH.
|
||||
|
||||
To build Kubernetes using your local Go development environment (generate linux
|
||||
binaries):
|
||||
|
||||
hack/build-go.sh
|
||||
You may pass build options and packages to the script as necessary. To build binaries for all platforms:
|
||||
You may pass build options and packages to the script as necessary. To build
|
||||
binaries for all platforms:
|
||||
|
||||
hack/build-cross.sh
|
||||
|
||||
@ -82,7 +85,9 @@ Other git workflows are also valid.
|
||||
|
||||
### Clone your fork
|
||||
|
||||
The commands below require that you have $GOPATH set ([$GOPATH docs](https://golang.org/doc/code.html#GOPATH)). We highly recommend you put Kubernetes' code into your GOPATH. Note: the commands below will not work if
|
||||
The commands below require that you have $GOPATH set ([$GOPATH
|
||||
docs](https://golang.org/doc/code.html#GOPATH)). We highly recommend you put
|
||||
Kubernetes' code into your GOPATH. Note: the commands below will not work if
|
||||
there is more than one directory in your `$GOPATH`.
|
||||
|
||||
```sh
|
||||
@ -108,7 +113,9 @@ git fetch upstream
|
||||
git rebase upstream/master
|
||||
```
|
||||
|
||||
Note: If you have write access to the main repository at github.com/kubernetes/kubernetes, you should modify your git configuration so that you can't accidentally push to upstream:
|
||||
Note: If you have write access to the main repository at
|
||||
github.com/kubernetes/kubernetes, you should modify your git configuration so
|
||||
that you can't accidentally push to upstream:
|
||||
|
||||
```sh
|
||||
git remote set-url --push upstream no_push
|
||||
@ -116,9 +123,10 @@ git remote set-url --push upstream no_push
|
||||
|
||||
### Committing changes to your fork
|
||||
|
||||
Before committing any changes, please link/copy the pre-commit hook
|
||||
into your .git directory. This will keep you from accidentally
|
||||
committing non-gofmt'd Go code. In addition this hook will do a build.
|
||||
Before committing any changes, please link/copy the pre-commit hook into your
|
||||
.git directory. This will keep you from accidentally committing non-gofmt'd Go
|
||||
code. This hook will also do a build and test whether documentation generation
|
||||
scripts need to be executed.
|
||||
|
||||
The hook requires both Godep and etcd on your `PATH`.
|
||||
|
||||
@ -156,15 +164,22 @@ See [Faster Reviews](faster_reviews.md) for more details.
|
||||
|
||||
## godep and dependency management
|
||||
|
||||
Kubernetes uses [godep](https://github.com/tools/godep) to manage dependencies. It is not strictly required for building Kubernetes but it is required when managing dependencies under the Godeps/ tree, and is required by a number of the build and test scripts. Please make sure that ``godep`` is installed and in your ``$PATH``.
|
||||
Kubernetes uses [godep](https://github.com/tools/godep) to manage dependencies.
|
||||
It is not strictly required for building Kubernetes but it is required when
|
||||
managing dependencies under the Godeps/ tree, and is required by a number of the
|
||||
build and test scripts. Please make sure that ``godep`` is installed and in your
|
||||
``$PATH``.
|
||||
|
||||
### Installing godep
|
||||
|
||||
There are many ways to build and host Go binaries. Here is an easy way to get utilities like `godep` installed:
|
||||
There are many ways to build and host Go binaries. Here is an easy way to get
|
||||
utilities like `godep` installed:
|
||||
|
||||
1) Ensure that [mercurial](http://mercurial.selenic.com/wiki/Download) is installed on your system. (some of godep's dependencies use the mercurial
|
||||
source control system). Use `apt-get install mercurial` or `yum install mercurial` on Linux, or [brew.sh](http://brew.sh) on OS X, or download
|
||||
directly from mercurial.
|
||||
1) Ensure that [mercurial](http://mercurial.selenic.com/wiki/Download) is
|
||||
installed on your system. (some of godep's dependencies use the mercurial
|
||||
source control system). Use `apt-get install mercurial` or `yum install
|
||||
mercurial` on Linux, or [brew.sh](http://brew.sh) on OS X, or download directly
|
||||
from mercurial.
|
||||
|
||||
2) Create a new GOPATH for your tools and install godep:
|
||||
|
||||
@ -182,7 +197,8 @@ export PATH=$PATH:$GOPATH/bin
|
||||
```
|
||||
|
||||
Note:
|
||||
At this time, godep update in the Kubernetes project only works properly if your version of godep is < 54.
|
||||
At this time, godep update in the Kubernetes project only works properly if your
|
||||
version of godep is < 54.
|
||||
|
||||
To check your version of godep:
|
||||
|
||||
@ -193,11 +209,14 @@ godep v53 (linux/amd64/go1.5.3)
|
||||
|
||||
### Using godep
|
||||
|
||||
Here's a quick walkthrough of one way to use godeps to add or update a Kubernetes dependency into `vendor/`. For more details, please see the instructions in [godep's documentation](https://github.com/tools/godep).
|
||||
Here's a quick walkthrough of one way to use godeps to add or update a
|
||||
Kubernetes dependency into `vendor/`. For more details, please see the
|
||||
instructions in [godep's documentation](https://github.com/tools/godep).
|
||||
|
||||
1) Devote a directory to this endeavor:
|
||||
|
||||
_Devoting a separate directory is not required, but it is helpful to separate dependency updates from other changes._
|
||||
_Devoting a separate directory is not required, but it is helpful to separate
|
||||
dependency updates from other changes._
|
||||
|
||||
```sh
|
||||
export KPATH=$HOME/code/kubernetes
|
||||
@ -240,20 +259,27 @@ go get -u path/to/dependency
|
||||
godep update path/to/dependency/...
|
||||
```
|
||||
|
||||
_If `go get -u path/to/dependency` fails with compilation errors, instead try `go get -d -u path/to/dependency`
|
||||
to fetch the dependencies without compiling them. This can happen when updating the cadvisor dependency._
|
||||
_If `go get -u path/to/dependency` fails with compilation errors, instead try
|
||||
`go get -d -u path/to/dependency` to fetch the dependencies without compiling
|
||||
them. This can happen when updating the cadvisor dependency._
|
||||
|
||||
|
||||
5) Before sending your PR, it's a good idea to sanity check that your Godeps.json file is ok by running `hack/verify-godeps.sh`
|
||||
5) Before sending your PR, it's a good idea to sanity check that your
|
||||
Godeps.json file is ok by running `hack/verify-godeps.sh`
|
||||
|
||||
_If hack/verify-godeps.sh fails after a `godep update`, it is possible that a transitive dependency was added or removed but not
|
||||
updated by godeps. It then may be necessary to perform a `godep save ./...` to pick up the transitive dependency changes._
|
||||
_If hack/verify-godeps.sh fails after a `godep update`, it is possible that a
|
||||
transitive dependency was added or removed but not updated by godeps. It then
|
||||
may be necessary to perform a `godep save ./...` to pick up the transitive
|
||||
dependency changes._
|
||||
|
||||
It is sometimes expedient to manually fix the /Godeps/godeps.json file to minimize the changes.
|
||||
It is sometimes expedient to manually fix the /Godeps/godeps.json file to
|
||||
minimize the changes.
|
||||
|
||||
Please send dependency updates in separate commits within your PR, for easier reviewing.
|
||||
Please send dependency updates in separate commits within your PR, for easier
|
||||
reviewing.
|
||||
|
||||
6) If you updated the Godeps, please also update `Godeps/LICENSES` by running `hack/update-godep-licenses.sh`.
|
||||
6) If you updated the Godeps, please also update `Godeps/LICENSES` by running
|
||||
`hack/update-godep-licenses.sh`.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@ -34,32 +34,39 @@ Documentation for other releases can be found at
|
||||
|
||||
# Node End-To-End tests
|
||||
|
||||
Node e2e tests start kubelet and minimal supporting infrastructure to validate the kubelet on a host.
|
||||
Tests can be run either locally, against a remote host or against a GCE image.
|
||||
Node e2e tests start kubelet and minimal supporting infrastructure to validate
|
||||
the kubelet on a host. Tests can be run either locally, against a remote host or
|
||||
against a GCE image.
|
||||
|
||||
*Note: Linux only. Mac and Windows unsupported.*
|
||||
|
||||
## Running tests locally
|
||||
|
||||
etcd must be installed and on the PATH to run the node e2e tests. To verify etcd is installed: `which etcd`.
|
||||
You can find instructions for installing etcd [on the etcd releases page](https://github.com/coreos/etcd/releases).
|
||||
etcd must be installed and on the PATH to run the node e2e tests. To verify
|
||||
etcd is installed: `which etcd`. You can find instructions for installing etcd
|
||||
[on the etcd releases page](https://github.com/coreos/etcd/releases).
|
||||
|
||||
Run the tests locally: `make test_e2e_node`
|
||||
|
||||
Running the node e2e tests locally will build the kubernetes go source files and then start the
|
||||
kubelet, kube-apiserver, and etcd binaries on localhost before executing the ginkgo tests under
|
||||
test/e2e_node against the local kubelet instance.
|
||||
Running the node e2e tests locally will build the kubernetes go source files and
|
||||
then start the kubelet, kube-apiserver, and etcd binaries on localhost before
|
||||
executing the ginkgo tests under test/e2e_node against the local kubelet
|
||||
instance.
|
||||
|
||||
## Running tests against a remote host
|
||||
|
||||
The node e2e tests can be run against one or more remote hosts using one of
|
||||
* [e2e-node-jenkins.sh](../../test/e2e_node/jenkins/e2e-node-jenkins.sh) (gce only)
|
||||
* [run_e2e.go](../../test/e2e_node/runner/run_e2e.go) (requires passwordless ssh and remote passwordless sudo access over ssh)
|
||||
* using [run_e2e.go](../../test/e2e_node/runner/run_e2e.go) to build a tar.gz and executing on host (requires host access w/ remote sudo)
|
||||
The node e2e tests can be run against one or more remote hosts using one of:
|
||||
* [e2e-node-jenkins.sh](../../test/e2e_node/jenkins/e2e-node-jenkins.sh) (gce
|
||||
only)
|
||||
* [run_e2e.go](../../test/e2e_node/runner/run_e2e.go) (requires passwordless ssh
|
||||
and remote passwordless sudo access over ssh)
|
||||
* using [run_e2e.go](../../test/e2e_node/runner/run_e2e.go) to build a tar.gz
|
||||
and executing on host (requires host access w/ remote sudo)
|
||||
|
||||
### Configuring a new remote host for testing
|
||||
|
||||
The host must contain a environment capable of supporting a mini-kubernetes cluster. Includes:
|
||||
The host must contain a environment capable of supporting a mini-kubernetes
|
||||
cluster. Includes:
|
||||
* install etcd
|
||||
* install docker
|
||||
* install lxc and update grub commandline
|
||||
@ -70,35 +77,60 @@ See [setup_host.sh](../../test/e2e_node/environment/setup_host.sh)
|
||||
### Running the tests
|
||||
|
||||
1. If running against a host on gce
|
||||
|
||||
* Copy [template.properties](../../test/e2e_node/jenkins/template.properties)
|
||||
|
||||
* Fill in `GCE_HOSTS`
|
||||
* Set `INSTALL_GODEP=true` to install `godep`, `gomega`, `ginkgo`
|
||||
|
||||
* Make sure host names are resolvable to ssh `ssh <host>`.
|
||||
* If needed, you can run `gcloud compute config-ssh` to add gce hostnames to your .ssh/config so they are resolvable by ssh.
|
||||
|
||||
* If needed, you can run `gcloud compute config-ssh` to add gce hostnames to
|
||||
your .ssh/config so they are resolvable by ssh.
|
||||
|
||||
* Run `test/e2e_node/jenkins/e2e-node-jenkins.sh <path to properties file>`
|
||||
* **Must be run from kubernetes root**
|
||||
|
||||
2. If running against a host anywhere else
|
||||
* **Requires password-less ssh and sudo access**
|
||||
* Make sure this works - e.g. `ssh <hostname> -- sudo echo "ok"`
|
||||
* If ssh flags are required (e.g. `-i`), they can be used and passed to the tests with `--ssh-options`
|
||||
* `go run test/e2e_node/runner/run_e2e.go --logtostderr --hosts <comma separated hosts>`
|
||||
* **Must be run from kubernetes root**
|
||||
* requires (go get): `github.com/tools/godep`, `github.com/onsi/gomega`, `github.com/onsi/ginkgo/ginkgo`
|
||||
|
||||
3. Alternatively, manually build and copy `e2e_node_test.tar.gz` to a remote host
|
||||
* Build the tar.gz `go run test/e2e_node/runner/run_e2e.go --logtostderr --build-only`
|
||||
* requires (go get): `github.com/tools/godep`, `github.com/onsi/gomega`, `github.com/onsi/ginkgo/ginkgo`
|
||||
* **Requires password-less ssh and sudo access**
|
||||
|
||||
* Make sure this works - e.g. `ssh <hostname> -- sudo echo "ok"`
|
||||
* If ssh flags are required (e.g. `-i`), they can be used and passed to the
|
||||
tests with `--ssh-options`
|
||||
|
||||
* `go run test/e2e_node/runner/run_e2e.go --logtostderr --hosts <comma
|
||||
separated hosts>`
|
||||
|
||||
* **Must be run from kubernetes root**
|
||||
* requires (go get): `github.com/tools/godep`, `github.com/onsi/gomega`,
|
||||
`github.com/onsi/ginkgo/ginkgo`
|
||||
|
||||
3. Alternatively, manually build and copy `e2e_node_test.tar.gz` to a remote
|
||||
host
|
||||
|
||||
* Build the tar.gz `go run test/e2e_node/runner/run_e2e.go --logtostderr
|
||||
--build-only`
|
||||
|
||||
* requires (go get): `github.com/tools/godep`, `github.com/onsi/gomega`,
|
||||
`github.com/onsi/ginkgo/ginkgo`
|
||||
|
||||
* Copy `e2e_node_test.tar.gz` to the remote host
|
||||
|
||||
* Extract the archive on the remote host `tar -xzvf e2e_node_test.tar.gz`
|
||||
* Run the tests `./e2e_node.test --logtostderr --vmodule=*=2 --build-services=false --node-name=<hostname>`
|
||||
* Note: This must be run from the directory containing the kubelet and kube-apiserver binaries.
|
||||
|
||||
* Run the tests `./e2e_node.test --logtostderr --vmodule=*=2
|
||||
--build-services=false --node-name=<hostname>`
|
||||
|
||||
* Note: This must be run from the directory containing the kubelet and
|
||||
kube-apiserver binaries.
|
||||
|
||||
## Running tests against a gce image
|
||||
|
||||
* Build a gce image from a prepared gce host
|
||||
* Create the host from a base image and configure it (see above)
|
||||
* Run tests against this remote host to ensure that it is setup correctly before doing anything else
|
||||
* Run tests against this remote host to ensure that it is setup correctly
|
||||
before doing anything else
|
||||
* Create a gce *snapshot* of the instance
|
||||
* Create a gce *disk* from the snapshot
|
||||
* Create a gce *image* from the disk
|
||||
@ -112,8 +144,9 @@ See [setup_host.sh](../../test/e2e_node/environment/setup_host.sh)
|
||||
|
||||
## Kubernetes Jenkins CI and PR builder
|
||||
|
||||
Node e2e tests are run against a static list of host environments continuously or when manually triggered on a github.com
|
||||
pull requests using the trigger phrase `@k8s-bot test node e2e experimental` - *results not yet publish, pending
|
||||
Node e2e tests are run against a static list of host environments continuously
|
||||
or when manually triggered on a github.com pull requests using the trigger
|
||||
phrase `@k8s-bot test node e2e experimental` - *results not yet publish, pending
|
||||
evaluation of test stability.*.
|
||||
|
||||
|
||||
|
||||
@ -65,19 +65,40 @@ Updated: 5/3/2016
|
||||
|
||||
## Overview
|
||||
|
||||
End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications. Although unit and integration tests should ideally provide a good signal, the reality is in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at the system level. e2e testing is very costly, both in time to run tests and difficulty debugging, though: it takes a long time to build, deploy, and exercise a cluster. Thus, the primary objectives of the e2e tests are to ensure a consistent and reliable behavior of the kubernetes code base, and to catch hard-to-test bugs before users do, when unit and integration tests are insufficient.
|
||||
End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end
|
||||
behavior of the system, and is the last signal to ensure end user operations
|
||||
match developer specifications. Although unit and integration tests provide a
|
||||
good signal, in a distributed system like Kubernetes it is not uncommon that a
|
||||
minor change may pass all unit and integration tests, but cause unforeseen
|
||||
changes at the system level.
|
||||
|
||||
The e2e tests in kubernetes are built atop of [Ginkgo](http://onsi.github.io/ginkgo/) and [Gomega](http://onsi.github.io/gomega/). There are a host of features that this BDD testing framework provides, and it is recommended that the developer read the documentation prior to diving into the tests.
|
||||
The primary objectives of the e2e tests are to ensure a consistent and reliable
|
||||
behavior of the kubernetes code base, and to catch hard-to-test bugs before
|
||||
users do, when unit and integration tests are insufficient.
|
||||
|
||||
The purpose of *this* document is to serve as a primer for developers who are looking to execute or add tests using a local development environment.
|
||||
The e2e tests in kubernetes are built atop of
|
||||
[Ginkgo](http://onsi.github.io/ginkgo/) and
|
||||
[Gomega](http://onsi.github.io/gomega/). There are a host of features that this
|
||||
Behavior-Driven Development (BDD) testing framework provides, and it is
|
||||
recommended that the developer read the documentation prior to diving into the
|
||||
tests.
|
||||
|
||||
Before writing new tests or making substantive changes to existing tests, you should also read [Writing Good e2e Tests](writing-good-e2e-tests.md)
|
||||
The purpose of *this* document is to serve as a primer for developers who are
|
||||
looking to execute or add tests using a local development environment.
|
||||
|
||||
Before writing new tests or making substantive changes to existing tests, you
|
||||
should also read [Writing Good e2e Tests](writing-good-e2e-tests.md)
|
||||
|
||||
## Building and Running the Tests
|
||||
|
||||
There are a variety of ways to run e2e tests, but we aim to decrease the number of ways to run e2e tests to a canonical way: `hack/e2e.go`.
|
||||
There are a variety of ways to run e2e tests, but we aim to decrease the number
|
||||
of ways to run e2e tests to a canonical way: `hack/e2e.go`.
|
||||
|
||||
You can run an end-to-end test which will bring up a master and nodes, perform some tests, and then tear everything down. Make sure you have followed the getting started steps for your chosen cloud platform (which might involve changing the `KUBERNETES_PROVIDER` environment variable to something other than "gce").
|
||||
You can run an end-to-end test which will bring up a master and nodes, perform
|
||||
some tests, and then tear everything down. Make sure you have followed the
|
||||
getting started steps for your chosen cloud platform (which might involve
|
||||
changing the `KUBERNETES_PROVIDER` environment variable to something other than
|
||||
"gce").
|
||||
|
||||
To build Kubernetes, up a cluster, run tests, and tear everything down, use:
|
||||
|
||||
@ -130,11 +151,16 @@ go run hack/e2e.go -v -ctl='get events'
|
||||
go run hack/e2e.go -v -ctl='delete pod foobar'
|
||||
```
|
||||
|
||||
The tests are built into a single binary which can be run used to deploy a Kubernetes system or run tests against an already-deployed Kubernetes system. See `go run hack/e2e.go --help` (or the flag definitions in `hack/e2e.go`) for more options, such as reusing an existing cluster.
|
||||
The tests are built into a single binary which can be run used to deploy a
|
||||
Kubernetes system or run tests against an already-deployed Kubernetes system.
|
||||
See `go run hack/e2e.go --help` (or the flag definitions in `hack/e2e.go`) for
|
||||
more options, such as reusing an existing cluster.
|
||||
|
||||
### Cleaning up
|
||||
|
||||
During a run, pressing `control-C` should result in an orderly shutdown, but if something goes wrong and you still have some VMs running you can force a cleanup with this command:
|
||||
During a run, pressing `control-C` should result in an orderly shutdown, but if
|
||||
something goes wrong and you still have some VMs running you can force a cleanup
|
||||
with this command:
|
||||
|
||||
```sh
|
||||
go run hack/e2e.go -v --down
|
||||
@ -144,24 +170,49 @@ go run hack/e2e.go -v --down
|
||||
|
||||
### Bringing up a cluster for testing
|
||||
|
||||
If you want, you may bring up a cluster in some other manner and run tests against it. To do so, or to do other non-standard test things, you can pass arguments into Ginkgo using `--test_args` (e.g. see above). For the purposes of brevity, we will look at a subset of the options, which are listed below:
|
||||
If you want, you may bring up a cluster in some other manner and run tests
|
||||
against it. To do so, or to do other non-standard test things, you can pass
|
||||
arguments into Ginkgo using `--test_args` (e.g. see above). For the purposes of
|
||||
brevity, we will look at a subset of the options, which are listed below:
|
||||
|
||||
```
|
||||
-ginkgo.dryRun=false: If set, ginkgo will walk the test hierarchy without actually running anything. Best paired with -v.
|
||||
-ginkgo.failFast=false: If set, ginkgo will stop running a test suite after a failure occurs.
|
||||
-ginkgo.failOnPending=false: If set, ginkgo will mark the test suite as failed if any specs are pending.
|
||||
-ginkgo.focus="": If set, ginkgo will only run specs that match this regular expression.
|
||||
-ginkgo.skip="": If set, ginkgo will only run specs that do not match this regular expression.
|
||||
-ginkgo.trace=false: If set, default reporter prints out the full stack trace when a failure occurs
|
||||
-ginkgo.dryRun=false: If set, ginkgo will walk the test hierarchy without
|
||||
actually running anything. Best paired with -v.
|
||||
|
||||
-ginkgo.failFast=false: If set, ginkgo will stop running a test suite after a
|
||||
failure occurs.
|
||||
|
||||
-ginkgo.failOnPending=false: If set, ginkgo will mark the test suite as failed
|
||||
if any specs are pending.
|
||||
|
||||
-ginkgo.focus="": If set, ginkgo will only run specs that match this regular
|
||||
expression.
|
||||
|
||||
-ginkgo.skip="": If set, ginkgo will only run specs that do not match this
|
||||
regular expression.
|
||||
|
||||
-ginkgo.trace=false: If set, default reporter prints out the full stack trace
|
||||
when a failure occurs
|
||||
|
||||
-ginkgo.v=false: If set, default reporter print out all specs as they begin.
|
||||
|
||||
-host="": The host, or api-server, to connect to
|
||||
|
||||
-kubeconfig="": Path to kubeconfig containing embedded authinfo.
|
||||
-prom-push-gateway="": The URL to prometheus gateway, so that metrics can be pushed during e2es and scraped by prometheus. Typically something like 127.0.0.1:9091.
|
||||
-provider="": The name of the Kubernetes provider (gce, gke, local, vagrant, etc.)
|
||||
-repo-root="../../": Root directory of kubernetes repository, for finding test files.
|
||||
|
||||
-prom-push-gateway="": The URL to prometheus gateway, so that metrics can be
|
||||
pushed during e2es and scraped by prometheus. Typically something like
|
||||
127.0.0.1:9091.
|
||||
|
||||
-provider="": The name of the Kubernetes provider (gce, gke, local, vagrant,
|
||||
etc.)
|
||||
|
||||
-repo-root="../../": Root directory of kubernetes repository, for finding test
|
||||
files.
|
||||
```
|
||||
|
||||
Prior to running the tests, you may want to first create a simple auth file in your home directory, e.g. `$HOME/.kube/config` , with the following:
|
||||
Prior to running the tests, you may want to first create a simple auth file in
|
||||
your home directory, e.g. `$HOME/.kube/config`, with the following:
|
||||
|
||||
```
|
||||
{
|
||||
@ -170,12 +221,16 @@ Prior to running the tests, you may want to first create a simple auth file in y
|
||||
}
|
||||
```
|
||||
|
||||
As mentioned earlier there are a host of other options that are available, but they are left to the developer.
|
||||
As mentioned earlier there are a host of other options that are available, but
|
||||
they are left to the developer.
|
||||
|
||||
**NOTE:** If you are running tests on a local cluster repeatedly, you may need to periodically perform some manual cleanup.
|
||||
**NOTE:** If you are running tests on a local cluster repeatedly, you may need
|
||||
to periodically perform some manual cleanup:
|
||||
|
||||
- `rm -rf /var/run/kubernetes`, clear kube generated credentials, sometimes stale permissions can cause problems.
|
||||
- `sudo iptables -F`, clear ip tables rules left by the kube-proxy.
|
||||
- `rm -rf /var/run/kubernetes`, clear kube generated credentials, sometimes
|
||||
stale permissions can cause problems.
|
||||
|
||||
- `sudo iptables -F`, clear ip tables rules left by the kube-proxy.
|
||||
|
||||
### Debugging clusters
|
||||
|
||||
@ -184,22 +239,22 @@ state to debug a failed e2e test, you can use the `cluster/log-dump.sh` script
|
||||
to gather logs.
|
||||
|
||||
This script requires that the cluster provider supports ssh. Assuming it does,
|
||||
running
|
||||
running:
|
||||
|
||||
```
|
||||
cluster/log-dump.sh <directory>
|
||||
````
|
||||
|
||||
will ssh to the master and all nodes
|
||||
and download a variety of useful logs to the provided directory (which should
|
||||
already exist).
|
||||
will ssh to the master and all nodes and download a variety of useful logs to
|
||||
the provided directory (which should already exist).
|
||||
|
||||
The Google-run Jenkins builds automatically collected these logs for every
|
||||
build, saving them in the `artifacts` directory uploaded to GCS.
|
||||
|
||||
### Local clusters
|
||||
|
||||
It can be much faster to iterate on a local cluster instead of a cloud-based one. To start a local cluster, you can run:
|
||||
It can be much faster to iterate on a local cluster instead of a cloud-based
|
||||
one. To start a local cluster, you can run:
|
||||
|
||||
```sh
|
||||
# The PATH construction is needed because PATH is one of the special-cased
|
||||
@ -207,11 +262,13 @@ It can be much faster to iterate on a local cluster instead of a cloud-based one
|
||||
sudo PATH=$PATH hack/local-up-cluster.sh
|
||||
```
|
||||
|
||||
This will start a single-node Kubernetes cluster than runs pods using the local docker daemon. Press Control-C to stop the cluster.
|
||||
This will start a single-node Kubernetes cluster than runs pods using the local
|
||||
docker daemon. Press Control-C to stop the cluster.
|
||||
|
||||
#### Testing against local clusters
|
||||
|
||||
In order to run an E2E test against a locally running cluster, point the tests at a custom host directly:
|
||||
In order to run an E2E test against a locally running cluster, point the tests
|
||||
at a custom host directly:
|
||||
|
||||
```sh
|
||||
export KUBECONFIG=/path/to/kubeconfig
|
||||
@ -226,26 +283,72 @@ go run hack/e2e.go -v --test_args="--host=http://127.0.0.1:8080" --ginkgo.focus=
|
||||
|
||||
## Kinds of tests
|
||||
|
||||
We are working on implementing clearer partitioning of our e2e tests to make running a known set of tests easier (#10548). Tests can be labeled with any of the following labels, in order of increasing precedence (that is, each label listed below supersedes the previous ones):
|
||||
We are working on implementing clearer partitioning of our e2e tests to make
|
||||
running a known set of tests easier (#10548). Tests can be labeled with any of
|
||||
the following labels, in order of increasing precedence (that is, each label
|
||||
listed below supersedes the previous ones):
|
||||
|
||||
- If a test has no labels, it is expected to run fast (under five minutes), be able to be run in parallel, and be consistent.
|
||||
- `[Slow]`: If a test takes more than five minutes to run (by itself or in parallel with many other tests), it is labeled `[Slow]`. This partition allows us to run almost all of our tests quickly in parallel, without waiting for the stragglers to finish.
|
||||
- `[Serial]`: If a test cannot be run in parallel with other tests (e.g. it takes too many resources or restarts nodes), it is labeled `[Serial]`, and should be run in serial as part of a separate suite.
|
||||
- `[Disruptive]`: If a test restarts components that might cause other tests to fail or break the cluster completely, it is labeled `[Disruptive]`. Any `[Disruptive]` test is also assumed to qualify for the `[Serial]` label, but need not be labeled as both. These tests are not run against soak clusters to avoid restarting components.
|
||||
- `[Flaky]`: If a test is found to be flaky and we have decided that it's too hard to fix in the short term (e.g. it's going to take a full engineer-week), it receives the `[Flaky]` label until it is fixed. The `[Flaky]` label should be used very sparingly, and should be accompanied with a reference to the issue for de-flaking the test, because while a test remains labeled `[Flaky]`, it is not monitored closely in CI. `[Flaky]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given.
|
||||
- `[Feature:.+]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:.+]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`. `[Feature:.+]` tests are not run in our core suites, instead running in custom suites. If a feature is experimental or alpha and is not enabled by default due to being incomplete or potentially subject to breaking changes, it does *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s) (see #continuous_integration below).
|
||||
- If a test has no labels, it is expected to run fast (under five minutes), be
|
||||
able to be run in parallel, and be consistent.
|
||||
|
||||
- `[Slow]`: If a test takes more than five minutes to run (by itself or in
|
||||
parallel with many other tests), it is labeled `[Slow]`. This partition allows
|
||||
us to run almost all of our tests quickly in parallel, without waiting for the
|
||||
stragglers to finish.
|
||||
|
||||
- `[Serial]`: If a test cannot be run in parallel with other tests (e.g. it
|
||||
takes too many resources or restarts nodes), it is labeled `[Serial]`, and
|
||||
should be run in serial as part of a separate suite.
|
||||
|
||||
- `[Disruptive]`: If a test restarts components that might cause other tests
|
||||
to fail or break the cluster completely, it is labeled `[Disruptive]`. Any
|
||||
`[Disruptive]` test is also assumed to qualify for the `[Serial]` label, but
|
||||
need not be labeled as both. These tests are not run against soak clusters to
|
||||
avoid restarting components.
|
||||
|
||||
- `[Flaky]`: If a test is found to be flaky and we have decided that it's too
|
||||
hard to fix in the short term (e.g. it's going to take a full engineer-week), it
|
||||
receives the `[Flaky]` label until it is fixed. The `[Flaky]` label should be
|
||||
used very sparingly, and should be accompanied with a reference to the issue for
|
||||
de-flaking the test, because while a test remains labeled `[Flaky]`, it is not
|
||||
monitored closely in CI. `[Flaky]` tests are by default not run, unless a
|
||||
`focus` or `skip` argument is explicitly given.
|
||||
|
||||
- `[Feature:.+]`: If a test has non-default requirements to run or targets
|
||||
some non-core functionality, and thus should not be run as part of the standard
|
||||
suite, it receives a `[Feature:.+]` label, e.g. `[Feature:Performance]` or
|
||||
`[Feature:Ingress]`. `[Feature:.+]` tests are not run in our core suites,
|
||||
instead running in custom suites. If a feature is experimental or alpha and is
|
||||
not enabled by default due to being incomplete or potentially subject to
|
||||
breaking changes, it does *not* block the merge-queue, and thus should run in
|
||||
some separate test suites owned by the feature owner(s)
|
||||
(see [Continuous Integration](#continuous-integration) below).
|
||||
|
||||
### Conformance tests
|
||||
|
||||
Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede any other labels.
|
||||
Finally, `[Conformance]` tests represent a subset of the e2e-tests we expect to
|
||||
pass on **any** Kubernetes cluster. The `[Conformance]` label does not supersede
|
||||
any other labels.
|
||||
|
||||
As each new release of Kubernetes providers new functionality, the subset of tests necessary to demonstrate conformance grows with each release. Conformance is thus considered versioned, with the same backwards compatibility guarantees as laid out in [our versioning policy](../design/versioning.md#supported-releases). Conformance tests for a given version should be run off of the release branch that corresponds to that version. Thus `v1.2` conformance tests would be run from the head of the `release-1.2` branch. eg:
|
||||
As each new release of Kubernetes providers new functionality, the subset of
|
||||
tests necessary to demonstrate conformance grows with each release. Conformance
|
||||
is thus considered versioned, with the same backwards compatibility guarantees
|
||||
as laid out in [our versioning policy](../design/versioning.md#supported-releases).
|
||||
Conformance tests for a given version should be run off of the release branch
|
||||
that corresponds to that version. Thus `v1.2` conformance tests would be run
|
||||
from the head of the `release-1.2` branch. eg:
|
||||
|
||||
- A v1.3 development cluster should pass v1.1, v1.2 conformance tests
|
||||
- A v1.2 cluster should pass v1.1, v1.2 conformance tests
|
||||
- A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2 conformance tests
|
||||
|
||||
Conformance tests are designed to be run with no cloud provider configured. Conformance tests can be run against clusters that have not been created with `hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and credentials.
|
||||
- A v1.2 cluster should pass v1.1, v1.2 conformance tests
|
||||
|
||||
- A v1.1 cluster should pass v1.0, v1.1 conformance tests, and fail v1.2
|
||||
conformance tests
|
||||
|
||||
Conformance tests are designed to be run with no cloud provider configured.
|
||||
Conformance tests can be run against clusters that have not been created with
|
||||
`hack/e2e.go`, just provide a kubeconfig with the appropriate endpoint and
|
||||
credentials.
|
||||
|
||||
```sh
|
||||
# setup for conformance tests
|
||||
@ -257,20 +360,30 @@ go run hack/e2e.go -v --test --test_args="--ginkgo.focus=\[Conformance\]"
|
||||
|
||||
# run all parallel-safe conformance tests in parallel
|
||||
GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Conformance\] --ginkgo.skip=\[Serial\]"
|
||||
|
||||
# ... and finish up with remaining tests in serial
|
||||
go run hack/e2e.go --v --test --test_args="--ginkgo.focus=\[Serial\].*\[Conformance\]"
|
||||
```
|
||||
|
||||
### Defining Conformance Subset
|
||||
|
||||
It is impossible to define the entire space of Conformance tests without knowing the future, so instead, we define the compliment of conformance tests, below.
|
||||
It is impossible to define the entire space of Conformance tests without knowing
|
||||
the future, so instead, we define the compliment of conformance tests, below
|
||||
(`Please update this with companion PRs as necessary`):
|
||||
|
||||
Please update this with companion PRs as necessary.
|
||||
- A conformance test cannot test cloud provider specific features (i.e. GCE
|
||||
monitoring, S3 Bucketing, ...)
|
||||
|
||||
- A conformance test cannot test cloud provider specific features (i.e. GCE monitoring, S3 Bucketing, ...)
|
||||
- A conformance test cannot rely on any particular non-standard file system permissions granted to containers or users (i.e. sharing writable host /tmp with a container)
|
||||
- A conformance test cannot rely on any binaries that are not required for the linux kernel or for a kubelet to run (i.e. git)
|
||||
- A conformance test cannot test a feature which obviously cannot be supported on a broad range of platforms (i.e. testing of multiple disk mounts, GPUs, high density)
|
||||
- A conformance test cannot rely on any particular non-standard file system
|
||||
permissions granted to containers or users (i.e. sharing writable host /tmp with
|
||||
a container)
|
||||
|
||||
- A conformance test cannot rely on any binaries that are not required for the
|
||||
linux kernel or for a kubelet to run (i.e. git)
|
||||
|
||||
- A conformance test cannot test a feature which obviously cannot be supported
|
||||
on a broad range of platforms (i.e. testing of multiple disk mounts, GPUs, high
|
||||
density)
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
@ -278,74 +391,149 @@ A quick overview of how we run e2e CI on Kubernetes.
|
||||
|
||||
### What is CI?
|
||||
|
||||
We run a battery of `e2e` tests against `HEAD` of the master branch on a continuous basis, and block merges via the [submit queue](http://submit-queue.k8s.io/) on a subset of those tests if they fail (the subset is defined in the [munger config](https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go) via the `jenkins-jobs` flag; note we also block on `kubernetes-build` and `kubernetes-test-go` jobs for build and unit and integration tests).
|
||||
We run a battery of `e2e` tests against `HEAD` of the master branch on a
|
||||
continuous basis, and block merges via the [submit
|
||||
queue](http://submit-queue.k8s.io/) on a subset of those tests if they fail (the
|
||||
subset is defined in the [munger config]
|
||||
(https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go)
|
||||
via the `jenkins-jobs` flag; note we also block on `kubernetes-build` and
|
||||
`kubernetes-test-go` jobs for build and unit and integration tests).
|
||||
|
||||
CI results can be found at [ci-test.k8s.io](http://ci-test.k8s.io), e.g. [ci-test.k8s.io/kubernetes-e2e-gce/10594](http://ci-test.k8s.io/kubernetes-e2e-gce/10594).
|
||||
CI results can be found at [ci-test.k8s.io](http://ci-test.k8s.io), e.g.
|
||||
[ci-test.k8s.io/kubernetes-e2e-gce/10594](http://ci-test.k8s.io/kubernetes-e2e-gce/10594).
|
||||
|
||||
### What runs in CI?
|
||||
|
||||
We run all default tests (those that aren't marked `[Flaky]` or `[Feature:.+]`) against GCE and GKE. To minimize the time from regression-to-green-run, we partition tests across different jobs:
|
||||
We run all default tests (those that aren't marked `[Flaky]` or `[Feature:.+]`)
|
||||
against GCE and GKE. To minimize the time from regression-to-green-run, we
|
||||
partition tests across different jobs:
|
||||
|
||||
- `kubernetes-e2e-<provider>` runs all non-`[Slow]`, non-`[Serial]`, non-`[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in parallel.
|
||||
- `kubernetes-e2e-<provider>-slow` runs all `[Slow]`, non-`[Serial]`, non-`[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in parallel.
|
||||
- `kubernetes-e2e-<provider>-serial` runs all `[Serial]` and `[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in serial.
|
||||
- `kubernetes-e2e-<provider>` runs all non-`[Slow]`, non-`[Serial]`,
|
||||
non-`[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in parallel.
|
||||
|
||||
We also run non-default tests if the tests exercise general-availability ("GA") features that require a special environment to run in, e.g. `kubernetes-e2e-gce-scalability` and `kubernetes-kubemark-gce`, which test for Kubernetes performance.
|
||||
- `kubernetes-e2e-<provider>-slow` runs all `[Slow]`, non-`[Serial]`,
|
||||
non-`[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in parallel.
|
||||
|
||||
- `kubernetes-e2e-<provider>-serial` runs all `[Serial]` and `[Disruptive]`,
|
||||
non-`[Flaky]`, non-`[Feature:.+]` tests in serial.
|
||||
|
||||
We also run non-default tests if the tests exercise general-availability ("GA")
|
||||
features that require a special environment to run in, e.g.
|
||||
`kubernetes-e2e-gce-scalability` and `kubernetes-kubemark-gce`, which test for
|
||||
Kubernetes performance.
|
||||
|
||||
#### Non-default tests
|
||||
|
||||
Many `[Feature:.+]` tests we don't run in CI. These tests are for features that are experimental (often in the `experimental` API), and aren't enabled by default.
|
||||
Many `[Feature:.+]` tests we don't run in CI. These tests are for features that
|
||||
are experimental (often in the `experimental` API), and aren't enabled by
|
||||
default.
|
||||
|
||||
### The PR-builder
|
||||
|
||||
We also run a battery of tests against every PR before we merge it. These tests are equivalent to `kubernetes-gce`: it runs all non-`[Slow]`, non-`[Serial]`, non-`[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in parallel. These tests are considered "smoke tests" to give a decent signal that the PR doesn't break most functionality. Results for you PR can be found at [pr-test.k8s.io](http://pr-test.k8s.io), e.g. [pr-test.k8s.io/20354](http://pr-test.k8s.io/20354) for #20354.
|
||||
We also run a battery of tests against every PR before we merge it. These tests
|
||||
are equivalent to `kubernetes-gce`: it runs all non-`[Slow]`, non-`[Serial]`,
|
||||
non-`[Disruptive]`, non-`[Flaky]`, non-`[Feature:.+]` tests in parallel. These
|
||||
tests are considered "smoke tests" to give a decent signal that the PR doesn't
|
||||
break most functionality. Results for your PR can be found at
|
||||
[pr-test.k8s.io](http://pr-test.k8s.io), e.g.
|
||||
[pr-test.k8s.io/20354](http://pr-test.k8s.io/20354) for #20354.
|
||||
|
||||
### Adding a test to CI
|
||||
|
||||
As mentioned above, prior to adding a new test, it is a good idea to perform a `-ginkgo.dryRun=true` on the system, in order to see if a behavior is already being tested, or to determine if it may be possible to augment an existing set of tests for a specific use case.
|
||||
As mentioned above, prior to adding a new test, it is a good idea to perform a
|
||||
`-ginkgo.dryRun=true` on the system, in order to see if a behavior is already
|
||||
being tested, or to determine if it may be possible to augment an existing set
|
||||
of tests for a specific use case.
|
||||
|
||||
If a behavior does not currently have coverage and a developer wishes to add a new e2e test, navigate to the ./test/e2e directory and create a new test using the existing suite as a guide.
|
||||
If a behavior does not currently have coverage and a developer wishes to add a
|
||||
new e2e test, navigate to the ./test/e2e directory and create a new test using
|
||||
the existing suite as a guide.
|
||||
|
||||
TODO(#20357): Create a self-documented example which has been disabled, but can be copied to create new tests and outlines the capabilities and libraries used.
|
||||
TODO(#20357): Create a self-documented example which has been disabled, but can
|
||||
be copied to create new tests and outlines the capabilities and libraries used.
|
||||
|
||||
When writing a test, consult #kinds_of_tests above to determine how your test should be marked, (e.g. `[Slow]`, `[Serial]`; remember, by default we assume a test can run in parallel with other tests!).
|
||||
When writing a test, consult #kinds_of_tests above to determine how your test
|
||||
should be marked, (e.g. `[Slow]`, `[Serial]`; remember, by default we assume a
|
||||
test can run in parallel with other tests!).
|
||||
|
||||
When first adding a test it should *not* go straight into CI, because failures block ordinary development. A test should only be added to CI after is has been running in some non-CI suite long enough to establish a track record showing that the test does not fail when run against *working* software. Note also that tests running in CI are generally running on a well-loaded cluster, so must contend for resources; see above about [kinds of tests](#kinds_of_tests).
|
||||
When first adding a test it should *not* go straight into CI, because failures
|
||||
block ordinary development. A test should only be added to CI after is has been
|
||||
running in some non-CI suite long enough to establish a track record showing
|
||||
that the test does not fail when run against *working* software. Note also that
|
||||
tests running in CI are generally running on a well-loaded cluster, so must
|
||||
contend for resources; see above about [kinds of tests](#kinds_of_tests).
|
||||
|
||||
Generally, a feature starts as `experimental`, and will be run in some suite owned by the team developing the feature. If a feature is in beta or GA, it *should* block the merge-queue. In moving from experimental to beta or GA, tests that are expected to pass by default should simply remove the `[Feature:.+]` label, and will be incorporated into our core suites. If tests are not expected to pass by default, (e.g. they require a special environment such as added quota,) they should remain with the `[Feature:.+]` label, and the suites that run them should be incorporated into the [munger config](https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go) via the `jenkins-jobs` flag.
|
||||
Generally, a feature starts as `experimental`, and will be run in some suite
|
||||
owned by the team developing the feature. If a feature is in beta or GA, it
|
||||
*should* block the merge-queue. In moving from experimental to beta or GA, tests
|
||||
that are expected to pass by default should simply remove the `[Feature:.+]`
|
||||
label, and will be incorporated into our core suites. If tests are not expected
|
||||
to pass by default, (e.g. they require a special environment such as added
|
||||
quota,) they should remain with the `[Feature:.+]` label, and the suites that
|
||||
run them should be incorporated into the
|
||||
[munger config](https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go)
|
||||
via the `jenkins-jobs` flag.
|
||||
|
||||
Occasionally, we'll want to add tests to better exercise features that are already GA. These tests also shouldn't go straight to CI. They should begin by being marked as `[Flaky]` to be run outside of CI, and once a track-record for them is established, they may be promoted out of `[Flaky]`.
|
||||
Occasionally, we'll want to add tests to better exercise features that are
|
||||
already GA. These tests also shouldn't go straight to CI. They should begin by
|
||||
being marked as `[Flaky]` to be run outside of CI, and once a track-record for
|
||||
them is established, they may be promoted out of `[Flaky]`.
|
||||
|
||||
### Moving a test out of CI
|
||||
|
||||
If we have determined that a test is known-flaky and cannot be fixed in the short-term, we may move it out of CI indefinitely. This move should be used sparingly, as it effectively means that we have no coverage of that test. When a test if demoted, it should be marked `[Flaky]` with a comment accompanying the label with a reference to an issue opened to fix the test.
|
||||
If we have determined that a test is known-flaky and cannot be fixed in the
|
||||
short-term, we may move it out of CI indefinitely. This move should be used
|
||||
sparingly, as it effectively means that we have no coverage of that test. When a
|
||||
test is demoted, it should be marked `[Flaky]` with a comment accompanying the
|
||||
label with a reference to an issue opened to fix the test.
|
||||
|
||||
## Performance Evaluation
|
||||
|
||||
Another benefit of the e2e tests is the ability to create reproducible loads on the system, which can then be used to determine the responsiveness, or analyze other characteristics of the system. For example, the density tests load the system to 30,50,100 pods per/node and measures the different characteristics of the system, such as throughput, api-latency, etc.
|
||||
Another benefit of the e2e tests is the ability to create reproducible loads on
|
||||
the system, which can then be used to determine the responsiveness, or analyze
|
||||
other characteristics of the system. For example, the density tests load the
|
||||
system to 30,50,100 pods per/node and measures the different characteristics of
|
||||
the system, such as throughput, api-latency, etc.
|
||||
|
||||
For a good overview of how we analyze performance data, please read the following [post](http://blog.kubernetes.io/2015/09/kubernetes-performance-measurements-and.html)
|
||||
For a good overview of how we analyze performance data, please read the
|
||||
following [post](http://blog.kubernetes.io/2015/09/kubernetes-performance-measurements-and.html)
|
||||
|
||||
For developers who are interested in doing their own performance analysis, we recommend setting up [prometheus](http://prometheus.io/) for data collection, and using [promdash](http://prometheus.io/docs/visualization/promdash/) to visualize the data. There also exists the option of pushing your own metrics in from the tests using a [prom-push-gateway](http://prometheus.io/docs/instrumenting/pushing/). Containers for all of these components can be found [here](https://hub.docker.com/u/prom/).
|
||||
For developers who are interested in doing their own performance analysis, we
|
||||
recommend setting up [prometheus](http://prometheus.io/) for data collection,
|
||||
and using [promdash](http://prometheus.io/docs/visualization/promdash/) to
|
||||
visualize the data. There also exists the option of pushing your own metrics in
|
||||
from the tests using a
|
||||
[prom-push-gateway](http://prometheus.io/docs/instrumenting/pushing/).
|
||||
Containers for all of these components can be found
|
||||
[here](https://hub.docker.com/u/prom/).
|
||||
|
||||
For more accurate measurements, you may wish to set up prometheus external to kubernetes in an environment where it can access the major system components (api-server, controller-manager, scheduler). This is especially useful when attempting to gather metrics in a load-balanced api-server environment, because all api-servers can be analyzed independently as well as collectively. On startup, configuration file is passed to prometheus that specifies the endpoints that prometheus will scrape, as well as the sampling interval.
|
||||
For more accurate measurements, you may wish to set up prometheus external to
|
||||
kubernetes in an environment where it can access the major system components
|
||||
(api-server, controller-manager, scheduler). This is especially useful when
|
||||
attempting to gather metrics in a load-balanced api-server environment, because
|
||||
all api-servers can be analyzed independently as well as collectively. On
|
||||
startup, configuration file is passed to prometheus that specifies the endpoints
|
||||
that prometheus will scrape, as well as the sampling interval.
|
||||
|
||||
```
|
||||
#prometheus.conf
|
||||
job: {
|
||||
name: "kubernetes"
|
||||
scrape_interval: "1s"
|
||||
target_group: {
|
||||
# apiserver(s)
|
||||
target: "http://localhost:8080/metrics"
|
||||
# scheduler
|
||||
target: "http://localhost:10251/metrics"
|
||||
# controller-manager
|
||||
target: "http://localhost:10252/metrics"
|
||||
}
|
||||
name: "kubernetes"
|
||||
scrape_interval: "1s"
|
||||
target_group: {
|
||||
# apiserver(s)
|
||||
target: "http://localhost:8080/metrics"
|
||||
# scheduler
|
||||
target: "http://localhost:10251/metrics"
|
||||
# controller-manager
|
||||
target: "http://localhost:10252/metrics"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Once prometheus is scraping the kubernetes endpoints, that data can then be plotted using promdash, and alerts can be created against the assortment of metrics that kubernetes provides.
|
||||
Once prometheus is scraping the kubernetes endpoints, that data can then be
|
||||
plotted using promdash, and alerts can be created against the assortment of
|
||||
metrics that kubernetes provides.
|
||||
|
||||
## One More Thing
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user