Merge pull request #25089 from janetkuo/test-convention-doc

Automatic merge from submit-queue

Update testing convention docs

@bgrant0607 @kubernetes/docs 
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]()

docs/devel/coding-conventions.md

@@ -31,7 +31,24 @@ Documentation for other releases can be found at
 <!-- END STRIP_FOR_RELEASE -->
 
 <!-- END MUNGE: UNVERSIONED_WARNING -->
-Code conventions
+
+# Coding Conventions
+
+Updated: 5/3/2016
+
+**Table of Contents**
+<!-- BEGIN MUNGE: GENERATED_TOC -->
+
+- [Coding Conventions](#coding-conventions)
+  - [Code conventions](#code-conventions)
+  - [Testing conventions](#testing-conventions)
+  - [Directory and file conventions](#directory-and-file-conventions)
+  - [Coding advice](#coding-advice)
+
+<!-- END MUNGE: GENERATED_TOC -->
+
+## Code conventions
+
   - Bash
     - https://google-styleguide.googlecode.com/svn/trunk/shell.xml
     - Ensure that build, release, test, and cluster-management scripts run on OS X
@@ -58,15 +75,19 @@ Code conventions
     - [Kubectl conventions](kubectl-conventions.md)
     - [Logging conventions](logging.md)
 
-Testing conventions
+## Testing conventions
+
   - All new packages and most new significant functionality must come with unit tests
   - Table-driven tests are preferred for testing multiple scenarios/inputs; for example, see [TestNamespaceAuthorization](../../test/integration/auth_test.go)
   - Significant features should come with integration (test/integration) and/or [end-to-end (test/e2e) tests](e2e-tests.md)
     - Including new kubectl commands and major features of existing commands
   - Unit tests must pass on OS X and Windows platforms - if you use Linux specific features, your test case must either be skipped on windows or compiled out (skipped is better when running Linux specific commands, compiled out is required when your code does not compile on Windows).
+  - Avoid relying on Docker hub (e.g. pull from Docker hub). Use gcr.io instead.
+  - Avoid waiting for a short amount of time (or without waiting) and expect an asynchronous thing to happen (e.g. wait for 1 seconds and expect a Pod to be running). Wait and retry instead.
   - See the [testing guide](testing.md) for additional testing advice.
 
-Directory and file conventions
+## Directory and file conventions
+
   - Avoid package sprawl. Find an appropriate subdirectory for new packages. (See [#4851](http://issues.k8s.io/4851) for discussion.)
     - Libraries with no more appropriate home belong in new package subdirectories of pkg/util
   - Avoid general utility packages. Packages called "util" are suspect. Instead, derive a name that describes your desired function. For example, the utility functions dealing with waiting for operations are in the "wait" package and include functionality like Poll. So the full name is wait.Poll
@@ -85,7 +106,8 @@ Directory and file conventions
     - Third-party code must include licenses
     - This includes modified third-party code and excerpts, as well
 
-Coding advice
+## Coding advice
+
   - Go
     - [Go landmines](https://gist.github.com/lavalamp/4bd23295a9f32706a48f)
 

docs/devel/e2e-tests.md

@@ -34,6 +34,35 @@ Documentation for other releases can be found at
 
 # End-to-End Testing in Kubernetes
 
+Updated: 5/3/2016
+
+**Table of Contents**
+<!-- BEGIN MUNGE: GENERATED_TOC -->
+
+- [End-to-End Testing in Kubernetes](#end-to-end-testing-in-kubernetes)
+  - [Overview](#overview)
+  - [Building and Running the Tests](#building-and-running-the-tests)
+    - [Cleaning up](#cleaning-up)
+  - [Advanced testing](#advanced-testing)
+    - [Bringing up a cluster for testing](#bringing-up-a-cluster-for-testing)
+    - [Debugging clusters](#debugging-clusters)
+    - [Local clusters](#local-clusters)
+      - [Testing against local clusters](#testing-against-local-clusters)
+  - [Kinds of tests](#kinds-of-tests)
+    - [Conformance tests](#conformance-tests)
+    - [Defining Conformance Subset](#defining-conformance-subset)
+  - [Continuous Integration](#continuous-integration)
+    - [What is CI?](#what-is-ci)
+    - [What runs in CI?](#what-runs-in-ci)
+      - [Non-default tests](#non-default-tests)
+    - [The PR-builder](#the-pr-builder)
+    - [Adding a test to CI](#adding-a-test-to-ci)
+    - [Moving a test out of CI](#moving-a-test-out-of-ci)
+  - [Performance Evaluation](#performance-evaluation)
+  - [One More Thing](#one-more-thing)
+
+<!-- END MUNGE: GENERATED_TOC -->
+
 ## Overview
 
 End-to-end (e2e) tests for Kubernetes provide a mechanism to test end-to-end behavior of the system, and is the last signal to ensure end user operations match developer specifications.  Although unit and integration tests should ideally provide a good signal, the reality is in a distributed system like Kubernetes it is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at the system level.  e2e testing is very costly, both in time to run tests and difficulty debugging, though: it takes a long time to build, deploy, and exercise a cluster.  Thus, the primary objectives of the e2e tests are to ensure a consistent and reliable behavior of the kubernetes code base, and to catch hard-to-test bugs before users do, when unit and integration tests are insufficient.
@@ -318,6 +347,10 @@ job: {
 
 Once prometheus is scraping the kubernetes endpoints, that data can then be plotted using promdash, and alerts can be created against the assortment of metrics that kubernetes provides.
 
+## One More Thing
+
+You should also know the [testing conventions](coding-conventions.md#testing-conventions).
+
 **HAPPY TESTING!**
 
 

docs/devel/testing.md

@@ -29,6 +29,25 @@ Documentation for other releases can be found at
 
 # Testing guide
 
+Updated: 5/3/2016
+
+**Table of Contents**
+<!-- BEGIN MUNGE: GENERATED_TOC -->
+
+- [Testing guide](#testing-guide)
+  - [Unit tests](#unit-tests)
+    - [Run all unit tests](#run-all-unit-tests)
+    - [Run some unit tests](#run-some-unit-tests)
+    - [Stress running unit tests](#stress-running-unit-tests)
+    - [Unit test coverage](#unit-test-coverage)
+    - [Benchmark unit tests](#benchmark-unit-tests)
+  - [Integration tests](#integration-tests)
+    - [Install etcd dependency](#install-etcd-dependency)
+    - [Run integration tests](#run-integration-tests)
+  - [End-to-End tests](#end-to-end-tests)
+
+<!-- END MUNGE: GENERATED_TOC -->
+
 This assumes you already read the [development guide](development.md) to
 install go, godeps, and configure your git client.