github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2026-01-04 23:17:50 +00:00
Code Issues Projects Releases Wiki Activity

devel/ tree 80col wrap and other minor edits

Browse Source 
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
This commit is contained in:
Mike Brown
2016-05-04 16:14:05 -05:00
parent 1642083e60
commit 77e63c5043
4 changed files with 254 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
docs/devel/node-performance-testing.md
View File

@@ -34,69 +34,76 @@ Documentation for other releases can be found at
# Measuring Node Performance
This document outlines the issues and pitfalls of measuring Node performance, as well as the tools
available.
This document outlines the issues and pitfalls of measuring Node performance, as
well as the tools available.
## Cluster Set-up
There are lots of factors which can affect node performance numbers, so care must be taken in
setting up the cluster to make the intended measurements. In addition to taking the following steps
into consideration, it is important to document precisely which setup was used. For example,
performance can vary wildly from commit-to-commit, so it is very important to **document which commit
There are lots of factors which can affect node performance numbers, so care
must be taken in setting up the cluster to make the intended measurements. In
addition to taking the following steps into consideration, it is important to
document precisely which setup was used. For example, performance can vary
wildly from commit-to-commit, so it is very important to **document which commit
or version** of Kubernetes was used, which Docker version was used, etc.
### Addon pods
Be aware of which addon pods are running on which nodes. By default Kubernetes runs 8 addon pods,
plus another 2 per node (`fluentd-elasticsearch` and `kube-proxy`) in the `kube-system`
namespace. The addon pods can be disabled for more consistent results, but doing so can also have
performance implications.
Be aware of which addon pods are running on which nodes. By default Kubernetes
runs 8 addon pods, plus another 2 per node (`fluentd-elasticsearch` and
`kube-proxy`) in the `kube-system` namespace. The addon pods can be disabled for
more consistent results, but doing so can also have performance implications.
For example, Heapster polls each node regularly to collect stats data. Disabling Heapster will hide
the performance cost of serving those stats in the Kubelet.
For example, Heapster polls each node regularly to collect stats data. Disabling
Heapster will hide the performance cost of serving those stats in the Kubelet.
#### Disabling Add-ons
Disabling addons is simple. Just ssh into the Kubernetes master and move the addon from
`/etc/kubernetes/addons/` to a backup location. More details [here](../../cluster/addons/).
Disabling addons is simple. Just ssh into the Kubernetes master and move the
addon from `/etc/kubernetes/addons/` to a backup location. More details
[here](../../cluster/addons/).
### Which / how many pods?
Performance will vary a lot between a node with 0 pods and a node with 100 pods. In many cases
you'll want to make measurements with several different amounts of pods. On a single node cluster
scaling a replication controller makes this easy, just make sure the system reaches a steady-state
before starting the measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100`
Performance will vary a lot between a node with 0 pods and a node with 100 pods.
In many cases you'll want to make measurements with several different amounts of
pods. On a single node cluster scaling a replication controller makes this easy,
just make sure the system reaches a steady-state before starting the
measurement. E.g. `kubectl scale replicationcontroller pause --replicas=100`
In most cases pause pods will yield the most consistent measurements since the system will not be
affected by pod load. However, in some special cases Kubernetes has been tuned to optimize pods that
are not doing anything, such as the cAdvisor housekeeping (stats gathering). In these cases,
performing a very light task (such as a simple network ping) can make a difference.
In most cases pause pods will yield the most consistent measurements since the
system will not be affected by pod load. However, in some special cases
Kubernetes has been tuned to optimize pods that are not doing anything, such as
the cAdvisor housekeeping (stats gathering). In these cases, performing a very
light task (such as a simple network ping) can make a difference.
Finally, you should also consider which features yours pods should be using. For example, if you
want to measure performance with probing, you should obviously use pods with liveness or readiness
probes configured. Likewise for volumes, number of containers, etc.
Finally, you should also consider which features yours pods should be using. For
example, if you want to measure performance with probing, you should obviously
use pods with liveness or readiness probes configured. Likewise for volumes,
number of containers, etc.
### Other Tips
**Number of nodes** - On the one hand, it can be easier to manage logs, pods, environment etc. with
a single node to worry about. On the other hand, having multiple nodes will let you gather more
data in parallel for more robust sampling.
**Number of nodes** - On the one hand, it can be easier to manage logs, pods,
environment etc. with a single node to worry about. On the other hand, having
multiple nodes will let you gather more data in parallel for more robust
sampling.
## E2E Performance Test
There is an end-to-end test for collecting overall resource usage of node components:
[kubelet_perf.go](../../test/e2e/kubelet_perf.go). To
run the test, simply make sure you have an e2e cluster running (`go run hack/e2e.go -up`) and
[set up](#cluster-set-up) correctly.
There is an end-to-end test for collecting overall resource usage of node
components: [kubelet_perf.go](../../test/e2e/kubelet_perf.go). To
run the test, simply make sure you have an e2e cluster running (`go run
hack/e2e.go -up`) and [set up](#cluster-set-up) correctly.
Run the test with `go run hack/e2e.go -v -test
--test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to customise the number of
pods or other parameters of the test (remember to rerun `make WHAT=test/e2e/e2e.test` after you do).
--test_args="--ginkgo.focus=resource\susage\stracking"`. You may also wish to
customise the number of pods or other parameters of the test (remember to rerun
`make WHAT=test/e2e/e2e.test` after you do).
## Profiling
Kubelet installs the [go pprof handlers](https://golang.org/pkg/net/http/pprof/), which can be
queried for CPU profiles:
Kubelet installs the [go pprof handlers]
(https://golang.org/pkg/net/http/pprof/), which can be queried for CPU profiles:
```console
$ kubectl proxy &
@@ -109,13 +116,15 @@ $ go tool pprof -web $KUBELET_BIN $OUTPUT
`pprof` can also provide heap usage, from the `/debug/pprof/heap` endpoint
(e.g. `http://localhost:8001/api/v1/proxy/nodes/${NODE}:10250/debug/pprof/heap`).
More information on go profiling can be found [here](http://blog.golang.org/profiling-go-programs).
More information on go profiling can be found
[here](http://blog.golang.org/profiling-go-programs).
## Benchmarks
Before jumping through all the hoops to measure a live Kubernetes node in a real cluster, it is
worth considering whether the data you need can be gathered through a Benchmark test. Go provides a
really simple benchmarking mechanism, just add a unit test of the form:
Before jumping through all the hoops to measure a live Kubernetes node in a real
cluster, it is worth considering whether the data you need can be gathered
through a Benchmark test. Go provides a really simple benchmarking mechanism,
just add a unit test of the form:
```go
// In foo_test.go