From 7910b43cd39c847adad73cc909424cc3ab472fdc Mon Sep 17 00:00:00 2001
From: Patrick Ohly <patrick.ohly@intel.com>
Date: Thu, 26 Jun 2025 07:57:59 +0200
Subject: [PATCH] scheduler_perf: document benchstat usage

With benchstat it's easy to do before/after comparisons, but the section for
running benchmark didn't mention it at all and didn't work as shown there:
- benchmark results must be printed (FULL_LOG)
- timeout might have been too short (KUBE_TIMEOUT)
- only "short" benchmarks ran (SHORT)
- klog log output must be redirected (ARTIFACTS)
---
 test/integration/scheduler_perf/README.md | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/test/integration/scheduler_perf/README.md b/test/integration/scheduler_perf/README.md
index 9571ffd4435..1ebc61657ac 100644
--- a/test/integration/scheduler_perf/README.md
+++ b/test/integration/scheduler_perf/README.md
@@ -33,9 +33,14 @@ Currently the test suite has the following:
 
 ```shell
 # In Kubernetes root path
-make test-integration WHAT=./test/integration/scheduler_perf/... ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling"
+make test-integration WHAT=./test/integration/scheduler_perf/... KUBE_TIMEOUT=-timeout=1h ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" FULL_LOG=y ARTIFACTS=/tmp SHORT=-short=false KUBE_TEST_ARGS='-run=^$ -benchtime=1ns -bench=BenchmarkPerfScheduling'
 ```
 
+The output can used for [`benchstat`](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)
+to summarize results or to do before/after
+comparisons. Add `-count=6` to `KUBE_TEST_ARGS` to get statistically relevant
+summaries.
+
 The benchmark suite runs all the tests specified under subdirectories split by topic (`<topic>/performance-config.yaml`).
 By default, it runs all workloads that have the "performance" label. In the configuration,
 labels can be added to a test case and/or individual workloads. Each workload also has
@@ -46,7 +51,7 @@ a comma-separated list of label names. Each label may have a `+` or `-` as prefi
 be set. For example, this runs all performance benchmarks except those that are labeled
 as "integration-test":
 ```shell
-make test-integration WHAT=./test/integration/scheduler_perf/... ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling -perf-scheduling-label-filter=performance,-integration-test"
+make test-integration WHAT=./test/integration/scheduler_perf/... KUBE_TIMEOUT=-timeout=1h ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" SHORT=-short=false ARTIFACTS=/tmp FULL_LOG=y KUBE_TEST_ARGS='-run=^$ -benchtime=1ns -bench=BenchmarkPerfScheduling -perf-scheduling-label-filter=performance,-integration-test'
 ```
 
 Once the benchmark is finished, JSON files with metrics are available in the subdirectories (`test/integration/scheduler_perf/config/<topic>`). 
@@ -60,16 +65,23 @@ Otherwise, the golang benchmark framework will try to run a test more than once
 
 ```shell
 # In Kubernetes root path
-make test-integration WHAT=./test/integration/scheduler_perf/... ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling/SchedulingBasic/5000Nodes/5000InitPods/1000PodsToSchedule"
+make test-integration WHAT=./test/integration/scheduler_perf/... KUBE_TIMEOUT=-timeout=1h ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" SHORT=-short=false ARTIFACTS=/tmp FULL_LOG=y KUBE_TEST_ARGS='-run=^$ -benchtime=1ns -bench=BenchmarkPerfScheduling/SchedulingBasic/5000Nodes/5000InitPods/1000PodsToSchedule'
 ```
 
 To produce a cpu profile:
 
 ```shell
 # In Kubernetes root path
-make test-integration WHAT=./test/integration/scheduler_perf/... KUBE_TIMEOUT="-timeout=3600s" ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=^$$ -benchtime=1ns -bench=BenchmarkPerfScheduling -cpuprofile ~/cpu-profile.out"
+make test-integration WHAT=./test/integration/scheduler_perf/... KUBE_TIMEOUT=-timeout=1h ETCD_LOGLEVEL=warn KUBE_TEST_VMODULE="''" FULL_LOG=y ARTIFACTS=/tmp SHORT=-short=false KUBE_TEST_ARGS='-run=^$ -benchtime=1ns -bench=BenchmarkPerfScheduling -cpuprofile ~/cpu-profile.out'
 ```
 
+Here some explanations for those parameters:
+- `KUBE_TIMEOUT=-timeout=1h`: benchmarks may have to run longer than the default 10 minutes
+- `FULL_LOG=y`: ensures that benchmark results are shown
+- `ARTIFACTS=/tmp`: redirects non-test log output, which is lengthy and breaks parsing by `benchstat`
+- `SHORT=-short=false`: enables running benchmarks which run sufficiently long to be meaningful
+- `-run=^$`: disables running unit tests
+
 ### How to configure benchmark tests
 
 Configuration files located under `<topic>/performance-config.yaml` contain a list of templates.