Most of the time the managed fields were returned in the order as expected,
but not always. Semantically the order is irrelevant, so the comparison gets
relaxed to ignore it by normalizing first.
A mutex lock around decrementing runningPods was missing, leading to a data
race report in ci-kubernetes-integration-race-master:
WARNING: DATA RACE
Read at 0x00c001bd20a8 by goroutine 95696:
k8s.io/kubernetes/test/integration/scheduler_perf.createPodsSteadily.func7()
/home/prow/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:2133 +0x238
...
Previous write at 0x00c001bd20a8 by goroutine 101407:
k8s.io/kubernetes/test/integration/scheduler_perf.createPodsSteadily.func5()
/home/prow/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:2064 +0x1a4
...
This reverts commit 2e756e7412,
which addressed the data race only for test/integration/apiserver/oidc.
It also occurs for test/integration/controlplane/transformation:
WARNING: DATA RACE
Read at 0x000008b9de38 by goroutine 104616:
k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds.RegisterFlags()
/home/prow/go/src/k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds/admission.go:42 +0x30
k8s.io/kubernetes/pkg/kubeapiserver/options.registerAllAdmissionPluginFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/plugins.go:115 +0x64
k8s.io/kubernetes/pkg/kubeapiserver/options.(*AdmissionOptions).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/admission.go:73 +0x55
k8s.io/kubernetes/pkg/controlplane/apiserver/options.(*Options).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/options/options.go:154 +0x388
k8s.io/kubernetes/cmd/kube-apiserver/app/options.(*ServerRunOptions).Flags()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go:101 +0x73
k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:219 +0xb77
k8s.io/kubernetes/test/integration/controlplane/transformation.newTransformTest()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/transformation_test.go:131 +0x6d0
k8s.io/kubernetes/test/integration/controlplane/transformation.TestKMSv2Healthz()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/kmsv2_transformation_test.go:874 +0x1c8
testing.tRunner()
/usr/local/go/src/testing/testing.go:1792 +0x225
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1851 +0x44
Previous write at 0x000008b9de38 by goroutine 111926:
github.com/spf13/pflag.newInt64Value()
/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/int64.go:9 +0x49
github.com/spf13/pflag.(*FlagSet).Int64Var()
/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/int64.go:41 +0x3d
k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds.RegisterFlags()
/home/prow/go/src/k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds/admission.go:42 +0x25
k8s.io/kubernetes/pkg/kubeapiserver/options.registerAllAdmissionPluginFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/plugins.go:115 +0x64
k8s.io/kubernetes/pkg/kubeapiserver/options.(*AdmissionOptions).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/admission.go:73 +0x55
k8s.io/kubernetes/pkg/controlplane/apiserver/options.(*Options).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/options/options.go:154 +0x388
k8s.io/kubernetes/cmd/kube-apiserver/app/options.(*ServerRunOptions).Flags()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go:101 +0x73
k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:219 +0xb77
k8s.io/kubernetes/test/integration/controlplane/transformation.newTransformTest()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/transformation_test.go:131 +0x6d0
k8s.io/kubernetes/test/integration/controlplane/transformation.TestKMSv2FeatureFlag()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/kmsv2_transformation_test.go:1009 +0x117
testing.tRunner()
/usr/local/go/src/testing/testing.go:1792 +0x225
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1851 +0x44
A more general solution is to lock while registering the variables (= write)
and while reading the variables once in InspectFeatureGates.
TestKMSv2ProviderKeyIDStaleness runs two tests which both modify
kmsv2.NowFunc temporarily. Those two cannot run in parallel. They can be run in
parallel with other tests because nothing else changes kmsv2.NowFunc.
Found during local testing.
Explicitly shutting down background flushing addresses the following data race:
WARNING: DATA RACE
Write at 0x0000091173b8 by goroutine 166:
k8s.io/component-base/logs/api/v1.apply.FlushLogger.func2()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/contextual.go:95 +0x35
k8s.io/klog/v2.SetLoggerWithOptions()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/contextual.go:75 +0xb4
k8s.io/component-base/logs/api/v1.apply()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:277 +0x10fb
k8s.io/component-base/logs/api/v1.validateAndApply()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:132 +0x90
k8s.io/component-base/logs/api/v1.ValidateAndApply()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:94 +0xd4
k8s.io/component-base/logs/api/v1.ResetForTest()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/api/v1/options.go:314 +0xd5
k8s.io/kubernetes/test/integration/scheduler_perf.setupTestCase.func1()
/nvme/gopath/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:1062 +0x95
testing.(*common).Cleanup.func1()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1211 +0x16f
testing.(*common).runCleanup()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1445 +0x2b3
testing.(*common).runCleanup()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1445 +0x2b3
testing.tRunner.func2()
/nvme/gopath/go-1.24.0/src/testing/testing.go:1786 +0x4c
runtime.deferreturn()
/nvme/gopath/go-1.24.0/src/runtime/panic.go:605 +0x5d
k8s.io/apiserver/pkg/endpoints.(*APIInstaller).Install()
/nvme/gopath/src/k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/installer.go:208 +0x3ad
...
Previous read at 0x0000091173b8 by goroutine 169:
k8s.io/klog/v2.(*loggingT).flushAll()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1215 +0x166
k8s.io/klog/v2.(*loggingT).lockAndFlushAll()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1193 +0x58
k8s.io/klog/v2.(*loggingT).lockAndFlushAll-fm()
<autogenerated>:1 +0x33
k8s.io/klog/v2.(*flushDaemon).run.func1()
/nvme/gopath/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1143 +0x179
That the cleanup code is shown as being called by runtime/panic.go and
k8s.io/apiserver/pkg/endpoints/installer.go seems to be an oddity of `go test
-race`. Under a debugger it gets called normally, without a panic, and the test
finishes normally either way.
It's debatable whether StopFlushDaemon should be called by the caller of
ResetForTest or by ResetForTest itself. It's explicitly called out as "not
thread safe", so making the caller responsible seems appropriate.
