* Add stress test for pod cleanup when rejected due to VolumeAttachmentLimitExceeded
Signed-off-by: Eddie Torres <torredil@amazon.com>
* Move newTestPodsWithResources below TestVolumeAttachLimitExceededCleanup
Signed-off-by: Eddie Torres <torredil@amazon.com>
---------
Signed-off-by: Eddie Torres <torredil@amazon.com>
Listing all keys from etcd turned out to be too expensive, negativly
impacting events POST latency. Events resource is the only resource that
by default has watch cache disabled and which includes very
large number of small objects making it very costly to list keys.
Expected impact:
* No apiserver_resource_size_estimate_bytes metric for events.
* APF overestimating LIST request cost to events. Fallback assumes
object size of 1.5MB, meaning LIST events will always get maxSeats
Most of the time the managed fields were returned in the order as expected,
but not always. Semantically the order is irrelevant, so the comparison gets
relaxed to ignore it by normalizing first.
A mutex lock around decrementing runningPods was missing, leading to a data
race report in ci-kubernetes-integration-race-master:
WARNING: DATA RACE
Read at 0x00c001bd20a8 by goroutine 95696:
k8s.io/kubernetes/test/integration/scheduler_perf.createPodsSteadily.func7()
/home/prow/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:2133 +0x238
...
Previous write at 0x00c001bd20a8 by goroutine 101407:
k8s.io/kubernetes/test/integration/scheduler_perf.createPodsSteadily.func5()
/home/prow/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf.go:2064 +0x1a4
...
This doubles the termination timeout for the eviction test from 5min to
10min. Reason for that is that the eviction manager relies on pod stats
metrics, which may not be acceessible during a period of time because of
the kubelet API unreachable. This could be reasoned in hardware or
network pressure when multiple tests run in parallel.
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
This reverts commit 2e756e7412,
which addressed the data race only for test/integration/apiserver/oidc.
It also occurs for test/integration/controlplane/transformation:
WARNING: DATA RACE
Read at 0x000008b9de38 by goroutine 104616:
k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds.RegisterFlags()
/home/prow/go/src/k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds/admission.go:42 +0x30
k8s.io/kubernetes/pkg/kubeapiserver/options.registerAllAdmissionPluginFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/plugins.go:115 +0x64
k8s.io/kubernetes/pkg/kubeapiserver/options.(*AdmissionOptions).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/admission.go:73 +0x55
k8s.io/kubernetes/pkg/controlplane/apiserver/options.(*Options).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/options/options.go:154 +0x388
k8s.io/kubernetes/cmd/kube-apiserver/app/options.(*ServerRunOptions).Flags()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go:101 +0x73
k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:219 +0xb77
k8s.io/kubernetes/test/integration/controlplane/transformation.newTransformTest()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/transformation_test.go:131 +0x6d0
k8s.io/kubernetes/test/integration/controlplane/transformation.TestKMSv2Healthz()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/kmsv2_transformation_test.go:874 +0x1c8
testing.tRunner()
/usr/local/go/src/testing/testing.go:1792 +0x225
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1851 +0x44
Previous write at 0x000008b9de38 by goroutine 111926:
github.com/spf13/pflag.newInt64Value()
/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/int64.go:9 +0x49
github.com/spf13/pflag.(*FlagSet).Int64Var()
/home/prow/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/int64.go:41 +0x3d
k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds.RegisterFlags()
/home/prow/go/src/k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds/admission.go:42 +0x25
k8s.io/kubernetes/pkg/kubeapiserver/options.registerAllAdmissionPluginFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/plugins.go:115 +0x64
k8s.io/kubernetes/pkg/kubeapiserver/options.(*AdmissionOptions).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/admission.go:73 +0x55
k8s.io/kubernetes/pkg/controlplane/apiserver/options.(*Options).AddFlags()
/home/prow/go/src/k8s.io/kubernetes/pkg/controlplane/apiserver/options/options.go:154 +0x388
k8s.io/kubernetes/cmd/kube-apiserver/app/options.(*ServerRunOptions).Flags()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go:101 +0x73
k8s.io/kubernetes/cmd/kube-apiserver/app/testing.StartTestServer()
/home/prow/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/testserver.go:219 +0xb77
k8s.io/kubernetes/test/integration/controlplane/transformation.newTransformTest()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/transformation_test.go:131 +0x6d0
k8s.io/kubernetes/test/integration/controlplane/transformation.TestKMSv2FeatureFlag()
/home/prow/go/src/k8s.io/kubernetes/test/integration/controlplane/transformation/kmsv2_transformation_test.go:1009 +0x117
testing.tRunner()
/usr/local/go/src/testing/testing.go:1792 +0x225
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1851 +0x44
A more general solution is to lock while registering the variables (= write)
and while reading the variables once in InspectFeatureGates.
TestKMSv2ProviderKeyIDStaleness runs two tests which both modify
kmsv2.NowFunc temporarily. Those two cannot run in parallel. They can be run in
parallel with other tests because nothing else changes kmsv2.NowFunc.
Found during local testing.
As mentioned for SystemPriorityClasses:
NOTE: be careful not to modify any of elements of the returned array directly.
Create violates that because it temporarily changes the kind during encoding.
This led to a data race in test/integration/apiserver/oidc, found during local
testing.
Co-authored-by: Jordan Liggitt <liggitt@google.com>
This updates kubernetes/kubernetes#132798 to show simple values and NO types.
This also reverts the logic that omits values when there is a message expression,
so we can discuss that behavior further.
Signed-off-by: Chris Bandy <bandy.chris@gmail.com>
The build log already shows whether a binary is built with race detection
enabled:
$ KUBE_RACE=-race KUBE_CGO_OVERRIDES="kube-apiserver" make WHAT=cmd/kube-apiserver
+++ [0902 16:30:07] Building go targets for linux/amd64
k8s.io/kubernetes/cmd/kube-apiserver (non-static, race detection)
But it's also useful to see that race detection is on in the log output of a
command, just to be sure:
I0902 16:31:39.976438 797652 withrace.go:28] Data race detection enabled
If disabled, nothing is logged.
Since a few releases, Go supports `go build -race` and then produces
binaries which do data race detection when invoked. Some changes are needed to
enable using this in a kind cluster:
- `-race` must be passed when building dynamically linked binaries.
Only those support -race because CGO is required.
To avoid adding yet another env variables, the existing KUBE_RACE
gets used to convey the intent.
- KUBE_RACE must be passed into the dockerized build.
- Logging the base image of a release image makes it easier
to figure out whether the binary has a chance to run.
The base image is important because dynamically linked binaries need a base
image with libc. By default, control plane components are linked statically,
so users need to explicitly override the defaults:
KUBE_RACE=-race KUBE_CGO_OVERRIDES="kube-apiserver kube-controller-manager kube-scheduler" KUBE_GORUNNER_IMAGE=gcr.io/k8s-staging-test-infra/kubekins-e2e:v20250815-171060767f-master kind build node-image ...
KUBE_GORUNNER_IMAGE changes the base image for kube-apiserver,
kube-controller-manager and kube-scheduler. The kubekins image was picked for
this example because a Prow job definition already uses it. Reusing
it in a job avoids the need to maintain another image definition.
Running conformance tests against such a cluster with alpha+beta features
enabled revealed one new data race:
$ kubectl logs -n kube-system kube-controller-manager-kind-control-plane
...
WARNING: DATA RACE
Write at 0x00c00019a730 by goroutine 216:
k8s.io/client-go/tools/leaderelection.(*LeaderElector).setObservedRecord()
k8s.io/client-go/tools/leaderelection/leaderelection.go:529 +0x179
k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryCoordinatedRenew()
k8s.io/client-go/tools/leaderelection/leaderelection.go:367 +0x5ca
...
There is a custom error message returned from resource.Builder when
SingleResourceType is set and multiple resources types are specified,
which makes it impossible to check for the condition in code easily.
This patch adds resource.ErrMultipleResourceTypes that is returned
precisely in this case so that it can be checked.
This patch also removes an extra custom error message in kubectl get,
which is actually never reached, because the builder fails before.
