Modify() was replacing components one at a time: stop X, start X, stop Y,
start Y, ... in version-skew order (apiserver last on downgrade). This
caused a crash during downgrade: KCM-1.35 started against the still-
running apiserver-1.36, passed its /healthz, and then immediately lose
its connection when apiserver-1.36 was killed by the localupcluster.
KCM-1.35 would reconnect to the not-yet-ready apiserver-1.35, hit a
403 RBAC error during controller initialization, and exit — because that
initialization phase does not retry on RBAC errors.
Fix by splitting Modify() into two phases:
Phase 1 — stop all components to be replaced, in reverse startup order
(kube-proxy down to apiserver), so dependent components release their
connections before the apiserver is stopped.
Phase 2 — start all replacement components in standard startup order
(apiserver first), so each component connects to a fully-ready apiserver.
Without an explicit interval, Gomega's default polling is very frequent,
generating a large volume of /readyz and /healthz requests in the component
logs. Set an explicit 1-second interval to reduce noise while still
detecting readiness promptly.
Despite being called checkReadiness, the function was only performing
a liveness check: /healthz was polled over HTTPS without verifying the
certificate or authenticating, and any HTTP response was accepted as a
signal that the component was up. The only exception was kubelet,
where a node readiness check was added on top.
Switched to /readyz for kube-apiserver and kube-scheduler,
kept /healthz for the rest and require HTTP 200 in all cases.
This ensures that the kube-apiserver is fully initialized before
dependent components are started.
The latest pause version is 3.10.2 but due to the introduction
of the PATCH level version to the pause image (previously was
only MAJOR.MINOR), various files have remained on an older
version. Either 3.10 or 3.10.1. Our validation with
build/dependencies.yaml ./hack/verify-external-dependencies.sh
did not account for that.
Not canceling the parent context made sense, but the new context should
be cancelable like any other TContext. Found when passing tCtx.WithoutCancel()
to StartTestServer and the tear-down function got stuck because it couldn't
cancel the context.
add integration test
Use proper test header, change to etcdMain to recognize test flags
fix goroutine leak in integration test
redo integration test with kubeapiserverapptesting
fix comment capitalization, use existing client libraries
fix comment capitalization, use existing client libraries
consolidate http connect handler logic from odic and tls_server-name into helper
add expected SNI, remove unused test
move oidc helpers.go to right dir, remove copyright year
split helpers.go into descriptive file names
use atomic ptr for SNI, refactor generateTestCerts, remove errors from runTLSEgressProxy, explain jwksHandler in comment
use testify, add back context messages
Clean up tests
Signed-off-by: Monis Khan <mok@microsoft.com>
This change allows slow impersonation requests to be tracked via the
apiserver.latency.k8s.io/impersonation audit event annotation.
Updated tests to assert that the audit event log:
- Contains the new latency annotation
- Contains the impersonationConstraint field
- Failed impersonation attempts are observable by the response status
Signed-off-by: Monis Khan <mok@microsoft.com>
In some (all?) CI jobs the initial kubelet instance keeps running, despite
command context cancellation. Not reproducible locally, so additional output
was necessary to track down the root cause in CI runs: signal propagation via
sudo didn't work for kube-proxy and kubelet, but only for those two and only in
the CI. The fix is to change the CI jobs so that they disable the usage of
sudo.
While at it, simplify by replacing atomic.Pointer with atomic.Boole.
The type alias made `go doc ./test/utils/ktesting.TContext` useless and was a
weird workaround for preserving the original interface type name. Passing a
TContext instance by value (almost) preserves the original API and is
acceptable because the struct is still small. The only consumers which need to
be updated are those which relied on passing nil as tCtx.
If we ever find that TContext is or becomes too large, then we can make it
a wrapper around some pointer.
I've not been able to trigger the flake, but it could happen:
- time.Sleep unblocks some background goroutines inside the synctest bubble.
- Those goroutines do not actually run yet.
- The main test checks for the result of those goroutines.
Adding a `synctest.Wait` ensures that all background processing is complete
because it waits for all goroutines to be durably blocked.
If the goroutine happens to log after the test has already terminated,
testing.T.Log panics. We must ensure that the goroutine has stopped before
allowing the test to terminate.
Replace all imports of k8s.io/apimachinery/pkg/util/dump with
k8s.io/utils/dump across the repo. The apimachinery dump package
now contains deprecated wrapper functions that delegate to
k8s.io/utils/dump for backwards compatibility.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
The tests validate the sidecar's functionality,
integration with the CSI driver and correctness of
metadata retrieval for snapshot backups.
This will help CSI vendors test their implementation
of the snapshot-metadata feature.
Issue: kubernetes-csi/external-snapshot-metadata#120
Signed-off-by: Praveen M <m.praveen@ibm.com>
When aborting an integration test with CTRL-C while it runs,
the current test fails and etcd exits. But additional tests were still being
started and the failed slowly because they couldn't connect to etcd.
It's better to fail additional tests in ktesting.Init when the test run has
already been interrupted.
While at it, also make it a bit more obvious that testing was interrupted by
logging it and update one comment about this and clean up the naming of
contexts in the code.
Example:
$ go test -v ./test/integration/quota
...
I1106 11:42:48.857162 147325 etcd.go:416] "Not using watch cache" resource="events.events.k8s.io"
I1106 11:42:48.857204 147325 handler.go:286] Adding GroupVersion events.k8s.io v1 to ResourceManager
W1106 11:42:48.857209 147325 genericapiserver.go:765] Skipping API events.k8s.io/v1beta1 because it has no resources.
^C
INFO: canceling test context: received interrupt signal
{"level":"warn","ts":"2024-11-06T11:42:48.984676+0100","caller":"embed/serve.go:160","msg":"stopping insecure grpc server due to error","error":"accept tcp 127.0.0.1:44177: use of closed network connection"}
...
I1106 11:42:50.042430 147325 handler.go:142] kube-apiserver: GET "/apis/rbac.authorization.k8s.io/v1/clusterroles" satisfied by gorestful with webservice /apis/rbac.authorization.k8s.io/v1
test_server.go:241: timed out waiting for the condition
--- FAIL: TestQuota (11.45s)
=== RUN TestQuotaLimitedResourceDenial
quota_test.go:292: testing has been interrupted: received interrupt signal
--- FAIL: TestQuotaLimitedResourceDenial (0.00s)
=== RUN TestQuotaLimitService
quota_test.go:418: testing has been interrupted: received interrupt signal
--- FAIL: TestQuotaLimitService (0.00s)
FAIL
When cleaning up the progress channel properly (stop signal delivery, closing
the channel), the loop dumping progress reports no longer needs to check for
the separate shutdown context. Instead, it can distinguish between "signal
received" and "channel closed".
The signal context was getting cleanup by canceling it, but a channel is better
because it avoids the slightly misleading "received interrupt signal"
cancellation when the test was only shutting down.
The "received interrupt signal" is useful also when running with "go test"
without -v because it shows that the shutdown has started.
But more important is that a progress report gets shown because that feature is
useful in particular when "go test" produces no output while it runs.
The recent change to support importing ktesting into an E2E suite
without progress reporting was flawed:
- If a Go unit test had a deadline (the default when invoked
by `go test`!), the early return skipped initializing progress
reporting.
- When it didn't, for example when invoking a test binary directly
under stress, a test created goroutines which were kept running,
which broke leak checking in e.g. an integration tests TestMain.
The revised approach uses reference counting: as long as some unit test is
running, the progress reporting with the required goroutines are active.
When the last one ends, they get cleaned up, which keeps the goleak
checker happy.
The TestCause tests were already unreliable in the CI. The others failed under
stress.
As synctest we have to be more careful how to construct and clean up the parent
context for TestCause (must happen inside bubble), but once that's handled we
can reliably measure the (fake) time and compare exactly against expected
results.
Replace the TensorFlow-based wide-deep workload with the PyTorch
implementation. This change:
- Adds pytorchWideDeepWorkload using the new pytorch-wide-deep image (1.0.0)
- Removes tfWideDeepWorkload and tf-wide-deep image references
- Enables arm64 support (PyTorch image is multi-arch)
- Uses the same log parsing (time -p output format)
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Updates the codebase to use the new glibc-dns-testing image which replaces
the deprecated jessie-dnsutils image.
This PR depends on the glibc-dns-testing image being available in the
registry (registry.k8s.io/e2e-test-images/glibc-dns-testing:2.0.0).
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Update outdated image versions across test manifests and add tracking
to build/dependencies.yaml for version drift detection via zeitgeist:
- agnhost: 2.32/2.53/2.54/2.57 → 2.61 (latest)
- etcd: 3.2.24 → 3.6.7-0
- kitten/nautilus BASEIMAGE: agnhost 2.57 → 2.61
and added etcd statefulset reference to existing etcd entry.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
