* client-go: add DNS resolver latency metrics
* client-go: add locking to DNS latency metrics
* client-go: add locking for whole DNSStart and DNSDone
Signed-off-by: Vu Dinh <vudinh@outlook.com>
* Fix a mismatched ctx on the request
Signed-off-by: Vu Dinh <vudinh@outlook.com>
* Clean up request code and fix comments
Signed-off-by: Vu Dinh <vudinh@outlook.com>
---------
Signed-off-by: Vu Dinh <vudinh@outlook.com>
Co-authored-by: Vu Dinh <vudinh@outlook.com>
Kubernetes-commit: 1c7e87cff27aa009488a9d55342220e223d5c146
Requests can accumulate errors with no obvious indication, e.g. if
their primary purpose is to construct a URL: URL() itself doesn't
return an error if r.err is non-nil.
Instead of changing URL() to return an error, which has quite a large
impact, add an Error() function and indicate on URL() that it should
be checked.
Signed-off-by: Stephen Kitt <skitt@redhat.com>
Kubernetes-commit: f69c1c47463ff70ad61adf6f38c4d5b7373e9d0a
The functionality provided by the finalURLTemplate is still used by
certain external projects to track the request latency for requests
performed to kube-apiserver.
Using a template of the URL, instead of the URL itself, prevents the
explosion of label cardinality in exposed metrics since it aggregates
the URLs in a way that common URLs requests are reported as being the
same.
This reverts commit bebf5a608f68523fc430a44f6db26b16022dc862.
Signed-off-by: André Martins <aanm90@gmail.com>
Kubernetes-commit: f8f190cdd2fa76296f8b1b019ac77128b5d40b79
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: a9593d634c6a053848413e600dadbf974627515f
This commit refactors the retry logic to include resetting the
request body. The reset logic will be called iff it is not the
first attempt. This refactor is nescessary mainly because now
as per the retry logic, we always ensure that the request body
is reset *after* the response body is *fully* read and closed
in order to reuse the same TCP connection.
Previously, the reset of the request body and the call to read
and close the response body were not in the right order, which
leads to race conditions.
This commit also adds a test that verifies the order in which
the function calls are made to ensure that we seek only after
the response body is closed.
Co-authored-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: 68c8c458ee8f6629eef806c48c1a776dedad3ec4
Some of these changes are cosmetic (repeatedly calling klog.V instead of
reusing the result), others address real issues:
- Logging a message only above a certain verbosity threshold without
recording that verbosity level (if klog.V().Enabled() { klog.Info... }):
this matters when using a logging backend which records the verbosity
level.
- Passing a format string with parameters to a logging function that
doesn't do string formatting.
All of these locations where found by the enhanced logcheck tool from
https://github.com/kubernetes/klog/pull/297.
In some cases it reports false positives, but those can be suppressed with
source code comments.
Kubernetes-commit: edffc700a43e610f641907290a5152ca593bad79
Get metrics for the request and response size, so we can correlate latency
and size on a request, otherwise we could get confused because we don't know if the
network is slow or just the request size huge.
Kubernetes-commit: 64d9d0585f6dbc9266f31b6d0f795d6c0421495e
The restclient metrics were updated to track only the host field of the
url, the finalURLTemplate is not longer needed, its only goal was to
replace name and namespace in the path to avoid cardinality.
Kubernetes-commit: bebf5a608f68523fc430a44f6db26b16022dc862
As `%v` doesn't allow error unwrapping, checks like `errors.Is` are not
working properly.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Kubernetes-commit: 6c0463bd2b616d0f22f47905bb26d66fa3b04e37
This reverts commit 5a59a43957c6743995dac67fdda42bf8e0a9ca77, reversing
changes made to 81b9789eaa7bc067f417b5e74d5695dd6dd88a46.
Kubernetes-commit: 892d4fabb845e2461e3655aa414beb6ac322fc99
The message argument is mistakenly used as the format specifier, if it
contains the special '%' characters. This causes many '[%d|%s](MISSING)'
errors in the API server logs.
Signed-off-by: Ivan Sim <isim@redhat.com>
Kubernetes-commit: b1d0d401875b2076e73183f8468ecb95c3fe61aa
In some environments, where url base is "/", it can cause all paths to
be presented in metrics with "{prefix}" as `groupIndex` is with the wrong
index. To fix the behavior in such environments, it was added a
conditional branch to check if the URL base is "/" and, thus, print the
metrics with the correct path, for example "api/v1/nodes/{name}" instead
of "{prefix}".
Fixes: 99248b8fe1fe ("Rewrite finalURLTemplate used only for metrics because of dynamic client change")
Signed-off-by: André Martins <aanm90@gmail.com>
Kubernetes-commit: c039b02fa7281fc061455e23b6530ed8b4d19645
Chore: Correct words and format codes
Revert three changes
Revert 1 change
Revert again
Revert 2 changes
Kubernetes-commit: af7cf4abc6bfeb0d2cfaca76097cf7a0603c4495
Close outbound connections when using a cert callback and certificates rotate. This means that we won't get into a situation where we have open TLS connections using expires certs, which would get unauthorized errors at the apiserver
Attempt to retrieve a new certificate if open connections near expiry, to prevent the case where the cert expires but we haven't yet opened a new TLS connection and so GetClientCertificate hasn't been called.
Move certificate rotation logic to a separate function
Rely on generic transport approach to handle closing TLS client connections in exec plugin; no need to use a custom dialer as this is now the default behaviour of the transport when faced with a cert callback. As a result of handling this case, it is now safe to apply the transport approach even in cases where there is a custom Dialer (this will not affect kubelet connrotation behaviour, because that uses a custom transport, not just a dialer).
Check expiry of the full TLS certificate chain that will be presented, not only the leaf. Only do this check when the certificate actually rotates. Start the certificate as a zero value, not nil, so that we don't see a rotation when there is in fact no client certificate
Drain the timer when we first initialize it, to prevent immediate rotation. Additionally, calling Stop() on the timer isn't necessary.
Don't close connections on the first 'rotation'
Remove RotateCertFromDisk and RotateClientCertFromDisk flags.
Instead simply default to rotating certificates from disk whenever files are exclusively provided.
Add integration test for client certificate rotation
Simplify logic; rotate every 5 mins
Instead of trying to be clever and checking for rotation just before an
expiry, let's match the logic of the new apiserver cert rotation logic
as much as possible. We write a controller that checks for rotation
every 5 mins. We also check on every new connection.
Respond to review
Fix kubelet certificate rotation logic
The kubelet rotation logic seems to be broken because it expects its
cert files to end up as cert data whereas in fact they end up as a
callback. We should just call the tlsConfig GetCertificate callback
as this obtains a current cert even in cases where a static cert is
provided, and check that for validity.
Later on we can refactor all of the kubelet logic so that all it does is
write files to disk, and the cert rotation work does the rest.
Only read certificates once a second at most
Respond to review
1) Don't blat the cert file names
2) Make it more obvious where we have a neverstop
3) Naming
4) Verbosity
Avoid cache busting
Use filenames as cache keys when rotation is enabled, and add the
rotation later in the creation of the transport.
Caller should start the rotating dialer
Add continuous request rotation test
Rebase: use context in List/Watch
Swap goroutine around
Retry GETs on net.IsProbableEOF
Refactor certRotatingDialer
For simplicity, don't affect cert callbacks
To reduce change surface, lets not try to handle the case of a changing
GetCert callback in this PR. Reverting this commit should be sufficient
to handle that case in a later PR.
This PR will focus only on rotating certificate and key files.
Therefore, we don't need to modify the exec auth plugin.
Fix copyright year
Kubernetes-commit: 929b1559a0b855d996257ab3ad5364605edc253d
* Move all usage of r.ctx to the beginning of Do, DoRaw, Stream, Watch
* Move tryThrottle from Do and DoRaw into request()
* Make request() and tryThrottle take a context
* In request(), remove the timeout context setting out of the loop
These changes should be entirely behavior preserving.
Kubernetes-commit: ed48ed0122c7289f458a6bc3ac616319d5c17e91
* Move all usage of r.ctx to the beginning of Do, DoRaw, Stream, Watch
* Move tryThrottle from Do and DoRaw into request()
* Make request() and tryThrottle take a context
* In request(), remove the timeout context setting out of the loop
These changes should be entirely behavior preserving.
Kubernetes-commit: d95ed2c8470158256466fb24728e63ac3afe0899
This commit performs two refactors and fixes a bug.
Refactor 1 changes the signature of Request to take a RESTClient, which
removes the extra copy of everything on RESTClient from Request. A pair
of optional constructors are added for testing. The major functional
change is that Request no longer has the shim HTTPClient interface and
so some test cases change slightly because we are now going through
http.Client code paths instead of direct to our test stubs.
Refactor 2 changes the signature of RESTClient to take a
ClientContentConfig instead of ContentConfig - the primary difference
being that ClientContentConfig uses ClientNegotiator instead of
NegotiatedSerializer and the old Serializers type. We also collapse
some redundancies (like the rate limiter can be created outside the
constructor).
The bug fix is to negotiate the streaming content type on a Watch()
like we do for requests. We stop caching the decoder and simply
resolve it on the request. We also clean up the dynamic client
and remove the extra WatchSpecificVersions() method by providing
a properly wrapped dynamic client.
Kubernetes-commit: 3b780c64b89606f4e6b21f48fb9c305d5998b9e5
Migrated code that checks for common programmer errors to a separated
function and added test coverage for it. Wrong comment stating that a
typed error is returned was also removed.
Kubernetes-commit: ad5fafd6ade2838098890a4e7727c8e347686867
Clients are required to handle watch events of type ERROR, so instead
of eating the decoding error we should pass it on to the client. Use
NewGenericServerError with isUnexpectedResponse to indicate that we
didn't get the bytes from the server we were expecting. For watch, the
415 error code is roughly correct and we will return an error to the
client that makes debugging a failure in either server watch or client
machinery much easier.
We do not alter the behavior when it appears the response is an EOF
or other disconnection.
Kubernetes-commit: 89620d5667adec6c132b2713b79efb1dd2391723
