With the ClientsAllowCBOR client-go feature gate enabled, a 415 response to a CBOR-encoded REST
causes all subsequent requests from the client to fall back to a JSON request encoding. This
mechanism had only worked as intended when CBOR was explicitly configured in the
ClientContentConfig. When both ClientsAllowCBOR and ClientsPreferCBOR are enabled, an
unconfigured (empty) content type defaults to CBOR instead of JSON. Both ways of configuring a
client to use the CBOR request encoding are now subject to the same fallback mechanism.
Kubernetes-commit: a77f4c7ba2e761461daaf115a38903fc91916dd6
For alpha, there is one apiserver feature gate and two client-go feature gates controlling
CBOR. They were initially wired to separate test-only feature gate instances in order to prevent
them from being configurable at runtime via command-line flags or environment variables (for
client-go feature gates outside of Kubernetes components). All of the integration tests required by
the KEP as alpha criteria have been implemented. This adds the feature gates to the usual feature
gate instances and removes the temporary code to support separate test-only feature gate instances.
Kubernetes-commit: 072dfcb416fd4e1ddab0a89ac4faf519e268bc96
If a client is configured to encode request bodies to CBOR, but the server does not support CBOR,
the server will respond with HTTP 415 (Unsupported Media Type). By feeding this response back to the
RESTClient, subsequent requests can fall back to JSON, which is assumed to be acceptable.
Kubernetes-commit: 1745dfdd154b1a838765e70b81c861c644bfcffe
Logging in rest.Request.Body cannot be made context-aware without an API
change. Such a change is complicated if done in a backwards-compatible
fashion (must change lots of callers in Kubernetes) and prohibitive if not (all
callers of Body would have to pass a context).
Instead, logging of the request body gets moved into the functions which send
the request. This is a change of behavior, but it is limited to log levels >= 8
and thus should have no impact in production.
A request which gets sent multiple times will also log the body multiple
times. This might even be a good thing because it serves as reminder what is
being sent when it is being sent.
While at it, stack backtracing gets enhanced so that the caller of the REST API
is logged and tests for the new behavior get added.
Kubernetes-commit: 57f9b7c7a2412865e7817dbf7638881b00ac9721
The Error method of the error returned from Request.Watch was "unknown"
even the server returned clear message in the Status struct. It was
because Request.Watch used the Result's err member directly, which is an
unstructured error from the response which the Result object may use if
the caller did not return a structured error.
The patch fixes it by calling the Result's Error method instead, which
returns the structured error when it's present.
It also removes the wrong expectation about events.
Kubernetes-commit: 596c5696c64023808af164284263647d795b0ac2
* client-go: add DNS resolver latency metrics
* client-go: add locking to DNS latency metrics
* client-go: add locking for whole DNSStart and DNSDone
Signed-off-by: Vu Dinh <vudinh@outlook.com>
* Fix a mismatched ctx on the request
Signed-off-by: Vu Dinh <vudinh@outlook.com>
* Clean up request code and fix comments
Signed-off-by: Vu Dinh <vudinh@outlook.com>
---------
Signed-off-by: Vu Dinh <vudinh@outlook.com>
Co-authored-by: Vu Dinh <vudinh@outlook.com>
Kubernetes-commit: 1c7e87cff27aa009488a9d55342220e223d5c146
Requests can accumulate errors with no obvious indication, e.g. if
their primary purpose is to construct a URL: URL() itself doesn't
return an error if r.err is non-nil.
Instead of changing URL() to return an error, which has quite a large
impact, add an Error() function and indicate on URL() that it should
be checked.
Signed-off-by: Stephen Kitt <skitt@redhat.com>
Kubernetes-commit: f69c1c47463ff70ad61adf6f38c4d5b7373e9d0a
The functionality provided by the finalURLTemplate is still used by
certain external projects to track the request latency for requests
performed to kube-apiserver.
Using a template of the URL, instead of the URL itself, prevents the
explosion of label cardinality in exposed metrics since it aggregates
the URLs in a way that common URLs requests are reported as being the
same.
This reverts commit bebf5a608f68523fc430a44f6db26b16022dc862.
Signed-off-by: André Martins <aanm90@gmail.com>
Kubernetes-commit: f8f190cdd2fa76296f8b1b019ac77128b5d40b79
- Run hack/update-codegen.sh
- Run hack/update-generated-device-plugin.sh
- Run hack/update-generated-protobuf.sh
- Run hack/update-generated-runtime.sh
- Run hack/update-generated-swagger-docs.sh
- Run hack/update-openapi-spec.sh
- Run hack/update-gofmt.sh
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Kubernetes-commit: a9593d634c6a053848413e600dadbf974627515f
This commit refactors the retry logic to include resetting the
request body. The reset logic will be called iff it is not the
first attempt. This refactor is nescessary mainly because now
as per the retry logic, we always ensure that the request body
is reset *after* the response body is *fully* read and closed
in order to reuse the same TCP connection.
Previously, the reset of the request body and the call to read
and close the response body were not in the right order, which
leads to race conditions.
This commit also adds a test that verifies the order in which
the function calls are made to ensure that we seek only after
the response body is closed.
Co-authored-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: 68c8c458ee8f6629eef806c48c1a776dedad3ec4
Some of these changes are cosmetic (repeatedly calling klog.V instead of
reusing the result), others address real issues:
- Logging a message only above a certain verbosity threshold without
recording that verbosity level (if klog.V().Enabled() { klog.Info... }):
this matters when using a logging backend which records the verbosity
level.
- Passing a format string with parameters to a logging function that
doesn't do string formatting.
All of these locations where found by the enhanced logcheck tool from
https://github.com/kubernetes/klog/pull/297.
In some cases it reports false positives, but those can be suppressed with
source code comments.
Kubernetes-commit: edffc700a43e610f641907290a5152ca593bad79
Get metrics for the request and response size, so we can correlate latency
and size on a request, otherwise we could get confused because we don't know if the
network is slow or just the request size huge.
Kubernetes-commit: 64d9d0585f6dbc9266f31b6d0f795d6c0421495e
The restclient metrics were updated to track only the host field of the
url, the finalURLTemplate is not longer needed, its only goal was to
replace name and namespace in the path to avoid cardinality.
Kubernetes-commit: bebf5a608f68523fc430a44f6db26b16022dc862
As `%v` doesn't allow error unwrapping, checks like `errors.Is` are not
working properly.
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
Kubernetes-commit: 6c0463bd2b616d0f22f47905bb26d66fa3b04e37
This reverts commit 5a59a43957c6743995dac67fdda42bf8e0a9ca77, reversing
changes made to 81b9789eaa7bc067f417b5e74d5695dd6dd88a46.
Kubernetes-commit: 892d4fabb845e2461e3655aa414beb6ac322fc99
The message argument is mistakenly used as the format specifier, if it
contains the special '%' characters. This causes many '[%d|%s](MISSING)'
errors in the API server logs.
Signed-off-by: Ivan Sim <isim@redhat.com>
Kubernetes-commit: b1d0d401875b2076e73183f8468ecb95c3fe61aa
In some environments, where url base is "/", it can cause all paths to
be presented in metrics with "{prefix}" as `groupIndex` is with the wrong
index. To fix the behavior in such environments, it was added a
conditional branch to check if the URL base is "/" and, thus, print the
metrics with the correct path, for example "api/v1/nodes/{name}" instead
of "{prefix}".
Fixes: 99248b8fe1fe ("Rewrite finalURLTemplate used only for metrics because of dynamic client change")
Signed-off-by: André Martins <aanm90@gmail.com>
Kubernetes-commit: c039b02fa7281fc061455e23b6530ed8b4d19645
Chore: Correct words and format codes
Revert three changes
Revert 1 change
Revert again
Revert 2 changes
Kubernetes-commit: af7cf4abc6bfeb0d2cfaca76097cf7a0603c4495
Close outbound connections when using a cert callback and certificates rotate. This means that we won't get into a situation where we have open TLS connections using expires certs, which would get unauthorized errors at the apiserver
Attempt to retrieve a new certificate if open connections near expiry, to prevent the case where the cert expires but we haven't yet opened a new TLS connection and so GetClientCertificate hasn't been called.
Move certificate rotation logic to a separate function
Rely on generic transport approach to handle closing TLS client connections in exec plugin; no need to use a custom dialer as this is now the default behaviour of the transport when faced with a cert callback. As a result of handling this case, it is now safe to apply the transport approach even in cases where there is a custom Dialer (this will not affect kubelet connrotation behaviour, because that uses a custom transport, not just a dialer).
Check expiry of the full TLS certificate chain that will be presented, not only the leaf. Only do this check when the certificate actually rotates. Start the certificate as a zero value, not nil, so that we don't see a rotation when there is in fact no client certificate
Drain the timer when we first initialize it, to prevent immediate rotation. Additionally, calling Stop() on the timer isn't necessary.
Don't close connections on the first 'rotation'
Remove RotateCertFromDisk and RotateClientCertFromDisk flags.
Instead simply default to rotating certificates from disk whenever files are exclusively provided.
Add integration test for client certificate rotation
Simplify logic; rotate every 5 mins
Instead of trying to be clever and checking for rotation just before an
expiry, let's match the logic of the new apiserver cert rotation logic
as much as possible. We write a controller that checks for rotation
every 5 mins. We also check on every new connection.
Respond to review
Fix kubelet certificate rotation logic
The kubelet rotation logic seems to be broken because it expects its
cert files to end up as cert data whereas in fact they end up as a
callback. We should just call the tlsConfig GetCertificate callback
as this obtains a current cert even in cases where a static cert is
provided, and check that for validity.
Later on we can refactor all of the kubelet logic so that all it does is
write files to disk, and the cert rotation work does the rest.
Only read certificates once a second at most
Respond to review
1) Don't blat the cert file names
2) Make it more obvious where we have a neverstop
3) Naming
4) Verbosity
Avoid cache busting
Use filenames as cache keys when rotation is enabled, and add the
rotation later in the creation of the transport.
Caller should start the rotating dialer
Add continuous request rotation test
Rebase: use context in List/Watch
Swap goroutine around
Retry GETs on net.IsProbableEOF
Refactor certRotatingDialer
For simplicity, don't affect cert callbacks
To reduce change surface, lets not try to handle the case of a changing
GetCert callback in this PR. Reverting this commit should be sufficient
to handle that case in a later PR.
This PR will focus only on rotating certificate and key files.
Therefore, we don't need to modify the exec auth plugin.
Fix copyright year
Kubernetes-commit: 929b1559a0b855d996257ab3ad5364605edc253d
