The previous attempt to fix this in
6aa779f4ed (diff-efa2cd1347df22ace5a516ea794152d00ef2a079db135c81787ed920ecb73658)
didn't address the root cause (or perhaps created it, not sure): the goroutine
must not be started if watch creation failed.
Instead, the error gets logged (as before) and an empty watch gets returned to
the caller (new). This is necessary because the function doesn't have an error
return value and changing that now would be disruptive. The empty watch is
valid and usable, so callers won't crash when they calls Stop.
This showed up recently in failed unit tests, probably because test
cancellation makes this error more likely:
"Unable start event watcher (will not retry!)" err="broadcaster already
stopped" logger="TestGarbageCollectorConstruction leaked goroutine"
The logger value and a preceding warning show that this occurs after test
completion.
Kubernetes-commit: 080432c46a7a49c3abf86d7fc5f2a5d7abc92239
Constructing a Broadcaster already starts a watch which runs in the
background. Shutdown must be called to avoid leaking the goroutine. Providing
a context was supposed to remove the need to call Shutdown, but that did not
actually work because the logic for "must check for cancellation" was
accidentally inverted.
While at it, structured log output also gets tested together with checking for
goroutine leaks.
Kubernetes-commit: ff779f1cb56cf896405e52f7923188b99b88bb00
the signature of the method was tightly connected to the reflector,
making it difficult to use for anything other than a reflector.
this simple refactor makes the method more generic.
Kubernetes-commit: 83c7542abc8c542c01ecb67376f134b2071c5304
Hang when canceling leader election information.
Occasionally, two leaders may run simultaneously.
Kubernetes-commit: b6b46a0e00682517d2ca7b7e9c2706b8e407e52e
In particular, document that ListAllByNamespace delegates to ListAll
if no namespace is specified.
Signed-off-by: Stephen Kitt <skitt@redhat.com>
Kubernetes-commit: 54e899317ef46e3b70827cacee244717022db0ad
The path module has a few different functions:
Clean, Split, Join, Ext, Dir, Base, IsAbs. These functions do not
take into account the OS-specific path separator, meaning that they
won't behave as intended on Windows.
For example, Dir is supposed to return all but the last element of the
path. For the path "C:\some\dir\somewhere", it is supposed to return
"C:\some\dir\", however, it returns ".".
Instead of these functions, the ones in filepath should be used instead.
Kubernetes-commit: 856bb5c8f266f5276f1a576f47be622d7cb384e7
If, for whatever reason, the context was context.Background(), the additional
goroutine was started and then got stuck forever because
context.Background().Done() is a nil channel. Found when indirectly
instantiating a broadcaster with such a context:
found unexpected goroutines:
[Goroutine 9106 in state chan receive (nil chan), with k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.NewBroadcaster.func1 on top of the stack:
goroutine 9106 [chan receive (nil chan)]:
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.NewBroadcaster.func1()
/home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:206 +0x2c
created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.NewBroadcaster in goroutine 8957
/home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:205 +0x1a5
This can be fixed by checking for a nil channel.
Another problem also gets addressed: if Shutdown was called without canceling
the context, the goroutine also didn't stop. Now it waits for the cancelation
context and thus terminates in both cases.
Kubernetes-commit: eed6e29a5b8cfaa20fbc426541d9c74105d430ee
until #115478(use streaming against the etcd storage)
is resolved the cacher need a way to disable the streaming.
Kubernetes-commit: 41e706600aea7468f486150d951d3b8948ce89d5
27a68aee3a4834 introduced context support. In order to use that
also with NewEventBroadcasterAdapter, a variant of the call is needed to allow
the caller to specify the context.
The `logcheck:context` comment ensures that code which is meant to be
contextual uses the new call.
Kubernetes-commit: f8e25eff926c640c86daa46222bfaf8d625e75d7
checkWatchListConsistencyIfRequested performs a data consistency check only when
the KUBE_WATCHLIST_INCONSISTENCY_DETECTOR environment variable was set during a binary startup.
The consistency check is meant to be enforced only in the CI, not in production.
The check ensures that data retrieved by the watch-list api call
is exactly the same as data received by the standard list api call.
Note that this function will panic when data inconsistency is detected.
This is intentional because we want to catch it in the CI.
Kubernetes-commit: b31e7793d0d873a71c90caf8455556aa905cf88d
Because the EventBroadcaster code now has a a context, changing the EventSink
interface so that the methods accepts a context instead of using context.TODO
becomes possible.
Kubernetes-commit: 5dc540ff57eed5d0e6bc8476f95fe03b7a983cf5
Using StartRecordingToSinkWithContext instead of StartRecordingToSink and
StartLogging instead of StartStructuredLogging has several advantages:
- Spawned goroutines no longer get stuck for extended periods of
time during shutdown when passing in a context that gets canceled.
- Log output can be directed towards a specific logger instead of the global
default, for example one which writes to a testing.T instance.
- The new methods return an error when something went wrong instead of
merely recording the error.
That last point is the reason for deprecating the old methods instead of merely
adding new alternatives.
Setting a context when constructing an EventBroadcaster makes calling Shutdown
optional. It can also be used to specify the logger.
Both EventRecorder interfaces in tools/events and tools/record now have a
WithLogger helper. Using that method is optional, but recommended to support
contextual logging properly. Without it, errors that occur while emitting an
event are not associated with the caller.
Kubernetes-commit: 27a68aee3a48340f7c14235f7fc24aa69aaeb8f6
originally we honored only apierrors.IsInvalid
but decided to fallback on every error
because it is better to make progress than deadlocking
Kubernetes-commit: 4b3915017950a114124a88c5d308bd8bfb9ec48e
