The event Object is created from the referenced Object name, however,
there is no guarantee that the name from the referenced Object will be a
valid Event Name.
For those Objects with name not valid for an Event, generate a new valid
name using an UUID.
Kubernetes-commit: ee36b817df06f84ce1a48ef4d65ed559c3775404
The previous attempt to fix this in
6aa779f4ed (diff-efa2cd1347df22ace5a516ea794152d00ef2a079db135c81787ed920ecb73658)
didn't address the root cause (or perhaps created it, not sure): the goroutine
must not be started if watch creation failed.
Instead, the error gets logged (as before) and an empty watch gets returned to
the caller (new). This is necessary because the function doesn't have an error
return value and changing that now would be disruptive. The empty watch is
valid and usable, so callers won't crash when they calls Stop.
This showed up recently in failed unit tests, probably because test
cancellation makes this error more likely:
"Unable start event watcher (will not retry!)" err="broadcaster already
stopped" logger="TestGarbageCollectorConstruction leaked goroutine"
The logger value and a preceding warning show that this occurs after test
completion.
Kubernetes-commit: 080432c46a7a49c3abf86d7fc5f2a5d7abc92239
Constructing a Broadcaster already starts a watch which runs in the
background. Shutdown must be called to avoid leaking the goroutine. Providing
a context was supposed to remove the need to call Shutdown, but that did not
actually work because the logic for "must check for cancellation" was
accidentally inverted.
While at it, structured log output also gets tested together with checking for
goroutine leaks.
Kubernetes-commit: ff779f1cb56cf896405e52f7923188b99b88bb00
If, for whatever reason, the context was context.Background(), the additional
goroutine was started and then got stuck forever because
context.Background().Done() is a nil channel. Found when indirectly
instantiating a broadcaster with such a context:
found unexpected goroutines:
[Goroutine 9106 in state chan receive (nil chan), with k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.NewBroadcaster.func1 on top of the stack:
goroutine 9106 [chan receive (nil chan)]:
k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.NewBroadcaster.func1()
/home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:206 +0x2c
created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.NewBroadcaster in goroutine 8957
/home/prow/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:205 +0x1a5
This can be fixed by checking for a nil channel.
Another problem also gets addressed: if Shutdown was called without canceling
the context, the goroutine also didn't stop. Now it waits for the cancelation
context and thus terminates in both cases.
Kubernetes-commit: eed6e29a5b8cfaa20fbc426541d9c74105d430ee
Using StartRecordingToSinkWithContext instead of StartRecordingToSink and
StartLogging instead of StartStructuredLogging has several advantages:
- Spawned goroutines no longer get stuck for extended periods of
time during shutdown when passing in a context that gets canceled.
- Log output can be directed towards a specific logger instead of the global
default, for example one which writes to a testing.T instance.
- The new methods return an error when something went wrong instead of
merely recording the error.
That last point is the reason for deprecating the old methods instead of merely
adding new alternatives.
Setting a context when constructing an EventBroadcaster makes calling Shutdown
optional. It can also be used to specify the logger.
Both EventRecorder interfaces in tools/events and tools/record now have a
WithLogger helper. Using that method is optional, but recommended to support
contextual logging properly. Without it, errors that occur while emitting an
event are not associated with the caller.
Kubernetes-commit: 27a68aee3a48340f7c14235f7fc24aa69aaeb8f6
Create events are forbidden in terminating namespaces, use info
instead of error to log the failed event.
Signed-off-by: Sunil Shivanand <sunil.shivanand@statnett.no>
Kubernetes-commit: 7a6d58001b7d824f92601fd246b3aad9fbb9c583
When Shutdown was called, delivery of each pending event would still be retried
12 times with a delay of ~10s between each retry. In apiserver integration
tests that caused the goroutine to linger long after the corresponding
apiserver of the test was shut down.
Kubernetes-commit: 15b01af9c18a0840d71e2bb7dff4d8c29b158aad
Currenlty an event recorder can send an event to a
broadcaster that is already stopped, resulting
in a panic. This ensures the broadcaster holds
a lock while it is shutting down and then forces
any senders to drop queued events following
broadcaster shutdown.
It also updates the Action, ActionOrDrop, Watch,
and WatchWithPrefix functions to return an error
in the case where data is sent on the closed bradcaster
channel rather than panicing.
Lastly it updates unit tests to ensure the fix works correctly
fixes: https://github.com/kubernetes/kubernetes/issues/108518
Signed-off-by: Andrew Stoycos <astoycos@redhat.com>
Kubernetes-commit: 6aa779f4ed3d3acdad2f2bf17fb27e11e23aabe4
- Introduce PassiveRateLimiter which implements all methods of previous RateLimiter except Accept() and Wait()
- Change RateLimiter interface to extend PassiveRateLimiter by additionally implementing Accept() and Wait()
- Make client-go/tools/record use PassiveRateLimiter
Refactor EventSourceObjectSpamFilter, EventAggregator, EventCorrelator
- EventSourceObjectSpamFilter, EventAggregator, EventCorrelator use clock.PassiveClock now.
- This won't be a breaking change because even if a clock.Clock is passed, it still implements the clock.PassiveClock interface.
- Extend clock.PassiveClock through Clock.
- Replace pacakge local implementation of realClock with clock.RealClock
- In flowcontrol/throttle.go split tokenBucketRateLimiters to use Clock and clock.PassiveClock.
- Migrate client-go/tools/record tests from using IntervalClock to using SimpleIntervalClock (honest implementation of clock.PassiveClock)
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
Kubernetes-commit: ac5c55f0bd853fcf883d9b8e1f5ef728a2fb5309
This changes the event recorder to use the equivalent of a select
statement instead of a goroutine to record events.
Previously, we used a goroutine to make event recording non-blocking.
Unfortunately, this writes to a channel, and during shutdown we then
race to write to a closed channel, panicing (caught by the error
handler, but still) and making the race detector unhappy.
Instead, we now use the select statement to make event emitting
non-blocking, and if we'd block, we just drop the event. We already
drop events if a particular sink is overloaded, so this just moves the
incoming event queue to match that behavior (and makes the incoming
event queue much longer).
This means that, if the user uses `Eventf` and friends correctly (i.e.
ensure they've returned by the time we call `Shutdown`), it's
now safe to call Shutdown. This matches the conventional go guidance on
channels: the writer should call close.
Kubernetes-commit: e90e67bd002e70a525d3ee9045b213a5d826074d
The `recorder.PastEventf` method wasn't actually working as advertised.
It was supposed to accept a timestamp, which would be used when
generating the event. However, as the
[source code](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/record/event.go#L316)
shows, this `timestamp` was never actually used.
In other words, `PastEventf` is identical to `Eventf`.
We have two options: one would be to fix `PastEventf` so that it works
as advertised. The other would be to delete `PastEventf` and only
support `Eventf`.
Ultimately, I could only find one use of `PastEventf` in the code base,
so I propose we just delete `PastEventf` and convert all uses to
`Eventf`.
Kubernetes-commit: 92940fa80d67593c7a2333267da4424c8b45ac88
Added CorrelatorOptions that contains options to change the
defaults in EventSourceObjectSpamFilter and EventAggregator
in EventCorrelator. Added a eventCorrelator property to the
eventBroadcasterImpl to help with this.
Kubernetes-commit: 9d8e6fb1b9cf2d3fac8139a97334287e33ff911f
- Move from the old github.com/golang/glog to k8s.io/klog
- klog as explicit InitFlags() so we add them as necessary
- we update the other repositories that we vendor that made a similar
change from glog to klog
* github.com/kubernetes/repo-infra
* k8s.io/gengo/
* k8s.io/kube-openapi/
* github.com/google/cadvisor
- Entirely remove all references to glog
- Fix some tests by explicit InitFlags in their init() methods
Change-Id: I92db545ff36fcec83afe98f550c9e630098b3135
Kubernetes-commit: 954996e231074dc7429f7be1256a579bedd8344c
