The default queue implementation is mostly FIFO and it is not
exchangeable unless we implement the whole `workqueue.Interface` which
is less desirable as we have to duplicate a lot of code. There was one
attempt done in [kubernetes/kubernetes#109349][1] which tried to
implement a priority queue. That is really useful and [knative/pkg][2]
implemented something called two-lane-queue. While two lane queue is
great, but isn't perfect since a full slow queue can still slow down
items in fast queue.
This change proposes a swappable queue implementation while not adding
extra maintenance effort in kubernetes community. We are happy to
maintain our own queue implementation (similar to two-lane-queue) in
downstream.
[1]: https://github.com/kubernetes/kubernetes/pull/109349
[2]: https://github.com/knative/pkg/blob/main/controller/two_lane_queue.go
Kubernetes-commit: 87b4279e07349b3c68f16f69a349a02bddd12f25
See https://github.com/grpc/grpc-go/issues/4758 for a real world example
of this leaking 2gb+ of data.
Basically, when we do `q.queue[1:]` we are just repositioning the slice.
The underlying array is still active, which contains the object formerly
known as `q.queue[0]`. Because its referencing this object, it will not
be GCed. The only thing that will trigger it to free is eventually when
we add enough to the queue that we allocate a whole new array.
Instead, we should explicitly clear out the old space when we remove it
from the queue. This ensures the object can be GCed, assuming the users'
application doesn't reference it anymore.
Kubernetes-commit: 2a34801168dc1c70ba25b1d4200b534bf515cbc2
