Merge pull request #27784 from deads2k/catch-mutators

Automatic merge from submit-queue

add optional mutation checks for shared informer cache

We need to make sure that no one is mutating caches if they're using a shared informer.  It is important that whatever is tracking those changes gets the object *before* anyone else possibly could.

This adds the ability to track the original objects in the cache and their current values.  Go doesn't have an exit hook or a way to say "wait for non-daemon go-funcs to complete before exit", so this runs a gofunc on a loop that can panic the entire process.  It's gated behind an env var.

@derekwaynecarr did I get the right spots to make sure that e2e runs with this flag?
@smarterclayton @kubernetes/rh-cluster-infra

hack/make-rules/test.sh

@@ -23,6 +23,10 @@ source "${KUBE_ROOT}/hack/lib/init.sh"
 
 kube::golang::setup_env
 
+# start the cache mutation detector by default so that cache mutators will be found
+KUBE_CACHE_MUTATION_DETECTOR="${KUBE_CACHE_MUTATION_DETECTOR:-true}"
+export KUBE_CACHE_MUTATION_DETECTOR
+
 kube::test::find_dirs() {
   (
     cd ${KUBE_ROOT}