Patrick Ohly f199fa2ea1 DRA resourceslice controller: fix update + delete sequence ...

When updating a ResourceSlice, a more recent copy than in the underlying store
gets cached by the MutationCache. Then when that ResourceSlice gets deleted on
the apiserver and then the store, the MutationCache kept the stale copy and
returned it when the controller synced again, causing the controller to
erroneously not recreate the deleted ResourceSlice.

Depending on timing, this could have happened while updating the DRA driver:
- Driver restarts, updating the existing ResourceSlices.
- kubelet deletes the slices before realizing that the driver
  is running again.

The fix is to address a long-standing TODO in the MutationCache: it needs to
react to changes in the store. This allows it to drop cached objects sooner and
let's it remove cached objects that are known to be deleted. Users of a
MutationCache must inform the cache in their own informer event
handlers *before* triggering sync operations.

The integration test demonstrates the problem when run against master:

    core.go:560: I0629 09:43:22.656382] Deleting slice...
    resourceslicecontroller.go:610: I0629 09:43:22.659446] ResourceSlice delete slice="00000-ga-publishresourceslices-5s47j-slice"
    resourceslicecontroller.go:612: I0629 09:43:22.659496] Scheduled sync poolName="global" at="2026-06-29 09:43:25.659481942 +0200 CEST m=+5.886096424"
    resourceslicecontroller.go:755: I0629 09:43:24.639116] Matched existing slice by index poolName="global" slice="00000-ga-publishresourceslices-5s47j-slice" matchIndex=0
    resourceslicecontroller.go:788: I0629 09:43:24.639317] Completed comparison poolName="global" numObsolete=0 numMatchedSlices=1 numChangedMatchedSlices=0 numNewSlices=0
    resourceslicecontroller.go:804: I0629 09:43:24.639391] Kept generation because at most one update API call is necessary poolName="global" generation=1
    core.go:566: FATAL ERROR: I0629 09:43:30.661588]
        	Timed out after 8.002s.
        	Expected
        	    <resourceslice.Stats>: {NumCreates: 0, NumUpdates: 1, NumDeletes: 0}
        	to equal
        	    <resourceslice.Stats>: {NumCreates: 1, NumUpdates: 1, NumDeletes: 0}

The "matched existing slice" doesn't actually exist anymore, so the controller
does nothing and the test fails.

Kubernetes-commit: 495f81169d8b2fe9e6e5dafd6b31fca1ae0483fa

2026-06-29 08:55:40 +02:00

8.2 KiB

Raw Blame History

View Raw