If a ResourceSlice got published by the ResourceSlice controller in a DRA
driver and then that ResourceSlice got deleted quickly (within one minute, the
mutation cache TTL) by someone (for example, the kubelet because of a restart),
then the controller did not react properly to the deletion unless some other
event triggered the syncing of the pool.
Found while adding upgrade/downgrade tests with a driver which keeps running
across the upgrade/downgrade.
The exact sequence leading to this were:
- controller adds ResourceSlice, schedules a sync for one minute in the future (the TTL)
- someone else deletes the ResourceSlice
- add and delete events schedule another sync 30 seconds in the future (the delay),
*overwriting* the other scheduled sync
- sync runs once, finds deleted slices in the mutation cache,
does not re-create them, and also does not run again
One possible fix would be to set a resync period. But then work is done
periodically, whether it's necessary or not.
Another fix is to ensure that the TTL is shorter than the delay. Then when a
sync occurs, all locally stored additional slices are expired. But that renders
the whole storing of recently created slices in the cache pointless.
So the fix used here is to keep track of when another sync has to run because
of added slices. At the end of each sync, the next sync gets scheduled if (and
only if) needed, until eventually syncing can stop.
* Move ClusterEvent type to staging repo, leaving some functions (that contain logic internal to scheduler) in kubernetes/kubernetes
apply review comment and fix linter warning
* update-vendor.sh
* update doc comments
* run update-vendor.sh
Writes to policy resources don't instantaneously take effect in admission. ValidatingAdmissionPolicy
integration tests determine that the policies under test have taken effect by adding a sentinel
policy rule and polling until that rule is applied to a request.
If the marker resource names are the same for each test case in a series of test cases, then
observing a policy's effect on a marker request only indicates that _any_ test policy is in effect,
but it's not necessarily the policy the current test case is waiting for. For example:
1. Test 1 creates a policy and binding.
2. The policy and binding are observed by the admission plugin and take effect.
3. Test 1 observes that a policy is in effect via marker requests.
4. Test 1 exercises the behavior under test and successfully deletes the policy and binding it
created.
5. Test 2 creates a policy and binding.
6. Test 2 observes that a policy is in effect via marker requests, but the policy in effect is still
the one created by Test 1.
7. Test 2 exercises the behavior under test, which fails because it was evaluated against Test 1's
policy.
Generating a per-policy name for the marker resource in each test resolves the timing issue. In the
example, step (6) will not proceed until the admission plugin has observed the policy and binding
created in (5).
Once received job deletion event, it cleans the backoff records for that
job before enqueueing this job so that we can avoid a race condition
that the syncJob() may incorrect use stale backoff records for a newly created
job with same key.
Co-authored-by: Michal Wozniak <michalwozniak@google.com>
The v1alpha3 version is still needed for DeviceTaintRule, but the rest of the
types and most structs became obsolete in v1.32 when we introduced v1beta1 and
bumped the storage version to v1beta1.
Removing them now simplifies adding new features because new fields don't need
to be added to these obsolete types. This could have been done already in 1.33,
but wasn't to minimize disrupting on-going work.
