From 5686664a1d6507cfcab11a20a4aada13077b6c9b Mon Sep 17 00:00:00 2001 From: Patrick Ohly Date: Tue, 27 Oct 2020 11:16:58 +0100 Subject: [PATCH] PV controller: don't delete PVs when PVC is not known yet Normally, the PV controller knows about the PVC that triggers the creation of a PV before it sees the PV, because the PV controller must set the volume.beta.kubernetes.io/storage-provisioner annotation that tells an external provisioner to create the PV. When restarting, the PV controller first syncs its caches, so that case is also covered. However, the creator of a PVC might decided to set that annotation itself to speed up volume creation. While unusual, it's not forbidden and thus part of the external Kubernetes API. Whether it makes sense depends on the intentions of the user. When that is done and there is heavy load, an external provisioner might see the PVC and create a PV before the PV controller sees the PVC. If the PV controller then encounters the PV before the PVC, it incorrectly concludes that the PV needs to be deleted instead of being bound. The same issue occurred earlier for external binding and the existing code for looking up a PVC in the cache or in the apiserver solves the issue also for volume provisioning, it just needs to be enabled also for PVs without the pv.kubernetes.io/bound-by-controller annotation. --- pkg/controller/volume/persistentvolume/pv_controller.go | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/pkg/controller/volume/persistentvolume/pv_controller.go b/pkg/controller/volume/persistentvolume/pv_controller.go index 61b1ed1f3ef..e3d7dfbe2a8 100644 --- a/pkg/controller/volume/persistentvolume/pv_controller.go +++ b/pkg/controller/volume/persistentvolume/pv_controller.go @@ -577,9 +577,10 @@ func (ctrl *PersistentVolumeController) syncVolume(volume *v1.PersistentVolume) if err != nil { return err } - if !found && metav1.HasAnnotation(volume.ObjectMeta, pvutil.AnnBoundByController) { - // If PV is bound by external PV binder (e.g. kube-scheduler), it's - // possible on heavy load that corresponding PVC is not synced to + if !found { + // If the PV was created by an external PV provisioner or + // bound by external PV binder (e.g. kube-scheduler), it's + // possible under heavy load that the corresponding PVC is not synced to // controller local cache yet. So we need to double-check PVC in // 1) informer cache // 2) apiserver if not found in informer cache