github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-07-30 15:05:27 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #63321 from sjenning/fix-pod-deletor

Browse Source 
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

 kubelet: force filterContainerID to empty string when removeAll is true 

fixes https://github.com/kubernetes/kubernetes/issues/57865

alternative to https://github.com/kubernetes/kubernetes/pull/62170

There is a bug in the container deletor where if `removeAll` is `true` in `deleteContainersInPod()` the `filterContainerID` is still used to filter results in `getContainersToDeleteInPod()`.  If the filter container is not found, no containers are returned for deletion.

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pod_container_deletor.go#L74-L77

This is the case for the delayed deletion a pod in `CrashLoopBackoff` as the death of the infra container in response to the `DELETE` is detected by PLEG and triggers an attempt to clean up all containers but uses the infra container id as a filter with `removeAll` set to `true`.  The infra container is immediately deleted and thus can not be found when `getContainersToDeleteInPod()` tries to find it.  Thus the dead app container from the previous restart attempt still exists.

`canBeDeleted()` in the status manager will return `false` until all the pod containers are deleted, delaying the deletion of the pod on the API server.

The removal of the containers is eventually forced by the API server `REMOVE` after the grace period.
This commit is contained in:
Kubernetes Submit Queue 2018-05-01 09:50:13 -07:00 committed by GitHub
commit 8eb7eeef39
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 6 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
pkg/kubelet/kubelet_pods.go
View File

@ -903,7 +903,11 @@ func (kl *Kubelet) PodResourcesAreReclaimed(pod *v1.Pod, status v1.PodStatus) bo
return false
}
if len(runtimeStatus.ContainerStatuses) > 0 {
glog.V(3).Infof("Pod %q is terminated, but some containers have not been cleaned up: %+v", format.Pod(pod), runtimeStatus.ContainerStatuses)
var statusStr string
for _, status := range runtimeStatus.ContainerStatuses {
statusStr += fmt.Sprintf("%+v ", *status)
}
glog.V(3).Infof("Pod %q is terminated, but some containers have not been cleaned up: %s", format.Pod(pod), statusStr)
return false
}
if kl.podVolumesExist(pod.UID) && !kl.keepTerminatedPodVolumes {

1
pkg/kubelet/pod_container_deletor.go
View File

@ -99,6 +99,7 @@ func (p *podContainerDeletor) deleteContainersInPod(filterContainerID string, po
containersToKeep := p.containersToKeep
if removeAll {
containersToKeep = 0
filterContainerID = ""
}
for _, candidate := range getContainersToDeleteInPod(filterContainerID, podStatus, containersToKeep) {