mirror of
https://github.com/k3s-io/kubernetes.git
synced 2026-01-30 05:43:58 +00:00
e86b3f266c
Automatic merge from submit-queue Fix wrapped volume race **EDIT:** now covers configmap, secret, downwardapi & git_repo volume plugins. Fixes #29297. wrappedVolumeSpec used by configMapVolumeMounter and configMapVolumeUnmounter contained a pointer to api.Volume which was being patched by NewWrapperMounter/NewWrapperUnmounter, causing race condition during configmap volume mounts. See https://github.com/kubernetes/kubernetes/issues/29297#issuecomment-235403806 for complete explanation. The subtle bug was introduced by #18445, it also can affect other volume plugins utilizing `wrappedVolumeSpec` technique, if this PR is correct/accepted will make more PRs for secrets etc. Although tmpfs variety of inner `emptyDir` volume appears to be less susceptible to this race, there's chance it can fail too. The errors produced by this race look like this: ```Jul 19 17:05:21 ubuntu1604 kubelet[17097]: I0719 17:05:21.854303 17097 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/configmap/foo-files" (spec.Name: "files") to pod "11786582-4dbf-11e6-9fc9-64cca009c636" (UID: "11786582-4dbf-11e6-9fc9-64cca009c636"). Jul 19 17:05:21 ubuntu1604 kubelet[17097]: I0719 17:05:21.854842 17097 reconciler.go:253] MountVolume operation started for volume "kubernetes.io/configmap/bar-file s" (spec.Name: "files") to pod "117d2c22-4dbf-11e6-9fc9-64cca009c636" (UID: "117d2c22-4dbf-11e6-9fc9-64cca009c636"). Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.860796 17097 configmap.go:171] Error creating atomic writer: stat /var/lib/kubelet/pods/117d2c22-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.861070 17097 goroutinemap.go:155] Operation for "kubernetes.io/configmap/bar-files" failed. No retries permitted until 2016-07-19 17:07:21.861036886 +0200 CEST (durationBeforeRetry 2m0s). error: MountVolume.SetUp failed for volume "kubernetes.io/configmap/bar-files" (spec.Name: "files") pod "117d2c22-4dbf-11e6-9fc9-64cca009c636" (UID: "117d2c22-4dbf-11e6-9fc9-64cca009c636") with: stat /var/lib/kubelet/pods/117d2c22-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.861271 17097 configmap.go:171] Error creating atomic writer: stat /var/lib/kubelet/pods/11786582-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory Jul 19 17:05:21 ubuntu1604 kubelet[17097]: E0719 17:05:21.862284 17097 goroutinemap.go:155] Operation for "kubernetes.io/configmap/foo-files" failed. No retries permitted until 2016-07-19 17:07:21.862275753 +0200 CEST (durationBeforeRetry 2m0s). error: MountVolume.SetUp failed for volume "kubernetes.io/configmap/foo-files" (spec.Name: "files") pod "11786582-4dbf-11e6-9fc9-64cca009c636" (UID: "11786582-4dbf-11e6-9fc9-64cca009c636") with: stat /var/lib/kubelet/pods/11786582-4dbf-11e6-9fc9-64cca009c636/volumes/kubernetes.io~configmap/files: no such file or directory``` Note "Error creating atomic writer" errors. This problem can be reproduced by making kubelet mount multiple config map volumes in parallel.