Currently, openapiv3 discovery uses requestURI to discover resources.
However, that does not work when the rest endpoint contains prefixes
(e.g. `http://localhost/test-endpoint/`).
Because requestURI overwrites prefixes also in rest endpoint
(e.g. `http://localhost/openapiv3/apis/apps/v1`).
Since `absPath` keeps the prefixes in the rest endpoint,
this PR changes to absPath instead requestURI.
In most cases `dir` arg of `SetUpAt()` method of `volume.Mounter` interface is the same as `mounter.GetPath()` because we usually call `SetUpAt()` from `SetUp()` like this:"
```
func (ed *emptyDir) SetUp(mounterArgs volume.MounterArgs) error {
return ed.SetUpAt(ed.GetPath(), mounterArgs)
}
```
(this example is from `volume/emptydir/empty_dir.go`, but there are plenty other examples like that in `volume/*`)
However, there is currently one exception. This is from `volume/projected/projected.go`:
```
if err := wrapped.SetUpAt(dir, mounterArgs); err != nil {
return err
}
```
(see 96306f144a/pkg/volume/projected/projected.go (L203))
In this case `dir` is not equal to `wrapped.GetPath()` and `volume.SetVolumeOwnership()` fails when called from `SetUpAt()` of wrapped volume:
```
lstat /var/lib/kubelet/pods/a2f6e58f-7edf-4c48-a97c-ef1b8fd3caf6/volumes/kubernetes.io~empty-dir/wrapped_kube-api-access-knvkv: no such file or directory
```
To fix the issue let's pass `dir` arg to `volume.SetVolumeOwnership()` explicitly, and use it instead of `mounter.GetPath()`.
The pod worker may recieve a new pod which is marked as terminal in
the runtime cache. This can occur if a pod is marked as terminal and the
kubelet is restarted.
The kubelet needs to drive these pods through the termination state
machine. If upon restart, the kubelet receives a pod which is terminal
based on runtime cache, it indicates that pod finished
`SyncTerminatingPod`, but it did not complete `SyncTerminatedPod`. The
pod worker needs ensure that `SyncTerminatedPod` will run on these pods.
To accomplish this, set `finished=False`, on the pod sync status, to
drive the pod through the rest of the state machine.
This will ensure that status manager and other kubelet subcomponents
(e.g. volume manager), will be aware of this pod and properly cleanup
all of the resources of the pod after the kubelet is restarted.
While making change, also update the comments to provide a bit more
background around why the kubelet needs to read the runtime pod cache
for newly synced terminal pods.
Signed-off-by: David Porter <david@porter.me>
Add a regression test for https://issues.k8s.io/116925. The test
exercises the following:
1) Start a restart never pod which will exit with
`v1.PodSucceeded` phase.
2) Start a graceful deletion of the pod (set a deletion timestamp)
3) Restart the kubelet as soon as the kubelet reports the pod is
terminal (but before the pod is deleted).
4) Verify that after kubelet restart, the pod is deleted.
As of v1.27, there is a delay between the pod being marked terminal
phaes, and the status manager deleting the pod. If the kubelet is
restarted in the middle, after starting up again, the kubelet needs to
ensure the pod will be deleted on the API server.
Signed-off-by: David Porter <david@porter.me>
A pod that cannot be started yet (due to static pod fullname
exclusion when UIDs are reused) must be accounted for in the
pod worker since it is considered to have been admitted and will
eventually start.
Due to a bug we accidentally cleared pendingUpdate for pods that
cannot start yet which means we can't report the right metric to
users in kubelet_working_pods and in theory we might fail to start
the pod in the future (although we currently have not observed
that in tests that should catch such an error). Describe, implement,
and test the invariant that when startPodSync returns in every path
that either activeUpdate OR pendingUpdate is set on the status, but
never both, and is only nil when the pod can never start.
This bug was detected by a "programmer error" assertion we added
on metrics that were not being reported, suggesting that we should
be more aggressive on using log assertions and automating detection
in tests.
