Continue streaming kubelet logs when runtime is unavailable

Container runtimes are able to run existing containers even when their
main CRI server is not available for any reason. The call to the
container status RPC happens quite frequently during log parsing, means
that a single CRI interruption will also abort streaming the logs.

We now check that specific use case and continue following the log
streaming if the CRI is unavailable. We still abort the streaming
accordingly if the CRI comes back and the container status reports that
the workload has exited.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
This commit is contained in:
Sascha Grunert 2024-03-22 09:04:40 +01:00
parent 9d63e575f8
commit ff95ae0d3c
No known key found for this signature in database
GPG Key ID: 09D97D153EF94D93

View File

@ -30,6 +30,8 @@ import (
"time"
"github.com/fsnotify/fsnotify"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
v1 "k8s.io/api/core/v1"
internalapi "k8s.io/cri-api/pkg/apis"
@ -422,6 +424,13 @@ func ReadLogs(ctx context.Context, logger *klog.Logger, path, containerID string
func isContainerRunning(ctx context.Context, logger *klog.Logger, id string, r internalapi.RuntimeService) (bool, error) {
resp, err := r.ContainerStatus(ctx, id, false)
if err != nil {
// Assume that the container is still running when the runtime is
// unavailable. Most runtimes support that containers can be in running
// state even if their CRI server is not available right now.
if status.Code(err) == codes.Unavailable {
return true, nil
}
return false, err
}
status := resp.GetStatus()