Commit Graph

236 Commits

Author SHA1 Message Date
Ed Bartosh
597c20d050 migrate kubelet/stats to contextual logging 2026-01-12 12:36:59 +02:00
Patrick Ohly
ad79e479c2 build: remove deprecated '// +build' tag
This has been replaced by `//build:...` for a long time now.

Removal of the old build tag was automated with:

    for i in $(git grep -l '^// +build' | grep -v -e '^vendor/'); do if ! grep -q '^// Code generated' "$i"; then sed -i -e '/^\/\/ +build/d' "$i"; fi; done
2025-12-18 12:16:21 +01:00
Davanum Srinivas
914ddf4468 kubelet: improve CRI stats for resource metrics and testing
properly support the resource metrics endpoint when `PodAndContainerStatsFromCRI` is enabled and fix the related e2e tests.

Stats Provider:
- add container-level CPU and memory stats to `ListPodCPUAndMemoryStats` so the resource metrics endpoint has complete data
- add `aggregatePodSwapStats` to compute pod-level swap from container stats (CRI doesn't provide pod-level swap directly)
- add missing memory stats fields: `AvailableBytes`, `PageFaults`, and `MajorPageFaults`
- add platform-specific implementations for Linux and Windows

Tests:
- skip cAdvisor metrics test when `PodAndContainerStatsFromCRI` is enabled (cAdvisor metrics aren't available in that mode)
- fix expected metrics in `ResourceMetricsAPI` test
- `node_swap_usage_bytes` is only available with cAdvisor (need to verify!)
- Add `dumpResourceMetricsForPods` helper to log actual metric values when tests fail, making debugging easier

Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2025-12-17 10:52:32 -05:00
Ed Bartosh
b276c96ef8 Migrate kubelet/server to contextual logging 2025-09-09 12:49:42 +03:00
Lan
cfeeff7ace chore(kubelet): migrate stats to contextual logging
Signed-off-by: Lan <gcslyp@gmail.com>
Co-Authored-By: Ed Bartosh <eduard.bartosh@intel.com>
2025-09-08 15:35:02 +03:00
ylink-lfs
065899e95d chore: use ptr.Deref to replace int deref utils 2025-07-27 16:33:22 +08:00
ylink-lfs
03acd266a3 chore: residual uint64ptr removal with ptr.To 2025-07-20 22:16:45 +08:00
Tim Allclair
a964468638 Add PodCPUAndMemoryStats to stats.Provider interface 2025-07-16 15:59:55 -07:00
Tim Allclair
5eab2b7a6e Remove unuse runtimeCache parameter from stats.Provider 2025-07-11 10:59:20 -07:00
ylink-lfs
3115651410 chore: remove uint64Ptr using ptr.To instead 2025-07-07 21:51:48 +08:00
Sascha Grunert
b464bbeb8f Remove gogo-protobuf from CRI
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2025-07-04 08:55:57 +02:00
Yuan Wang
c5f061e0df Fix pod and container level swap metrics for CRI 2025-06-23 17:57:12 +00:00
Benjamin Elder
139fab0ebd kubelet: don't import testing dependency in non-test code
refactor NewFakeHostStatsProvider to take an injected kubecontainer.OSInterface just like NewFakeHostStatsProviderWithData

this prevents importing a testing package into non-test files, so we can enforce not linking test packages into prod binaries
2025-05-15 12:46:03 -07:00
Haowei Cai
c86ff2339d Unit and E2E tests 2025-03-21 15:54:22 +00:00
Haowei Cai
6bbaf8cb10 Extend CRI stats provider to support PSI 2025-03-21 07:42:52 +00:00
Haowei Cai
92a42e65ec Surface cadvisor PSI metrics to summary API and prometheus metrics 2025-03-21 07:42:51 +00:00
Tim Hockin
e54719bb66 Use randfill, do API renames 2025-03-08 15:18:00 -08:00
Itamar Holder
54500bfe69 cadvisor_provider, unit tests: ensure container-level metrics are collected
Signed-off-by: Itamar Holder <iholder@redhat.com>
2025-01-27 13:13:17 +02:00
Itamar Holder
ceeba21d3d cadvisor_provider, unit test: Add swap stats to cadvisor CPU and Memory stats
Signed-off-by: Itamar Holder <iholder@redhat.com>
2025-01-27 13:13:17 +02:00
Itamar Holder
c111266609 cadvisor_provider, bugfix: Add swap stats to CPU and Memory stats
Signed-off-by: Itamar Holder <iholder@redhat.com>
2025-01-27 13:13:17 +02:00
Itamar Holder
e6c19f315f cri_provider, unit tests: ensure container-level metrics are collected
Signed-off-by: Itamar Holder <iholder@redhat.com>
2025-01-27 13:13:17 +02:00
Itamar Holder
748b52a130 cri_provider, bugfix: Add cadvisor container stats
Without this fix, when CRI stats provided collects cadvisor
stats, pod swap stats are being collected but corresponding
container swap stats are not. This commit fixes this.

Signed-off-by: Itamar Holder <iholder@redhat.com>
2025-01-27 13:13:17 +02:00
Ed Bartosh
e5cb071c2e kubelet: Migrate CAdvisor to contextual logging 2024-11-06 01:49:20 +02:00
Kevin Hannon
c25563b9ab Fix a bug where container fs is considered split if imagefs and
containerfs are on the same drive but different path
2024-10-25 18:51:22 -04:00
Anish Shah
ac82e8a005 Revert "Fix issue in split filesystem where layers are stored on same disk but different paths"
This reverts commit aea0b90652.
2024-10-25 02:09:55 -07:00
Kubernetes Prow Robot
c4a14d7ef5 Merge pull request #126562 from kannon92/split-filesystem-on-same-disk
[KEP-4191]: Fix issue in split filesystem
2024-10-22 19:48:53 +01:00
Prince Pereira
3448455083 Replacing hcsshim library with new hnslib library. 2024-10-14 10:44:30 -07:00
Kirtana Ashok
4a5513c19c Add local reference to hcs structs in windows cri stats test
Signed-off-by: Kirtana Ashok <kiashok@microsoft.com>
2024-09-25 18:56:03 -07:00
Kevin Hannon
aea0b90652 Fix issue in split filesystem where layers are stored on same disk but different paths 2024-08-07 10:43:19 -04:00
Sascha Grunert
50e430b3e9 Fix kubelet cadvisor stats runtime panic
Fixing a kubelet runtime panic when the runtime returns incomplete data:

```
E0729 08:17:47.260393    5218 panic.go:115] "Observed a panic" panic="runtime error: index out of range [0] with length 0" panicGoValue="runtime.boundsError{x:0, y:0, signed:true, code:0x0}" stacktrace=<
        goroutine 174 [running]:
        k8s.io/apimachinery/pkg/util/runtime.logPanic({0x33631e8, 0x4ddf5c0}, {0x2c9bfe0, 0xc000a563f0})
                k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc
        k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x33631e8, 0x4ddf5c0}, {0x2c9bfe0, 0xc000a563f0}, {0x4ddf5c0, 0x0, 0x10000000043c9e5?})
                k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e
        k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000ae08c0?})
                k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
        panic({0x2c9bfe0?, 0xc000a563f0?})
                runtime/panic.go:785 +0x132
        k8s.io/kubernetes/pkg/kubelet/stats.(*cadvisorStatsProvider).ImageFsStats(0xc000535d10, {0x3363348, 0xc000afa330})
                k8s.io/kubernetes/pkg/kubelet/stats/cadvisor_stats_provider.go:277 +0xaba
        k8s.io/kubernetes/pkg/kubelet/images.(*realImageGCManager).GarbageCollect(0xc000a3c820, {0x33631e8?, 0x4ddf5c0?}, {0x0?, 0x0?, 0x4dbca20?})
                k8s.io/kubernetes/pkg/kubelet/images/image_gc_manager.go:354 +0x1d3
        k8s.io/kubernetes/pkg/kubelet.(*Kubelet).StartGarbageCollection.func2()
                k8s.io/kubernetes/pkg/kubelet/kubelet.go:1472 +0x58
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
                k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000add110, {0x3330380, 0xc000afa300}, 0x1, 0xc0000ac150)
                k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
        k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000add110, 0x45d964b800, 0x0, 0x1, 0xc0000ac150)
                k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
        k8s.io/apimachinery/pkg/util/wait.Until(...)
                k8s.io/apimachinery/pkg/util/wait/backoff.go:161
        created by k8s.io/kubernetes/pkg/kubelet.(*Kubelet).StartGarbageCollection in goroutine 1
                k8s.io/kubernetes/pkg/kubelet/kubelet.go:1470 +0x247
```

This commit fixes panics if:

- `len(imageStats.ImageFilesystems) == 0`
- `len(imageStats.ContainerFilesystems) == 0`
- `imageStats.ImageFilesystems[0].FsId == nil`
- `imageStats.ContainerFilesystems[0].FsId == nil`
- `imageStats.ImageFilesystems[0].UsedBytes == nil`
- `imageStats.ContainerFilesystems[0].UsedBytes == nil`

It also fixes the wrapped `nil` error for the check: `err != nil ||
imageStats == nil` in case that `imageStats == nil`.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
2024-07-29 14:13:47 +02:00
Kubernetes Prow Robot
e9d9a82839 Merge pull request #124101 from haircommander/process_stats-with-pid-fix
kubelet: fix PID based eviction
2024-07-25 11:59:57 -07:00
Kevin Hannon
3e642aee3f move container fs check so that we only check if system is split 2024-07-24 11:22:23 -04:00
Peter Hunt
5fd7219cf4 kubelet/stats: fix pid stats for cadvisor stats provider
the process stats aren't correct coming from only the pod stats.
They need to be summed for all of the containers, as cadvisor
is only reading per pid (per container process)

Signed-off-by: Peter Hunt <pehunt@redhat.com>
2024-07-22 10:54:42 -04:00
David Porter
6e6b2b76a3 test: Update summary test to check for process count
The process count is expected to always be >= 1 for pods in the test.

Let's check it's >= 1, so we can catch issues if the proecss count is
not reported.

Signed-off-by: David Porter <david@porter.me>
Signed-off-by: Paco Xu <paco.xu@daocloud.io>
2024-07-22 10:54:42 -04:00
David Porter
f58b46cb97 fix process stats
Signed-off-by: David Porter <david@porter.me>
2024-07-22 10:54:42 -04:00
Kubernetes Prow Robot
27fa59a8af Merge pull request #125656 from gyuho/recent-stats-check-error-for-error-level-logging
feat(kubelet/stats): match cadvisor error to lower not found stats log level
2024-07-18 19:24:01 -07:00
Matthieu MOREL
f014b754fb fix: enable empty and len rules from testifylint on pkg package
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>

Co-authored-by: Patrick Ohly <patrick.ohly@intel.com>
2024-07-06 23:15:43 +00:00
Gyuho Lee
ac992f9a92 remove unncessary errors.Unwrap calls
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2024-07-03 21:31:57 +08:00
Gyuho Lee
d29bdab951 feat(kubelet/stats): match cadvisor error to lower not found stats log level
This "RecentStats: unable to find data in memory cache" error is not actionable,
in terms of kubelt, if the entry is not found in the memory cache.

Thus, proposing it to lower the log level to info.

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2024-06-24 12:25:03 +08:00
Stephen Kitt
3f36c83c68 Switch to stretchr/testify / mockery for mocks
testify is used throughout the codebase; this switches mocks from
gomock to testify with the help of mockery for code generation.

Handlers and mocks in test/utils/oidc are moved to a new package:
mockery operates package by package, and requires packages to build
correctly; test/utils/oidc/testserver.go relies on the mocks and fails
to build when they are removed. Moving the interface and mocks to a
different package allows mockery to process that package without
having to build testserver.go.

Signed-off-by: Stephen Kitt <skitt@redhat.com>
2024-06-20 19:42:53 +02:00
Claudiu Belu
2be8baeaef unittests: Skip failing Windows tests
Some of the unit tests are currently failing on Windows.

Skip them for now, and remove the skips later, once the underlying issues
have been resolved.
2024-04-25 14:24:16 +00:00
Marek Siarkowicz
3ee8178768 Cleanup defer from SetFeatureGateDuringTest function call 2024-04-24 20:25:29 +02:00
Stephen Kitt
6bf667af06 Switch from golang/mock to uber-go/mock
See https://github.com/golang/mock#gomock: golang/mock is no longer
maintained, and should be replaced by go.uber.org/mock.

This allows golang/mock to be dropped from the status and vendored
fields in unwanted-dependencies.json.

Signed-off-by: Stephen Kitt <skitt@redhat.com>
2024-03-07 09:12:16 +01:00
Kubernetes Prow Robot
dc3f5ec6cc Merge pull request #112957 from mxpv/log-dir
Allow changing pod log directory
2024-03-04 21:07:06 -08:00
Paco Xu
0f49a1e36e fix panic in CRI stats of windows 2024-02-18 15:03:11 +08:00
Maksym Pavlenko
ae0a813be1 Fix tests after rebase
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2024-02-16 16:02:10 -08:00
Maksym Pavlenko
ff4f2907c5 Use non-default directory for pod logs and limit path to ASCII characters
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2024-02-16 09:57:17 -08:00
Maksym Pavlenko
d9e2487d0c Add PodLogsPath to kubelet config
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2024-02-16 09:55:59 -08:00
Mark Rossetti
5ba3883be0 Adding unit test for makeWinContainerStats
Signed-off-by: Mark Rossetti <marosset@microsoft.com>
2024-02-12 09:56:58 -08:00
Mark Rossetti
5691a7899e Fixing bug with windows criStatsProvider sometimes not reporting AvailableBytes correctly
Signed-off-by: Mark Rossetti <marosset@microsoft.com>
2024-01-18 10:13:49 -08:00