As pointed out during code review, the CEL cost estimates are not considered
perfectly reliable. Therefore it is better to also do runtime checks.
Some downstream users might decide to allow CEL expressions to run
longer. Therefore the cost limit is now part of an Options struct.
kube-scheduler uses the default cost limit defined in the resource.k8s.io API,
which is the same cost limit that also the apiserver uses during validation.
Expression evaluation in all scenarios gets benchmarked where compilation
works. A pending optimization in another PR caches compiled expressions, so the
time for compilation will become less important. What matters is the actual
evaluation.
In DRA, the cost check is done only at validation time. At runtime, any
expression that passed validation gets executed without interrupting it. The
advantage is that it becomes easier to change the limit because stored
expression do not suddenly fail after an up- or downgrade. The limit could
even become a configuration parameter of the apiserver because that is the only
place where the limit gets checked
The main purpose is to protect against denial-of-service attacks. Scheduling
time depends a lot on unpredictable factors and expected scheduling time also
varies, so no attempt is made to limit the overall time spent on evaluating CEL
expressions per claim.
When PodAndContainerStatsFromCRI FG is enabled, Kubelet tries to get
list of metrics from the CRI runtime using CRI API 'ListMetricDescriptors'.
As this API is not implemented in neither CRI-O nor Containerd versions
used in the test-infra, ResourceMetrics test case fails to gather
certain container metrics.
Excluding container metrics from the expected list of metrics if
PodAndContainerStatsFromCRI is enabled should solve the issue.
This commit allows the configuration of topology manager
option with the use of environment variable, following
same approach as done for configuring cpumanager policy
options.
In order to improve the observability of the resource management
in kubelet, cpu allocation and NUMA alignment, we add more metrics
to report if resource alignment is in effect.
The more precise reporting would probably be using pod status,
but this would require more invasive and riskier changes,
and possibly extra interactions to the APIServer.
We start adding metrics to report if containers got their
compute resources aligned.
If metrics are growing, the assingment is working as expected;
If metrics stay consistent, perhaps at zero, no resource
alignment is done.
Extra fixes brought by this work
- retroactively add labels for existing tests
- running metrics test demands precision accounting to avoid flakes;
ensure the node state is restored pristine between each test, to
minimize the aforementioned risk of flakes.
- The test pod command line was wrong, with this the pod could not
reach Running state. That gone unnoticed so far because
no test using this utility function actually needed a pod
in running state.
Signed-off-by: Francesco Romani <fromani@redhat.com>
