Commit Graph

106947 Commits

Author SHA1 Message Date
Tim Hockin
bf27cad256 Build flags: use all= syntax
This has somewhat subtle implications.  For a concrete example, this
changes the `-trimpath` behavior from only affecting the named pkg to
affecting all pkgs, which broke ginkgo, which seems to try to strip its
own `pwd` prefix.  But since that runs in run-in-gopath, and not in
KUBE_ROOT, it fails to strip anything.

e.g.

before this, strings in the binary would be like
    /home/user/kube/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/...
Ginkgo would find its own root as
    /home/user/kube/_output/local/go/src/k8s.io/kubernetes/
so it would produce
    vendor/github.com/onsi/ginkgo/...
in logs.

after this, strings in the binary strip the KUBE_ROOT and be like:
    _output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/...
Ginkgo would find its own root as
    /home/user/kube/_output/local/go/src/k8s.io/kubernetes/
so it would not strip anything, and produce
    _output/local/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/...
in logs.
2022-03-01 08:48:31 -08:00
Tim Hockin
ed5e549cde Makefile: Add a DBG flag to build debug binaries
Now `make DBG=1` will produce binaries with no optimizaions and no
inlining, but with symbols and DWARF information.
2022-03-01 08:47:34 -08:00
Tim Hockin
56ad63913a Make builds fail if go2make misbehaves
Rather than an obscure error.
2022-03-01 08:37:40 -08:00
Davanum Srinivas
22cef5094e
Log where we pull images from
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
2022-03-01 09:53:19 -05:00
Kevin Klues
99c57828ce Update TopologyManager algorithm for selecting "best" non-preferred hint
For the 'single-numa' and 'restricted' TopologyManager policies, pods are only
admitted if all of their containers have perfect alignment across the set of
resources they are requesting. The best-effort policy, on the other hand, will
prefer allocations that have perfect alignment, but fall back to a non-preferred
alignment if perfect alignment can't be achieved.

The existing algorithm of how to choose the best hint from the set of
"non-preferred" hints is fairly naive and often results in choosing a
sub-optimal hint. It works fine in cases where all resources would end up
coming from a single NUMA node (even if its not the same NUMA nodes), but
breaks down as soon as multiple NUMA nodes are required for the "best"
alignment.  We will never be able to achieve perfect alignment with these
non-preferred hints, but we should try and do something more intelligent than
simply choosing the hint with the narrowest mask.

In an ideal world, we would have the TopologyManager return a set of
"resources-relative" hints (as opposed to a common hint for all resources as is
done today). Each resource-relative hint would indicate how many other
resources could be aligned to it on a given NUMA node, and a  hint provider
would use this information to allocate its resources in the most aligned way
possible. There are likely some edge cases to consider here, but such an
algorithm would allow us to do partial-perfect-alignment of "some" resources,
even if all resources could not be perfectly aligned.

Unfortunately, supporting something like this would require a major redesign to
how the TopologyManager interacts with its hint providers (as well as how those
hint providers make decisions based on the hints they get back).

That said, we can still do better than the naive algorithm we have today, and
this patch provides a mechanism to do so.

We start by looking at the set of hints passed into the TopologyManager for
each resource and generate a list of the minimum number of NUMA nodes required
to satisfy an allocation for a given resource. Each entry in this list then
contains the 'minNUMAAffinity.Count()' for a given resources. Once we have this
list, we find the *maximum* 'minNUMAAffinity.Count()' from the list and mark
that as the 'bestNonPreferredAffinityCount' that we would like to have
associated with whatever "bestHint" we ultimately generate. The intuition being
that we would like to (at the very least) get alignment for those resources
that *require* multiple NUMA nodes to satisfy their allocation. If we can't
quite get there, then we should try to come as close to it as possible.

Once we have this 'bestNonPreferredAffinityCount', the algorithm proceeds as
follows:

If the mergedHint and bestHint are both non-preferred, then try and find a hint
whose affinity count is as close to (but not higher than) the
bestNonPreferredAffinityCount as possible. To do this we need to consider the
following cases and react accordingly:

  1. bestHint.NUMANodeAffinity.Count() >  bestNonPreferredAffinityCount
  2. bestHint.NUMANodeAffinity.Count() == bestNonPreferredAffinityCount
  3. bestHint.NUMANodeAffinity.Count() <  bestNonPreferredAffinityCount

For case (1), the current bestHint is larger than the
bestNonPreferredAffinityCount, so updating to any narrower mergeHint is
preferred over staying where we are.

For case (2), the current bestHint is equal to the
bestNonPreferredAffinityCount, so we would like to stick with what we have
*unless* the current mergedHint is also equal to bestNonPreferredAffinityCount
and it is narrower.

For case (3), the current bestHint is less than bestNonPreferredAffinityCount,
so we would like to creep back up to bestNonPreferredAffinityCount as close as
we can. There are three cases to consider here:

  3a. mergedHint.NUMANodeAffinity.Count() >  bestNonPreferredAffinityCount
  3b. mergedHint.NUMANodeAffinity.Count() == bestNonPreferredAffinityCount
  3c. mergedHint.NUMANodeAffinity.Count() <  bestNonPreferredAffinityCount

For case (3a), we just want to stick with the current bestHint because choosing
a new hint that is greater than bestNonPreferredAffinityCount would be
counter-productive.

For case (3b), we want to immediately update bestHint to the current
mergedHint, making it now equal to bestNonPreferredAffinityCount.

For case (3c), we know that *both* the current bestHint and the current
mergedHint are less than bestNonPreferredAffinityCount, so we want to choose
one that brings us back up as close to bestNonPreferredAffinityCount as
possible. There are three cases to consider here:

  3ca. mergedHint.NUMANodeAffinity.Count() >  bestHint.NUMANodeAffinity.Count()
  3cb. mergedHint.NUMANodeAffinity.Count() <  bestHint.NUMANodeAffinity.Count()
  3cc. mergedHint.NUMANodeAffinity.Count() == bestHint.NUMANodeAffinity.Count()

For case (3ca), we want to immediately update bestHint to mergedHint because
that will bring us closer to the (higher) value of
bestNonPreferredAffinityCount.

For case (3cb), we want to stick with the current bestHint because choosing the
current mergedHint would strictly move us further away from the
bestNonPreferredAffinityCount.

Finally, for case (3cc), we know that the current bestHint and the current
mergedHint are equal, so we simply choose the narrower of the 2.

This patch implements this algorithm for the case where we must choose from a
set of non-preferred hints and provides a set of unit-tests to verify its
correctness.

Signed-off-by: Kevin Klues <kklues@nvidia.com>
2022-03-01 14:38:26 +00:00
Kubernetes Prow Robot
50e07c9a12
Merge pull request #108381 from thockin/add-generated-openapi
Add the last zz_generated.openapi.go file
2022-03-01 06:35:48 -08:00
Antonio Ojea
2660ac8a6d integration framework: log number of leaked goroutines 2022-03-01 15:31:13 +01:00
kerthcet
9c5898ba90 feat: update feature gate DefaultPodTopologySpread release note
Signed-off-by: kerthcet <kerthcet@gmail.com>
2022-03-01 19:06:24 +08:00
Piotr Betkier
13c3f091a7 Small review fixes 2022-03-01 09:24:26 +01:00
Piotr Betkier
50200dc54f Add e2e test for HPA behavior: decreased downscale stabilization 2022-03-01 09:24:26 +01:00
Kubernetes Prow Robot
77f6476d34
Merge pull request #108327 from SergeyKanzhelev/unused_jenkins_files
two files not being used any longer
2022-02-28 19:43:46 -08:00
ZhangJian He
d09947a5b5 Fix typo in watch_based_manager_test 2022-03-01 10:21:17 +08:00
Kubernetes Prow Robot
bef9d807a0
Merge pull request #108325 from pacoxu/donotReturnErrWhenPauseLose
do not return err when PodSandbox not exist
2022-02-28 18:15:46 -08:00
Kubernetes Prow Robot
1cc8edb838
Merge pull request #108286 from ii/replace-pod-template
Write replaceCoreV1NamespacedPodTemplate test - +1 endpoint coverage
2022-02-28 17:11:46 -08:00
Kubernetes Prow Robot
effff78a1f
Merge pull request #107744 from Shubham82/Adding_logs-node_authorizer
Added Logs Statement for a Pod in graph_populator.
2022-02-28 16:07:58 -08:00
Kubernetes Prow Robot
e9ba9dc4e4
Merge pull request #107201 from pacoxu/add-metrics-volume-stats-cal
add VolumeStatCalDuration metrics for fsquato monitoring benchmark
2022-02-28 16:07:46 -08:00
Kubernetes Prow Robot
5ee80dee04
Merge pull request #108296 from aojea/client_go_size_metrics
client-go: add request and response size metrics
2022-02-28 14:27:46 -08:00
Kevin Klues
f8601cb5a3 Refactor TopologyManager to be more explicit about bestHint calculation
Signed-off-by: Kevin Klues <kklues@nvidia.com>
2022-02-28 20:30:01 +00:00
Kubernetes Prow Robot
f4df7d0eb2
Merge pull request #108393 from ialidzhikov/nit/ingressclassnamespacedparams
Correct comment related to IngressClassNamespacedParams feature gate
2022-02-28 12:23:46 -08:00
Kubernetes Prow Robot
2c91952fcf
Merge pull request #106486 from Ahmed-Aghadi/codeEnhanceNode
test/e2e/node + test/e2e/scheduling: improve checks
2022-02-28 11:17:46 -08:00
Tim Hockin
f9e19fc83e Add the last zz_generated.openapi.go file
We had 4 of 5 checked in.
2022-02-28 10:17:54 -08:00
AHMED AGHADI
ff0a3009db Improve checks for test/e2e/node and test/e2e/scheduling 2022-02-28 23:44:21 +05:30
Kubernetes Prow Robot
cc4c95181e
Merge pull request #108187 from stevekuznetsov/skuznets/storage-tests-diff
storage: etcd: use cmp.Diff for comparisons
2022-02-28 10:08:57 -08:00
ialidzhikov
856cf94d55 Correct comment related to IngressClassNamespacedParams feature gate
Signed-off-by: ialidzhikov <i.alidjikov@gmail.com>
2022-02-28 18:55:13 +02:00
Steve Kuznetsov
dfdd486f09
storage: etcd: use cmp.Diff for comparisons
This commit simply modernizes the comparisons made in the storage tests
to use `cmp.Diff()` so that pointer comparisons and length checks do not
have to be made by hand. We also get nice diffs in the test output this
way instead of large pasted blobs.

Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
2022-02-28 08:41:45 -08:00
Kubernetes Prow Robot
bf7b9119f0
Merge pull request #108278 from kerthcet/feature/graduate-defaultPodTopologySpread-to-ga
graduate default pod topology spread to ga
2022-02-28 08:02:57 -08:00
Antonio Ojea
64d9d0585f client-go: add request and response size metrics
Get metrics for the request and response  size, so we can correlate latency
and size on a request, otherwise we could get confused because we don't know if the
network is slow or just the request size huge.
2022-02-28 16:59:12 +01:00
Marek Siarkowicz
8f5250e3fe Start building etcd v3.6.0-alpha.0 image for scalability tests 2022-02-28 13:58:18 +01:00
Emre Sülün
d208389055
Fix typo in bug report template 2022-02-28 12:34:43 +03:00
Kubernetes Prow Robot
5b1e538759
Merge pull request #108378 from thockin/makefile-use-loglib
Makefile: emit codegen info via kube::log::status
2022-02-27 14:27:18 -08:00
Kubernetes Prow Robot
6d7c252906
Merge pull request #108377 from thockin/makefile-check-restarts
Makefile: avoid redundant work upon make restart
2022-02-27 13:25:17 -08:00
Kubernetes Prow Robot
8b46a0d276
Merge pull request #108376 from thockin/makefile-static-nonstatic-output
Makefile: clean up static/nonstatic build output
2022-02-27 12:01:17 -08:00
Tim Hockin
457e8778f7 Makefile: emit codegen info via kube::log::status 2022-02-27 11:33:52 -08:00
Tim Hockin
88fef96e75 Makefile: avoid redundant work upon make restart
Only build and run go2make on the first pass.  If that generates a new
GO_PKGDEPS_FILE, make will restart the whole process and set MAKE_RESTARTS to
a numeric value.

We can use this to avoid re-running go2make, which saves a few seconds
each build.
2022-02-27 10:50:19 -08:00
Tim Hockin
4a0c6c2444 Slightly nicer output when building
before:

```
$ make generated_files
+++ [0226 13:42:17] Building go targets for linux/amd64:
    hack/make-rules/helpers/go2make
> non-static build: k8s.io/kubernetes/hack/make-rules/helpers/go2make
```

after:

```
$ make generated_files
+++ [0226 14:30:08] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
```
2022-02-27 10:29:59 -08:00
Tim Hockin
30b20f184e Use build.sh to build go2make
This allows common flags to be passed in (upcoming commits).
2022-02-27 10:28:44 -08:00
Kubernetes Prow Robot
25c9bca49c
Merge pull request #108367 from thockin/makefile-hack-tools
Remove hack/tools/Makefile
2022-02-26 20:33:17 -08:00
Kubernetes Prow Robot
e9760925f9
Merge pull request #108369 from thockin/makefile-errexit
Makefile: use errexit, pipefail, and nounset
2022-02-26 17:45:16 -08:00
Kubernetes Prow Robot
0436117231
Merge pull request #108368 from thockin/makefile-superfluous-at
Makefile: remove superfluous @
2022-02-26 13:35:23 -08:00
Tim Hockin
e671fea954 Remove hack/tools/Makefile 2022-02-26 12:57:10 -08:00
Tim Hockin
9ad4a659ae Makefile: use errexit, pipefail, and nounset 2022-02-26 12:32:48 -08:00
Tim Hockin
7a3dded74d Makefile: remove superfluous @ 2022-02-26 12:17:45 -08:00
Kubernetes Prow Robot
05a71be79d
Merge pull request #108365 from liggitt/fix-proto-gen
Fix proto gen
2022-02-26 11:17:23 -08:00
Jordan Liggitt
48a1c729a0 Regenerate protobuf
Change-Id: I2a563514955d7fc7559ceb7afb73df08ace8fd8b
2022-02-26 18:02:52 +00:00
Jordan Liggitt
f68adbcd59 Fix protobuf generation 2022-02-26 13:00:42 -05:00
Kubernetes Prow Robot
109cb7cbe7
Merge pull request #108339 from cheftako/anp-release
Bump konnectivity-client to v0.0.28
2022-02-25 17:49:23 -08:00
Kevin Delgado
bbf1208480 PR feedback 2022-02-25 23:07:41 +00:00
Shubham Kuchhal
cdfbcf94fc Recording the Duration. 2022-02-25 13:20:01 +05:30
Kubernetes Prow Robot
5bafc8306a
Merge pull request #108334 from helayoty/fix-kep-kube-sched-link
fix: fix kube-scheduler KEP link
2022-02-24 23:39:54 -08:00
Kubernetes Prow Robot
356ec6f12f
Merge pull request #108295 from adisky/install-cni-if-not-installed
Detect CNI installation in local-up-cluster.sh
2022-02-24 22:35:51 -08:00