Commit Graph

7893 Commits

Author SHA1 Message Date
Mike Spreitzer
05abd0c07d More work on shuffle sharding utils
Changes following up on PR #807810 .

Made the validation checking function return a slice of error messages
rather than just a bit.

Replaced all the `int32` with `int` because this is intended for more
than just the priority-and-faireness feature and so should not be a
slave to its configuration datatypes.

Introduced ShuffleAndDealIntoHand, to make memory allocation the
caller's problem/privilege.

Made the hand uniformity tester avoid reflection, evaluate the
histogram against the expected range of counts, and run multiple test
cases, including one in which the number of hash values is a power of
two with four extra bits (as the validation check requires) and one in
which the deck size is not a power of two.

Updated the fairqueuing implementation to use the shuffle sharding
utility package.
2019-09-26 01:06:00 -04:00
yue9944882
639be9133f ensure flow-control api enabled 2019-09-26 01:06:00 -04:00
Mike Spreitzer
313597b96b More uniform and informative debugging of request handling 2019-09-26 01:06:00 -04:00
Mike Spreitzer
fda5d9277c Better goroutine tracking in the API priority and fairness filter
Introduced `OptionalWaitGroup` as a convenient abstraction of
`*WaitGroup`.

Made fairqueuing and emptyRelay::HandleEmpty update the
OptionalWaitGroup and plumbed requestManagementSystem to be able to do
that. Making HandleEmpty update the OptionalWaitGroup illustrates how
a test function that is testing the quiesce functionality of a
QueueSet would do that. This is also a step toward making
requestManagementSystem update the OptionalWaitGroup properly when
there are configuration changes; the remaining steps will include
plumbing the OptionalWaitGroup through the work queue and its worker.

Even without this change it is already possible to test the rest of
QueueSet functionality without any special entry methods. It is also
easy to test the functionality of a requestManagementSystem excepting
its config controller.
2019-09-26 01:06:00 -04:00
yue9944882
77edfe378c fixes some problems in status syncing 2019-09-26 01:06:00 -04:00
Aaron Prindle
fb9dc1b2bb add fair queuing implementation and passing test suite 2019-09-26 01:06:00 -04:00
yue9944882
7a156d2914 flow-control system
compute flow-distinguisher

config controller

plumbs default-confifuration into post-start-hook

addressing reviews

apiserver flow-control
2019-09-26 01:06:00 -04:00
yue9944882
99a4e51fee fixes local-up failures
disable server-side printing

fix test
2019-09-26 01:06:00 -04:00
Bruce Ma
6e0f50c974 make benchmark tests ever-changing and add uniform distribution test
Signed-off-by: Bruce Ma <brucema19901024@gmail.com>
2019-09-26 01:06:00 -04:00
Mike Spreitzer
a8e9921880 Start drafting a unit test of QueueSet
The test runs a scenario over time and evaluates fairness of the
result.

There is no real QueueSet implementation now, so only a dummy is
tested.

In other news, this change introduces a dummy QueueSet implementation
that exerts no control.
2019-09-26 01:06:00 -04:00
Bruce Ma
e7f224651c some refactors on shuffle sharding
Signed-off-by: Bruce Ma <brucema19901024@gmail.com>
2019-09-26 01:06:00 -04:00
Bruce Ma
3328046a34 shuffle sharding: add some benchmarks and tests
Signed-off-by: Bruce Ma <brucema19901024@gmail.com>
2019-09-26 01:06:00 -04:00
Bruce Ma
1d30ce12bc add shuffle sharding utils
Signed-off-by: Bruce Ma <brucema19901024@gmail.com>
2019-09-26 01:06:00 -04:00
Mike Spreitzer
3926050733 Continue refactoring started in PR 80628
Moved the top-level logic associated with the data in
k8s.io/apiserver/pkg/util/flowcontrol into the same package, greatly
simplifying its interface.

Also plumbed the call to NewRequestManagementSystem (1) into
`RecommendedOptions::ApplyTo(RecommendedConfig)` for running in
aggregated servers and (2) into buildGenericConfig in the
kube-apiserver code.

Also moved the definitions of FairQueuingSystem and FairQueuingFactory
into the directory where Aaron is developing his implementation of
those.

Also did some renaming to satisfy golint: FairQueuingSytem became
QueueSet, and FairQueuingFactory became QueueSetFactory.

Also realized that the clock should be passed in every Create... call
to the Factory, the Factory should hold the clock to use (as well as
the WaitGroup that will be helpful for testing).
2019-09-26 01:06:00 -04:00
yue9944882
7b6ef2555c refactor request management filter 2019-09-26 01:05:59 -04:00
Aaron Prindle
dfadb6c9ff update generated code 2019-09-26 01:05:59 -04:00
Aaron Prindle
e17f0671dd update api/flowcontrol/v1alpha1 to have correct Group and Version 2019-09-26 01:05:59 -04:00
Mike Spreitzer
c42702f87a Simplified FairQueuingSystem interface
.. and introduced a tiny bit of starting to work with config objects.
2019-09-26 01:05:59 -04:00
Mike Spreitzer
faf3723ea6 A little more thought about concurrency in the reqmgmt filter
(1) removed the idea of multiple config queue workers; it will be
simpler and adequate if there is only 1.

(2) attempted to clarify the comment on FairQueuingSystem::DoOnEmpty
to (a) make the division of responsibilities clearer, (b) make a
precise statement of the semantics, and (c) be less prescriptive of
how the filter uses this method.

(this is the third edition of this change)
2019-09-26 01:05:59 -04:00
Mike Spreitzer
c79ec2fcf3 Expanded comment on FairQueuingSystem::DoOnEmpty 2019-09-26 01:05:59 -04:00
Mike Spreitzer
1c8f89b554 update bazel by hack 2019-09-26 01:05:59 -04:00
Mike Spreitzer
89caade3cf Started drafting skeleton of the request management filter 2019-09-26 01:05:59 -04:00
yue9944882
24f7323248 [DO NOT REVIEW]: trival codes 2019-09-26 01:05:59 -04:00
yue9944882
601da29814 versioned flow-control api model 2019-09-26 01:05:59 -04:00
Kubernetes Prow Robot
fb1db2eba7 Merge pull request #82957 from RainbowMango/pr_handle_stability_level_default_better
Handle stability level default better
2019-09-25 19:43:07 -07:00
Kubernetes Prow Robot
45f7f70479 Merge pull request #80465 from answer1991/remove-apiserver-loopback-client-qps-limit
remove apiserver loopback client QPS limit
2019-09-25 17:15:07 -07:00
Kubernetes Prow Robot
28bcf55acc Merge pull request #83113 from yastij/bump-utils
bump k8s.io/utils to pickup ipallocator changes
2019-09-25 13:52:36 -07:00
Kubernetes Prow Robot
ea9a559093 Merge pull request #83075 from yutedz/pod-ready-time
Remove unnecessary traversal of pod.Status.Conditions
2019-09-25 13:52:03 -07:00
Kubernetes Prow Robot
6555c34743 Merge pull request #83022 from mm4tt/controller_ref
Optimize GetControllerOf method and add a benchmark for it
2019-09-25 13:51:14 -07:00
Kubernetes Prow Robot
08f46e425b Merge pull request #82932 from ashish-billore/mybranch2
Corrected the pod reporting and messages
2019-09-25 11:07:13 -07:00
Mahendra Kariya
3698100224 Fix golint errors in pkg/apis/core (#82919)
* Fix lint errors related to receiver name

Ref #68026

* Fix lint errors related to comments

Ref #68026

* Fix package name in comments

Ref #68026

* Rename Cpu to CPU

Ref #68026

* Fix lint errors related to naming convention

Ref #68026

* Remove deprecated field

DoNotUse_ExternalID has been deprecated and is not in use anymore.
It has been removed to fix lint errors related to underscores in field
names.

Ref #68026, #61966

* Include pkg/apis/core in golint check

Ref #68026

* Rename var to fix lint errors

Ref #68026

* Revert "Remove deprecated field"

This reverts commit 75e9bfc168077fcb9346e334b59d60a2c997735b.

Ref #82919

* Remove math from godoc

Ref #82919, #68026

* Remove underscore from var name

Ref #68026

* Rename var in staging core api type

Ref #68026
2019-09-25 11:06:51 -07:00
Kubernetes Prow Robot
dd8a23ca2e Merge pull request #82643 from ZP-AlwaysWin/dev02
Fix bug The statefulset have duplicate revision after resource was up…
2019-09-25 11:06:16 -07:00
Kubernetes Prow Robot
973629f742 Merge pull request #82410 from nikhita/informer-gen-package-name-dots
[v1.16.1] informer-gen: allow package names containing dots
2019-09-25 11:05:50 -07:00
Kubernetes Prow Robot
07025a5d9e Merge pull request #82409 from RainbowMango/pr_add_authentication_overall_latency_metrics
Add authentication overall latency metrics
2019-09-25 11:05:32 -07:00
Kubernetes Prow Robot
0676af36c5 Merge pull request #82388 from marun/unstructured-reflector
Add support for type checking Unstructured event objects via GVK to client-go's reflector
2019-09-25 11:05:14 -07:00
Kubernetes Prow Robot
327f53ba57 Merge pull request #83064 from liggitt/propagate-context
Propagate context to remote authorize/authenticate webhook calls
2019-09-25 09:32:01 -07:00
Yassine TIJANI
16fe4d7627 bump k8s.io/utils to pickup ipallocator changes
Signed-off-by: Yassine TIJANI <ytijani@vmware.com>
2019-09-25 16:15:30 +02:00
Matt Matejczyk
01ccd2e19e Optimize GetControllerOf method
In addition create a similar method that doesn't copy objects.

Benchmark for the new no-copy method vs the old one:
```
benchmark                       old ns/op     new ns/op     delta
BenchmarkGetControllerOf-12     214           14.8          -93.08%

benchmark                       old allocs     new allocs     delta
BenchmarkGetControllerOf-12     1              0              -100.00%

benchmark                       old bytes     new bytes     delta
BenchmarkGetControllerOf-12     80            0             -100.00%
```

Benchamrk for the new (copy) method vs the old one:
```
benchmark                       old ns/op     new ns/op     delta
BenchmarkGetControllerOf-12     128           114           -10.94%

benchmark                       old allocs     new allocs     delta
BenchmarkGetControllerOf-12     1              1              +0.00%

benchmark                       old bytes     new bytes     delta
BenchmarkGetControllerOf-12     80            80            +0.00%

```

Overall there is a 10% improvement for the old vs new (copy) method and
huge improvent (x10) for the old vs new (no-copy).

I changed the IsControlledBy and a few other methods to use the new (no-copy) method.
2019-09-25 15:07:37 +02:00
ashish-billore
31e427b54c Corrected the pod reporting and messages
It is inconsistent and confusing to report pod count from all
namespaces but report message for only default namespace.
Added the namespace (default) reporting to clarify this.
Updated comments for usage clarity.
2019-09-25 14:17:44 +09:00
Jordan Liggitt
b78edd86b8 Plumb context to webhook calls 2019-09-24 21:59:59 -04:00
Jordan Liggitt
4c686ddc1c Propagate context to ExponentialBackoff 2019-09-24 21:59:59 -04:00
Ted Yu
936ae632a0 Remove unnecessary traversal of pod.Status.Conditions 2019-09-24 10:47:34 -07:00
Jordan Liggitt
92eb072989 Propagate context to Authorize() calls 2019-09-24 11:14:54 -04:00
Jordan Liggitt
4ffa91a388 Add context-propagating CreateContext methods to *Review clients 2019-09-24 11:08:00 -04:00
Maru Newby
2a836d1710 Run hack/update-bazel.sh 2019-09-24 14:47:02 +00:00
Maru Newby
237dbfd8ad Add support for type checking Unstructured via GVK in reflector
It was previously possible to instantiate `Reflector` with
`*unstructured.Unstructured` as the expected type but this did not
support checking that event objects were of the correct API
type (e.g. if event object was `v1.Pod` in `Unstructured` form but
`v1.Service` was expected). This commit adds support for providing a
GVK via an `Unstructured` expected type to compare with the GVK of
event objects. The GVK will also be used in reflector log output.
2019-09-24 14:46:59 +00:00
ZP-AlwaysWin
e081d1dca6 Fix bug The statefulset have duplicate revision after resource was updated 2019-09-24 21:20:24 +08:00
Kubernetes Prow Robot
0541d0bb79 Merge pull request #83018 from jfbai/fix-remove-duplicated-field-in-eventKey
fix: remove reportingInstance field in eventKey.
2019-09-24 04:53:27 -07:00
Kubernetes Prow Robot
9be907d792 Merge pull request #83019 from RainbowMango/pr_wrap_prom_Labels
Wrap promethues.Labels to stability framework.
2019-09-23 21:33:25 -07:00
RainbowMango
4127525e9b Migrate stability level handle functionality overall metrics package 2019-09-24 12:27:21 +08:00