Production-Grade Container Scheduling and Management
Go to file
Kubernetes Submit Queue 15b1d165fb
Merge pull request #58107 from ironcladlou/quota-controller-deadlock
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix quota controller worker deadlock

The resource quota controller worker pool can deadlock when:

* Worker goroutines are idle waiting for work from queues
* The Sync() method detects discovery updates to apply

The problem is workers acquire a read lock while idle, making write lock
acquisition dependent upon the presence of work in the queues.

The Sync() method blocks on a pending write lock acquisition and won't unblock
until every existing worker processes one item from their queue and releases
their read lock. While the Sync() method's lock is pending, all new read lock
acquisitions will block; if a worker does process work and release its lock, it
will then become blocked on a read lock acquisition; they become blocked on
Sync(). This can easily deadlock all the workers processing from one queue while
any workers on the other queue remain blocked waiting for work.

Fix the deadlock by refactoring workers to acquire a read lock *after* work is
popped from the queue. This allows writers to get locks while workers are idle,
while preserving the worker pause semantics necessary to allow safe sync.

```release-note
Fixes an infrequent problem causing the resource quota controller to become stuck in clusters with low ResourceQuota churn, potentially preventing quota from being recalculated until the controller is restarted or until bursts of diverse quota activity unstick the controller.
```

/cc @kubernetes/sig-api-machinery-bugs
2018-01-10 15:59:50 -08:00
.github
api fixed some bad url 2018-01-10 22:05:00 +08:00
build Fix scheduler refs in BUILD files. 2018-01-05 15:05:01 -08:00
cluster Merge pull request #58063 from monotek/detect-exceptions 2018-01-10 14:21:47 -08:00
cmd Fix cadvisor flag registration for cross build 2018-01-10 13:00:28 -05:00
docs Run update-api-reference-docs.sh. 2018-01-03 00:18:03 +00:00
examples Fix scheduler refs in BUILD files. 2018-01-05 15:05:01 -08:00
Godeps Merge pull request #57989 from NickrenREN/spec-0.1 2018-01-09 00:27:22 -08:00
hack Merge pull request #57571 from spiffxp/mk-staging-repos-authoritative 2018-01-10 11:46:49 -08:00
logo
pkg Merge pull request #58107 from ironcladlou/quota-controller-deadlock 2018-01-10 15:59:50 -08:00
plugin Fix scheduler refs in BUILD files. 2018-01-05 15:05:01 -08:00
staging Merge pull request #58091 from deads2k/client-01-scale 2018-01-10 12:37:41 -08:00
test Merge pull request #54230 from jsafrane/ceph-test-sleep 2018-01-10 13:19:33 -08:00
third_party
translations
vendor Merge pull request #57989 from NickrenREN/spec-0.1 2018-01-09 00:27:22 -08:00
.bazelrc
.generated_files
.gitattributes
.gitignore
.kazelcfg.json
BUILD.bazel
CHANGELOG-1.2.md
CHANGELOG-1.3.md
CHANGELOG-1.4.md
CHANGELOG-1.5.md
CHANGELOG-1.6.md
CHANGELOG-1.7.md Update CHANGELOG-1.7.md for v1.7.12. 2017-12-29 13:42:49 +01:00
CHANGELOG-1.8.md
CHANGELOG-1.9.md Update CHANGELOG-1.9.md for v1.9.1. 2018-01-04 10:52:59 -08:00
CHANGELOG-1.10.md
CHANGELOG.md
code-of-conduct.md
CONTRIBUTING.md
labels.yaml
LICENSE
Makefile
Makefile.generated_files
OWNERS
OWNERS_ALIASES
README.md
SUPPORT.md
Vagrantfile
WORKSPACE

Kubernetes

Submit Queue Widget GoDoc Widget CII Best Practices


Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.

Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.

Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF). If you are a company that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how Kubernetes plays a role, read the CNCF announcement.


To start using Kubernetes

See our documentation on kubernetes.io.

Try our interactive tutorial.

Take a free course on Scalable Microservices with Kubernetes.

To start developing Kubernetes

The community repository hosts all information about building Kubernetes from source, how to contribute code and documentation, who to contact about what, etc.

If you want to build Kubernetes right away there are two options:

You have a working Go environment.
$ go get -d k8s.io/kubernetes
$ cd $GOPATH/src/k8s.io/kubernetes
$ make
You have a working Docker environment.
$ git clone https://github.com/kubernetes/kubernetes
$ cd kubernetes
$ make quick-release

For the full story, head over to the developer's documentation.

Support

If you need support, start with the troubleshooting guide and work your way through the process that we've outlined.

That said, if you have questions, reach out to us one way or another.

Analytics