Production-Grade Container Scheduling and Management
Go to file
Masashi Honma 3266136c1d Fire an event when failing to open NodePort
[issue]
When creating a NodePort service with the kubectl create command, the NodePort
assignment may fail.

Failure to assign a NodePort can be simulated with the following malicious
command[1].

$ kubectl create service nodeport temp-svc --tcp=`python3 <<EOF
print("1", end="")
for i in range(2, 1026):
  print("," + str(i), end="")
EOF
`

The command succeeds and shows following output.

service/temp-svc created

The service has been successfully generated and can also be referenced with the
get command.

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)
temp-svc     NodePort    10.0.0.139   <none>        1:31335/TCP,2:32367/TCP,3:30263/TCP,(omitted),1023:31821/TCP,1024:32475/TCP,1025:30311/TCP   12s

The user does not recognize failure to assign a NodePort because
create/get/describe command does not show any error. This is the issue.

[solution]
Users can notice errors by looking at the kube-proxy logs, but it may be difficult to see the kube-proxy logs of all nodes.

E0327 08:50:10.216571  660960 proxier.go:1286] "can't open port, skipping this nodePort" err="listen tcp4 :30641: socket: too many open files" port="\"nodePort for default/temp-svc:744\" (:30641/tcp4)"
E0327 08:50:10.216611  660960 proxier.go:1286] "can't open port, skipping this nodePort" err="listen tcp4 :30827: socket: too many open files" port="\"nodePort for default/temp-svc:857\" (:30827/tcp4)"
...
E0327 08:50:10.217119  660960 proxier.go:1286] "can't open port, skipping this nodePort" err="listen tcp4 :32484: socket: too many open files" port="\"nodePort for default/temp-svc:805\" (:32484/tcp4)"
E0327 08:50:10.217293  660960 proxier.go:1612] "Failed to execute iptables-restore" err="pipe2: too many open files ()"
I0327 08:50:10.217341  660960 proxier.go:1615] "Closing local ports after iptables-restore failure"

So, this patch will fire an event when NodePort assignment fails.
In fact, when the externalIP assignment fails, it is also notified by event.

The event will be displayed like this.

$ kubectl get event
LAST SEEN   TYPE      REASON                                            OBJECT           MESSAGE
...
2s          Warning   listen tcp4 :31055: socket: too many open files   node/127.0.0.1   can't open "nodePort for default/temp-svc:901" (:31055/tcp4), skipping this nodePort: listen tcp4 :31055: socket: too many open files
2s          Warning   listen tcp4 :31422: socket: too many open files   node/127.0.0.1   can't open "nodePort for default/temp-svc:474" (:31422/tcp4), skipping this nodePort: listen tcp4 :31422: socket: too many open files
...

This PR fixes iptables and ipvs proxier.
Since userspace proxier does not seem to be affected by this issue, it is not fixed.

[1] Assume that fd limit is 1024(default).
$ ulimit -n
1024
2021-04-01 08:27:51 +09:00
.github
api Merge pull request #99375 from ehashman/probe-kep-2238 2021-03-11 23:10:18 -08:00
build Switch to newer agnhost image 2021-03-30 20:15:36 -04:00
CHANGELOG CHANGELOG: Update directory for v1.21.0-rc.0 release 2021-03-26 22:35:18 +00:00
cluster Fix the containerd service check 2021-03-19 23:35:05 +00:00
cmd Merge pull request #100171 from chenyw1990/fixGlobalFlagChange 2021-03-23 22:08:04 -07:00
docs hack/update-bazel.sh 2021-02-28 15:17:29 -08:00
hack Merge pull request #100550 from dims/add-new-iptables-rule-for-local-up-cluster.sh 2021-03-29 13:24:09 -07:00
LICENSES vendor: cadvisor v0.39.0 2021-03-08 22:10:29 -08:00
logo
pkg Fire an event when failing to open NodePort 2021-04-01 08:27:51 +09:00
plugin generic ephemeral volumes: refresh rbac testdata 2021-03-09 08:24:52 +01:00
staging Merge pull request #100660 from dims/common-auth-plugins-should-always-be-available 2021-03-30 07:11:57 -07:00
test Switch to newer agnhost image 2021-03-30 20:15:36 -04:00
third_party remove unused third_party/intemp 2021-03-08 21:17:37 -08:00
translations hack/update-bazel.sh 2021-02-28 15:17:29 -08:00
vendor Bump klog to 2.8.0, fixing nil panics in KObj 2021-03-12 12:12:00 -08:00
.generated_files
.gitattributes
.gitignore
CHANGELOG.md
code-of-conduct.md
CONTRIBUTING.md
go.mod Bump klog to 2.8.0, fixing nil panics in KObj 2021-03-12 12:12:00 -08:00
go.sum Bump klog to 2.8.0, fixing nil panics in KObj 2021-03-12 12:12:00 -08:00
LICENSE
Makefile
Makefile.generated_files
OWNERS
OWNERS_ALIASES Merge pull request #98191 from cmluciano/cml/signetown 2021-03-09 05:32:59 -08:00
README.md
SECURITY_CONTACTS
SUPPORT.md

Kubernetes (K8s)

GoPkg Widget CII Best Practices


Kubernetes, also known as K8s, is an open source system for managing containerized applications across multiple hosts. It provides basic mechanisms for deployment, maintenance, and scaling of applications.

Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.

Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF). If your company wants to help shape the evolution of technologies that are container-packaged, dynamically scheduled, and microservices-oriented, consider joining the CNCF. For details about who's involved and how Kubernetes plays a role, read the CNCF announcement.


To start using K8s

See our documentation on kubernetes.io.

Try our interactive tutorial.

Take a free course on Scalable Microservices with Kubernetes.

To use Kubernetes code as a library in other applications, see the list of published components. Use of the k8s.io/kubernetes module or k8s.io/kubernetes/... packages as libraries is not supported.

To start developing K8s

The community repository hosts all information about building Kubernetes from source, how to contribute code and documentation, who to contact about what, etc.

If you want to build Kubernetes right away there are two options:

You have a working Go environment.
mkdir -p $GOPATH/src/k8s.io
cd $GOPATH/src/k8s.io
git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make
You have a working Docker environment.
git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make quick-release

For the full story, head over to the developer's documentation.

Support

If you need support, start with the troubleshooting guide, and work your way through the process that we've outlined.

That said, if you have questions, reach out to us one way or another.