github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-07-23 19:56:01 +00:00
Code Issues Projects Releases Wiki Activity

Fire an event when failing to open NodePort

Browse Source 
[issue]
When creating a NodePort service with the kubectl create command, the NodePort
assignment may fail.

Failure to assign a NodePort can be simulated with the following malicious
command[1].

$ kubectl create service nodeport temp-svc --tcp=`python3 <<EOF
print("1", end="")
for i in range(2, 1026):
  print("," + str(i), end="")
EOF
`

The command succeeds and shows following output.

service/temp-svc created

The service has been successfully generated and can also be referenced with the
get command.

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)
temp-svc     NodePort    10.0.0.139   <none>        1:31335/TCP,2:32367/TCP,3:30263/TCP,(omitted),1023:31821/TCP,1024:32475/TCP,1025:30311/TCP   12s

The user does not recognize failure to assign a NodePort because
create/get/describe command does not show any error. This is the issue.

[solution]
Users can notice errors by looking at the kube-proxy logs, but it may be difficult to see the kube-proxy logs of all nodes.

E0327 08:50:10.216571  660960 proxier.go:1286] "can't open port, skipping this nodePort" err="listen tcp4 :30641: socket: too many open files" port="\"nodePort for default/temp-svc:744\" (:30641/tcp4)"
E0327 08:50:10.216611  660960 proxier.go:1286] "can't open port, skipping this nodePort" err="listen tcp4 :30827: socket: too many open files" port="\"nodePort for default/temp-svc:857\" (:30827/tcp4)"
...
E0327 08:50:10.217119  660960 proxier.go:1286] "can't open port, skipping this nodePort" err="listen tcp4 :32484: socket: too many open files" port="\"nodePort for default/temp-svc:805\" (:32484/tcp4)"
E0327 08:50:10.217293  660960 proxier.go:1612] "Failed to execute iptables-restore" err="pipe2: too many open files ()"
I0327 08:50:10.217341  660960 proxier.go:1615] "Closing local ports after iptables-restore failure"

So, this patch will fire an event when NodePort assignment fails.
In fact, when the externalIP assignment fails, it is also notified by event.

The event will be displayed like this.

$ kubectl get event
LAST SEEN   TYPE      REASON                                            OBJECT           MESSAGE
...
2s          Warning   listen tcp4 :31055: socket: too many open files   node/127.0.0.1   can't open "nodePort for default/temp-svc:901" (:31055/tcp4), skipping this nodePort: listen tcp4 :31055: socket: too many open files
2s          Warning   listen tcp4 :31422: socket: too many open files   node/127.0.0.1   can't open "nodePort for default/temp-svc:474" (:31422/tcp4), skipping this nodePort: listen tcp4 :31422: socket: too many open files
...

This PR fixes iptables and ipvs proxier.
Since userspace proxier does not seem to be affected by this issue, it is not fixed.

[1] Assume that fd limit is 1024(default).
$ ulimit -n
1024
This commit is contained in:
Masashi Honma 2021-03-27 17:51:07 +09:00
parent a651804427
commit 3266136c1d
2 changed files with 18 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
pkg/proxy/iptables/proxier.go
View File

@ -1283,6 +1283,15 @@ func (proxier *Proxier) syncProxyRules() {
} else if svcInfo.Protocol() != v1.ProtocolSCTP {
socket, err := proxier.portMapper.OpenLocalPort(&lp)
if err != nil {
msg := fmt.Sprintf("can't open %s, skipping this nodePort: %v", lp.String(), err)
proxier.recorder.Eventf(
&v1.ObjectReference{
Kind: "Node",
Name: proxier.hostname,
UID: types.UID(proxier.hostname),
Namespace: "",
}, v1.EventTypeWarning, err.Error(), msg)
klog.ErrorS(err, "can't open port, skipping this nodePort", "port", lp.String())
continue
}

9
pkg/proxy/ipvs/proxier.go
View File

@ -1466,6 +1466,15 @@ func (proxier *Proxier) syncProxyRules() {
} else if svcInfo.Protocol() != v1.ProtocolSCTP {
socket, err := proxier.portMapper.OpenLocalPort(&lp)
if err != nil {
msg := fmt.Sprintf("can't open %s, skipping this nodePort: %v", lp.String(), err)
proxier.recorder.Eventf(
&v1.ObjectReference{
Kind: "Node",
Name: proxier.hostname,
UID: types.UID(proxier.hostname),
Namespace: "",
}, v1.EventTypeWarning, err.Error(), msg)
klog.Errorf("can't open %s, skipping this nodePort: %v", lp.String(), err)
continue
}