github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-09-03 10:17:46 +00:00
Code Issues Projects Releases Wiki Activity

Add streaming command execution & port forwarding

Browse Source 
Add streaming command execution & port forwarding via HTTP connection
upgrades (currently using SPDY).
This commit is contained in:
Andy Goldstein
2015-01-08 15:41:38 -05:00
parent 25d38c175b
commit 5bd0e9ab05
45 changed files with 4439 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files

144
docs/design/command_execution_port_forwarding.md Normal file
View File

@@ -0,0 +1,144 @@
# Container Command Execution & Port Forwarding in Kubernetes
## Abstract
This describes an approach for providing support for:
- executing commands in containers, with stdin/stdout/stderr streams attached
- port forwarding to containers
## Background
There are several related issues/PRs:
- [Support attach](https://github.com/GoogleCloudPlatform/kubernetes/issues/1521)
- [Real container ssh](https://github.com/GoogleCloudPlatform/kubernetes/issues/1513)
- [Provide easy debug network access to services](https://github.com/GoogleCloudPlatform/kubernetes/issues/1863)
- [OpenShift container command execution proposal](https://github.com/openshift/origin/pull/576)
## Motivation
Users and administrators are accustomed to being able to access their systems
via SSH to run remote commands, get shell access, and do port forwarding.
Supporting SSH to containers in Kubernetes is a difficult task. You must
specify a "user" and a hostname to make an SSH connection, and `sshd` requires
real users (resolvable by NSS and PAM). Because a container belongs to a pod,
and the pod belongs to a namespace, you need to specify namespace/pod/container
to uniquely identify the target container. Unfortunately, a
namespace/pod/container is not a real user as far as SSH is concerned. Also,
most Linux systems limit user names to 32 characters, which is unlikely to be
large enough to contain namespace/pod/container. We could devise some scheme to
map each namespace/pod/container to a 32-character user name, adding entries to
`/etc/passwd` (or LDAP, etc.) and keeping those entries fully in sync all the
time. Alternatively, we could write custom NSS and PAM modules that allow the
host to resolve a namespace/pod/container to a user without needing to keep
files or LDAP in sync.
As an alternative to SSH, we are using a multiplexed streaming protocol that
runs on top of HTTP. There are no requirements about users being real users,
nor is there any limitation on user name length, as the protocol is under our
control. The only downside is that standard tooling that expects to use SSH
won't be able to work with this mechanism, unless adapters can be written.
## Constraints and Assumptions
- SSH support is not currently in scope
- CGroup confinement is ultimately desired, but implementing that support is not currently in scope
- SELinux confinement is ultimately desired, but implementing that support is not currently in scope
## Use Cases
- As a user of a Kubernetes cluster, I want to run arbitrary commands in a container, attaching my local stdin/stdout/stderr to the container
- As a user of a Kubernetes cluster, I want to be able to connect to local ports on my computer and have them forwarded to ports in the container
## Process Flow
### Remote Command Execution Flow
1. The client connects to the Kubernetes Master to initiate a remote command execution
request
2. The Master proxies the request to the Kubelet where the container lives
3. The Kubelet executes nsenter + the requested command and streams stdin/stdout/stderr back and forth between the client and the container
### Port Forwarding Flow
1. The client connects to the Kubernetes Master to initiate a remote command execution
request
2. The Master proxies the request to the Kubelet where the container lives
3. The client listens on each specified local port, awaiting local connections
4. The client connects to one of the local listening ports
4. The client notifies the Kubelet of the new connection
5. The Kubelet executes nsenter + socat and streams data back and forth between the client and the port in the container
## Design Considerations
### Streaming Protocol
The current multiplexed streaming protocol used is SPDY. This is not the
long-term desire, however. As soon as there is viable support for HTTP/2 in Go,
we will switch to that.
### Master as First Level Proxy
Clients should not be allowed to communicate directly with the Kubelet for
security reasons. Therefore, the Master is currently the only suggested entry
point to be used for remote command execution and port forwarding. This is not
necessarily desirable, as it means that all remote command execution and port
forwarding traffic must travel through the Master, potentially impacting other
API requests.
In the future, it might make more sense to retrieve an authorization token from
the Master, and then use that token to initiate a remote command execution or
port forwarding request with a load balanced proxy service dedicated to this
functionality. This would keep the streaming traffic out of the Master.
### Kubelet as Backend Proxy
The kubelet is currently responsible for handling remote command execution and
port forwarding requests. Just like with the Master described above, this means
that all remote command execution and port forwarding streaming traffic must
travel through the Kubelet, which could result in a degraded ability to service
other requests.
In the future, it might make more sense to use a separate service on the node.
Alternatively, we could possibly inject a process into the container that only
listens for a single request, expose that process's listening port on the node,
and then issue a redirect to the client such that it would connect to the first
level proxy, which would then proxy directly to the injected process's exposed
port. This would minimize the amount of proxying that takes place.
### Scalability
There are at least 2 different ways to execute a command in a container:
`docker exec` and `nsenter`. While `docker exec` might seem like an easier and
more obvious choice, it has some drawbacks.
#### `docker exec`
We could expose `docker exec` (i.e. have Docker listen on an exposed TCP port
on the node), but this would require proxying from the edge and securing the
Docker API. `docker exec` calls go through the Docker daemon, meaning that all
stdin/stdout/stderr traffic is proxied through the Daemon, adding an extra hop.
Additionally, you can't isolate 1 malicious `docker exec` call from normal
usage, meaning an attacker could initiate a denial of service or other attack
and take down the Docker daemon, or the node itself.
We expect remote command execution and port forwarding requests to be long
running and/or high bandwidth operations, and routing all the streaming data
through the Docker daemon feels like a bottleneck we can avoid.
#### `nsenter`
The implementation currently uses `nsenter` to run commands in containers,
joining the appropriate container namespaces. `nsenter` runs directly on the
node and is not proxied through any single daemon process.
### Security
Authentication and authorization hasn't specifically been tested yet with this
functionality. We need to make sure that users are not allowed to execute
remote commands or do port forwarding to containers they aren't allowed to
access.
Additional work is required to ensure that multiple command execution or port forwarding connections from different clients are not able to see each other's data. This can most likely be achieved via SELinux labeling and unique process contexts.

60
docs/kubectl-exec.md Normal file
View File

@@ -0,0 +1,60 @@
## kubectl exec
Execute a command in a container.
### Synopsis
Execute a command in a container.
Examples:
$ kubectl exec -p 123456-7890 -c ruby-container date
<returns output from running 'date' in ruby-container from pod 123456-7890>
$ kubectl exec -p 123456-7890 -c ruby-container -i -t -- bash -il
<switches to raw terminal mode, sends stdin to 'bash' in ruby-container from
pod 123456-780 and sends stdout/stderr from 'bash' back to the client
kubectl exec -p <pod> -c <container> -- <command> [<args...>]
### Options
```
-c, --container="": Container name
-p, --pod="": Pod name
-i, --stdin=false: Pass stdin to the container
-t, --tty=false: Stdin is a TTY
```
### Options inherrited from parent commands
```
--alsologtostderr=false: log to standard error as well as files
--api-version="": The API version to use when talking to the server
-a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https.
--certificate-authority="": Path to a cert. file for the certificate authority.
--client-certificate="": Path to a client key file for TLS.
--client-key="": Path to a client key file for TLS.
--cluster="": The name of the kubeconfig cluster to use
--context="": The name of the kubeconfig context to use
-h, --help=false: help for kubectl
--insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
--kubeconfig="": Path to the kubeconfig file to use for CLI requests.
--log_backtrace_at=:0: when logging hits line file:N, emit a stack trace
--log_dir=: If non-empty, write log files in this directory
--log_flush_frequency=5s: Maximum number of seconds between log flushes
--logtostderr=true: log to standard error instead of files
--match-server-version=false: Require server version to match client version
--namespace="": If present, the namespace scope for this CLI request.
--password="": Password for basic authentication to the API server.
-s, --server="": The address and port of the Kubernetes API server
--stderrthreshold=2: logs at or above this threshold go to stderr
--token="": Bearer token for authentication to the API server.
--user="": The name of the kubeconfig user to use
--username="": Username for basic authentication to the API server.
--v=0: log level for V logs
--validate=false: If true, use a schema to validate the input before sending it
--vmodule=: comma-separated list of pattern=N settings for file-filtered logging
```
### SEE ALSO
* [kubectl](kubectl.md)

64
docs/kubectl-port-forward.md Normal file
View File

@@ -0,0 +1,64 @@
## kubectl port-forward
Forward 1 or more local ports to a pod.
### Synopsis
Forward 1 or more local ports to a pod.
Examples:
$ kubectl port-forward -p mypod 5000 6000
<listens on ports 5000 and 6000 locally, forwarding data to/from ports 5000
and 6000 in the pod>
$ kubectl port-forward -p mypod 8888:5000
<listens on port 8888 locally, forwarding to 5000 in the pod>
$ kubectl port-forward -p mypod :5000
<listens on a random port locally, forwarding to 5000 in the pod>
$ kubectl port-forward -p mypod 0:5000
<listens on a random port locally, forwarding to 5000 in the pod>
kubectl port-forward -p <pod> [<local port>:]<remote port> [<port>...]
### Options
```
-p, --pod="": Pod name
```
### Options inherrited from parent commands
```
--alsologtostderr=false: log to standard error as well as files
--api-version="": The API version to use when talking to the server
-a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https.
--certificate-authority="": Path to a cert. file for the certificate authority.
--client-certificate="": Path to a client key file for TLS.
--client-key="": Path to a client key file for TLS.
--cluster="": The name of the kubeconfig cluster to use
--context="": The name of the kubeconfig context to use
-h, --help=false: help for kubectl
--insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
--kubeconfig="": Path to the kubeconfig file to use for CLI requests.
--log_backtrace_at=:0: when logging hits line file:N, emit a stack trace
--log_dir=: If non-empty, write log files in this directory
--log_flush_frequency=5s: Maximum number of seconds between log flushes
--logtostderr=true: log to standard error instead of files
--match-server-version=false: Require server version to match client version
--namespace="": If present, the namespace scope for this CLI request.
--password="": Password for basic authentication to the API server.
-s, --server="": The address and port of the Kubernetes API server
--stderrthreshold=2: logs at or above this threshold go to stderr
--token="": Bearer token for authentication to the API server.
--user="": The name of the kubeconfig user to use
--username="": Username for basic authentication to the API server.
--v=0: log level for V logs
--validate=false: If true, use a schema to validate the input before sending it
--vmodule=: comma-separated list of pattern=N settings for file-filtered logging
```
### SEE ALSO
* [kubectl](kubectl.md)

2
docs/kubectl.md
View File

@@ -54,6 +54,8 @@ kubectl
* [kubectl-log](kubectl-log.md)
* [kubectl-rollingupdate](kubectl-rollingupdate.md)
* [kubectl-resize](kubectl-resize.md)
* [kubectl-exec](kubectl-exec.md)
* [kubectl-port-forward](kubectl-port-forward.md)
* [kubectl-run-container](kubectl-run-container.md)
* [kubectl-stop](kubectl-stop.md)
* [kubectl-expose](kubectl-expose.md)

52
docs/man/man1/kubectl-exec.1 Normal file
View File

@@ -0,0 +1,52 @@
.TH "KUBERNETES" "1" " kubernetes User Manuals" "Eric Paris" "Jan 2015" ""
.SH NAME
.PP
kubectl exec \- Execute a command in a container.
.SH SYNOPSIS
.PP
\fBkubectl exec\fP [OPTIONS]
.SH DESCRIPTION
.PP
Execute a command in a container.
Examples:
$ kubectl exec \-p 123456\-7890 \-c ruby\-container date
<returns output from running 'date' in ruby-container from pod 123456-7890>
.PP
$ kubectl exec \-p 123456\-7890 \-c ruby\-container \-i \-t \-\- bash \-il
<switches to raw terminal mode, sends stdin to 'bash' in ruby\-container from
pod 123456\-780 and sends stdout/stderr from 'bash' back to the client
.SH OPTIONS
.PP
\fB\-c\fP, \fB\-\-container\fP=""
Container name
.PP
\fB\-p\fP, \fB\-\-pod\fP=""
Pod name
.PP
\fB\-i\fP, \fB\-\-stdin\fP=false
Pass stdin to the container
.PP
\fB\-t\fP, \fB\-\-tty\fP=false
Stdin is a TTY
.SH SEE ALSO
.PP
\fBkubectl(1)\fP,
.SH HISTORY
.PP
January 2015, Originally compiled by Eric Paris (eparis at redhat dot com) based on the kubernetes source material, but hopefully they have been automatically generated since!

52
docs/man/man1/kubectl-port-forward.1 Normal file
View File

@@ -0,0 +1,52 @@
.TH "KUBERNETES" "1" " kubernetes User Manuals" "Eric Paris" "Jan 2015" ""
.SH NAME
.PP
kubectl port\-forward \- Forward 1 or more local ports to a pod.
.SH SYNOPSIS
.PP
\fBkubectl port\-forward\fP [OPTIONS]
.SH DESCRIPTION
.PP
Forward 1 or more local ports to a pod.
Examples:
$ kubectl port\-forward \-p mypod 5000 6000
<listens on ports 5000 and 6000 locally, forwarding data to/from ports 5000
and 6000 in the pod>
.PP
.RS
.nf
$ kubectl port\-forward \-p mypod 8888:5000
<listens on port 8888 locally, forwarding to 5000 in the pod>
$ kubectl port\-forward \-p mypod :5000
<listens on a random port locally, forwarding to 5000 in the pod>
$ kubectl port\-forward \-p mypod 0:5000
<listens on a random port locally, forwarding to 5000 in the pod>
.fi
.RE
.SH OPTIONS
.PP
\fB\-p\fP, \fB\-\-pod\fP=""
Pod name
.SH SEE ALSO
.PP
\fBkubectl(1)\fP,
.SH HISTORY
.PP
January 2015, Originally compiled by Eric Paris (eparis at redhat dot com) based on the kubernetes source material, but hopefully they have been automatically generated since!

2
docs/man/man1/kubectl.1
View File

@@ -128,7 +128,7 @@ Find more information at
.SH SEE ALSO
.PP
\fBkubectl\-version(1)\fP, \fBkubectl\-proxy(1)\fP, \fBkubectl\-get(1)\fP, \fBkubectl\-describe(1)\fP, \fBkubectl\-create(1)\fP, \fBkubectl\-update(1)\fP, \fBkubectl\-delete(1)\fP, \fBkubectl\-config(1)\fP, \fBkubectl\-namespace(1)\fP, \fBkubectl\-log(1)\fP, \fBkubectl\-rollingupdate(1)\fP, \fBkubectl\-resize(1)\fP, \fBkubectl\-run\-container(1)\fP, \fBkubectl\-stop(1)\fP, \fBkubectl\-expose(1)\fP, \fBkubectl\-label(1)\fP,
\fBkubectl\-version(1)\fP, \fBkubectl\-proxy(1)\fP, \fBkubectl\-get(1)\fP, \fBkubectl\-describe(1)\fP, \fBkubectl\-create(1)\fP, \fBkubectl\-update(1)\fP, \fBkubectl\-delete(1)\fP, \fBkubectl\-config(1)\fP, \fBkubectl\-namespace(1)\fP, \fBkubectl\-log(1)\fP, \fBkubectl\-rollingupdate(1)\fP, \fBkubectl\-resize(1)\fP, \fBkubectl\-exec(1)\fP, \fBkubectl\-port\-forward(1)\fP, \fBkubectl\-run\-container(1)\fP, \fBkubectl\-stop(1)\fP, \fBkubectl\-expose(1)\fP, \fBkubectl\-label(1)\fP,
.SH HISTORY