github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-07-29 14:37:00 +00:00
Code Issues Projects Releases Wiki Activity

Add variable expansion and design doc

Browse Source
This commit is contained in:
Paul Morie 2015-05-12 19:48:29 -04:00
parent d7834f5033
commit e84092f870
3 changed files with 790 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

407
docs/design/expansion.md Normal file
View File

@ -0,0 +1,407 @@
# Variable expansion in pod command, args, and env
## Abstract
A proposal for the expansion of environment variables using a simple `$(var)` syntax.
## Motivation
It is extremely common for users to need to compose environment variables or pass arguments to
their commands using the values of environment variables. Kubernetes should provide a facility for
the 80% cases in order to decrease coupling and the use of workarounds.
## Goals
1. Define the syntax format
2. Define the scoping and ordering of substitutions
3. Define the behavior for unmatched variables
4. Define the behavior for unexpected/malformed input
## Constraints and Assumptions
* This design should describe the simplest possible syntax to accomplish the use-cases
* Expansion syntax will not support more complicated shell-like behaviors such as default values
(viz: `$(VARIABLE_NAME:"default")`), inline substitution, etc.
## Use Cases
1. As a user, I want to compose new environment variables for a container using a substitution
syntax to reference other variables in the container's environment and service environment
variables
1. As a user, I want to substitute environment variables into a container's command
1. As a user, I want to do the above without requiring the container's image to have a shell
1. As a user, I want to be able to specify a default value for a service variable which may
not exist
1. As a user, I want to see an event associated with the pod if an expansion fails (ie, references
variable names that cannot be expanded)
### Use Case: Composition of environment variables
Currently, containers are injected with docker-style environment variables for the services in
their pod's namespace. There are several variables for each service, but users routinely need
to compose URLs based on these variables because there is not a variable for the exact format
they need. Users should be able to build new environment variables with the exact format they need.
Eventually, it should also be possible to turn off the automatic injection of the docker-style
variables into pods and let the users consume the exact information they need via the downward API
and composition.
#### Expanding expanded variables
It should be possible to reference an variable which is itself the result of an expansion, if the
referenced variable is declared in the container's environment prior to the one referencing it.
Put another way -- a container's environment is expanded in order, and expanded variables are
available to subsequent expansions.
### Use Case: Variable expansion in command
Users frequently need to pass the values of environment variables to a container's command.
Currently, Kubernetes does not perform any expansion of varibles. The workaround is to invoke a
shell in the container's command and have the shell perform the substitution, or to write a wrapper
script that sets up the environment and runs the command. This has a number of drawbacks:
1. Solutions that require a shell are unfriendly to images that do not contain a shell
2. Wrapper scripts make it harder to use images as base images
3. Wrapper scripts increase coupling to kubernetes
Users should be able to do the 80% case of variable expansion in command without writing a wrapper
script or adding a shell invocation to their containers' commands.
### Use Case: Images without shells
The current workaround for variable expansion in a container's command requires the container's
image to have a shell. This is unfriendly to images that do not contain a shell (`scratch` images,
for example). Users should be able to perform the other use-cases in this design without regard to
the content of their images.
### Use Case: See an event for incomplete expansions
It is possible that a container with incorrect variable values or command line may continue to run
for a long period of time, and that the end-user would have no visual or obvious warning of the
incorrect configuration. If the kubelet creates an event when an expansion references a variable
that cannot be expanded, it will help users quickly detect problems with expansions.
## Design Considerations
### What features should be supported?
In order to limit complexity, we want to provide the right amount of functionality so that the 80%
cases can be realized and nothing more. We felt that the essentials boiled down to:
1. Ability to perform direct expansion of variables in a string
2. Ability to specify default values via a prioritized mapping function but without support for
defaults as a syntax-level feature
### What should the syntax be?
The exact syntax for variable expansion has a large impact on how users perceive and relate to the
feature. We considered implementing a very restrictive subset of the shell `${var}` syntax. This
syntax is an attractive option on some level, because many people are familiar with it. However,
this syntax also has a large number of lesser known features such as the ability to provide
default values for unset variables, perform inline substitution, etc.
In the interest of preventing conflation of the expansion feature in Kubernetes with the shell
feature, we chose a different syntax similar to the one in Makefiles, `$(var)`. We also chose not
to support the bar `$var` format, since it is not required to implement the required use-cases.
Nested references, ie, variable expansion within variable names, are not supported.
#### How should unmatched references be treated?
Ideally, it should be extremely clear when a variable reference couldn't be expanded. We decided
the best experience for unmatched variable references would be to have the entire reference, syntax
included, show up in the output. As an example, if the reference `$(VARIABLE_NAME)` cannot be
expanded, then `$(VARIABLE_NAME)` should be present in the output.
#### Escaping the operator
Although the `$(var)` syntax does overlap with the `$(command)` form of command substitution
supported by many shells, because unexpanded variables are present verbatim in the output, we
expect this will not present a problem to many users. If there is a collision between a varible
name and command substitution syntax, the syntax can be escaped with the form `$$(VARIABLE_NAME)`,
which will evaluate to `$(VARIABLE_NAME)` whether `VARIABLE_NAME` can be expanded or not.
## Design
This design encompasses the variable expansion syntax and specification and the changes needed to
incorporate the expansion feature into the container's environment and command.
### Syntax and expansion mechanics
This section describes the expansion syntax, evaluation of variable values, and how unexpected or
malformed inputs are handled.
#### Syntax
The inputs to the expansion feature are:
1. A utf-8 string (the input string) which may contain variable references
2. A function (the mapping function) that maps the name of a variable to the variable's value, of
type `func(string) string`
Variable references in the input string are indicated exclusively with the syntax
`$(<variable-name>)`. The syntax tokens are:
- `$`: the operator
- `(`: the reference opener
- `)`: the reference closer
The operator has no meaning unless accompanied by the reference opener and closer tokens. The
operator can be escaped using `$$`. One literal `$` will be emitted for each `$$` in the input.
The reference opener and closer characters have no meaning when not part of a variable reference.
If a variable reference is malformed, viz: `$(VARIABLE_NAME` without a closing expression, the
operator and expression opening characters are treated as ordinary characters without special
meanings.
#### Scope and ordering of substitutions
The scope in which variable references are expanded is defined by the mapping function. Within the
mapping function, any arbitrary strategy may be used to determine the value of a variable name.
The most basic implementation of a mapping function is to use a `map[string]string` to lookup the
value of a variable.
In order to support default values for variables like service variables presented by the kubelet,
which may not be bound because the service that provides them does not yet exist, there should be a
mapping function that uses a list of `map[string]string` like:
```go
func MakeMappingFunc(maps ...map[string]string) func(string) string {
return func(input string) string {
for _, context := range maps {
val, ok := context[input]
if ok {
return val
}
}
return ""
}
}
// elsewhere
containerEnv := map[string]string{
"FOO": "BAR",
"ZOO": "ZAB",
"SERVICE2_HOST": "some-host",
}
serviceEnv := map[string]string{
"SERVICE_HOST": "another-host",
"SERVICE_PORT": "8083",
}
// single-map variation
mapping := MakeMappingFunc(containerEnv)
// default variables not found in serviceEnv
mappingWithDefaults := MakeMappingFunc(serviceEnv, containerEnv)
```
### Implementation changes
The necessary changes to implement this functionality are:
1. Add a new interface, `ObjectEventRecorder`, which is like the `EventRecorder` interface, but
scoped to a single object, and a function that returns an `ObjectEventRecorder` given an
`ObjectReference` and an `EventRecorder`
2. Introduce `third_party/golang/expansion` package that provides:
1. An `Expand(string, func(string) string) string` function
2. A `MappingFuncFor(ObjectEventRecorder, ...map[string]string) string` function
3. Add a new EnvVarSource for expansions and associated tests
4. Make the kubelet expand environment correctly
5. Make the kubelet expand command correctly
#### Event Recording
In order to provide an event when an expansion references undefined variables, the mapping function
must be able to create an event. In order to facilitate this, we should create a new interface in
the `api/client/record` package which is similar to `EventRecorder`, but scoped to a single object:
```go
// ObjectEventRecorder knows how to record events about a single object.
type ObjectEventRecorder interface {
// Event constructs an event from the given information and puts it in the queue for sending.
// 'reason' is the reason this event is generated. 'reason' should be short and unique; it will
// be used to automate handling of events, so imagine people writing switch statements to
// handle them. You want to make that easy.
// 'message' is intended to be human readable.
//
// The resulting event will be created in the same namespace as the reference object.
Event(reason, message string)
// Eventf is just like Event, but with Sprintf for the message field.
Eventf(reason, messageFmt string, args ...interface{})
// PastEventf is just like Eventf, but with an option to specify the event's 'timestamp' field.
PastEventf(timestamp util.Time, reason, messageFmt string, args ...interface{})
}
```
There should also be a function that can construct an `ObjectEventRecorder` from a `runtime.Object`
and an `EventRecorder`:
```go
type objectRecorderImpl struct {
object runtime.Object
recorder EventRecorder
}
func (r *objectRecorderImpl) Event(reason, message string) {
r.recorder.Event(r.object, reason, message)
}
func ObjectEventRecorderFor(object runtime.Object, recorder EventRecorder) ObjectEventRecorder {
return &objectRecorderImpl{object, recorder}
}
```
#### Expansion package
The expansion package should provide two methods:
```go
// MappingFuncFor returns a mapping function for use with Expand that
// implements the expansion semantics defined in the expansion spec; it
// returns the input string wrapped in the expansion syntax if no mapping
// for the input is found. If no expansion is found for a key, an event
// is raised on the given recorder.
func MappingFuncFor(recorder record.ObjectEventRecorder, context ...map[string]string) func(string) string {
// ...
}
// Expand replaces variable references in the input string according to
// the expansion spec using the given mapping function to resolve the
// values of variables.
func Expand(input string, mapping func(string) string) string {
// ...
}
```
#### Expansion `EnvVarSource`
In order to avoid changing the existing behavior of the `EnvVar.Value` field, there should be a new
`EnvVarSource` that represents a variable expansion that an env var's value should come from:
```go
// EnvVarSource represents a source for the value of an EnvVar.
type EnvVarSource struct {
// Other fields omitted
Expansion *EnvVarExpansion
}
type EnvVarExpansion struct {
// The input string to be expanded
Expand string
}
```
#### Kubelet changes
The Kubelet should change to:
1. Correctly expand environment variables with `Expansion` sources
2. Correctly expand references in the Command and Args
### Examples
#### Inputs and outputs
These examples are in the context of the mapping:
| Name | Value |
|-------------|------------|
| `VAR_A` | `"A"` |
| `VAR_B` | `"B"` |
| `VAR_C` | `"C"` |
| `VAR_REF` | `$(VAR_A)` |
| `VAR_EMPTY` | `""` |
No other variables are defined.
| Input | Result |
|--------------------------------|----------------------------|
| `"$(VAR_A)"` | `"A"` |
| `"___$(VAR_B)___"` | `"___B___"` |
| `"___$(VAR_C)"` | `"___C"` |
| `"$(VAR_A)-$(VAR_A)"` | `"A-A"` |
| `"$(VAR_A)-1"` | `"A-1"` |
| `"$(VAR_A)_$(VAR_B)_$(VAR_C)"` | `"A_B_C"` |
| `"$$(VAR_B)_$(VAR_A)"` | `"$(VAR_B)_A"` |
| `"$$(VAR_A)_$$(VAR_B)"` | `"$(VAR_A)_$(VAR_B)"` |
| `"f000-$$VAR_A"` | `"f000-$VAR_A"` |
| `"foo\\$(VAR_C)bar"` | `"foo\Cbar"` |
| `"foo\\\\$(VAR_C)bar"` | `"foo\\Cbar"` |
| `"foo\\\\\\\\$(VAR_A)bar"` | `"foo\\\\Abar"` |
| `"$(VAR_A$(VAR_B))"` | `"$(VAR_A$(VAR_B))"` |
| `"$(VAR_A$(VAR_B)"` | `"$(VAR_A$(VAR_B)"` |
| `"$(VAR_REF)"` | `"$(VAR_A)"` |
| `"%%$(VAR_REF)--$(VAR_REF)%%"` | `"%%$(VAR_A)--$(VAR_A)%%"` |
| `"foo$(VAR_EMPTY)bar"` | `"foobar"` |
| `"foo$(VAR_Awhoops!"` | `"foo$(VAR_Awhoops!"` |
| `"f00__(VAR_A)__"` | `"f00__(VAR_A)__"` |
| `"$?_boo_$!"` | `"$?_boo_$!"` |
| `"$VAR_A"` | `"$VAR_A"` |
| `"$(VAR_DNE)"` | `"$(VAR_DNE)"` |
| `"$$$$$$(BIG_MONEY)"` | `"$$$(BIG_MONEY)"` |
| `"$$$$$$(VAR_A)"` | `"$$$(VAR_A)"` |
| `"$$$$$$$(GOOD_ODDS)"` | `"$$$$(GOOD_ODDS)"` |
| `"$$$$$$$(VAR_A)"` | `"$$$A"` |
| `"$VAR_A)"` | `"$VAR_A)"` |
| `"${VAR_A}"` | `"${VAR_A}"` |
| `"$(VAR_B)_______$(A"` | `"B_______$(A"` |
| `"$(VAR_C)_______$("` | `"C_______$("` |
| `"$(VAR_A)foobarzab$"` | `"Afoobarzab$"` |
| `"foo-\\$(VAR_A"` | `"foo-\$(VAR_A"` |
| `"--$($($($($--"` | `"--$($($($($--"` |
| `"$($($($($--foo$("` | `"$($($($($--foo$("` |
| `"foo0--$($($($("` | `"foo0--$($($($("` |
| `"$(foo$$var)` | `$(foo$$var)` |
#### In a pod: building a URL
Notice the `$(var)` syntax.
```yaml
apiVersion: v1beta3
kind: Pod
metadata:
name: expansion-pod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: PUBLIC_URL
valueFrom:
expansion:
expand: "http://$(GITSERVER_SERVICE_HOST):$(GITSERVER_SERVICE_PORT)"
restartPolicy: Never
```
#### In a pod: building a URL using downward API
```yaml
apiVersion: v1beta3
kind: Pod
metadata:
name: expansion-pod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: "metadata.namespace"
- name: PUBLIC_URL
valueFrom:
expansion:
expand: "http://gitserver.$(POD_NAMESPACE):$(SERVICE_PORT)"
restartPolicy: Never
```
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/design/expansion.md?pixel)]()

98
third_party/golang/expansion/expand.go vendored Normal file
View File

@ -0,0 +1,98 @@
package expansion
const (
operator = '$'
referenceOpener = '('
referenceCloser = ')'
)
// syntaxWrap returns the input string wrapped the expansion syntax.
func syntaxWrap(input string) string {
return string(operator) + string(referenceOpener) + input + string(referenceCloser)
}
// MappingFuncFor returns a mapping function for use with Expand that
// implements the expansion semantics defined in the expansion spec; it
// returns the input string wrapped in the expansion syntax if no mapping
// for the input is found.
func MappingFuncFor(context ...map[string]string) func(string) string {
return func(input string) string {
for _, vars := range context {
val, ok := vars[input]
if ok {
return val
}
}
return syntaxWrap(input)
}
}
// Expand replaces variable references in the input string according to
// the expansion spec using the given mapping function to resolve the
// values of variables.
func Expand(input string, mapping func(string) string) string {
buf := make([]byte, 0, 2*len(input))
checkpoint := 0
for cursor := 0; cursor < len(input); cursor++ {
if input[cursor] == operator && cursor+1 < len(input) {
// Copy the portion of the input string since the last
// checkpoint into the buffer
buf = append(buf, input[checkpoint:cursor]...)
// Attempt to read the variable name as defined by the
// syntax from the input string
read, isVar, advance := tryReadVariableName(input[cursor+1:])
if isVar {
// We were able to read a variable name correctly;
// apply the mapping to the variable name and copy the
// bytes into the buffer
buf = append(buf, mapping(read)...)
} else {
// Not a variable name; copy the read bytes into the buffer
buf = append(buf, read...)
}
// Advance the cursor in the input string to account for
// bytes consumed to read the variable name expression
cursor += advance
// Advance the checkpoint in the input string
checkpoint = cursor + 1
}
}
// Return the buffer and any remaining unwritten bytes in the
// input string.
return string(buf) + input[checkpoint:]
}
// tryReadVariableName attempts to read a variable name from the input
// string and returns the content read from the input, whether that content
// represents a variable name to perform mapping on, and the number of bytes
// consumed in the input string.
//
// The input string is assumed not to contain the initial operator.
func tryReadVariableName(input string) (string, bool, int) {
switch input[0] {
case operator:
// Escaped operator; return it.
return input[0:1], false, 1
case referenceOpener:
// Scan to expression closer
for i := 1; i < len(input); i++ {
if input[i] == referenceCloser {
return input[1:i], true, i + 1
}
}
// Incomplete reference; return it.
return string(operator) + string(referenceOpener), false, 1
default:
// Not the beginning of an expression, ie, an operator
// that doesn't begin an expression. Return the operator
// and the first rune in the string.
return (string(operator) + string(input[0])), false, 1
}
}

285
third_party/golang/expansion/expand_test.go vendored Normal file
View File

@ -0,0 +1,285 @@
package expansion
import (
"testing"
"github.com/GoogleCloudPlatform/kubernetes/pkg/api"
)
func TestMapReference(t *testing.T) {
envs := []api.EnvVar{
{
Name: "FOO",
Value: "bar",
},
{
Name: "ZOO",
Value: "$(FOO)-1",
},
{
Name: "BLU",
Value: "$(ZOO)-2",
},
}
declaredEnv := map[string]string{
"FOO": "bar",
"ZOO": "$(FOO)-1",
"BLU": "$(ZOO)-2",
}
serviceEnv := map[string]string{}
mapping := MappingFuncFor(declaredEnv, serviceEnv)
for _, env := range envs {
declaredEnv[env.Name] = Expand(env.Value, mapping)
}
expectedEnv := map[string]string{
"FOO": "bar",
"ZOO": "bar-1",
"BLU": "bar-1-2",
}
for k, v := range expectedEnv {
if e, a := v, declaredEnv[k]; e != a {
t.Errorf("Expected %v, got %v", e, a)
} else {
delete(declaredEnv, k)
}
}
if len(declaredEnv) != 0 {
t.Errorf("Unexpected keys in declared env: %v", declaredEnv)
}
}
func TestMapping(t *testing.T) {
context := map[string]string{
"VAR_A": "A",
"VAR_B": "B",
"VAR_C": "C",
"VAR_REF": "$(VAR_A)",
"VAR_EMPTY": "",
}
mapping := MappingFuncFor(context)
doExpansionTest(t, mapping)
}
func TestMappingDual(t *testing.T) {
context := map[string]string{
"VAR_A": "A",
"VAR_EMPTY": "",
}
context2 := map[string]string{
"VAR_B": "B",
"VAR_C": "C",
"VAR_REF": "$(VAR_A)",
}
mapping := MappingFuncFor(context, context2)
doExpansionTest(t, mapping)
}
func doExpansionTest(t *testing.T, mapping func(string) string) {
cases := []struct {
name string
input string
expected string
}{
{
name: "whole string",
input: "$(VAR_A)",
expected: "A",
},
{
name: "repeat",
input: "$(VAR_A)-$(VAR_A)",
expected: "A-A",
},
{
name: "beginning",
input: "$(VAR_A)-1",
expected: "A-1",
},
{
name: "middle",
input: "___$(VAR_B)___",
expected: "___B___",
},
{
name: "end",
input: "___$(VAR_C)",
expected: "___C",
},
{
name: "compound",
input: "$(VAR_A)_$(VAR_B)_$(VAR_C)",
expected: "A_B_C",
},
{
name: "escape & expand",
input: "$$(VAR_B)_$(VAR_A)",
expected: "$(VAR_B)_A",
},
{
name: "compound escape",
input: "$$(VAR_A)_$$(VAR_B)",
expected: "$(VAR_A)_$(VAR_B)",
},
{
name: "mixed in escapes",
input: "f000-$$VAR_A",
expected: "f000-$VAR_A",
},
{
name: "backslash escape ignored",
input: "foo\\$(VAR_C)bar",
expected: "foo\\Cbar",
},
{
name: "backslash escape ignored",
input: "foo\\\\$(VAR_C)bar",
expected: "foo\\\\Cbar",
},
{
name: "lots of backslashes",
input: "foo\\\\\\\\$(VAR_A)bar",
expected: "foo\\\\\\\\Abar",
},
{
name: "nested var references",
input: "$(VAR_A$(VAR_B))",
expected: "$(VAR_A$(VAR_B))",
},
{
name: "nested var references second type",
input: "$(VAR_A$(VAR_B)",
expected: "$(VAR_A$(VAR_B)",
},
{
name: "value is a reference",
input: "$(VAR_REF)",
expected: "$(VAR_A)",
},
{
name: "value is a reference x 2",
input: "%%$(VAR_REF)--$(VAR_REF)%%",
expected: "%%$(VAR_A)--$(VAR_A)%%",
},
{
name: "empty var",
input: "foo$(VAR_EMPTY)bar",
expected: "foobar",
},
{
name: "unterminated expression",
input: "foo$(VAR_Awhoops!",
expected: "foo$(VAR_Awhoops!",
},
{
name: "expression without operator",
input: "f00__(VAR_A)__",
expected: "f00__(VAR_A)__",
},
{
name: "shell special vars pass through",
input: "$?_boo_$!",
expected: "$?_boo_$!",
},
{
name: "bare operators are ignored",
input: "$VAR_A",
expected: "$VAR_A",
},
{
name: "undefined vars are passed through",
input: "$(VAR_DNE)",
expected: "$(VAR_DNE)",
},
{
name: "multiple (even) operators, var undefined",
input: "$$$$$$(BIG_MONEY)",
expected: "$$$(BIG_MONEY)",
},
{
name: "multiple (even) operators, var defined",
input: "$$$$$$(VAR_A)",
expected: "$$$(VAR_A)",
},
{
name: "multiple (odd) operators, var undefined",
input: "$$$$$$$(GOOD_ODDS)",
expected: "$$$$(GOOD_ODDS)",
},
{
name: "multiple (odd) operators, var defined",
input: "$$$$$$$(VAR_A)",
expected: "$$$A",
},
{
name: "missing open expression",
input: "$VAR_A)",
expected: "$VAR_A)",
},
{
name: "shell syntax ignored",
input: "${VAR_A}",
expected: "${VAR_A}",
},
{
name: "trailing incomplete expression not consumed",
input: "$(VAR_B)_______$(A",
expected: "B_______$(A",
},
{
name: "trailing incomplete expression, no content, is not consumed",
input: "$(VAR_C)_______$(",
expected: "C_______$(",
},
{
name: "operator at end of input string is preserved",
input: "$(VAR_A)foobarzab$",
expected: "Afoobarzab$",
},
{
name: "shell escaped incomplete expr",
input: "foo-\\$(VAR_A",
expected: "foo-\\$(VAR_A",
},
{
name: "lots of $( in middle",
input: "--$($($($($--",
expected: "--$($($($($--",
},
{
name: "lots of $( in beginning",
input: "$($($($($--foo$(",
expected: "$($($($($--foo$(",
},
{
name: "lots of $( at end",
input: "foo0--$($($($(",
expected: "foo0--$($($($(",
},
{
name: "escaped operators in variable names are not escaped",
input: "$(foo$$var)",
expected: "$(foo$$var)",
},
{
name: "newline not expanded",
input: "\n",
expected: "\n",
},
}
for _, tc := range cases {
expanded := Expand(tc.input, mapping)
if e, a := tc.expected, expanded; e != a {
t.Errorf("%v: expected %q, got %q", tc.name, e, a)
}
}
}