test images: Adds building README

Adds a README explaining the image building process, including the
Windows Container image building process.
This commit is contained in:
Claudiu Belu 2019-05-01 21:20:25 -07:00
parent b71fbdb364
commit 91dc590cde
2 changed files with 92 additions and 1 deletions

View File

@ -14,7 +14,6 @@
REGISTRY ?= gcr.io/kubernetes-e2e-test-images
GOARM ?= 7
REMOTE_DOCKER_URL ?=
QEMUVERSION=v2.9.1
GOLANG_VERSION=1.13.6
export

View File

@ -18,6 +18,75 @@ is recommended in order to avoid certain issues.
The node must be able to push the images to the desired container registry, make sure you are
authenticated with the registry you're pushing to.
Windows Container images are not built by default, since they cannot be built on Linux. For
that, a Windows node with Docker installed and configured for remote management is required.
### Windows node(s) setup
In order to build the Windows container images, a node with Windows 10 or Windows Server 2019
with the latest updates installed is required. The node will have to have Docker installed,
preferably version 18.06.0 or newer.
Keep in mind that the Windows node might not be able to build container images for newer OS versions
than itself (even with `--isolation=hyperv`), so keeping the node up to date and / or upgrading it
to the latest Windows Server edition is ideal.
Windows test images must be built for Windows Server 2019 (1809) and Windows Server 1903, thus,
if the node does not have Hyper-V enabled, or it is not supported, multiple Windows nodes are required,
one per OS version.
Additionally, remote management must be configured for the node's Docker daemon. Exposing the
Docker daemon without requiring any authentication is not recommended, and thus, it must be
configured with TLS to ensure that only authorised people can interact with it. For this, the
following `powershell` script can be executed:
```powershell
mkdir .docker
docker run --isolation=hyperv --user=ContainerAdministrator --rm `
-e SERVER_NAME=$(hostname) `
-e IP_ADDRESSES=127.0.0.1,YOUR_WINDOWS_BUILD_NODE_IP `
-v "c:\programdata\docker:c:\programdata\docker" `
-v "$env:USERPROFILE\.docker:c:\users\containeradministrator\.docker" stefanscherer/dockertls-windows:2.5.5
# restart the Docker daemon.
Restart-Service docker
```
For more information about the above commands, you can check [here](https://hub.docker.com/r/stefanscherer/dockertls-windows/).
A firewall rule to allow connections to the Docker daemon is necessary:
```powershell
New-NetFirewallRule -DisplayName 'Docker SSL Inbound' -Profile @('Domain', 'Public', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort 2376
```
If your Windows build node is hosted by a cloud provider, make sure the port `2376` is open for the node.
For example, in Azure, this is done by running the following command:
```console
az vm open-port -g GROUP-NAME -n NODE-NAME --port 2376
```
The `ca.pem`, `cert.pem`, and `key.pem` files that can be found in `$env:USERPROFILE\.docker`
will have to copied to the `~/.docker-${os_version)/` on the Linux build node, where `${os_version}`
is `1809` or `1903`.
```powershell
scp.exe -r $env:USERPROFILE\.docker ubuntu@YOUR_LINUX_BUILD_NODE:/home/ubuntu/.docker-$os_version
```
After all this, the Linux build node should be able to connect to the Windows build node:
```bash
docker --tlsverify --tlscacert ~/.docker-${os_version}/ca.pem --tlscert ~/.docker-${os_version}/cert.pem --tlskey ~/.docker-${os_version}/key.pem -H "$REMOTE_DOCKER_URL" version
```
For more information and troubleshooting about enabling Docker remote management, see
[here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost)
Finally, the node must be able to push the images to the desired container registry, make sure you are
authenticated with the registry you're pushing to.
## Making changes to images
@ -63,6 +132,9 @@ For this, you will need the image manifest list's digest, which can be obtained
manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest'
```
The images are built through `make`. Since some images (e.g.: `busybox`) are used as a base for
other images, it is recommended to build them first, if needed.
## Building images
@ -88,6 +160,14 @@ registry. That can changed by running this command instead:
REGISTRY=foo_registry make all-push WHAT=agnhost
```
In order to also include Windows Container images into the final manifest lists, the `REMOTE_DOCKER_URL` argument
in the form `tcp://[host]:[port][path]` (for more details, see [here]([https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option]/))
will also have to be specified:
```bash
REMOTE_DOCKER_URL_1909=remote_docker_url_1909 REMOTE_DOCKER_URL_1903=remote_docker_url_1903 REMOTE_DOCKER_URL_1809=remote_docker_url_1809 REGISTRY=foo_registry make all-push WHAT=test-webserver
```
*NOTE* (for test `gcr.io` image publishers): Some tests (e.g.: `should serve a basic image on each replica with a private image`)
require the `agnhost` image to be published in an authenticated repo as well:
@ -135,3 +215,15 @@ After all the above has been done, run the desired tests.
```bash
sudo chmod o+x /etc/docker
```
`nc` is being used by some E2E tests, which is why we are including a Linux-like `nc.exe` into the Windows `busybox` image. The image could fail to build during that step with an error that looks like this:
```console
re-exec error: exit status 1: output: time="..." level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) path=\\\\?\\C:\\ProgramData\\...
```
The issue is caused by the Windows Defender which is removing the `nc.exe` binary from the filesystem. For more details on this issue, see [here](https://github.com/diegocr/netcat/issues/6). To fix this, you can simply run the following powershell command to temporarily disable Windows Defender:
```powershell
Set-MpPreference -DisableRealtimeMonitoring $true
```