mirror of
https://github.com/k3s-io/kubernetes.git
synced 2025-07-30 15:05:27 +00:00
test images: Adds building README
Adds a README explaining the image building process, including the Windows Container image building process.
This commit is contained in:
parent
b71fbdb364
commit
91dc590cde
@ -14,7 +14,6 @@
|
||||
|
||||
REGISTRY ?= gcr.io/kubernetes-e2e-test-images
|
||||
GOARM ?= 7
|
||||
REMOTE_DOCKER_URL ?=
|
||||
QEMUVERSION=v2.9.1
|
||||
GOLANG_VERSION=1.13.6
|
||||
export
|
||||
|
@ -18,6 +18,75 @@ is recommended in order to avoid certain issues.
|
||||
The node must be able to push the images to the desired container registry, make sure you are
|
||||
authenticated with the registry you're pushing to.
|
||||
|
||||
Windows Container images are not built by default, since they cannot be built on Linux. For
|
||||
that, a Windows node with Docker installed and configured for remote management is required.
|
||||
|
||||
|
||||
### Windows node(s) setup
|
||||
|
||||
In order to build the Windows container images, a node with Windows 10 or Windows Server 2019
|
||||
with the latest updates installed is required. The node will have to have Docker installed,
|
||||
preferably version 18.06.0 or newer.
|
||||
|
||||
Keep in mind that the Windows node might not be able to build container images for newer OS versions
|
||||
than itself (even with `--isolation=hyperv`), so keeping the node up to date and / or upgrading it
|
||||
to the latest Windows Server edition is ideal.
|
||||
|
||||
Windows test images must be built for Windows Server 2019 (1809) and Windows Server 1903, thus,
|
||||
if the node does not have Hyper-V enabled, or it is not supported, multiple Windows nodes are required,
|
||||
one per OS version.
|
||||
|
||||
Additionally, remote management must be configured for the node's Docker daemon. Exposing the
|
||||
Docker daemon without requiring any authentication is not recommended, and thus, it must be
|
||||
configured with TLS to ensure that only authorised people can interact with it. For this, the
|
||||
following `powershell` script can be executed:
|
||||
|
||||
```powershell
|
||||
mkdir .docker
|
||||
docker run --isolation=hyperv --user=ContainerAdministrator --rm `
|
||||
-e SERVER_NAME=$(hostname) `
|
||||
-e IP_ADDRESSES=127.0.0.1,YOUR_WINDOWS_BUILD_NODE_IP `
|
||||
-v "c:\programdata\docker:c:\programdata\docker" `
|
||||
-v "$env:USERPROFILE\.docker:c:\users\containeradministrator\.docker" stefanscherer/dockertls-windows:2.5.5
|
||||
# restart the Docker daemon.
|
||||
Restart-Service docker
|
||||
```
|
||||
|
||||
For more information about the above commands, you can check [here](https://hub.docker.com/r/stefanscherer/dockertls-windows/).
|
||||
|
||||
A firewall rule to allow connections to the Docker daemon is necessary:
|
||||
|
||||
```powershell
|
||||
New-NetFirewallRule -DisplayName 'Docker SSL Inbound' -Profile @('Domain', 'Public', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort 2376
|
||||
```
|
||||
|
||||
If your Windows build node is hosted by a cloud provider, make sure the port `2376` is open for the node.
|
||||
For example, in Azure, this is done by running the following command:
|
||||
|
||||
```console
|
||||
az vm open-port -g GROUP-NAME -n NODE-NAME --port 2376
|
||||
```
|
||||
|
||||
The `ca.pem`, `cert.pem`, and `key.pem` files that can be found in `$env:USERPROFILE\.docker`
|
||||
will have to copied to the `~/.docker-${os_version)/` on the Linux build node, where `${os_version}`
|
||||
is `1809` or `1903`.
|
||||
|
||||
```powershell
|
||||
scp.exe -r $env:USERPROFILE\.docker ubuntu@YOUR_LINUX_BUILD_NODE:/home/ubuntu/.docker-$os_version
|
||||
```
|
||||
|
||||
After all this, the Linux build node should be able to connect to the Windows build node:
|
||||
|
||||
```bash
|
||||
docker --tlsverify --tlscacert ~/.docker-${os_version}/ca.pem --tlscert ~/.docker-${os_version}/cert.pem --tlskey ~/.docker-${os_version}/key.pem -H "$REMOTE_DOCKER_URL" version
|
||||
```
|
||||
|
||||
For more information and troubleshooting about enabling Docker remote management, see
|
||||
[here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost)
|
||||
|
||||
Finally, the node must be able to push the images to the desired container registry, make sure you are
|
||||
authenticated with the registry you're pushing to.
|
||||
|
||||
|
||||
## Making changes to images
|
||||
|
||||
@ -63,6 +132,9 @@ For this, you will need the image manifest list's digest, which can be obtained
|
||||
manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest'
|
||||
```
|
||||
|
||||
The images are built through `make`. Since some images (e.g.: `busybox`) are used as a base for
|
||||
other images, it is recommended to build them first, if needed.
|
||||
|
||||
|
||||
## Building images
|
||||
|
||||
@ -88,6 +160,14 @@ registry. That can changed by running this command instead:
|
||||
REGISTRY=foo_registry make all-push WHAT=agnhost
|
||||
```
|
||||
|
||||
In order to also include Windows Container images into the final manifest lists, the `REMOTE_DOCKER_URL` argument
|
||||
in the form `tcp://[host]:[port][path]` (for more details, see [here]([https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option]/))
|
||||
will also have to be specified:
|
||||
|
||||
```bash
|
||||
REMOTE_DOCKER_URL_1909=remote_docker_url_1909 REMOTE_DOCKER_URL_1903=remote_docker_url_1903 REMOTE_DOCKER_URL_1809=remote_docker_url_1809 REGISTRY=foo_registry make all-push WHAT=test-webserver
|
||||
```
|
||||
|
||||
*NOTE* (for test `gcr.io` image publishers): Some tests (e.g.: `should serve a basic image on each replica with a private image`)
|
||||
require the `agnhost` image to be published in an authenticated repo as well:
|
||||
|
||||
@ -135,3 +215,15 @@ After all the above has been done, run the desired tests.
|
||||
```bash
|
||||
sudo chmod o+x /etc/docker
|
||||
```
|
||||
|
||||
`nc` is being used by some E2E tests, which is why we are including a Linux-like `nc.exe` into the Windows `busybox` image. The image could fail to build during that step with an error that looks like this:
|
||||
|
||||
```console
|
||||
re-exec error: exit status 1: output: time="..." level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) path=\\\\?\\C:\\ProgramData\\...
|
||||
```
|
||||
|
||||
The issue is caused by the Windows Defender which is removing the `nc.exe` binary from the filesystem. For more details on this issue, see [here](https://github.com/diegocr/netcat/issues/6). To fix this, you can simply run the following powershell command to temporarily disable Windows Defender:
|
||||
|
||||
```powershell
|
||||
Set-MpPreference -DisableRealtimeMonitoring $true
|
||||
```
|
||||
|
Loading…
Reference in New Issue
Block a user