mirror of
https://github.com/k3s-io/kubernetes.git
synced 2025-08-01 07:47:56 +00:00
test images: Adds building README
Adds a README explaining the image building process, including the Windows Container image building process.
This commit is contained in:
parent
b71fbdb364
commit
91dc590cde
@ -14,7 +14,6 @@
|
|||||||
|
|
||||||
REGISTRY ?= gcr.io/kubernetes-e2e-test-images
|
REGISTRY ?= gcr.io/kubernetes-e2e-test-images
|
||||||
GOARM ?= 7
|
GOARM ?= 7
|
||||||
REMOTE_DOCKER_URL ?=
|
|
||||||
QEMUVERSION=v2.9.1
|
QEMUVERSION=v2.9.1
|
||||||
GOLANG_VERSION=1.13.6
|
GOLANG_VERSION=1.13.6
|
||||||
export
|
export
|
||||||
|
@ -18,6 +18,75 @@ is recommended in order to avoid certain issues.
|
|||||||
The node must be able to push the images to the desired container registry, make sure you are
|
The node must be able to push the images to the desired container registry, make sure you are
|
||||||
authenticated with the registry you're pushing to.
|
authenticated with the registry you're pushing to.
|
||||||
|
|
||||||
|
Windows Container images are not built by default, since they cannot be built on Linux. For
|
||||||
|
that, a Windows node with Docker installed and configured for remote management is required.
|
||||||
|
|
||||||
|
|
||||||
|
### Windows node(s) setup
|
||||||
|
|
||||||
|
In order to build the Windows container images, a node with Windows 10 or Windows Server 2019
|
||||||
|
with the latest updates installed is required. The node will have to have Docker installed,
|
||||||
|
preferably version 18.06.0 or newer.
|
||||||
|
|
||||||
|
Keep in mind that the Windows node might not be able to build container images for newer OS versions
|
||||||
|
than itself (even with `--isolation=hyperv`), so keeping the node up to date and / or upgrading it
|
||||||
|
to the latest Windows Server edition is ideal.
|
||||||
|
|
||||||
|
Windows test images must be built for Windows Server 2019 (1809) and Windows Server 1903, thus,
|
||||||
|
if the node does not have Hyper-V enabled, or it is not supported, multiple Windows nodes are required,
|
||||||
|
one per OS version.
|
||||||
|
|
||||||
|
Additionally, remote management must be configured for the node's Docker daemon. Exposing the
|
||||||
|
Docker daemon without requiring any authentication is not recommended, and thus, it must be
|
||||||
|
configured with TLS to ensure that only authorised people can interact with it. For this, the
|
||||||
|
following `powershell` script can be executed:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
mkdir .docker
|
||||||
|
docker run --isolation=hyperv --user=ContainerAdministrator --rm `
|
||||||
|
-e SERVER_NAME=$(hostname) `
|
||||||
|
-e IP_ADDRESSES=127.0.0.1,YOUR_WINDOWS_BUILD_NODE_IP `
|
||||||
|
-v "c:\programdata\docker:c:\programdata\docker" `
|
||||||
|
-v "$env:USERPROFILE\.docker:c:\users\containeradministrator\.docker" stefanscherer/dockertls-windows:2.5.5
|
||||||
|
# restart the Docker daemon.
|
||||||
|
Restart-Service docker
|
||||||
|
```
|
||||||
|
|
||||||
|
For more information about the above commands, you can check [here](https://hub.docker.com/r/stefanscherer/dockertls-windows/).
|
||||||
|
|
||||||
|
A firewall rule to allow connections to the Docker daemon is necessary:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
New-NetFirewallRule -DisplayName 'Docker SSL Inbound' -Profile @('Domain', 'Public', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort 2376
|
||||||
|
```
|
||||||
|
|
||||||
|
If your Windows build node is hosted by a cloud provider, make sure the port `2376` is open for the node.
|
||||||
|
For example, in Azure, this is done by running the following command:
|
||||||
|
|
||||||
|
```console
|
||||||
|
az vm open-port -g GROUP-NAME -n NODE-NAME --port 2376
|
||||||
|
```
|
||||||
|
|
||||||
|
The `ca.pem`, `cert.pem`, and `key.pem` files that can be found in `$env:USERPROFILE\.docker`
|
||||||
|
will have to copied to the `~/.docker-${os_version)/` on the Linux build node, where `${os_version}`
|
||||||
|
is `1809` or `1903`.
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
scp.exe -r $env:USERPROFILE\.docker ubuntu@YOUR_LINUX_BUILD_NODE:/home/ubuntu/.docker-$os_version
|
||||||
|
```
|
||||||
|
|
||||||
|
After all this, the Linux build node should be able to connect to the Windows build node:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker --tlsverify --tlscacert ~/.docker-${os_version}/ca.pem --tlscert ~/.docker-${os_version}/cert.pem --tlskey ~/.docker-${os_version}/key.pem -H "$REMOTE_DOCKER_URL" version
|
||||||
|
```
|
||||||
|
|
||||||
|
For more information and troubleshooting about enabling Docker remote management, see
|
||||||
|
[here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost)
|
||||||
|
|
||||||
|
Finally, the node must be able to push the images to the desired container registry, make sure you are
|
||||||
|
authenticated with the registry you're pushing to.
|
||||||
|
|
||||||
|
|
||||||
## Making changes to images
|
## Making changes to images
|
||||||
|
|
||||||
@ -63,6 +132,9 @@ For this, you will need the image manifest list's digest, which can be obtained
|
|||||||
manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest'
|
manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The images are built through `make`. Since some images (e.g.: `busybox`) are used as a base for
|
||||||
|
other images, it is recommended to build them first, if needed.
|
||||||
|
|
||||||
|
|
||||||
## Building images
|
## Building images
|
||||||
|
|
||||||
@ -88,6 +160,14 @@ registry. That can changed by running this command instead:
|
|||||||
REGISTRY=foo_registry make all-push WHAT=agnhost
|
REGISTRY=foo_registry make all-push WHAT=agnhost
|
||||||
```
|
```
|
||||||
|
|
||||||
|
In order to also include Windows Container images into the final manifest lists, the `REMOTE_DOCKER_URL` argument
|
||||||
|
in the form `tcp://[host]:[port][path]` (for more details, see [here]([https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option]/))
|
||||||
|
will also have to be specified:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
REMOTE_DOCKER_URL_1909=remote_docker_url_1909 REMOTE_DOCKER_URL_1903=remote_docker_url_1903 REMOTE_DOCKER_URL_1809=remote_docker_url_1809 REGISTRY=foo_registry make all-push WHAT=test-webserver
|
||||||
|
```
|
||||||
|
|
||||||
*NOTE* (for test `gcr.io` image publishers): Some tests (e.g.: `should serve a basic image on each replica with a private image`)
|
*NOTE* (for test `gcr.io` image publishers): Some tests (e.g.: `should serve a basic image on each replica with a private image`)
|
||||||
require the `agnhost` image to be published in an authenticated repo as well:
|
require the `agnhost` image to be published in an authenticated repo as well:
|
||||||
|
|
||||||
@ -135,3 +215,15 @@ After all the above has been done, run the desired tests.
|
|||||||
```bash
|
```bash
|
||||||
sudo chmod o+x /etc/docker
|
sudo chmod o+x /etc/docker
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`nc` is being used by some E2E tests, which is why we are including a Linux-like `nc.exe` into the Windows `busybox` image. The image could fail to build during that step with an error that looks like this:
|
||||||
|
|
||||||
|
```console
|
||||||
|
re-exec error: exit status 1: output: time="..." level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) path=\\\\?\\C:\\ProgramData\\...
|
||||||
|
```
|
||||||
|
|
||||||
|
The issue is caused by the Windows Defender which is removing the `nc.exe` binary from the filesystem. For more details on this issue, see [here](https://github.com/diegocr/netcat/issues/6). To fix this, you can simply run the following powershell command to temporarily disable Windows Defender:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
Set-MpPreference -DisableRealtimeMonitoring $true
|
||||||
|
```
|
||||||
|
Loading…
Reference in New Issue
Block a user