github/kubernetes
1
0
Fork 0
mirror of https://github.com/k3s-io/kubernetes.git synced 2025-07-22 11:21:47 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #97457 from claudiubelu/test-images-readme-updates

Browse Source 
test images: Updates Image Builder process in the README
This commit is contained in:
Kubernetes Prow Robot 2021-02-04 09:52:47 -08:00 committed by GitHub
commit aacd157c2e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 113 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
test/images/README.md
View File

@ -54,26 +54,33 @@ to build the image and push it to your own registry first, and run some tests th
For these steps, see the sections below. For these steps, see the sections below.
After the desired changes have been made, the affected images will have to be built and published, After the desired changes have been made, the affected images will have to be built and published,
and then tested. After the pull request with those changes has been approved, the new images will be and then tested. After the pull request with those changes has been approved and merged, a postsubmit
built and published to the `gcr.io/kubernetes-e2e-test-images` registry as well. job will then be triggered which will build the images that were changed (for example, if a change was
made in `test/images/agnhost`, then the job [post-kubernetes-push-e2e-agnhost-test-images](
Currently, the image building process has been automated with the Image Promoter, but *only* for the https://testgrid.k8s.io/sig-testing-images#post-kubernetes-push-e2e-agnhost-test-images)
Conformance images (`agnhost`, `jessie-dnsutils`, `kitten`, `nautilus`, `nonewprivs`, `resource-consumer`, will be triggered). The image will then be pushed to the `gcr.io/k8s-staging-e2e-test-images` registry,
`sample-apiserver`). After the pull request merges, a postsubmit job will be started with the new changes, from which it can then promoted to the `k8s.gcr.io/e2e-test-images` registry by adding a line in
which can be tracked [here](https://testgrid.k8s.io/sig-testing-images#post-kubernetes-push-images). [kubernetes/k8s.io](https://github.com/kubernetes/k8s.io/blob/master/k8s.gcr.io/images/k8s-staging-e2e-test-images/images.yaml). For this, you will need the image manifest list's digest, which can be obtained by running:
After it passes successfully, the new image will reside in the `gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION}`
registry, from which it will have to be promoted by adding a line for it
[here](https://github.com/kubernetes/k8s.io/blob/master/k8s.gcr.io/images/k8s-staging-e2e-test-images/images.yaml).
For this, you will need the image manifest list's digest, which can be obtained by running:
```bash ```bash
manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest' manifest-tool inspect --raw gcr.io/k8s-staging-e2e-test-images/${IMAGE_NAME}:${VERSION} | jq '.[0].Digest'
``` ```
All the postsubmit jobs and their logs for all the images can be seen in
[testgrid](https://testgrid.k8s.io/sig-testing-images).
The images are built through `make`. Since some images (e.g.: `busybox`) are used as a base for The images are built through `make`. Since some images (e.g.: `busybox`) are used as a base for
other images, it is recommended to build them first, if needed. other images, it is recommended to build them first, if needed.
### Creating and promoting new images
If you intend to add an entirely different image and have it automatically built by the Image Builder
and used in E2E tests, you will also have to define the postsubmit prow job for it. This can easily
be done by running [this script](https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/k8s-staging-e2e-test-images.sh)
in `kubernetes/test-infra`.
### Windows test images considerations ### Windows test images considerations
Ideally, the same `Dockerfile` can be used to build both Windows and Linux images. However, that isn't Ideally, the same `Dockerfile` can be used to build both Windows and Linux images. However, that isn't
@ -105,6 +112,17 @@ limitations when it comes to the Windows images:
(make sure that process isolation is used, not Hyper-V isolation). (make sure that process isolation is used, not Hyper-V isolation).
[This](https://stefanscherer.github.io/find-dependencies-in-windows-containers/) is a useful guide on how to use `procmon.exe`. [This](https://stefanscherer.github.io/find-dependencies-in-windows-containers/) is a useful guide on how to use `procmon.exe`.
Because of the docker buildx limitations regarding the `RUN` commands mentioned above, we need to use a
Windows helper image in order to properly build the Windows images without requiring any Windows nodes in
the regular image building process. This helper image can be found in at `e2eteam/powershell-helper:6.2.7`.
It can be used by anyone, but if you need to build your own, you can read more about it [here](windows/README.md).
For Windows, in order to spawn process-isolated containers, the container OS version should closely match
the host OS version. For this reason, we build test images for different Windows OS Versions: 1809, 1903,
1909, 2004, 20H2. In order add support for a new Windows OS version, a new entry for that OS version will have
to be first added to the `windows-servercore-cache` and `busybox` images, followed by the rest of the images.
These images are then used by the rest of the E2E test images as a cache / base image.
## Building images ## Building images
@ -138,6 +156,9 @@ REGISTRY=gcr.io/kubernetes-e2e-test-images make all-push WHAT=agnhost
REGISTRY=gcr.io/k8s-authenticated-test make all-push WHAT=agnhost REGISTRY=gcr.io/k8s-authenticated-test make all-push WHAT=agnhost
``` ```
Additionally, `WHAT=all-conformance` can be used to build / push the images most commonly used in E2E
Conformance tests.
## Testing the new image ## Testing the new image
@ -155,6 +176,7 @@ to pull the images from. Sample file:
```yaml ```yaml
dockerLibraryRegistry: your-awesome-registry dockerLibraryRegistry: your-awesome-registry
e2eRegistry: your-awesome-registry e2eRegistry: your-awesome-registry
promoterE2eRegistry: your-awesome-registry
gcRegistry: your-awesome-registry gcRegistry: your-awesome-registry
sampleRegistry: your-awesome-registry sampleRegistry: your-awesome-registry
``` ```

80
test/images/windows/README.md Normal file
View File

@ -0,0 +1,80 @@
# Windows image builder helpers
Currently, the image building process implies using a Linux node with docker buildx, which
allows us to build multiarch images, as well as building Windows images. An additional benefit
to this approach is that we wouldn't have to create a new Windows node for each new Windows
release (you cannot spawn containers with an OS version newer than the host OS version),
simplifying the building process.
However, there are few constraints when it comes to building Windows images using docker buildx:
you cannot use any `RUN` or `WORKDIR` instructions in the Windows stage in the image Dockerfile,
but there are a few cases in which we need to `RUN` some commands. As a workaround to this, we
can simply build some helper images using a Windows docker node, publish it, and then use them
in the regular Image Builder which will only have to use a Linux node. This folder contains
such helper images.
## Windows node(s) setup
In order to build your own helper images, a node with Windows 10 or Windows Server 2019
with the latest updates installed is required. The node will have to have Docker installed,
preferably version 18.06.0 or newer.
Remote management must be configured for the node's Docker daemon. Exposing the Docker daemon
without requiring any authentication is not recommended, and thus, it must be configured with
TLS to ensure that only authorised users can interact with it. For this, the following
`powershell` script can be executed:
```powershell
mkdir .docker
docker run --isolation=hyperv --user=ContainerAdministrator --rm `
-e SERVER_NAME=$(hostname) `
-e IP_ADDRESSES=127.0.0.1,YOUR_WINDOWS_BUILD_NODE_IP `
-v "c:\programdata\docker:c:\programdata\docker" `
-v "$env:USERPROFILE\.docker:c:\users\containeradministrator\.docker" stefanscherer/dockertls-windows:2.5.5
# restart the Docker daemon.
Restart-Service docker
```
For more information about the above commands, you can check [here](https://hub.docker.com/r/stefanscherer/dockertls-windows/).
A firewall rule to allow connections to the Docker daemon is necessary:
```powershell
New-NetFirewallRule -DisplayName 'Docker SSL Inbound' -Profile @('Domain', 'Public', 'Private') -Direction Inbound -Action Allow -Protocol TCP -LocalPort 2376
```
If your Windows build node is hosted by a cloud provider, make sure the port `2376` is open for the node.
For example, in Azure, this is done by running the following command:
```console
az vm open-port -g GROUP-NAME -n NODE-NAME --port 2376
```
The `ca.pem`, `cert.pem`, and `key.pem` files that can be found in `$env:USERPROFILE\.docker`
will have to copied to the `~/.docker/` on the Linux build node.
```powershell
scp.exe $env:USERPROFILE\.docker\*.pem ubuntu@YOUR_LINUX_BUILD_NODE:/home/ubuntu/.docker/
```
After all this, the Linux build node should be able to connect to the Windows build node:
```bash
docker --tlsverify --tlscacert ~/.docker/ca.pem --tlscert ~/.docker/cert.pem --tlskey ~/.docker/key.pem -H "$REMOTE_DOCKER_URL" version
```
For more information and troubleshooting about enabling Docker remote management, see
[here](https://docs.microsoft.com/en-us/virtualization/windowscontainers/management/manage_remotehost)
Finally, the node must be able to push the images to the desired container registry, make sure you are
authenticated with the registry you're pushing to.
## Building images
The images are built through `make`:
```bash
make REGISTRY=foo_registry REMOTE_DOCKER_URL=$REMOTE_DOCKER_URL all-push
```