26 Commits

Author SHA1 Message Date
Avi Deitcher
dc8c6d5985 Merge pull request #4089 from deitch/tag-in-build-yml
support --tag in build.yml for packages
2024-12-23 18:00:06 +02:00
Avi Deitcher
4f765b5da0 support --tag in build.yml for packages
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-12-23 17:28:49 +02:00
Avi Deitcher
ad95c6fc2e Merge pull request #4085 from deitch/volume-image
additional volume support in building
2024-10-01 15:57:17 +03:00
Avi Deitcher
76f4802ccf additional volume support in building
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-10-01 15:27:55 +03:00
Avi Deitcher
e4d41061b6 Merge pull request #4084 from deitch/cache-platform-instead-of-arch
internal restructure to use explicit platform instead of implicit arch in cache
2024-10-01 15:14:21 +03:00
Avi Deitcher
81f0c3eff2 internal restructure to use explicit platform instead of implicit arch in cache
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-10-01 14:30:03 +03:00
Avi Deitcher
5e3f7dd9a5 Merge pull request #4083 from deitch/restructure-logging
restructure logging
2024-10-01 14:00:06 +03:00
Avi Deitcher
67e9e22a36 restructure logging
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-10-01 12:50:43 +03:00
Avi Deitcher
8556f024ef Merge pull request #4082 from kolyshkin/moby-cap
vendor: switch to moby/sys/capability
2024-10-01 11:07:29 +03:00
Kir Kolyshkin
da3be29998 vendor: switch to moby/sys/capability
github.com/moby/sys/capability is a fork of the (no longer maintained)
github.com/syndtr/gocapability package.

For changes since the fork took place, see
https://github.com/moby/sys/blob/main/capability/CHANGELOG.md

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2024-09-30 18:10:16 -07:00
Avi Deitcher
d7a6bc8899 Merge pull request #4077 from deitch/docker-bump
bump docker deps to v27.2.0
2024-09-08 13:00:19 +03:00
Avi Deitcher
2159aacb09 bump docker deps to v27.2.0
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-09-08 12:22:57 +03:00
Avi Deitcher
fa3207c86e Merge pull request #4072 from christoph-zededa/docker_cache_consider_architecture
moby: check architecture for docker image
2024-08-29 22:15:19 +03:00
Avi Deitcher
1d6d5fa612 Merge pull request #4074 from deitch/efi-kernel
remove linuxefi grub EFI handover to normal linux loading
2024-08-29 21:13:48 +03:00
Avi Deitcher
ba25e59640 remove linuxefi grub EFI handover to normal linux loading
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-08-29 17:30:57 +03:00
Avi Deitcher
6979859e76 Merge pull request #4073 from deitch/init-debug-no-control
use only stdout/stderr or file for runc output
2024-08-28 15:28:33 +03:00
Avi Deitcher
5848a2856f use only stdout/stderr or file for runc output
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-08-28 14:45:26 +03:00
Christoph Ostarek
cb8f36adf3 moby: check architecture for docker image
under certain cases the container image is already in the local docker
registry, but with the wrong architecture; in this case just pretend
it is not there and let the caller decide if they want to build it

Signed-off-by: Christoph Ostarek <christoph@zededa.com>
2024-08-27 15:49:21 +02:00
Avi Deitcher
5f09346e1e Merge pull request #4070 from deitch/verbose-runc
more verbose runc messages
2024-08-22 20:55:44 +03:00
Avi Deitcher
15c808c4ee more verbose runc messages
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-08-22 20:23:44 +03:00
Avi Deitcher
745da8f4c0 Merge pull request #4069 from deitch/fix-ro-volumes
when building read-only volumes, still use overlayfs
2024-08-22 19:20:35 +03:00
Avi Deitcher
b36cad081b when building read-only volumes, still use overlayfs
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-08-22 18:49:08 +03:00
Avi Deitcher
370bf51cdf Merge pull request #4067 from deitch/runc-debug-options
support cmdline-driven debugging mode for runc
2024-08-22 15:53:46 +03:00
Avi Deitcher
2af30c5503 support cmdline-driven debugging mode for runc
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-08-22 15:14:35 +03:00
Avi Deitcher
270fd1c5aa Merge pull request #4066 from deitch/ssh-support
support for pkg build ssh
2024-07-28 13:32:42 +03:00
Avi Deitcher
51727db254 support for pkg build ssh
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-07-28 11:39:55 +03:00
243 changed files with 9197 additions and 848 deletions

View File

@@ -122,13 +122,13 @@ jobs:
- name: Build Packages
# Skip s390x as emulation is unreliable
run: |
make OPTIONS="-v --skip-platforms linux/s390x" -C pkg build
make OPTIONS="-v 2 --skip-platforms linux/s390x" -C pkg build
- name: Build Test Packages
# ensures that the test packages are in linuxkit cache when we need them for tests later
# Skip s390x as emulation is unreliable
run: |
make OPTIONS="-v --skip-platforms linux/s390x" -C test/pkg build
make OPTIONS="-v 2 --skip-platforms linux/s390x" -C test/pkg build
- name: Check Kernel Dependencies up to date
# checks that any kernel dependencies are up to date.
@@ -145,7 +145,7 @@ jobs:
# ensures that the kernel packages are in linuxkit cache when we need them for tests later
# no need for excluding s390x, as each build.yml in the kernel explicitly lists archs
run: |
make OPTIONS="-v" -C kernel build
make OPTIONS="-v 2" -C kernel build
- name: list cache contents
run: |

19
docs/cmdline.md Normal file
View File

@@ -0,0 +1,19 @@
# Kernel command-line options
The kernel command-line is a string of text that the kernel parses as it is starting up. It is passed by the boot loader
to the kernel and specifies parameters that the kernel uses to configure the system. The command-line is a list of command-line
options separated by spaces. The options are parsed by the kernel and can be used to enable or disable certain features.
LinuxKit passes all command-line options to the kernel, which uses them in the usual way.
There are several options that can be used to control the behaviour of linuxkit itself, or specifically packages
within linuxkit. Unless standard Linux options exist, these all are prefaced with `linuxkit.`.
| Option | Description |
|---|---|
| `linuxkit.unified_cgroup_hierarchy=0` | Start up cgroups v1. If not present or set to 1, default to cgroups v1. |
| `linuxkit.runc_debug=1` | Start runc for `onboot` and `onshutdown` containers to run with `--debug`, and add extra logging messages for each stage of starting those containers. If not present or set to 0, default to usual mode. |
| `linuxkit.runc_console=1` | Send logs for runc for `onboot` and `onshutdown` containers, as well as the output of the containers themselves, to the console, instead of the normal output to logfiles. If not present or set to 0, default to usual mode. |
It often is useful to combine both of the `linuxkit.runc_debug` and `linuxkit.runc_console` options to get the most
information about what is happening with `onboot` containers.

View File

@@ -50,6 +50,7 @@ A package source consists of a directory containing at least two files:
- `image` _(string)_: *(mandatory)* The name of the image to build
- `org` _(string)_: The hub/registry organisation to which this package belongs
- `tag` _(string)_: The tag to use for the image, can be fixed string or template (default: `{{.Hash}}`)
- `dockerfile` _(string)_: The dockerfile to use to build this package, must be in this directory or below (default: `Dockerfile`)
- `arches` _(list of string)_: The architectures which this package should be built for (valid entries are `GOARCH` names)
- `extra-sources` _(list of strings)_: Additional sources for the package outside the package directory. The format is `src:dst`, where `src` can be relative to the package directory and `dst` is the destination in the build context. This is useful for sharing files, such as vendored go code, between packages.

View File

@@ -18,8 +18,17 @@ For private registries or private repositories on a registry credentials provide
## Sections
The configuration file is processed in the order `kernel`, `init`, `onboot`, `onshutdown`,
`services`, `files`, `volumes`. Each section adds files to the root file system. Sections may be omitted.
The configuration file is processed in the order:
1. `kernel`
1. `init`
1. `volumes`
1. `onboot`
1. `onshutdown`
1. `services`
1. `files`
Each section adds files to the root file system. Sections may be omitted.
Each container that is specified is allocated a unique `uid` and `gid` that it may use if it
wishes to run as an isolated user (or user namespace). Anywhere you specify a `uid` or `gid`
@@ -52,6 +61,9 @@ which should contain a `kernel` file that will be booted (eg a `bzImage` for `am
called `kernel.tar` which is a tarball that is unpacked into the root, which should usually
contain a kernel modules directory. `cmdline` specifies the kernel command line options if required.
The contents of `cmdline` are passed to the kernel as-is. There are several special values that are
used to control the behaviour of linuxkit packages. See [kernel command line options](../docs/cmdline.md).
To override the names, you can specify the kernel image name with `binary: bzImage` and the tar image
with `tar: kernel.tar` or the empty string or `none` if you do not want to use a tarball at all.
@@ -97,8 +109,13 @@ including those in `services`, `onboot` and `onshutdown`. The volumes are create
chosen by linuxkit at build-time. The volumes then can be referenced by other containers and
mounted into them.
Volumes normally are blank directories. If an image is provided, the contents of that image
will be used to populate the volume.
Volumes can be in one of several formats:
* Blank directory: This is the default, and is an empty directory that is created at build-time. It is an overlayfs mount, and can be shared among multiple containers.
* Image laid out as filesystem: The contents of the image are used to populate the volume. Default format when an image is provided.
* Image as OCI v1-layout: The image is used as an [OCI v1-layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md). Indicated by `format: oci`.
Examples of each are given later in this section.
The `volumes` section can declare a volume to be read-write or read-only. If the volume is read-write,
a volume that is mounted into a container can be mounted read-only or read-write. If the volume is read-only,
@@ -108,7 +125,36 @@ By default, volumes are created read-write, and are mounted read-write.
Volume names **must** be unique, and must contain only lower-case alphanumeric characters, hyphens, and
underscores.
Sample `volumes` section:
#### Samples of `volumes`
##### Empty directory
Yaml showing both read-only and read-write:
```yml
volumes:
- name: dira
readonly: true
- name: dirb
readonly: true
```
Contents:
```sh
$ cd dir && ls -la
drwxr-xr-x 19 root wheel 608 Sep 30 15:03 .
drwxrwxrwt 130 root wheel 4160 Sep 30 15:03 ..
```
In the above example:
* `dira` is empty and is read-only.
* `volb` is empty and is read-write.
##### Image directory
Yaml showing both read-only and read-write:
```yml
volumes:
@@ -117,8 +163,7 @@ volumes:
readonly: true
- name: volb
image: alpine:latest
readonly: false
- name: volc
format: filesystem # optional, as this is the default format
readonly: false
```
@@ -126,7 +171,56 @@ In the above example:
* `vola` is populated by the contents of `alpine:latest` and is read-only.
* `volb` is populated by the contents of `alpine:latest` and is read-write.
* `volc` is an empty volume and is read-write.
Contents:
```sh
$ cd dir && ls -la
drwxr-xr-x 19 root wheel 608 Sep 30 15:03 .
drwxrwxrwt 130 root wheel 4160 Sep 30 15:03 ..
drwxr-xr-x 84 root wheel 2688 Sep 6 14:34 bin
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 dev
drwxr-xr-x 37 root wheel 1184 Sep 6 14:34 etc
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 home
drwxr-xr-x 13 root wheel 416 Sep 6 14:34 lib
drwxr-xr-x 5 root wheel 160 Sep 6 14:34 media
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 mnt
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 opt
dr-xr-xr-x 2 root wheel 64 Sep 6 14:34 proc
drwx------ 2 root wheel 64 Sep 6 14:34 root
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 run
drwxr-xr-x 63 root wheel 2016 Sep 6 14:34 sbin
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 srv
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 sys
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 tmp
drwxr-xr-x 7 root wheel 224 Sep 6 14:34 usr
drwxr-xr-x 13 root wheel 416 Sep 6 14:34 var
```
##### Image OCI Layout
Yaml showing both read-only and read-write, and both all architectures and a limited subset:
```yml
volumes:
- name: volo
image: alpine:latest
format: oci
readonly: true
- name: volp
image: alpine:latest
readonly: false
format: oci
platforms:
- linux/amd64
```
In the above example:
* `volo` is populated by the contents of `alpine:latest` as an OCI v1-layout for all architectures and is read-only.
* `volb` is populated by the contents of `alpine:latest` as an OCI v1-layout just for linux/amd64 and is read-write.
##### Volumes in `services`
Sample usage of volumes in `services` section:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -3,7 +3,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -4,7 +4,7 @@ kernel:
cmdline: "console=ttyS0 page_poison=1"
init:
- linuxkit/vpnkit-expose-port:77e45e4681c78d59f1d8a48818260948d55f9d05 # install vpnkit-expose-port and vpnkit-iptables-wrapper on host
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -3,7 +3,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
onboot:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
services:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -3,7 +3,7 @@ kernel:
cmdline: console=ttyS1
ucode: intel-ucode.cpio
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -3,7 +3,7 @@ kernel:
cmdline: console=ttyS1
ucode: intel-ucode.cpio
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13-rt
cmdline: "console=tty0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0 root=/dev/vda"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -4,7 +4,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
onboot:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
onboot:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 console=ttysclp0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -3,7 +3,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
onboot:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
onboot:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -7,6 +7,7 @@ import (
"path"
"path/filepath"
"strconv"
"strings"
log "github.com/sirupsen/logrus"
"golang.org/x/sys/unix"
@@ -62,10 +63,40 @@ func runcInit(rootPath, serviceType string) int {
}
logger := GetLog(logDir)
v2, err := isCgroupV2()
if err != nil {
log.Fatalf("Cannot determine cgroup version: %v", err)
}
msg := "cgroup v1"
if v2 {
msg = "cgroup v2"
}
log.Printf("Using %s", msg)
// did we choose to run in debug mode? If so, runc will be in debug, and all messages will go to stdout/stderr in addition to the log
var runcDebugMode, runcConsoleMode bool
dt, err := os.ReadFile("/proc/cmdline")
if err != nil {
log.Fatalf("error reading /proc/cmdline: %v", err)
}
debugLogger := log.New()
debugLogger.Level = log.InfoLevel
for _, s := range strings.Fields(string(dt)) {
if s == "linuxkit.runc_debug=1" {
runcDebugMode = true
debugLogger.Level = log.DebugLevel
}
if s == "linuxkit.runc_console=1" {
runcConsoleMode = true
}
}
for _, file := range files {
name := file.Name()
path := filepath.Join(rootPath, name)
log.Printf("%s %s: from %s", serviceType, name, path)
runtimeConfig := getRuntimeConfig(path)
@@ -74,8 +105,13 @@ func runcInit(rootPath, serviceType string) int {
status = 1
continue
}
debugLogger.Debugf("%s %s: creating", serviceType, name)
pidfile := filepath.Join(tmpdir, name)
cmd := exec.Command(runcBinary, "create", "--bundle", path, "--pid-file", pidfile, name)
cmdArgs := []string{"create", "--bundle", path, "--pid-file", pidfile, name}
if runcDebugMode {
cmdArgs = append([]string{"--debug"}, cmdArgs...)
}
cmd := exec.Command(runcBinary, cmdArgs...)
stdoutLog := serviceType + "." + name + ".out"
stdout, err := logger.Open(stdoutLog)
@@ -98,6 +134,15 @@ func runcInit(rootPath, serviceType string) int {
cmd.Stdout = stdout
cmd.Stderr = stderr
// if in console mode, send output to stdout/stderr instead of the log
// do not try io.MultiWriter(os.Stdout, stdout) as console messages will hang.
// it is not clear why, but since this is all for debugging anyways, it doesn't matter
// much.
if runcConsoleMode {
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
}
if err := cmd.Run(); err != nil {
log.Printf("Error creating %s: %v", name, err)
status = 1
@@ -117,6 +162,7 @@ func runcInit(rootPath, serviceType string) int {
continue
}
debugLogger.Debugf("%s %s: preparing", serviceType, name)
if err := prepareProcess(pid, runtimeConfig); err != nil {
log.Printf("Cannot prepare process: %v", err)
status = 1
@@ -134,7 +180,12 @@ func runcInit(rootPath, serviceType string) int {
waitFor <- state
}()
cmd = exec.Command(runcBinary, "start", name)
debugLogger.Debugf("%s %s: starting", serviceType, name)
cmdArgs = []string{"start", name}
if runcDebugMode {
cmdArgs = append([]string{"--debug"}, cmdArgs...)
}
cmd = exec.Command(runcBinary, cmdArgs...)
cmd.Stdout = stdout
cmd.Stderr = stderr
@@ -144,8 +195,10 @@ func runcInit(rootPath, serviceType string) int {
continue
}
debugLogger.Debugf("%s %s: waiting for completion", serviceType, name)
_ = <-waitFor
debugLogger.Debugf("%s %s: cleaning up", serviceType, name)
cleanup(path)
_ = os.Remove(pidfile)
@@ -154,6 +207,7 @@ func runcInit(rootPath, serviceType string) int {
// once that is fixed, this can be cleaned up
logger.Dump(stdoutLog)
logger.Dump(stderrLog)
debugLogger.Debugf("%s %s: complete", serviceType, name)
}
_ = os.RemoveAll(tmpdir)

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"os"
"path/filepath"
"strings"
log "github.com/sirupsen/logrus"
"golang.org/x/sys/unix"
@@ -59,36 +60,35 @@ func volumeInitCmd(ctx context.Context) int {
}
lowerDir := filepath.Join(*path, vol.Name(), "lower")
mergedDir := filepath.Join(*path, vol.Name(), "merged")
// need a tmpfs to create the workdir and upper
tmpDir := filepath.Join(*path, vol.Name(), "tmp")
if err := unix.Mount("tmpfs", tmpDir, "tmpfs", unix.MS_RELATIME, ""); err != nil {
log.WithError(err).Errorf("Error creating tmpDir for volume %s", vol.Name())
return 1
}
workDir := filepath.Join(tmpDir, "work")
upperDir := filepath.Join(tmpDir, "upper")
if err := os.Mkdir(upperDir, 0755); err != nil {
log.WithError(err).Errorf("Error creating upper dir for volume %s", vol.Name())
return 1
}
if err := os.Mkdir(workDir, 0755); err != nil {
log.WithError(err).Errorf("Error creating work dir for volume %s", vol.Name())
return 1
}
// and let's mount the actual dir
mountOps := []string{fmt.Sprintf("lowerdir=%s", lowerDir), fmt.Sprintf("upperdir=%s", upperDir), fmt.Sprintf("workdir=%s", workDir)}
if !readWrite {
log.Infof("Volume %s is read-only, bind-mounting read-only", vol.Name())
if err := unix.Mount(lowerDir, mergedDir, "bind", unix.MS_RDONLY, ""); err != nil {
log.WithError(err).Errorf("Error bind-mounting volume %s", vol.Name())
return 1
}
log.Infof("Volume %s is read-only", vol.Name())
mountOps = append(mountOps, "ro")
} else {
log.Infof("Volume %s is read-write, overlay mounting", vol.Name())
// need a tmpfs to create the workdir and upper
tmpDir := filepath.Join(*path, vol.Name(), "tmp")
if err := unix.Mount("tmpfs", tmpDir, "tmpfs", unix.MS_RELATIME, ""); err != nil {
log.WithError(err).Errorf("Error creating tmpDir for volume %s", vol.Name())
return 1
}
workDir := filepath.Join(tmpDir, "work")
upperDir := filepath.Join(tmpDir, "upper")
if err := os.Mkdir(upperDir, 0755); err != nil {
log.WithError(err).Errorf("Error creating upper dir for volume %s", vol.Name())
return 1
}
if err := os.Mkdir(workDir, 0755); err != nil {
log.WithError(err).Errorf("Error creating work dir for volume %s", vol.Name())
return 1
}
// and let's mount the actual dir
data := fmt.Sprintf("lowerdir=%s,upperdir=%s,workdir=%s", lowerDir, upperDir, workDir)
if err := unix.Mount("overlay", mergedDir, "overlay", unix.MS_RELATIME, data); err != nil {
log.WithError(err).Errorf("Error overlay-mounting volume %s", vol.Name())
return 1
}
log.Infof("Volume %s is read-write", vol.Name())
}
data := strings.Join(mountOps, ",")
if err := unix.Mount("overlay", mergedDir, "overlay", unix.MS_RELATIME, data); err != nil {
log.WithError(err).Errorf("Error overlay-mounting volume %s", vol.Name())
return 1
}
}
return 0

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel-clear-containers:4.9.x
cmdline: "root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off quiet cryptomgr.notests page_poison=on"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
onboot:
- name: sysctl
image: mobylinux/sysctl:2cf2f9d5b4d314ba1bfc22b2fe931924af666d8c

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel-ima:4.11.1-186dd3605ee7b23214850142f8f02b4679dbd148
cmdline: "console=ttyS0 console=tty0 page_poison=1 ima_appraise=enforce_ns"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: mobylinux/kernel-landlock:4.9.x
cmdline: "console=ttyS0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- mobylinux/runc:b0fb122e10dbb7e4e45115177a61a3f8d68c19a9
- mobylinux/containerd:18eaf72f3f4f9a9f29ca1951f66df701f873060b
- mobylinux/ca-certificates:eabc5a6e59f05aa91529d80e9a595b85b046f935

View File

@@ -2,7 +2,7 @@ kernel:
image: "linuxkitprojects/kernel-memorizer:4.10_dbg"
cmdline: "console=ttyS0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
onboot:

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
onboot:

View File

@@ -2,7 +2,7 @@ kernel:
image: okernel:latest
cmdline: "console=tty0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkitprojects/kernel-shiftfs:4.11.4-881a041fc14bd95814cf140b5e98d97dd65160b5
cmdline: "console=ttyS0 console=tty0 page_poison=1"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
- linuxkit/containerd:39301e7312f13eedf19bd5d5551af7b37001d435
- linuxkit/ca-certificates:5aaa343474e5ac3ac01f8b917e82efb1063d80ff

View File

@@ -22,12 +22,13 @@ func createPackageResolver(baseDir string) spec.PackageResolver {
pkgValue = pkgTmpl
case strings.HasPrefix(pkgTmpl, templateFlag+templatePkg):
pkgPath := strings.TrimPrefix(pkgTmpl, templateFlag+templatePkg)
piBase := pkglib.NewPkgInfo()
var pkgs []pkglib.Pkg
pkgConfig := pkglib.PkglibConfig{
BuildYML: defaultPkgBuildYML,
HashCommit: defaultPkgCommit,
Tag: defaultPkgTag,
Tag: piBase.Tag,
}
pkgs, err = pkglib.NewFromConfig(pkgConfig, path.Join(baseDir, pkgPath))
if err != nil {

View File

@@ -8,6 +8,7 @@ import (
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/match"
"github.com/google/go-containerregistry/pkg/v1/partial"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// matchPlatformsOSArch because match.Platforms rejects it if the provided
@@ -46,7 +47,7 @@ func matchAllAnnotations(annotations map[string]string) match.Matcher {
}
}
func (p *Provider) findImage(imageName, architecture string) (v1.Image, error) {
func (p *Provider) findImage(imageName string, platform imagespec.Platform) (v1.Image, error) {
root, err := p.FindRoot(imageName)
if err != nil {
return nil, err
@@ -58,7 +59,7 @@ func (p *Provider) findImage(imageName, architecture string) (v1.Image, error) {
ii, err := root.ImageIndex()
if err == nil {
// we have the index, get the manifest that represents the manifest for the desired architecture
platform := v1.Platform{OS: "linux", Architecture: architecture}
platform := v1.Platform{OS: platform.OS, Architecture: platform.Architecture}
images, err := partial.FindImages(ii, matchPlatformsOSArch(platform))
if err != nil || len(images) < 1 {
return nil, fmt.Errorf("error retrieving image %s for platform %v from cache: %v", imageName, platform, err)

View File

@@ -10,7 +10,6 @@ import (
"github.com/containerd/containerd/reference"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
"github.com/google/go-containerregistry/pkg/v1/match"
"github.com/google/go-containerregistry/pkg/v1/mutate"
"github.com/google/go-containerregistry/pkg/v1/partial"
@@ -32,10 +31,10 @@ const (
// ImageSource a source for an image in the OCI distribution cache.
// Implements a spec.ImageSource.
type ImageSource struct {
ref *reference.Spec
provider *Provider
architecture string
descriptor *v1.Descriptor
ref *reference.Spec
provider *Provider
platform *imagespec.Platform
descriptor *v1.Descriptor
}
type spdxStatement struct {
@@ -45,12 +44,12 @@ type spdxStatement struct {
// NewSource return an ImageSource for a specific ref and architecture in the given
// cache directory.
func (p *Provider) NewSource(ref *reference.Spec, architecture string, descriptor *v1.Descriptor) lktspec.ImageSource {
func (p *Provider) NewSource(ref *reference.Spec, platform *imagespec.Platform, descriptor *v1.Descriptor) lktspec.ImageSource {
return ImageSource{
ref: ref,
provider: p,
architecture: architecture,
descriptor: descriptor,
ref: ref,
provider: p,
platform: platform,
descriptor: descriptor,
}
}
@@ -58,7 +57,7 @@ func (p *Provider) NewSource(ref *reference.Spec, architecture string, descripto
// architecture, if necessary.
func (c ImageSource) Config() (imagespec.ImageConfig, error) {
imageName := c.ref.String()
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return imagespec.ImageConfig{}, err
}
@@ -84,7 +83,7 @@ func (c ImageSource) TarReader() (io.ReadCloser, error) {
imageName := c.ref.String()
// get a reference to the image
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return nil, err
}
@@ -104,7 +103,7 @@ func (c ImageSource) V1TarReader(overrideName string) (io.ReadCloser, error) {
return nil, fmt.Errorf("error parsing image name: %v", err)
}
// get a reference to the image
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return nil, err
}
@@ -129,7 +128,7 @@ func (c ImageSource) OCITarReader(overrideName string) (io.ReadCloser, error) {
return nil, fmt.Errorf("error parsing image name: %v", err)
}
// get a reference to the image
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return nil, err
}
@@ -139,160 +138,37 @@ func (c ImageSource) OCITarReader(overrideName string) (io.ReadCloser, error) {
defer w.Close()
tw := tar.NewWriter(w)
defer tw.Close()
// layout file
layoutFileBytes := []byte(layoutFile)
if err := tw.WriteHeader(&tar.Header{
Name: "oci-layout",
Mode: 0644,
Size: int64(len(layoutFileBytes)),
Typeflag: tar.TypeReg,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(layoutFileBytes); err != nil {
if err := writeLayoutHeader(tw); err != nil {
_ = w.CloseWithError(err)
return
}
// make blobs directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
if err := writeLayoutImage(tw, image); err != nil {
_ = w.CloseWithError(err)
return
}
// make blobs/sha256 directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/sha256/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
_ = w.CloseWithError(err)
return
}
// write config, each layer, manifest, saving the digest for each
config, err := image.RawConfigFile()
imageDigest, err := image.Digest()
if err != nil {
_ = w.CloseWithError(err)
return
}
configDigest, configSize, err := v1.SHA256(bytes.NewReader(config))
imageSize, err := image.Size()
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", configDigest.Hex),
Mode: 0644,
Typeflag: tar.TypeReg,
Size: configSize,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(config); err != nil {
_ = w.CloseWithError(err)
return
}
layers, err := image.Layers()
if err != nil {
_ = w.CloseWithError(err)
return
}
for _, layer := range layers {
blob, err := layer.Compressed()
if err != nil {
_ = w.CloseWithError(err)
return
}
defer blob.Close()
blobDigest, err := layer.Digest()
if err != nil {
_ = w.CloseWithError(err)
return
}
blobSize, err := layer.Size()
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", blobDigest.Hex),
Mode: 0644,
Size: blobSize,
Typeflag: tar.TypeReg,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := io.Copy(tw, blob); err != nil {
_ = w.CloseWithError(err)
return
}
}
// write the manifest
manifest, err := image.RawManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
manifestDigest, manifestSize, err := v1.SHA256(bytes.NewReader(manifest))
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", manifestDigest.Hex),
Mode: 0644,
Size: int64(len(manifest)),
Typeflag: tar.TypeReg,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(manifest); err != nil {
_ = w.CloseWithError(err)
return
}
// write the index file
desc := v1.Descriptor{
MediaType: types.OCIImageIndex,
Size: manifestSize,
Digest: manifestDigest,
Size: imageSize,
Digest: imageDigest,
Annotations: map[string]string{
imagespec.AnnotationRefName: refName.String(),
},
}
ii := empty.Index
index, err := ii.IndexManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
index.Manifests = append(index.Manifests, desc)
rawIndex, err := json.MarshalIndent(index, "", " ")
if err != nil {
_ = w.CloseWithError(err)
return
}
// write the index
if err := tw.WriteHeader(&tar.Header{
Name: "index.json",
Mode: 0644,
Size: int64(len(rawIndex)),
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(rawIndex); err != nil {
if err := writeLayoutIndex(tw, desc); err != nil {
_ = w.CloseWithError(err)
return
}
@@ -314,15 +190,15 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
}
// get the digest of the manifest that represents our targeted architecture
descs, err := partial.FindManifests(index, matchPlatformsOSArch(v1.Platform{OS: "linux", Architecture: c.architecture}))
descs, err := partial.FindManifests(index, matchPlatformsOSArch(v1.Platform{OS: c.platform.OS, Architecture: c.platform.Architecture}))
if err != nil {
return nil, err
}
if len(descs) < 1 {
return nil, fmt.Errorf("no manifest found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("no manifest found for %s platform %s", c.ref.String(), c.platform)
}
if len(descs) > 1 {
return nil, fmt.Errorf("multiple manifests found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("multiple manifests found for %s platform %s", c.ref.String(), c.platform)
}
// get the digest of the manifest that represents our targeted architecture
desc := descs[0]
@@ -336,7 +212,7 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
return nil, err
}
if len(descs) > 1 {
return nil, fmt.Errorf("multiple manifests found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("multiple manifests found for %s platform %s", c.ref.String(), c.platform)
}
if len(descs) < 1 {
return nil, nil
@@ -348,10 +224,10 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
return nil, err
}
if len(images) < 1 {
return nil, fmt.Errorf("no attestation image found for %s arch %s, even though the manifest exists", c.ref.String(), c.architecture)
return nil, fmt.Errorf("no attestation image found for %s platform %s, even though the manifest exists", c.ref.String(), c.platform)
}
if len(images) > 1 {
return nil, fmt.Errorf("multiple attestation images found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("multiple attestation images found for %s platform %s", c.ref.String(), c.platform)
}
image := images[0]
manifest, err := image.Manifest()
@@ -363,7 +239,7 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
return nil, err
}
if len(manifest.Layers) != len(layers) {
return nil, fmt.Errorf("manifest layers and image layers do not match for the attestation for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("manifest layers and image layers do not match for the attestation for %s platform %s", c.ref.String(), c.platform)
}
var readers []io.ReadCloser
for i, layer := range manifest.Layers {

162
src/cmd/linuxkit/cache/indexsource.go vendored Normal file
View File

@@ -0,0 +1,162 @@
package cache
import (
"archive/tar"
"bytes"
"fmt"
"io"
"github.com/containerd/containerd/reference"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/types"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// IndexSource a source for an image in the OCI distribution cache.
// Implements a spec.ImageSource.
type IndexSource struct {
ref *reference.Spec
provider *Provider
descriptor *v1.Descriptor
platforms []imagespec.Platform
}
// NewIndexSource return an IndexSource for a specific ref in the given
// cache directory.
func (p *Provider) NewIndexSource(ref *reference.Spec, descriptor *v1.Descriptor, platforms []imagespec.Platform) lktspec.IndexSource {
return IndexSource{
ref: ref,
provider: p,
descriptor: descriptor,
platforms: platforms,
}
}
// Config return the imagespec.ImageConfig for the given source. Resolves to the
// architecture, if necessary.
func (c IndexSource) Image(platform imagespec.Platform) (spec.ImageSource, error) {
imageName := c.ref.String()
index, err := c.provider.findIndex(imageName)
if err != nil {
return nil, err
}
manifests, err := index.IndexManifest()
if err != nil {
return nil, err
}
for _, manifest := range manifests.Manifests {
if manifest.Platform != nil && manifest.Platform.Architecture == platform.Architecture && manifest.Platform.OS == platform.OS {
return c.provider.NewSource(c.ref, &platform, &manifest), nil
}
}
return nil, fmt.Errorf("no manifest found for platform %q", platform)
}
// OCITarReader return an io.ReadCloser to read the image as a v1 tarball whose contents match OCI v1 layout spec
func (c IndexSource) OCITarReader(overrideName string) (io.ReadCloser, error) {
imageName := c.ref.String()
saveName := imageName
if overrideName != "" {
saveName = overrideName
}
refName, err := name.ParseReference(saveName)
if err != nil {
return nil, fmt.Errorf("error parsing image name: %v", err)
}
// get a reference to the image
index, err := c.provider.findIndex(c.ref.String())
if err != nil {
return nil, err
}
// convert the writer to a reader
r, w := io.Pipe()
go func() {
defer w.Close()
tw := tar.NewWriter(w)
defer tw.Close()
if err := writeLayoutHeader(tw); err != nil {
_ = w.CloseWithError(err)
return
}
manifests, err := index.IndexManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
// for each manifest, write the manifest blob, then go through each manifest and find the image for it
// and write its blobs
for _, manifest := range manifests.Manifests {
// if we restricted this image source to certain platforms, we should only write those
if len(c.platforms) > 0 {
found := false
for _, platform := range c.platforms {
if platform.Architecture == manifest.Platform.Architecture && platform.OS == manifest.Platform.OS &&
(platform.Variant == "" || platform.Variant == manifest.Platform.Variant) {
found = true
break
}
}
if !found {
continue
}
}
switch manifest.MediaType {
case types.OCIManifestSchema1, types.DockerManifestSchema2:
// this is an image manifest
image, err := index.Image(manifest.Digest)
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := writeLayoutImage(tw, image); err != nil {
_ = w.CloseWithError(err)
return
}
}
}
// write the index directly as a blob
indexSize, err := index.Size()
if err != nil {
_ = w.CloseWithError(err)
return
}
indexDigest, err := index.Digest()
if err != nil {
_ = w.CloseWithError(err)
return
}
indexBytes, err := index.RawManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := writeLayoutBlob(tw, indexDigest.Hex, indexSize, bytes.NewReader(indexBytes)); err != nil {
_ = w.CloseWithError(err)
return
}
desc := v1.Descriptor{
MediaType: types.OCIImageIndex,
Size: indexSize,
Digest: indexDigest,
Annotations: map[string]string{
imagespec.AnnotationRefName: refName.String(),
},
}
if err := writeLayoutIndex(tw, desc); err != nil {
_ = w.CloseWithError(err)
return
}
}()
return r, nil
}
// Descriptor return the descriptor of the index.
func (c IndexSource) Descriptor() *v1.Descriptor {
return c.descriptor
}

145
src/cmd/linuxkit/cache/layout.go vendored Normal file
View File

@@ -0,0 +1,145 @@
package cache
import (
"archive/tar"
"bytes"
"encoding/json"
"fmt"
"io"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
)
func writeLayoutHeader(tw *tar.Writer) error {
// layout file
layoutFileBytes := []byte(layoutFile)
if err := tw.WriteHeader(&tar.Header{
Name: "oci-layout",
Mode: 0644,
Size: int64(len(layoutFileBytes)),
Typeflag: tar.TypeReg,
}); err != nil {
return err
}
if _, err := tw.Write(layoutFileBytes); err != nil {
return err
}
// make blobs directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
return err
}
// make blobs/sha256 directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/sha256/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
return err
}
return nil
}
func writeLayoutImage(tw *tar.Writer, image v1.Image) error {
// write config, each layer, manifest, saving the digest for each
manifest, err := image.Manifest()
if err != nil {
return err
}
configDesc := manifest.Config
configBytes, err := image.RawConfigFile()
if err != nil {
return err
}
if err := writeLayoutBlob(tw, configDesc.Digest.Hex, configDesc.Size, bytes.NewReader(configBytes)); err != nil {
return err
}
layers, err := image.Layers()
if err != nil {
return err
}
for _, layer := range layers {
blob, err := layer.Compressed()
if err != nil {
return err
}
defer blob.Close()
blobDigest, err := layer.Digest()
if err != nil {
return err
}
blobSize, err := layer.Size()
if err != nil {
return err
}
if err := writeLayoutBlob(tw, blobDigest.Hex, blobSize, blob); err != nil {
return err
}
}
// write the manifest
manifestSize, err := image.Size()
if err != nil {
return err
}
manifestDigest, err := image.Digest()
if err != nil {
return err
}
manifestBytes, err := image.RawManifest()
if err != nil {
return err
}
if err := writeLayoutBlob(tw, manifestDigest.Hex, manifestSize, bytes.NewReader(manifestBytes)); err != nil {
return err
}
return nil
}
func writeLayoutBlob(tw *tar.Writer, digest string, size int64, blob io.Reader) error {
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", digest),
Mode: 0644,
Size: size,
Typeflag: tar.TypeReg,
}); err != nil {
return err
}
if _, err := io.Copy(tw, blob); err != nil {
return err
}
return nil
}
func writeLayoutIndex(tw *tar.Writer, desc v1.Descriptor) error {
ii := empty.Index
index, err := ii.IndexManifest()
if err != nil {
return err
}
index.Manifests = append(index.Manifests, desc)
rawIndex, err := json.MarshalIndent(index, "", " ")
if err != nil {
return err
}
// write the index
if err := tw.WriteHeader(&tar.Header{
Name: "index.json",
Mode: 0644,
Size: int64(len(rawIndex)),
}); err != nil {
return err
}
if _, err := tw.Write(rawIndex); err != nil {
return err
}
return nil
}

33
src/cmd/linuxkit/cache/platform.go vendored Normal file
View File

@@ -0,0 +1,33 @@
package cache
import (
"fmt"
"strings"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
func platformString(p imagespec.Platform) string {
parts := []string{p.OS, p.Architecture}
if p.Variant != "" {
parts = append(parts, p.Variant)
}
return strings.Join(parts, "/")
}
func platformMessageGenerator(platforms []imagespec.Platform) string {
var platformMessage string
switch {
case len(platforms) == 0:
platformMessage = "all platforms"
case len(platforms) == 1:
platformMessage = fmt.Sprintf("platform %s", platformString(platforms[0]))
default:
var platStrings []string
for _, p := range platforms {
platStrings = append(platStrings, platformString(p))
}
platformMessage = fmt.Sprintf("platforms %s", strings.Join(platStrings, ","))
}
return platformMessage
}

View File

@@ -3,6 +3,7 @@ package cache
import (
"errors"
"fmt"
"strings"
"github.com/containerd/containerd/reference"
"github.com/google/go-containerregistry/pkg/authn"
@@ -30,12 +31,13 @@ const (
// architecture, and any manifests that have no architecture at all. It will ignore manifests
// for other architectures. If no architecture is provided, it will validate all manifests.
// It also calculates the hash of each component.
func (p *Provider) ValidateImage(ref *reference.Spec, architecture string) (lktspec.ImageSource, error) {
func (p *Provider) ValidateImage(ref *reference.Spec, platforms []imagespec.Platform) (lktspec.ImageSource, error) {
var (
imageIndex v1.ImageIndex
image v1.Image
imageName = ref.String()
desc *v1.Descriptor
imageIndex v1.ImageIndex
image v1.Image
imageName = ref.String()
desc *v1.Descriptor
platformMessage = platformMessageGenerator(platforms)
)
// next try the local cache
root, err := p.FindRoot(imageName)
@@ -71,7 +73,15 @@ func (p *Provider) ValidateImage(ref *reference.Spec, architecture string) (lkts
if err != nil {
return ImageSource{}, fmt.Errorf("could not get index manifest: %w", err)
}
var architectures = make(map[string]bool)
var (
targetPlatforms = make(map[string]bool)
foundPlatforms = make(map[string]bool)
)
for _, plat := range platforms {
pString := platformString(plat)
targetPlatforms[pString] = false
foundPlatforms[pString] = false
}
// ignore only other architectures; manifest entries that have no architectures at all
// are going to be additional metadata, so we need to check them
for _, m := range im.Manifests {
@@ -80,29 +90,50 @@ func (p *Provider) ValidateImage(ref *reference.Spec, architecture string) (lkts
return ImageSource{}, fmt.Errorf("invalid image: %w", err)
}
}
if architecture != "" && m.Platform.Architecture == architecture && m.Platform.OS == linux {
if err := validateManifestContents(imageIndex, m.Digest); err != nil {
return ImageSource{}, fmt.Errorf("invalid image: %w", err)
// go through each target platform, and see if this one matched. If it did, mark the target as
for _, plat := range platforms {
if plat.Architecture == m.Platform.Architecture && plat.OS == m.Platform.OS &&
(plat.Variant == "" || plat.Variant == m.Platform.Variant) {
targetPlatforms[platformString(plat)] = true
break
}
architectures[architecture] = true
}
}
if architecture == "" || architectures[architecture] {
if len(platforms) == 0 {
return p.NewSource(
ref,
architecture,
nil,
desc,
), nil
}
return ImageSource{}, fmt.Errorf("index for %s did not contain image for platform linux/%s", imageName, architecture)
// we have cycled through all of the manifests, let's check if we have all of the platforms
var missing []string
for plat, found := range targetPlatforms {
if !found {
missing = append(missing, plat)
}
}
if len(missing) == 0 {
return p.NewSource(
ref,
nil,
desc,
), nil
}
return ImageSource{}, fmt.Errorf("index for %s did not contain image for platforms %s", imageName, strings.Join(missing, ", "))
case image != nil:
if len(platforms) > 1 {
return ImageSource{}, fmt.Errorf("image %s is not a multi-arch image, but asked for %s", imageName, platformMessage)
}
// we found a local image, make sure it is up to date
if err := validate.Image(image); err != nil {
return ImageSource{}, fmt.Errorf("invalid image, %s", err)
}
return p.NewSource(
ref,
architecture,
&platforms[0],
desc,
), nil
}
@@ -164,7 +195,7 @@ func (p *Provider) Pull(name string, withArchReferences bool) error {
if err := p.cache.WriteIndex(ii); err != nil {
return fmt.Errorf("unable to write index: %v", err)
}
if _, err := p.DescriptorWrite(&v1ref, desc.Descriptor); err != nil {
if err := p.DescriptorWrite(&v1ref, desc.Descriptor); err != nil {
return fmt.Errorf("unable to write index descriptor to cache: %v", err)
}
if withArchReferences {
@@ -179,7 +210,7 @@ func (p *Provider) Pull(name string, withArchReferences bool) error {
if err != nil {
return fmt.Errorf("unable to parse arch-specific reference %s: %v", archSpecific, err)
}
if _, err := p.DescriptorWrite(&archRef, m); err != nil {
if err := p.DescriptorWrite(&archRef, m); err != nil {
return fmt.Errorf("unable to write index descriptor to cache: %v", err)
}
}

View File

@@ -19,9 +19,7 @@ import (
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/remote"
"github.com/google/go-containerregistry/pkg/v1/types"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
lktutil "github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
log "github.com/sirupsen/logrus"
)
@@ -41,45 +39,42 @@ const (
// If you just want to check the status of a local ref, use ValidateImage.
// Note that ImagePull does try ValidateImage first, so if the image is already in the cache, it will not
// do any network activity at all.
func (p *Provider) ImagePull(ref *reference.Spec, trustedRef, architecture string, alwaysPull bool) (lktspec.ImageSource, error) {
func (p *Provider) ImagePull(ref *reference.Spec, platforms []imagespec.Platform, alwaysPull bool) error {
imageName := util.ReferenceExpand(ref.String())
canonicalRef, err := reference.Parse(imageName)
if err != nil {
return ImageSource{}, fmt.Errorf("invalid image name %s: %v", imageName, err)
return fmt.Errorf("invalid image name %s: %v", imageName, err)
}
ref = &canonicalRef
image := ref.String()
pullImageName := image
platformMessage := platformMessageGenerator(platforms)
remoteOptions := []remote.Option{remote.WithAuthFromKeychain(authn.DefaultKeychain)}
if trustedRef != "" {
pullImageName = trustedRef
}
log.Debugf("ImagePull to cache %s trusted reference %s", image, pullImageName)
// unless alwaysPull is set to true, check locally first
if alwaysPull {
log.Printf("Instructed always to pull, so pulling image %s arch %s", image, architecture)
log.Debugf("Instructed always to pull, so pulling image %s %s", image, platformMessage)
} else {
imgSrc, err := p.ValidateImage(ref, architecture)
imgSrc, err := p.ValidateImage(ref, platforms)
switch {
case err == nil && imgSrc != nil:
log.Printf("Image %s arch %s found in local cache, not pulling", image, architecture)
return imgSrc, nil
log.Debugf("Image %s %s found in local cache, not pulling", image, platformMessage)
return nil
case err != nil && errors.Is(err, &noReferenceError{}):
log.Printf("Image %s arch %s not found in local cache, pulling", image, architecture)
log.Debugf("Image %s %s not found in local cache, pulling", image, platformMessage)
default:
log.Printf("Image %s arch %s incomplete or invalid in local cache, error %v, pulling", image, architecture, err)
log.Debugf("Image %s %s incomplete or invalid in local cache, error %v, pulling", image, platformMessage, err)
}
// there was an error, so try to pull
}
remoteRef, err := name.ParseReference(pullImageName)
if err != nil {
return ImageSource{}, fmt.Errorf("invalid image name %s: %v", pullImageName, err)
return fmt.Errorf("invalid image name %s: %v", pullImageName, err)
}
desc, err := remote.Get(remoteRef, remoteOptions...)
if err != nil {
return ImageSource{}, fmt.Errorf("error getting manifest for trusted image %s: %v", pullImageName, err)
return fmt.Errorf("error getting manifest for image %s: %v", pullImageName, err)
}
// use the original image name in the annotation
@@ -90,46 +85,57 @@ func (p *Provider) ImagePull(ref *reference.Spec, trustedRef, architecture strin
// first attempt as an index
ii, err := desc.ImageIndex()
if err == nil {
log.Debugf("ImageWrite retrieved %s is index, saving, first checking if it contains target arch %s", pullImageName, architecture)
log.Debugf("ImageWrite retrieved %s is index, saving, first checking if it contains target %s", pullImageName, platformMessage)
im, err := ii.IndexManifest()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to get IndexManifest: %v", err)
return fmt.Errorf("unable to get IndexManifest: %v", err)
}
// only useful if it contains our architecture
var foundArch bool
var foundPlatforms []*v1.Platform
for _, m := range im.Manifests {
if m.MediaType.IsImage() && m.Platform != nil && m.Platform.Architecture == architecture && m.Platform.OS == linux {
foundArch = true
break
if m.MediaType.IsImage() && m.Platform != nil {
foundPlatforms = append(foundPlatforms, m.Platform)
}
}
if !foundArch {
return ImageSource{}, fmt.Errorf("index %s does not contain target architecture %s", pullImageName, architecture)
// now see if we have all of the platforms we need
var missing []string
for _, requiredPlatform := range platforms {
// we did not find it, so maybe one satisfies it
var matchedPlatform bool
for _, p := range foundPlatforms {
if p.OS == requiredPlatform.OS && p.Architecture == requiredPlatform.Architecture && (p.Variant == requiredPlatform.Variant || requiredPlatform.Variant == "") {
// this one satisfies it, so do not count it missing
matchedPlatform = true
break
}
}
if !matchedPlatform {
missing = append(missing, platformString(requiredPlatform))
}
}
if len(missing) > 0 {
return fmt.Errorf("index %s does not contain target platforms %s", pullImageName, strings.Join(missing, ","))
}
if err := p.cache.WriteIndex(ii); err != nil {
return ImageSource{}, fmt.Errorf("unable to write index: %v", err)
return fmt.Errorf("unable to write index: %v", err)
}
if _, err := p.DescriptorWrite(ref, desc.Descriptor); err != nil {
return ImageSource{}, fmt.Errorf("unable to write index descriptor to cache: %v", err)
if err := p.DescriptorWrite(ref, desc.Descriptor); err != nil {
return fmt.Errorf("unable to write index descriptor to cache: %v", err)
}
} else {
var im v1.Image
// try an image
im, err = desc.Image()
if err != nil {
return ImageSource{}, fmt.Errorf("provided image is neither an image nor an index: %s", image)
return fmt.Errorf("provided image is neither an image nor an index: %s", image)
}
log.Debugf("ImageWrite retrieved %s is image, saving", pullImageName)
if err = p.cache.ReplaceImage(im, match.Name(image), layout.WithAnnotations(annotations)); err != nil {
return ImageSource{}, fmt.Errorf("unable to save image to cache: %v", err)
return fmt.Errorf("unable to save image to cache: %v", err)
}
}
return p.NewSource(
ref,
architecture,
&desc.Descriptor,
), nil
return nil
}
// ImageLoad takes an OCI format image tar stream and writes it locally. It should be
@@ -227,27 +233,27 @@ func (p *Provider) ImageLoad(r io.Reader) ([]v1.Descriptor, error) {
// does not pull down any images; entirely assumes that the subjects of the manifests are present.
// If a reference to the provided already exists and it is an index, updates the manifests in the
// existing index.
func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) (lktspec.ImageSource, error) {
func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) error {
image := ref.String()
log.Debugf("writing an index for %s", image)
if len(descriptors) < 1 {
return ImageSource{}, errors.New("cannot create index without any manifests")
return errors.New("cannot create index without any manifests")
}
ii, err := p.cache.ImageIndex()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to get root index: %v", err)
return fmt.Errorf("unable to get root index: %v", err)
}
images, err := partial.FindImages(ii, match.Name(image))
if err != nil {
return ImageSource{}, fmt.Errorf("error parsing index: %v", err)
return fmt.Errorf("error parsing index: %v", err)
}
if err == nil && len(images) > 0 {
return ImageSource{}, fmt.Errorf("image named %s already exists in cache and is not an index", image)
return fmt.Errorf("image named %s already exists in cache and is not an index", image)
}
indexes, err := partial.FindIndexes(ii, match.Name(image))
if err != nil {
return ImageSource{}, fmt.Errorf("error parsing index: %v", err)
return fmt.Errorf("error parsing index: %v", err)
}
var im v1.IndexManifest
// do we update an existing one? Or create a new one?
@@ -255,11 +261,11 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
// we already had one, so update just the referenced index and return
manifest, err := indexes[0].IndexManifest()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to convert index for %s into its manifest: %v", image, err)
return fmt.Errorf("unable to convert index for %s into its manifest: %v", image, err)
}
oldhash, err := indexes[0].Digest()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to get hash of existing index: %v", err)
return fmt.Errorf("unable to get hash of existing index: %v", err)
}
// we only care about avoiding duplicate arch/OS/Variant
var (
@@ -318,7 +324,7 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
appliedManifests[m.Digest] = true
continue
}
value, ok := m.Annotations[lktutil.AnnotationDockerReferenceDigest]
value, ok := m.Annotations[util.AnnotationDockerReferenceDigest]
if !ok {
manifest.Manifests = append(manifest.Manifests, m)
appliedManifests[m.Digest] = true
@@ -336,7 +342,7 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
im = *manifest
// remove the old index
if err := p.cache.RemoveBlob(oldhash); err != nil {
return ImageSource{}, fmt.Errorf("unable to remove old index file: %v", err)
return fmt.Errorf("unable to remove old index file: %v", err)
}
} else {
// we did not have one, so create an index, store it, update the root index.json, and return
@@ -350,18 +356,18 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
// write the updated index, remove the old one
b, err := json.Marshal(im)
if err != nil {
return ImageSource{}, fmt.Errorf("unable to marshal new index to json: %v", err)
return fmt.Errorf("unable to marshal new index to json: %v", err)
}
hash, size, err := v1.SHA256(bytes.NewReader(b))
if err != nil {
return ImageSource{}, fmt.Errorf("error calculating hash of index json: %v", err)
return fmt.Errorf("error calculating hash of index json: %v", err)
}
if err := p.cache.WriteBlob(hash, io.NopCloser(bytes.NewReader(b))); err != nil {
return ImageSource{}, fmt.Errorf("error writing new index to json: %v", err)
return fmt.Errorf("error writing new index to json: %v", err)
}
// finally update the descriptor in the root
if err := p.cache.RemoveDescriptors(match.Name(image)); err != nil {
return ImageSource{}, fmt.Errorf("unable to remove old descriptor from index.json: %v", err)
return fmt.Errorf("unable to remove old descriptor from index.json: %v", err)
}
desc := v1.Descriptor{
MediaType: types.OCIImageIndex,
@@ -372,21 +378,17 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
},
}
if err := p.cache.AppendDescriptor(desc); err != nil {
return ImageSource{}, fmt.Errorf("unable to append new descriptor to index.json: %v", err)
return fmt.Errorf("unable to append new descriptor to index.json: %v", err)
}
return p.NewSource(
ref,
"",
&desc,
), nil
return nil
}
// DescriptorWrite writes a descriptor to the cache index; it validates that it has a name
// and replaces any existing one
func (p *Provider) DescriptorWrite(ref *reference.Spec, desc v1.Descriptor) (lktspec.ImageSource, error) {
func (p *Provider) DescriptorWrite(ref *reference.Spec, desc v1.Descriptor) error {
if ref == nil {
return ImageSource{}, errors.New("cannot write descriptor without reference name")
return errors.New("cannot write descriptor without reference name")
}
image := ref.String()
if desc.Annotations == nil {
@@ -397,22 +399,18 @@ func (p *Provider) DescriptorWrite(ref *reference.Spec, desc v1.Descriptor) (lkt
// do we update an existing one? Or create a new one?
if err := p.cache.RemoveDescriptors(match.Name(image)); err != nil {
return ImageSource{}, fmt.Errorf("unable to remove old descriptors for %s: %v", image, err)
return fmt.Errorf("unable to remove old descriptors for %s: %v", image, err)
}
if err := p.cache.AppendDescriptor(desc); err != nil {
return ImageSource{}, fmt.Errorf("unable to append new descriptor for %s: %v", image, err)
return fmt.Errorf("unable to append new descriptor for %s: %v", image, err)
}
return p.NewSource(
ref,
"",
&desc,
), nil
return nil
}
func (p *Provider) ImageInCache(ref *reference.Spec, trustedRef, architecture string) (bool, error) {
img, err := p.findImage(ref.String(), architecture)
img, err := p.findImage(ref.String(), imagespec.Platform{OS: linux, Architecture: architecture})
if err != nil {
return false, err
}

View File

@@ -4,17 +4,20 @@ import (
"io"
"os"
"runtime"
"strings"
"github.com/containerd/containerd/reference"
v1 "github.com/google/go-containerregistry/pkg/v1"
cachepkg "github.com/linuxkit/linuxkit/src/cmd/linuxkit/cache"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)
func cacheExportCmd() *cobra.Command {
var (
arch string
platform string
outputFile string
format string
tagName string
@@ -42,7 +45,16 @@ func cacheExportCmd() *cobra.Command {
log.Fatalf("unable to find image named %s: %v", name, err)
}
src := p.NewSource(&ref, arch, desc)
plat, err := v1.ParsePlatform(platform)
if err != nil {
log.Fatalf("invalid platform %s: %v", platform, err)
}
platspec := imagespec.Platform{
Architecture: plat.Architecture,
OS: plat.OS,
Variant: plat.Variant,
}
src := p.NewSource(&ref, &platspec, desc)
var reader io.ReadCloser
switch format {
case "docker":
@@ -88,7 +100,7 @@ func cacheExportCmd() *cobra.Command {
},
}
cmd.Flags().StringVar(&arch, "arch", runtime.GOARCH, "Architecture to resolve an index to an image, if the provided image name is an index")
cmd.Flags().StringVar(&platform, "platform", strings.Join([]string{"linux", runtime.GOARCH}, "/"), "Platform to resolve an index to an image, if the provided image name is an index")
cmd.Flags().StringVar(&outputFile, "outfile", "", "Path to file to save output, '-' for stdout")
cmd.Flags().StringVar(&format, "format", "oci", "export format, one of 'oci' (OCI tar), 'docker' (docker tar), 'filesystem'")
cmd.Flags().StringVar(&tagName, "name", "", "override the provided image name in the exported tar file; useful only for format=oci")

View File

@@ -43,8 +43,9 @@ func readConfig() {
func newCmd() *cobra.Command {
var (
flagQuiet bool
flagVerbose bool
flagQuiet bool
flagVerbose int
flagVerboseName = "verbose"
)
cmd := &cobra.Command{
Use: "linuxkit",
@@ -54,7 +55,7 @@ func newCmd() *cobra.Command {
readConfig()
// Set up logging
return util.SetupLogging(flagQuiet, flagVerbose)
return util.SetupLogging(flagQuiet, flagVerbose, cmd.Flag(flagVerboseName).Changed)
},
}
@@ -69,7 +70,7 @@ func newCmd() *cobra.Command {
cmd.PersistentFlags().StringVar(&cacheDir, "cache", defaultLinuxkitCache(), fmt.Sprintf("Directory for caching and finding cached image, overrides env var %s", envVarCacheDir))
cmd.PersistentFlags().BoolVarP(&flagQuiet, "quiet", "q", false, "Quiet execution")
cmd.PersistentFlags().BoolVarP(&flagVerbose, "verbose", "v", false, "Verbose execution")
cmd.PersistentFlags().IntVarP(&flagVerbose, flagVerboseName, "v", 1, "Verbosity of logging: 0 = quiet, 1 = info, 2 = debug, 3 = trace. Default is info. Setting it explicitly will create structured logging lines.")
return cmd
}

View File

@@ -3,5 +3,4 @@ package main
const (
defaultPkgBuildYML = "build.yml"
defaultPkgCommit = "HEAD"
defaultPkgTag = "{{.Hash}}"
)

View File

@@ -3,6 +3,7 @@ package docker
import (
"context"
"errors"
"fmt"
"io"
"os"
"sync"
@@ -51,13 +52,19 @@ func createClient() (*client.Client, error) {
}
// HasImage check if the provided ref is available in the docker cache.
func HasImage(ref *reference.Spec) error {
func HasImage(ref *reference.Spec, architecture string) error {
log.Debugf("docker inspect image: %s", ref)
cli, err := Client()
if err != nil {
return err
}
_, err = InspectImage(cli, ref)
imageInspect, err := InspectImage(cli, ref)
if err != nil {
return err
}
if imageInspect.Architecture != "" && imageInspect.Architecture != architecture {
return fmt.Errorf("image not found for right architecture (%s != %s)", imageInspect.Architecture, architecture)
}
return err
}

View File

@@ -14,7 +14,7 @@ require (
github.com/containerd/containerd v1.7.15
github.com/docker/buildx v0.14.1
github.com/docker/cli v26.1.3+incompatible
github.com/docker/docker v27.0.3+incompatible
github.com/docker/docker v27.2.0+incompatible
github.com/docker/go-units v0.5.0
github.com/google/go-containerregistry v0.14.0
github.com/google/uuid v1.6.0
@@ -54,9 +54,9 @@ require (
github.com/Code-Hex/vz/v3 v3.0.0
github.com/equinix/equinix-sdk-go v0.42.0
github.com/in-toto/in-toto-golang v0.5.0
github.com/moby/sys/capability v0.3.0
github.com/spdx/tools-golang v0.5.3
github.com/spf13/cobra v1.8.0
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635
gopkg.in/yaml.v3 v3.0.1
)
@@ -102,6 +102,7 @@ require (
github.com/gorilla/websocket v1.5.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.16.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/klauspost/compress v1.17.4 // indirect

View File

@@ -103,8 +103,8 @@ github.com/docker/cli v26.1.3+incompatible h1:bUpXT/N0kDE3VUHI2r5VMsYQgi38kYuoC0
github.com/docker/cli v26.1.3+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/distribution v2.8.2+incompatible h1:T3de5rq0dB1j30rp0sA2rER+m322EBzniBPB6ZIzuh8=
github.com/docker/distribution v2.8.2+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
github.com/docker/docker v27.0.3+incompatible h1:aBGI9TeQ4MPlhquTQKq9XbK79rKFVwXNUAYz9aXyEBE=
github.com/docker/docker v27.0.3+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v27.2.0+incompatible h1:Rk9nIVdfH3+Vz4cyI/uhbINhEZ/oLmc+CBXmH6fbNk4=
github.com/docker/docker v27.2.0+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker-credential-helpers v0.8.0 h1:YQFtbBQb4VrpoPxhFuzEBPQ9E16qz5SpHLS+uswaCp8=
github.com/docker/docker-credential-helpers v0.8.0/go.mod h1:UGFXcuoQ5TxPiB54nHOZ32AWRqQdECoh/Mg0AlEYb40=
github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
@@ -203,6 +203,8 @@ github.com/grpc-ecosystem/go-grpc-middleware v1.3.0/go.mod h1:z0ButlSOZa5vEBq9m2
github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.16.0 h1:YBftPWNWd4WwGqtY2yeZL2ef8rHAxPBD8KFhJpmcqms=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.16.0/go.mod h1:YN5jB8ie0yfIUg6VvR9Kz84aCaG7AsGZnLjhHbUqwPg=
github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
github.com/hashicorp/go-version v1.2.0 h1:3vNe/fWF5CBgRIguda1meWhsZHy3m8gCJ5wx+dIzX/E=
github.com/hashicorp/go-version v1.2.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/in-toto/in-toto-golang v0.5.0 h1:hb8bgwr0M2hGdDsLjkJ3ZqJ8JFLL/tgYdAxF/XEFBbY=
@@ -243,6 +245,8 @@ github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/capability v0.3.0 h1:kEP+y6te0gEXIaeQhIi0s7vKs/w0RPoH1qPa6jROcVg=
github.com/moby/sys/capability v0.3.0/go.mod h1:4g9IK291rVkms3LKCDOoYlnV8xKwoDTpIrNEE35Wq0I=
github.com/moby/sys/mountinfo v0.7.1 h1:/tTvQaSJRr2FshkhXiIpux6fQ2Zvc4j7tAhMTStAG2g=
github.com/moby/sys/mountinfo v0.7.1/go.mod h1:IJb6JQeOklcdMU9F5xQ8ZALD+CUr5VlGpwtX+VE0rpI=
github.com/moby/sys/sequential v0.5.0 h1:OPvI35Lzn9K04PBbCLW0g4LcFAJgHsvXsRyewg5lXtc=
@@ -327,8 +331,6 @@ github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsT
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/surma/gocpio v1.0.2-0.20160926205914-fcb68777e7dc h1:iA3Eg1OVd2o0M4M+0PBsBBssMz98L8CUH7x0xVkuyUA=
github.com/surma/gocpio v1.0.2-0.20160926205914-fcb68777e7dc/go.mod h1:zaLNaN+EDnfSnNdWPJJf9OZxWF817w5dt8JNzF9LCVI=
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 h1:kdXcSzyDtseVEc4yCz2qF8ZrQvIDBJLl4S1c3GCXmoI=
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
github.com/tonistiigi/fsutil v0.0.0-20240424095704-91a3fc46842c h1:+6wg/4ORAbnSoGDzg2Q1i3CeMcT/jjhye/ZfnBHy7/M=
github.com/tonistiigi/fsutil v0.0.0-20240424095704-91a3fc46842c/go.mod h1:vbbYqJlnswsbJqWUcJN8fKtBhnEgldDrcagTgnBVKKM=
github.com/tonistiigi/units v0.0.0-20180711220420-6950e57a87ea h1:SXhTLE6pb6eld/v/cCndK0AMpt1wiVFb/YYmqB3/QG0=

View File

@@ -16,6 +16,9 @@ import (
"strings"
"github.com/containerd/containerd/reference"
v1 "github.com/google/go-containerregistry/pkg/v1"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
// drop-in 100% compatible replacement and 17% faster than compress/gzip.
gzip "github.com/klauspost/pgzip"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/moby"
@@ -84,6 +87,8 @@ func OutputTypes() []string {
return ts
}
// outputImage given an image and a section, such as onboot, onshutdown or services, lay it out with correct location
// config, etc. in the filesystem, so runc can use it.
func outputImage(image *moby.Image, section string, index int, prefix string, m moby.Moby, idMap map[string]uint32, dupMap map[string]string, iw *tar.Writer, opts BuildOpts) error {
log.Infof(" Create OCI config for %s", image.Image)
imageName := util.ReferenceExpand(image.Image)
@@ -91,7 +96,7 @@ func outputImage(image *moby.Image, section string, index int, prefix string, m
if err != nil {
return fmt.Errorf("could not resolve references for image %s: %v", image.Image, err)
}
src, err := imagePull(&ref, opts.Pull, opts.CacheDir, opts.DockerCache, opts.Arch)
src, err := imageSource(&ref, opts.Pull, opts.CacheDir, opts.DockerCache, imagespec.Platform{OS: "linux", Architecture: opts.Arch})
if err != nil {
return fmt.Errorf("could not pull image %s: %v", image.Image, err)
}
@@ -280,29 +285,46 @@ func Build(m moby.Moby, w io.Writer, opts BuildOpts) error {
lowerPath := strings.TrimPrefix(lower, "/") + "/"
// get volume tarball from container
if err := ImageTar(location, vol.ImageRef(), lowerPath, apkTar, resolvconfSymlink, opts); err != nil {
return fmt.Errorf("failed to build volume tarball from %s: %v", vol.Name, err)
switch {
case vol.ImageRef() == nil || vol.Format == "" || vol.Format == "filesystem":
if err := ImageTar(location, vol.ImageRef(), lowerPath, apkTar, resolvconfSymlink, opts); err != nil {
return fmt.Errorf("failed to build volume filesystem tarball from %s: %v", vol.Name, err)
}
case vol.Format == "oci":
// convert platforms into imagespec platforms
platforms := make([]imagespec.Platform, len(vol.Platforms))
for i, p := range vol.Platforms {
platform, err := v1.ParsePlatform(p)
if err != nil {
return fmt.Errorf("failed to parse platform %s: %v", p, err)
}
platforms[i] = imagespec.Platform{
Architecture: platform.Architecture,
OS: platform.OS,
Variant: platform.Variant,
}
}
if err := ImageOCITar(location, vol.ImageRef(), lowerPath, apkTar, opts, platforms); err != nil {
return fmt.Errorf("failed to build volume OCI v1 layout tarball from %s: %v", vol.Name, err)
}
}
// make upper and merged dirs which will be used for mounting
// no need to make lower dir, as it is made automatically by ImageTar()
// the existence of an upper dir indicates that it is read-write and should be overlayfs.
if !vol.ReadOnly {
// need the tmp dir where work gets done, since the whole thing fs is read-only
tmpPath := strings.TrimPrefix(tmpDir, "/") + "/"
tmphdr := &tar.Header{
Name: tmpPath,
Mode: 0755,
Typeflag: tar.TypeDir,
ModTime: defaultModTime,
Format: tar.FormatPAX,
PAXRecords: map[string]string{
moby.PaxRecordLinuxkitSource: "linuxkit.volumes",
moby.PaxRecordLinuxkitLocation: location,
},
}
if err := apkTar.WriteHeader(tmphdr); err != nil {
return err
}
tmpPath := strings.TrimPrefix(tmpDir, "/") + "/"
tmphdr := &tar.Header{
Name: tmpPath,
Mode: 0755,
Typeflag: tar.TypeDir,
ModTime: defaultModTime,
Format: tar.FormatPAX,
PAXRecords: map[string]string{
moby.PaxRecordLinuxkitSource: "linuxkit.volumes",
moby.PaxRecordLinuxkitLocation: location,
},
}
if err := apkTar.WriteHeader(tmphdr); err != nil {
return err
}
mergedPath := strings.TrimPrefix(merged, "/") + "/"
mhdr := &tar.Header{

View File

@@ -11,6 +11,7 @@ import (
"github.com/containerd/containerd/reference"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/moby"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/opencontainers/runtime-spec/specs-go"
log "github.com/sirupsen/logrus"
)
@@ -175,14 +176,15 @@ func tarPrefix(path, location, refName string, tw tarWriter) error {
return nil
}
// ImageTar takes a Docker image and outputs it to a tar stream
// ImageTar takes a Docker image and outputs it to a tar stream as a merged filesystem for a specific architecture
// defined in opts.
// location is where it is in the linuxkit.yaml file
func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter, resolv string, opts BuildOpts) (e error) {
refName := "empty"
if ref != nil {
refName = ref.String()
}
log.Debugf("image tar: %s %s", refName, prefix)
log.Debugf("image filesystem tar: %s %s %s", refName, prefix, opts.Arch)
if prefix != "" && prefix[len(prefix)-1] != '/' {
return fmt.Errorf("prefix does not end with /: %s", prefix)
}
@@ -197,9 +199,8 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
return nil
}
// pullImage first checks in the cache, then pulls the image.
// If pull==true, then it always tries to pull from registry.
src, err := imagePull(ref, opts.Pull, opts.CacheDir, opts.DockerCache, opts.Arch)
// get a handle on the image, optionally from docker, pulling from registry if necessary.
src, err := imageSource(ref, opts.Pull, opts.CacheDir, opts.DockerCache, imagespec.Platform{OS: "linux", Architecture: opts.Arch})
if err != nil {
return fmt.Errorf("could not pull image %s: %v", ref, err)
}
@@ -237,7 +238,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.PAXRecords[moby.PaxRecordLinuxkitSource] = ref.String()
hdr.PAXRecords[moby.PaxRecordLinuxkitLocation] = location
if exclude[hdr.Name] {
log.Debugf("image tar: %s %s exclude %s", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s exclude %s", ref, prefix, hdr.Name)
_, err = io.Copy(io.Discard, tr)
if err != nil {
return err
@@ -248,7 +249,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.Size = int64(len(contents))
hdr.Name = prefix + hdr.Name
hdr.ModTime = defaultModTime
log.Debugf("image tar: %s %s add %s (replaced)", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s add %s (replaced)", ref, prefix, hdr.Name)
if err := tw.WriteHeader(hdr); err != nil {
return err
}
@@ -263,7 +264,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.Typeflag = tar.TypeSymlink
hdr.Linkname = resolv
hdr.ModTime = defaultModTime
log.Debugf("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
log.Tracef("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
if err := tw.WriteHeader(hdr); err != nil {
return err
}
@@ -274,12 +275,12 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
}
} else {
if found, ok := touch[hdr.Name]; ok {
log.Debugf("image tar: %s %s add %s (touch)", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s add %s (touch)", ref, prefix, hdr.Name)
hdr.ModTime = found.ModTime
// record that we saw this one
touchFound[hdr.Name] = true
} else {
log.Debugf("image tar: %s %s add %s (original)", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s add %s (original)", ref, prefix, hdr.Name)
}
hdr.Name = prefix + hdr.Name
if hdr.Typeflag == tar.TypeLink {
@@ -304,7 +305,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
sort.Strings(touchNames)
for _, name := range touchNames {
if touchFound[name] {
log.Debugf("image tar: %s already found in original image", name)
log.Tracef("image tar: %s already found in original image", name)
continue
}
hdr := touch[name]
@@ -326,9 +327,9 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.Size = 0
hdr.Typeflag = tar.TypeSymlink
hdr.Linkname = resolv
log.Debugf("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
log.Tracef("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
}
log.Debugf("image tar: creating %s", name)
log.Tracef("image tar: creating %s", name)
if err := tw.WriteHeader(&hdr); err != nil {
return err
}
@@ -356,6 +357,79 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
return nil
}
// ImageOCITar takes an OCI image and outputs it to a tar stream as a v1 layout format.
// Will include all architectures, or, if specific ones provided, then only those.
// location is where it is in the linuxkit.yaml file
func ImageOCITar(location string, ref *reference.Spec, prefix string, tw tarWriter, opts BuildOpts, platforms []imagespec.Platform) (e error) {
refName := "empty"
if ref != nil {
refName = ref.String()
}
log.Debugf("image v1 layout tar: %s %s %s", refName, prefix, opts.Arch)
if prefix != "" && prefix[len(prefix)-1] != '/' {
return fmt.Errorf("prefix does not end with /: %s", prefix)
}
err := tarPrefix(prefix, location, refName, tw)
if err != nil {
return err
}
// if the image is blank, we do not need to do any more
if ref == nil {
return fmt.Errorf("no image reference provided")
}
// indexSource first checks in the cache, then pulls the image.
// If pull==true, then it always tries to pull from registry.
src, err := indexSource(ref, opts.Pull, opts.CacheDir, platforms)
if err != nil {
return fmt.Errorf("could not pull image %s: %v", ref, err)
}
contents, err := src.OCITarReader("")
if err != nil {
return fmt.Errorf("could not unpack image %s: %v", ref, err)
}
defer contents.Close()
tr := tar.NewReader(contents)
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return err
}
// force PAX format, since it allows for unlimited Name/Linkname
// and we move all files below prefix.
hdr.Format = tar.FormatPAX
// ensure we record the source of the file in the PAX header
if hdr.PAXRecords == nil {
hdr.PAXRecords = make(map[string]string)
}
hdr.PAXRecords[moby.PaxRecordLinuxkitSource] = ref.String()
hdr.PAXRecords[moby.PaxRecordLinuxkitLocation] = location
hdr.Name = prefix + hdr.Name
if hdr.Typeflag == tar.TypeLink {
// hard links are referenced by full path so need to be adjusted
hdr.Linkname = prefix + hdr.Linkname
}
if err := tw.WriteHeader(hdr); err != nil {
return err
}
_, err = io.Copy(tw, tr)
if err != nil {
return err
}
}
return nil
}
// ImageBundle produces an OCI bundle at the given path in a tarball, given an image and a config.json
func ImageBundle(prefix, location string, ref *reference.Spec, config []byte, runtime moby.Runtime, tw tarWriter, readonly bool, dupMap map[string]string, opts BuildOpts) error { // nolint: lll
// if read only, just unpack in rootfs/ but otherwise set up for overlay
@@ -489,7 +563,7 @@ func ImageBundle(prefix, location string, ref *reference.Spec, config []byte, ru
return err
}
log.Debugf("image bundle: %s %s cfg: %s runtime: %s", prefix, ref, string(config), string(runtimeConfig))
log.Tracef("image bundle: %s %s cfg: %s runtime: %s", prefix, ref, string(config), string(runtimeConfig))
return nil
}

View File

@@ -5,20 +5,23 @@ import (
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/cache"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/docker"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// imagePull pull an image from the OCI registry to the cache.
// If the image root already is in the cache, use it, unless
// the option pull is set to true.
// if alwaysPull, then do not even bother reading locally
func imagePull(ref *reference.Spec, alwaysPull bool, cacheDir string, dockerCache bool, architecture string) (lktspec.ImageSource, error) {
// imageSource given an image ref, get a handle on the image so it can be used as a source for its configuration
// and layers. If the image root already is in the cache, use it.
// If not in cache, pull it down from the OCI registry.
// Optionally can look in docker image cache first, before falling back to linuxkit cache and OCI registry.
// Optionally can be told to alwaysPull, in which case it always pulls from the OCI registry.
// Always works for a single architecture, as we are referencing a specific image.
func imageSource(ref *reference.Spec, alwaysPull bool, cacheDir string, dockerCache bool, platform imagespec.Platform) (lktspec.ImageSource, error) {
// several possibilities:
// - alwaysPull: try to pull it down from the registry to linuxkit cache, then fail
// - !alwaysPull && dockerCache: try to read it from docker, then try linuxkit cache, then try to pull from registry, then fail
// - !alwaysPull && !dockerCache: try linuxkit cache, then try to pull from registry, then fail
// first, try docker, if that is available
if !alwaysPull && dockerCache {
if err := docker.HasImage(ref); err == nil {
if err := docker.HasImage(ref, platform.Architecture); err == nil {
return docker.NewSource(ref), nil
}
// docker is not required, so any error - image not available, no docker, whatever - just gets ignored
@@ -31,5 +34,44 @@ func imagePull(ref *reference.Spec, alwaysPull bool, cacheDir string, dockerCach
}
// if we made it here, we either did not have the image, or it was incomplete
return c.ImagePull(ref, ref.String(), architecture, alwaysPull)
if err := c.ImagePull(ref, []imagespec.Platform{platform}, alwaysPull); err != nil {
return nil, err
}
desc, err := c.FindDescriptor(ref)
if err != nil {
return nil, err
}
return c.NewSource(
ref,
&platform,
desc,
), nil
}
// indexSource given an image ref, get a handle on the index so it can be used as a source for its underlying images.
// If the index root already is in the cache, use it.
// If not in cache, pull it down from the OCI registry.
// Optionally can look in docker image cache first, before falling back to linuxkit cache and OCI registry.
// Optionally can be told to alwaysPull, in which case it always pulls from the OCI registry.
// Can provide architectures to list which ones to limit, or leave empty for all available.
func indexSource(ref *reference.Spec, alwaysPull bool, cacheDir string, platforms []imagespec.Platform) (lktspec.IndexSource, error) {
// get a reference to the local cache; we either will find the ref there or will pull to it
c, err := cache.NewProvider(cacheDir)
if err != nil {
return nil, err
}
// if we made it here, we either did not have the image, or it was incomplete
if err := c.ImagePull(ref, platforms, alwaysPull); err != nil {
return nil, err
}
desc, err := c.FindDescriptor(ref)
if err != nil {
return nil, err
}
return c.NewIndexSource(
ref,
desc,
platforms,
), nil
}

View File

@@ -1,12 +1,12 @@
iso: linuxkit/mkimage-iso:08d19f8acf285bdce65dd4aea24f01d8adbedfbc
iso-bios: linuxkit/mkimage-iso-bios:96d5dac296345c308b8ad9e6cae7467e76ba8fd1
iso-efi: linuxkit/mkimage-iso-efi:cf2a3ff1dacfcacfb1f165abc210d9e33dc9d161
iso-efi-initrd: linuxkit/mkimage-iso-efi-initrd:a9e61bc810ae9928bab92f41f6e810e5b4f6183a
iso-efi: linuxkit/mkimage-iso-efi:8b538605a581db7523c021bf92a715d2054f609e
iso-efi-initrd: linuxkit/mkimage-iso-efi-initrd:d390030aae1069f3142523e9f433aad946838911
raw-bios: linuxkit/mkimage-raw-bios:4c21d66c81fd3641c62b9e80ddf5494000a1a442
raw-efi: linuxkit/mkimage-raw-efi:df0979572e8d0251a5cf55b6f73131868d036bb4
raw-efi: linuxkit/mkimage-raw-efi:14b66a308b2047c59b0fe7c43996f73c653a9fcd
squashfs: linuxkit/mkimage-squashfs:a61fd76227ab4998d6c1ba17229cd8bd749e8f13
gcp: linuxkit/mkimage-gcp:035c2c2b4b958060c0b6bdd41d9cbc886a335098
qcow2-efi: linuxkit/mkimage-qcow2-efi:98a6e3e7b6eed965f879cd77c009c7c404d4457d
qcow2-efi: linuxkit/mkimage-qcow2-efi:8b76f4118e6640db6f39196dc365529f401e7670
vhd: linuxkit/mkimage-vhd:91bcc7a6475f46a3d5d84cf6161f07c583dd9c21
dynamic-vhd: linuxkit/mkimage-dynamic-vhd:b755f8ff82c8631d18decaebb09867e7b88c2533
vmdk: linuxkit/mkimage-vmdk:20a370a55bd8d58c2ae9d634c297a955bb006efd

View File

@@ -2,7 +2,7 @@ kernel:
image: linuxkit/kernel:6.6.13
cmdline: "console=ttyS0"
init:
- linuxkit/init:872d2e1be745f1acb948762562cf31c367303a3b
- linuxkit/init:e120ea2a30d906bd1ee1874973d6e4b1403b5ca3
- linuxkit/runc:6062483d748609d505f2bcde4e52ee64a3329f5f
onboot:
- name: mkimage

View File

@@ -13,10 +13,10 @@ import (
"github.com/containerd/containerd/reference"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
"github.com/moby/sys/capability"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/opencontainers/runtime-spec/specs-go"
log "github.com/sirupsen/logrus"
"github.com/syndtr/gocapability/capability"
"github.com/xeipuuv/gojsonschema"
"gopkg.in/yaml.v3"
)
@@ -76,10 +76,12 @@ type File struct {
// Volume is the type of a volume specification
type Volume struct {
Name string `yaml:"name" json:"name"`
Image string `yaml:"image,omitempty" json:"image,omitempty"`
ReadOnly bool `yaml:"readonly,omitempty" json:"readonly,omitempty"`
ref *reference.Spec
Name string `yaml:"name" json:"name"`
Image string `yaml:"image,omitempty" json:"image,omitempty"`
ReadOnly bool `yaml:"readonly,omitempty" json:"readonly,omitempty"`
Format string `yaml:"format,omitempty" json:"format,omitempty"`
Platforms []string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
ref *reference.Spec
}
func (v Volume) ImageRef() *reference.Spec {
@@ -446,7 +448,7 @@ func AppendConfig(m0, m1 Moby) (Moby, error) {
// NewImage validates an parses yaml or json for a Image
func NewImage(config []byte) (Image, error) {
log.Debugf("Reading label config: %s", string(config))
log.Tracef("Reading label config: %s", string(config))
mi := Image{}
@@ -781,7 +783,7 @@ func assignStringEmpty4(v1, v2, v3, v4 string) string {
func getAllCapabilities() []string {
var caps []string
for _, cap := range capability.List() {
for _, cap := range capability.ListKnown() {
caps = append(caps, "CAP_"+strings.ToUpper(cap.String()))
}
return caps

View File

@@ -43,7 +43,9 @@ var schema = `
"properties": {
"name": {"type": "string"},
"image": {"type": "string"},
"readonly": {"type": "boolean"}
"readonly": {"type": "boolean"},
"format": {"enum": ["oci","filesystem"]},
"platforms": {"$ref": "#/definitions/strings"}
}
},
"volumes": {

View File

@@ -37,7 +37,6 @@ func pkgCmd() *cobra.Command {
HashPath: hashPath,
Dirty: dirty,
Dev: devMode,
Tag: tag,
}
if cmd.Flags().Changed("disable-cache") && cmd.Flags().Changed("enable-cache") {
return errors.New("cannot set but disable-cache and enable-cache")
@@ -65,6 +64,9 @@ func pkgCmd() *cobra.Command {
if cmd.Flags().Changed("org") {
pkglibConfig.Org = &argOrg
}
if cmd.Flags().Changed("tag") {
pkglibConfig.Tag = tag
}
return nil
},
@@ -88,7 +90,7 @@ func pkgCmd() *cobra.Command {
cmd.PersistentFlags().StringVar(&argOrg, "org", piBase.Org, "Override the hub org")
cmd.PersistentFlags().StringVar(&buildYML, "build-yml", defaultPkgBuildYML, "Override the name of the yml file")
cmd.PersistentFlags().StringVar(&hash, "hash", "", "Override the image hash (default is to query git for the package's tree-sh)")
cmd.PersistentFlags().StringVar(&tag, "tag", defaultPkgTag, "Override the tag using fixed strings and/or text templates. Acceptable are .Hash for the hash")
cmd.PersistentFlags().StringVar(&tag, "tag", piBase.Tag, "Override the tag using fixed strings and/or text templates. Acceptable are .Hash for the hash")
cmd.PersistentFlags().StringVar(&hashCommit, "hash-commit", defaultPkgCommit, "Override the git commit to use for the hash")
cmd.PersistentFlags().StringVar(&hashPath, "hash-path", "", "Override the directory to use for the image hash, must be a parent of the package dir (default is to use the package dir)")
cmd.PersistentFlags().BoolVar(&dirty, "force-dirty", false, "Force the pkg(s) to be considered dirty")

View File

@@ -49,6 +49,7 @@ func addCmdRunPkgBuildPush(cmd *cobra.Command, withPush bool) *cobra.Command {
dockerfile string
buildArgFiles []string
progress string
ssh []string
)
cmd.RunE = func(cmd *cobra.Command, args []string) error {
@@ -162,6 +163,9 @@ func addCmdRunPkgBuildPush(cmd *cobra.Command, withPush bool) *cobra.Command {
opts = append(opts, pkglib.WithBuildBuilderImage(builderImage))
opts = append(opts, pkglib.WithBuildBuilderRestart(builderRestart))
opts = append(opts, pkglib.WithProgress(progress))
if len(ssh) > 0 {
opts = append(opts, pkglib.WithSSH(ssh))
}
for _, p := range pkgs {
// things we need our own copies of
@@ -229,6 +233,7 @@ func addCmdRunPkgBuildPush(cmd *cobra.Command, withPush bool) *cobra.Command {
cmd.Flags().StringVar(&dockerfile, "dockerfile", "", "Dockerfile to use for building the image, must be in this directory or below, overrides what is in build.yml")
cmd.Flags().StringArrayVar(&buildArgFiles, "build-arg-file", nil, "Files containing build arguments, one key=value per line, contents augment and override buildArgs in build.yml. Can be specified multiple times. File is relative to working directory when running `linuxkit pkg build`")
cmd.Flags().StringVar(&progress, "progress", "auto", "Set type of progress output (auto, plain, tty). Use plain to show container output, tty for interactive build")
cmd.Flags().StringArrayVar(&ssh, "ssh", nil, "SSH agent config to use for build, follows the syntax used for buildx and buildctl, see https://docs.docker.com/reference/dockerfile/#run---mounttypessh")
return cmd
}

View File

@@ -13,9 +13,9 @@ import (
"strings"
"github.com/containerd/containerd/reference"
"github.com/docker/docker/api/types"
registry "github.com/google/go-containerregistry/pkg/v1"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/cache"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/version"
@@ -46,6 +46,7 @@ type buildOpts struct {
dockerfile string
buildArgs []string
progress string
ssh []string
}
// BuildOpt allows callers to specify options to Build
@@ -215,6 +216,14 @@ func WithProgress(progress string) BuildOpt {
}
}
// WithSSH sets up the package to use SSH in the build
func WithSSH(ssh []string) BuildOpt {
return func(bo *buildOpts) error {
bo.ssh = ssh
return nil
}
}
// Build builds the package
func (p Pkg) Build(bos ...BuildOpt) error {
var bo buildOpts
@@ -318,7 +327,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
case bo.pull:
// need to pull the image from the registry, else build
fmt.Fprintf(writer, "%s %s not found in local cache, trying to pull\n", ref, platform.Architecture)
if _, err := c.ImagePull(&ref, "", platform.Architecture, false); err == nil {
if err := c.ImagePull(&ref, []imagespec.Platform{platform}, false); err == nil {
fmt.Fprintf(writer, "%s pulled\n", ref)
// successfully pulled, no need to build, continue with next platform
continue
@@ -350,7 +359,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
}
fmt.Fprintf(writer, "building %s for arches: %s\n", ref, strings.Join(arches, ","))
var (
imageBuildOpts = types.ImageBuildOptions{
imageBuildOpts = spec.ImageBuildOptions{
Labels: map[string]string{},
BuildArgs: map[string]*string{},
}
@@ -440,6 +449,8 @@ func (p Pkg) Build(bos ...BuildOpt) error {
imageBuildOpts.BuildArgs["PKG_IMAGE"] = &ret
}
imageBuildOpts.SSH = bo.ssh
// build for each arch and save in the linuxkit cache
for _, platform := range platformsToBuild {
builtDescs, err := p.buildArch(ctx, d, c, bo.builderImage, platform.Architecture, bo.builderRestart, writer, bo, imageBuildOpts)
@@ -459,7 +470,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
// - potentially create a release, including push and load into docker
// create a multi-arch index
if _, err := c.IndexWrite(&ref, descs...); err != nil {
if err := c.IndexWrite(&ref, descs...); err != nil {
return err
}
}
@@ -479,7 +490,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
if err != nil {
return err
}
cacheSource := c.NewSource(&ref, platform.Architecture, desc)
cacheSource := c.NewSource(&ref, &platform, desc)
reader, err := cacheSource.V1TarReader(fmt.Sprintf("%s-%s", p.FullTag(), platform.Architecture))
if err != nil {
return fmt.Errorf("unable to get reader from cache: %v", err)
@@ -551,7 +562,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
if err != nil {
return err
}
if _, err := c.DescriptorWrite(&ref, *desc); err != nil {
if err := c.DescriptorWrite(&ref, *desc); err != nil {
return err
}
if err := c.Push(fullRelTag, "", bo.manifest, true); err != nil {
@@ -592,7 +603,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
// C - manifest, saved in cache as is, referenced by the index (E), and returned as a descriptor
// D - attestations (if any), saved in cache as is, referenced by the index (E), and returned as a descriptor
// E - index, saved in cache as is, stored in cache as tag "image:tag-arch", *not* returned as a descriptor
func (p Pkg) buildArch(ctx context.Context, d dockerRunner, c lktspec.CacheProvider, builderImage, arch string, restart bool, writer io.Writer, bo buildOpts, imageBuildOpts types.ImageBuildOptions) ([]registry.Descriptor, error) {
func (p Pkg) buildArch(ctx context.Context, d dockerRunner, c lktspec.CacheProvider, builderImage, arch string, restart bool, writer io.Writer, bo buildOpts, imageBuildOpts spec.ImageBuildOptions) ([]registry.Descriptor, error) {
var (
tagArch string
tag = p.FullTag()
@@ -606,7 +617,7 @@ func (p Pkg) buildArch(ctx context.Context, d dockerRunner, c lktspec.CacheProvi
if err != nil {
return nil, fmt.Errorf("could not resolve references for image %s: %v", p.Tag(), err)
}
if _, err := c.ImagePull(&ref, "", arch, false); err == nil {
if err := c.ImagePull(&ref, []imagespec.Platform{{Architecture: arch, OS: "linux"}}, false); err == nil {
fmt.Fprintf(writer, "image already found %s for arch %s", ref, arch)
desc, err := c.FindDescriptor(&ref)
if err != nil {
@@ -659,7 +670,9 @@ func (p Pkg) buildArch(ctx context.Context, d dockerRunner, c lktspec.CacheProvi
passCache = nil
}
if err := d.build(ctx, tagArch, p.path, bo.dockerfile, builderName, builderImage, platform, restart, passCache, buildCtx.Reader(), stdout, bo.sbomScan, bo.sbomScannerImage, bo.progress, imageBuildOpts); err != nil {
imageBuildOpts.Dockerfile = bo.dockerfile
if err := d.build(ctx, tagArch, p.path, builderName, builderImage, platform, restart, passCache, buildCtx.Reader(), stdout, bo.sbomScan, bo.sbomScannerImage, bo.progress, imageBuildOpts); err != nil {
stdoutCloser()
if strings.Contains(err.Error(), "executor failed running [/dev/.buildkit_qemu_emulator") {
return nil, fmt.Errorf("buildkit was unable to emulate %s. check binfmt has been set up and works for this platform: %v", platform, err)

View File

@@ -16,10 +16,10 @@ import (
"github.com/containerd/containerd/content"
"github.com/containerd/containerd/images"
"github.com/containerd/containerd/reference"
dockertypes "github.com/docker/docker/api/types"
registry "github.com/google/go-containerregistry/pkg/v1"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/types"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
buildkitClient "github.com/moby/buildkit/client"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
@@ -58,7 +58,7 @@ func (d *dockerMocker) contextSupportCheck() error {
func (d *dockerMocker) builder(_ context.Context, _, _, _ string, _ bool) (*buildkitClient.Client, error) {
return nil, fmt.Errorf("not implemented")
}
func (d *dockerMocker) build(ctx context.Context, tag, pkg, dockerfile, dockerContext, builderImage, platform string, builderRestart bool, c lktspec.CacheProvider, r io.Reader, stdout io.Writer, sbomScan bool, sbomScannerImage, progress string, imageBuildOpts dockertypes.ImageBuildOptions) error {
func (d *dockerMocker) build(ctx context.Context, tag, pkg, dockerContext, builderImage, platform string, builderRestart bool, c lktspec.CacheProvider, r io.Reader, stdout io.Writer, sbomScan bool, sbomScannerImage, progress string, imageBuildOpts spec.ImageBuildOptions) error {
if !d.enableBuild {
return errors.New("build disabled")
}
@@ -240,21 +240,24 @@ type cacheMocker struct {
hashes map[string][]byte
}
func (c *cacheMocker) ImagePull(ref *reference.Spec, trustedRef, architecture string, alwaysPull bool) (lktspec.ImageSource, error) {
func (c *cacheMocker) ImagePull(ref *reference.Spec, platforms []imagespec.Platform, alwaysPull bool) error {
if !c.enableImagePull {
return nil, errors.New("ImagePull disabled")
return errors.New("ImagePull disabled")
}
// make some random data for a layer
b := make([]byte, 256)
_, _ = rand.Read(b)
descs, err := c.imageWriteStream(bytes.NewReader(b))
if err != nil {
return nil, err
return err
}
if len(descs) != 1 {
return nil, fmt.Errorf("expected 1 descriptor, got %d", len(descs))
return fmt.Errorf("expected 1 descriptor, got %d", len(descs))
}
return c.NewSource(ref, architecture, &descs[1]), nil
if len(platforms) != 1 {
return fmt.Errorf("cache does not support multiple platforms %s", platforms)
}
return nil
}
func (c *cacheMocker) ImageInCache(ref *reference.Spec, trustedRef, architecture string) (bool, error) {
@@ -359,9 +362,9 @@ func (c *cacheMocker) imageWriteStream(r io.Reader) ([]registry.Descriptor, erro
return []registry.Descriptor{desc}, nil
}
func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.Descriptor) (lktspec.ImageSource, error) {
func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.Descriptor) error {
if !c.enableIndexWrite {
return nil, errors.New("disabled")
return errors.New("disabled")
}
image := ref.String()
im := registry.IndexManifest{
@@ -373,11 +376,11 @@ func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.De
// write the updated index, remove the old one
b, err := json.Marshal(im)
if err != nil {
return nil, fmt.Errorf("unable to marshal new index to json: %v", err)
return fmt.Errorf("unable to marshal new index to json: %v", err)
}
hash, size, err := registry.SHA256(bytes.NewReader(b))
if err != nil {
return nil, fmt.Errorf("error calculating hash of index json: %v", err)
return fmt.Errorf("error calculating hash of index json: %v", err)
}
c.assignHash(hash.String(), b)
desc := registry.Descriptor{
@@ -390,7 +393,7 @@ func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.De
}
c.appendImage(image, desc)
return c.NewSource(ref, "", &desc), nil
return nil
}
func (c *cacheMocker) Push(name, remoteName string, withManifest, override bool) error {
if !c.enablePush {
@@ -402,9 +405,9 @@ func (c *cacheMocker) Push(name, remoteName string, withManifest, override bool)
return nil
}
func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descriptor) (lktspec.ImageSource, error) {
func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descriptor) error {
if !c.enabledDescriptorWrite {
return nil, errors.New("descriptor disabled")
return errors.New("descriptor disabled")
}
var (
image = ref.String()
@@ -417,11 +420,11 @@ func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descrip
// write the updated index, remove the old one
b, err := json.Marshal(im)
if err != nil {
return nil, fmt.Errorf("unable to marshal new index to json: %v", err)
return fmt.Errorf("unable to marshal new index to json: %v", err)
}
hash, size, err := registry.SHA256(bytes.NewReader(b))
if err != nil {
return nil, fmt.Errorf("error calculating hash of index json: %v", err)
return fmt.Errorf("error calculating hash of index json: %v", err)
}
c.assignHash(hash.String(), b)
root := registry.Descriptor{
@@ -434,7 +437,7 @@ func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descrip
}
c.appendImage(image, root)
return c.NewSource(ref, "", &root), nil
return nil
}
func (c *cacheMocker) FindDescriptor(ref *reference.Spec) (*registry.Descriptor, error) {
name := ref.String()
@@ -443,8 +446,8 @@ func (c *cacheMocker) FindDescriptor(ref *reference.Spec) (*registry.Descriptor,
}
return nil, fmt.Errorf("not found %s", name)
}
func (c *cacheMocker) NewSource(ref *reference.Spec, architecture string, descriptor *registry.Descriptor) lktspec.ImageSource {
return cacheMockerSource{c, ref, architecture, descriptor}
func (c *cacheMocker) NewSource(ref *reference.Spec, platform *imagespec.Platform, descriptor *registry.Descriptor) lktspec.ImageSource {
return cacheMockerSource{c, ref, platform, descriptor}
}
func (c *cacheMocker) assignHash(hash string, b []byte) {
if c.hashes == nil {
@@ -473,10 +476,10 @@ func (c *cacheMocker) GetContent(hash v1.Hash) (io.ReadCloser, error) {
}
type cacheMockerSource struct {
c *cacheMocker
ref *reference.Spec
architecture string
descriptor *registry.Descriptor
c *cacheMocker
ref *reference.Spec
platform *imagespec.Platform
descriptor *registry.Descriptor
}
func (c cacheMockerSource) Config() (imagespec.ImageConfig, error) {

View File

@@ -30,7 +30,9 @@ import (
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
buildkitClient "github.com/moby/buildkit/client"
"github.com/moby/buildkit/cmd/buildctl/build"
"github.com/moby/buildkit/frontend/dockerui"
"github.com/moby/buildkit/session"
"github.com/moby/buildkit/util/progress/progressui"
// golint requires comments on non-main(test)
@@ -40,6 +42,7 @@ import (
"github.com/moby/buildkit/frontend/dockerfile/instructions"
"github.com/moby/buildkit/frontend/dockerfile/parser"
"github.com/moby/buildkit/frontend/dockerfile/shell"
"github.com/moby/buildkit/session/sshforward/sshprovider"
"github.com/moby/buildkit/session/upload/uploadprovider"
log "github.com/sirupsen/logrus"
)
@@ -54,7 +57,7 @@ const (
type dockerRunner interface {
tag(ref, tag string) error
build(ctx context.Context, tag, pkg, dockerfile, dockerContext, builderImage, platform string, restart bool, c spec.CacheProvider, r io.Reader, stdout io.Writer, sbomScan bool, sbomScannerImage, platformType string, imageBuildOpts types.ImageBuildOptions) error
build(ctx context.Context, tag, pkg, dockerContext, builderImage, platform string, restart bool, c spec.CacheProvider, r io.Reader, stdout io.Writer, sbomScan bool, sbomScannerImage, platformType string, imageBuildOpts spec.ImageBuildOptions) error
save(tgt string, refs ...string) error
load(src io.Reader) error
pull(img string) (bool, error)
@@ -403,7 +406,7 @@ func (dr *dockerRunnerImpl) tag(ref, tag string) error {
return dr.command(nil, nil, nil, "image", "tag", ref, tag)
}
func (dr *dockerRunnerImpl) build(ctx context.Context, tag, pkg, dockerfile, dockerContext, builderImage, platform string, restart bool, c spec.CacheProvider, stdin io.Reader, stdout io.Writer, sbomScan bool, sbomScannerImage, progressType string, imageBuildOpts types.ImageBuildOptions) error {
func (dr *dockerRunnerImpl) build(ctx context.Context, tag, pkg, dockerContext, builderImage, platform string, restart bool, c spec.CacheProvider, stdin io.Reader, stdout io.Writer, sbomScan bool, sbomScannerImage, progressType string, imageBuildOpts spec.ImageBuildOptions) error {
// ensure we have a builder
client, err := dr.builder(ctx, dockerContext, builderImage, platform, restart)
if err != nil {
@@ -453,6 +456,32 @@ func (dr *dockerRunnerImpl) build(ctx context.Context, tag, pkg, dockerfile, doc
frontendAttrs[sbomFrontEndKey] = sbomValue
}
attachable := []session.Attachable{}
localDirs := map[string]string{}
if len(imageBuildOpts.SSH) > 0 {
configs, err := build.ParseSSH(imageBuildOpts.SSH)
if err != nil {
return err
}
sp, err := sshprovider.NewSSHAgentProvider(configs)
if err != nil {
return err
}
attachable = append(attachable, sp)
}
if stdin != nil {
buf := bufio.NewReader(stdin)
up := uploadprovider.New()
frontendAttrs["context"] = up.Add(buf)
attachable = append(attachable, up)
} else {
localDirs[dockerui.DefaultLocalNameDockerfile] = pkg
localDirs[dockerui.DefaultLocalNameContext] = pkg
}
solveOpts := buildkitClient.SolveOpt{
Frontend: "dockerfile.v0",
FrontendAttrs: frontendAttrs,
@@ -465,24 +494,15 @@ func (dr *dockerRunnerImpl) build(ctx context.Context, tag, pkg, dockerfile, doc
Output: fixedWriteCloser(&writeNopCloser{stdout}),
},
},
Session: attachable,
LocalDirs: localDirs,
}
if stdin != nil {
buf := bufio.NewReader(stdin)
up := uploadprovider.New()
frontendAttrs["context"] = up.Add(buf)
solveOpts.Session = append(solveOpts.Session, up)
} else {
solveOpts.LocalDirs = map[string]string{
dockerui.DefaultLocalNameDockerfile: pkg,
dockerui.DefaultLocalNameContext: pkg,
}
}
frontendAttrs["filename"] = dockerfile
frontendAttrs["filename"] = imageBuildOpts.Dockerfile
// go through the dockerfile to see if we have any provided images cached
if c != nil {
dockerfileRef := path.Join(pkg, dockerfile)
dockerfileRef := path.Join(pkg, imageBuildOpts.Dockerfile)
f, err := os.Open(dockerfileRef)
if err != nil {
return fmt.Errorf("error opening dockerfile %s: %v", dockerfileRef, err)

View File

@@ -20,6 +20,7 @@ import (
type pkgInfo struct {
Image string `yaml:"image"`
Org string `yaml:"org"`
Tag string `yaml:"tag,omitempty"` // default to {{.Hash}}
Dockerfile string `yaml:"dockerfile"`
Arches []string `yaml:"arches"`
ExtraSources []string `yaml:"extra-sources"`
@@ -60,6 +61,7 @@ func NewPkgInfo() pkgInfo {
return pkgInfo{
Org: "linuxkit",
Arches: []string{"amd64", "arm64"},
Tag: "{{.Hash}}",
GitRepo: "https://github.com/linuxkit/linuxkit",
Network: false,
DisableCache: false,
@@ -257,9 +259,16 @@ func NewFromConfig(cfg PkglibConfig, args ...string) ([]Pkg, error) {
}
}
}
tagTmpl := pi.Tag
if cfg.Tag != "" {
tagTmpl = cfg.Tag
}
if tagTmpl == "" {
tagTmpl = "{{.Hash}}"
}
// calculate the tag to use based on the template and the pkgHash
tmpl, err := template.New("tag").Parse(cfg.Tag)
tmpl, err := template.New("tag").Parse(tagTmpl)
if err != nil {
return nil, fmt.Errorf("invalid tag template: %v", err)
}

View File

@@ -0,0 +1,9 @@
package spec
type ImageBuildOptions struct {
Labels map[string]string
BuildArgs map[string]*string
NetworkMode string
Dockerfile string
SSH []string
}

View File

@@ -6,6 +6,7 @@ import (
"github.com/containerd/containerd/content"
"github.com/containerd/containerd/reference"
v1 "github.com/google/go-containerregistry/pkg/v1"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// CacheProvider interface for a provide of a cache.
@@ -19,7 +20,7 @@ type CacheProvider interface {
// ImagePull takes an image name and pulls it from a registry to the cache. It should be
// efficient and only write missing blobs, based on their content hash. If the ref already
// exists in the cache, it should not pull anything, unless alwaysPull is set to true.
ImagePull(ref *reference.Spec, trustedRef, architecture string, alwaysPull bool) (ImageSource, error)
ImagePull(ref *reference.Spec, platform []imagespec.Platform, alwaysPull bool) error
// ImageInCache takes an image name and checks if it exists in the cache, including checking that the given
// architecture is complete. Like ImagePull, it should be efficient and only write missing blobs, based on
// their content hash.
@@ -30,20 +31,20 @@ type CacheProvider interface {
// Cache implementation determines whether it should pull missing blobs from a remote registry.
// If the provided reference already exists and it is an index, updates the manifests in the
// existing index.
IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) (ImageSource, error)
IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) error
// ImageLoad takes an OCI format image tar stream in the io.Reader and writes it to the cache. It should be
// efficient and only write missing blobs, based on their content hash.
ImageLoad(r io.Reader) ([]v1.Descriptor, error)
// DescriptorWrite writes a descriptor to the cache index; it validates that it has a name
// and replaces any existing one
DescriptorWrite(ref *reference.Spec, descriptors v1.Descriptor) (ImageSource, error)
DescriptorWrite(ref *reference.Spec, descriptors v1.Descriptor) error
// Push an image along with a multi-arch index from local cache to remote registry.
// name is the name as referenced in the local cache, remoteName is the name to give it remotely.
// If remoteName is empty, it is the same as name.
// if withManifest defined will push a multi-arch manifest
Push(name, remoteName string, withManifest, override bool) error
// NewSource return an ImageSource for a specific ref and architecture in the cache.
NewSource(ref *reference.Spec, architecture string, descriptor *v1.Descriptor) ImageSource
NewSource(ref *reference.Spec, platform *imagespec.Platform, descriptor *v1.Descriptor) ImageSource
// GetContent returns an io.Reader to the provided content as is, given a specific digest. It is
// up to the caller to validate it.
GetContent(hash v1.Hash) (io.ReadCloser, error)

View File

@@ -10,12 +10,12 @@ import (
// ImageSource interface to an image. It can have its config read, and a its containers
// can be read via an io.ReadCloser tar stream.
type ImageSource interface {
// Descriptor get the v1.Descriptor of the image
Descriptor() *v1.Descriptor
// Config get the config for the image
Config() (imagespec.ImageConfig, error)
// TarReader get the flattened filesystem of the image as a tar stream
TarReader() (io.ReadCloser, error)
// Descriptor get the v1.Descriptor of the image
Descriptor() *v1.Descriptor
// V1TarReader get the image as v1 tarball, also compatible with `docker load`. If name arg is not "", override name of image in tarfile from default of image.
V1TarReader(overrideName string) (io.ReadCloser, error)
// OCITarReader get the image as an OCI tarball, also compatible with `docker load`. If name arg is not "", override name of image in tarfile from default of image.
@@ -23,3 +23,14 @@ type ImageSource interface {
// SBoM get the sbom for the image, if any is available
SBoMs() ([]io.ReadCloser, error)
}
// IndexSource interface to an image. It can have its config read, and a its containers
// can be read via an io.ReadCloser tar stream.
type IndexSource interface {
// Descriptor get the v1.Descriptor of the index
Descriptor() *v1.Descriptor
// Image get image for a specific architecture
Image(platform imagespec.Platform) (ImageSource, error)
// OCITarReader get the image as an OCI tarball, also compatible with `docker load`. If name arg is not "", override name of image in tarfile from default of image.
OCITarReader(overrideName string) (io.ReadCloser, error)
}

View File

@@ -25,23 +25,38 @@ func (f *infoFormatter) Format(entry *log.Entry) ([]byte, error) {
}
// SetupLogging once the flags have been parsed, setup the logging
func SetupLogging(quiet, verbose bool) error {
func SetupLogging(quiet bool, verbose int, verboseSet bool) error {
// Set up logging
log.SetFormatter(new(infoFormatter))
log.SetLevel(log.InfoLevel)
if quiet && verbose {
if quiet && verboseSet && verbose > 0 {
return errors.New("can't set quiet and verbose flag at the same time")
}
if quiet {
switch {
case quiet, verbose == 0:
log.SetLevel(log.ErrorLevel)
}
if verbose {
case verbose == 1:
if verboseSet {
// Switch back to the standard formatter
log.SetFormatter(defaultLogFormatter)
}
log.SetLevel(log.InfoLevel)
case verbose == 2:
// Switch back to the standard formatter
log.SetFormatter(defaultLogFormatter)
log.SetLevel(log.DebugLevel)
// set go-containerregistry logging as well
ggcrlog.Warn = stdlog.New(log.StandardLogger().WriterLevel(log.WarnLevel), "", 0)
ggcrlog.Debug = stdlog.New(log.StandardLogger().WriterLevel(log.DebugLevel), "", 0)
case verbose == 3:
// Switch back to the standard formatter
log.SetFormatter(defaultLogFormatter)
log.SetLevel(log.TraceLevel)
// set go-containerregistry logging as well
ggcrlog.Warn = stdlog.New(log.StandardLogger().WriterLevel(log.WarnLevel), "", 0)
ggcrlog.Debug = stdlog.New(log.StandardLogger().WriterLevel(log.DebugLevel), "", 0)
default:
return errors.New("verbose flag can only be set to 0, 1, 2 or 3")
}
ggcrlog.Progress = stdlog.New(log.StandardLogger().WriterLevel(log.InfoLevel), "", 0)
return nil

View File

@@ -3,7 +3,7 @@ package api // import "github.com/docker/docker/api"
// Common constants for daemon and client.
const (
// DefaultVersion of the current REST API.
DefaultVersion = "1.46"
DefaultVersion = "1.47"
// MinSupportedAPIVersion is the minimum API version that can be supported
// by the API server, specified as "major.minor". Note that the daemon

View File

@@ -19,10 +19,10 @@ produces:
consumes:
- "application/json"
- "text/plain"
basePath: "/v1.46"
basePath: "/v1.47"
info:
title: "Docker Engine API"
version: "1.46"
version: "1.47"
x-logo:
url: "https://docs.docker.com/assets/images/logo-docker-main.png"
description: |
@@ -55,8 +55,8 @@ info:
the URL is not supported by the daemon, a HTTP `400 Bad Request` error message
is returned.
If you omit the version-prefix, the current version of the API (v1.46) is used.
For example, calling `/info` is the same as calling `/v1.46/info`. Using the
If you omit the version-prefix, the current version of the API (v1.47) is used.
For example, calling `/info` is the same as calling `/v1.47/info`. Using the
API without a version-prefix is deprecated and will be removed in a future release.
Engine releases in the near future should support this version of the API,
@@ -2265,6 +2265,19 @@ definitions:
x-nullable: false
type: "integer"
example: 2
Manifests:
description: |
Manifests is a list of manifests available in this image.
It provides a more detailed view of the platform-specific image manifests
or other image-attached data like build attestations.
WARNING: This is experimental and may change at any time without any backward
compatibility.
type: "array"
x-nullable: false
x-omitempty: true
items:
$ref: "#/definitions/ImageManifestSummary"
AuthConfig:
type: "object"
@@ -5318,7 +5331,7 @@ definitions:
description: |
The default (and highest) API version that is supported by the daemon
type: "string"
example: "1.46"
example: "1.47"
MinAPIVersion:
description: |
The minimum API version that is supported by the daemon
@@ -5334,7 +5347,7 @@ definitions:
The version Go used to compile the daemon, and the version of the Go
runtime in use.
type: "string"
example: "go1.21.11"
example: "go1.21.13"
Os:
description: |
The operating system that the daemon is running on ("linux" or "windows")
@@ -5830,13 +5843,13 @@ definitions:
- "/var/run/cdi"
Containerd:
$ref: "#/definitions/ContainerdInfo"
x-nullable: true
ContainerdInfo:
description: |
Information for connecting to the containerd instance that is used by the daemon.
This is included for debugging purposes only.
type: "object"
x-nullable: true
properties:
Address:
description: "The address of the containerd socket."
@@ -6644,6 +6657,120 @@ definitions:
additionalProperties:
type: "string"
ImageManifestSummary:
x-go-name: "ManifestSummary"
description: |
ImageManifestSummary represents a summary of an image manifest.
type: "object"
required: ["ID", "Descriptor", "Available", "Size", "Kind"]
properties:
ID:
description: |
ID is the content-addressable ID of an image and is the same as the
digest of the image manifest.
type: "string"
example: "sha256:95869fbcf224d947ace8d61d0e931d49e31bb7fc67fffbbe9c3198c33aa8e93f"
Descriptor:
$ref: "#/definitions/OCIDescriptor"
Available:
description: Indicates whether all the child content (image config, layers) is fully available locally.
type: "boolean"
example: true
Size:
type: "object"
x-nullable: false
required: ["Content", "Total"]
properties:
Total:
type: "integer"
format: "int64"
example: 8213251
description: |
Total is the total size (in bytes) of all the locally present
data (both distributable and non-distributable) that's related to
this manifest and its children.
This equal to the sum of [Content] size AND all the sizes in the
[Size] struct present in the Kind-specific data struct.
For example, for an image kind (Kind == "image")
this would include the size of the image content and unpacked
image snapshots ([Size.Content] + [ImageData.Size.Unpacked]).
Content:
description: |
Content is the size (in bytes) of all the locally present
content in the content store (e.g. image config, layers)
referenced by this manifest and its children.
This only includes blobs in the content store.
type: "integer"
format: "int64"
example: 3987495
Kind:
type: "string"
example: "image"
enum:
- "image"
- "attestation"
- "unknown"
description: |
The kind of the manifest.
kind | description
-------------|-----------------------------------------------------------
image | Image manifest that can be used to start a container.
attestation | Attestation manifest produced by the Buildkit builder for a specific image manifest.
ImageData:
description: |
The image data for the image manifest.
This field is only populated when Kind is "image".
type: "object"
x-nullable: true
x-omitempty: true
required: ["Platform", "Containers", "Size", "UnpackedSize"]
properties:
Platform:
$ref: "#/definitions/OCIPlatform"
description: |
OCI platform of the image. This will be the platform specified in the
manifest descriptor from the index/manifest list.
If it's not available, it will be obtained from the image config.
Containers:
description: |
The IDs of the containers that are using this image.
type: "array"
items:
type: "string"
example: ["ede54ee1fda366ab42f824e8a5ffd195155d853ceaec74a927f249ea270c7430", "abadbce344c096744d8d6071a90d474d28af8f1034b5ea9fb03c3f4bfc6d005e"]
Size:
type: "object"
x-nullable: false
required: ["Unpacked"]
properties:
Unpacked:
type: "integer"
format: "int64"
example: 3987495
description: |
Unpacked is the size (in bytes) of the locally unpacked
(uncompressed) image content that's directly usable by the containers
running this image.
It's independent of the distributable content - e.g.
the image might still have an unpacked data that's still used by
some container even when the distributable/compressed content is
already gone.
AttestationData:
description: |
The image data for the attestation manifest.
This field is only populated when Kind is "attestation".
type: "object"
x-nullable: true
x-omitempty: true
required: ["For"]
properties:
For:
description: |
The digest of the image manifest that this attestation is for.
type: "string"
example: "sha256:95869fbcf224d947ace8d61d0e931d49e31bb7fc67fffbbe9c3198c33aa8e93f"
paths:
/containers/json:
get:
@@ -8622,6 +8749,11 @@ paths:
description: "Show digest information as a `RepoDigests` field on each image."
type: "boolean"
default: false
- name: "manifests"
in: "query"
description: "Include `Manifests` in the image summary."
type: "boolean"
default: false
tags: ["Image"]
/build:
post:
@@ -9563,7 +9695,7 @@ paths:
Containers report these events: `attach`, `commit`, `copy`, `create`, `destroy`, `detach`, `die`, `exec_create`, `exec_detach`, `exec_start`, `exec_die`, `export`, `health_status`, `kill`, `oom`, `pause`, `rename`, `resize`, `restart`, `start`, `stop`, `top`, `unpause`, `update`, and `prune`
Images report these events: `create, `delete`, `import`, `load`, `pull`, `push`, `save`, `tag`, `untag`, and `prune`
Images report these events: `create`, `delete`, `import`, `load`, `pull`, `push`, `save`, `tag`, `untag`, and `prune`
Volumes report these events: `create`, `mount`, `unmount`, `destroy`, and `prune`

View File

@@ -0,0 +1,99 @@
package image
import (
"github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
type ManifestKind string
const (
ManifestKindImage ManifestKind = "image"
ManifestKindAttestation ManifestKind = "attestation"
ManifestKindUnknown ManifestKind = "unknown"
)
type ManifestSummary struct {
// ID is the content-addressable ID of an image and is the same as the
// digest of the image manifest.
//
// Required: true
ID string `json:"ID"`
// Descriptor is the OCI descriptor of the image.
//
// Required: true
Descriptor ocispec.Descriptor `json:"Descriptor"`
// Indicates whether all the child content (image config, layers) is
// fully available locally
//
// Required: true
Available bool `json:"Available"`
// Size is the size information of the content related to this manifest.
// Note: These sizes only take the locally available content into account.
//
// Required: true
Size struct {
// Content is the size (in bytes) of all the locally present
// content in the content store (e.g. image config, layers)
// referenced by this manifest and its children.
// This only includes blobs in the content store.
Content int64 `json:"Content"`
// Total is the total size (in bytes) of all the locally present
// data (both distributable and non-distributable) that's related to
// this manifest and its children.
// This equal to the sum of [Content] size AND all the sizes in the
// [Size] struct present in the Kind-specific data struct.
// For example, for an image kind (Kind == ManifestKindImage),
// this would include the size of the image content and unpacked
// image snapshots ([Size.Content] + [ImageData.Size.Unpacked]).
Total int64 `json:"Total"`
} `json:"Size"`
// Kind is the kind of the image manifest.
//
// Required: true
Kind ManifestKind `json:"Kind"`
// Fields below are specific to the kind of the image manifest.
// Present only if Kind == ManifestKindImage.
ImageData *ImageProperties `json:"ImageData,omitempty"`
// Present only if Kind == ManifestKindAttestation.
AttestationData *AttestationProperties `json:"AttestationData,omitempty"`
}
type ImageProperties struct {
// Platform is the OCI platform object describing the platform of the image.
//
// Required: true
Platform ocispec.Platform `json:"Platform"`
Size struct {
// Unpacked is the size (in bytes) of the locally unpacked
// (uncompressed) image content that's directly usable by the containers
// running this image.
// It's independent of the distributable content - e.g.
// the image might still have an unpacked data that's still used by
// some container even when the distributable/compressed content is
// already gone.
//
// Required: true
Unpacked int64 `json:"Unpacked"`
}
// Containers is an array containing the IDs of the containers that are
// using this image.
//
// Required: true
Containers []string `json:"Containers"`
}
type AttestationProperties struct {
// For is the digest of the image manifest that this attestation is for.
For digest.Digest `json:"For"`
}

View File

@@ -76,6 +76,9 @@ type ListOptions struct {
// ContainerCount indicates whether container count should be computed.
ContainerCount bool
// Manifests indicates whether the image manifests should be returned.
Manifests bool
}
// RemoveOptions holds parameters to remove images.

View File

@@ -1,10 +1,5 @@
package image
// This file was generated by the swagger tool.
// Editing this file might prove futile when you re-run the swagger generate command
// Summary summary
// swagger:model Summary
type Summary struct {
// Number of containers using this image. Includes both stopped and running
@@ -47,6 +42,14 @@ type Summary struct {
// Required: true
ParentID string `json:"ParentId"`
// Manifests is a list of image manifests available in this image. It
// provides a more detailed view of the platform-specific image manifests or
// other image-attached data like build attestations.
//
// WARNING: This is experimental and may change at any time without any backward
// compatibility.
Manifests []ManifestSummary `json:"Manifests,omitempty"`
// List of content-addressable digests of locally available image manifests
// that the image is referenced from. Multiple manifests can refer to the
// same image.

View File

@@ -34,10 +34,9 @@ type AuthConfig struct {
}
// EncodeAuthConfig serializes the auth configuration as a base64url encoded
// RFC4648, section 5) JSON string for sending through the X-Registry-Auth header.
// ([RFC4648, section 5]) JSON string for sending through the X-Registry-Auth header.
//
// For details on base64url encoding, see:
// - RFC4648, section 5: https://tools.ietf.org/html/rfc4648#section-5
// [RFC4648, section 5]: https://tools.ietf.org/html/rfc4648#section-5
func EncodeAuthConfig(authConfig AuthConfig) (string, error) {
buf, err := json.Marshal(authConfig)
if err != nil {
@@ -46,15 +45,14 @@ func EncodeAuthConfig(authConfig AuthConfig) (string, error) {
return base64.URLEncoding.EncodeToString(buf), nil
}
// DecodeAuthConfig decodes base64url encoded (RFC4648, section 5) JSON
// DecodeAuthConfig decodes base64url encoded ([RFC4648, section 5]) JSON
// authentication information as sent through the X-Registry-Auth header.
//
// This function always returns an AuthConfig, even if an error occurs. It is up
// This function always returns an [AuthConfig], even if an error occurs. It is up
// to the caller to decide if authentication is required, and if the error can
// be ignored.
//
// For details on base64url encoding, see:
// - RFC4648, section 5: https://tools.ietf.org/html/rfc4648#section-5
// [RFC4648, section 5]: https://tools.ietf.org/html/rfc4648#section-5
func DecodeAuthConfig(authEncoded string) (*AuthConfig, error) {
if authEncoded == "" {
return &AuthConfig{}, nil
@@ -69,7 +67,7 @@ func DecodeAuthConfig(authEncoded string) (*AuthConfig, error) {
// clients and API versions. Current clients and API versions expect authentication
// to be provided through the X-Registry-Auth header.
//
// Like DecodeAuthConfig, this function always returns an AuthConfig, even if an
// Like [DecodeAuthConfig], this function always returns an [AuthConfig], even if an
// error occurs. It is up to the caller to decide if authentication is required,
// and if the error can be ignored.
func DecodeAuthConfigBody(rdr io.ReadCloser) (*AuthConfig, error) {

View File

@@ -77,9 +77,6 @@ type Info struct {
Containerd *ContainerdInfo `json:",omitempty"`
// Legacy API fields for older API versions.
legacyFields
// Warnings contains a slice of warnings that occurred while collecting
// system information. These warnings are intended to be informational
// messages for the user, and are not intended to be parsed / used for
@@ -124,10 +121,6 @@ type ContainerdNamespaces struct {
Plugins string
}
type legacyFields struct {
ExecutionDriver string `json:",omitempty"` // Deprecated: deprecated since API v1.25, but returned for older versions.
}
// PluginsInfo is a temp struct holding Plugins name
// registered with docker daemon. It is used by [Info] struct
type PluginsInfo struct {

View File

@@ -11,6 +11,11 @@ import (
)
// ImageList returns a list of images in the docker host.
//
// Experimental: Setting the [options.Manifest] will populate
// [image.Summary.Manifests] with information about image manifests.
// This is experimental and might change in the future without any backward
// compatibility.
func (cli *Client) ImageList(ctx context.Context, options image.ListOptions) ([]image.Summary, error) {
var images []image.Summary
@@ -47,6 +52,9 @@ func (cli *Client) ImageList(ctx context.Context, options image.ListOptions) ([]
if options.SharedSize && versions.GreaterThanOrEqualTo(cli.version, "1.42") {
query.Set("shared-size", "1")
}
if options.Manifests && versions.GreaterThanOrEqualTo(cli.version, "1.47") {
query.Set("manifests", "1")
}
serverResp, err := cli.get(ctx, "/images/json", query, nil)
defer ensureReaderClosed(serverResp)

View File

@@ -0,0 +1,363 @@
Mozilla Public License, version 2.0
1. Definitions
1.1. "Contributor"
means each individual or legal entity that creates, contributes to the
creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used by a
Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached the
notice in Exhibit A, the Executable Form of such Source Code Form, and
Modifications of such Source Code Form, in each case including portions
thereof.
1.5. "Incompatible With Secondary Licenses"
means
a. that the initial Contributor has attached the notice described in
Exhibit B to the Covered Software; or
b. that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the terms of
a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in a
separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible, whether
at the time of the initial grant or subsequently, any and all of the
rights conveyed by this License.
1.10. "Modifications"
means any of the following:
a. any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered Software; or
b. any new file in Source Code Form that contains any Covered Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the License,
by the making, using, selling, offering for sale, having made, import,
or transfer of either its Contributions or its Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU Lesser
General Public License, Version 2.1, the GNU Affero General Public
License, Version 3.0, or any later versions of those licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that controls, is
controlled by, or is under common control with You. For purposes of this
definition, "control" means (a) the power, direct or indirect, to cause
the direction or management of such entity, whether by contract or
otherwise, or (b) ownership of more than fifty percent (50%) of the
outstanding shares or beneficial ownership of such entity.
2. License Grants and Conditions
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
a. under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
b. under Patent Claims of such Contributor to make, use, sell, offer for
sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
a. for any code that a Contributor has removed from Covered Software; or
b. for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
c. under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights to
grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in
Section 2.1.
3. Responsibilities
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
a. such Covered Software must also be made available in Source Code Form,
as described in Section 3.1, and You must inform recipients of the
Executable Form how they can obtain a copy of such Source Code Form by
reasonable means in a timely manner, at a charge no more than the cost
of distribution to the recipient; and
b. You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter the
recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty, or
limitations of liability) contained within the Source Code Form of the
Covered Software, except that You may alter any license notices to the
extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
If it is impossible for You to comply with any of the terms of this License
with respect to some or all of the Covered Software due to statute,
judicial order, or regulation then You must: (a) comply with the terms of
this License to the maximum extent possible; and (b) describe the
limitations and the code they affect. Such description must be placed in a
text file included with all distributions of the Covered Software under
this License. Except to the extent prohibited by statute or regulation,
such description must be sufficiently detailed for a recipient of ordinary
skill to be able to understand it.
5. Termination
5.1. The rights granted under this License will terminate automatically if You
fail to comply with any of its terms. However, if You become compliant,
then the rights granted under this License from a particular Contributor
are reinstated (a) provisionally, unless and until such Contributor
explicitly and finally terminates Your grants, and (b) on an ongoing
basis, if such Contributor fails to notify You of the non-compliance by
some reasonable means prior to 60 days after You have come back into
compliance. Moreover, Your grants from a particular Contributor are
reinstated on an ongoing basis if such Contributor notifies You of the
non-compliance by some reasonable means, this is the first time You have
received notice of non-compliance with this License from such
Contributor, and You become compliant prior to 30 days after Your receipt
of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user
license agreements (excluding distributors and resellers) which have been
validly granted by You or Your distributors under this License prior to
termination shall survive termination.
6. Disclaimer of Warranty
Covered Software is provided under this License on an "as is" basis,
without warranty of any kind, either expressed, implied, or statutory,
including, without limitation, warranties that the Covered Software is free
of defects, merchantable, fit for a particular purpose or non-infringing.
The entire risk as to the quality and performance of the Covered Software
is with You. Should any Covered Software prove defective in any respect,
You (not any Contributor) assume the cost of any necessary servicing,
repair, or correction. This disclaimer of warranty constitutes an essential
part of this License. No use of any Covered Software is authorized under
this License except under this disclaimer.
7. Limitation of Liability
Under no circumstances and under no legal theory, whether tort (including
negligence), contract, or otherwise, shall any Contributor, or anyone who
distributes Covered Software as permitted above, be liable to You for any
direct, indirect, special, incidental, or consequential damages of any
character including, without limitation, damages for lost profits, loss of
goodwill, work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses, even if such party shall have been
informed of the possibility of such damages. This limitation of liability
shall not apply to liability for death or personal injury resulting from
such party's negligence to the extent applicable law prohibits such
limitation. Some jurisdictions do not allow the exclusion or limitation of
incidental or consequential damages, so this exclusion and limitation may
not apply to You.
8. Litigation
Any litigation relating to this License may be brought only in the courts
of a jurisdiction where the defendant maintains its principal place of
business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions. Nothing
in this Section shall prevent a party's ability to bring cross-claims or
counter-claims.
9. Miscellaneous
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides that
the language of a contract shall be construed against the drafter shall not
be used to construe this License against a Contributor.
10. Versions of the License
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses If You choose to distribute Source Code Form that is
Incompatible With Secondary Licenses under the terms of this version of
the License, the notice described in Exhibit B of this License must be
attached.
Exhibit A - Source Code Form License Notice
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular file,
then You may include the notice in a location (such as a LICENSE file in a
relevant directory) where a recipient would be likely to look for such a
notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
This Source Code Form is "Incompatible
With Secondary Licenses", as defined by
the Mozilla Public License, v. 2.0.

View File

@@ -0,0 +1,30 @@
# cleanhttp
Functions for accessing "clean" Go http.Client values
-------------
The Go standard library contains a default `http.Client` called
`http.DefaultClient`. It is a common idiom in Go code to start with
`http.DefaultClient` and tweak it as necessary, and in fact, this is
encouraged; from the `http` package documentation:
> The Client's Transport typically has internal state (cached TCP connections),
so Clients should be reused instead of created as needed. Clients are safe for
concurrent use by multiple goroutines.
Unfortunately, this is a shared value, and it is not uncommon for libraries to
assume that they are free to modify it at will. With enough dependencies, it
can be very easy to encounter strange problems and race conditions due to
manipulation of this shared value across libraries and goroutines (clients are
safe for concurrent use, but writing values to the client struct itself is not
protected).
Making things worse is the fact that a bare `http.Client` will use a default
`http.Transport` called `http.DefaultTransport`, which is another global value
that behaves the same way. So it is not simply enough to replace
`http.DefaultClient` with `&http.Client{}`.
This repository provides some simple functions to get a "clean" `http.Client`
-- one that uses the same default values as the Go standard library, but
returns a client that does not share any state with other clients.

View File

@@ -0,0 +1,58 @@
package cleanhttp
import (
"net"
"net/http"
"runtime"
"time"
)
// DefaultTransport returns a new http.Transport with similar default values to
// http.DefaultTransport, but with idle connections and keepalives disabled.
func DefaultTransport() *http.Transport {
transport := DefaultPooledTransport()
transport.DisableKeepAlives = true
transport.MaxIdleConnsPerHost = -1
return transport
}
// DefaultPooledTransport returns a new http.Transport with similar default
// values to http.DefaultTransport. Do not use this for transient transports as
// it can leak file descriptors over time. Only use this for transports that
// will be re-used for the same host(s).
func DefaultPooledTransport() *http.Transport {
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
DualStack: true,
}).DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
ForceAttemptHTTP2: true,
MaxIdleConnsPerHost: runtime.GOMAXPROCS(0) + 1,
}
return transport
}
// DefaultClient returns a new http.Client with similar default values to
// http.Client, but with a non-shared Transport, idle connections disabled, and
// keepalives disabled.
func DefaultClient() *http.Client {
return &http.Client{
Transport: DefaultTransport(),
}
}
// DefaultPooledClient returns a new http.Client with similar default values to
// http.Client, but with a shared Transport. Do not use this function for
// transient clients as it can leak file descriptors over time. Only use this
// for clients that will be re-used for the same host(s).
func DefaultPooledClient() *http.Client {
return &http.Client{
Transport: DefaultPooledTransport(),
}
}

View File

@@ -0,0 +1,20 @@
// Package cleanhttp offers convenience utilities for acquiring "clean"
// http.Transport and http.Client structs.
//
// Values set on http.DefaultClient and http.DefaultTransport affect all
// callers. This can have detrimental effects, esepcially in TLS contexts,
// where client or root certificates set to talk to multiple endpoints can end
// up displacing each other, leading to hard-to-debug issues. This package
// provides non-shared http.Client and http.Transport structs to ensure that
// the configuration will not be overwritten by other parts of the application
// or dependencies.
//
// The DefaultClient and DefaultTransport functions disable idle connections
// and keepalives. Without ensuring that idle connections are closed before
// garbage collection, short-term clients/transports can leak file descriptors,
// eventually leading to "too many open files" errors. If you will be
// connecting to the same hosts repeatedly from the same client, you can use
// DefaultPooledClient to receive a client that has connection pooling
// semantics similar to http.DefaultClient.
//
package cleanhttp

View File

@@ -0,0 +1,48 @@
package cleanhttp
import (
"net/http"
"strings"
"unicode"
)
// HandlerInput provides input options to cleanhttp's handlers
type HandlerInput struct {
ErrStatus int
}
// PrintablePathCheckHandler is a middleware that ensures the request path
// contains only printable runes.
func PrintablePathCheckHandler(next http.Handler, input *HandlerInput) http.Handler {
// Nil-check on input to make it optional
if input == nil {
input = &HandlerInput{
ErrStatus: http.StatusBadRequest,
}
}
// Default to http.StatusBadRequest on error
if input.ErrStatus == 0 {
input.ErrStatus = http.StatusBadRequest
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r != nil {
// Check URL path for non-printable characters
idx := strings.IndexFunc(r.URL.Path, func(c rune) bool {
return !unicode.IsPrint(c)
})
if idx != -1 {
w.WriteHeader(input.ErrStatus)
return
}
if next != nil {
next.ServeHTTP(w, r)
}
}
return
})
}

View File

@@ -0,0 +1,18 @@
package build
import (
"github.com/moby/buildkit/util/entitlements"
)
// ParseAllow parses --allow
func ParseAllow(inp []string) ([]entitlements.Entitlement, error) {
ent := make([]entitlements.Entitlement, 0, len(inp))
for _, v := range inp {
e, err := entitlements.Parse(v)
if err != nil {
return nil, err
}
ent = append(ent, e)
}
return ent, nil
}

View File

@@ -0,0 +1,19 @@
package build
import (
"strings"
"github.com/pkg/errors"
)
func attrMap(sl []string) (map[string]string, error) {
m := map[string]string{}
for _, v := range sl {
parts := strings.SplitN(v, "=", 2)
if len(parts) != 2 {
return nil, errors.Errorf("invalid value %s", v)
}
m[parts[0]] = parts[1]
}
return m, nil
}

View File

@@ -0,0 +1,71 @@
package build
import (
"encoding/csv"
"strings"
"github.com/moby/buildkit/client"
"github.com/moby/buildkit/util/bklog"
"github.com/pkg/errors"
)
func parseExportCacheCSV(s string) (client.CacheOptionsEntry, error) {
ex := client.CacheOptionsEntry{
Type: "",
Attrs: map[string]string{},
}
csvReader := csv.NewReader(strings.NewReader(s))
fields, err := csvReader.Read()
if err != nil {
return ex, err
}
for _, field := range fields {
key, value, ok := strings.Cut(field, "=")
if !ok {
return ex, errors.Errorf("invalid value %s", field)
}
key = strings.ToLower(key)
switch key {
case "type":
ex.Type = value
default:
ex.Attrs[key] = value
}
}
if ex.Type == "" {
return ex, errors.New("--export-cache requires type=<type>")
}
if _, ok := ex.Attrs["mode"]; !ok {
ex.Attrs["mode"] = "min"
}
if ex.Type == "gha" {
return loadGithubEnv(ex)
}
return ex, nil
}
// ParseExportCache parses --export-cache
func ParseExportCache(exportCaches []string) ([]client.CacheOptionsEntry, error) {
var exports []client.CacheOptionsEntry
for _, exportCache := range exportCaches {
legacy := !strings.Contains(exportCache, "type=")
if legacy {
// Deprecated since BuildKit v0.4.0, but no plan to remove: https://github.com/moby/buildkit/pull/2783#issuecomment-1093449772
bklog.L.Warnf("--export-cache <ref> is deprecated. Please use --export-cache type=registry,ref=<ref>,<opt>=<optval>[,<opt>=<optval>] instead")
exports = append(exports, client.CacheOptionsEntry{
Type: "registry",
Attrs: map[string]string{
"mode": "min",
"ref": exportCache,
},
})
} else {
ex, err := parseExportCacheCSV(exportCache)
if err != nil {
return nil, err
}
exports = append(exports, ex)
}
}
return exports, nil
}

View File

@@ -0,0 +1,65 @@
package build
import (
"encoding/csv"
"strings"
"github.com/moby/buildkit/client"
"github.com/moby/buildkit/util/bklog"
"github.com/pkg/errors"
)
func parseImportCacheCSV(s string) (client.CacheOptionsEntry, error) {
im := client.CacheOptionsEntry{
Type: "",
Attrs: map[string]string{},
}
csvReader := csv.NewReader(strings.NewReader(s))
fields, err := csvReader.Read()
if err != nil {
return im, err
}
for _, field := range fields {
key, value, ok := strings.Cut(field, "=")
if !ok {
return im, errors.Errorf("invalid value %s", field)
}
key = strings.ToLower(key)
switch key {
case "type":
im.Type = value
default:
im.Attrs[key] = value
}
}
if im.Type == "" {
return im, errors.New("--import-cache requires type=<type>")
}
if im.Type == "gha" {
return loadGithubEnv(im)
}
return im, nil
}
// ParseImportCache parses --import-cache
func ParseImportCache(importCaches []string) ([]client.CacheOptionsEntry, error) {
var imports []client.CacheOptionsEntry
for _, importCache := range importCaches {
legacy := !strings.Contains(importCache, "type=")
if legacy {
// Deprecated since BuildKit v0.4.0, but no plan to remove: https://github.com/moby/buildkit/pull/2783#issuecomment-1093449772
bklog.L.Warn("--import-cache <ref> is deprecated. Please use --import-cache type=registry,ref=<ref>,<opt>=<optval>[,<opt>=<optval>] instead.")
imports = append(imports, client.CacheOptionsEntry{
Type: "registry",
Attrs: map[string]string{"ref": importCache},
})
} else {
im, err := parseImportCacheCSV(importCache)
if err != nil {
return nil, err
}
imports = append(imports, im)
}
}
return imports, nil
}

View File

@@ -0,0 +1,25 @@
package build
import (
"github.com/pkg/errors"
"github.com/tonistiigi/fsutil"
)
// ParseLocal parses --local
func ParseLocal(locals []string) (map[string]fsutil.FS, error) {
localDirs, err := attrMap(locals)
if err != nil {
return nil, errors.WithStack(err)
}
mounts := make(map[string]fsutil.FS, len(localDirs))
for k, v := range localDirs {
mounts[k], err = fsutil.NewFS(v)
if err != nil {
return nil, errors.WithStack(err)
}
}
return mounts, nil
}

View File

@@ -0,0 +1,27 @@
package build
import (
"strings"
"github.com/containerd/containerd/content"
"github.com/containerd/containerd/content/local"
"github.com/pkg/errors"
)
// ParseOCILayout parses --oci-layout
func ParseOCILayout(layouts []string) (map[string]content.Store, error) {
contentStores := make(map[string]content.Store)
for _, idAndDir := range layouts {
parts := strings.SplitN(idAndDir, "=", 2)
if len(parts) != 2 {
return nil, errors.Errorf("oci-layout option must be 'id=path/to/layout', instead had invalid %s", idAndDir)
}
cs, err := local.NewStore(parts[1])
if err != nil {
return nil, errors.Wrapf(err, "oci-layout context at %s failed to initialize", parts[1])
}
contentStores[parts[0]] = cs
}
return contentStores, nil
}

View File

@@ -0,0 +1,13 @@
package build
func ParseOpt(opts []string) (map[string]string, error) {
m := loadOptEnv()
m2, err := attrMap(opts)
if err != nil {
return nil, err
}
for k, v := range m2 {
m[k] = v
}
return m, nil
}

Some files were not shown because too many files have changed in this diff Show More