Commit Graph

474 Commits

Author SHA1 Message Date
Petr Fedchenkov
3540a1bc34 Update buildkit to have platform fix
We noticed that we use host arch when we want to use previously build
image in oci-layout. Let's use fix on buildkit side and improve test.

Signed-off-by: Petr Fedchenkov <giggsoff@gmail.com>
2022-07-22 12:25:39 +03:00
Petr Fedchenkov
5763c4f4bc Ability to parse args from Dockerfile
We should check if we have args in "FROM" and replace them:

ARG IMAGE=linuxkit/img
FROM ${IMAGE} as src

will be parsed as

FROM linuxkit/img as src

Signed-off-by: Petr Fedchenkov <giggsoff@gmail.com>
2022-07-20 17:07:43 +03:00
Avi Deitcher
403e8f9353 Merge pull request #3797 from deitch/use-images-from-cache
add support for building from cached images
2022-07-20 09:19:47 -04:00
Petr Fedchenkov
39f1649995 Allow store to docker multiarch builds
We do not allow to load into docker images that are targets another
platform differ from current arch. Assume this is because of no support
of manifest. But we can keep all images in place by adding arch suffix
and using tag without arch suffix to point onto current system arch. It
will help to use images from docker for another arch.

Signed-off-by: Petr Fedchenkov <giggsoff@gmail.com>
2022-07-15 16:51:18 +03:00
Avi Deitcher
a90ff542cd add support for building from cached images
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-07-14 12:53:34 +03:00
Davide Brini
860934d5d9 New output format: iso-efi-initrd
This option was previously not available and required postprocessing of a `tar-kernel-initrd` output.

Comparison with `iso-efi`:

`iso-efi` only loads the kernel at boot, and the root filesystem is mounted from the actual boot media (eg, a CD-ROM - physical or emulated). This can often cause trouble (it has for us) for multiple reasons:
- the linuxkit kernel might not have the correct drivers built-in for the hardware (see #3154)
- especially with virtual or emulated CD-ROMs, performance can be abysmal: we saw the case where the server IPMI allowed using a ISO stored in AWS S3 over HTTP...you can imagine what happens when you start doing random I/O on the root fs in that case.
- The ISO image has the root device name baked in (ie, `/dev/sr0`) which fails if for some reason the CD-ROM we're running from doesn't end up using that device, so manual tweaking is required (see #2375)

`iso-efi-initrd`, on the other hand, packs the root filesystem as an initramfs (ie similar to what the raw output does, except that in this case we're preparing an ISO image), so both the kernel and the initramfs are loaded in memory by the boot loader and, once running, we don't need to worry about root devices or kernel drivers (and the speed is good, as everything runs in RAM).

Also, the generated ISO can be copied verbatim (eg with `dd`) onto a USB media and it still works.

Finally, the image size is much smaller compared to `iso-efi`.

IMHO, `iso-efi-initrd` could be used almost anywhere `iso-efi` would be used, or might even supersede it. I can't think of a scenario where one might explicitly want to use `iso-efi`.

Points to consider:

- Not tested under aarch64 as I don't have access to that arch. If the automated CI tests also test that, then it should be fine.
- I'm not sure what to put inside `images.yaml` for the `iso-efi-initrd` image. As it is it works of course (my personal image on docker hub), but I guess it'll have to be some more "official" image. However, that cannot be until this PR is merged, so it's kind of a chicken and egg situation. Please advise.
- I can look into adding the corresponding `iso-bios-initrd` builder if there is interest.

![cute seal](https://sites.psu.edu/siowfa16/files/2016/09/baby-seal-29vsgyf-288x300.jpg)

Signed-off-by: Davide Brini <waldner@katamail.com>
2022-07-11 20:26:14 +02:00
Davide Brini
9edeba42d7 Fix: #3799 golint requires comments on non-main package for blank import
Signed-off-by: Davide Brini <waldner@katamail.com>
2022-07-10 11:55:51 +02:00
Avi Deitcher
9c95286a35 add docs to the cache spec
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-07-01 15:30:22 +03:00
Avi Deitcher
7778f34c0c Merge pull request #3791 from deitch/direct-buildkit
build directly with buildkit
2022-07-01 12:29:20 +03:00
Avi Deitcher
0929aabe50 build directly with buildkit
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-06-28 10:36:30 +03:00
Avi Deitcher
fb111d3bbf add buildkit module and update packages and vendor
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-06-28 10:35:45 +03:00
Petr Fedchenkov
a51f40f68b Support for vTPM on GCP
Add options to support vTPM-enabled VMs on GCP

Signed-off-by: Petr Fedchenkov <giggsoff@gmail.com>
2022-06-21 16:53:41 +03:00
Avi Deitcher
9218f2577d when no archs for pkg to build, should continue rather than return, or it misses other packages
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-06-07 19:45:35 +03:00
Avi Deitcher
02fd6c48ee do not push if the tag and digest are identical
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-06-07 10:45:29 +03:00
Avi Deitcher
6cd8a874a6 do not try to build or push if no architectures are available
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-06-07 10:25:41 +03:00
Yuri Volchkov
1a013f4424 Declare build-args in build.yml
This allows multiple build flavors for a single codebase, without
sacrificing reproducible builds. The build-args are set in build.yml,
which is typically under the source control (if it is not, then no
reproducible builds are possible anyways). Meaning that mutating
build-args would result in setting "dirty" flag.

Intended use of this commit is to switch between build flavors by
specifying a different yaml file (presumably also under the version
control)  by  `-build-yml` option.

Because it is impossible to build a final image from packages in
cache, the test for this feature relies on the `RUN echo $build-arg`
output during the `pkg build` process.

Signed-off-by: Yuri Volchkov <yuri@zededa.com>
2022-04-13 17:36:55 +00:00
Avi Deitcher
b3f276f8f9 update linuxkit/init dependencies
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-03-21 14:15:28 +02:00
Avi Deitcher
200e0ee94a update dependencies of pkg/init and pkg/containerd
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-03-21 01:22:19 +02:00
David Scott
f14ad44ce4 update shas for pkg/init
Signed-off-by: David Scott <dave@recoil.org>
2022-03-13 19:57:25 +00:00
Avi Deitcher
1df038e1b0 Merge pull request #3753 from deitch/rationalize-args
replace Build many args with opts struct
2022-01-14 11:21:37 +02:00
Avi Deitcher
f6b9dc1717 replace Build many args with opts struct
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-01-13 12:50:58 +02:00
Dave Tucker
0b739c80e9 pkg: update shas
Update the SHA of all packages that have been go-mod-ififed
such that they are use for testing in CI

Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
2022-01-12 00:04:37 +00:00
Avi Deitcher
84e3b80863 pass target architecture explicitly
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2022-01-10 10:36:22 +02:00
David Scott
6a11c118f4 Merge pull request #3678 from olljanat/docker-run-print-stderr
Print docker run stderr in-case of error
2021-12-30 10:37:30 +00:00
David Scott
341c4a4f5e Update shas for linuxkit/runc
./scripts/update-component-sha.sh linuxkit/runc:21dbbda709ae138de0af6b0c7e4ae49525db5e88  linuxkit/runc:9f7aad4eb5e4360cc9ed8778a5c501cce6e21601

Signed-off-by: David Scott <dave@recoil.org>
2021-12-16 19:10:58 +01:00
Frédéric Dalleau
36853c74ef Revert "runc: don't mount /dev with ro"
This reverts commit 380f36cc1a.

Now that runc includes a fix for this, this patch can be reverted

Signed-off-by: Frédéric Dalleau <frederic.dalleau@docker.com>
2021-12-16 19:10:58 +01:00
谢瑶瑶
f5a720bcbb fix: #3742 golint requires comments on non-main package for blank import
Signed-off-by: 谢瑶瑶 <yaoyao.xyy@alibaba-inc.com>
2021-12-14 13:50:59 +08:00
Avi Deitcher
46ef83c979 Merge pull request #3702 from deitch/alpine-314
bump alpine to 3.14, remove go version hack
2021-11-16 14:56:01 +02:00
Avi Deitcher
58813e8791 bump go-containerregistry and manifest-tool to latest
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-11-09 18:48:06 +02:00
Avi Deitcher
260cc85b52 Update package tags
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-11-09 14:02:43 +02:00
Avi Deitcher
bcbabcac28 Update use of test packages to latest
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-11-09 14:02:41 +02:00
Avi Deitcher
897d7a5071 Update use of tools to latest
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-11-07 16:40:26 +02:00
Avi Deitcher
e1dd1af1b9 extract more hard-coded yaml
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-11-05 13:07:06 +02:00
Avi Deitcher
0660ace86f extract hard-coded default image builders into file
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-11-04 14:51:15 -04:00
David Scott
f5f5dce318 linuxkit: fix pkg build on Windows
Previously when we set `cmd.Stderr = os.Stderr`, the stderr from buildx
would be mixed with the image tar, corrupting it.

Work around this (Windows-specific) problem by adding an explicit
indirection via a io.Pipe()

Signed-off-by: David Scott <dave@recoil.org>
2021-10-29 12:01:35 +01:00
David Scott
e8f8a409e8 Update hashes for pkg/init
Signed-off-by: David Scott <dave@recoil.org>
2021-10-26 19:52:22 +01:00
David Scott
d4c6ab742b Update hashes for pkg/...
Signed-off-by: David Scott <dave@recoil.org>
2021-10-18 21:02:44 +01:00
David Scott
380f36cc1a runc: don't mount /dev with ro
After runc 1.0.0-rc92 mounting /dev with ro will fail to start the
container with an error trying to `mkdir /dev/...` (for example
`/dev/pts`). This can be observed following the runc example

Comparing our `config.json` with the working one generated by
`runc spec`, both have a readonly rootfs (good) but the `runc spec`
one does not set `ro` in the `/dev` mount options.

This patch fixes readonly onboot containers by removing the "ro"
option from `/dev`, to match the `runc spec` example.

Signed-off-by: David Scott <dave@recoil.org>
2021-10-18 21:00:04 +01:00
Rolf Neugebauer
0dd8086d39 Update YAMLs to latest runc/containerd/test-containerd
Signed-off-by: Rolf Neugebauer <rn@rneugeba.io>
2021-10-16 16:57:15 +01:00
David Scott
46ea02f65b moby: device "all" will add to the cgroup whitelist
After the runc security advisory[1] the default cgroup device
whitelist was changed.

In previous versions every container had "rwm" (read, write, mknod)
for every device ("a" for all). Typically this was overridden by
container engines like Docker. In LinuxKit we left the permissive
default.

In recent `runc` versions the default allow-all rule was removed,
so a container can only access a device if it is specifically
granted access, which LinuxKit handles via a device: entry.

However it is inconvenient for pkg/format, pkg/mount, pkg/swap
to list all possible block devices up-front. Therefore we add the
ability to grant access to an entire class of device with a single
rule:

```
- path: all
  type: b
```

Obviously a paranoid user can still override this with a specific
major/minor number in a device: rule.

[1] https://github.com/opencontainers/runc/security/advisories/GHSA-g54h-m393-cpwq

Signed-off-by: David Scott <dave@recoil.org>
2021-10-14 16:14:21 +01:00
David Scott
24db42dd68 moby: add a Devices array to the image yml
According to https://github.com/linuxkit/linuxkit/pull/3684#issuecomment-860128095

runc removed the console as a default device, so now it must be specified
explicitly in the OCI config.

See 60e21ec26e

The similar code in moby/moby is here: https://github.com/moby/moby/blob/master/oci/devices_linux.go

This patch allows packages to declare a `devices` array, which can contain `/dev/console` etc.

Signed-off-by: David Scott <dave@recoil.org>
2021-10-14 16:14:05 +01:00
David Scott
c779e894da Fix linuxkit run qemu on macOS on Apple Silicon
Signed-off-by: David Scott <dave@recoil.org>
2021-10-13 14:38:20 +01:00
Avi Deitcher
ebbb1281f3 one-more
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-06-23 13:40:55 +03:00
Avi Deitcher
a05b0ac923 check for arch when pulling to cache, push by descriptor
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-06-21 23:33:34 +03:00
Avi Deitcher
873f61e695 bump manifest-tool to latest
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-06-15 20:28:44 +03:00
Avi Deitcher
df36b7aa7d Push arch-specific tags, always build index from registry
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-06-10 13:33:16 +03:00
Olli Janatuinen
49cbe047ce Print docker run stderr in-case of error
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
2021-05-26 22:55:52 +03:00
Dave Tucker
6312d58032 Merge pull request #3659 from edvinerikson/fix-install
fix: make command
2021-05-26 14:27:52 +01:00
Dave Tucker
452910c625 Makefile: Don't encode bindir in the tar archive
Prior to this commit we go build -o bin/foo, archive it, and
expand the archive, leaving the resulting artifact in bin.

This doesn't allow us to easily change the bin directory, or
move parts of the makefile around to make things more modular.

This commit changes the behaviour to:
go build -o foo, archive it, expand to `bin`

Signed-off-by: Dave Tucker <dave@dtucker.co.uk>
2021-05-26 11:18:41 +01:00
Avi Deitcher
4795c993ee Add support for cache export
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2021-05-26 13:13:37 +03:00