Commit Graph

193 Commits

Author SHA1 Message Date
Garth Bushell
e1b4f769cb Add qcow2 EFI image building support
Signed-off-by: Garth Bushell <garth.bushell@oracle.com>
2018-03-01 11:11:23 +00:00
Rolf Neugebauer
8fb1a88757 The vmdk and dynamic-vhd packages had a bug in them
This picks up a newer version

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2018-01-25 18:04:07 +00:00
Rolf Neugebauer
5c1ebdbaa2 Update LinuxKit images to the latest
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2018-01-25 14:15:52 +00:00
Justin Cormack
2dd65d27b6 Add support for setting containerd namespace in runtime config
This adds a namespace field to override the LinuxKit containerd
default namespace, in case you want to run a container in another
namespace.

Needs a patch in LinuxKit to implement this that I will open soon.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2018-01-17 14:12:15 +00:00
Justin Cormack
929a837d65 Add support for OCI annotations
Annotations do not do anything by default but get passed through to the runtime,
which can be useful. I never metadata I didn't like...

Also fix sysctl to be a map in the validation, not an array. I can't see any
examples using this in LinuxKit, but this matches OCI so is correct.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2018-01-17 12:26:08 +00:00
Rolf Neugebauer
5d5a13526b output: Add support for CPU ucode to the kernel+initrd format
This prepends 'ucode.cpio' to the initrd if present. Padding
should not be necessary as the ucode.cpio should be padded
to the right size.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2018-01-15 16:49:43 +00:00
Rolf Neugebauer
1d24454438 output: Add ucode.cpio to the tar-kernel-initrd format
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2018-01-15 16:49:42 +00:00
Rolf Neugebauer
2b1a611bab output: Extract ucode if present
For now the backends for the different formats do not yet
use the extracted ucode cpio archive, but '// TODO' are
placed for the backends which should eventually handle it.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2018-01-15 16:49:42 +00:00
Rolf Neugebauer
0c1b1def13 build: Add ucode to the intermediate tar ball if specified
This extends the kernel filter to also look for the CPU microcode
file if specified in the YAML. If found, the ucode cpio archive
is placed into the intermediate tar file as '/boot/ucode.cpio'.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2018-01-15 16:49:42 +00:00
Rolf Neugebauer
b4704b8ba1 config: Add 'ucode' to the kernel schema
This optional option will allow users to specify a CPU
microcode cpio archive to be prepended to the initrd file.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2018-01-15 16:49:42 +00:00
Ian Campbell
0a6d69d2d3 Prepend the rootfs mounts to the user specified runtime.mounts.
User specified mounts should be able to rely on the rootfs being mounted, in
particular for a writeable container they should expect the writeable overlay
to already be in place.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-01-11 11:42:25 +00:00
Justin Cormack
b5615a3714 Merge pull request #192 from w9n/fix_empty_container_config
allow ImageInspect.Config to be nil when parsing
2018-01-04 11:27:42 +00:00
Robin Winkelewski
175fadb3d4 add omitempty for yml config
Signed-off-by: Robin Winkelewski <w9ncontact@gmail.com>
2017-12-27 03:08:44 +01:00
Robin Winkelewski
1b06de3136 fix inspect.Config usage
Signed-off-by: Robin Winkelewski <w9ncontact@gmail.com>
2017-12-14 07:11:17 +01:00
Ian Campbell
307f13b129 Defer dockerRm until we are finished with the contents
This was introduced by #191 but somehow did not trigger either for me in local
testing or in CI.

It did trigger in initial CI of https://github.com/linuxkit/linuxkit/pull/2811
which can be seen at https://linuxkit.datakit.ci/linuxkit/linuxkit/pr/2811?history=1637690296123e9a15307b3a41b290da6e27e7cc
The error is:

    Failed to docker rm container «...»: «...»: aufs: unmount error after retries: «...»: device or resource busy

No doubt because we were still holding an open fd while trying to remove the
container.

Unclear why this didn't repro for me (docker 17.11.0-ce with overlay2) or
whatever CI uses.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-12-13 11:44:32 +00:00
Ian Campbell
3045a80c85 Stream docker export directly to consumer
Rather than queueing up into a `bytes.Buffer`.

In my test case (building kube master image) this reduces Maximum RSS (as
measured by time(1)) compared with the previous patch from 2.8G to 110M. The
tar output case goes from 2.1G to 110M also. Overall allocations are ~715M in
both cases.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-12-06 16:41:42 +00:00
Ian Campbell
9f44acf8e3 Generate intermediate image into a temp file
All of the `output*` functions took a `[]byte` and immediately wrapped it in a
`bytes.Buffer` to produce an `io.Reader`. Make them take an `io.Reader` instead
and satisfy this further up the call chain by directing `moby.Build` to output
to a temp file instead of another `bytes.Buffer`.

In my test case (building kube master image) this reduces Maximum RSS (as
measured by time(1)) from 6.7G to 2.8G and overall allocations from 9.7G to
5.3G. When building a tar (output to /dev/null) the Maximum RSS fell slightly
from 2.2G to 2.1G. Overall allocations remained stable at around 5.3G.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-12-06 16:07:46 +00:00
Ian Campbell
f9893d9a9b Fixup test after Image → ImageConfig split.
The syntax used for the yaml definitions is changed by the need to include the
substruct in the struct literal.

For the label switch to `ImageConfig` directly, which is actually more correct
in that it avoids spurious `name` and `image` fields in the label.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-11-27 14:47:21 +00:00
Ian Campbell
aec82c4cdf Split config-related fields of Image into a substruct.
Where "config-related" here means "ones you might find in the
"org.mobyproject.config" label on an image.

By making this new struct an anonymous member of the existing Image struct the
Go json parser does the right thing (i.e. inlines into the parent) when parsing
a complete image (from a yml assembly) by default. The Go yaml library which we
use requires a tag on the anonymous field to achieve the same.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-11-27 14:16:59 +00:00
Justin Cormack
3389f89c44 Create tmp dir in case required
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-11-20 16:50:58 +00:00
Justin Cormack
6395e339b9 Do not error if cache/config directory not specified
Just use the default. Easier to use as an external library.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-11-20 13:52:53 +00:00
Justin Cormack
c9bb1ecdf1 Add Cgroups to runtime config
This will create the list of cgroups specified

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-11-17 13:22:38 +00:00
Dennis Chen
90f1582988 EFI image hash value update
Update the hash value of EFI-* docker image

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2017-11-10 08:10:22 +00:00
Justin Cormack
45333d2480 Update dependencies to use sirupsen not Sirupsen
That entailed a few other fixes, eg small Notary API changes.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-11-01 15:14:58 +00:00
Avi Deitcher
31187652ce Fix prereqs, raw images with correct calcs
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2017-10-31 10:51:55 +02:00
Justin Cormack
14ed0953cd Fix some lint errors with error checking
This is due to lack of rebase after linter changes.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-10-30 13:25:22 +00:00
Justin Cormack
4a95c04300 Merge pull request #169 from deitch/add-raw-efi-bios-rename-aws
Add raw file formats, rename raw to aws and qcow2 to qcow2-bios
2017-10-30 11:28:53 +00:00
Justin Cormack
626b41fd30 Merge pull request #162 from ebriand/init-processing-logging
Add back init image processing logging
2017-10-30 11:24:53 +00:00
Justin Cormack
4ed9d49d3a Merge pull request #168 from dnephin/improve-the-build
Improvements to Makefile, linting, and circleci.yml
2017-10-30 11:24:24 +00:00
Avi Deitcher
60c1ce5156 Add raw file formats, rename raw to aws and qcow2 to qcow2-bios
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2017-10-23 20:27:47 -04:00
Ian Campbell
d9d6b56153 Do an explicit docker pull before docker run.
This avoids https://github.com/docker/cli/issues/631

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-20 15:51:47 +01:00
Daniel Nephin
a224547540 Fix lint errors
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-10-17 17:30:47 -04:00
Rolf Neugebauer
67ce80d213 Fix mkimage-iso-bios hash
Looks like a6b89f1137 ("Update linuxkit/mkimage-*") updated to a
non-existing tag.

linuxkit pkg show-tag tools/mkimage-iso-bios
linuxkit/mkimage-iso-bios:165b051322578cb0c2a4f16253b20f7d2797a502

and docker pull of that image works.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-16 12:52:02 +02:00
ebriand
0754e4d662 Add back init image processing logging
Signed-off-by: ebriand <eric.briand@gmail.com>
2017-10-16 11:00:19 +02:00
Rolf Neugebauer
c1da07b3a8 output: Add support for creating Raspberry Pi images
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-13 18:02:17 +01:00
Ian Campbell
a6b89f1137 Update linuxkit/mkimage-*
These versions were created by https://github.com/linuxkit/linuxkit/pull/2607
which enables content trust, so drop the sha256 from all of them and ensure
DOCKER_CONTENT_TRUST is unconditionally set when running, since these
references are hardcoded we know they must be signed.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-13 10:56:21 +01:00
Ian Campbell
8825ca2e66 Disable network in dockerRun
AFAICT none of the callers (which all involve one of `linuxkit/mkimage-*`) have
any reason to hit the network.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-13 10:42:40 +01:00
Justin Cormack
70c6aebd26 Merge pull request #159 from rn/img-ref
Restructure Image and image reference code
2017-10-09 12:29:13 +01:00
Rolf Neugebauer
1aee0c3d30 Update Image strings before writing them out
If the YAML file contains:
  - path: etc/linuxkit.yml
    metadata: yaml
in the fil section, the image was build with content trust,
then the linuxkit.yml file image contains fully qualified
image references (including the sha256).

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:59:02 +01:00
Rolf Neugebauer
ad83cb8928 Use containerd reference.Spec in place of the Image string
Instead of passing the image name as string use the a reference
to a containerd reference.Spec. This allows us, for example,
to update the reference in place when verifying content trust
with more specific information, such as the sha256

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:37:22 +01:00
Rolf Neugebauer
d9b79548a5 Add a containerd reference.Spec for the Image name
When constructing a Moby structure from a YAML also
extract a containerd reference.Spec for each image
and the kernel.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:26:42 +01:00
Rolf Neugebauer
1713f59e4f Turn Images into references
We want to modify some of the content of the Image structure
and thus have to pass them by reference.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:26:42 +01:00
Justin Cormack
e73a7b0315 Add a "tar-kernel-initrd" output format
This is a tarball of the kernel, initrd and cmdline files, suitable for
sending to the mkimage images that expect this format.

Note you can't currently stream this output format using `-o` will clean this
up in future commits.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-10-04 17:01:13 +01:00
Justin Cormack
aca26f00c2 Remove the hyperkit option
We are going to phase out the LinuxKit build option, in favour of keeping Docker
or a native Linux build option for CI use cases, as it is faster. So the
hyperkit option that only worked in one very limited use case is not needed.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-10-03 15:55:02 +01:00
Justin Cormack
cc823286f4 Merge pull request #153 from ijc/subassign-runtime
More granular merge of Runtime from labels & yaml
2017-10-02 11:40:14 +01:00
Justin Cormack
1fc16c4580 Update to use new common iso EFI for amd64 and arm64
See https://github.com/linuxkit/linuxkit/pull/2524

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-09-25 11:02:37 -07:00
Ian Campbell
a5a3b778c0 More granular merge of Runtime from labels & yaml
Previously any Runtime specified in yml would completely override anything from
the image label, even if they set distinct fields. This pushes the merging down
to the next layer, and in the case of BindNS down two layers.

Most of the fields involved needed to become pointers to support this, which
required a smattering of other changes to cope. As well as the local test suite
this has been put through the linuxkit test suite (as of cc200d296a).

I also tested in the scenario which caused me to file #152.

Fixes #152.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-09-22 13:18:36 +01:00
Justin Cormack
63eae8abee Merge pull request #149 from justincormack/shared-init
Used shared init with LinuxKit
2017-09-18 18:10:08 -07:00
Justin Cormack
dff6dc2876 Used shared init with LinuxKit
With https://github.com/linuxkit/linuxkit/pull/2500 the init script
can be used for both bare metal and in Docker.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-09-18 13:11:37 -07:00
Ian Campbell
411a21ae07 Make "metadata: yaml" actually output yaml
The existing code made json...

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-09-18 15:13:32 +01:00