Commit Graph

156 Commits

Author SHA1 Message Date
Justin Cormack
70c6aebd26 Merge pull request #159 from rn/img-ref
Restructure Image and image reference code
2017-10-09 12:29:13 +01:00
Rolf Neugebauer
1aee0c3d30 Update Image strings before writing them out
If the YAML file contains:
  - path: etc/linuxkit.yml
    metadata: yaml
in the fil section, the image was build with content trust,
then the linuxkit.yml file image contains fully qualified
image references (including the sha256).

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:59:02 +01:00
Rolf Neugebauer
ad83cb8928 Use containerd reference.Spec in place of the Image string
Instead of passing the image name as string use the a reference
to a containerd reference.Spec. This allows us, for example,
to update the reference in place when verifying content trust
with more specific information, such as the sha256

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:37:22 +01:00
Rolf Neugebauer
d9b79548a5 Add a containerd reference.Spec for the Image name
When constructing a Moby structure from a YAML also
extract a containerd reference.Spec for each image
and the kernel.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:26:42 +01:00
Rolf Neugebauer
1713f59e4f Turn Images into references
We want to modify some of the content of the Image structure
and thus have to pass them by reference.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-10-04 17:26:42 +01:00
Justin Cormack
e73a7b0315 Add a "tar-kernel-initrd" output format
This is a tarball of the kernel, initrd and cmdline files, suitable for
sending to the mkimage images that expect this format.

Note you can't currently stream this output format using `-o` will clean this
up in future commits.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-10-04 17:01:13 +01:00
Justin Cormack
aca26f00c2 Remove the hyperkit option
We are going to phase out the LinuxKit build option, in favour of keeping Docker
or a native Linux build option for CI use cases, as it is faster. So the
hyperkit option that only worked in one very limited use case is not needed.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-10-03 15:55:02 +01:00
Justin Cormack
cc823286f4 Merge pull request #153 from ijc/subassign-runtime
More granular merge of Runtime from labels & yaml
2017-10-02 11:40:14 +01:00
Justin Cormack
1fc16c4580 Update to use new common iso EFI for amd64 and arm64
See https://github.com/linuxkit/linuxkit/pull/2524

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-09-25 11:02:37 -07:00
Ian Campbell
a5a3b778c0 More granular merge of Runtime from labels & yaml
Previously any Runtime specified in yml would completely override anything from
the image label, even if they set distinct fields. This pushes the merging down
to the next layer, and in the case of BindNS down two layers.

Most of the fields involved needed to become pointers to support this, which
required a smattering of other changes to cope. As well as the local test suite
this has been put through the linuxkit test suite (as of cc200d296a).

I also tested in the scenario which caused me to file #152.

Fixes #152.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-09-22 13:18:36 +01:00
Justin Cormack
63eae8abee Merge pull request #149 from justincormack/shared-init
Used shared init with LinuxKit
2017-09-18 18:10:08 -07:00
Justin Cormack
dff6dc2876 Used shared init with LinuxKit
With https://github.com/linuxkit/linuxkit/pull/2500 the init script
can be used for both bare metal and in Docker.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-09-18 13:11:37 -07:00
Ian Campbell
411a21ae07 Make "metadata: yaml" actually output yaml
The existing code made json...

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-09-18 15:13:32 +01:00
Justin Cormack
11b573c6fb Rename "output" to "format"
This was confusing as there is an option to output to a file as well.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-09-01 23:45:25 +01:00
Justin Cormack
cfa5d273b7 De-dup container root filesystems
With the mount framework we can de-dup containers that share the same image.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-22 17:27:08 +01:00
Justin Cormack
6b98aff58b Use the runtime config to specify how overlay should be created
This puts the build side in charge of the runtime layout, which enables
additional optimisations later, like sharing the rootfs if it is
used multiple times.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-22 16:21:56 +01:00
Justin Cormack
ee0901762f Add support for mount in runtime config
This could be used in LinuxKit now, as there are some examples, eg
https://github.com/linuxkit/linuxkit/blob/master/blueprints/docker-for-mac/base.yml#L33
which are creating containers to do a mount.

The main reason though is to in future change the ad hoc code that generates
overlay mounts for writeable containers with a runtime config which does
the same thing; this code needs to create both tmpfs and overlay mounts.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-22 15:17:54 +01:00
Justin Cormack
ea60eff557 Add support for runtime configuration
This adds a `runtime` section in the config that can be used
to move network interfaces into a container, create directories,
and bind mount container namespaces into the filesystem.

See also https://github.com/linuxkit/linuxkit/pull/2413

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-17 17:36:49 +01:00
Justin Cormack
4da3c09e19 Do not use filepath.Join if constructing LinuxKit paths
This will do the wrong thing on Windows, and construct paths with \.

fix #142

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-14 15:38:58 +01:00
Justin Cormack
13b1eb7d0c Do not use rm as may not be in root filesystem
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-10 14:59:32 +01:00
Justin Cormack
13a1ffdd44 Add Cgroup resource limits into Yaml spec
This adds all the cgroup restrictions into the spec.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-09 16:39:10 +01:00
Justin Cormack
7b36db8358 Output EFI ISO as for BIOS ISO with filesystem on.
The image is multi-arch and should work on arm64 as it has no boot loader...

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-07 16:44:13 +01:00
Justin Cormack
2d930bc8e5 Use new style ISO BIOS mkimage from Linuxkit which unpacks filesystem
Rather than using an initrd, unpack full filesystem for ISO BIOS.

Stream docker output direct to file rather than via a buffer, to save
memory.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-02 14:24:24 +01:00
Justin Cormack
48e42540d2 Fix hardlinks in tar output
When we converted these to cpio we were not noticing that they
were invalid as they had incorrect paths as we converted the
path to a symlink anyway. Only the busybox images have hard links
in, the Alpine ones are symlinks anyway, which is why it was
less visible too.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-01 17:04:57 +01:00
Justin Cormack
3dec6855e0 Fix read only output when defined in a label
Also do some code cleanup.

Related to #131 we need to read the OCI config to find if the container
is read only, not rely on the yaml, as it may just be set in the label.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-28 14:34:18 +01:00
Justin Cormack
7010eae061 Update mkimage used for building to lastest hashes
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-28 13:35:59 +01:00
Justin Cormack
e388c0ff14 Rework setup of container for read only/read write
To work with truly immutable filesystems, rather than ones
we sneakily remount `rw`, we are going to use overlay for
writeable containers. To leave the final mount as `rootfs`,
in the writeable case we make a new `lower` path for the read
only filesystem, and leave `rootfs` as a mount point for an
overlay, with the writable layer and workdir mounted as a tmpfs
on `tmp`.

See https://github.com/linuxkit/linuxkit/issues/2288

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-27 14:52:13 +01:00
Justin Cormack
6403215635 In the init section use a symlink for /etc/resolv.conf
Unfortunately there are a lot of issues with resolv.conf as we
cannot actually write it into the image from any docker image, as docker will
always have something bind mounted in.

In addition, normally we expect the filesystem to br read only for images
that moby generates, so the actual etc/resolv.conf is likely not to be writeable.

Previously we were adding in a default resolv.conf into every image pointing at
Google's name servers but that is really a bad idea.

Instead, normal images now get an empty default, while images in the `init`
section will get a symlink, currently hard coded to `/run/resolvconf/resolv.conf`
but you can override this with the `files` section to be static or a different
link.

In future, if we have an easy way to build and extract images with user control
of this, we can drop this.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-25 14:40:40 +01:00
Justin Cormack
2c4d567781 Merge pull request #125 from justincormack/tmpdir
Add a tmp directory for each container to be used as a mount point
2017-07-24 17:19:32 +01:00
Ian Campbell
e8d5005b0e Synchronise vendoring with containerd v1.0.0-alpha1
Some of these are arbitrary and just syncing for the sake of it, however the
image- and runtime-spec are relevant. Interesting changes:

- runtime spec:
  - LinuxRLimit is now POSIXRLimit.
  - Specs.Config is now a pointer.
  - LinuxResources.DisableOOMKiller moved to
    LinuxResources.LinuxMemory.DisableOOMKiller
- image spec:
  - Platform.Features is removed (unused here).

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-07-20 12:03:22 +01:00
Justin Cormack
4720490c92 Add a tmp directory for each container to be used as a mount point
This is to clean up how we do writeable containers in LinuxKit.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-19 11:29:03 +01:00
Justin Cormack
6eb74b2f2b Add support for onshutdown
This is a list of images to run on a clean shutdown. Note that you must not rely on these
being run at all, as machines may be be powered off or shut down without having time to run
these scripts. If you add anything here you should test both in the case where they are
run and when they are not. Most systems are likely to be "crash only" and not have any setup here,
but you can attempt to deregister cleanly from a network service here, rather than relying
on timeouts, for example.

Fix https://github.com/linuxkit/linuxkit/issues/1988

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-18 13:55:52 +01:00
Justin Cormack
55f7df577a Add json config for top level config
Otherwise JSON output as metadata cannot actually be used as it
has capitalised field names.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-18 13:16:58 +01:00
Justin Cormack
12439d947d Do not modify data structure while building
This will give odd effects if we output the yaml structure into the image.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-17 15:42:26 +01:00
Justin Cormack
040c500648 Add more omitempty to json spec
The metadata config was rather verbose.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-17 15:31:31 +01:00
Justin Cormack
389dd8c0fa Add a "metadata" file contents flag
Currently this supports "yaml" as the only option, which will output
the yaml config (as JSON) into the file specified in the image.

Fix #107

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-17 15:31:24 +01:00
Riyaz Faizullabhoy
c3b2c7d575 trust: do not call x509.SystemCertPool as nil will use this as default
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
2017-07-14 14:39:39 -07:00
Justin Cormack
cc33e67492 Allow uid, gid fields to be numeric or names
Previously I was forcing them to be strings, which is horrible. Now you
can either specify a numeric uid or the name of a service to use the
allocated id for that service.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-14 16:57:12 +01:00
Justin Cormack
830be373da Allow specifying the kernel and tarball names, or omitting tarball
fix #113

Use `tar: none` or `tar: ""` to omit the tarball.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-14 14:09:42 +01:00
Justin Cormack
7bc763a628 Move Contributing to top level
- accidentally committed to wrong directory

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-14 11:50:51 +01:00
Ian Campbell
f20828bf7e Update LinuxKit components used for build
This updates to the current set of images as of
https://github.com/linuxkit/linuxkit/pull/2159

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-07-13 14:43:34 +01:00
Justin Cormack
b4804b69e8 Add Contributing doc
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-12 12:04:33 +01:00
Justin Cormack
51b4e20154 Merge pull request #103 from ijc/master
Build for Darwin and Windows
2017-07-11 11:07:36 +01:00
Justin Cormack
3ee4fdc43a Error if there are duplicate service names
Continue to allow onboot to have duplicates as we do not run simultaneously
so that is ok (and we number them anyway), but services are run together
so we will get a runtime error if duplicated as this is the containerd/runc
id.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-10 14:50:04 +01:00
Ian Campbell
3c5fad4539 Remove dependency on docker/docker/cli
We were pulling in this whole stack of packages just for `trust.ReleasesRole`.
Just define it locally.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-07-10 14:38:21 +01:00
Justin Cormack
ec7e73b304 Merge pull request #100 from justincormack/userns
Add partial user namespace support
2017-07-10 14:30:30 +01:00
Ian Campbell
c3edfec759 revendor runc + runtime-spec to match containerd 8e1a04ff9ee3
Note that various fields have changed moved around in the JSON as a result:

* `Platform` has been removed.
* `Process` is now a pointer.
* `OOMScoreAdj` has moved into `Process`, from `Linux.Resources` (resolving a
  TODO here).

Also updates golang.org/x/sys which is less critical.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-07-06 15:34:01 +01:00
Justin Cormack
a73c3d3667 Add partial user namespace support
This adds the OCI parts needed into the yaml, but there are still
permissions issues in practise so marked as experimental.

It may just need further documentation to resolve the issues.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-03 14:45:58 +01:00
Justin Cormack
0acaaa71fe Assign each container a uid and gid it can use
In order to support not running containers as root, allocate
each of them a uid and gid, a bit like traditional Unix system
service IDs. These can be referred to elsewhere by the name of
the container, eg if you wish to create a file owned by a
particular esrvice.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-30 17:40:08 +01:00
Justin Cormack
66b4aed239 Add support for ambient capabilities
Allow setting ambient capabilities, as a seperate option to the standard
ones. If you are running as a non root user you should use these.

Note that unless you add `CAP_DAC_OVERRIDE` and similar permissions you
need to be careful about file ownership. Added support to set ownership
in the `files` section to help out with this.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-30 14:08:16 +01:00