This was done with the following "script":
git rm pkg/{auditd,binfmt,init}/Makefile
sed -e 's/IMAGE=/image: /g' -i pkg/*/Makefile
sed -e 's/NETWORK=1/network: true/g' -i pkg/*/Makefile
sed -e 's/ARCHES=x86_64/arches:\n - amd64/g' -i pkg/*/Makefile
sed -e '/DEPS:\?=/d' -i pkg/*/Makefile
sed -e '/ARCHES=SKIP/d' -i pkg/node_exporter/Makefile
sed -e 's/include \.\.\/package.mk//g' -i pkg/*/Makefile
sed -e '/^$/d' -i pkg/*/Makefile
git mv pkg/node_exporter/Makefile pkg/node_exporter/build.yml-skip
for i in pkg/*/Makefile ; do git mv $i ${i%Makefile}build.yml ; done
and manual update of pkg/Makefile.
Signed-off-by: Ian Campbell <ijc@docker.com>
Using explicit targets is cleaner and easier to follow.
Also add `forcepush` to pkg and test/pkg and make `push` explicit.
The `dirs` phony was non-existent in all three cases. Likewise `$(DIR)` wasn't
used in that way in pkg or test/pkg and is nolonger used in that way in tools.
Finally, `clean` was bogus in all cases (tools cleaned a thing which never
exists and pkg and test/pkg were a syntax error).
Signed-off-by: Ian Campbell <ijc@docker.com>
This implements the proposal in #2564 and converts a handful of representative
or especially interesting (from a build PoV) packages to use it.
For now those pkg/* affected get a stub-`Makefile`, once all packages are
converted then `pkg/Makefile` can be adjusted and those stubs can be removed.
For now only `pkg/package.mk`'s functionality is implemented. In particular:
- `push-manifest.sh` remains a separate script, to enable calling it on systems
with just the LinuxKit tools installed arrange to install it under a less
generic name.
- `kernel` and `tools/alpine` do not use `pkg/package.mk` and those cases are
not yet fully considered/covered.
I have updated the documentation assuming that the existing uses of
`pkg/package.mk` will be removed quite soon in a follow up PR rather than
trying to document the situation which results after just this commit.
Due to `cmd/linuxkit` now gaining a library the build needs adjusting slightly to
allow both `make bin/linuxkit` and `go build` to work.
`go vet` has forced me to write some rather asinine comments for things that
are rather obvious from the name.
Signed-off-by: Ian Campbell <ijc@docker.com>
These packages include the wireguard tools which need to be
updated due to the wireguard kernel bump in:
43db718f14 ("wireguard: version bump").
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
- Added new argument -force to the format utility. This will force formatting of the specified single device if it exists and is a block device
- By default, the format package will no longer format the specified single device if content exists on that device
- Added new blkid-based check for content, in addition to the existing check for partitions on autoformat candidate devices.
- Cleanup of old unused code.
- Refactoring of block device verification.
- Added test/cases/040_packages/006_format_mount/005_by_device_force
- Added document of new arguments to /docs/external-disk.md. Also sorted the arguments in the doc alphabetically and added them as bullets so they do not run together on the page.
Signed-off-by: Isaac Rodman <isaac@eyz.us>
Some software (such as cri-containerd and kubelet) rely on the presence of
these links, which are usually created by udev.
Inspired by 74b145b5c7/sourceroot/functions.sh (L93...L124)
but rewritten in go.
Signed-off-by: Ian Campbell <ijc@docker.com>
golint on pkg/init now complains:
golint...
./init.go:199:2: redundant if ...; err != nil check, just return error instead.
Resulting in a change which doesn't seem like an improvement to me.
Signed-off-by: Ian Campbell <ijc@docker.com>
We always had 1G swap to work better with small memory setups, but this
was omitted in the update to LinuxKit.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This removes more shell scripts to improve maintainability.
This now also works correctly in userspace, so it can be used for
running LinuxKit images in Docker and other such use cases.
It is a literal conversion of the shell scripts with a few small
tweaks.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This removes things we do not need and expects the interface to
already be created.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Previously we would pass the path `/var/log/service.log` for both
stdout and stderr to containerd. containerd would construct a dict
with the paths as keys[1] and, due to the duplicate key, would only
open one of the files and start one `io.Copy` instance. Writes to
the other stream would be buffered by the pipe connected to
containerd-shim and would eventually block.
If we modified containerd to open the file twice and start 2
`io.Copy` instances, we would end up with the two streams interleaved
together. It seems cleaner to keep the streams separate; therefore
this patch logs stdout to `/var/log/service.out.log` and stderr to
`/var/log/service.err.log`.
[1]
49437711c3/linux/shim/io.go (L51)
Signed-off-by: David Scott <dave.scott@docker.com>
This removes all the code that had knowledge of how to do read only
and read write container mounts, and just uses the runtime config.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This could be used in LinuxKit now, as there are some examples, eg
https://github.com/linuxkit/linuxkit/blob/master/blueprints/docker-for-mac/base.yml#L33
which are creating containers to do a mount.
The main reason though is to in future change the ad hoc code that generates
overlay mounts for writeable containers with a runtime config which does
the same thing; this code needs to create both tmpfs and overlay mounts.
See https://github.com/moby/tool/pull/145
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
The metadata package has binds
- /dev
- /var
- /sys
- /etc/resolv.conf
- /etc/ssl/certs
but unfortunately `/etc/ssl/certs` doesn't exist and this causes the
following commands:
cd blueprints/docker-for-mac # easy example
moby build -name docker-for-mac base.yml docker-17.06-ce.yml
linuxkit run hyperkit -networking=vpnkit -vsock-ports=2376 -disk size=500M docker-for-mac
to produce the following error on the VM console:
container_linux.go:265: starting container process caused "process_linux.go:348: container init caused \"rootfs_linux.go:57: mounting \\\"/etc/ssl/certs\\\" to rootfs \\\"/containers/onboot/000-metadata/rootfs\\\" at \\\"/etc/ssl/certs\\\" caused \\\"stat /etc/ssl/certs: no such file or directory\\\"\""
2017/08/21 16:39:40 Error creating 000-metadata: exit status 1
This patch creates /etc/ssl/certs in the `init` package. The metadata package
will now say things like
2017/08/21 16:44:39 No metadata/userdata found. Bye
Signed-off-by: David Scott <dave.scott@docker.com>
As discussed before, as we use this in three places, cloning in
base makes more sense.
Update base image.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Use the packet metadata to set up bonding, also get ssh keys and hostname.
This does not yet do anything with disk metadata.
Userdata is not used if it has been used for ipxe, but is otherwise available.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This contans the upstream (non busybox) version of `ip` and also the
Wireguard tools as these aere not yet integrated into `ip`. It is designed
to be used for network configuration.
For example a Wireguard interface can be setup with something like:
```
- name: wg
image: linuxkit/ip
net: new
binds:
- /etc/wireguard:/etc/wireguard
command: ["sh", "-c", "ip link set dev wg0 up; ip address add dev wg0 192.168.2.1 peer 192.168.2.2; wg setconf wg0 /etc/wireguard/wg0.conf; wg show wg0"]
runtime:
interfaces:
- name: wg0
add: wireguard
createInRoot: true
bindNS:
net: /run/netns/wg
```
This binds the net namespace so other containers can use it later.
Note that `ip` supports batch scripts with `ip -b file` which is quite convenient for
this use case.
Alas `wg-quick` requires `bash` so this is currently a bit bloated, maybe @zx2c4
might be persuaded to rewrite it without bashisms...
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This adds support for a runtime configuration file that can do:
- `mkdir` to make a directory at runtime, eg in `/var` or `/tmp`, to avoid workarounds
- `interface` that can create network interfaces in a container or move them
- `bindNS` that can bind mount namespaces of an `onboot` container to a file so a service can be started in that namespace.
It merges the `service` and `onboot` tools (in `init`) to avoid duplication. This also saves some size for
eg LCOW which did not use the `onboot` code in `runc`.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
During development I tend to do:
make -C pkg/foo tag HASH=dev ORG=ijc
and I expect to get an image `ijc/foo:dev` and not `ijc/foo:dev-amd64`. In
general I think if the HASH has been explicitly provided the build should
just honour that.
To compensate and avoid mistaken pushes adjust the check for dirtiness to also
insist on a non-empty suffix.
Signed-off-by: Ian Campbell <ijc@docker.com>
At least when running under Docker this seems to cause some issues,
possibly also explains some other oddnesses.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This has a number of fixes/improvements
- console resize
- better error handling (pass stacktrace/errors back to docker
- fix for container images which have a resolve.conf file
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Chown clears suid bits even for root on Linux.
Also move a few functions to x/sys/unix from syscall, to be
more arm64 friendly.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Runc is master as of this morning, slightly newer than what is vendored into
containerd v1.0.0-alpha3 but the differences[0] seemed benign and/or good to have.
Signed-off-by: Ian Campbell <ijc@docker.com>
[0] 429a538712...45bde006ca
This requires that users have `CAP_SYSLOG` in order to access `dmesg`.
This means that containers by default have no access to `dmesg` (which
can leak information about the host or other containers) unless they
have this capability added.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
They either won't work or the packages are specific
to x86_64 like the DfM or VMware packages.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
A package makefile can specify a list of architectures ('ARCHES')
it supports. If build on a unsupported architecture, a message is
printed and make exits without an error.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This checks if any CDROM in the system has metadata. Useful if
you boot off CDROM and also have a metadata CDROM.
Also switch the metadata container to read only, as it is only
writing to bind mounted in directories, and /tmp.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Epoll in `syscall` package does not work on arm64. We do not have
arm64 yet here, but we will when we add tpm support etc.
Signed-off-by: Justin Cormack <justin@specialbusservice.com>
Only supports the use cases we currently need, currently support
for using Intel hardware rng to initialise and add entropy.
Supports oneshot and service mode. Call as `rngd -1` for one shot
mode.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Previously we were cheating and remounting /var `rw` but this does not
work if the filesystem is really read only. Nount a tmpfs, which may
be overmounted later by a persistent filesystem.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Previously we would sneakily remount as `rw` but of course you can't
really do that on a truly immutable filesystem.
See https://github.com/moby/tool/pull/129 for the `moby` side.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
We need to add 'ttyAMA0' console used on ARM64 platform into
securetty file to make it's possible to log into the system
as root. Also it will dismiss the below warning message before
login:
"getty: cmdline has console=ttyAMA0 but does not exist in
/etc/securetty; will not be able to log in as root on this tty ttyAMA0."
'ttyAMA1' is also added order to handle some system happens have that one
instead of 'ttyAMA0'
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Current package.mk will operate on the first dependence object
'check-dirty', which will result the make process abort earlier
with below message:
pkg/package.mk:60: *** Your repository is not clean. Will not push package image. Stop.
This is not expected behavior since we have no chance to build
a docker image locally in case of the user has no intention to
push that image.
This patch adjust the dependece order for the default target, thus
we can build the docker image locally but can't push that image
since we're in a dirty git repository.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
The logrus import path has changed, so adjust. Also there is a minor API change
to the containerd.IOCreation() function spec, it now takes a string id which we
can ignore.
Signed-off-by: Ian Campbell <ijc@docker.com>
At present they use a small shared function called "prepare"
that does the read-write remounts, that I will switch to doing overlay
mounts soon.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This was split out from pkg/format into its own package.
It has the ability to extend ext4, btrfs and xfs partitions.
Signed-off-by: Dave Tucker <dt@docker.com>
This commit rewrites the mount package in Go.
It adds the ability to mount the by label, UUID or name.
It also fixes the automatic mount logic to check that a drive isn't
already mounted before attempting to mount it. This allows for multiple
uses of the mount pkg in a single YAML file.
Signed-off-by: Dave Tucker <dt@docker.com>
This commit re-writes the format package in Go and adds the ability to
add partition labels and also specify either ext4, btrfs or xfs
Signed-off-by: Dave Tucker <dt@docker.com>
This moves most of the initialisation of containers to the
service init in the `service` command.
Still leaves remounting root file systems read only but this
will go away shortly. Another step closer to removing shell
scripts in base system.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This is like the `push` target but omits the pulls and depends on forcetag
instead. With the git commit now being embedded into the image this is now a
necessary part of rebasing a PR for which images have already been pushed.
Also adds PHONY targets for existing forcetag and push targets which were
missing.
NB $(error) appends a "." to omit the final one from the error message
Signed-off-by: Ian Campbell <ijc@docker.com>
The filesystem is supposed to be immutable, so do not try to make
a symlink; new versions of moby tool should add one anyway. But
try to make the directory a symlink points to, assuming that it
will be on a writeable filesystem.
fix#1920
see also #2288
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This uses a more memory efficient copy, and gets us closer to
not having a shell in the base system if not required.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
People really want to play around with this, so adding them here makes
it possible. Just as iproute2 is part of these, so should
wireguard-tools.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Since these are the user login containers, having the ability to add packages
is useful (e.g. I quite often find I want strace).
Doing this requires that we not share `/var` with the login containers since we
want the apk database therein. Previously it was thought that the containers
might need some parts of `/var` for `ctr` to work (e.g. `/var/lib/containerd`)
but this is not the case now (if it ever was) based on my testing.
Fixes#2206.
Signed-off-by: Ian Campbell <ijc@docker.com>
This is actually containerd#1141 rebased onto v1.0.0-alpha1.
The `dist` command has been integreated into `ctr` and so is removed, including
from the getty and sshd bind mounts and the test which uses it is updated..
There is no change to the version of runc vendored by containerd, so this is
unchanged.
Signed-off-by: Ian Campbell <ijc@docker.com>
If the configuration .json has contents like:
{
"etc": {
"ssl": {
"certs": {
"ca-certificates.crt": {
"perm": "0644",
"content": "large amount of certificate text"
}
}
}
},
...
}
then we print a warning because the node "ssl" has no "perm".
Previously the warning would include the contents of "ssl", which
would be large (and in theory could include secret information).
This patch modifies the warning print to only print the key and
not the value.
Without this patch, I see on the console:
2017/07/20 10:03:04 CDROM: Probe succeeded
2017/07/20 10:03:04 No permission provided ssl:map[certs:map[ca-certificates.crt:map[perm:0644 content:large amount of certificate text]]]
- 000-metadata
With this patch, I see on the console:
2017/07/20 09:54:18 CDROM: Probe succeeded
2017/07/20 09:54:18 No permission provided ssl
- 000-metadata
Signed-off-by: David Scott <dave.scott@docker.com>
The sample init script from opengcs is quite basic (and doesn't
set up several mounts and symlinks). Use the relevant portion
from rc.init from the LinuxKit init package instead.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This updates the build of the two image caches to use the `pkg/package.mk`
infrastructure, albeit in a slightly (ok, very) atypical way.
In order to share the bulk of the build code (including the `Dockerfile` and
the `Makefile` machinery to download the images) we arrange for the necessary
bits to be copied at build time into distinct subdirectories and for the
`pkg/package.mk` to be aware of this possibility.
Since pkg/package.mk is only set up to build a single package we use a single
`image-cache/Makefile` to drive the whole process and recurse into
`Makefile.pkg` to build individual packages.
One particular subtlety is that the package hash is based on the `image-cache`
directory (which is in `git`) rather than the generated subdirectories (which
are not in `git`). Since all the generators (and their inputs) are in the
`image-cache` directory this is what we want. This means that the two images
are given the same tag, but this is deliberate and desirable.
The generated directories are completely temporary to avoid picking up stale
versions of images when versions are updated. Images are hardlinked into place.
The images are moved to the linuxkitprojects org. Using a dev tag for now, will
update once everything is in place.
Also use "tag" rather than "build" where appropriate in the Makefile.
There is no point in the .dockerignore now, but add a .gitignore.
Signed-off-by: Ian Campbell <ijc@docker.com>
These can be added by other packages if they need to do something on
clean shutdown.
Crash only software can ignore this.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
It is pretty close to our docker package, if we adjust the command
that is run to avoid the actual dind startup script. We can't use
the normal docker image as it does not have mkfs and so on.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
We want them to run in sequence. For example we want mounts to be done (done by
`pkg/runc/etc/init.d/010-onboot`) before we start services (done by
`pkg/containerd/etc/init.d/020-containerd`). This was most likely introduced by
28b4245b12 ("Move onboot startup script to runc package").
None of the initscripts in pkg/* block, but some in projects (selinux and
logging, not updated here) do.
Signed-off-by: Ian Campbell <ijc@docker.com>
Do not copy host securetty file - this one should be comprehensive
or bind mount host one in yourself.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This package build the init filesystem for LCOW (Linux Containers
on Windows) based on the instructions from:
https://github.com/Microsoft/opengcs.git
We also pull in a udhcpd config script from a specific version of
busybox which was the tip of master at the time this was added.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
As this does not use containerd at all, this means you can run very
minimal setups with just `runc` if you use no services, for example
most of our tests do not actually use services, or if you have other
similar very minimal use cases.
Move ulimit setup to `init` which makes more sense.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
We expect this (or something very similar) to be merged soon, it fixes#2131 so
moving ahead now.
The new alpine mirror is linuxkit/alpine:6832775a7e861ee2d7842e157688ece52d007142
Signed-off-by: Ian Campbell <ijc@docker.com>
This will make it simpler to temporarily in PRs which are not yet merged (but
are expected to be soon).
Tools alpine is not rebuilt here since we are going to do just that in the next
commit.
Signed-off-by: Ian Campbell <ijc@docker.com>
pull in newer containerd v1.0.0-alpha0 via updated alpine base, update runc to
429a5387123625040bacfbb60d96b1cbd02293ab which is vendored by that version of
containerd (and also update alpine base for runc)
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
These are not needed, but we are inconsistent. Been waiting for a
quiet moment to fix this since I noticed while doing a presentation...
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Use os.Mkdirall() instead of os.Mkdir() as it does not
error if the path already exists.
This enables specifying a default config file with the image
and then over writing it with metadata.
While at it, also update to the latest alpine base image.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Some hypervisors (e.g. hyperkit / xhyve) don't provide a good way to keep
the VM's clock in sync with the Host's clock. NTP will usually keep the
clocks together, but after a the host or VM is suspended and resumed the
clocks can be suddenly too far apart for NTP to work properly.
This simple daemon listens on an AF_VSOCK port and resynchronises the VM
clock from the virtualised hardware clock.
This is a Go conversion of original C code written by
Magnus Skjegstad <magnus@skjegstad.com>
Signed-off-by: David Scott <dave.scott@docker.com>
For DIRTY and RELEASED this means simply performing a static assignment with
`:=`. For HASH it is a little more complex since it is (and we want/need it to
be) a conditional assigment. However it is only used for defining TAG, so make
that a static assignment.
This reduces the number of times the complex DIRTY shell command in particular
is evaluated.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Firstly add option to disable content trust, for the use of e.g. projects which
are pushing to the linuxkitprojects org (which has no trust setup) rather than
the main linuxkit org.
Secondly, when trust _is_ enabled then enable it globally, in particular it is
now active for the `docker build` and hence containers referenced in
Dockerfiles via "FROM" will be checked.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Adds a new service for running `/sbin/acpid` that comes with busybox.
The VM will shut down if the power button is pressed.
Signed-off-by: Magnus Skjegstad <magnus@skjegstad.com>
Otherwise files which have an updated timestamp but no actual changes are
marked as changes because `git diff-index` only uses the `lstat` result and not
the actual file contents. Running `git update-index --refresh` updates the
cache.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
If a user passes a HASH e.g. HASH=dev then assume they know what they are doing
and don't need dirty tracking.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Useful to answer the question of what the hash was at the point of some older
commit.
$ make --no-print-directory -C pkg/init/ show-tag
linuxkit/init:36c56f0664d49c5a6adc1120d1bf5ba6ac30b389
$ make --no-print-directory -C pkg/init/ show-tag HASH_COMMIT=4699f80ef73141ee87a77d03f93065e541fab382
linuxkit/init:17693d233dd009b2a3a8d23673cb85969e1dce80
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
The current rune for $(HASH) returns nothing with the Jessie version of git
(2.1.4). Using `--full-tree` works corectly but requires an absolute path (else
it produces e.g. "fatal: ../init: '../init' is outside repository").
Both `ls-tree` and `diff-index` are happy with a full absolute path, which
simplifies things since we can use `$(CURDIR)` directly.
Tested with a dirty `pkg/init` on both Jessie (git 2.1.4) and Stretch (git
2.11.0) with the following command, which produces identical output in both
cases:
$ for pkg in init containerd ; do make -C pkg/$pkg --no-print-directory show-tag; ( cd pkg/$pkg && make show-tag ); done
linuxkit/init:36c56f0664d49c5a6adc1120d1bf5ba6ac30b389-dirty
linuxkit/init:36c56f0664d49c5a6adc1120d1bf5ba6ac30b389-dirty
linuxkit/containerd:1e3e8f207421de8deac8cedc26a138d6b1661a0d
linuxkit/containerd:1e3e8f207421de8deac8cedc26a138d6b1661a0d
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
By setting `DIRTY` to either "-dirty" or "" directly we can simply use
`$(HASH)$(DIRTY)` and avoid make adding a space as it does with `+=`.
For the push check we now block pushing if `$(DIRTY)` is non-empty.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
$ make --no-print-directory -C pkg/init/ show-tag
linuxkit/init:36c56f0664d49c5a6adc1120d1bf5ba6ac30b389
Useful for scripting etc.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
As of commit b2439251dd ("blueprints: update docker-for-mac.yml
to latest vpnkit packages") the docker-ce package does not need
the vpnkit-expose-port binary anymore. It was also removed from
the linuxkit/vpnkit-forwarder forwarder package with commit
f48a2bfe12 ("pkg: update vpnkit-forwarder") causing the docker-ce
package build to fail.
This commit fixes this by removing the linuxkit/vpnkit-forwarder
package from the docker-ce package build.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This fixes a regression introduced in:
5a225b9d ("Makefile: Fix bug where network was not disabled")
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This package contains vpnkit binaries needed inside the dockerd
container that can be either copied into the container with a multistage
build or used as an init-container and bind-mounted to the right
locations.
vpnkit-expose-port is a userland proxy that interacts with vpnkit and
the vpnkit-forwarder service to open ports on the host.
vpnkit-iptables-wrapper wraps iptables to launch
vpnkit-expose-port automatically when ports are opened to a swarm
service.
Signed-off-by: Magnus Skjegstad <magnus@skjegstad.com>
This commit moves the include statement to the bottom of the file to
ensure that all variables are set before conditionals are evaluated.
I also changed the ifndef NETWORK to ifdef NETWORK as the former was
incorrect. We want `NET_OPTS="--network=none"` in cases where NETWORK is
not defined.
Fixes: #2134
Signed-off-by: Dave Tucker <dt@docker.com>
Warn the user in the MOTD
Add "(ns: getty)" or "(ns: sshd)" to the PS1
Use `agetty` and `-a root` to ensure we get a login shell when insecure
Signed-off-by: Dave Tucker <dt@docker.com>
Two things to note here: we unfortunately can't just exec audit, because
something needs to load the rules in beforehand.
Second, it also dies if it can't re-nice itself, so we have to give it
CAP_SYS_NICE as well as the audit caps.
I didn't add this to the default linuxkit.yml because the linuxkit/audit
repo doesn't exist yet, but we should probably (?) do that. Additionally,
we should provide the kernel parameter audit=1, otherwise according to
auditd's man pages, we can't audit some early tasks.
Closes#52
Signed-off-by: Tycho Andersen <tycho@docker.com>
- Use the new style kernel tags with the full kernel version
- Update packages with new alpine base and new/simplified Makefiles.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
People typically update hashes in YAML files with
'git grep ... | sed ...' this will change the README.md files
in the ./pkg directory and thus change the git tree hash.
Remove the use of explicit hashes from those files to avoid
the obvious chicken and egg problem.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This adds a helper which monitors the Docker event stream and, when it
receives an image delete event, queues a call to `fstrim /var/lib/docker`
to trigger a space reclamation. Previously we would rely on a cron job
running every 15 minutes.
Signed-off-by: David Scott <dave.scott@docker.com>
There seems to be a change in docker 17.06-rc5 on how remote
URLs and tar archives work with ADD. Use curl instead.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
In a subsequent commit, all YAML files will be updated with
new package hashes since all packages needed rebuild due to
build system changes in commit adae27b8d1 ("Simplify
Makefiles for Packages"). So, we might as well bring all
packages up to the latest alpine base package.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
- Only build if the hash has changed. Previosuly
we'd unconditionally build and not push of the tag existed.
- Add '.m' to the hash if the repository is dirty.
This allows developers to build locally without committing
but makes it clear that the hash is from a modified repository.
- Don't push to hub if the repository is dirty
- If the current commit has a tag, also push this to hub.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This adds all the binds from pkg/getty/Dockerfile to pkg/sshd/Dockerfile and
retains the existing /root/.ssh and /etc/resolv.conf binds.
With this `dist` and `ctr` function in a ssh session.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Otherwise:
dist pull docker.io/library/redis:alpine
dist: failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/alpine: x509: failed to load system roots and no roots provided
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
ctr (which runs in the getty or ssh container) relies on sharing files in /tmp
with containerd (which runs in the host mount namespace). Specifically it
currently uses paths under /tmp/containerd for the stdio FIFOs of containers, resulting in:
# ctr run -t docker.io/library/redis:alpine test
ctr: rpc error: code = Unknown desc = runtime create failed: runc create failed: container with id exists: test
Currently it is not possible to specify a non-existent source directory for a
bind mount, so we cannot easily bind just /tmp/containerd. Sharing all of /tmp
doesn't sound like a terrible idea anyway.
Defering updating the sha in *.yml until after some further changes to these packages.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Vultr uses a very similar approach to AWS, including using the
same IP address for serving metadata. In fact, it seems
as though if AWS appears first in the list of providers, that
provider mistakenly believes to be running on AWS (hence the
insertion of `NewVultr` in between GCP and AWS. I don't believe
AWS servers will accidentally try to use the Vultr provider,
as it seems that the `/v1/` endpoint doesn't exist on AWS.
Signed-off-by: Luke Hodkinson <furious.luke@gmail.com>
These now inherit from a top-level package.mk
Options like use of the network can be enabled on a per package basis
This removes a lot of duplicate code and make the maintenace of these
Makefiles much easier
Signed-off-by: Dave Tucker <dt@docker.com>
This includes https://github.com/containerd/containerd/pull/994 and hence
requires updating the various instances of `/etc/containerd/config.toml`.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Currently it supports only `service start <SERVICE>`, but it could grow e.g.
`stop`, `exec` etc in the future (although you can still use `ctr` for those).
In order to be able to use go-compile.sh the containerd build needs to move
from /root/go to /go as the GOPATH.
The vendoring situation is not ideal, but since this tool wants to be an exact
match for the containerd it seems tollerable to reuse its vendoring.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
This doesn't exist with newer ctr or in systems where service containers are
not started using the ctr tool. All it contains today are the stdio FIFOs,
which are not in general useful to access after container creation.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
vpnkit-expose-port is the dockerd userland proxy used to expose
forwarding ports with vpnkit. This adds the binary to the image in
/usr/bin/vpnkit-expose-port, but does not enable it by default.
Signed-off-by: Magnus Skjegstad <magnus@skjegstad.com>
The binary is used in tandem with CONFIG_STATIC_USERMODEHELPER=y in 4.11+,
see the big comment in the binary for the current whitelist of binaries.
Signed-off-by: Tycho Andersen <tycho@docker.com>
mdev only supports hot-plug, ie devices which are added
after the kernel booted. For cold-plugged devices, ie devices
present when the kernel boots, we need to modprobe the drivers
differently. The additions to rc.init does just that.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Adds /pkg/vpnkit-forwarder which downloads and builds vpnkit-forwarder
and vpnkit-expose-port from moby/vpnkit. Also includes an example for
forwarding `sshd` and updates the documentation for `hyperkit` to
include a `vpnkit` section.
`vpnkit` still requires a 9P mount for coordination, so the `socat`
forwarding is simpler to set up at the moment.
Signed-off-by: Magnus Skjegstad <magnus@skjegstad.com>
This at least checks for buildability of packages, even if we
are not yet pushing them.
See https://github.com/linuxkit/linuxkit/issues/1991 for what it mitigates.
Will not pass until this is fixed.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
0eb21735ae accidentally broke some package
builds by switching linuxkit/alpine to linuxkit/containerd. Let's revert
the ones that shouldn't be there.
Closes#1991
Signed-off-by: Tycho Andersen <tycho@docker.com>
This can be turned on if required but we'll default to leaving it off
It's been enabled in the top-level linuxkit.yml
This port is also non-standard and there doesn't appear to be a standard
port for this as yet.
Signed-off-by: Dave Tucker <dt@docker.com>
- Prefix platform documentation with 'platform-'
- Add/Correct links from top-level README.md
- Tweak the Azure documentation
- Move the vsudd README to platform-hyperkit.md
- Add a dummy document for qemu/kvm
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Note that this is not the latest (which was 95efd45db073 at time of writing)
but the next commit 6428b4bad0c2 merges "Port ctr to use client package" breaks
the use of `ctr run --runtime-config` (by removing that option).
This contains https://github.com/containerd/containerd/pull/954 which was
causing some services to fail to start.
All previous uses of 15541037b9 are updated to
5749f2e9e6.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Update everything including the current
linuxkit/containerd:b1766e4c4c09f63ac4925a6e4612852a93f7e73b to the new
linuxkit/containerd:deaf5bf838bf7f131c2287ecff3ed9835b0497e2.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
This should make it easier to switch out `init` for other versions,
although the `getty` config still needs to be removed.
- use `/bin/rc.init` for start
- use `rc.shutdown` for stop
- make `rc.init` run other code.
- merge `containers` and `containerd` startup code
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
We don't actually build it here, we just use the alpine package, which is
already included in the linuxkit/alpine base.
Can be instantiated with a service stanza such as:
- name: qemu-ga
image: "linuxkit/qemu-ga:e5fbcf55926f6e1a96b3e49a392e547e8be1022c"
binds:
- /dev/vport0p1:/dev/vport0p1
The only functionality I've tested is to report IP addresses to the host (e.g.
via `virsh domifaddr`).
The image is configured to use `/dev/vport0p1` by default. If your VM image is
using a different port number then you can either incorporate the rename into
the bind ("/dev/vport1p1:/dev/vport0p1") or override the command with:
command: ["/usr/bin/qemu-ga", "-p", "/dev/vport1p1"]
The mdev tool appears to not create the symlinks which udev would provide
`/dev/virtio-ports/org.qemu.guest_agent.0` hence the need to hardcode the
potentially unstable `/dev/vportNp1`, `N` seems to depend on the number and
order of virtio devices in use. I don't know if it is possible to get mdev to
create these links. For reference the udev rule is:
SUBSYSTEM=="virtio-ports", KERNEL=="vport*", ATTR{name}=="?*", SYMLINK+="virtio-ports/$attr{name}"
See https://wiki.libvirt.org/page/Qemu_guest_agent for more info on the guest
agent.
A suitable stanza is added to `projects/swarmd/swarmd.yml` since that is where
I am using it.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Note this also removes 'net:host' and 'pid:host' as
this does not seem to be necessary.
Also update to latest Alpine base image.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
The remaining packages will be updated with subsequent commits
to also include the config label.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This specifies the capabilities and bind mounts the dhcpcd daemon
needs.
While at it also update the Alpine base image
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
- use new Makefile flags to add our build flags correctly now
- restore `dist` for now as it is useful for testing still, for now
Will remove both `dist` and `ctr` once we have our own tool, but will add
them to dev container instead.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This makes the package slightly bigger as tini from Alpine
is not statically linked but removes the dependency on
./tools/tini
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Also convert it to a multi-stage build and make
the HUB org/registry build time configurable.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Also convert to a multi-stage build and make the
make the HUB org/registry build time configurable.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
The dhcpcd package only needs a basic alpine-baselayout and
dhcpcd. This reduces the size of the package from 4.3MB to 1.8MB.
Also make the HUB org/registry build time configurable.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
We only need the alpine-baselayout packe to run init. This also
removes the sha256 import from alpine:edge.
This also reduces the size of the init package by more than
50% to just 1.5MB.
Also make the HUB org/registry build time configurable.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Also make the HUB org/registry build time configurable
for these packages.
Others will be reworked in sub sequent commits.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Works around https://github.com/moby/moby/issues/33176 and fixes#1807.
Updated al users of linuxkit/runc:2649198589ef0020d99f613adaeda45ce0093a38 to
this new build.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
If a file is a simple string (as opposed to a map) then write it with the
default perms. This makes for slightly terser metadata when creating a simple
datafile.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
`ok` was not assigned when type casting to `map[string]interface{}` so the
subsequent check was testing a stale value.
Also, the error message upon failure had the wrong sense.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
There is no need for start-stop-daemon since this package
is using containerd. Remove it and update the init
package to alpine:edge
Also use the git tree hash for the package hash while at it.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Directly install the required packages from the Alpine base
image to compile the rngd package.
Also use the git tree hash for the package hash while at it.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This builds the dhcpcd package from a fixed Alpine 3.5 base and adds
the dhcpcd daemon from the Alpine base image. This should make the
build reproducible and allows us to use the git tree hash.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Bring it into line with official images: remove `jq`, `sfdisk` which
are not needed; we used them for disk mounting but this is down standalone.
Add the tools for other filesystems, which are needed for `dm`, `btrfs`.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Update examples/swap.yml to use latest linuxkit/swap image
Add comment to examples/swap.yml how to enable encryption
Add encrypt options to README
Make encrypted swap the default
Signed-off-by: Avi Deitcher <avi@deitcher.net>
for swap.sh, take any command-line options, not just those with an arg
that in turn requires a new hash on the example
Use latest 4.9.x kernel to keep in line with other examples
Replace (bash) herestring with (posix) heredoc in swap.sh, and update hash to match
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Not sure when this arrived but it was stopping anything running.
Appears not to be in the older test `init` containers.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Fix spaces after commas
Remove MAINTAINER
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Simplify swap.yml example to remove files section
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Switch swap.sh to sh from bash and remove bash from image
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Replace fallocate with dd and update calculation function to support it
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Fix indentation
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Change link to swap to just swap.sh
Signed-off-by: Avi Deitcher <avi@deitcher.net>
Fix indent
Signed-off-by: Avi Deitcher <avi@deitcher.net>
We were using Debian but Alpine more consistent. Use nested build.
Currently extract the hash in a nasty way but this can be fixed later
when we switch over hashing method.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This was missed when things were renamed.
The intention with this code was (apparently) to provide a (pseudo)unique
hostname in the case where something more specific was not provided (e.g. by
DHCP). Make this a little clearer by using '(none)' rather than 'linuxkit' as
the default, in the normal case this will be overwritten by something more
specific and if it isn't we will change it to something somewhat unique derived
from the MAC address (as before). nb: '(none)' is already used by Debian so I
think it is a safe choice as the sentinel value.
The use of both CONFIG_DEFAULT_HOSTNAME and the explicit /etc/hostname from
mkimage.sh is likely to be redundant in some cases, but neither seems to
completely cover all cases so keep both.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Previously the "tini" contained the entire build. Using the
multi-stage build creates a new "tini" base image, which only
contains the "/bin/tini". This image can be used as the base
image for packages requiring "tini".
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Because we are not yet using the image store, this requires a
patched `ctr` that can run a specified rootfs rather than an
image.
We will switch to the image store later, but this requires
extensive build changes.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Implementing a String() interface for each provider makes it
easier for users to prefix log strings with the provider.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Previously only `/var` was `rshared` but some people need to share
mounts in `/opt` etc so let us make everything `rshared` for now.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This package handles meta and user data for different cloud
and other platforms. It should be easy to extend to new
platforms.
Currently, it handles GCP metadata and a simple CDROM userdata
provider.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This is needed for cloud environments that want to get their metadata in
the onboot phase over the network.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Added to Docker example as Swarm mode likes time sync.
Note uses pool.ntp.org at present, on some providers it
should use a different source.
Constraints not enabled as they seem to have an issue;
possibly trying to set before network is up.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Instead of mounting a new filesystem, revert to doing a `rw` bind.
However do not make `/` `rshared`, just `/var` as that is where we expect
filesystems to be mounted for persistence. Also only make the actual
container rootfs writeable, not the whole directory.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
As suggested by @shykes these are clearer
- onboot for things that are run at boot time to completion
- services for persistent services
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Also add ca-certificates to base, needed to use `dist` to pull.
Make two stage builds for `containerd` and `runc` so they have a
from `scratch` second stage.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
It is not necessary to bring up `eth0`, the program does it fine.
This means we can remove shell script, clean up build.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This works and runs containers now, if you eg `runc exec` into it.
Needs a few tweaks for rlimits, but will pull and run containers.
Will integrate better with ssh/dev containers to make more usable.
For a simple test use
```
./bin/moby build examples/docker.yml
./bin/moby run hyperkit -disk-size 100 docker
```
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This is based on the code we used for Docker Editions, and
will format an external drive, to be used for example for
`/var` for Docker image persistence.
It does not `mount` the drive yet, as splitting format and mount
gives better modularity.
Example yaml fragment:
```
- name: format
image: "mobylinux/format:097d4f22b20f976b1f89d8f0b8a5d074d35b856c"
binds:
- /dev:/dev
capabilities:
- CAP_SYS_ADMIN
- CAP_MKNOD
```
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This is compatible with containerd 8353da59c6ae7e1933aac2228df23541ef8b163f
which was picked up by d2caae4c1a.
This required jiggering with riddler output some more to update to new OCI
config.json format for capabilities.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Since Makefile contains the hashes we need a dependency otherwise changing the
hashes does not necessarily result in the binaries getting updated.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
- now supports image pull and run end to end
- update runc to last version before spec update fix#1302
- remove ext2 utils from init
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This does not get everything where we want it finally, see #1266
nor the optimal way of building, but it gets it out of top level.
Added instructions to build if you have a Go installation.
Not moving `vendor` yet.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- remove remainder of editions code
- add a new check container to run tests without Docker
- switch over `make test` to use new command to build tests
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
