Note that a237b51511 ("Bump alpine base across the board.") inadvertently
corrupted `tools/alpine/versions.aarch64` (by updating it to the amd64 hash),
this commit corrects this as part of the rebuild.
Signed-off-by: Ian Campbell <ijc@docker.com>
Signing is not setup for these repositories. It seems they haven't been updated
since before trust was added since their previous Makefile metadata (which was
translated in build.yml) did not contain `NOTRUST=1`.
Signed-off-by: Ian Campbell <ijc@docker.com>
By running:
./scripts/update-component-sha.sh --image linuxkit/alpine ad35b6ddbc70faa07e59a9d7dee7707c08122e8d
Signed-off-by: Ian Campbell <ijc@docker.com>
tools/alpine and tools/guestfs are omitted since they do not currently use
pkg/package.mk and do their own thing. Slightly hacky arrangements are made for
these. Note that previously they were only recursed into for their default
target (push) and that behaviour is retained.
Signed-off-by: Ian Campbell <ijc@docker.com>
Current versions of Docker complain:
Sending build context to Docker daemon 5.12kB
Step 1/4 : FROM linuxkit/guestfs@sha256:62e810869e83dabe04af7ac8fcd81d9f177d986a3424dcb9c4eb5439993c962f
...
Successfully tagged linuxkit/mkimage-gcp:736880776838c8f41c081276d9c017ae6adbf1bb-amd64
Tagging linuxkit/guestfs@sha256:62e810869e83dabe04af7ac8fcd81d9f177d986a3424dcb9c4eb5439993c962f as linuxkit/guestfs:aecc40bf84ce4f4238e06934c46d7cc570a39eed@sha256:62e810869e83dabe04af7ac8fcd81d9f177d986a3424dcb9c4eb5439993c962f
refusing to create a tag with a digest reference
../../pkg/package.mk:82: recipe for target 'tag-y' failed
Just use the digest for now. Perhaps someone will have time to fixup this build
properly at some point.
Make the same change to test/pkg/ltp.
Signed-off-by: Ian Campbell <ijc@docker.com>
Using explicit targets is cleaner and easier to follow.
Also add `forcepush` to pkg and test/pkg and make `push` explicit.
The `dirs` phony was non-existent in all three cases. Likewise `$(DIR)` wasn't
used in that way in pkg or test/pkg and is nolonger used in that way in tools.
Finally, `clean` was bogus in all cases (tools cleaned a thing which never
exists and pkg and test/pkg were a syntax error).
Signed-off-by: Ian Campbell <ijc@docker.com>
On x86_64, the blkid package gets installed as a dependency, but
not on arm64. Explicitly add it as the new format package depends
on it.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Currently projects/kubernetes pulls in the binaries, adding these dependencies
will allow us to build them as part of our packaging process.
Signed-off-by: Ian Campbell <ijc@docker.com>
This patch uses coreos grub2 instead of the built-in gummiboot
tool with Alpine distribution.
Coreos grub2 has the security feature such as TPM and kernel
verification, so we can setup a trust chain when loading
modules with grub2.
GNU grub2[1] also has the plan to add those security related
features, they have a 'verifiers' branch to do that, but
there're some build issue need to fix,so this patch use
coreos as an alternative.
This patch is used to address the #2359#2375.
Thanks Avi Deitcher <avi@deitcher.net> for the contribution
to build the GRUB2 from ubuntu 16.04 to alpine base image.
Change Log:
1.Address the comments raised by @rn
2.Change the '/dev/vda' as the device name point by kernel command
line 'root=' on arm64, '/dev/sr0' on amd64. As next plan, we can adapt
a more flexible method to get the dev name of the CDROM.
3.Switch the base image to build grub2 from ubuntu 16.04 to alpine.
4.'linux' as the grub2 menu entry on arm64, while 'linuxefi' on amd64.
[1] https://git.savannah.gnu.org/git/grub.git
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This makes it easier to add the containerd testing tools like
`containerd-stress` to test packages, for example, and also at
some point move `ctr` out of the base image as it can be installed
from the alpine image.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
The first is needed by the okernel project and last two to run
configure on the zfs kernel modules.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
As discussed before, as we use this in three places, cloning in
base makes more sense.
Update base image.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
These are needed by Kubernetes which is currently not using the standard abse image.
Also sort it.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
In line with BIOS ISO changes previously.
Remove boot options, user can add if required, they made boot very slow.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Also adjust the 'linuxkit/alpine' script to follow the
same pattern.
The new version of the script extract username/password from
the credential helper (or docker) and build and 'expect'
script to feed the info to 'notary'.
They can be invoked by:
DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE="phrase" ./push-manifest.sh ...
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Now that LinuxKit works on a truly immutable filesystem, for an
ISO it makes sense to use the ISO filesystem rather than using
an initrd. Only major difference is you may need to specify the
device for the root filesystem in the kernel command line, but
we set a sane default of `/dev/sr0` if unspecified.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- append a architecture specific suffix (-amd64, -arm64) to the
image
- add a script which looks up the hashes from the 'versions' file
and creates a multi-arch manifest
- the manifest is pushed to hub and signed with notary
The new linuxkit/alpine multi-arch image is available with the
x86_64 hash without the '-amd64' suffix.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This should reduce instability due to different people's environment. This
undoes the reordering done by "alpine: Update to containerd v1.0.0-alpha2",
leaving just the hash update (as expected).
Signed-off-by: Ian Campbell <ijc@docker.com>
The previous commit moved ovmf to the packages instead
of stashing it on the filesystem. This requires the
package to be installed here.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
We add the 'ovmf' to the mirror instead of installing
it in the rootfs. This way we record the version we are
using.
We already pull a few images (go, tini) from community
so may as well pull ovmf directly via 'packages.x86_64'.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This makes the need to pull $(BASE) redundant.
Rebuild resulted in linuxkit/guestfs:aecc40bf84ce4f4238e06934c46d7cc570a39eed@sha256:62e810869e83dabe04af7ac8fcd81d9f177d986a3424dcb9c4eb5439993c962f
Signed-off-by: Ian Campbell <ijc@docker.com>
We should always pull and push linuxkit/alpine with content trust
irrespective of architecture, unless explicitly disabled.
Currently, we have to dance around various other issues on arm64,
which are now documented in the Makefile.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Somewhere between the various updates yesterday the hash in
'versions.x86_64' went wrong and there is no image with hash
available on hub.
This commit updates the alpine base to the latest version and
thus rectifies the issue
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This was added in 17.06 and allows us to avoid using `$(BASE):build` which is
not safe against parallel builds etc.
Having done this restructure the build to not always delete the built container
and to separate out the `hash` and `version` file rules so that they can be
included in both the `tag` and `push` targets.
Signed-off-by: Ian Campbell <ijc@docker.com>
Alpine is the base docker image for the LinuxKit, but currently
it only supports amd64 architecture. This patch is try to unify
the alpine tool docker image build process order to suport other
architectures, such as AArch64, by using '--build-arg' to override
the alpine base image specified by 'FROM' in the Dockerfile.
Also this patch splits the standalone packages into 2 parts:
one is common for all archs, another is arch-specific.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This is actually containerd#1141 rebased onto v1.0.0-alpha1.
The `dist` command has been integreated into `ctr` and so is removed, including
from the getty and sshd bind mounts and the test which uses it is updated..
There is no change to the version of runc vendored by containerd, so this is
unchanged.
Signed-off-by: Ian Campbell <ijc@docker.com>
I noticed that running the runes in docs/vendoring.md made a bunch of files
which are omitted by modern vndr come back, presumably due to the old version
of vndr referenced there. So update the go-compile package to the current
version everywhere.
Also correct the use of `make` variable syntax (`$(PWD)`) where shell syntax
command substitution for `pwd` was desired.
Signed-off-by: Ian Campbell <ijc@docker.com>
tools/qemu and toos/go-compile define the IMAGE after the
package.mk, which result in below error if 'make ORG=other_org':
...
invalid argument "other_org/:2c6d9e1d9c52167f4f2b7a8fd235eda318175c99"for t: invalid reference format
See 'docker build --help'.
../../pkg/package.mk:47: recipe for target 'tag' failed
make: *** [tag] Error 125
This because '../../pkg/package.mk' need to use IMAGE variable first.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Otherwise the only record is if the pusher happens to remember to put it in the
commit message (tollerable) or the github PR (not great).
Also add the same show-tag target as `pkg/package.mk` supports, although with a
very different implementation.
The actual hash is unchanged from the previous commit.
Signed-off-by: Ian Campbell <ijc@docker.com>
If a user of linuxkit/alpine wants to produce an image with apk in it then it
is useful for them to have the original upstream repository list.
The new hash is linuxkit/alpine:3744607156e6b67e3e7d083b15be9e7722215e73
Signed-off-by: Ian Campbell <ijc@docker.com>
We expect this (or something very similar) to be merged soon, it fixes#2131 so
moving ahead now.
The new alpine mirror is linuxkit/alpine:6832775a7e861ee2d7842e157688ece52d007142
Signed-off-by: Ian Campbell <ijc@docker.com>
This will make it simpler to temporarily in PRs which are not yet merged (but
are expected to be soon).
Tools alpine is not rebuilt here since we are going to do just that in the next
commit.
Signed-off-by: Ian Campbell <ijc@docker.com>
We should use this from a container at top level instead; it makes it
much harder to port to other architectures if we have to have this as
it is written in Haskell making a multi arch build much harder.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- iproute2 provides a more complete suite of 'ip' utilities than
the busybox implementation. Useful, for debugging.
- iperf3 for network testing
- tcpdump for network debugging
- expect is used in some tests. Adding it to the base image should
make it easier to create a LinuxKit image to self-host CI.
The new version of the base package is:
linuxkit/alpine:3d57ded3edd54e616210cf0c17e3bc15eed9d03a
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Since we have to push all new packages due to 5a225b9d
("Makefile: Fix bug where network was not disabled") we
may as well update the Alpine base to the latest.
The new tag is: 9bcf61f605ef0ce36cc94d59b8eac307862de6e1
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Commit 4699f80e ("test: add test-containerd") added
the containerd commit hash to the alpine base image.
Unfortunately, setting it as ARG does not work and it
needs to be set with ENV for other builds to pick it up.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
We'll use abuild in the next commit for building auditd from source.
Also add these audit deps:
* swig
* python3
* libcap-ng-dev
Signed-off-by: Tycho Andersen <tycho@docker.com>
A subsequent commit will update all packages, so we might as well
update all packages to the latest base.
The new hash is: linuxkit/alpine:c608d404c1cb36cce0c7d9303e30b52c9d81ccf0
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Moved qemu-img to keep list fully alphabetical.
Rebuild pushed as linuxkit/alpine:c96f56ef989ccb31fd115d25c1b90f05b992271d
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
iAdd a bunch of packages which are useful for compiling
some of the tools shipped with the Linux kernel source code.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
- no longer uses several of the `mkimage-*` tools in favour of dogfooding
with `linuxkit` and using the `mkimage` package.
- fix the qemu docker container fallbacks to work better when multiple
paths are used for disks and the image.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Adding a mkimage package to create dynamic 25GB VHD images (static
VHD images are already supported by LinuxKit).
Signed-off-by: Dave Freitag <dcfreita@us.ibm.com>
This ovmf package is part of the community repository. We don't
want to "pollute" the Alpine base with it. Luckily it's juts a single
file which we can install and then copy to the base image.
The package is needed for qemu EFI boot.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
While not used anywhere, adding it to the based makes it easy
to add temporarily add it to init (or elsewhere) for debugging.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
The Dockerfile is now an input to the contents of the base image and
needs to be included in the hash calculation.
Also, make the Makefile, Dockerfile and pacakges file a dependency.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
We use the "official" Alpine based Docker image and extract
the binary and libraries from it and add to the base. Compiling
it from source would require a Haskell setup...
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
The added packages are necessary to create a base Alpine
container image and for converting more packages.
Update the versions file to the latest versions
Also make the HUB org/registry build time configurable.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This is a fairly generic bootable disk with syslinux. Should
work if you `dd` it onto a USB stick, and should also work for AWS.
You need to uncompress it of course! Default size is 1G.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
It is not longer used by any package. Some projects
still use it and I haven't updated their builds but they
should continue to work.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Directly install the required packages from the Alpine base
image to compile the toybox-media package.
Also use the git tree hash for the package hash while at it.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Use the alpine base image and install the tools to compile
tini manually. This removes the dependency on c-compile.
Also use the git tree hash for the package hash while at it.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This means that we limit to one place where we use network access
in building, and in future all other package builds can be deterministic.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Previously we hardcoded `bzImage` which is not used for all
use cases or architectures.
fix#1630
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Updated go-compile to be able to compile remotely. Note I
did not update the oter users of go-compile as it does not affect
them.
Update `go get` instructions to fetch new one, or `make && make install`
will still work.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This was missed when things were renamed.
The intention with this code was (apparently) to provide a (pseudo)unique
hostname in the case where something more specific was not provided (e.g. by
DHCP). Make this a little clearer by using '(none)' rather than 'linuxkit' as
the default, in the normal case this will be overwritten by something more
specific and if it isn't we will change it to something somewhat unique derived
from the MAC address (as before). nb: '(none)' is already used by Debian so I
think it is a safe choice as the sentinel value.
The use of both CONFIG_DEFAULT_HOSTNAME and the explicit /etc/hostname from
mkimage.sh is likely to be redundant in some cases, but neither seems to
completely cover all cases so keep both.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
Previously the "tini" contained the entire build. Using the
multi-stage build creates a new "tini" base image, which only
contains the "/bin/tini". This image can be used as the base
image for packages requiring "tini".
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Previously, the cmdline file was never sent in the build context which
caused the script to pick the default options.
Also, when mulitple options are specified (not the default) the right
escaping needs to be used.
Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>
This flags allows passing additional ldflags to the build.
It is primarily there to pass -C arguments to specify values
of variables at compile time.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Generated largely from the specified config; small parts taken from `docker image inspect`,
such as the command line.
Renamed some of the yaml keys to match the OCI spec rather than Docker Compose as
we decided they are more readable, no more underscores.
Add some extra functionality
- tmpfs specification
- fully general mount specification
- no new privileges can be specified now
For nostalgic reasons, using engine-api to talk to the docker cli as
we only need an old API version, and it is nice and easy to vendor...
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This is compatible with containerd 8353da59c6ae7e1933aac2228df23541ef8b163f
which was picked up by d2caae4c1a.
This required jiggering with riddler output some more to update to new OCI
config.json format for capabilities.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
mkfs.vfat prints output (even without the -v option) which ends up
in front of the ISO, which is cat'ed to stdout. This made the generated
ISO unbootable. Redirect stdout of mkfs.vfat to /dev/null.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Guestfs is not really a base package and is only used
byt other images in the ./tools directory. Move it there.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
- The tools directory ideally should not contain source code
- Removes double vendoring of packagages
- Makes it easer to hook the build into the top-level Makefile
Eventually, the plugin should be moved to the infrakit repo.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
- this removes the use of riddler to extract the rootfs, use code
we were using for rootfs. riddler now just geenrates the config,
next stage is to generate this ourselves
- change the naming of the daemons so no longer include number as we
do not guarantee ordering as they start up simultaneously
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Corrected naming from vmware->vmdk and fixed Makfile
Fixed mistake outputting a vhd instead of a vmdk in output.go
Build vmdk image and added to Docker Hub, corrected link in output.go
Modified directories to confirm to standard mkimage-<imgType>
Signed-off-by: Dan Finneran <dan@thebsdbox.co.uk>
Removing the left over indirect creates that use the Docker socket
and run in containers not directly.
See #1347
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This does not yet move the `make test` options in the Makefile, will
probably move those too later.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Trying to find the relevant yaml file was an issue as we now support
`--name` and it might be in a different directory, so although it is
a bit verbose outputing a whole file at least it is more consistent.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- this needs improvements to make it more "platform native", in
particular GCP supports multiple users and more ssh key mangement
options.
- at present you can login as root with any platform ssh key
- add support for uts=host and ipc=host
- set the hostname from the metadata as well
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This requires switching to the dosfstools from alpine:edge since neither the
busybox nor alpine:3.5 dosfstools supports the -C option (in fact alpine:3.5
only has mkfs.fat and not mkfs.vfat).
The 511k slack seems like a lot to me, but 256k was somehow not enough.
Fixes#1304.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
LTP is compiled in on debian container and the binaries
then copied into another container, which is pushed to hub.
LTP does not compile on Alpine as it uses glibc specific
pthread extensions.
I tried to link LTP statically to not require a glibc based
base image but that failed too.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Passing accel=kvm:tcg causes it to try KVM first if available with a fallback
to TCG (emulated/JIT mode) if it is not available. With this the boot logs gain:
+Hypervisor detected: KVM
and also
-Booting paravirtualized kernel on bare hardware
+Booting paravirtualized kernel on KVM
Among various other noise.
If I rename my host /dev/kvm then this is reversed, although with the following
message:
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Q35 is a more modern emulated platform based on the ICH9 host chipset rather
than the default "pc" I440FX (Pentium Pro / Pentium II era) emulation. See
http://wiki.qemu-project.org/Features/Q35 for more info. Switching to Q35 is
not a requirement for enabling KVM but seemed like a reasonable change.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
- VHD is uncompressed VHD. Currently hard coded at 1GB, which may need to change. Use `format: vhd`
- GCE is the GCE compressed tarred raw image. Use `format: gce-img` - reserving `gce` for actually
uploading the image.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
os.FindProcess() does not return an error when then process does not
exist. It even returns a dummy process object.
Use the go-ps package to find out if the hyperkit process is actually
running.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
the `moby` tool uses a prefix for the build output. Specify the prefix
(and optional path) in the `Moby` property in the JSON instead of assuming
a different path for each initrd/kernel.
Also, adopt the new name for the kernel image (vmlinuz64 -> <prefix>-bzImage).
This should make it simpler to boot up hyperkit with Moby images
build from by the moby tool.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
- remove remainder of editions code
- add a new check container to run tests without Docker
- switch over `make test` to use new command to build tests
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Note that the EFI ISO is not yet automatically sized, and the
kernel command lines are currently hard coded in the builders.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Don't return if the process does not exist. Instead proceed and delete
the state directory.
This allows us to use the Destroy() method elsewhere to clean up the state.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Instead pass the directory where to find initrd/vmlinuz via the
"Moby" entry in the instance JSON config.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This change now allows setting the Disk size, memory, and number of CPUs
for an instance via the JSON config file.
We now also write the hyperkit command line to the VM directory.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
This is the first cut of a InfraKit instance plugin which
primarily allows one to manage Moby instances (though, other Linux's
may work too).
It's very rough on the edges and needs work and update to handle
templates and better configuration etc.
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Use aws cli to pull docker engine binaries from s3 bucket. Conditional and
happens if DOCKER_BIN_URL begins with 's3://'
Example usage:
```
export AWS_ACCESS_KEY_ID=key
export AWS_SECRET_ACCESS_KEY=secret
make DOCKER_BIN_URL=s3://path-to-engine-binaries
```
Signed-off-by: Robb Kistler <robb.kistler@docker.com>
Signed-off-by: Robb Kistler <robb.kistler@docker.com>
This includes most things except the containerd and runc builds
which I will do next time I update them.
Note that all golang:alpine images now for 1.8 are ALpine 3.5 based.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- `CONFIG_BUG_ON_DATA_CORRUPTION` is a 4.10 kernel only flag, so gate this.
- some config was duplicated on gated and normal config
- some 4.4 issues with the new config
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
