Commit Graph

258 Commits

Author SHA1 Message Date
Ian Campbell
e42e8ceed7 linuxkit pkg: improve handling of git working directory
Ensure that all git commands are run as if from the package directory using the
`-C` option. Otherwise the various attempts to use git fail if `linuxkit pkg`
is invoked from outside the git repo.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-20 16:57:33 +01:00
Daniel Nephin
60ef22ef60 Fix GCP error printing
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
2017-10-18 13:46:04 -04:00
Ian Campbell
fc31e3bddd linuxkit pkg: defer content trust passphrase setup until we know it is needed
Otherwise "linuxkit pkg build" etc will needlessly run the command (which might
prompt the user).

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-13 15:56:27 +01:00
Ian Campbell
dd8e3d49e8 linuxkit: allow user to configure a content trust passphrase command
Could be e.g.:
```
pkg:
  content-trust-passphrase-command: "lpass show <key> --password"
```
or
```
pkg:
  content-trust-passphrase-command: "gpg -d ~/.notary/passphrase.gpg"
```

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-13 11:41:22 +01:00
Ian Campbell
4304d5f4fd linuxkit pkg: correct behaviour when user explictly specifies a hash.
With the old package.mk arrangements we would only append -dirty and -$arch to
the hash if the user didn't specify a HASH= directly. So the following would
work
    make ORG=ijc HASH=dev tag
and produce an image `ijc/«image»:dev`.

However the new equivalent:
    linuxkit pkg build -org=ijc --hash-=dev
instead produces an image `ijc/«image»:dev-dirty-$arch`. Which is undesirable.

This commit refactors the code in two main ways:

Firstly directly append `-dirty` to the hash as we extract it from git (rather
than on use in the `Tag()` method), and only do this if the user has not
specified an explicit tag. Note that we still track dirtiness in the `Pkg`
object and so will not allow a push (or release) from a dirty tree (the
makefile version would have tried this with unpredictable results), nor will we
apply the `org.opencontainers.image.revision` label to a dirty build.

Secondly if we are not pushing the image+manifest then we retag the -$arch
suffixed image without the the -$arch. This differs from the Makefile version
which would simply have built without the -$arch in the first place, I think
this is an improvement. If we are pushing the manifest-tool remains responsible
for creating the non -$arch image.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-12 10:17:56 +01:00
Ian Campbell
2eb5dfffaa Merge pull request #2589 from ijc/linuxkit-pkg-release
linuxkit pkg: Support --release option to pkg push.
2017-10-11 11:10:23 +01:00
Ian Campbell
861b2cbe80 linuxkit pkg: Support --release option to pkg push.
This overrides the default (which is to look for an exact git tag) and releases
just that.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-10 16:54:04 +01:00
Ian Campbell
2264bc8021 Correct typo in linuxkit pkg push help.
Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-10 16:11:48 +01:00
Ian Campbell
8f1d39cea1 Log before building or pushing.
Previously there would be a make "entering directory" hint in the logs, but
with the switch to `linuxkit pkg` that no longer occurs.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-10 16:05:15 +01:00
Ian Campbell
4df5d394e8 Bump yml
Done as follows:

    find -name build.yml | xargs dirname | while read d ; do
        t=$(linuxkit pkg show-tag $d)
        ./scripts/update-component-sha.sh --image ${t%:*} ${t#*:}
    done
    git commit -s test pkg tools blueprints examples projects/kubernetes projects/swarmd docs linuxkit.yml Makefile src

This explicitly excludes projects/* which I did not know whether to update.

Then:

    git reset --hard
    for i in init runc containerd ca-certificates sysctl dhcpcd getty rngd ; do
        o=$(git grep -h "\(image:\|-\) *linuxkit/$i:[0-9a-f]\{40\}" origin/master:linuxkit.yml | awk '// { print $2 }')
        n=$(linuxkit pkg show-tag pkg/$i)
        ./scripts/update-component-sha.sh "$o" "$n"
    done
    git commit --amend projects

This updates any projects which were using components with the same hash as the
top-level linuxkit.yml.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-10 13:23:45 +01:00
Ian Campbell
fbf23b4b9f New option linuxkit pkg build --force-dirty
Will tag as `-dirty` and prevent push etc even if the tree isn't actually
dirty.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-10 10:43:53 +01:00
Ian Campbell
12e6a85661 Make linuxkit pkg build on an unsupported arch a nop
Rather than a hard fail. This allows batch builds of a set of packages without
the surrounding loop needing to be away of the possibility.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-10 10:42:58 +01:00
Ian Campbell
57225335bc Correct manifest push without content trust
The script expects an empty second argument to mean no trust and anything else
to mean trust.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-10 10:22:24 +01:00
Ian Campbell
d11e49d113 Fixup linuxkit pkg debug output (space wrong side of variable)
Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-09 16:19:20 +01:00
Ian Campbell
896cfaab63 Rename some variables following rename of PkgSrc to Pkg
Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-09 11:24:03 +01:00
Ian Campbell
4cdc90e221 Build fields default to false, rename accordingly
Added some test cases for the boolean fields because I kept getting confused.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-06 17:43:38 +01:00
Ian Campbell
a9b0c6f889 Pass argumements to NewFromCLI as varargs
Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-06 17:42:05 +01:00
Ian Campbell
7748b59d5d Add missing file
Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-06 15:56:33 +01:00
Ian Campbell
52d1bc6395 More specific error message when docker not installed
Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-06 14:14:50 +01:00
Ian Campbell
3141ca0bef Embed scripts/push-manifest.sh into linuxkit tool
This is a bit gross, but hopefully this script will be written in go sooner
rather than later and in the meantime this avoids the need to install yet
another tool on user's systems (and get it into brew etc).

Checkin the result of `go:generate` for now. Once there are no more users of
push-manifest.sh it can be moved alongside the go code (if it hasn't been
rewritten in go along the way).

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-06 14:02:51 +01:00
Ian Campbell
c84c997383 Rename pkgsrc to pkglib
Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-06 13:38:37 +01:00
Ian Campbell
ba60937754 Implement package build, push and show-tag in linuxkit tool
This implements the proposal in #2564 and converts a handful of representative
or especially interesting (from a build PoV) packages to use it.

For now those pkg/* affected get a stub-`Makefile`, once all packages are
converted then `pkg/Makefile` can be adjusted and those stubs can be removed.

For now only `pkg/package.mk`'s functionality is implemented. In particular:

- `push-manifest.sh` remains a separate script, to enable calling it on systems
  with just the LinuxKit tools installed arrange to install it under a less
  generic name.
- `kernel` and `tools/alpine` do not use `pkg/package.mk` and those cases are
  not yet fully considered/covered.

I have updated the documentation assuming that the existing uses of
`pkg/package.mk` will be removed quite soon in a follow up PR rather than
trying to document the situation which results after just this commit.

Due to `cmd/linuxkit` now gaining a library the build needs adjusting slightly to
allow both `make bin/linuxkit` and `go build` to work.

`go vet` has forced me to write some rather asinine comments for things that
are rather obvious from the name.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-10-06 12:59:25 +01:00
Avi Deitcher
ff889ef763 Better error message when running qemu on macOS when --fw not supplied
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2017-09-24 11:18:19 +03:00
Nick Jones
0556812b63 [OpenStack] Support specifying an SSH key name when creating an instance
This commit introduces a new option - `keyname` - to the OpenStack
runner, which allows the user to specify the name of a keypair they want
to associate with the instance at the time of creation.

Signed-off-by: Nick Jones <nick@dischord.org>
2017-09-18 21:36:04 +01:00
Ian Campbell
16147a7591 qemu: restore persistent mac address support
If the state dir is preserved then the VM should come up with the same mac
address on subsequent reboots.

Fixes regression in #2498.

Rather than fixing the typo in the associated comments just remove them since
they don't really convey any useful information.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-09-12 12:12:03 +01:00
Dennis Chen
147964b241 qemu: Fix the networking issue in 'tap' mode on arm64
This PR is used to fix the issue #2488.

Currently we use '-net' the old way to initialize a host nic
interface, this method will not work on arm64 platform(#2488 issue),
so we use the '-netdev' method which will work on both arm64 and amd64.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2017-09-11 05:33:06 +00:00
Rolf Neugebauer
57394620b8 cmd: On KVM/aarch64 add gic_version=host
This picks the correct GIC version to be emulated.
Without it you get an error on Thunder-X (and possibly others):

KVM_SET_DEVICE_ATTR failed: No such device

See also https://bugzilla.redhat.com/show_bug.cgi?id=1375571#c6

Fixes #2360

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-09-08 16:34:33 +01:00
Magnus Skjegstad
c8ba942a80 Update to new Hyperkit API / VPNKit protocol
This adds support for the updated Hyperkit API, which is needed to
request a specific IP address in new versions of VPNKit / Docker for
Mac. IPs encoded in the UUID (the old method) will now be ignored by
VPNKit.

A preferred IPv4 address can be requested directly via the new API. The
IP is then associated with the VPNKit UUID identifying the connection.
The UUID is either user specified or randomly assigned if left empty.
VMs launched with the same VPNKit UUID it will get the same IP address.

To avoid having to copy the assigned UUID manually, a file `uuid.vpnkit`
is now saved in the state directory when the UUID is generated.  The UUID
from this file is reused automatically if it exists, unless a different
VPNKit UUID is specified on the command line. This also means that VMs
that use dynamically assigned IPs will by default get the same IP each
time they are started, as long as the state directory exists.

This change is incompatible with earlier versions of VPNKit / Hyperkit
and a recent version of Docker for Mac has to be installed. If the
feature is unsupported using the `--ip` parameter will exit with an
error message.

Signed-off-by: Magnus Skjegstad <magnus@skjegstad.com>
2017-09-07 12:15:49 +02:00
Magnus Skjegstad
241136e910 Update Hyperkit to latest version
Signed-off-by: Magnus Skjegstad <magnus@skjegstad.com>
2017-09-07 10:49:45 +02:00
Ian Campbell
4bdca70c5d qemu: Support ISO boot at the same time as metadata
Tested with:
 - boot from kernel + initrd with metadata (appears as sr0)
 - boot from iso with no metadata (root is sr0)
 - boot from iso with metadata (root is sr0, metadata is sr1)
 - boot from iso with metadata using containerized qemu

In all cases where it was present the metadata was correctly expanded to
/var/config.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-08-29 21:16:10 +01:00
Justin Cormack
4a374eb175 Add missing case when running non existent file in hyperkit
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-23 16:08:30 +01:00
Rolf Neugebauer
c6ab35f721 cmd: Use the main packet.net API repo again
The local changes were merged, so switching back to the original
repository. A minor change in API was needed.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-18 10:21:36 +01:00
Rolf Neugebauer
e28d960635 Merge pull request #2428 from justincormack/packet-fork
Vendor fork correctly
2017-08-17 15:36:43 +01:00
Justin Cormack
ad4131ad4b Merge pull request #2426 from arm64b/firmware-file-format
qemu: Specify the firmware file format explicitly
2017-08-17 14:56:35 +01:00
Justin Cormack
63f521fd42 Vendor fork correctly
`vndr` has support for forks!

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-17 14:16:48 +01:00
Dennis Chen
023dba65c9 qemu: Specify the firmware file format explicitly
Currently we depend on the qemu to detect the firmware file format
automatically, which is dangerous. This patch specify the 'raw'
format explicitly to remove the kind of restrictions.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2017-08-17 09:27:28 +00:00
Rolf Neugebauer
56ae6d5055 cmd: Allow re-using an existing packet.net machine
This add a -device option which enables the re-use of an existing
device. The device state is update to use a potentially new base-url
ro PXE boot from. This maybe faster than provisioning a new server.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-16 16:27:50 +01:00
Rolf Neugebauer
d246ea9130 cmd: Temporarily use my fork of the packet.net API
This adds a new call to update an existing device.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-16 16:27:50 +01:00
Rolf Neugebauer
4d5bf9d90c Merge pull request #2416 from rn/p4
iPXE boot of arm64 on packet.net
2017-08-15 23:24:11 +01:00
Rolf Neugebauer
a6e25b0825 cmd: Print the packet machine ID when booting
This is useful if the connection to the SOS fails and you
want to debug manually after.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-15 21:36:10 +01:00
Rolf Neugebauer
f8c45539cc cmd: Add iPXE support for arm64 on packet.net
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-15 17:54:38 +01:00
Justin Cormack
b7fbabe2ba Simpler known_hosts parsing
Note it cannot deal with hashed host keys still.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-08-15 17:22:43 +01:00
Rolf Neugebauer
7ae9fab6b6 cmd: Unify Logging for packet backend
Be less verbose when running.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-15 14:31:44 +01:00
Rolf Neugebauer
2590c66511 cmd: Remove the packet machine when done
Adds a new option, '-keep' which will keep the machine around,
but by default remove the machine when done.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-15 14:31:44 +01:00
Rolf Neugebauer
fced67dab8 cmd: Add a -serve command to the packet.net backend
The -serve command starts a local HTTP server which serves
the kernel and initrd files.  The server can then easily be
made available via ngrok or other means.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-15 14:31:44 +01:00
Rolf Neugebauer
33e8afb292 cmd: Use the proper packet.net go bindings again
The support for always PXE has been merged.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-08-15 14:31:44 +01:00
Justin Cormack
a3de645048 Merge pull request #2397 from dave-tucker/aws-panic
aws: Fix panic when there is no console output
2017-08-10 21:59:54 +01:00
Dave Tucker
90bf0cc2f6 aws: Fix panic when there is no console output
When an AWS machine is stopped and there is no console output the CLI
panics due to a nil pointer dereference. This commit checks that the
console output is not null before attempting to decode it.

Signed-off-by: Dave Tucker <dt@docker.com>
2017-08-10 17:28:09 +01:00
Dave Tucker
f574ada213 aws: Honour the zone variable when creating an instance
Instances were being created in a different AZ to the one specified in
the zone variable. This could lead to situations where the disks and the
instance were in different AZs and would result in an error.

This commit adds placement information to the API call used to create
the instance.

Fixes: #2388

Signed-off-by: Dave Tucker <dt@docker.com>
2017-08-10 16:49:56 +01:00
Rolf Neugebauer
1a9d736f75 Merge pull request #2392 from rn/p2
Better kernel cmdline handling on packet.net
2017-08-09 14:20:55 +01:00