Commit Graph

613 Commits

Author SHA1 Message Date
Rolf Neugebauer
3219daa674 Merge pull request #2264 from tych0/support-tap-qemu
support tap devices in qemu backend
2017-07-21 10:56:24 +01:00
Tycho Andersen
d237c92273 support tap devices in qemu backend
The motivation for this is networking out (in particular, testing NFS
support) from the VM.

We could be a lot more user friendly (a la libvirt) by creating the tap
device for users and allowing them to specify a bridge instead, but then
we'd need root to create this tap device. For now, let's make people do
their own tap devices, and just use them. A tap device can be created for a
bridge as follows:

    # ip tuntap add linuxkit0 mode tap user `whoami`
    # ip link set linuxkit0 up
    # ip link set linuxkit0 master $bridge_name

and then used by:

    $ ./bin/linuxkit run qemu -tap-device linuxkit0 linuxkit

Signed-off-by: Tycho Andersen <tycho@docker.com>
2017-07-20 12:40:07 -06:00
thebsdbox
41f2d2c256 OpenStack Push support
Signed-off-by: Dan Finneran <daniel.finneran@gmail.com>
2017-07-20 16:24:48 +01:00
Ian Campbell
e8d5005b0e Synchronise vendoring with containerd v1.0.0-alpha1
Some of these are arbitrary and just syncing for the sake of it, however the
image- and runtime-spec are relevant. Interesting changes:

- runtime spec:
  - LinuxRLimit is now POSIXRLimit.
  - Specs.Config is now a pointer.
  - LinuxResources.DisableOOMKiller moved to
    LinuxResources.LinuxMemory.DisableOOMKiller
- image spec:
  - Platform.Features is removed (unused here).

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-07-20 12:03:22 +01:00
Justin Cormack
4720490c92 Add a tmp directory for each container to be used as a mount point
This is to clean up how we do writeable containers in LinuxKit.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-19 11:29:03 +01:00
Justin Cormack
6eb74b2f2b Add support for onshutdown
This is a list of images to run on a clean shutdown. Note that you must not rely on these
being run at all, as machines may be be powered off or shut down without having time to run
these scripts. If you add anything here you should test both in the case where they are
run and when they are not. Most systems are likely to be "crash only" and not have any setup here,
but you can attempt to deregister cleanly from a network service here, rather than relying
on timeouts, for example.

Fix https://github.com/linuxkit/linuxkit/issues/1988

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-18 13:55:52 +01:00
Justin Cormack
55f7df577a Add json config for top level config
Otherwise JSON output as metadata cannot actually be used as it
has capitalised field names.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-18 13:16:58 +01:00
Justin Cormack
12439d947d Do not modify data structure while building
This will give odd effects if we output the yaml structure into the image.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-17 15:42:26 +01:00
Justin Cormack
040c500648 Add more omitempty to json spec
The metadata config was rather verbose.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-17 15:31:31 +01:00
Justin Cormack
389dd8c0fa Add a "metadata" file contents flag
Currently this supports "yaml" as the only option, which will output
the yaml config (as JSON) into the file specified in the image.

Fix #107

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-17 15:31:24 +01:00
Riyaz Faizullabhoy
c3b2c7d575 trust: do not call x509.SystemCertPool as nil will use this as default
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
2017-07-14 14:39:39 -07:00
Justin Cormack
cc33e67492 Allow uid, gid fields to be numeric or names
Previously I was forcing them to be strings, which is horrible. Now you
can either specify a numeric uid or the name of a service to use the
allocated id for that service.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-14 16:57:12 +01:00
Justin Cormack
830be373da Allow specifying the kernel and tarball names, or omitting tarball
fix #113

Use `tar: none` or `tar: ""` to omit the tarball.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-14 14:09:42 +01:00
Justin Cormack
7bc763a628 Move Contributing to top level
- accidentally committed to wrong directory

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-14 11:50:51 +01:00
Dennis Chen
3b1dfb69c7 ARM64: specify external firware binary for containerized qemu
Current implementation uses a fixed firmware(bios) binary
installed by the build process of the qemu container image,
which will prevent us from providing an external firmware binary
outside the container. This patch removes this limitation, thus we
can assign a firware binary image file with "-fw" option.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2017-07-14 09:10:42 +00:00
Justin Cormack
944793312a Merge pull request #2197 from ijc/qemu-envvar-overrides
qemu: allow kvm and containerized options to be overridden by the env…
2017-07-13 16:25:51 +01:00
Ian Campbell
cc8bd94960 qemu: allow kvm and containerized options to be overridden by the environment
This is useful in the case where Moby is shelling out to LinuxKit for certain
image types (currently raw and qcow2). Currently to experiment with different
options (e.g. when comparing performance to CI) you have to edit either the
moby or linuxkit tool to change the options used.

The environment variables take precedence over any explict command line options
given.

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-07-13 15:23:14 +01:00
Ian Campbell
f20828bf7e Update LinuxKit components used for build
This updates to the current set of images as of
https://github.com/linuxkit/linuxkit/pull/2159

Signed-off-by: Ian Campbell <ijc@docker.com>
2017-07-13 14:43:34 +01:00
Dave Tucker
94f2d2cd9a linuxkit: Fix qemu run behaviour when file does not exist
This commit fixes an issue reported on Slack where `linuxkit run` will
assume that a file that is neither a kernel or iso must be a disk image
without first checking that it exists. This would result in `qemu-img`
attempting to create a disk with 0 size due to the default behaviour of
creating disk images that do not exist.

Signed-off-by: Dave Tucker <dt@docker.com>
2017-07-13 11:46:22 +01:00
Justin Cormack
b4804b69e8 Add Contributing doc
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-12 12:04:33 +01:00
Justin Cormack
107c014a9d Re-run vendoring
Latest `vndr` has changed its algorithm a bit. It also pointed out that we were missing
some things.

Move the `vendor` directory up to the `linuxkit` command, else it gets confused by packages
that have Go code in.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-11 13:12:00 +01:00
Justin Cormack
51b4e20154 Merge pull request #103 from ijc/master
Build for Darwin and Windows
2017-07-11 11:07:36 +01:00
Stefan Bourlon
846e814a7f Backend vCenter: Remove network param for push cmd
The network parameter is not used in the command
"linuxkit push vcenter ..."

Signed-off-by: Stefan Bourlon <stefan.bourlon@ca.com>
2017-07-10 14:03:55 -07:00
Justin Cormack
3ee4fdc43a Error if there are duplicate service names
Continue to allow onboot to have duplicates as we do not run simultaneously
so that is ok (and we number them anyway), but services are run together
so we will get a runtime error if duplicated as this is the containerd/runc
id.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-10 14:50:04 +01:00
Ian Campbell
3c5fad4539 Remove dependency on docker/docker/cli
We were pulling in this whole stack of packages just for `trust.ReleasesRole`.
Just define it locally.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-07-10 14:38:21 +01:00
Justin Cormack
ec7e73b304 Merge pull request #100 from justincormack/userns
Add partial user namespace support
2017-07-10 14:30:30 +01:00
Stefan Bourlon
ffef020a09 Backend vCenter: add the datacenter parameter
Signed-off-by: Stefan Bourlon <stefan.bourlon@ca.com>
2017-07-07 14:32:14 -07:00
Stefan Bourlon
ea5114dc93 Backend vCenter: Upload ISO only with linuxkit push
Signed-off-by: Stefan Bourlon <stefan.bourlon@ca.com>
2017-07-07 14:26:58 -07:00
Dennis Chen
3a9c88c0c5 ARM64: Remove the hardcode of virtual machine type
Current implementation hardcodes the virtual machine as
"q35" for x86, this patch remove this hardcode and config
the machine type according to the arch the VM is running.
Also, in order to make sure the qemu can run on arm64 platform,
we need to specify the vcpu type in the command line.

Signed-off-by: Dennis Chen <dennis.chen@arm.com>
2017-07-07 21:54:15 +08:00
Ian Campbell
c3edfec759 revendor runc + runtime-spec to match containerd 8e1a04ff9ee3
Note that various fields have changed moved around in the JSON as a result:

* `Platform` has been removed.
* `Process` is now a pointer.
* `OOMScoreAdj` has moved into `Process`, from `Linux.Resources` (resolving a
  TODO here).

Also updates golang.org/x/sys which is less critical.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-07-06 15:34:01 +01:00
Justin Cormack
a73c3d3667 Add partial user namespace support
This adds the OCI parts needed into the yaml, but there are still
permissions issues in practise so marked as experimental.

It may just need further documentation to resolve the issues.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-07-03 14:45:58 +01:00
Rolf Neugebauer
4a9c00a5e0 src: Update qemu image to latest
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-07-01 19:24:15 +01:00
Justin Cormack
0acaaa71fe Assign each container a uid and gid it can use
In order to support not running containers as root, allocate
each of them a uid and gid, a bit like traditional Unix system
service IDs. These can be referred to elsewhere by the name of
the container, eg if you wish to create a file owned by a
particular esrvice.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-30 17:40:08 +01:00
Justin Cormack
66b4aed239 Add support for ambient capabilities
Allow setting ambient capabilities, as a seperate option to the standard
ones. If you are running as a non root user you should use these.

Note that unless you add `CAP_DAC_OVERRIDE` and similar permissions you
need to be careful about file ownership. Added support to set ownership
in the `files` section to help out with this.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-30 14:08:16 +01:00
Justin Cormack
70c5c2e1a0 Merge pull request #94 from davefreitag/ibm-vhd
Adding 'dynamic-vhd' Output Format
2017-06-27 15:29:20 +01:00
Tycho Andersen
c2e460fee5 check that caps are valid
Rather than build the image and have something weird happen, let's check
that the capabilities specified are actually valid capabilities.

Signed-off-by: Tycho Andersen <tycho@docker.com>
2017-06-27 08:25:10 -06:00
Dave Freitag
b605e61d1f Adding dynamic-vhd Output Format
Adding support for the dynamic-vhd output format.

Signed-off-by: Dave Freitag <dcfreita@us.ibm.com>
2017-06-27 08:53:56 -05:00
Ian Campbell
7151f1af52 qemu: Add -enable-kvm to force (attempted) use (or not) of KVM
The default is set by probing for /dev/kvm. Use of KVM can be forcibly disabled
with `-enable-kvm=false`. Note that passing `-enable-kvm=true` when `/dev/kvm` is
not present still boots but complains:

    Could not access KVM kernel module: No such file or directory
    failed to initialize KVM: No such file or directory
    Back to tcg accelerator.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-06-27 12:20:25 +01:00
Justin Cormack
e7ebabdb05 Split out into a small stub command line and a library
- this is pretty much the smallest change to split this out and it
  exposes a few things that can be improved later
- no change to logging yet

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-21 16:19:31 -07:00
Justin Cormack
c20f10f7f8 Add vmnet support to linuxkit run hyperkit
Allows routed networking, so long as you runhyperkit as root.

This has quite a few downsides, including the requirement to
run as root in order to set up the networking, but some people
really want VMs that are routable from the host.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-16 10:54:32 -07:00
Rolf Neugebauer
2f212e6c74 cmd: Unify the push coding style
- Use 'flags' for the subcommand FlagSet
- Use %v to print errors
- Use 'path' for the path
- Fix cases where the 'path' refers to a different directory
- Don't use CamelCase for command line options

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
2017-06-15 17:31:28 -07:00
Justin Cormack
d194cf7cf7 Update hyperkit and support multiple disks
Now hyperkit Go API has multiple disk support, allow using them from linuxkit

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-15 13:31:46 -07:00
Ian Campbell
e2eb17ef6c Only create qemu runner state dir if it is going to be used.
Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-06-13 17:22:08 +01:00
Ian Campbell
13a94c7a3c Add metadata support to Qemu runner.
Based on the hyperkit runner's code.

project/kubernetes/boot.sh now works, although lack of network connectivity
between individual VMs remains an issue.

Also manually validated containerized operation with:

    rm -rf kube-node-0-state && ../../bin/linuxkit run qemu --containerized  -cpus 2 -mem 4096 -state kube-node-0-state -disk size=4G -data "foo bar" kube-node

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-06-13 11:08:53 +01:00
Ian Campbell
a09090e6cd Add state dir support to Qemu runner.
Modelled on the Hyperkit runner, for now only used for the disk.

This is one step closer to having project/kubernetes/boot.sh work on Linux.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
2017-06-13 11:08:53 +01:00
Justin Cormack
6f62d083df Remove initrd library, it has moved to moby tool
As we are not using it here, makes more sense for it to go there.

Moved with history in https://github.com/moby/tool/pull/78

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-12 15:41:27 +02:00
Justin Cormack
322d06f906 Update vendoring to use initrd library from this repo
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-12 15:36:30 +02:00
Justin Cormack
528b430789 Clean up initrd creation library
Remove duplicated code and clean up file handling a little.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-12 15:34:30 +02:00
Justin Cormack
ddc92b8846 Code in the initrd generator to split a single tarball
In the WIP code in `moby` we now have a standard base tarball format,
that includes the kernel and cmdline as files in `/boot` so that the
entire output of the yaml file can default to a single tarball. Then
this can be split back up by LinuxKit into initrd, kernel and cmdline
as needed. This will probably become the only output of the `moby build`
stage, with a `moby package` stage dealing with output formats.

We may remove the output format specification from the yaml file as well,
and just have it in the command.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-12 15:34:22 +02:00
Justin Cormack
ecf14815f7 Update Go path to linuxkit/linuxkit for libraries
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
2017-06-12 15:34:11 +02:00