Commit Graph

6135 Commits

Author SHA1 Message Date
Carlos Venegas
ec36883fe3
Merge pull request #1638 from jcvenegas/2021-04-06/linux-5.10.x-qemu-experimental
kernel: update experimental kernel to 5.10.x
2021-04-07 16:02:44 -05:00
Wainer Moschetta
69dbcaa32b
Merge pull request #1633 from fidencio/wip/fix-test-kata-deploy-script
kata-deploy: Fix `test-kata.sh` and do some small cleanups / improvements in the kata-deploy script
2021-04-07 16:13:17 -03:00
GabyCT
a374b007bd
Merge pull request #1646 from fidencio/wip/fix-slash-command-action-usage
github: Fix slash-command-action usage
2021-04-07 10:17:04 -05:00
GabyCT
d922070c50
Merge pull request #1644 from lifupan/fix_env
rustjail: fix the issue of missing default home env
2021-04-07 10:16:07 -05:00
GabyCT
81bcded9a3
Merge pull request #1492 from dgibson/uevent
Make uevent watching mechanism more flexible
2021-04-07 10:15:33 -05:00
Fabiano Fidêncio
d2fda148fa
Merge pull request #1637 from wainersm/openshift_ci_centos8
ci/openshift-ci: Prepare to build on CentOS 8
2021-04-07 15:37:41 +02:00
Fabiano Fidêncio
514ba369fd
Merge pull request #1630 from liubin/1629-update-containerd-config
docs: update configuration for passing annotations in conatinerd
2021-04-07 15:36:21 +02:00
Fabiano Fidêncio
7873b7a1f9 github: Fix slash-command-action usage
`/test-kata-deploy` command does **not** work, and the output returned
is:
```
Error: Comment didn't contain a valid slash command
```

So, why does this happen?

This is the regex used: `^\/([\w]+)\b *(.*)?$`, being the important part
of the command "\/([\w]+)\b", with the rest being arguments to it.
Okay, `\w` is the key here, as `\w` means: a-z, A-Z, 0-9, including the
_.

Our command is `/test-kata-deploy`, and `-` is not present as part of
`\w`.  Knowing this we need to update the command to something like:
`/test_kata_deploy`

Fixes: #1645

Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2021-04-07 14:46:21 +02:00
Fabiano Fidêncio
b09fda36bd
Merge pull request #1641 from fidencio/wip/re-add-kata-deploy-test-action
Revert "github: Remove kata-deploy-test action"
2021-04-07 13:18:25 +02:00
Fabiano Fidêncio
eda8da1ec5 github: Revert "github: Remove kata-deploy-test action"
This partially reverts commit 1cce930071.

As mentioned in #1635, the malformed yaml wouldn't allow us to actually
test changes that were supposed to be test by this action.

So, this is now reverted and adapted accordingly.

Main differences from what we had before:
* As it tests kata-deploy itself, not the statically built binaries,
  let's just use the binaries from 2.0.0 release;
* Adapt download and deploy location to the
  `kata-containers/kata-containers` repo, as the original action was
  based on 1.x repos;

Fixes: #1640

Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2021-04-07 12:07:30 +02:00
fupan.lfp
a938d90310 rustjail: fix the issue of missing default home env
first get the "HOME" env from "/etc/passwd", if
there's no corresponding uid entry in /etc/passwd,
then set "/" as the home env.

Fixes: #1643

Signed-off-by: fupan.lfp <fupan.lfp@antfin.com>
2021-04-07 15:11:28 +08:00
bin
b0e4618e84 docs: update configuration for passing annotations in conatinerd
Using "io.containerd.kata.v2" instead of deprecated "io.containerd.runc.v1".

Fixes: #1629

Signed-off-by: bin <bin@hyper.sh>
2021-04-07 10:16:02 +08:00
Fabiano Fidêncio
d43098ec21 kata-deploy: Adapt regex for testing kata-deploy
On commit 60f6315 we've started adding the specific version of the image
to be used, in order to ensure people using our content from a tarball
would be relying on the correct image.

However, later on, @bergwolf figured out it had some undesired side
effects, such as
https://github.com/kata-containers/kata-containers/runs/2235812941?check_suite_focus=true

What happens there is that the regular expression used to point the
image to a testing one doesn't take into consideration the $VERSION, and
that breaks the deployment.

Depends-on: github.com/kata-containers/kata-containers#1641
Fixes: #1632

Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2021-04-06 23:41:24 +02:00
GabyCT
0b87fd436f
Merge pull request #1544 from snir911/timeout
runtime: increase dial timeout
2021-04-06 16:10:51 -05:00
Carlos Venegas
84e453a643
Merge pull request #1625 from jcvenegas/2021-04-05/qemu-experimental-6.0
qemu experimental: Move to latest tree on virtio-fs-dev (qemu 6.0 + DAX patches).
2021-04-06 15:29:05 -05:00
Carlos Venegas
107ceca680 kernel: update experimental kernel to 5.10.x
Relevant changes for experimental :

42d3e2d04 virtiofs: calculate number of scatter-gather elements accurately
413daa1a3 fuse: connection remove fix
bf109c640 fuse: implement crossmounts
1866d779d fuse: Allow fuse_fill_super_common() for submounts
fcee216be fuse: split fuse_mount off of fuse_conn
8f622e949 fuse: drop fuse_conn parameter where possible
24754db27 fuse: store fuse_conn in fuse_req
c6ff213fe fuse: add submount support to <uapi/linux/fuse.h>
d78092e49 fuse: fix page dereference after free
9a752d18c virtiofs: add logic to free up a memory range
d0cfb9dcb virtiofs: maintain a list of busy elements
6ae330cad virtiofs: serialize truncate/punch_hole and dax fault path
9483e7d58 virtiofs: define dax address space operations
2a9a609a0 virtiofs: add DAX mmap support
c2d0ad00d virtiofs: implement dax read/write operations
ceec02d43 virtiofs: introduce setupmapping/removemapping commands
fd1a1dc6f virtiofs: implement FUSE_INIT map_alignment field
45f2348ec virtiofs: keep a list of free dax memory ranges
1dd539577 virtiofs: add a mount option to enable dax
22f3787e9 virtiofs: set up virtio_fs dax_device
f4fd4ae35 virtiofs: get rid of no_mount_options
b43b7e81e virtiofs: provide a helper function for virtqueue initialization

Fixes: #1639

Signed-off-by: Carlos Venegas <jos.c.venegas.munoz@intel.com>
2021-04-06 19:06:09 +00:00
Fabiano Fidêncio
ca4dccf980 release: Get rid of "master"
We don't use the "master" branch for anything in
`kata-containers/kata-containers`.

Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2021-04-06 20:28:42 +02:00
Fabiano Fidêncio
c2197cbf2b release: Use sudo to install hub
This doesn't make much difference for the automated process we have in
place, but makes a whole lot of difference for those trying to have the
binaries deployed locally.

Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2021-04-06 20:28:42 +02:00
Fabiano Fidêncio
dc373d0161
Merge pull request #1635 from fidencio/wip/remove-kata-deploy-test-action
github: Remove kata-deploy-test action
2021-04-06 20:27:54 +02:00
Wainer dos Santos Moschetta
16f732fc18 ci/lib: Use git to clone the tests repository
On clone_tests_repo() use git instead of `go get` to clone and/or
update the tests repository.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
2021-04-06 13:44:02 -04:00
Wainer dos Santos Moschetta
9281e56705 ci/openshift-ci: Add build root dockerfile
This adds the dockerfile which is used by the OpenShift CI operator to build
the build root image. It is installed git as it is required by the operator
to clone repositories. The sudo package is also installed because many scripts
relies on the command but it is not installed by tests/.ci/setup_env_centos.sh.

Fixes #1636

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
2021-04-06 13:44:02 -04:00
Fabiano Fidêncio
1cce930071 github: Remove kata-deploy-test action
Currently the action is not running because it's broken, and it was
broken by 50fea9f.

Sadly, I cannot just test a fix on a PR as every single time we end up
triggering what's currently on main, rather than triggering the content
of the PR itself.

With this in mind, let me just remove the file and re-add it as part of
a new PR and, hopefully, have it tested in this way.

Sorry for the breakage, by the way.

Fixes: #1634

Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>
2021-04-06 19:01:53 +02:00
GabyCT
aac852a0bc
Merge pull request #1561 from Jakob-Naucke/s390x-statfs-constants
agent: s390x statfs constants
2021-04-06 11:11:40 -05:00
GabyCT
201ad249c2
Merge pull request #1597 from jongwu/PA
kernel: upgrade kernel to 5.10.x for arm64.
2021-04-06 09:44:16 -05:00
David Gibson
0828f9ba70 agent/uevent: Introduce wait_for_uevent() helper
get_device_name() contains logic to wait for a specific uevent, then
extract the /dev node name from it.  In future we're going to want similar
logic to wait on uevents, but using different match criteria, or getting
different information out.

To simplify this, add a wait_for_uevent() helper in the uevent module,
which takes an explicit UeventMatcher object and returns the whole uevent
found.

To make testing easier, we also extract the cut down uevent watcher from
test_get_device_name() into a new spawn_test_watcher() helper.  Its used
for both test_get_device_name() and a new test_wait_for_uevent() amd will
be useful for more tests in future.

fixes #1484

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 21:14:52 +10:00
David Gibson
16ed55e440 agent/device: Use consistent matching for past and future uevents
get_device_name() looks at kernel uevents to work out the device name for
a given PCI (usually) address.  However, when we call it we can't know if
the uevent we're interested in has already happened (in which case it will
have been recorded in Sandbox::uevent_map) or yet to come, in which case
we need to register to watch it.

However, we currently match differently against past and future events.
For past events we simply look for a sysfs path including the address, but
for future events we use a complex bit of logic in the is_match() closure.
Change it to use the exact same matching logic in both cases.

fixes #1397

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 21:14:33 +10:00
David Gibson
4b16681d87 agent/uevent: Put matcher object rather than "device address" in watch list
Currently, Sandbox::uevent_watchers lists uevents to watch for by a
"device address" string.  This is not very clearly defined, and is
matched against events with a rather complex closure created in
Uevent::process_add().

That closure makes a bunch of fragile assumptions about what sort of
events we could ever be interested in.  In some ways it is too
restrictive (requires everything to be a block device), but in others
is not restrictive enough (allows things matching NVDIMM paths, even
if we're looking for a PCI block device).

To allow the clients more precise control over uevent matching, we
define a new UeventMatcher trait with a method to match uevents.  We
then have the atchers list include UeventMatcher trait objects which
are used directly by Uevent::process_add(), instead of constructing
our match directly from dev_addr.

For now we don't actually change the matching function, or even use
multiple different trait implementations, but we'll refine that in
future.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 21:14:18 +10:00
David Gibson
b8b322482c agent/uevent: Consolidate event matching logic
The event matching logic in Uevent::process_add() is split into two parts.
The first checks if we care about the event at all, the second checks
whether the event is relevant to a particular watcher.

However, we're going to be adding more types of watchers in future, which
will make the global filter too restrictive.  Fold the two bits of logic
together into a per-watcher filter function.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:59:43 +10:00
David Gibson
d2caff6c55 agent: Re-organize uevent processing
Uevent::process() is a bit oddly organized.  It treats the onlining of
hotplugged memory as the "default" case, although that's quite specific,
while treating the handling of hotplugged block devices more like a special
case, although that's pretty close to being very general.

Furthermore splitting Uevent::is_block_add_event() from
Uevent::handle_block_add_event() doesn't make a lot of sense, since their
logic is intimately related to each other.

Alter the code to be a bit more sensible: first split on the "action" type
since that's the most fundamental difference, then handle the memory
onlining special case, then the block device add (which will become a lot
more general in future changes).

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:59:20 +10:00
David Gibson
55ed2ddd07 agent: Store uevent watchers in Vec rather than HashMap
Sandbox:dev_watcher is a HashMap from a "device address" to a channel used
to notify get_device_name() that a suitable uevent has been found.
However, "device address" isn't well defined, having somewhat different
meanings for different device/event types.  We never actually look up this
HashMap by key, except to remove entries.

Not looking up by key suggests that a map is not the appropriate data
structure here.  Furthermore, HashMap imposes limitations on the types
which will prevent some future extensions we want.

So, replace the HashMap with a Vec<Option<>>.  We need the Option<> so that
we can remove entries by index (removing them from the Vec completely would
hange the indices of other entries, possibly breaking concurrent work.

This does mean that the vector will keep growing as we watch for different
events during startup.  However, we don't expect the number of device
events we watch for during a run to be very large, so that shouldn't be
a problem.  We can optimize this later if it becomes a problem.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:59:19 +10:00
David Gibson
91e0ef5c90 agent/uevent: Report whole Uevents to device watchers
Currently, when Uevent::handle_block_add_event() receives an event matching
a registered watcher, it reports the /dev node name from the event back
to the watcher.

This changes it to report the entire uevent, not just the /dev node name.
This will allow various future extensions.  It also makes the client side
of the uevent watching - get_device_name() - more consistent between its
two paths: finding a past uevent in Sandbox::uevent_map() or waiting for
a new uevent via a watcher.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:58:47 +10:00
David Gibson
3642005479 agent: Store whole Uevent in map, rather than just /dev name
Sandbox::pci_device_map contains a mapping from sysfs paths to /dev entries
which is used by get_device_name() to look up the right /dev node.  But,
the map only supplies the answer if the uevent for the device has already
been received, otherwise get_device_name() has to wait for it.

However the matching for already-received and yet-to-come uevents isn't
quite the same which makes the whole system fragile.

In order to make sure the matching for both cases is identical, we need the
already-received side to store the whole uevent to match against, not just
the sysfs path and device name.

So, rename pci_device_map to uevent_map and store the whole uevent there
verbatim.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:58:47 +10:00
David Gibson
0616202580 agent/device: Move GLOBAL_DEVICE_WATCHER into Sandbox
In Kata 1.x, both the sysToDevMap and the deviceWatchers are in the sandbox
structure.  For some reason in Kata 2.x, the device watchers have moved to
a separate global variable, GLOBAL_DEVICE_WATCHER.

This is a bad idea: apart from introducing an extra global variable
unnecessarily, it means that Sandbox::pci_device_map and
GLOBAL_DEVICE_WATCHER are protected by separate mutexes.  Since the
information in these two structures has to be kept in sync with each other,
it makes much more sense to keep them both under the same single Sandbox
mutex.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:58:45 +10:00
David Gibson
11ae32e3c0 agent/device: Fix path matching for PCI devices
For the case of virtio-blk PCI devices, when matching uevents we create
a pci_p temporary.  However, we build it incorrectly: the dev_addr values
we use for PCI devices are a relative sysfs paths from the PCI root to the
device in question *including an initial /*.  But when we construct pci_p
we add an extra /, meaning the resulting path will *not* match properly.

AFAICT the only reason we got away with this is because in practice the
virtio-blk devices where discovered by the kernel before we looked for them
meaning the loosed matching in get_device_name() was used, rather than the
pci_p logic in handle_block_add_event().

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:58:06 +10:00
David Gibson
4f60880414 agent/device: Update test_get_device_name()
The current test_get_device_name(), ported from Kata 1.x doesn't really
reflect how the function is used in practice.  The example path appears
to be for a virtio-blk device, but it's an s390 specific variant, not a
PCI device.  The s390 form isn't actually supported by any of the existing
users of get_device_name().

Change it to a plausible virtio-blk-pci style path to better test how
get_device_name() will actually be used in practice.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 20:49:48 +10:00
Bin Liu
117c59150d
Merge pull request #1613 from Tim-Zhang/pipestream-shutdown-do-nothing
Don't do anything in Pipestream::shutdown
2021-04-06 14:03:00 +08:00
Tim Zhang
ee6a590db1 agent: add test test_pipestream_shutdown
Make sure PipeStream::shutdown() do not close the inner fd.

Signed-off-by: Tim Zhang <tim@hyper.sh>
2021-04-06 11:44:56 +08:00
Tim Zhang
4a2d437043 agent: don't do anything in Pipestream::shutdown
The only right way to shutdown pipe is drop it
Otherwise PipeStream will conflict with its twins
Because they both have the same fd, and both registered.

Fixes: #1614

Signed-off-by: Tim Zhang <tim@hyper.sh>
2021-04-06 11:44:38 +08:00
Peng Tao
d5600641dd
Merge pull request #1603 from lifupan/fix_fsgroup
Fix fsgroup
2021-04-06 11:35:03 +08:00
David Gibson
e3e670c56f agent/device: Forward port test for get_device_name() from Kata 1.x
Kata 1.x had a testcase for the equivalent getDeviceName function in Go,
this adapts it to Rust and adds it to Kata 2.x.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 13:29:37 +10:00
David Gibson
b980965c6b
Merge pull request #1627 from dgibson/bug1626
agent: Remove many "panic message is not string literal" warnings
2021-04-06 13:29:11 +10:00
David Gibson
ed08980fc1 agent: Remove many "panic message is not string literal" warnings
Rust 1.51 appears to have added a new warning in anticipation of Rust 2021,
which requires the format string for panic!()s (including via the various
assert!() macros) to be a string literal.  This triggers quite a few times
in the agent code.  This patch fixes them.

fixes #1626

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2021-04-06 11:51:34 +10:00
Carlos Venegas
f365bdb7cf versions: qemu-experimental: 6.0~rc 470dd6
Move to next 6.0 dev tree for qemu experimental,
the qemu version is the same base as:

https://gitlab.com/virtio-fs/qemu/-/commits/virtio-fs-dev/

Using qemu 6.0-rc1 some patches does not apply.

Fixes: #1624

Signed-off-by: Carlos Venegas <jos.c.venegas.munoz@intel.com>
2021-04-05 21:01:58 +00:00
Carlos Venegas
6491b9d7aa qemu: Add support to build static qemu for dev tree
Update static build scripts to allow build qemu dev tree.
When qemu starts the process for a new version the patch number
from the qemu version is more than 50. Add logic to detect it
and not apply patches fro the base branch.

For example:

Qemu 5.2.50 means the beginning for 6.0 development. After detect a
development version, patches for 5.2.x will not be applied.

Signed-off-by: Carlos Venegas <jos.c.venegas.munoz@intel.com>
2021-04-05 20:55:55 +00:00
GabyCT
503039482b
Merge pull request #1620 from eadamsintel/update-qat-dockerfile
osbuilder: Update QAT Dockerfile with new QAT driver version
2021-04-05 09:51:14 -05:00
Snir Sheriber
13653e7b55 runtime: increase dial timeout
On some setups, starting multiple kata pods (qemu) simultaneously on the same node
might cause kata VMs booting time to increase and the pods to fail with:
Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed
out connecting to vsock 1358662990:1024: unknown

Increasing default dialing timeout to 30s should cover most cases.

Signed-off-by: Snir Sheriber <ssheribe@redhat.com>
Fixes: #1543
2021-04-04 09:37:38 +03:00
Eric Ernst
d44412fe24
Merge pull request #1623 from egernst/custom-image-registry
osbuilder: update dockerfiles to utilize IMAGE_REGISTRY
2021-04-02 14:18:36 -07:00
Chelsea Mafrica
17b1452c2a
Merge pull request #1607 from fidencio/wip/only-keep-one-VERSION-file
Only keep one VERSION file
2021-04-02 11:14:12 -07:00
Eric Ernst
935460e549 osbuilder: update dockerfiles to utilize IMAGE_REGISTRY
While we introduced IMAGE_REGISTRY, we didn't actually update the
corresponding Dockerfiles to utilize it. Let's add

Fixes: #1622

Signed-off-by: Eric Ernst <eric.g.ernst@gmail.com>
2021-04-02 09:46:16 -07:00
Eric Adams
010d57f4e2 osbuilder: Update QAT Dockerfile with new QAT driver version
This fixes the QAT driver version and provides a check early in the
building process to make sure the driver exists. It also provides
hints to users on how to fix themselves if the driver changes again.

Fixes: #1618

Signed-off-by: Eric Adams <eric.adams@intel.com>
2021-04-01 19:20:44 +00:00