Commit Graph

237 Commits

Author SHA1 Message Date
Ettore Di Giacinto
7bfff97f57 Fixup race when updating compiler options
Instead of updating the compiler options that can be accessed by each
worker, update the compilespec BuildOptions directly
2021-04-23 12:02:41 +02:00
Ettore Di Giacinto
0288eedbc3 Always resolve buildhash image, add --rebuild to build 2021-04-23 12:02:41 +02:00
Ettore Di Giacinto
b27237b7ff Allow to pull images from multiple-repo while generating all artifacts
before, without --only-target-package, we were forcing to build the
images locally. Now we extend it also to that use-case.

Also revisit how we pass by the builder image hash, so it's easier to
read.

This change also re-enables tagging and bulding builder images all the
times.
Previously, we regressed into not tagging build images coming out-of-the
tree, with the unpleasant side-effect of not be able to re-build the
same artifacts for those leaf packages.
2021-04-23 12:02:41 +02:00
Ettore Di Giacinto
c9aed37fa7 Fixup on values interpolation and metadata retrieve
- Fixup search path on metadata spec load. Previously we were reading
  the package being passed, and not the one resolved (it failed against
selectors)
- Do inherit first pushrepositories, so they take precedence over pull
- Add test cases to cover build values interpolation by remote
  repositories
- Enhance test cases to check image cache repository inheritance when
  --from-repositories is passed
- Fix race condition when inheriting buildspec options: Instead of consuming the compiler one, annotate the updates in the
package BuildOption spec which is passed by
- Update vendor
2021-04-23 12:02:19 +02:00
Ettore Di Giacinto
ef92f23221 Inherit pullimages as pushimages while parsing compilespec metadata
Fixes #200
2021-04-19 17:18:37 +02:00
Ettore Di Giacinto
612477718e Add values interpolation inheritance test 2021-04-16 14:01:23 +02:00
Ettore Di Giacinto
c27d4d258e Allow create-repo to source trees from remote repositories
This makes possible to create the repositorie from external ones,
It also required to address #26.

Fixes #26
2021-04-16 14:01:13 +02:00
Ettore Di Giacinto
9202bcbbbe Convert raw value to templatedata before merging 2021-04-16 14:00:11 +02:00
Ettore Di Giacinto
7ba7add2a8 Allow to pull specfiles from published repositories
- Interpolates values from the repositories compilespec if present
- Automatically merge cache images coming from specified repository when
  necessary

Fixes #194
2021-04-16 13:58:51 +02:00
Ettore Di Giacinto
57c769b4a5 Refactor compiler and annotate buildoptions into compiler metadata
This allows to later pick up values used during build of each package
2021-04-16 13:57:54 +02:00
Ettore Di Giacinto
44cae094e8 Allow multiple build values files
Fixes #198
2021-04-16 13:56:51 +02:00
Ettore Di Giacinto
ec19b34ca8 Allow to pull from multiple repositories
Adds a new cli flag to luet build `--pull-repository` which allows to
pass-by a list of docker image references which are used to pull the
cache from

Fixes #185
Fixes #184
Closes #161
2021-04-08 09:17:54 +02:00
Ettore Di Giacinto
4f1e4c0b41 Switch to containerd when unpacking layers 2021-03-18 10:47:58 +01:00
Ettore Di Giacinto
662742851a Generate backend bus events in the backends 2021-03-16 14:46:28 +01:00
Ettore Di Giacinto
749a4cb615 Add --backend-args
Allow to add arguments to the backend build arguments

Fixes #146
2021-02-22 13:49:29 +01:00
Ettore Di Giacinto
21bd76af9c Uncomplicate runCommand and return command output
Fixes #189
2021-02-22 11:44:46 +01:00
Ettore Di Giacinto
b3e3abec8f Fixup spinner data race
Add spinner lock
2021-02-17 13:01:32 +01:00
Ettore Di Giacinto
101df40eec
Merge pull request #183 from mudler/docker-debug-output
Add realtime output for building phase
2021-02-13 12:02:20 +01:00
Daniele Rondina
c22adb3a47 compiler: Move spinner at the low level 2021-02-13 09:28:54 +01:00
Ettore Di Giacinto
518fb16067 Add IsVirtual() to compile spec 2021-02-09 19:05:16 +01:00
Ettore Di Giacinto
4d9297e3da Be sure to copy exact folder structure when generating final images
'COPY *' has a different behavior than 'COPY .' - when regexes are
involved, COPY behave differently, by unpacking directory content to the
root.

Enhance unit test to cover the scenario as well
2021-02-09 18:27:53 +01:00
Ettore Di Giacinto
544895e051 Handle empty packages when pushing final images
We used to create dockerfiles blindly assuming there is content, but
that's not the case for virtual packages.

Due to https://github.com/moby/moby/issues/38039 we are forced for a
"unpleasant" workaround, as we can't create empty FROM scratch images
and export them.
2021-02-09 18:27:53 +01:00
Daniele Rondina
c1fe3278fa backend: Add realtime output on building phase
The realtime output could be configured through
LUET_GENERAL__SHOW_BUILD_OUTPUT environment
variable or related config option or through
`--live-output` option.
2021-02-02 12:58:34 +01:00
David Cassany
18e9ce4557
Trim the Domain Name from cached image references
This commit removes the Domain Name, if any, from the cached image
reference before computing the image fingerprint. This way the same
image, if stored in some oter mirror, is still seen as the same one.

Fixes #158
2021-01-29 15:11:52 +01:00
Ettore Di Giacinto
685bbf46a6 Refactor common code while compiling regexes 2021-01-25 12:20:04 +01:00
Ettore Di Giacinto
d89225f37d Handle namedpipe copy
CopyFile relies on copy.Copy from https://github.com/otiai10/copy which
doesn't handle named pipes copy. Handle it here until
https://github.com/otiai10/copy/issues/47 is fixed.

This causes luet to hang while copying packages that have named pipes in
it.

Also invert compression argument for gzip, it causes slowliness.
2021-01-25 12:20:04 +01:00
Ettore Di Giacinto
55d34a3b40 Adapt artifact tests after delta changes 2021-01-24 20:07:33 +01:00
Ettore Di Giacinto
85b5c96bdd Promote to info building image messages
The user wants to know whats going on in this case. Image builds can
take up also long time
2021-01-24 19:09:09 +01:00
Ettore Di Giacinto
6f5f400765 Don't bail out if image doesn't exist locally
The backend will figure out if we have the image or not, otherwise will
atempt to pull if not there.

Skip retrieve integration test with img as its not supported.
2021-01-24 19:05:21 +01:00
Ettore Di Giacinto
be87861657 img: pull image if not locally present while extracting 2021-01-24 13:17:11 +01:00
Ettore Di Giacinto
be7b56bae3 Split ImageDefinitionToTar test
ImageDefinitionToTar it is not actually used by compiler code, but can
be handy from an API perspective, so we keep it.
2021-01-24 12:56:25 +01:00
Ettore Di Giacinto
eae2382764 unpackDelta needs a rootfs where to extract files from 2021-01-24 12:41:56 +01:00
Ettore Di Giacinto
7d11df3225 Simplify delta generation, and avoid two-pass with img backend
This changeset also drops --keep-exported-images, which is quite unused
and can be replaced with a plugin, or either by manually exporting the
resulting images.
2021-01-24 12:27:07 +01:00
Ettore Di Giacinto
b9f0ef1c55 Implement ImageExists in the img backend 2021-01-23 22:01:29 +01:00
Ettore Di Giacinto
d5f70aea26 Add test for Artifact GetUncompressedName() 2021-01-22 19:15:54 +01:00
Ettore Di Giacinto
a6e7a3059c Respect artifact extension when populating cache
This caused cache to not hit correctly
2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
91ea2ed99f Make docker image repositories actually working
Several changes are included:
- Expose ensureDir in helpers, and call it in the Docker client. In
  other implementations that was handled by CopyFile behind the scenes,
  but that's not the case here
- Create accessor in Artifact to create Artifact objects from files.
  This is handy when we have to carry over downloaded package content
  into caches when artifacts are already verified
- Fix various issues around the imagePush flag, so now trees are pushed
  forcefully each time
- Take into consideration the real artifact name when pushing single
  files in the docker image. This behavior should be changed eventually,
  because single files which aren't repository packages now are in its
  own docker image, but we should have just one that brings the required
  metadata alltogether.
2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
b27b146b45 Refactor artifact Verify() 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
dbd37afced Add docker client #169 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
322fe72ef2 Generate repository metadata and packages for docker repository type
Drop image-repository on create-repo. In case of a docker repository, --output is the image reference to use.
Also restore default output build dir.

See also: #169
2021-01-22 16:53:52 +01:00
Ettore Di Giacinto
88b5576611 Expect full image name to GenerateFinalImage
We will re-use this method also when generating repository metadata
2021-01-18 12:26:22 +01:00
Ettore Di Giacinto
a1f4c28973 Add GenerateFinalImage to package artifacts
GenerateFinalImage generates a docker image from scratch with the
artifact content.

Related to #169
2021-01-18 12:08:47 +01:00
Ettore Di Giacinto
43b0b11028 Define a build context for backends 2021-01-18 11:06:54 +01:00
Ettore Di Giacinto
ecae2873d6 zstd extension suffix is zst, not zstd
Fixes #163
2021-01-12 15:52:34 +01:00
Ettore Di Giacinto
43f5b69c18 Let the build fail when depending on virtuals
This is currently not a valid use case. Virtuals are empty packages and
if the `build.yaml` is completely empty, nothing could depend on them.

Let's try to not be too smart and build the package image if a source
image is supplied, and fail hardly when we depend on a virtual in build
time.
2021-01-03 23:03:01 +01:00
Ettore Di Giacinto
457acd0d8a Add virtual packages support 2021-01-03 20:08:04 +01:00
Niklas Engvall
bfbcb81210
Set import name for zstd 2020-12-29 22:22:53 +01:00
Ettore Di Giacinto
2b6fe2baa1 Add luet build --wait
It allows to wait for intermediate images to be available instead of
building all of them
2020-12-18 23:19:18 +01:00
Ettore Di Giacinto
f0fae82ad9 Add experimental zstd support
Closes #97
2020-12-18 21:24:01 +01:00
Ettore Di Giacinto
8e1a457bf1 Check if we have to pull images before generating delta
As we might skip building entirely, it's possible that the image is not
there yet, so we check if have to pull it or not
2020-12-15 17:01:56 +01:00