Commit Graph

681 Commits

Author SHA1 Message Date
Ettore Di Giacinto
7bfff97f57 Fixup race when updating compiler options
Instead of updating the compiler options that can be accessed by each
worker, update the compilespec BuildOptions directly
2021-04-23 12:02:41 +02:00
Ettore Di Giacinto
0288eedbc3 Always resolve buildhash image, add --rebuild to build 2021-04-23 12:02:41 +02:00
Ettore Di Giacinto
b27237b7ff Allow to pull images from multiple-repo while generating all artifacts
before, without --only-target-package, we were forcing to build the
images locally. Now we extend it also to that use-case.

Also revisit how we pass by the builder image hash, so it's easier to
read.

This change also re-enables tagging and bulding builder images all the
times.
Previously, we regressed into not tagging build images coming out-of-the
tree, with the unpleasant side-effect of not be able to re-build the
same artifacts for those leaf packages.
2021-04-23 12:02:41 +02:00
Ettore Di Giacinto
c9aed37fa7 Fixup on values interpolation and metadata retrieve
- Fixup search path on metadata spec load. Previously we were reading
  the package being passed, and not the one resolved (it failed against
selectors)
- Do inherit first pushrepositories, so they take precedence over pull
- Add test cases to cover build values interpolation by remote
  repositories
- Enhance test cases to check image cache repository inheritance when
  --from-repositories is passed
- Fix race condition when inheriting buildspec options: Instead of consuming the compiler one, annotate the updates in the
package BuildOption spec which is passed by
- Update vendor
2021-04-23 12:02:19 +02:00
Ettore Di Giacinto
ef92f23221 Inherit pullimages as pushimages while parsing compilespec metadata
Fixes #200
2021-04-19 17:18:37 +02:00
Ettore Di Giacinto
562fcc2421 Strip invalid chars from docker images also for metadata files
Fixes #199
2021-04-19 17:18:37 +02:00
Ettore Di Giacinto
ecc41ce370 Check if there are upgrades before attempting install 2021-04-16 23:38:10 +02:00
Ettore Di Giacinto
dd6501a642 Check if there are upgrades before attempting install 2021-04-16 22:28:30 +02:00
Ettore Di Giacinto
9f3e7fd0b2 PackageIndex is in repository metadata file only
This was a regression in those changeset, don't replicate the info, and
at the same time, write the artifact path containing the name of the
file only (excluding the path)
2021-04-16 15:12:51 +02:00
Ettore Di Giacinto
612477718e Add values interpolation inheritance test 2021-04-16 14:01:23 +02:00
Ettore Di Giacinto
c27d4d258e Allow create-repo to source trees from remote repositories
This makes possible to create the repositorie from external ones,
It also required to address #26.

Fixes #26
2021-04-16 14:01:13 +02:00
Ettore Di Giacinto
9202bcbbbe Convert raw value to templatedata before merging 2021-04-16 14:00:11 +02:00
Ettore Di Giacinto
fadd5a10a3 Load trees separately while reconstructing spectrees 2021-04-16 14:00:11 +02:00
Ettore Di Giacinto
7ba7add2a8 Allow to pull specfiles from published repositories
- Interpolates values from the repositories compilespec if present
- Automatically merge cache images coming from specified repository when
  necessary

Fixes #194
2021-04-16 13:58:51 +02:00
Ettore Di Giacinto
57c769b4a5 Refactor compiler and annotate buildoptions into compiler metadata
This allows to later pick up values used during build of each package
2021-04-16 13:57:54 +02:00
Ettore Di Giacinto
44cae094e8 Allow multiple build values files
Fixes #198
2021-04-16 13:56:51 +02:00
Ettore Di Giacinto
c022c75239 Write BuildTree as an additional repository file
Also split Docker repository generation a bit more for readability
2021-04-16 13:56:51 +02:00
Ettore Di Giacinto
3250d63072 Don't merge install and compilation DB. Use a temporary DB for generation 2021-04-16 13:56:51 +02:00
Ettore Di Giacinto
b8961be793 Add accessor to embed custom files to a repository while writing
Split up into methods so it's easier to add custom files
2021-04-16 13:56:45 +02:00
Ettore Di Giacinto
036b5c08c6 Split up repository generators 2021-04-08 15:24:32 +02:00
Ettore Di Giacinto
c7b79bf630 Compiler recipe now saves the entire tree 2021-04-08 15:22:00 +02:00
Ettore Di Giacinto
83cb6a2804 Check target with PathSeparator in finalizer.go 2021-04-08 10:22:24 +02:00
Ettore Di Giacinto
ec19b34ca8 Allow to pull from multiple repositories
Adds a new cli flag to luet build `--pull-repository` which allows to
pass-by a list of docker image references which are used to pull the
cache from

Fixes #185
Fixes #184
Closes #161
2021-04-08 09:17:54 +02:00
Ettore Di Giacinto
b8352a81a2 Use Lchown when copying bits, lower message to warn 2021-03-18 11:23:42 +01:00
Ettore Di Giacinto
f0a34f1cf0 Enable NoLchown
This caused to drop file permissions
2021-03-18 10:48:23 +01:00
Ettore Di Giacinto
4f1e4c0b41 Switch to containerd when unpacking layers 2021-03-18 10:47:58 +01:00
Ettore Di Giacinto
ce349ca1b7 Se the path relative to the artifact when generating docker repositories 2021-03-16 14:46:46 +01:00
Ettore Di Giacinto
662742851a Generate backend bus events in the backends 2021-03-16 14:46:28 +01:00
Ettore Di Giacinto
f8ef1c0889 Print error when failing downloading docker image 2021-03-16 11:34:36 +01:00
Ettore Di Giacinto
268239a561 Propagate verify when serializing 2021-03-15 11:37:56 +01:00
Ettore Di Giacinto
0028dd3a92 Support content trust images and pull with authentication
Contact the notary server if ```--verify``` is specified (or `verify:
true` is enabled on the repo config) and verify if the image is signed,
use the returned value to pull the verified image.
2021-03-11 17:57:59 +01:00
Ettore Di Giacinto
675170939d Expose DownloadAndExtractDockerImage as a util
Create a util sub cmd to add all utils that are handy for development
and already present in the luet codebase. We expose in this case `luet
util unpack` to unpack a docker image without a docker daemon running.
2021-03-09 09:22:37 +01:00
Ettore Di Giacinto
42f5210764 Add DownloadOnly option
It allows to download only the packages, without
installing/upgrading/replacing them

Fixes #179
2021-03-07 11:39:59 +01:00
Ettore Di Giacinto
9eda81667b Use common ImageID() when computing final images
It also adds tests and strip invalid "+" character which is not
supported in docker image tags
2021-03-07 11:01:08 +01:00
Ettore Di Giacinto
c13a2174c4 Try to chown after copy 2021-03-06 18:49:14 +01:00
Ettore Di Giacinto
f07cf6c245 Add tests for file search 2021-03-06 18:49:14 +01:00
Ettore Di Giacinto
233429bbeb Allow to search by file
Also make possible to retrieve the artifact when searching for matches
between repositories list. This made possible to show the package list
when calling `luet search`.
2021-02-28 18:42:58 +01:00
Ettore Di Giacinto
749a4cb615 Add --backend-args
Allow to add arguments to the backend build arguments

Fixes #146
2021-02-22 13:49:29 +01:00
Ettore Di Giacinto
57e19c61e7 Start spinner before pulling docker artifacts 2021-02-22 11:54:09 +01:00
Ettore Di Giacinto
5ee1e28b9c Display informative message when pulling docker artifacts 2021-02-22 11:50:52 +01:00
Ettore Di Giacinto
21bd76af9c Uncomplicate runCommand and return command output
Fixes #189
2021-02-22 11:44:46 +01:00
Ettore Di Giacinto
b3e3abec8f Fixup spinner data race
Add spinner lock
2021-02-17 13:01:32 +01:00
Ettore Di Giacinto
92e73051a0 Initialize cached in singleton 2021-02-17 09:34:54 +01:00
Ettore Di Giacinto
2448f3175e Initialize cache db if empty 2021-02-15 18:53:37 +01:00
Ettore Di Giacinto
101df40eec Merge pull request #183 from mudler/docker-debug-output
Add realtime output for building phase
2021-02-13 12:02:20 +01:00
Daniele Rondina
c22adb3a47 compiler: Move spinner at the low level 2021-02-13 09:28:54 +01:00
Ettore Di Giacinto
518fb16067 Add IsVirtual() to compile spec 2021-02-09 19:05:16 +01:00
Ettore Di Giacinto
4d9297e3da Be sure to copy exact folder structure when generating final images
'COPY *' has a different behavior than 'COPY .' - when regexes are
involved, COPY behave differently, by unpacking directory content to the
root.

Enhance unit test to cover the scenario as well
2021-02-09 18:27:53 +01:00
Ettore Di Giacinto
544895e051 Handle empty packages when pushing final images
We used to create dockerfiles blindly assuming there is content, but
that's not the case for virtual packages.

Due to https://github.com/moby/moby/issues/38039 we are forced for a
"unpleasant" workaround, as we can't create empty FROM scratch images
and export them.
2021-02-09 18:27:53 +01:00
Ettore Di Giacinto
fd80bb526e Add DirectoryIsEmpty 2021-02-09 16:51:10 +01:00