kairos/luet
1
0
Fork 0
mirror of https://github.com/mudler/luet.git synced 2025-06-25 15:01:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,277 Commits 12 Branches 164 Tags 38 MiB
2eeb464946
Commit Graph

274 Commits

Author SHA1 Message Date
Ettore Di Giacinto
3c0dd2b71d Adapt test 2020-12-07 00:07:57 +01:00
Ettore Di Giacinto
e9b4d66a3e Retrieve should be rendered also for step images 2020-12-07 00:00:32 +01:00
Ettore Di Giacinto
5047316b70 Try to build only when strictly necessary 2020-12-06 23:50:51 +01:00
Ettore Di Giacinto
6dbf19f085 Use single image to build packages 2020-12-02 21:18:12 +01:00
Ettore Di Giacinto
9423b7c1e3 Add image build events, and add luet replace 
Enhance also some commands descriptions
 2020-12-02 18:24:35 +01:00
Ettore Di Giacinto
9784d6192a Don't hide error on pulling image 2020-11-28 18:03:43 +01:00
Ettore Di Giacinto
0fe30ddcfd Add ability to interpolate during build 
Now build takes a --values argument, which is a yaml file that can be
used to interpolate the specs that are going to be compiled.
 2020-11-28 15:47:29 +01:00
Ettore Di Giacinto
44d33eceba Set workdir also on step image 
Otherwise with DOCKER_SQUASH=true it wouldn't be coherent on where to
find the package files
 2020-11-28 12:07:07 +01:00
Ettore Di Giacinto
8ce135fe12 Add DOCKER_SQUASH 2020-11-27 23:38:31 +01:00
Ettore Di Giacinto
ac149e9336 Use candidate for search, as doesn't have a selector 2020-11-15 11:47:32 +01:00
Ettore Di Giacinto
b9c8e50e42 Allow to define multiple templated packages with collections 
Collections, similarly to packages, have a `build.yaml` and
a `finalize.yaml` that are templated for each package.
They have a `collection.yaml` containing a list of
packages that are part of the tree.
 2020-11-15 00:13:46 +01:00
Ettore Di Giacinto
0e46e763d5 Move bus implementation to a separate repo, hook to events in luet 2020-11-13 18:25:44 +01:00
Ettore Di Giacinto
a8624fe451 Move image removal in compileWithImage and further cleanup 2020-11-10 18:48:39 +01:00
Ettore Di Giacinto
14c1d6ef24 Refactor and optimize build process 2020-11-10 18:14:18 +01:00
Ettore Di Giacinto
36c58307e2 Don't export unless needed 2020-11-10 16:57:24 +01:00
Ettore Di Giacinto
a765147c1d Add templated finalizers 2020-11-08 21:14:19 +01:00
Ettore Di Giacinto
b1316b50b4 Add excludes tests 2020-11-08 16:02:11 +01:00
Ettore Di Giacinto
d92ee9e1d9 Add preliminar support for excludes 2020-11-08 15:35:24 +01:00
Daniele Rondina
d219a2e0fb Run travis task with/without buildkit 2020-11-07 11:41:44 +01:00
Daniele Rondina
6819a28f07 Add support to DOCKER_BUILDKIT on test 2020-11-07 11:37:58 +01:00
Daniele Rondina
24eb6eaef5 Fix test with docker buildkit 2020-11-07 11:37:58 +01:00
Daniele Rondina
c72565e019 Integrate tests for config protects with uninstall 2020-11-06 23:30:37 +01:00
Daniele Rondina
b24d335538 GetProtectFiles() is used also for tree tarball without specs 2020-11-06 23:00:37 +01:00
Daniele Rondina
51417ecb5d pkg/compiler/artifact.go: permit to support config protect with only annotation 2020-11-06 20:23:46 +01:00
Daniele Rondina
130eb8de1a Integrate config protection on uninstall too 2020-11-06 20:14:25 +01:00
Ettore Di Giacinto
91dfb8ce3a Enhance CLI output 2020-10-30 19:15:04 +01:00
Ettore Di Giacinto
95e640c9d0 Make solver type switchable 2020-10-29 16:32:13 +01:00
Daniele Rondina
d67cf2fa33 Revert "Do image export only if we have to generate the package" 
This reverts commit 0857e53b03.
 2020-10-07 10:14:06 +02:00
Ettore Di Giacinto
0857e53b03 Do image export only if we have to generate the package 2020-10-06 19:01:25 +02:00
Ettore Di Giacinto
1c1bdca343 Add only-target-package option to luet build 2020-10-06 17:57:57 +02:00
Ettore Di Giacinto
74246780d4 Support templated packages 2020-10-04 19:33:15 +02:00
Daniele Rondina
097ea37c97 compiler/artefact: remove debug println 2020-10-02 22:37:38 +02:00
Daniele Rondina
c64660b8d1 Permit to ignore config protect rules 
* Added command line option --skip-config-protect

* Added config option config_protect_skip
 2020-10-02 22:25:21 +02:00
Daniele Rondina
93187182e5 pkg/compiler: Fix typo on error message 2020-08-19 19:24:46 +02:00
Ettore Di Giacinto
a076613f66
Fixup import path 2020-08-07 19:30:08 +02:00
Ettore Di Giacinto
95b125cb91
Pull images before executing diff tests 2020-08-06 18:03:00 +02:00
Ettore Di Giacinto
66513955c7
Compute image diffs internally 
Is it more faster in this way as we already have all the needed folders
to the comparison extracted. In this way we don't repeat I/O operation
twice by calling container-diff.

Do not depend on container-diff anymore
 2020-08-05 19:09:45 +02:00
Ettore Di Giacinto
07a1058ac1
Add cli option to skip packages if only metadata is present (without checking the image) 2020-07-17 22:42:03 +02:00
Ettore Di Giacinto
5dcf77c987
Add package buildtimestamp and luet upgrade --sync 
Annotate the package build time when compiling, and use that from the
client to force upgrade of packages that changed the artifact, but
didn't changed any version.

The client can trigger this behavior with `luet upgrade --sync`
 2020-07-12 15:29:38 +02:00
Ettore Di Giacinto
16e9d7b20c
Use packageImage as builder image fingerprint 
This allows to have an unique identifier for the builder image id against
different depgraphs combinations. The package fingerprint is not enough,
as an atom could have a difference deptree depending on the requires
constraints.

TODO: Don't use the full image name, but only the hash as a salt
(currently the salt contains ALSO a reference of the image-repository,
as such it doesn't allow to port a tree in a different docker registry)
 2020-06-23 18:59:18 +02:00
Ettore Di Giacinto
7b51e83902
Merge pull request #119 from mudler/config-protect 
Integrate feature config-protect like Gentoo/Funtoo do
 2020-06-07 11:33:55 +02:00
Daniele Rondina
3a365c709b Integrate config-protect from package spec 2020-06-06 13:12:54 +02:00
Ettore Di Giacinto
264e1e9652
Print inspect output as string 2020-06-06 11:32:47 +02:00
Daniele Rondina
c87db16d31 tarModifierWrapperFunc: Now use strings.HasPrefix for match path 2020-06-06 10:38:11 +02:00
Ettore Di Giacinto
90a25406a0
Add --full option to build 
Don't compile dependencies when computing all the compilation specs from
a tree if are among the target deps

Fixes #41
 2020-06-06 08:58:18 +02:00
Daniele Rondina
16f717f04b tarModifierWrapperFunc: Fix config protect filename 2020-06-05 23:12:19 +02:00
Ettore Di Giacinto
9e0e1199df
Check if image exists before skipping compilation 2020-06-03 21:00:30 +02:00
Daniele Rondina
8f0c528c08 Create helpers.UntarProtect for handle protected files 
Currently, it's used the archive.ReplaceFileTarWrapper
that requite a []byte of the files replaced. This is not
a good idea if files are big and instead could be better
in the near future reimplement ReplaceFileTarWrapper with
a callback that return io.Reader instead of []byte.

If a protected file is already present on target rootfs
it is created a file with the same prefix used in Gentoo:

._cfgXXXX_<filename>
 2020-06-02 11:08:37 +02:00
Ettore Di Giacinto
9971fe9f45
Unique hashes for packages without deps 2020-05-18 19:57:01 +02:00
Ettore Di Giacinto
51711dafba
Add package_dir to pack a spec dir as the main artifact 2020-05-16 21:34:27 +02:00
Daniele Rondina
11944873ea Integrate tmpdir_base params and tmpdirs cleanup 2020-04-30 20:29:28 +02:00
Ettore Di Giacinto
528f481a7b
Annotate package files in metadata 
Fixes #87
 2020-04-13 10:52:41 +02:00
Daniele Rondina
69a82a1ca5 simpledocker: Use debug option for print container-diff results 2020-04-10 09:32:12 +02:00
Daniele Rondina
02c33896d5 simpledocker: Move warning before return 2020-04-10 09:31:08 +02:00
Daniele Rondina
c2e9176ab2 solver.Order now return error 2020-04-09 17:07:57 +02:00
Ettore Di Giacinto
0d2668e452
Print warning if container-diffs return errors to stderr 2020-04-08 18:31:40 +02:00
Ettore Di Giacinto
5e31d940f0
Introduce Packages for []Package 2020-04-04 14:29:08 +02:00
Ettore Di Giacinto
fab2d21ac1
Drop unused function 2020-03-19 22:39:22 +01:00
Ettore Di Giacinto
487239334f
Treat base case with same algorithm of dependencies 2020-03-19 22:37:39 +01:00
Ettore Di Giacinto
69477a0c36
Make uniform build of seeds and packages 2020-03-19 22:34:26 +01:00
Ettore Di Giacinto
67362a3bc1
Drop unused condition 2020-03-19 22:29:34 +01:00
Ettore Di Giacinto
302d18e749
Tag the target image with the computed hash 2020-03-19 22:28:33 +01:00
Ettore Di Giacinto
e3ebfd6bfe
Respect ImageRepository with build --nodeps 2020-02-26 19:00:22 +01:00
Ettore Di Giacinto
07610bc216
Add build option to remove backend artifacts 
Add keep-exported-images to build to instruct to remove backend artifacts (docker images)
 2020-02-21 22:33:18 +01:00
Ettore Di Giacinto
7e16e0cdb6
Add HashFingerprint() to Package and consume it while generating image names 
In this way we don't depend on the package fingerpint characters while generating images, allowing
to support completely semver (e.g. + character is not allowed in docker images tag).

This change is not backward compatible with images currently already built and will invalidate the cache
 2020-02-19 18:28:29 +01:00
Ettore Di Giacinto
2ce427d601
Add Nodeps, Onlydeps and Force options to Compiler 
See #41
 2020-02-18 18:36:44 +01:00
Ettore Di Giacinto
26b94888c3
Skip image building if we pulled them successfully 2020-02-15 22:53:00 +01:00
Ettore Di Giacinto
fcec6c5699
Add Push to hub feature 
Add to the backends the Push capabilities to push images that were correctly built.
Also expose them to the CLI, along with KeepImg which was hidden.
 2020-02-15 14:45:05 +01:00
Ettore Di Giacinto
c7253ac8ad
Adapt test to spec changes 2020-02-14 08:04:26 +01:00
Ettore Di Giacinto
f329e1d5e0
Add retrieve to compilation spec 
It allows to copy build artifact during buildtime. In this way
artifacts between different seed images can be shared

```
retrieve:
- a-test-1.0.package.*
- https://...
```

They will be available under WORKDIR
 2020-02-13 14:15:43 +01:00
Ettore Di Giacinto
07a154474b
Consume SolverOptions in compiler 2020-02-12 11:21:55 +01:00
Daniele Rondina
524bbf990e Add support for same-owner config option 2020-02-01 19:01:15 +01:00
Ettore Di Giacinto
c6fe34b059
Support compression and checksum for trees 
Fixes #33 #34 #35
 2020-01-28 17:46:32 +01:00
Daniele Rondina
07633dc307 Add container-diff summary 2020-01-05 18:05:56 +01:00
Ettore Di Giacinto
d5fd14bceb
Merge branch 'build_clean' into develop 2020-01-05 16:27:31 +01:00
Ettore Di Giacinto
d2abaa9cc1
Add message asserting skip. Also return pointer to allow to edit Options 2020-01-05 16:26:42 +01:00
Ettore Di Giacinto
d23e1dee78
Default to Clean true in CompilerOptions 2020-01-05 16:08:40 +01:00
Ettore Di Giacinto
ee055e08b1
Set abs path when returning artifact from yaml 2020-01-05 16:08:39 +01:00
Ettore Di Giacinto
02c37c7451
Fix tests after contructor change 2020-01-05 16:08:39 +01:00
Ettore Di Giacinto
1d1efad18b
Skip building if artifact already exists 2020-01-05 15:30:16 +01:00
Ettore Di Giacinto
bcc6ce19ea
Move compiler options to its own struct 
Also add Clean attribute, to handle a future build clean method
 2020-01-05 14:32:26 +01:00
Daniele Rondina
7ec36da059 Add --fatal option 
Close #8
 2020-01-03 15:41:45 +01:00
Daniele Rondina
358b39b5dd
simpledocker: Support show_build_output option 2020-01-01 13:43:22 +01:00
Daniele Rondina
0cb49a40c0
Support logging level and spinner customization 2020-01-01 13:43:22 +01:00
Ettore Di Giacinto
efdfe72568
Treat CompressionType none as default 
To provide backward compatibility with repos that didn't declares it explictly
 2019-12-31 12:29:53 +01:00
Ettore Di Giacinto
3d5b723668
Add compression tests 
Refers to #33
 2019-12-30 16:35:35 +01:00
Ettore Di Giacinto
4f65d46d56
Drop CompressedPath, or we don't have a way to compare checksums 
Refers to #33
 2019-12-30 16:35:30 +01:00
Ettore Di Giacinto
d48f510f14
Propagate Checksum and CompressionType when building ArtifactIndex 
Refers to #33
 2019-12-30 16:35:28 +01:00
Ettore Di Giacinto
ea27ada6c0
Do not return errors after we uncompress successfully 
Refers to #33
 2019-12-30 16:35:25 +01:00
Ettore Di Giacinto
f71c9937c4
Add compression to build CLI 
Also handle how concurrency is set now.

Adds also an accessor to compiler to set the desired compression type

Refers to #33
 2019-12-30 16:35:21 +01:00
Ettore Di Giacinto
475b63be95
Consume concurrency from compiler 
Refers to #33
 2019-12-30 16:35:18 +01:00
Ettore Di Giacinto
a40ecaea40
Use a separate attribute to handle the compressed artifact 
Refers to #33
 2019-12-30 16:35:15 +01:00
Ettore Di Giacinto
5155681513
Fixup tests 
Refers to #33
 2019-12-30 16:35:12 +01:00
Ettore Di Giacinto
d2d72c3fc4
Add package compression type 
TODO: Handle the path substitution in a separate field
Adds GZip support and allows the compiler to switch compression type.

It also adds Concurrency as a compiler attribute (not consumed yet)

Refers to #33
 2019-12-30 16:34:41 +01:00
Ettore Di Giacinto
bb98259a48
Add sanity check test 
To verify that we are actually comparing with some data

Closes #28
 2019-12-29 14:14:06 +01:00
Ettore Di Giacinto
fea6061f89
Add hash test to artifact_test 
Refers to #28
 2019-12-29 14:14:03 +01:00
Ettore Di Giacinto
cb98a49917
Create new Checksum struct for Artifact 
Refers to #28
 2019-12-29 14:13:51 +01:00
Ettore Di Giacinto
eeb6719529
Add accessors to Hash and Verify artifacts 
Refers to #28
 2019-12-29 13:59:58 +01:00
Ettore Di Giacinto
17982e9527
Add package to calculate and compare artifact checksums 
Refers to #28
 2019-12-29 13:59:47 +01:00
Ettore Di Giacinto
2fa9c754ae
Move archive helpers to artifact 
This allow in the future to swap and provide archive/compression methods without hijacking the code.

Refers to #33
 2019-12-28 16:48:05 +01:00
Ettore Di Giacinto
d941c9755b Relax best match constraint rule 
Or we could return unsat
 2019-12-18 19:24:44 +01:00
Ettore Di Giacinto
15250bd991 Add support for Package provides 
Add "provides" field in packages (which affect both runtime and buildtime deps).
It replaces all the occurences in the deptree before solving, actually
allowing to swap packages and provide virtuals. Along with a mechanism
for package rename #25.
 2019-12-17 19:32:31 +01:00
Ettore Di Giacinto
fe608469d8
Do not call solver multiple times, reuse same assertion 
Introduce Cut(), it allows to filter in the assertion to achieve the same hashes
 2019-12-14 15:00:16 +01:00
Ettore Di Giacinto
556668fcc4 Make Expand and Revdeps consume a database 2019-12-06 16:28:42 +01:00
Ettore Di Giacinto
097f2ec827
Support includes also on packages coming from unpacked images 2019-12-03 18:15:07 +01:00
Ettore Di Giacinto
2f9d12a248
Add KeepImg and PullFirst options 
They allow to keep a cache,define an endpoint and pull it first.

Also slim logic in favor of more freedom on backend side
 2019-12-02 16:36:55 +01:00
Ettore Di Giacinto
1a584d0bfa
Fix img backend calls 
Options have to go first always.

Note: it needs runc when unpacking
 2019-12-02 16:36:29 +01:00
Ettore Di Giacinto
58169770e1
Add test for FindPackageCandidate 
Adapt tests for env
 2019-12-01 23:24:42 +01:00
Ettore Di Giacinto
ebf818ff08
Add Environment to compilespec #24 
Now compilespecs can define envs and have PACKAGE_NAME, PACKAGE_VERSION and PACKAGE_CATEGORY injected
 2019-12-01 21:20:16 +01:00
Ettore Di Giacinto
55fa7265e5
Move selection logic to a db accessor #19 
In this way the compiler consume such accessor, and make it possible to expand whenever used
 2019-12-01 19:11:31 +01:00
Ettore Di Giacinto
ddfb4fb8e8
Drop explicit directory copy 2019-12-01 19:10:40 +01:00
Ettore Di Giacinto
e23e38b571
Do not treat copy errors as fatal for now 2019-11-30 11:51:50 +01:00
Ettore Di Giacinto
f4fff415f4
Disable extra output from container-diff 2019-11-30 11:51:37 +01:00
Ettore Di Giacinto
c7e1803540
Fix path deletion on artifacts 2019-11-29 19:01:59 +01:00
Ettore Di Giacinto
e31f8820e2
Adapt tests 
Adapt docker and package tests

Fix tree test
 2019-11-29 19:01:54 +01:00
Ettore Di Giacinto
d0c84d24a0
Expand also when ordering assertions 2019-11-29 19:01:52 +01:00
Ettore Di Giacinto
4031a4ae81
Marshal JSON of package without escapes 
Otherwise >, < get escaped as JSON does that in favour of browsers
 2019-11-29 19:01:50 +01:00
Ettore Di Giacinto
fa46601638
Adapt compiler and installer to Tree removal 2019-11-29 19:01:49 +01:00
Ettore Di Giacinto
6d25e5d881
Check if a layer is provided in the manifest 2019-11-26 20:22:33 +01:00
Ettore Di Giacinto
393164da36
Extract all layers of the docker image 2019-11-26 20:12:06 +01:00
Ettore Di Giacinto
1cf5f2eb74
Enhance compile output 2019-11-26 20:11:51 +01:00
Ettore Di Giacinto
caca3fbf8c
Return immediately in case of deps failures 2019-11-26 20:11:27 +01:00
Ettore Di Giacinto
3b9337a03b
Keep the path around after writing the yaml, otherwise compile specs couldn't be found 2019-11-23 18:41:15 +01:00
Ettore Di Giacinto
e9c01b46a7
Use well defined structure for serializing, fixups to make test green 2019-11-23 15:42:53 +01:00
Ettore Di Giacinto
542d45a646
Do not use interface in struct used for serialization 2019-11-23 00:29:44 +01:00
Ettore Di Giacinto
df78308e98
Annotate artifact metadata after compiling 2019-11-22 21:01:29 +01:00
Ettore Di Giacinto
f107f47242
Prepare for 0.1 tag 2019-11-17 19:45:20 +01:00
Ettore Di Giacinto
6e662c75ce
Reuse docker backend in some Img functions 2019-11-17 18:38:57 +01:00
Ettore Di Giacinto
8ed2badefc
Add test for shared layer unpack 2019-11-17 15:59:52 +01:00
Ettore Di Giacinto
a39f596e8f
Do not compute diff when package is unpacked 2019-11-17 15:59:52 +01:00
Ettore Di Giacinto
d0b128000d
Clean build dirs for each package after build 2019-11-17 15:59:52 +01:00
Ettore Di Giacinto
dea8ea7f9a
Respect unpack also when compiling images #15 2019-11-17 15:59:48 +01:00
Ettore Di Giacinto
3499c0c580
Add test for compilation with version selection 2019-11-16 16:05:51 +01:00
Ettore Di Giacinto
32dea6f7fd
Add Best() to solver for package selection based on version 2019-11-16 14:40:58 +01:00
Ettore Di Giacinto
602f5c68d5
Use emojis for icons 2019-11-16 14:16:59 +01:00
Ettore Di Giacinto
588d06430c
Do not spin in the backend 
Signed-off-by: Ettore Di Giacinto <mudler@gentoo.org>
 2019-11-16 13:26:34 +01:00
Ettore Di Giacinto
3e850e9607
Adapt tests and fixup races in spinner 
Signed-off-by: Ettore Di Giacinto <mudler@gentoo.org>
 2019-11-16 13:26:33 +01:00
Ettore Di Giacinto
b2060c82e3
Make DB Switchable 
Fixes races conditions and make the DB Switchable. Also prepare inside
the CompilationSpec the tree of the deps to be built, and parallelize
only the building jobs.

Closes #7

Signed-off-by: Ettore Di Giacinto <mudler@gentoo.org>
 2019-11-16 13:26:33 +01:00
Ettore Di Giacinto
c95e0ed91d
Add revdep calculation 
Add CompilationSpecs type to handle slices of CompilationSpec, to
perform operation such as Unique() and Remove().

Add also dependencies and the spec associated to the Artifact, to track
how the artifact was generated.

Add revdeps compilation wrapper, and unit tests
 2019-11-15 18:11:26 +01:00
Ettore Di Giacinto
321edd1676
Cleans container-diff cache on each run 
It generates false positives if run with the same image paths
 2019-11-14 22:50:39 +01:00
Ettore Di Giacinto
cd73d950cc
Add fixture and test for includes and layered packages 2019-11-14 20:28:21 +01:00
Ettore Di Giacinto
ccdca66420
Small fixes and enhance output 
Fixes a small issue with the Spinner which prevented to update the
spinner style. Also respect the temp folders of the output when
extracting the rootfs.
 2019-11-14 20:22:49 +01:00
Ettore Di Giacinto
0eef18d75c
Add 'includes' to CompileSpec 
The includes field is an array of regexp used to indicate what include
inside the packag. It can be omitted to default behavior (all)
 2019-11-14 17:43:47 +01:00
Ettore Di Giacinto
08944a22ac
Add unicode icons 2019-11-14 00:14:43 +01:00
Ettore Di Giacinto
d5a8579454
Propagate also original Artifact name inside the copyjob 2019-11-13 17:06:55 +01:00
Ettore Di Giacinto
d752c0572b
Add unpack field to specify a package used as layer 
Closes #9
 2019-11-13 09:43:54 +01:00
Ettore Di Giacinto
747d0ef9ac
Do not fail in case we cannot remove cached images 
Add a TODO, we should have a cli flag to treat warnings as fatal
 2019-11-12 22:17:51 +01:00
Ettore Di Giacinto
2bbca653fd
Add tree test and minor comment fixups 2019-11-12 17:31:50 +01:00
Ettore Di Giacinto
a2c32aaaa1
Propagate reverse dep calculation 
Whenever we update the tree with new informations, all the sibilings
needs to be updated as well

Also switch to inmemory db for recipe - even if this should be a switch
 2019-11-12 17:30:06 +01:00
Ettore Di Giacinto
a7a20358ef
Search in world instead of db (or solver would mismatch) 2019-11-12 17:26:00 +01:00