kairos/luet
1
0
Fork 0
mirror of https://github.com/mudler/luet.git synced 2025-07-31 23:05:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,174 Commits 12 Branches 164 Tags 38 MiB
06fa8b1c87
Commit Graph

707 Commits

Author SHA1 Message Date
Ettore Di Giacinto
4b4c3a2e14 Adapt tests to new constructor changes 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
163f93067c Temporarly rework genuinetools/img code for pull/unpack without Docker 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
91ea2ed99f Make docker image repositories actually working 
Several changes are included:
- Expose ensureDir in helpers, and call it in the Docker client. In
  other implementations that was handled by CopyFile behind the scenes,
  but that's not the case here
- Create accessor in Artifact to create Artifact objects from files.
  This is handy when we have to carry over downloaded package content
  into caches when artifacts are already verified
- Fix various issues around the imagePush flag, so now trees are pushed
  forcefully each time
- Take into consideration the real artifact name when pushing single
  files in the docker image. This behavior should be changed eventually,
  because single files which aren't repository packages now are in its
  own docker image, but we should have just one that brings the required
  metadata alltogether.
 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
b27b146b45 Refactor artifact Verify() 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
dbd37afced Add docker client #169 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
2f459c0469 Use only one docker image reference to push repository. 
Instead of generating different images, which are harder to track and
clean, we generate a single image with various tags, corresponding to
the packages available in the repositories.

Tagging, and pushing separate images will be possible with the plugin
mechanism
 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
ad455edafc Allow to push images in create-repo 
Add also the --force flag to allow image overwrite

Related to #169
 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
d9286a1a1e Download repository metadata with client DownloadFile, uniform downloads for Docker repositories 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
322fe72ef2 Generate repository metadata and packages for docker repository type 
Drop image-repository on create-repo. In case of a docker repository, --output is the image reference to use.
Also restore default output build dir.

See also: #169
 2021-01-22 16:53:52 +01:00
Ettore Di Giacinto
88b5576611 Expect full image name to GenerateFinalImage 
We will re-use this method also when generating repository metadata
 2021-01-18 12:26:22 +01:00
Ettore Di Giacinto
a1f4c28973 Add GenerateFinalImage to package artifacts 
GenerateFinalImage generates a docker image from scratch with the
artifact content.

Related to #169
 2021-01-18 12:08:47 +01:00
Ettore Di Giacinto
43b0b11028 Define a build context for backends 2021-01-18 11:06:54 +01:00
Ettore Di Giacinto
ecae2873d6 zstd extension suffix is zst, not zstd 
Fixes #163
 2021-01-12 15:52:34 +01:00
Ettore Di Giacinto
bd3e483f0f Be sure to cache packages with same fingerprint only once 
this makes sure that we are fast to process also invalid trees
 2021-01-12 10:30:31 +01:00
Ettore Di Giacinto
40fc948c6e Stabilize tests after changes 
With BuildWorld() we get more results back (now we return the whole
model, including the false assertions).

Besides, now solving with BuildWorld() detects an invalid case:
when we supply a provided, the definitionDB shouldn't explictly supply
also the package that has to be provided. This would cause to 'shadow'
packages between repositories.

The test was invalid before, and shouldn't have contained A1. Moved the
test to Pending to inspect it further in subsequent dev iterations
 2021-01-11 23:35:18 +01:00
Ettore Di Giacinto
186fa58ab0 Use BuildWorld() instead of BuildPartialWorld() in solver.Solve 
We now have a stronger cache system while we pre-compute also RevDeps in
a hashmap, this makes now makes BuildWorld() much more performant.
 2021-01-11 20:11:51 +01:00
Ettore Di Giacinto
7390623e40 Don't warn user of accepting license in case of uninstall 
Closes #160
 2021-01-07 10:36:57 +01:00
Ettore Di Giacinto
bd0d2765aa Mark executed finalizers at beginning 
don't retry failing finalizers, but mark as executed right away
 2021-01-04 17:04:20 +01:00
Ettore Di Giacinto
43f5b69c18 Let the build fail when depending on virtuals 
This is currently not a valid use case. Virtuals are empty packages and
if the `build.yaml` is completely empty, nothing could depend on them.

Let's try to not be too smart and build the package image if a source
image is supplied, and fail hardly when we depend on a virtual in build
time.
 2021-01-03 23:03:01 +01:00
Ettore Di Giacinto
457acd0d8a Add virtual packages support 2021-01-03 20:08:04 +01:00
Ettore Di Giacinto
a81d0bc3a3 Build assertions when swapping 
When we are swapping packages, we do not run the solver to gather things
to install, but we trust the given list when calling computeInstall. In this case, the assertion
returned by computeInstall is empty, as we force l.Options.NoDeps.

This change generates the assertion list while calling computeSwap so
it's available later when we call ExecuteFinalizer.
 2021-01-02 21:28:54 +01:00
Ettore Di Giacinto
bb48326039 Adapt solver tests after changes 2020-12-30 02:05:55 +01:00
Ettore Di Giacinto
dce8b52293 Use Conflicts() which already lists revdeps on failure 2020-12-30 01:17:31 +01:00
Ettore Di Giacinto
0652fce55e Update revdeps table while populating Cache 
When we cycle, we don't necessarly have all the packages into the DB
yet.

With this change, luet annotates the reverse dependency without any version, and we try to
update revdeps table when new items gets added, by checking the version
required in the selector.

Thanks to @joostruis for noticing the issue
 2020-12-30 01:12:35 +01:00
Ettore Di Giacinto
38c9540a1d Use DB copy in GetRevdeps in BoltDB 2020-12-30 01:12:09 +01:00
Ettore Di Giacinto
90278a034b Use ConflictsWith to check conflicts when uninstalling packages 2020-12-29 23:43:39 +01:00
Ettore Di Giacinto
55ab1894e9 Add unit test for Uninstall in Installer 2020-12-29 22:58:03 +01:00
Niklas Engvall
bfbcb81210
Set import name for zstd 2020-12-29 22:22:53 +01:00
Ettore Di Giacinto
062e75bc25 Add unit test for Uninstall without full 2020-12-29 22:13:26 +01:00
Ettore Di Giacinto
b81ce66914 Reduce download verbosity 2020-12-27 20:21:05 +01:00
Ettore Di Giacinto
f871111e50 Collect errors from finalizer runs 
Instead of failing and depend on the --force flag, always execute
finalizer and collect errors to determine if install was successfull or
not
 2020-12-25 10:35:09 +01:00
Ettore Di Giacinto
736c9470cf Add db copy and clone 2020-12-19 17:45:50 +01:00
Ettore Di Giacinto
e52bc4f2b2 Refactor: get systemdb from config, which knows which one to load 2020-12-19 17:23:59 +01:00
Ettore Di Giacinto
96e877fc0b Allow uninstall to take multiple packages 
And treat those as a list, instead of each single of them
 2020-12-19 17:16:58 +01:00
Ettore Di Giacinto
525bfb5ebf Respect --nodeps when calling Swap from the public interface 2020-12-19 15:26:18 +01:00
Ettore Di Giacinto
f4e2f32aff Return candidate not found when appropriate 2020-12-19 14:57:42 +01:00
Ettore Di Giacinto
7cf650a8f6 Break Swap in computeSwap() and display uninstall dialog only when asked 2020-12-19 14:55:59 +01:00
Ettore Di Giacinto
2b6fe2baa1 Add luet build --wait 
It allows to wait for intermediate images to be available instead of
building all of them
 2020-12-18 23:19:18 +01:00
Ettore Di Giacinto
f0fae82ad9 Add experimental zstd support 
Closes #97
 2020-12-18 21:24:01 +01:00
Ettore Di Giacinto
1c9b821058 Drop unneeded if 2020-12-18 00:50:20 +01:00
Ettore Di Giacinto
0e21548bc0 Lookup uninstall and Install in installer.Swap 
In this way we resolve selectors from user inputs
 2020-12-18 00:49:51 +01:00
Ettore Di Giacinto
39c8895f80 Return provided if selector isn't a range in boltdb 2020-12-17 23:17:40 +01:00
Ettore Di Giacinto
ce169f49af If provided isn't a selector, it means we don't have to return a range 
Add also more tests about provides
 2020-12-16 22:17:34 +01:00
Ettore Di Giacinto
8e1a457bf1 Check if we have to pull images before generating delta 
As we might skip building entirely, it's possible that the image is not
there yet, so we check if have to pull it or not
 2020-12-15 17:01:56 +01:00
Ettore Di Giacinto
70f05f41e8 Check only if package image exists 
We don't need to look after the builder image as its optional. In this
way we can also reduce the compiler options, as we don't require a
--clean flag anymore. --only-target-package is sufficient to determine
what we can skip and how.
 2020-12-14 18:41:39 +01:00
Ettore Di Giacinto
ef034d87b0 Detect if images are available if we don't have to generate a Package 
While building, if we aren't doing a clean build, we scan now to see if
images are available and we skip, in case we don't find a metadata
already.
 2020-12-14 18:32:32 +01:00
Ettore Di Giacinto
265e2371b4 Add ArtifactNode to test, now we get the gen Dockerfile in the diff 2020-12-12 16:04:54 +01:00
Daniele Rondina
78442c91fc events: Review description of build_artifact events 2020-12-12 16:03:40 +01:00
Ettore Di Giacinto
d97e606a31 Adapt fixtures and tests 2020-12-12 12:10:24 +01:00
Ettore Di Giacinto
95da20e366 Context files are immutable 2020-12-12 11:55:25 +01:00
Ettore Di Giacinto
797a34ba49 Reuse same dockerfile gen logic between prelude and steps 
As now we build only when necessary, we need to make sure the images are
built similarly. The discrepancies between the two are less now, and
they can share the same logic.

This fixes a regresion where when no prelude is defined, the build
context isn't copied over
 2020-12-12 11:16:34 +01:00
Ettore Di Giacinto
a1453b7242 Fixup error messages 2020-12-11 23:03:56 +01:00
Ettore Di Giacinto
9eef7e5c6d Clean up if condition 2020-12-09 22:58:33 +01:00
Ettore Di Giacinto
332824fd42 Fail in the downloader goroutine and don't skip errors with force 2020-12-09 22:56:55 +01:00
Ettore Di Giacinto
737fbdbdc1 Don't make artifact checksum skippable 2020-12-09 21:31:07 +01:00
Ettore Di Giacinto
b5990b5333 Generate changes from CompilerBackendOptions and pass by image name so img can unpack images 2020-12-09 00:27:37 +01:00
Ettore Di Giacinto
9911888d18 Stabilize test 2020-12-08 14:56:51 +01:00
Ettore Di Giacinto
cf5e4e1305 Detect removed also when availables aren't found 2020-12-08 12:28:20 +01:00
Ettore Di Giacinto
519586f6bc Search for removed in Def DB 2020-12-08 12:07:28 +01:00
Ettore Di Giacinto
6dbc422b8f Apply solver change to UpgradeUniverse also to the parallel variant and adapt tests 
Similarly, we want just to consider what is being uninstalled and the
new rules of the package that is going to be upgraded
 2020-12-08 11:43:38 +01:00
Ettore Di Giacinto
a3cfebf438 Create BuildFormula from installed with InstallDatabase 
Instead of using the DefinitionDB which supposedly contains only the
relations present in the online repositories. In this way the solver its
more consistent and tries to solve with only the internal definitions.

This also fixes quirks with luet upgrade --universe
 2020-12-08 10:58:08 +01:00
Ettore Di Giacinto
24201b25ef Apply solver change also to the parallel variant 2020-12-08 10:41:03 +01:00
Ettore Di Giacinto
7c53296530 Adapt tests 2020-12-08 10:39:15 +01:00
Ettore Di Giacinto
a3cb0ed17f When attempting to uninstall, do it from the internal db so it can resolve the current versions 2020-12-08 02:04:54 +01:00
Ettore Di Giacinto
9a34296be0 Build step is always required for tagging images 2020-12-07 19:39:56 +01:00
Ettore Di Giacinto
ebd18ae22c Set builderTagged image afterwards 2020-12-07 18:58:14 +01:00
Ettore Di Giacinto
6bf7368993 Don't replace buildertaggedImage if there aren't build steps 2020-12-07 18:39:15 +01:00
Ettore Di Giacinto
338f310d67 Tag and push an image when virtual is supplied, to have a track of it in the image graph tree 2020-12-07 17:59:30 +01:00
Ettore Di Giacinto
59d78c3f5c While upgrading always use nodeps while computing uninstall 2020-12-07 17:20:55 +01:00
Ettore Di Giacinto
86c256a062 Generate empty tar 2020-12-07 17:20:32 +01:00
Ettore Di Giacinto
3c0dd2b71d Adapt test 2020-12-07 00:07:57 +01:00
Ettore Di Giacinto
e9b4d66a3e Retrieve should be rendered also for step images 2020-12-07 00:00:32 +01:00
Ettore Di Giacinto
5047316b70 Try to build only when strictly necessary 2020-12-06 23:50:51 +01:00
Ettore Di Giacinto
d479ada402 Don't consider deps while uninstalling during package Swap 
Beside being forced, it also doesn't need to look deep into the deps, as
we already have precalculated those
 2020-12-06 22:48:48 +01:00
Ettore Di Giacinto
7b800c9a20 Pre-compute swap step 
Otherwise, while upgrading, it could happen that package dependencies
aren't downloaded before, and they would just be installed in the middle
of installation, after removal already happened.
 2020-12-06 22:11:17 +01:00
Ettore Di Giacinto
18e6e085d5 Sort correctly also subfolders 2020-12-05 23:17:05 +01:00
Ettore Di Giacinto
67c43eb936 Don't bail out if package is installed and we have a list 2020-12-03 20:03:37 +01:00
Ettore Di Giacinto
cf80e5fc09 Resolvers might omit packages 2020-12-03 18:53:57 +01:00
Ettore Di Giacinto
d668d8344b Accept selectors on uninstall and fixup failure logic 2020-12-03 18:32:24 +01:00
Ettore Di Giacinto
b17ac447f1 Display matched packages only, and check if they are available 2020-12-03 17:25:29 +01:00
Ettore Di Giacinto
6dbf19f085 Use single image to build packages 2020-12-02 21:18:12 +01:00
Ettore Di Giacinto
9423b7c1e3 Add image build events, and add luet replace 
Enhance also some commands descriptions
 2020-12-02 18:24:35 +01:00
Ettore Di Giacinto
9784d6192a Don't hide error on pulling image 2020-11-28 18:03:43 +01:00
Ettore Di Giacinto
0fe30ddcfd Add ability to interpolate during build 
Now build takes a --values argument, which is a yaml file that can be
used to interpolate the specs that are going to be compiled.
 2020-11-28 15:47:29 +01:00
Ettore Di Giacinto
44d33eceba Set workdir also on step image 
Otherwise with DOCKER_SQUASH=true it wouldn't be coherent on where to
find the package files
 2020-11-28 12:07:07 +01:00
Ettore Di Giacinto
8ce135fe12 Add DOCKER_SQUASH 2020-11-27 23:38:31 +01:00
Ettore Di Giacinto
18d9366bca Minor fixes 2020-11-24 18:27:49 +01:00
Ettore Di Giacinto
db1b190fb5 Minor fixup and cleanups around the new prompt feature 2020-11-23 18:20:30 +01:00
Ettore Di Giacinto
b349665ff2 Add user prompts 
Fixes #106
 2020-11-22 23:43:29 +01:00
Ettore Di Giacinto
53ab0e0dd2
Merge pull request #151 from mudler/download-progress-bar 
Download progress bar
 2020-11-20 19:00:25 +01:00
Daniele Rondina
60d5c9dfd5 Add download progress bar 2020-11-20 18:12:23 +01:00
Ettore Di Giacinto
1f807f369a Move revdeps computation to db 2020-11-20 17:23:21 +01:00
Ettore Di Giacinto
0cc2b72831 Drop converter code, will be in a separate extension 2020-11-19 18:10:16 +01:00
Ettore Di Giacinto
f2df3faee5 Now Uninstall takes multiple packages 2020-11-19 18:05:27 +01:00
Daniele Rondina
f9a7113ab9 client/http: Add experimental download info 2020-11-19 00:56:28 +01:00
Ettore Di Giacinto
ac149e9336 Use candidate for search, as doesn't have a selector 2020-11-15 11:47:32 +01:00
Ettore Di Giacinto
b9c8e50e42 Allow to define multiple templated packages with collections 
Collections, similarly to packages, have a `build.yaml` and
a `finalize.yaml` that are templated for each package.
They have a `collection.yaml` containing a list of
packages that are part of the tree.
 2020-11-15 00:13:46 +01:00
Daniele Rondina
83f924da35 spectools: Add DefaultPackageSanitized.Clone() 2020-11-14 12:42:49 +01:00
Ettore Di Giacinto
0e46e763d5 Move bus implementation to a separate repo, hook to events in luet 2020-11-13 18:25:44 +01:00
Ettore Di Giacinto
a793b44e83 Wip 2020-11-12 23:21:10 +01:00
Ettore Di Giacinto
a8624fe451 Move image removal in compileWithImage and further cleanup 2020-11-10 18:48:39 +01:00
Ettore Di Giacinto
14c1d6ef24 Refactor and optimize build process 2020-11-10 18:14:18 +01:00
Ettore Di Giacinto
36c58307e2 Don't export unless needed 2020-11-10 16:57:24 +01:00
Ettore Di Giacinto
a765147c1d Add templated finalizers 2020-11-08 21:14:19 +01:00
Ettore Di Giacinto
cead09fb9f
Merge pull request #148 from mudler/respect_rootfs4conf 
Respect rootfs path for configs and url
 2020-11-08 18:25:29 +01:00
Daniele Rondina
9a1787ddaf client/local: Handle config_from_host on DownloadFile 2020-11-08 17:06:05 +01:00
Ettore Di Giacinto
b1316b50b4 Add excludes tests 2020-11-08 16:02:11 +01:00
Ettore Di Giacinto
d92ee9e1d9 Add preliminar support for excludes 2020-11-08 15:35:24 +01:00
Ettore Di Giacinto
6a1b64acea Order files before uninstall 
Fixes #149
 2020-11-08 12:36:41 +01:00
Ettore Di Giacinto
459eb01a59
Don't write err to stdout if not present 2020-11-08 10:02:00 +01:00
Daniele Rondina
e70cdbaaf7 Respect rootfs on repositories urls 2020-11-08 00:00:15 +01:00
Daniele Rondina
513f441bb3 Add option config_from_host 2020-11-07 18:56:25 +01:00
Daniele Rondina
ebe7466fdc Respect rootfs path for load config 2020-11-07 18:28:23 +01:00
Ettore Di Giacinto
46ed6423ad
Merge pull request #147 from mudler/fix-protect-uninstall 
Fix protect uninstall
 2020-11-07 12:28:24 +01:00
Daniele Rondina
d5df40512b installer: Improve message for protected files 2020-11-07 12:27:18 +01:00
Daniele Rondina
d219a2e0fb Run travis task with/without buildkit 2020-11-07 11:41:44 +01:00
Daniele Rondina
4048138dcb Add test suite for ConfigProtect 2020-11-07 11:39:31 +01:00
Daniele Rondina
e5f44eee09 ConfigProtect: support annotation without initial / 2020-11-07 11:39:13 +01:00
Daniele Rondina
6819a28f07 Add support to DOCKER_BUILDKIT on test 2020-11-07 11:37:58 +01:00
Daniele Rondina
24eb6eaef5 Fix test with docker buildkit 2020-11-07 11:37:58 +01:00
Daniele Rondina
c72565e019 Integrate tests for config protects with uninstall 2020-11-06 23:30:37 +01:00
Daniele Rondina
68bc8d4d27 ConfigProtect: Permit to obtain the list of files without initial / 2020-11-06 23:29:08 +01:00
Daniele Rondina
b24d335538 GetProtectFiles() is used also for tree tarball without specs 2020-11-06 23:00:37 +01:00
Ettore Di Giacinto
99bf9e291d Use LStat and attempt removing before bailing out on first failure 2020-11-06 21:34:56 +01:00
Daniele Rondina
51417ecb5d pkg/compiler/artifact.go: permit to support config protect with only annotation 2020-11-06 20:23:46 +01:00
Daniele Rondina
130eb8de1a Integrate config protection on uninstall too 2020-11-06 20:14:25 +01:00
Ettore Di Giacinto
5b5735266a Calculate provides for parallel solver too 2020-11-05 21:00:24 +01:00
Ettore Di Giacinto
984366d3a5 Consider provides during upgrades 2020-11-05 20:52:02 +01:00
Ettore Di Giacinto
25f69d4f1c Bump topsort 2020-11-03 17:20:52 +01:00
Ettore Di Giacinto
102a788c91 Revert "Revert "Stabilize ordering graph"" 
This reverts commit 2b23016a51.
 2020-11-02 15:43:35 +01:00
Ettore Di Giacinto
2b23016a51 Revert "Stabilize ordering graph" 
This reverts commit 940f553e1c.
 2020-11-02 15:43:15 +01:00
Ettore Di Giacinto
940f553e1c Stabilize ordering graph 
In this way when we order, we always return the same solution order in
case there are weak deps.

The following is optional - it doesn't change the "correctness" of the
solver results: We add an extra edge between deps that
share common dependendencies. This makes the link more stronger and
balances the graph so it doesn't show different results for the same query, as they
could be shuffled as don't have a direct connection.
 2020-11-02 14:30:41 +01:00
Ettore Di Giacinto
c3ef549673 Warn user only when required when uninstalling directories 2020-10-31 11:56:03 +01:00
Ettore Di Giacinto
0e764e525e Filter packages to install instead of looping solver result 2020-10-31 01:25:18 +01:00
Ettore Di Giacinto
f401e2b37f Add install benchmark test for solver 2020-10-30 22:20:08 +01:00
Ettore Di Giacinto
91dfb8ce3a Enhance CLI output 2020-10-30 19:15:04 +01:00
Ettore Di Giacinto
f6a4b634c1 Don't always walk all World() packages 
With this change the solver during install now considers only the part
of the tree which is required to calculate the solution, it doesn't
consider anymore World() as the search space.

The search space now is narrowed down to the packages that related to
the one which we are considering.

In this subset of changes we are also optimizing the Parallel solver
avoiding an useless loop.

This change boost overall performance on large datasets which don't
necessarly have relations touching the whole tree.
 2020-10-30 19:12:12 +01:00
Ettore Di Giacinto
39bc74fc73 Add boltDB test and fixup range over interface cast 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
99c59643a1 Add benchmarks tests 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
ffea4d8cf9 Fix priority constraint formula 
The parallel solver made the issue more visible, the constraints needed
to be less relaxed and needed to be exclusive so our candidate is looked
up at it first
 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
e459ddf470 Optimize BoltDB World() call 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
eb2c240e84 Adapt installer tests 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
95e640c9d0 Make solver type switchable 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
9f1a182eee Add tests and various fixes to parallel implementation 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
9a7d92b02e Make the parallel solver completely parallel in building formulas from dataset 2020-10-29 16:32:13 +01:00
Ettore Di Giacinto
c5ed36b2bd Sketch concourrent solver when building formulas 2020-10-29 16:32:13 +01:00
Daniele Rondina
6806103b3e installer: Start spinner of upgrade calculation 2020-10-25 14:01:45 +01:00
Ettore Di Giacinto
0aa0411c6e Bump copy dep and handle shallow symlinks 2020-10-19 17:58:43 +02:00
Daniele Rondina
c0cc9ec703 convert: Now use slot for category name 2020-10-18 19:58:15 +02:00
Daniele Rondina
07dff7f197 installer: log packages ignored on create-repo 2020-10-12 08:42:19 +02:00