Commit Graph

180 Commits

Author SHA1 Message Date
Ettore Di Giacinto
e70a543f42 ⚙️ Add ability to build from Dockerfiles directly 2022-04-27 21:39:55 +02:00
mudler
77c7eab1ee ⚙️ Fixup runtime tree detection during join images
We need to hook up the runtime DB in order to query the correct part of
the tree
2022-04-13 23:28:44 +02:00
Ettore Di Giacinto
c6170fabd6
⚙️ Enhancements to final images building
This commit is multi-fold and impacts several areas:

- Don't re-generate final artifact locally if already present while
building with `requires_final_images`.
- Expose to CLI a way to build final images without pushing them.
- The packages listed with `requires_final_images` now are evaluated by
  the solver so the full deptree is took into account

Fixes: https://github.com/mudler/luet/issues/294
2022-04-13 18:04:50 +02:00
Ettore Di Giacinto
6f77fa2b3a ♻️ Use sprig instead of helm
This alleviates luet build dependency graph while it simplifies template
logic too.
2022-01-27 17:42:53 +01:00
Ettore Di Giacinto
ba0d625b5f 🎨 Port package to types
This refactors DefaultPackage into types.Package and gets rid of the
interface. This is a preceeding for a follow up where accessors will be
removed from the code.
It also does several cleanup, so we get rid also of some unneeded
dependencies.
2022-01-08 00:05:20 +01:00
Ettore Di Giacinto
519058f13c 🔧 Do not store file list of hidden packages 2022-01-07 00:02:22 +01:00
Ettore Di Giacinto
c98f427156 🎨 Introduce contextualized logging
This commit is multi-fold as it also refactors internally context and logger
as interfaces so it is easier to plug luet as a library externally.

Introduces a garbage collector (related to #227) but doesn't handle yet
parallelism.

Closes #265
2021-12-21 21:54:14 +01:00
Ettore Di Giacinto
0a90129e34 🔧 Restore tree imglist hash output
Fixes #271
2021-12-15 18:38:47 +01:00
Ettore Di Giacinto
8e16d3abd3 🔧 Use ImageID for generating dockerfile names
It is safer, and plays better with buildx
2021-12-06 21:46:15 +01:00
Ettore Di Giacinto
da85a7306f 🔧 Consistently use Tempdir in compiler 2021-12-04 21:48:43 +01:00
Ettore Di Giacinto
79e98af604 Handle error if we can't generate a compilation spec from a package 2021-11-27 21:12:14 +01:00
Ettore Di Giacinto
a02ab16510 Don't load requires while parsing compilespec that consume final images
When depending on those package otherwise we try to compile the full
tree instead of reconstrucing the image which is result of a join while
keeping the revdep tree invariate
2021-11-25 14:18:15 +01:00
Ettore Di Giacinto
a2d91a2aee fixup: sanitize metadata images name 2021-11-15 21:10:15 +01:00
Ettore Di Giacinto
702a9f17db Drop code which is called already by containerd
Drop also direct xattrs handling
2021-11-10 16:28:22 +01:00
Ettore Di Giacinto
1e78570c50 Allow to push final images while compiling
Add --push-final-images, --push-final-images-repository and
--push-final-images-force to luet build.

--push-final-images enables pushing final artifact during build. Each
package built will be packed and pushed to the final repository
specified with --push-final-images-repository. By default if no
final-repository is specified and pushing final images is enabled will
default to the cache repository.

--push-final-images-force allows to force-push final images even if
there are already available on the specified registry
2021-11-05 23:03:29 +01:00
Ettore Di Giacinto
c7f9708f90 Add CreateTar to image API
Add api call which uses go-container registry to create OCI images from
standard tar archives.

Consume new API when generating final images instead of docker building them
and adapts/add tests as necessary.

This change now allows to carry over xattrs to final images.

Fixes #266
2021-10-29 10:35:03 +02:00
Ettore Di Giacinto
262d09dfbc Lower message levels 2021-10-26 16:56:49 +02:00
Ettore Di Giacinto
35fcd868ee Switch to ondisk also when unpacking FS
From benchmarks it seems to be still faster. Add a note for a future
improvement
2021-10-26 16:56:49 +02:00
Ettore Di Giacinto
aea3cdff8d Use ondisk reference for deltas 2021-10-26 16:56:49 +02:00
Ettore Di Giacinto
daa9eb98d2 Walk destination only once when computing delta
Avoid the double pass by constructing the list on the fly
2021-10-26 16:56:49 +02:00
Ettore Di Giacinto
f6bb7a9405 Make sure to pull images before generating artifacts
Fixes #262
2021-10-25 23:56:38 +02:00
Ettore Di Giacinto
c220eac061 Move bus to api/core 2021-10-24 19:07:41 +02:00
Ettore Di Giacinto
6ff22d923c Make default build dir over context temp 2021-10-24 18:26:30 +02:00
Ettore Di Giacinto
37a9a3ef55 use containerd to uncompress 2021-10-24 18:26:30 +02:00
Ettore Di Giacinto
ebbb3aad27 Use API also when pulling from helpers used in client 2021-10-24 18:26:30 +02:00
Ettore Di Giacinto
ad489c2157 tests: pull image before running 2021-10-24 18:26:30 +02:00
Ettore Di Giacinto
acd685b927 Extract with new image API 2021-10-24 18:26:25 +02:00
Ettore Di Giacinto
6a9f19941a Add crane-based methods for extraction
- create a new api package to encapsulate image manipulation
- use new api method to calculate delta

Fixes #258
Fixes #204
Fixes #90
2021-10-24 18:26:08 +02:00
Ettore Di Giacinto
6b45b1d61c ux: rework displaying of success messages 2021-10-22 17:55:38 +02:00
Ettore Di Giacinto
a1c669d3ae Define context for scoped operation across core types
It holds necessary state plus additional information relative to the
context which we are being run to (e.g. if we are in a terminal or not).
Besides in the future we can use it also as a contextual logger to
provide more smart logging capabilities.

This also replace the general global configuration instance that
previously was share between the core components.
2021-10-21 23:58:24 +02:00
Ettore Di Giacinto
fe14d56afe Massive UX rewrite
- Ditch multiple libraries for progressbar, spinner, colors and replace
  with pterm
- Detect when running on terminal and disable automatically spinner
- Add support for multiple progress bars
- Huge rewrite of the configuration part. No more crazy stuff with viper
  CLI commands now correctly overrides default config file as expected
- Limit banner to be displayed on relevant parts

Fixes #211
Fixes #105
Fixes #247
Fixes #233
2021-10-21 23:58:00 +02:00
Ettore Di Giacinto
5b4e930fc3 Do implement a real cache
Instead of merely storing files into disk use a real cache.

This also makes possible finally to reference artifacts in the cache with the
package checksum, which solves the cache hit checksum failures we had
previously.
2021-10-19 21:25:01 +02:00
Ettore Di Giacinto
e764b1cd29 Add CLI helpers and BuildTree heuristic
The BuildTree implementation was in parallel-build-tools, which actually makes
sense to share in the compiler as can be used as another way to trigger
builds.

The CLI helpers were split in several CIs
implementations. This moves a new layout where we expose several
packages, as we will refactor things slowly.
2021-10-15 21:14:48 +02:00
Ettore Di Giacinto
097e310d3a Drop join from compilation specs
Adapt also hash and integration tests
2021-10-10 00:28:54 +02:00
Ettore Di Giacinto
a348fd4835 Replace Yaml with YAML in function, add debug output when extracting runtime data 2021-09-17 14:46:32 +02:00
Daniele Rondina
540e8151ad
compiler: Speedup & troubleshooting debug messages (#240)
* compiler: Speedup stripFromRootfs and add debug log

* compiler: on stripFromRootfs show files removed with debug
2021-08-11 10:06:52 +02:00
Ettore Di Giacinto
4d6cccb2fa Give explaination when formulas are unsat
Fixes #168

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-08-07 14:46:05 +02:00
Ettore Di Giacinto
33b1c63815 Allow to have shared templates across packages
This changeset allows to have shared templates in a static folder
"templates" present in each luet tree. If the directory is present, it
gets scanned and templated accordingly on top of each package. This
allows to use such folder to store custom blocks to share between
packages.

This is still experimental and subject to change, this is just a first
pass version to provide the feature. It needs to be refined still as it
would be more elegant to use the helm engine properly and map our
structure to the engine instead of adapting it roughly.

Fixes #224
2021-08-04 16:35:28 +02:00
Ettore Di Giacinto
cb032dc714 Allow to manipulate requires/conflicts/provides
This allows to manipulate requires, conflicts and provides with
templating in build time.

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-08-03 13:27:44 +02:00
Ettore Di Giacinto
2efb17a06c Add specific field to resolve final images
This commit starts deprecation of `join` keyword in favor of
`requires_final_images` as boolean in the compilation spec.

The change is driven by two reasons: syntax and guarantee unique hashes.

- the hashtree when computing a hash it analizes the requires field of
  each spec, ignoring the join field
- the join field doesn't add much value. Having it separate suggests
  that a spec can contain both `requires` and `join`, but that's not
  actually true. We just act differently on the same list.

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-07-09 12:37:38 +02:00
Itxaka
1bd4d520a4
Add 2 new events for image unpacking (#226)
* Reduce possibility of circular dependency

Just by adding an import for bus to anything in the helper dir, we would
run into a circular dependency due to how things are structured. That
means that we cannot set any events for unpacking or docker helper
pulling an image.

This commit tries to work around this by doing several things.
 - Remove full imports of the helper module by segmentating some modules
   into their own submodule, like docker or match so just using a small match
   function doesnt bring the whole module
 - Removing a simple function to check if a dir exists from importing
   the full helper module and instead write the function (5 lines)
 - Using logrus in the bus module instead of logger, which avoids a
   circular dependency

Signed-off-by: Itxaka <igarcia@suse.com>

* Add two new events for unpacking an image

Both pre and post unpacking an image

Signed-off-by: Itxaka <igarcia@suse.com>
2021-06-01 16:43:31 +02:00
Ettore Di Giacinto
92ea69a2b9 Don't always generate artifact
Distinguish rather if we should generate artifact of the deps of a
package or not, and ensure that we just force artifact generation for
targets when computing the join image.

Fixes #217
2021-05-25 13:28:14 +02:00
Ettore Di Giacinto
06fa8b1c87 fixup: jointag index 2021-05-24 22:13:27 +02:00
Ettore Di Giacinto
ff153f367f fixup: enhance output of copy tag when cycling over packages 2021-05-24 21:35:22 +02:00
Ettore Di Giacinto
459676397c Enhance multi-stage output 2021-05-24 19:40:39 +02:00
Ettore Di Giacinto
93057fbf6d Respect PackageTargetOnly during multi-stage copy image generation 2021-05-24 19:40:17 +02:00
Ettore Di Giacinto
5e1a7c50df fixup: display hash of join image 2021-05-23 18:22:14 +02:00
Ettore Di Giacinto
0ceaf09615 fixup: propagate outputpath 2021-05-23 17:36:09 +02:00
Ettore Di Giacinto
27c2e3c51f fixup: make sure the outputdir exists 2021-05-21 22:25:18 +02:00
Ettore Di Giacinto
6344e47eb3 fixup: resolve multistage and join also when building deps 2021-05-21 19:47:23 +02:00