Commit Graph

91 Commits

Author SHA1 Message Date
Itxaka
03e72653c7
Fill referenceID on repo sync (#300)
Signed-off-by: Itxaka <igarcia@suse.com>
2022-06-06 17:36:19 +02:00
Ettore Di Giacinto
e70a543f42 ⚙️ Add ability to build from Dockerfiles directly 2022-04-27 21:39:55 +02:00
mudler
cca1a6dbc4
⚙️ Annotate build dir when building against system repo 2022-04-14 09:33:44 +02:00
Ettore Di Giacinto
114d4899f6 🐛 Update yaml to v3
The recent switch to mergo throws in a yaml version which is affected
by https://github.com/go-yaml/yaml/issues/139, and caused
https://github.com/rancher-sandbox/cOS-toolkit/issues/1189.

Updating to yaml.v3 where it was affected.

Note mergo still uses v2, so this can be problematic on other areas
https://github.com/imdario/mergo/issues/206.
2022-03-21 14:58:41 +01:00
Ettore Di Giacinto
4943ed6aef ⚙️ Resolve templates folder when syncing repositories
This fixes #284
2022-01-09 14:21:42 +01:00
Ettore Di Giacinto
ba0d625b5f 🎨 Port package to types
This refactors DefaultPackage into types.Package and gets rid of the
interface. This is a preceeding for a follow up where accessors will be
removed from the code.
It also does several cleanup, so we get rid also of some unneeded
dependencies.
2022-01-08 00:05:20 +01:00
Ettore Di Giacinto
9dcaeb0870 🔧 Defer write repository synctime 2021-12-28 12:06:09 +01:00
Ettore Di Giacinto
c98f427156 🎨 Introduce contextualized logging
This commit is multi-fold as it also refactors internally context and logger
as interfaces so it is easier to plug luet as a library externally.

Introduces a garbage collector (related to #227) but doesn't handle yet
parallelism.

Closes #265
2021-12-21 21:54:14 +01:00
Ettore Di Giacinto
20d01e43c7 🎨 Update repos automatically only if out-of-sync
Fixes #274
Fixes #212
2021-12-18 16:32:03 +01:00
Ettore Di Giacinto
37f4289cdd 🔧 Allow to specify a snapshot ID #276 2021-12-17 15:41:17 +01:00
Ettore Di Giacinto
52ad2b5cfa Fixup config protect 2021-10-24 18:26:30 +02:00
Ettore Di Giacinto
37a9a3ef55 use containerd to uncompress 2021-10-24 18:26:30 +02:00
Ettore Di Giacinto
5eb5a42bf7 Generate snapshot and push it along
Fixes #260
Fixes #223
2021-10-22 17:55:08 +02:00
Ettore Di Giacinto
0bacdc75f2 Allow clients to pick a specific referenceID
Related to #260
2021-10-22 12:39:36 +02:00
Ettore Di Giacinto
a1c669d3ae Define context for scoped operation across core types
It holds necessary state plus additional information relative to the
context which we are being run to (e.g. if we are in a terminal or not).
Besides in the future we can use it also as a contextual logger to
provide more smart logging capabilities.

This also replace the general global configuration instance that
previously was share between the core components.
2021-10-21 23:58:24 +02:00
Ettore Di Giacinto
5b4e930fc3 Do implement a real cache
Instead of merely storing files into disk use a real cache.

This also makes possible finally to reference artifacts in the cache with the
package checksum, which solves the cache hit checksum failures we had
previously.
2021-10-19 21:25:01 +02:00
Ettore Di Giacinto
2eeb464946 Make repositories arch-aware
Introduce an arch field that can be used to filter repositories based on
the local architecture.

If arch is provided and matching with the current GOARCH then the
repository is enabled
2021-10-19 14:19:53 +02:00
Ettore Di Giacinto
fc45eae80a create-repo: annotate runtime definition in artifacts 2021-09-17 14:30:06 +02:00
Ettore Di Giacinto
b73ac21004 create-repo: don't inherit build requirements in runtime 2021-09-17 12:06:39 +02:00
Ettore Di Giacinto
14914f3c8e Update tests for including packages in tree from metadata
Also switches GenerateRepository to functional interface to allow
more parametrization
2021-09-16 12:59:54 +02:00
Ettore Di Giacinto
e4fff77d43 Source packages metadata to update repository tree
This allows to generate trees with `create-repo` by just having
the metadata files, making tree(s) an optional requirement.
2021-09-16 10:56:09 +02:00
Itxaka
1bd4d520a4
Add 2 new events for image unpacking (#226)
* Reduce possibility of circular dependency

Just by adding an import for bus to anything in the helper dir, we would
run into a circular dependency due to how things are structured. That
means that we cannot set any events for unpacking or docker helper
pulling an image.

This commit tries to work around this by doing several things.
 - Remove full imports of the helper module by segmentating some modules
   into their own submodule, like docker or match so just using a small match
   function doesnt bring the whole module
 - Removing a simple function to check if a dir exists from importing
   the full helper module and instead write the function (5 lines)
 - Using logrus in the bus module instead of logger, which avoids a
   circular dependency

Signed-off-by: Itxaka <igarcia@suse.com>

* Add two new events for unpacking an image

Both pre and post unpacking an image

Signed-off-by: Itxaka <igarcia@suse.com>
2021-06-01 16:43:31 +02:00
Ettore Di Giacinto
9f3e7fd0b2 PackageIndex is in repository metadata file only
This was a regression in those changeset, don't replicate the info, and
at the same time, write the artifact path containing the name of the
file only (excluding the path)
2021-04-16 15:12:51 +02:00
Ettore Di Giacinto
c27d4d258e Allow create-repo to source trees from remote repositories
This makes possible to create the repositorie from external ones,
It also required to address #26.

Fixes #26
2021-04-16 14:01:13 +02:00
Ettore Di Giacinto
fadd5a10a3 Load trees separately while reconstructing spectrees 2021-04-16 14:00:11 +02:00
Ettore Di Giacinto
7ba7add2a8 Allow to pull specfiles from published repositories
- Interpolates values from the repositories compilespec if present
- Automatically merge cache images coming from specified repository when
  necessary

Fixes #194
2021-04-16 13:58:51 +02:00
Ettore Di Giacinto
57c769b4a5 Refactor compiler and annotate buildoptions into compiler metadata
This allows to later pick up values used during build of each package
2021-04-16 13:57:54 +02:00
Ettore Di Giacinto
c022c75239 Write BuildTree as an additional repository file
Also split Docker repository generation a bit more for readability
2021-04-16 13:56:51 +02:00
Ettore Di Giacinto
3250d63072 Don't merge install and compilation DB. Use a temporary DB for generation 2021-04-16 13:56:51 +02:00
Ettore Di Giacinto
b8961be793 Add accessor to embed custom files to a repository while writing
Split up into methods so it's easier to add custom files
2021-04-16 13:56:45 +02:00
Ettore Di Giacinto
036b5c08c6 Split up repository generators 2021-04-08 15:24:32 +02:00
Ettore Di Giacinto
ce349ca1b7 Se the path relative to the artifact when generating docker repositories 2021-03-16 14:46:46 +01:00
Ettore Di Giacinto
268239a561 Propagate verify when serializing 2021-03-15 11:37:56 +01:00
Ettore Di Giacinto
0028dd3a92 Support content trust images and pull with authentication
Contact the notary server if ```--verify``` is specified (or `verify:
true` is enabled on the repo config) and verify if the image is signed,
use the returned value to pull the verified image.
2021-03-11 17:57:59 +01:00
Ettore Di Giacinto
9eda81667b Use common ImageID() when computing final images
It also adds tests and strip invalid "+" character which is not
supported in docker image tags
2021-03-07 11:01:08 +01:00
Ettore Di Giacinto
233429bbeb Allow to search by file
Also make possible to retrieve the artifact when searching for matches
between repositories list. This made possible to show the package list
when calling `luet search`.
2021-02-28 18:42:58 +01:00
Ettore Di Giacinto
a2df02e1bf Add comments on repository files #159 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
928c305ff1 Respect metadata fields about the tree filename
This caused to always ignore what was explictly asked during repo
creation
2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
9b2b877a53 Avoid to build images if already present 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
75fad993f3 Fixup repository revision bump 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
91ea2ed99f Make docker image repositories actually working
Several changes are included:
- Expose ensureDir in helpers, and call it in the Docker client. In
  other implementations that was handled by CopyFile behind the scenes,
  but that's not the case here
- Create accessor in Artifact to create Artifact objects from files.
  This is handy when we have to carry over downloaded package content
  into caches when artifacts are already verified
- Fix various issues around the imagePush flag, so now trees are pushed
  forcefully each time
- Take into consideration the real artifact name when pushing single
  files in the docker image. This behavior should be changed eventually,
  because single files which aren't repository packages now are in its
  own docker image, but we should have just one that brings the required
  metadata alltogether.
2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
2f459c0469 Use only one docker image reference to push repository.
Instead of generating different images, which are harder to track and
clean, we generate a single image with various tags, corresponding to
the packages available in the repositories.

Tagging, and pushing separate images will be possible with the plugin
mechanism
2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
ad455edafc Allow to push images in create-repo
Add also the --force flag to allow image overwrite

Related to #169
2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
d9286a1a1e Download repository metadata with client DownloadFile, uniform downloads for Docker repositories 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
322fe72ef2 Generate repository metadata and packages for docker repository type
Drop image-repository on create-repo. In case of a docker repository, --output is the image reference to use.
Also restore default output build dir.

See also: #169
2021-01-22 16:53:52 +01:00
Ettore Di Giacinto
f4e2f32aff Return candidate not found when appropriate 2020-12-19 14:57:42 +01:00
Ettore Di Giacinto
9423b7c1e3 Add image build events, and add luet replace
Enhance also some commands descriptions
2020-12-02 18:24:35 +01:00
Ettore Di Giacinto
0e46e763d5 Move bus implementation to a separate repo, hook to events in luet 2020-11-13 18:25:44 +01:00
Daniele Rondina
07dff7f197 installer: log packages ignored on create-repo 2020-10-12 08:42:19 +02:00
Ettore Di Giacinto
cc21e6fa5e
Respect user-defined repository naming 2020-05-24 12:16:02 +02:00