Commit Graph

707 Commits

Author SHA1 Message Date
Ettore Di Giacinto
f0a34f1cf0 Enable NoLchown
This caused to drop file permissions
2021-03-18 10:48:23 +01:00
Ettore Di Giacinto
4f1e4c0b41 Switch to containerd when unpacking layers 2021-03-18 10:47:58 +01:00
Ettore Di Giacinto
ce349ca1b7 Se the path relative to the artifact when generating docker repositories 2021-03-16 14:46:46 +01:00
Ettore Di Giacinto
662742851a Generate backend bus events in the backends 2021-03-16 14:46:28 +01:00
Ettore Di Giacinto
f8ef1c0889 Print error when failing downloading docker image 2021-03-16 11:34:36 +01:00
Ettore Di Giacinto
268239a561 Propagate verify when serializing 2021-03-15 11:37:56 +01:00
Ettore Di Giacinto
0028dd3a92 Support content trust images and pull with authentication
Contact the notary server if ```--verify``` is specified (or `verify:
true` is enabled on the repo config) and verify if the image is signed,
use the returned value to pull the verified image.
2021-03-11 17:57:59 +01:00
Ettore Di Giacinto
675170939d Expose DownloadAndExtractDockerImage as a util
Create a util sub cmd to add all utils that are handy for development
and already present in the luet codebase. We expose in this case `luet
util unpack` to unpack a docker image without a docker daemon running.
2021-03-09 09:22:37 +01:00
Ettore Di Giacinto
42f5210764 Add DownloadOnly option
It allows to download only the packages, without
installing/upgrading/replacing them

Fixes #179
2021-03-07 11:39:59 +01:00
Ettore Di Giacinto
9eda81667b Use common ImageID() when computing final images
It also adds tests and strip invalid "+" character which is not
supported in docker image tags
2021-03-07 11:01:08 +01:00
Ettore Di Giacinto
c13a2174c4 Try to chown after copy 2021-03-06 18:49:14 +01:00
Ettore Di Giacinto
f07cf6c245 Add tests for file search 2021-03-06 18:49:14 +01:00
Ettore Di Giacinto
233429bbeb Allow to search by file
Also make possible to retrieve the artifact when searching for matches
between repositories list. This made possible to show the package list
when calling `luet search`.
2021-02-28 18:42:58 +01:00
Ettore Di Giacinto
749a4cb615 Add --backend-args
Allow to add arguments to the backend build arguments

Fixes #146
2021-02-22 13:49:29 +01:00
Ettore Di Giacinto
57e19c61e7 Start spinner before pulling docker artifacts 2021-02-22 11:54:09 +01:00
Ettore Di Giacinto
5ee1e28b9c Display informative message when pulling docker artifacts 2021-02-22 11:50:52 +01:00
Ettore Di Giacinto
21bd76af9c Uncomplicate runCommand and return command output
Fixes #189
2021-02-22 11:44:46 +01:00
Ettore Di Giacinto
b3e3abec8f Fixup spinner data race
Add spinner lock
2021-02-17 13:01:32 +01:00
Ettore Di Giacinto
92e73051a0 Initialize cached in singleton 2021-02-17 09:34:54 +01:00
Ettore Di Giacinto
2448f3175e Initialize cache db if empty 2021-02-15 18:53:37 +01:00
Ettore Di Giacinto
101df40eec
Merge pull request #183 from mudler/docker-debug-output
Add realtime output for building phase
2021-02-13 12:02:20 +01:00
Daniele Rondina
c22adb3a47 compiler: Move spinner at the low level 2021-02-13 09:28:54 +01:00
Ettore Di Giacinto
518fb16067 Add IsVirtual() to compile spec 2021-02-09 19:05:16 +01:00
Ettore Di Giacinto
4d9297e3da Be sure to copy exact folder structure when generating final images
'COPY *' has a different behavior than 'COPY .' - when regexes are
involved, COPY behave differently, by unpacking directory content to the
root.

Enhance unit test to cover the scenario as well
2021-02-09 18:27:53 +01:00
Ettore Di Giacinto
544895e051 Handle empty packages when pushing final images
We used to create dockerfiles blindly assuming there is content, but
that's not the case for virtual packages.

Due to https://github.com/moby/moby/issues/38039 we are forced for a
"unpleasant" workaround, as we can't create empty FROM scratch images
and export them.
2021-02-09 18:27:53 +01:00
Ettore Di Giacinto
fd80bb526e Add DirectoryIsEmpty 2021-02-09 16:51:10 +01:00
Daniele Rondina
c1fe3278fa backend: Add realtime output on building phase
The realtime output could be configured through
LUET_GENERAL__SHOW_BUILD_OUTPUT environment
variable or related config option or through
`--live-output` option.
2021-02-02 12:58:34 +01:00
Daniele Rondina
2854c68209 logger: Add ln option for writer log 2021-02-01 19:10:05 +01:00
David Cassany
18e9ce4557
Trim the Domain Name from cached image references
This commit removes the Domain Name, if any, from the cached image
reference before computing the image fingerprint. This way the same
image, if stored in some oter mirror, is still seen as the same one.

Fixes #158
2021-01-29 15:11:52 +01:00
Ettore Di Giacinto
685bbf46a6 Refactor common code while compiling regexes 2021-01-25 12:20:04 +01:00
Ettore Di Giacinto
d89225f37d Handle namedpipe copy
CopyFile relies on copy.Copy from https://github.com/otiai10/copy which
doesn't handle named pipes copy. Handle it here until
https://github.com/otiai10/copy/issues/47 is fixed.

This causes luet to hang while copying packages that have named pipes in
it.

Also invert compression argument for gzip, it causes slowliness.
2021-01-25 12:20:04 +01:00
Ettore Di Giacinto
55d34a3b40 Adapt artifact tests after delta changes 2021-01-24 20:07:33 +01:00
Ettore Di Giacinto
85b5c96bdd Promote to info building image messages
The user wants to know whats going on in this case. Image builds can
take up also long time
2021-01-24 19:09:09 +01:00
Ettore Di Giacinto
6f5f400765 Don't bail out if image doesn't exist locally
The backend will figure out if we have the image or not, otherwise will
atempt to pull if not there.

Skip retrieve integration test with img as its not supported.
2021-01-24 19:05:21 +01:00
Ettore Di Giacinto
be87861657 img: pull image if not locally present while extracting 2021-01-24 13:17:11 +01:00
Ettore Di Giacinto
be7b56bae3 Split ImageDefinitionToTar test
ImageDefinitionToTar it is not actually used by compiler code, but can
be handy from an API perspective, so we keep it.
2021-01-24 12:56:25 +01:00
Ettore Di Giacinto
eae2382764 unpackDelta needs a rootfs where to extract files from 2021-01-24 12:41:56 +01:00
Ettore Di Giacinto
7d11df3225 Simplify delta generation, and avoid two-pass with img backend
This changeset also drops --keep-exported-images, which is quite unused
and can be replaced with a plugin, or either by manually exporting the
resulting images.
2021-01-24 12:27:07 +01:00
Ettore Di Giacinto
b9f0ef1c55 Implement ImageExists in the img backend 2021-01-23 22:01:29 +01:00
Ettore Di Giacinto
715ee1db08 Add unit test for creating docker repositories 2021-01-22 19:59:26 +01:00
Ettore Di Giacinto
d5f70aea26 Add test for Artifact GetUncompressedName() 2021-01-22 19:15:54 +01:00
Ettore Di Giacinto
a037bc545b Add comment on required permission to unpack 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
d2e6409451 Add docker client unit test 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
a2df02e1bf Add comments on repository files #159 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
f0b8e4556e Make warning messages less prominent 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
161b5f40f7 Add file/line/function to debug messages 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
928c305ff1 Respect metadata fields about the tree filename
This caused to always ignore what was explictly asked during repo
creation
2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
a6e7a3059c Respect artifact extension when populating cache
This caused cache to not hit correctly
2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
9b2b877a53 Avoid to build images if already present 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
75fad993f3 Fixup repository revision bump 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
4b4c3a2e14 Adapt tests to new constructor changes 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
163f93067c Temporarly rework genuinetools/img code for pull/unpack without Docker 2021-01-22 16:55:51 +01:00
Ettore Di Giacinto
91ea2ed99f Make docker image repositories actually working
Several changes are included:
- Expose ensureDir in helpers, and call it in the Docker client. In
  other implementations that was handled by CopyFile behind the scenes,
  but that's not the case here
- Create accessor in Artifact to create Artifact objects from files.
  This is handy when we have to carry over downloaded package content
  into caches when artifacts are already verified
- Fix various issues around the imagePush flag, so now trees are pushed
  forcefully each time
- Take into consideration the real artifact name when pushing single
  files in the docker image. This behavior should be changed eventually,
  because single files which aren't repository packages now are in its
  own docker image, but we should have just one that brings the required
  metadata alltogether.
2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
b27b146b45 Refactor artifact Verify() 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
dbd37afced Add docker client #169 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
2f459c0469 Use only one docker image reference to push repository.
Instead of generating different images, which are harder to track and
clean, we generate a single image with various tags, corresponding to
the packages available in the repositories.

Tagging, and pushing separate images will be possible with the plugin
mechanism
2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
ad455edafc Allow to push images in create-repo
Add also the --force flag to allow image overwrite

Related to #169
2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
d9286a1a1e Download repository metadata with client DownloadFile, uniform downloads for Docker repositories 2021-01-22 16:54:19 +01:00
Ettore Di Giacinto
322fe72ef2 Generate repository metadata and packages for docker repository type
Drop image-repository on create-repo. In case of a docker repository, --output is the image reference to use.
Also restore default output build dir.

See also: #169
2021-01-22 16:53:52 +01:00
Ettore Di Giacinto
88b5576611 Expect full image name to GenerateFinalImage
We will re-use this method also when generating repository metadata
2021-01-18 12:26:22 +01:00
Ettore Di Giacinto
a1f4c28973 Add GenerateFinalImage to package artifacts
GenerateFinalImage generates a docker image from scratch with the
artifact content.

Related to #169
2021-01-18 12:08:47 +01:00
Ettore Di Giacinto
43b0b11028 Define a build context for backends 2021-01-18 11:06:54 +01:00
Ettore Di Giacinto
ecae2873d6 zstd extension suffix is zst, not zstd
Fixes #163
2021-01-12 15:52:34 +01:00
Ettore Di Giacinto
bd3e483f0f Be sure to cache packages with same fingerprint only once
this makes sure that we are fast to process also invalid trees
2021-01-12 10:30:31 +01:00
Ettore Di Giacinto
40fc948c6e Stabilize tests after changes
With BuildWorld() we get more results back (now we return the whole
model, including the false assertions).

Besides, now solving with BuildWorld() detects an invalid case:
when we supply a provided, the definitionDB shouldn't explictly supply
also the package that has to be provided. This would cause to 'shadow'
packages between repositories.

The test was invalid before, and shouldn't have contained A1. Moved the
test to Pending to inspect it further in subsequent dev iterations
2021-01-11 23:35:18 +01:00
Ettore Di Giacinto
186fa58ab0 Use BuildWorld() instead of BuildPartialWorld() in solver.Solve
We now have a stronger cache system while we pre-compute also RevDeps in
a hashmap, this makes now makes BuildWorld() much more performant.
2021-01-11 20:11:51 +01:00
Ettore Di Giacinto
7390623e40 Don't warn user of accepting license in case of uninstall
Closes #160
2021-01-07 10:36:57 +01:00
Ettore Di Giacinto
bd0d2765aa Mark executed finalizers at beginning
don't retry failing finalizers, but mark as executed right away
2021-01-04 17:04:20 +01:00
Ettore Di Giacinto
43f5b69c18 Let the build fail when depending on virtuals
This is currently not a valid use case. Virtuals are empty packages and
if the `build.yaml` is completely empty, nothing could depend on them.

Let's try to not be too smart and build the package image if a source
image is supplied, and fail hardly when we depend on a virtual in build
time.
2021-01-03 23:03:01 +01:00
Ettore Di Giacinto
457acd0d8a Add virtual packages support 2021-01-03 20:08:04 +01:00
Ettore Di Giacinto
a81d0bc3a3 Build assertions when swapping
When we are swapping packages, we do not run the solver to gather things
to install, but we trust the given list when calling computeInstall. In this case, the assertion
returned by computeInstall is empty, as we force l.Options.NoDeps.

This change generates the assertion list while calling computeSwap so
it's available later when we call ExecuteFinalizer.
2021-01-02 21:28:54 +01:00
Ettore Di Giacinto
bb48326039 Adapt solver tests after changes 2020-12-30 02:05:55 +01:00
Ettore Di Giacinto
dce8b52293 Use Conflicts() which already lists revdeps on failure 2020-12-30 01:17:31 +01:00
Ettore Di Giacinto
0652fce55e Update revdeps table while populating Cache
When we cycle, we don't necessarly have all the packages into the DB
yet.

With this change, luet annotates the reverse dependency without any version, and we try to
update revdeps table when new items gets added, by checking the version
required in the selector.

Thanks to @joostruis for noticing the issue
2020-12-30 01:12:35 +01:00
Ettore Di Giacinto
38c9540a1d Use DB copy in GetRevdeps in BoltDB 2020-12-30 01:12:09 +01:00
Ettore Di Giacinto
90278a034b Use ConflictsWith to check conflicts when uninstalling packages 2020-12-29 23:43:39 +01:00
Ettore Di Giacinto
55ab1894e9 Add unit test for Uninstall in Installer 2020-12-29 22:58:03 +01:00
Niklas Engvall
bfbcb81210
Set import name for zstd 2020-12-29 22:22:53 +01:00
Ettore Di Giacinto
062e75bc25 Add unit test for Uninstall without full 2020-12-29 22:13:26 +01:00
Ettore Di Giacinto
b81ce66914 Reduce download verbosity 2020-12-27 20:21:05 +01:00
Ettore Di Giacinto
f871111e50 Collect errors from finalizer runs
Instead of failing and depend on the --force flag, always execute
finalizer and collect errors to determine if install was successfull or
not
2020-12-25 10:35:09 +01:00
Ettore Di Giacinto
736c9470cf Add db copy and clone 2020-12-19 17:45:50 +01:00
Ettore Di Giacinto
e52bc4f2b2 Refactor: get systemdb from config, which knows which one to load 2020-12-19 17:23:59 +01:00
Ettore Di Giacinto
96e877fc0b Allow uninstall to take multiple packages
And treat those as a list, instead of each single of them
2020-12-19 17:16:58 +01:00
Ettore Di Giacinto
525bfb5ebf Respect --nodeps when calling Swap from the public interface 2020-12-19 15:26:18 +01:00
Ettore Di Giacinto
f4e2f32aff Return candidate not found when appropriate 2020-12-19 14:57:42 +01:00
Ettore Di Giacinto
7cf650a8f6 Break Swap in computeSwap() and display uninstall dialog only when asked 2020-12-19 14:55:59 +01:00
Ettore Di Giacinto
2b6fe2baa1 Add luet build --wait
It allows to wait for intermediate images to be available instead of
building all of them
2020-12-18 23:19:18 +01:00
Ettore Di Giacinto
f0fae82ad9 Add experimental zstd support
Closes #97
2020-12-18 21:24:01 +01:00
Ettore Di Giacinto
1c9b821058 Drop unneeded if 2020-12-18 00:50:20 +01:00
Ettore Di Giacinto
0e21548bc0 Lookup uninstall and Install in installer.Swap
In this way we resolve selectors from user inputs
2020-12-18 00:49:51 +01:00
Ettore Di Giacinto
39c8895f80 Return provided if selector isn't a range in boltdb 2020-12-17 23:17:40 +01:00
Ettore Di Giacinto
ce169f49af If provided isn't a selector, it means we don't have to return a range
Add also more tests about provides
2020-12-16 22:17:34 +01:00
Ettore Di Giacinto
8e1a457bf1 Check if we have to pull images before generating delta
As we might skip building entirely, it's possible that the image is not
there yet, so we check if have to pull it or not
2020-12-15 17:01:56 +01:00
Ettore Di Giacinto
70f05f41e8 Check only if package image exists
We don't need to look after the builder image as its optional. In this
way we can also reduce the compiler options, as we don't require a
--clean flag anymore. --only-target-package is sufficient to determine
what we can skip and how.
2020-12-14 18:41:39 +01:00
Ettore Di Giacinto
ef034d87b0 Detect if images are available if we don't have to generate a Package
While building, if we aren't doing a clean build, we scan now to see if
images are available and we skip, in case we don't find a metadata
already.
2020-12-14 18:32:32 +01:00
Ettore Di Giacinto
265e2371b4 Add ArtifactNode to test, now we get the gen Dockerfile in the diff 2020-12-12 16:04:54 +01:00
Daniele Rondina
78442c91fc events: Review description of build_artifact events 2020-12-12 16:03:40 +01:00
Ettore Di Giacinto
d97e606a31 Adapt fixtures and tests 2020-12-12 12:10:24 +01:00
Ettore Di Giacinto
95da20e366 Context files are immutable 2020-12-12 11:55:25 +01:00