Compare commits

...

57 Commits

Author SHA1 Message Date
Ettore Di Giacinto
86bd6c5fc0 Tag 0.17.4 2021-08-04 11:55:23 +02:00
Ettore Di Giacinto
658612fcf3 Check file conflicts also on packages that are going to be swapped out
Fixes #237
2021-08-04 10:43:12 +02:00
Ettore Di Giacinto
7128c88da6 Tag 0.17.3 2021-08-03 18:15:40 +02:00
Ettore Di Giacinto
74402fae81 Re-organize common CLI code
Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-08-03 16:48:34 +02:00
Ettore Di Giacinto
9d1594c036 Check if the system-target path supplied is absolute
docker.Untar
(https://github.com/mudler/luet/blob/master/vendor/github.com/docker/docker/pkg/archive/archive.go#L942) requires absolute paths.
We didn't do any input validation before, assuming the path passed by
were absolute since they were coming from YAML configuration files, now
that this is not the truth anymore we need to sanitize the input.

With this change we check if the given path is absolute or relative, if
it's relative we calculate the absolute path and use it in place.
2021-08-03 16:22:59 +02:00
Ettore Di Giacinto
75906c4198 Add missing check for error 2021-08-03 13:36:38 +02:00
Ettore Di Giacinto
cb032dc714 Allow to manipulate requires/conflicts/provides
This allows to manipulate requires, conflicts and provides with
templating in build time.

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-08-03 13:27:44 +02:00
Ettore Di Giacinto
2c7e495fa1 Add http timeout to grab
grab.NewClient() doesn't set a specific timeout
https://github.com/cavaliercoder/grab/blob/v2.0.0/client.go#L37 even if
the project advertize "default sane settings".

We default to 30, and allow to set it up with HTTP_TIMEOUT

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-08-03 11:57:44 +02:00
Ettore Di Giacinto
db8bf2b85e Tag 0.17.2 2021-07-30 13:33:29 +02:00
Ettore Di Giacinto
5eb586ddb0 Speedup conflict check by caching in memory files 2021-07-30 12:40:55 +02:00
Ettore Di Giacinto
f9747cdf87 Tag 0.17.1 2021-07-29 15:07:32 +02:00
Ettore Di Giacinto
becac7d853 Add tests 2021-07-29 11:03:02 +02:00
Ettore Di Giacinto
5aa5bffb48 Add reinstall command
Fixes #51
2021-07-29 11:03:02 +02:00
Ettore Di Giacinto
9aa3159787 Check for file conflicts before install
Fixes #88
2021-07-29 11:03:02 +02:00
Ettore Di Giacinto
9cb6e65bb6 Don't display banner on cleanup 2021-07-29 11:03:02 +02:00
Ettore Di Giacinto
92b243d7aa Add git-chglog template and config 2021-07-12 10:26:22 +02:00
Ettore Di Giacinto
29ec19a8a1 Tag 0.17.0 2021-07-11 11:55:35 +02:00
Ettore Di Giacinto
440e07c418 Display version banner only on specific commands
Avoid cumbersome text on areas that can be parsed by machines

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-07-11 11:00:59 +02:00
Ettore Di Giacinto
acf74f5896 config_protect: don't protect files during uninstall by default
Otherwise during uninstall we would retain the files which are
protected. We introduced a specific flag for it to pass during
uninstall, but for now we choose semplicity and expected default first.

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-07-11 10:40:36 +02:00
Ettore Di Giacinto
1ee1894ffa ci: split integration/unit tests 2021-07-09 17:15:17 +02:00
Ettore Di Giacinto
c8573f9535 Adapt hash tests
Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-07-09 17:15:17 +02:00
Ettore Di Giacinto
76b70ebeb4 Add dirhash to package signatures
While this breaks current hashing, it ties also the spec content to the
hash, in this way if we change something in the spec folder, it breaks
the hashing for the package.

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-07-09 12:37:38 +02:00
Ettore Di Giacinto
2efb17a06c Add specific field to resolve final images
This commit starts deprecation of `join` keyword in favor of
`requires_final_images` as boolean in the compilation spec.

The change is driven by two reasons: syntax and guarantee unique hashes.

- the hashtree when computing a hash it analizes the requires field of
  each spec, ignoring the join field
- the join field doesn't add much value. Having it separate suggests
  that a spec can contain both `requires` and `join`, but that's not
  actually true. We just act differently on the same list.

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-07-09 12:37:38 +02:00
Ettore Di Giacinto
64ab3711ca Display version and license on each run
Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-07-09 12:25:55 +02:00
Ettore Di Giacinto
eb5d7ba35b Merge branch 'master' into develop 2021-07-09 12:25:04 +02:00
Daniele Rondina
b6b91cfd7a compiler: Avoid generation of delta if there are only prelude steps (#232) 2021-07-09 12:23:32 +02:00
Daniele Rondina
4d8a9a544b cmd/database/create: Avoid creation of the same package multiple times (#222) 2021-06-25 13:09:32 +02:00
Ettore Di Giacinto
654b5b48cd Tag 0.16.7 2021-06-17 07:55:01 +02:00
Ettore Di Giacinto
92e18d5782 Support priv/unpriv image extraction
Optionally add back privileged extraction which can be enabled with
LUET_PRIVILEGED_EXTRACT=true

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-06-16 23:30:28 +02:00
Ettore Di Giacinto
8780e4f16f Unpack using our moby fork
Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-06-16 19:16:02 +02:00
Ettore Di Giacinto
a7b4ae67c9 Tag 0.16.6 2021-06-04 16:23:51 +02:00
Ettore Di Giacinto
68edfd58e7 Keep emitting plugin events 2021-06-04 12:08:57 +02:00
Ettore Di Giacinto
0658020c60 Update go.mod and vendor 2021-06-04 11:15:10 +02:00
Ettore Di Giacinto
c3b552103f Return a specific struct for the image
And drop imgworker
2021-06-04 11:09:41 +02:00
Darren Shepherd
796967cc9d Allow luet install to work inside a non-privileged container
This switches from using the containerd snapshotter to go-containerregistry
library which requires no additional privileges beyond root file system
access.

Signed-off-by: Ettore Di Giacinto <mudler@sabayon.org>
2021-06-04 10:29:57 +02:00
Ettore Di Giacinto
5cccc34f32 Tag 0.16.5 2021-06-01 22:54:10 +02:00
Ettore Di Giacinto
5ef1d04055 Create CONTRIBUTING.md 2021-06-01 17:13:23 +02:00
Itxaka
1bd4d520a4 Add 2 new events for image unpacking (#226)
* Reduce possibility of circular dependency

Just by adding an import for bus to anything in the helper dir, we would
run into a circular dependency due to how things are structured. That
means that we cannot set any events for unpacking or docker helper
pulling an image.

This commit tries to work around this by doing several things.
 - Remove full imports of the helper module by segmentating some modules
   into their own submodule, like docker or match so just using a small match
   function doesnt bring the whole module
 - Removing a simple function to check if a dir exists from importing
   the full helper module and instead write the function (5 lines)
 - Using logrus in the bus module instead of logger, which avoids a
   circular dependency

Signed-off-by: Itxaka <igarcia@suse.com>

* Add two new events for unpacking an image

Both pre and post unpacking an image

Signed-off-by: Itxaka <igarcia@suse.com>
2021-06-01 16:43:31 +02:00
Daniele Rondina
b12c7678d4 get_luet_root.sh: Fix installation on alpine/busybox env (#218)
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
2021-05-27 17:17:49 +02:00
Ettore Di Giacinto
32a99a4a49 Update readme with badges and a small logo 2021-05-26 17:57:41 +02:00
Ettore Di Giacinto
56e9c6f82e Tag 0.16.4 2021-05-25 14:55:55 +02:00
Ettore Di Giacinto
92ea69a2b9 Don't always generate artifact
Distinguish rather if we should generate artifact of the deps of a
package or not, and ensure that we just force artifact generation for
targets when computing the join image.

Fixes #217
2021-05-25 13:28:14 +02:00
Ettore Di Giacinto
838899aa83 Respect create-repo --force-push with metadata files
Don't force push package metadata with docker repository.

We were pushing regardless .metadata.yaml files even if packages were
not, also when calling create-repo without --force-push.
2021-05-25 12:13:23 +02:00
Ettore Di Giacinto
76695b2fc8 docker: check during extraction if image exists and pull if needed 2021-05-25 11:04:06 +02:00
Ettore Di Giacinto
5c84e5b0a7 Tag 0.16.3 2021-05-24 22:23:57 +02:00
Ettore Di Giacinto
06fa8b1c87 fixup: jointag index 2021-05-24 22:13:27 +02:00
Ettore Di Giacinto
ff153f367f fixup: enhance output of copy tag when cycling over packages 2021-05-24 21:35:22 +02:00
Ettore Di Giacinto
459676397c Enhance multi-stage output 2021-05-24 19:40:39 +02:00
Ettore Di Giacinto
93057fbf6d Respect PackageTargetOnly during multi-stage copy image generation 2021-05-24 19:40:17 +02:00
Ettore Di Giacinto
5e1a7c50df fixup: display hash of join image 2021-05-23 18:22:14 +02:00
Ettore Di Giacinto
0ceaf09615 fixup: propagate outputpath 2021-05-23 17:36:09 +02:00
Ettore Di Giacinto
0dc78ebe41 Tag 0.16.2 2021-05-22 14:49:58 +02:00
Ettore Di Giacinto
27c2e3c51f fixup: make sure the outputdir exists 2021-05-21 22:25:18 +02:00
Ettore Di Giacinto
e83f600ed3 Tag 0.16.1 2021-05-21 21:28:33 +02:00
Ettore Di Giacinto
6344e47eb3 fixup: resolve multistage and join also when building deps 2021-05-21 19:47:23 +02:00
Ettore Di Giacinto
8b1c5558b2 Tag 0.16.0 2021-05-21 16:10:28 +02:00
Ettore Di Giacinto
c277ac0f94 Add join keyword to generate parent image from final artifacts
A new keyword `join` is introduced to generate the parent image. It
takes precedence over a `requires` or a `image` already defined in a
spec.

It will generate all the artifacts from the packages listed and join
them in a single image which will be used as parent for the package
build process.

This is a change which invalidates priorly generated hashes.

Fixes #173
2021-05-21 14:52:48 +02:00
108 changed files with 3058 additions and 878 deletions

49
.chglog/CHANGELOG.tpl.md Normal file
View File

@@ -0,0 +1,49 @@
{{ if .Versions -}}
{{ if .Unreleased.CommitGroups -}}
{{ range .Unreleased.CommitGroups -}}
### {{ .Title }}
{{ range .Commits -}}
- {{ if .Scope }}**{{ .Scope }}:** {{ end }}{{ .Subject }}
{{ end }}
{{ end -}}
{{ else }}
{{ range .Unreleased.Commits -}}
- {{ if .Scope }}**{{ .Scope }}:** {{ end }}{{ .Subject }}
{{ end }}
{{ end -}}
{{ end -}}
{{ range .Versions }}
<a name="{{ .Tag.Name }}"></a>
{{ if .CommitGroups -}}
{{ range .CommitGroups -}}
### {{ .Title }}
{{ range .Commits -}}
- {{ if .Scope }}**{{ .Scope }}:** {{ end }}{{ .Subject }}
{{ end }}
{{ end -}}
{{ else }}
{{ range .Commits -}}
- {{ if .Scope }}**{{ .Scope }}:** {{ end }}{{ .Subject }} (https://github.com/mudler/luet/commit/{{.Hash.Short}})
{{ end }}
{{ end -}}
{{- if .NoteGroups -}}
{{ range .NoteGroups -}}
### {{ .Title }}
{{ range .Notes }}
{{ .Body }}
{{ end }}
{{ end -}}
{{ end -}}
{{ end -}}
{{- if .Versions }}
[Unreleased]: {{ .Info.RepositoryURL }}/compare/{{ $latest := index .Versions 0 }}{{ $latest.Tag.Name }}...HEAD
{{ range .Versions -}}
{{ if .Tag.Previous -}}
[{{ .Tag.Name }}]: {{ $.Info.RepositoryURL }}/compare/{{ .Tag.Previous.Name }}...{{ .Tag.Name }}
{{ end -}}
{{ end -}}
{{ end -}}

27
.chglog/config.yml Executable file
View File

@@ -0,0 +1,27 @@
style: github
template: CHANGELOG.tpl.md
info:
title: CHANGELOG
repository_url: https://github.com/mudler/luet
options:
commits:
# filters:
# Type:
# - feat
# - fix
# - perf
# - refactor
commit_groups:
title_maps:
feat: Features
fix: Bug Fixes
perf: Performance Improvements
refactor: Code Refactoring
ci: Continous Integration
header:
pattern: "(.*)"
pattern_maps:
- Subject
notes:
keywords:
- BREAKING CHANGE

View File

@@ -1,8 +1,46 @@
on: push
concurrency:
group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
cancel-in-progress: true
name: Build and release on push
jobs:
release:
name: Test and Release
tests-integration:
name: Integration tests
runs-on: ubuntu-latest
steps:
- name: Install Go
uses: actions/setup-go@v2
with:
go-version: 1.14.x
- name: Checkout code
uses: actions/checkout@v2
- name: setup-docker
uses: docker-practice/actions-setup-docker@0.0.1
- name: Login to quay
run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
- name: Install deps
run: |
sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
sudo chmod a+x "/usr/bin/img"
- name: Login to quay with img
run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo img login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
- name: Tests with Img backend
run: |
sudo -E env "PATH=$PATH" \
env "LUET_BACKEND=img" \
make test-integration
- name: Tests
run: |
sudo -E \
env "PATH=$PATH" \
env "TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
env "UNIT_TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
env "UNIT_TEST_DOCKER_IMAGE_REPOSITORY=${{ secrets.DOCKER_TESTING_UNIT_TEST_IMAGE }}" \
make test-integration
tests-unit:
name: Unit tests
runs-on: ubuntu-latest
steps:
- name: Install Go
@@ -22,13 +60,6 @@ jobs:
sudo chmod a+x "/usr/bin/img"
- name: Build test
run: sudo -E env "PATH=$PATH" make multiarch-build-small
- name: Login to quay with img
run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo img login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
- name: Tests with Img backend
run: |
sudo -E env "PATH=$PATH" \
env "LUET_BACKEND=img" \
make test-integration
- name: Tests
run: |
sudo -E \
@@ -36,7 +67,28 @@ jobs:
env "TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
env "UNIT_TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
env "UNIT_TEST_DOCKER_IMAGE_REPOSITORY=${{ secrets.DOCKER_TESTING_UNIT_TEST_IMAGE }}" \
make test-integration test-coverage
make test-coverage
release:
name: Test and Release
needs: ["tests-integration","tests-unit"]
runs-on: ubuntu-latest
steps:
- name: Install Go
uses: actions/setup-go@v2
with:
go-version: 1.14.x
- name: Checkout code
uses: actions/checkout@v2
- name: setup-docker
uses: docker-practice/actions-setup-docker@0.0.1
- name: Login to quay
run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
- name: Install deps
run: |
sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
sudo chmod a+x "/usr/bin/img"
- name: Build
run: sudo -E env "PATH=$PATH" make multiarch-build-small && sudo chmod -R 777 release/
- name: Release

View File

@@ -2,7 +2,31 @@
on: pull_request
name: Build and Test
jobs:
test:
tests-integration:
strategy:
matrix:
go-version: [1.14.x]
platform: [ubuntu-latest]
runs-on: ${{ matrix.platform }}
steps:
- name: Install Go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v2
- name: setup-docker
uses: docker-practice/actions-setup-docker@0.0.1
- name: Install deps
run: |
sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
sudo chmod a+x "/usr/bin/img"
- name: Tests with Img backend
run: sudo -E env "PATH=$PATH" env "LUET_BACKEND=img" make test-integration
- name: Tests
run: sudo -E env "PATH=$PATH" make test-integration
tests-unit:
strategy:
matrix:
go-version: [1.14.x]
@@ -24,7 +48,5 @@ jobs:
sudo chmod a+x "/usr/bin/img"
- name: Build
run: sudo -E env "PATH=$PATH" make multiarch-build-small
- name: Tests with Img backend
run: sudo -E env "PATH=$PATH" env "LUET_BACKEND=img" make test-integration
- name: Tests
run: sudo -E env "PATH=$PATH" make test-integration test-coverage
run: sudo -E env "PATH=$PATH" make test-coverage

53
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,53 @@
We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
- Reporting a bug
- Discussing the current state of the code
- Submitting a fix
- Proposing new features
- Becoming a maintainer
## We Develop with Github
We use github to host code, to track issues and feature requests, as well as accept pull requests.
## Stay in touch
Join us in [slack](https://luet.slack.com/join/shared_invite/enQtOTQxMjcyNDQ0MDUxLWQ5ODVlNTI1MTYzNDRkYzkyYmM1YWE5YjM0NTliNDEzNmQwMTkxNDRhNDIzM2Y5NDBlOTZjZTYxYWQyNDE4YzY#/) and hang out with the community! It will be much easier to get started and do your first steps in contributing to the project.
## All Code Changes Happen Through Pull Requests
Pull requests are the best way to propose changes to the codebase. We actively welcome your pull requests:
1. Fork the repo you want to contribute to and create your branch from `develop`.
2. If you've added code that should be tested, add tests.
3. If you've changed APIs, update the [documentation](https://github.com/Luet-lab/docs).
4. Ensure the test suite passes.
5. Make sure your code lints.
6. Issue that pull request!
## Any contributions you make will be under the Software License of the repository
In short, when you submit code changes, your submissions are understood to be under the same License that covers the project. Feel free to contact the maintainers if that's a concern.
## Report bugs using Github's [issues](https://github.com/mudler/luet/issues)
We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/mudler/luet/issues/new); it's that easy!
## Write bug reports with detail, background, and sample code
Try to be as more descriptive as possible. When opening a new issue you will be prompted to choose between a bug or a feature request, with a small template to fill details with. Be specific!
**Great Bug Reports** tend to have:
- A quick summary and/or background
- Steps to reproduce
- Be specific!
- Give sample code if you can.
- What you expected would happen
- What actually happens
- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
People *love* thorough bug reports.
## License
By contributing, you agree that your contributions will be licensed under the project Licenses.
## References
This document was adapted from the open-source contribution guidelines from https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62

View File

@@ -1,19 +1,24 @@
<p align="center">
<img width=150 height=150 src="https://user-images.githubusercontent.com/2420543/119691600-0293d700-be4b-11eb-827f-49ff1174a07a.png">
</p>
# luet - Container-based Package manager
[![Docker Repository on Quay](https://quay.io/repository/luet/base/status "Docker Repository on Quay")](https://quay.io/repository/luet/base)
[![Go Report Card](https://goreportcard.com/badge/github.com/mudler/luet)](https://goreportcard.com/report/github.com/mudler/luet)
[![Build Status](https://travis-ci.org/mudler/luet.svg?branch=master)](https://travis-ci.org/mudler/luet)
[![Build and release on push](https://github.com/mudler/luet/actions/workflows/release.yml/badge.svg)](https://github.com/mudler/luet/actions/workflows/release.yml)
[![GoDoc](https://godoc.org/github.com/mudler/luet?status.svg)](https://godoc.org/github.com/mudler/luet)
[![codecov](https://codecov.io/gh/mudler/luet/branch/master/graph/badge.svg)](https://codecov.io/gh/mudler/luet)
[![asciicast](https://asciinema.org/a/388348.svg)](https://asciinema.org/a/388348)
Luet is a multi-platform Package Manager based off from containers - it uses Docker (and others) to build packages. It has zero dependencies and it is well suitable for "from scratch" environments. It can also version entire rootfs and enables delivery of OTA-alike updates, making it a perfect fit for the Edge computing era and IoT embedded devices.
It offers a simple [specfile format](https://luet-lab.github.io/docs/docs/concepts/packages/specfile/) in YAML notation to define both [packages](https://luet-lab.github.io/docs/docs/concepts/packages/) and [rootfs](https://luet-lab.github.io/docs/docs/concepts/packages/#package-layers). As it is based on containers, it can be also used to build stages for Linux From Scratch installations and it can build and track updates for those systems.
It is written entirely in Golang and where used as package manager, it can run in from scratch environment, with zero dependencies.
[![asciicast](https://asciinema.org/a/388348.svg)](https://asciinema.org/a/388348)
## In a glance
- Luet can reuse Gentoo's portage tree hierarchy, and it is heavily inspired from it.

View File

@@ -21,6 +21,7 @@ import (
"github.com/ghodss/yaml"
helpers "github.com/mudler/luet/cmd/helpers"
"github.com/mudler/luet/cmd/util"
"github.com/mudler/luet/pkg/compiler"
"github.com/mudler/luet/pkg/compiler/types/artifact"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
@@ -83,10 +84,8 @@ Build packages specifying multiple definition trees:
viper.BindPFlag("wait", cmd.Flags().Lookup("wait"))
viper.BindPFlag("keep-images", cmd.Flags().Lookup("keep-images"))
LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
util.BindSolverFlags(cmd)
LuetCfg.Viper.BindPFlag("general.show_build_output", cmd.Flags().Lookup("live-output"))
LuetCfg.Viper.BindPFlag("backend-args", cmd.Flags().Lookup("backend-args"))
@@ -148,21 +147,11 @@ Build packages specifying multiple definition trees:
Info("Building in", dst)
stype := LuetCfg.Viper.GetString("solver.type")
discount := LuetCfg.Viper.GetFloat64("solver.discount")
rate := LuetCfg.Viper.GetFloat64("solver.rate")
attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
opts := util.SetSolverConfig()
pullRepo, _ := cmd.Flags().GetStringArray("pull-repository")
LuetCfg.GetGeneral().ShowBuildOutput = LuetCfg.Viper.GetBool("general.show_build_output")
opts := &LuetSolverOptions{
Type: stype,
LearnRate: float32(rate),
Discount: float32(discount),
MaxAttempts: attempts,
}
Debug("Solver", opts.CompactString())
if concurrent {

View File

@@ -20,9 +20,9 @@ import (
"os"
"path/filepath"
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
config "github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/logger"
"github.com/spf13/cobra"
@@ -33,21 +33,13 @@ var cleanupCmd = &cobra.Command{
Short: "Clean packages cache.",
Long: `remove downloaded packages tarballs and clean cache directory`,
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("installed", cmd.Flags().Lookup("installed"))
util.BindSystemFlags(cmd)
},
Run: func(cmd *cobra.Command, args []string) {
var cleaned int = 0
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := config.LuetCfg.Viper.GetString("system.rootfs")
engine := config.LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
// Check if cache dir exists
if helpers.Exists(LuetCfg.GetSystem().GetSystemPkgsCacheDirPath()) {
if fileHelper.Exists(LuetCfg.GetSystem().GetSystemPkgsCacheDirPath()) {
files, err := ioutil.ReadDir(LuetCfg.GetSystem().GetSystemPkgsCacheDirPath())
if err != nil {

View File

@@ -18,6 +18,7 @@ package cmd_database
import (
"io/ioutil"
"github.com/mudler/luet/cmd/util"
artifact "github.com/mudler/luet/pkg/compiler/types/artifact"
. "github.com/mudler/luet/pkg/logger"
@@ -45,21 +46,11 @@ The yaml must contain the package definition, and the file list at least.
For reference, inspect a "metadata.yaml" file generated while running "luet build"`,
Args: cobra.OnlyValidArgs,
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
util.BindSystemFlags(cmd)
},
Run: func(cmd *cobra.Command, args []string) {
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
systemDB := LuetCfg.GetSystemDB()
for _, a := range args {
@@ -74,6 +65,12 @@ For reference, inspect a "metadata.yaml" file generated while running "luet buil
files := art.Files
// Check if the package is already present
if p, err := systemDB.FindPackage(art.CompileSpec.GetPackage()); err == nil && p.GetName() != "" {
Fatal("Package", art.CompileSpec.GetPackage().HumanReadableString(),
" already present.")
}
if _, err := systemDB.CreatePackage(art.CompileSpec.GetPackage()); err != nil {
Fatal("Failed to create ", a, ": ", err.Error())
}

View File

@@ -19,6 +19,7 @@ import (
"fmt"
helpers "github.com/mudler/luet/cmd/helpers"
"github.com/mudler/luet/cmd/util"
"gopkg.in/yaml.v2"
. "github.com/mudler/luet/pkg/config"
@@ -38,20 +39,11 @@ To return also files:
$ luet database get --files system/foo`,
Args: cobra.OnlyValidArgs,
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
util.BindSystemFlags(cmd)
},
Run: func(cmd *cobra.Command, args []string) {
showFiles, _ := cmd.Flags().GetBool("files")
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
systemDB := LuetCfg.GetSystemDB()

View File

@@ -19,6 +19,7 @@ import (
. "github.com/mudler/luet/pkg/logger"
helpers "github.com/mudler/luet/cmd/helpers"
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
"github.com/spf13/cobra"
@@ -36,19 +37,10 @@ This commands takes multiple packages as arguments and prunes their entries from
`,
Args: cobra.OnlyValidArgs,
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
util.BindSystemFlags(cmd)
},
Run: func(cmd *cobra.Command, args []string) {
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
systemDB := LuetCfg.GetSystemDB()

View File

@@ -19,6 +19,7 @@ import (
"github.com/mudler/luet/pkg/solver"
helpers "github.com/mudler/luet/cmd/helpers"
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
@@ -47,13 +48,8 @@ To force install a package:
`,
Aliases: []string{"i"},
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
util.BindSystemFlags(cmd)
util.BindSolverFlags(cmd)
LuetCfg.Viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
LuetCfg.Viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
@@ -70,10 +66,6 @@ To force install a package:
toInstall = append(toInstall, pack)
}
stype := LuetCfg.Viper.GetString("solver.type")
discount := LuetCfg.Viper.GetFloat64("solver.discount")
rate := LuetCfg.Viper.GetFloat64("solver.rate")
attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
force := LuetCfg.Viper.GetBool("force")
nodeps := LuetCfg.Viper.GetBool("nodeps")
onlydeps := LuetCfg.Viper.GetBool("onlydeps")
@@ -81,18 +73,8 @@ To force install a package:
yes := LuetCfg.Viper.GetBool("yes")
downloadOnly, _ := cmd.Flags().GetBool("download-only")
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
LuetCfg.GetSolverOptions().Type = stype
LuetCfg.GetSolverOptions().LearnRate = float32(rate)
LuetCfg.GetSolverOptions().Discount = float32(discount)
LuetCfg.GetSolverOptions().MaxAttempts = attempts
util.SetSystemConfig()
util.SetSolverConfig()
if concurrent {
LuetCfg.GetSolverOptions().Implementation = solver.ParallelSimple

View File

@@ -15,6 +15,7 @@
package cmd
import (
"github.com/mudler/luet/cmd/util"
installer "github.com/mudler/luet/pkg/installer"
. "github.com/mudler/luet/pkg/config"
@@ -27,9 +28,7 @@ var reclaimCmd = &cobra.Command{
Use: "reclaim",
Short: "Reclaim packages to Luet database from available repositories",
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
util.BindSystemFlags(cmd)
LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
},
Long: `Reclaim tries to find association between packages in the online repositories and the system one.
@@ -39,13 +38,7 @@ var reclaimCmd = &cobra.Command{
It scans the target file system, and if finds a match with a package available in the repositories, it marks as installed in the system database.
`,
Run: func(cmd *cobra.Command, args []string) {
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
// This shouldn't be necessary, but we need to unmarshal the repositories to a concrete struct, thus we need to port them back to the Repositories type
repos := installer.Repositories{}

128
cmd/reinstall.go Normal file
View File

@@ -0,0 +1,128 @@
// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package cmd
import (
installer "github.com/mudler/luet/pkg/installer"
"github.com/mudler/luet/pkg/solver"
helpers "github.com/mudler/luet/cmd/helpers"
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
"github.com/spf13/cobra"
)
var reinstallCmd = &cobra.Command{
Use: "reinstall <pkg1> <pkg2> <pkg3>",
Short: "reinstall a set of packages",
Long: `Reinstall a group of packages in the system:
$ luet reinstall -y system/busybox shells/bash system/coreutils ...
`,
PreRun: func(cmd *cobra.Command, args []string) {
util.BindSystemFlags(cmd)
util.BindSolverFlags(cmd)
LuetCfg.Viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
LuetCfg.Viper.BindPFlag("for", cmd.Flags().Lookup("for"))
LuetCfg.Viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
},
Run: func(cmd *cobra.Command, args []string) {
var toUninstall pkg.Packages
var toAdd pkg.Packages
force := LuetCfg.Viper.GetBool("force")
onlydeps := LuetCfg.Viper.GetBool("onlydeps")
concurrent, _ := cmd.Flags().GetBool("solver-concurrent")
yes := LuetCfg.Viper.GetBool("yes")
downloadOnly, _ := cmd.Flags().GetBool("download-only")
util.SetSystemConfig()
for _, a := range args {
pack, err := helpers.ParsePackageStr(a)
if err != nil {
Fatal("Invalid package string ", a, ": ", err.Error())
}
toUninstall = append(toUninstall, pack)
toAdd = append(toAdd, pack)
}
// This shouldn't be necessary, but we need to unmarshal the repositories to a concrete struct, thus we need to port them back to the Repositories type
repos := installer.Repositories{}
for _, repo := range LuetCfg.SystemRepositories {
if !repo.Enable {
continue
}
r := installer.NewSystemRepository(repo)
repos = append(repos, r)
}
util.SetSolverConfig()
if concurrent {
LuetCfg.GetSolverOptions().Implementation = solver.ParallelSimple
} else {
LuetCfg.GetSolverOptions().Implementation = solver.SingleCoreSimple
}
Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
// Load config protect configs
installer.LoadConfigProtectConfs(LuetCfg)
inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
Concurrency: LuetCfg.GetGeneral().Concurrency,
SolverOptions: *LuetCfg.GetSolverOptions(),
NoDeps: true,
Force: force,
OnlyDeps: onlydeps,
PreserveSystemEssentialData: true,
Ask: !yes,
DownloadOnly: downloadOnly,
})
inst.Repositories(repos)
system := &installer.System{Database: LuetCfg.GetSystemDB(), Target: LuetCfg.GetSystem().Rootfs}
err := inst.Swap(toUninstall, toAdd, system)
if err != nil {
Fatal("Error: " + err.Error())
}
},
}
func init() {
reinstallCmd.Flags().String("system-dbpath", "", "System db path")
reinstallCmd.Flags().String("system-target", "", "System rootpath")
reinstallCmd.Flags().String("system-engine", "", "System DB engine")
reinstallCmd.Flags().String("solver-type", "", "Solver strategy ( Defaults none, available: "+AvailableResolvers+" )")
reinstallCmd.Flags().Float32("solver-rate", 0.7, "Solver learning rate")
reinstallCmd.Flags().Float32("solver-discount", 1.0, "Solver discount rate")
reinstallCmd.Flags().Int("solver-attempts", 9000, "Solver maximum attempts")
reinstallCmd.Flags().Bool("onlydeps", false, "Consider **only** package dependencies")
reinstallCmd.Flags().Bool("force", false, "Skip errors and keep going (potentially harmful)")
reinstallCmd.Flags().Bool("solver-concurrent", false, "Use concurrent solver (experimental)")
reinstallCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
reinstallCmd.Flags().Bool("download-only", false, "Download only")
RootCmd.AddCommand(reinstallCmd)
}

View File

@@ -19,6 +19,7 @@ import (
"github.com/mudler/luet/pkg/solver"
helpers "github.com/mudler/luet/cmd/helpers"
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
@@ -35,13 +36,8 @@ var replaceCmd = &cobra.Command{
$ luet replace -y system/busybox ... --for shells/bash --for system/coreutils ...
`,
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
util.BindSystemFlags(cmd)
util.BindSolverFlags(cmd)
LuetCfg.Viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
LuetCfg.Viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
@@ -54,24 +50,15 @@ var replaceCmd = &cobra.Command{
var toAdd pkg.Packages
f := LuetCfg.Viper.GetStringSlice("for")
stype := LuetCfg.Viper.GetString("solver.type")
discount := LuetCfg.Viper.GetFloat64("solver.discount")
rate := LuetCfg.Viper.GetFloat64("solver.rate")
attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
force := LuetCfg.Viper.GetBool("force")
nodeps := LuetCfg.Viper.GetBool("nodeps")
onlydeps := LuetCfg.Viper.GetBool("onlydeps")
concurrent, _ := cmd.Flags().GetBool("solver-concurrent")
yes := LuetCfg.Viper.GetBool("yes")
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
downloadOnly, _ := cmd.Flags().GetBool("download-only")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
util.SetSolverConfig()
for _, a := range args {
pack, err := helpers.ParsePackageStr(a)
if err != nil {
@@ -98,11 +85,6 @@ var replaceCmd = &cobra.Command{
repos = append(repos, r)
}
LuetCfg.GetSolverOptions().Type = stype
LuetCfg.GetSolverOptions().LearnRate = float32(rate)
LuetCfg.GetSolverOptions().Discount = float32(discount)
LuetCfg.GetSolverOptions().MaxAttempts = attempts
if concurrent {
LuetCfg.GetSolverOptions().Implementation = solver.ParallelSimple
} else {

View File

@@ -25,6 +25,7 @@ import (
"github.com/marcsauter/single"
bus "github.com/mudler/luet/pkg/bus"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
extensions "github.com/mudler/cobra-extensions"
config "github.com/mudler/luet/pkg/config"
@@ -40,8 +41,14 @@ var Verbose bool
var LockedCommands = []string{"install", "uninstall", "upgrade"}
const (
LuetCLIVersion = "0.15.0"
LuetCLIVersion = "0.17.4"
LuetEnvPrefix = "LUET"
license = `
Luet Copyright (C) 2019-2021 Ettore Di Giacinto
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions.
`
)
// Build time and commit information.
@@ -52,6 +59,47 @@ var (
BuildCommit string
)
func version() string {
return fmt.Sprintf("%s-g%s %s", LuetCLIVersion, BuildCommit, BuildTime)
}
var noBannerCommands = []string{"search", "exec", "tree", "database", "box", "cleanup"}
func displayVersionBanner() {
display := true
if len(os.Args) > 1 {
for _, c := range noBannerCommands {
if os.Args[1] == c {
display = false
}
}
}
if display {
Info("Luet version", version())
Info(license)
}
}
func handleLock() {
if os.Getenv("LUET_NOLOCK") != "true" {
if len(os.Args) > 1 {
for _, lockedCmd := range LockedCommands {
if os.Args[1] == lockedCmd {
s := single.New("luet")
if err := s.CheckLock(); err != nil && err == single.ErrAlreadyRunning {
Fatal("another instance of the app is already running, exiting")
} else if err != nil {
// Another error occurred, might be worth handling it as well
Fatal("failed to acquire exclusive app lock:", err.Error())
}
defer s.TryUnlock()
break
}
}
}
}
}
// RootCmd represents the base command when called without any subcommands
var RootCmd = &cobra.Command{
Use: "luet",
@@ -79,8 +127,9 @@ To build a package, from a tree definition:
$ luet build --tree tree/path package
`,
Version: fmt.Sprintf("%s-g%s %s", LuetCLIVersion, BuildCommit, BuildTime),
Version: version(),
PersistentPreRun: func(cmd *cobra.Command, args []string) {
err := LoadConfig(config.LuetCfg)
if err != nil {
Fatal("failed to load configuration:", err.Error())
@@ -154,23 +203,8 @@ func LoadConfig(c *config.LuetConfig) error {
// This is called by main.main(). It only needs to happen once to the rootCmd.
func Execute() {
if os.Getenv("LUET_NOLOCK") != "true" {
if len(os.Args) > 1 {
for _, lockedCmd := range LockedCommands {
if os.Args[1] == lockedCmd {
s := single.New("luet")
if err := s.CheckLock(); err != nil && err == single.ErrAlreadyRunning {
Fatal("another instance of the app is already running, exiting")
} else if err != nil {
// Another error occurred, might be worth handling it as well
Fatal("failed to acquire exclusive app lock:", err.Error())
}
defer s.TryUnlock()
break
}
}
}
}
handleLock()
displayVersionBanner()
if err := RootCmd.Execute(); err != nil {
fmt.Println(err)
@@ -252,7 +286,7 @@ func initConfig() {
}
homeDir := helpers.GetHomeDir()
if helpers.Exists(filepath.Join(pwdDir, ".luet.yaml")) || (homeDir != "" && helpers.Exists(filepath.Join(homeDir, ".luet.yaml"))) {
if fileHelper.Exists(filepath.Join(pwdDir, ".luet.yaml")) || (homeDir != "" && fileHelper.Exists(filepath.Join(homeDir, ".luet.yaml"))) {
viper.AddConfigPath(".")
if homeDir != "" {
viper.AddConfigPath(homeDir)

View File

@@ -21,6 +21,7 @@ import (
"github.com/ghodss/yaml"
"github.com/jedib0t/go-pretty/table"
"github.com/jedib0t/go-pretty/v6/list"
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
installer "github.com/mudler/luet/pkg/installer"
. "github.com/mudler/luet/pkg/logger"
@@ -306,14 +307,9 @@ Search can also return results in the terminal in different ways: as terminal ou
`,
Aliases: []string{"s"},
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
util.BindSystemFlags(cmd)
util.BindSolverFlags(cmd)
LuetCfg.Viper.BindPFlag("installed", cmd.Flags().Lookup("installed"))
LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
},
Run: func(cmd *cobra.Command, args []string) {
var results Results
@@ -325,32 +321,20 @@ Search can also return results in the terminal in different ways: as terminal ou
hidden, _ := cmd.Flags().GetBool("hidden")
installed := LuetCfg.Viper.GetBool("installed")
stype := LuetCfg.Viper.GetString("solver.type")
discount := LuetCfg.Viper.GetFloat64("solver.discount")
rate := LuetCfg.Viper.GetFloat64("solver.rate")
attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
searchWithLabel, _ := cmd.Flags().GetBool("by-label")
searchWithLabelMatch, _ := cmd.Flags().GetBool("by-label-regex")
revdeps, _ := cmd.Flags().GetBool("revdeps")
tableMode, _ := cmd.Flags().GetBool("table")
files, _ := cmd.Flags().GetBool("files")
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
util.SetSolverConfig()
out, _ := cmd.Flags().GetString("output")
if out != "terminal" {
LuetCfg.GetLogging().SetLogLevel("error")
}
LuetCfg.GetSolverOptions().Type = stype
LuetCfg.GetSolverOptions().LearnRate = float32(rate)
LuetCfg.GetSolverOptions().Discount = float32(discount)
LuetCfg.GetSolverOptions().MaxAttempts = attempts
l := list.NewWriter()
t := table.NewWriter()
t.AppendHeader(rows)

View File

@@ -16,6 +16,7 @@ package cmd
import (
helpers "github.com/mudler/luet/cmd/helpers"
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
installer "github.com/mudler/luet/pkg/installer"
. "github.com/mudler/luet/pkg/logger"
@@ -31,16 +32,11 @@ var uninstallCmd = &cobra.Command{
Long: `Uninstall packages`,
Aliases: []string{"rm", "un"},
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
util.BindSystemFlags(cmd)
util.BindSolverFlags(cmd)
LuetCfg.Viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
LuetCfg.Viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
},
Run: func(cmd *cobra.Command, args []string) {
toRemove := []pkg.Package{}
@@ -53,10 +49,6 @@ var uninstallCmd = &cobra.Command{
toRemove = append(toRemove, pack)
}
stype := LuetCfg.Viper.GetString("solver.type")
discount := LuetCfg.Viper.GetFloat64("solver.discount")
rate := LuetCfg.Viper.GetFloat64("solver.rate")
attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
force := LuetCfg.Viper.GetBool("force")
nodeps, _ := cmd.Flags().GetBool("nodeps")
full, _ := cmd.Flags().GetBool("full")
@@ -64,18 +56,12 @@ var uninstallCmd = &cobra.Command{
fullClean, _ := cmd.Flags().GetBool("full-clean")
concurrent, _ := cmd.Flags().GetBool("solver-concurrent")
yes := LuetCfg.Viper.GetBool("yes")
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
keepProtected, _ := cmd.Flags().GetBool("keep-protected-files")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
util.SetSystemConfig()
util.SetSolverConfig()
LuetCfg.GetSolverOptions().Type = stype
LuetCfg.GetSolverOptions().LearnRate = float32(rate)
LuetCfg.GetSolverOptions().Discount = float32(discount)
LuetCfg.GetSolverOptions().MaxAttempts = attempts
LuetCfg.ConfigProtectSkip = !keepProtected
if concurrent {
LuetCfg.GetSolverOptions().Implementation = solver.ParallelSimple
} else {
@@ -123,6 +109,7 @@ func init() {
uninstallCmd.Flags().Bool("full-clean", false, "(experimental) Uninstall packages and all the other deps/revdeps of it.")
uninstallCmd.Flags().Bool("solver-concurrent", false, "Use concurrent solver (experimental)")
uninstallCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
uninstallCmd.Flags().BoolP("keep-protected-files", "k", false, "Keep package protected files around")
RootCmd.AddCommand(uninstallCmd)
}

View File

@@ -15,6 +15,7 @@
package cmd
import (
"github.com/mudler/luet/cmd/util"
. "github.com/mudler/luet/pkg/config"
installer "github.com/mudler/luet/pkg/installer"
. "github.com/mudler/luet/pkg/logger"
@@ -28,15 +29,10 @@ var upgradeCmd = &cobra.Command{
Short: "Upgrades the system",
Aliases: []string{"u"},
PreRun: func(cmd *cobra.Command, args []string) {
LuetCfg.Viper.BindPFlag("system.database_path", installCmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", installCmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
util.BindSystemFlags(cmd)
util.BindSolverFlags(cmd)
LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
LuetCfg.Viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
},
Long: `Upgrades packages in parallel`,
Run: func(cmd *cobra.Command, args []string) {
@@ -51,10 +47,6 @@ var upgradeCmd = &cobra.Command{
repos = append(repos, r)
}
stype := LuetCfg.Viper.GetString("solver.type")
discount := LuetCfg.Viper.GetFloat64("solver.discount")
rate := LuetCfg.Viper.GetFloat64("solver.rate")
attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
force := LuetCfg.Viper.GetBool("force")
nodeps, _ := cmd.Flags().GetBool("nodeps")
full, _ := cmd.Flags().GetBool("full")
@@ -63,18 +55,11 @@ var upgradeCmd = &cobra.Command{
sync, _ := cmd.Flags().GetBool("sync")
concurrent, _ := cmd.Flags().GetBool("solver-concurrent")
yes := LuetCfg.Viper.GetBool("yes")
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
downloadOnly, _ := cmd.Flags().GetBool("download-only")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.Rootfs = rootfs
LuetCfg.GetSolverOptions().Type = stype
LuetCfg.GetSolverOptions().LearnRate = float32(rate)
LuetCfg.GetSolverOptions().Discount = float32(discount)
LuetCfg.GetSolverOptions().MaxAttempts = attempts
util.SetSystemConfig()
util.SetSolverConfig()
if concurrent {
LuetCfg.GetSolverOptions().Implementation = solver.ParallelSimple
} else {

65
cmd/util/cli.go Normal file
View File

@@ -0,0 +1,65 @@
// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package util
import (
"github.com/spf13/cobra"
"github.com/mudler/luet/pkg/config"
. "github.com/mudler/luet/pkg/config"
)
func BindSystemFlags(cmd *cobra.Command) {
LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
LuetCfg.Viper.BindPFlag("system.database_engine", cmd.Flags().Lookup("system-engine"))
}
func BindSolverFlags(cmd *cobra.Command) {
LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
}
func SetSystemConfig() {
dbpath := LuetCfg.Viper.GetString("system.database_path")
rootfs := LuetCfg.Viper.GetString("system.rootfs")
engine := LuetCfg.Viper.GetString("system.database_engine")
LuetCfg.System.DatabaseEngine = engine
LuetCfg.System.DatabasePath = dbpath
LuetCfg.System.SetRootFS(rootfs)
}
func SetSolverConfig() (c *config.LuetSolverOptions) {
stype := LuetCfg.Viper.GetString("solver.type")
discount := LuetCfg.Viper.GetFloat64("solver.discount")
rate := LuetCfg.Viper.GetFloat64("solver.rate")
attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
LuetCfg.GetSolverOptions().Type = stype
LuetCfg.GetSolverOptions().LearnRate = float32(rate)
LuetCfg.GetSolverOptions().Discount = float32(discount)
LuetCfg.GetSolverOptions().MaxAttempts = attempts
return &config.LuetSolverOptions{
Type: stype,
LearnRate: float32(rate),
Discount: float32(discount),
MaxAttempts: attempts,
}
}

View File

@@ -23,7 +23,7 @@ import (
"github.com/docker/docker/api/types"
"github.com/docker/go-units"
"github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
"github.com/mudler/luet/pkg/helpers/docker"
. "github.com/mudler/luet/pkg/logger"
"github.com/spf13/cobra"
@@ -77,7 +77,7 @@ func NewUnpackCommand() *cobra.Command {
RegistryToken: registryToken,
}
info, err := helpers.DownloadAndExtractDockerImage(temp, image, destination, auth, verify)
info, err := docker.DownloadAndExtractDockerImage(temp, image, destination, auth, verify)
if err != nil {
Error(err.Error())
os.Exit(1)

View File

@@ -7,7 +7,7 @@ fi
set -ex
export LUET_NOLOCK=true
LUET_VERSION=$(curl -s https://api.github.com/repos/mudler/luet/releases/latest | ( grep -oP '"tag_name": "\K(.*)(?=")' || echo "0.9.24" ))
LUET_VERSION=$(curl -s https://api.github.com/repos/mudler/luet/releases/latest | grep tag_name | awk '{ print $2 }' | sed -e 's/\"//g' -e 's/,//g' || echo "0.9.24" )
LUET_ROOTFS=${LUET_ROOTFS:-/}
LUET_DATABASE_PATH=${LUET_DATABASE_PATH:-/var/luet/db}
LUET_DATABASE_ENGINE=${LUET_DATABASE_ENGINE:-boltdb}

7
go.mod
View File

@@ -11,9 +11,10 @@ require (
github.com/cavaliercoder/grab v1.0.1-0.20201108051000-98a5bfe305ec
github.com/containerd/containerd v1.4.1-0.20201117152358-0edc412565dc
github.com/crillab/gophersat v1.3.2-0.20201023142334-3fc2ac466765
github.com/docker/cli v0.0.0-20200227165822-2298e6a3fe24
github.com/docker/cli v20.10.0-beta1.0.20201029214301-1d20b15adc38+incompatible
github.com/docker/distribution v2.7.1+incompatible
github.com/docker/docker v20.10.0-beta1.0.20201110211921-af34b94a78a1+incompatible
github.com/docker/go-events v0.0.0-20190806004212-e31b211e4f1c // indirect
github.com/docker/go-units v0.4.0
github.com/ecooper/qlearning v0.0.0-20160612200101-3075011a69fd
github.com/genuinetools/img v0.5.11
@@ -22,7 +23,7 @@ require (
github.com/google/renameio v1.0.0
github.com/hashicorp/go-multierror v1.0.0
github.com/hashicorp/go-version v1.2.1
github.com/imdario/mergo v0.3.8
github.com/imdario/mergo v0.3.9
github.com/jedib0t/go-pretty v4.3.0+incompatible
github.com/jedib0t/go-pretty/v6 v6.0.5
github.com/jinzhu/copier v0.0.0-20180308034124-7e38e58719c3
@@ -55,8 +56,10 @@ require (
go.uber.org/atomic v1.5.1 // indirect
go.uber.org/multierr v1.4.0
go.uber.org/zap v1.13.0
golang.org/x/mod v0.4.2
google.golang.org/grpc v1.29.1
gopkg.in/yaml.v2 v2.3.0
gotest.tools/v3 v3.0.2 // indirect
helm.sh/helm/v3 v3.3.4
)

469
go.sum

File diff suppressed because it is too large Load Diff

View File

@@ -23,9 +23,9 @@ import (
"strings"
"syscall"
"github.com/pkg/errors"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
helpers "github.com/mudler/luet/pkg/helpers"
"github.com/pkg/errors"
)
type Box interface {
@@ -107,7 +107,7 @@ func (b *DefaultBox) Exec() error {
func (b *DefaultBox) Run() error {
if !helpers.Exists(b.Root) {
if !fileHelper.Exists(b.Root) {
return errors.New(b.Root + " does not exist")
}

View File

@@ -1,9 +1,8 @@
package bus
import (
. "github.com/mudler/luet/pkg/logger"
"github.com/mudler/go-pluggable"
"github.com/sirupsen/logrus"
)
var (
@@ -46,6 +45,13 @@ var (
EventRepositoryPreBuild pluggable.EventType = "repository.pre.build"
// EventRepositoryPostBuild is the event fired after a repository was built
EventRepositoryPostBuild pluggable.EventType = "repository.post.build"
// Image unpack
// EventImagePreUnPack is the event fired before unpacking an image to a local dir
EventImagePreUnPack pluggable.EventType = "image.pre.unpack"
// EventImagePostUnPack is the event fired after unpacking an image to a local dir
EventImagePostUnPack pluggable.EventType = "image.post.unpack"
)
// Manager is the bus instance manager, which subscribes plugins to events emitted by Luet
@@ -66,6 +72,8 @@ var Manager *Bus = &Bus{
EventImagePostBuild,
EventImagePostPull,
EventImagePostPush,
EventImagePreUnPack,
EventImagePostUnPack,
},
),
}
@@ -80,9 +88,9 @@ func (b *Bus) Initialize(plugin ...string) {
for _, e := range b.Manager.Events {
b.Manager.Response(e, func(p *pluggable.Plugin, r *pluggable.EventResponse) {
if r.Errored() {
Fatal("Plugin", p.Name, "at", p.Executable, "Error", r.Error)
logrus.Fatal("Plugin", p.Name, "at", p.Executable, "Error", r.Error)
}
Debug(
logrus.Debug(
"plugin_event",
"received from",
p.Name,

View File

@@ -24,6 +24,7 @@ import (
"strings"
bus "github.com/mudler/luet/pkg/bus"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
capi "github.com/mudler/docker-companion/api"
@@ -181,6 +182,12 @@ func (b *SimpleDocker) ExtractRootfs(opts Options, keepPerms bool) error {
name := opts.ImageName
dst := opts.Destination
if !b.ImageExists(name) {
if err := b.DownloadImage(opts); err != nil {
return errors.Wrap(err, "failed pulling image "+name+" during extraction")
}
}
tempexport, err := ioutil.TempDir(dst, "tmprootfs")
if err != nil {
return errors.Wrap(err, "Error met while creating tempdir for rootfs")
@@ -231,7 +238,7 @@ func (b *SimpleDocker) ExtractRootfs(opts Options, keepPerms bool) error {
return errors.Wrap(err, "Error met while unpacking rootfs")
}
manifest, err := helpers.Read(filepath.Join(rootfs, "manifest.json"))
manifest, err := fileHelper.Read(filepath.Join(rootfs, "manifest.json"))
if err != nil {
return errors.Wrap(err, "Error met while reading image manifest")
}

View File

@@ -21,12 +21,12 @@ import (
"github.com/mudler/luet/pkg/compiler/backend"
. "github.com/mudler/luet/pkg/compiler/backend"
"github.com/mudler/luet/pkg/compiler/types/artifact"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
"io/ioutil"
"os"
"path/filepath"
helpers "github.com/mudler/luet/pkg/helpers"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/tree"
. "github.com/onsi/ginkgo"
@@ -60,7 +60,7 @@ var _ = Describe("Docker backend", func() {
err = lspec.WriteBuildImageDefinition(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err := helpers.Read(filepath.Join(tmpdir, "Dockerfile"))
dockerfile, err := fileHelper.Read(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM alpine
@@ -79,11 +79,11 @@ ENV PACKAGE_CATEGORY=app-admin`))
Expect(b.BuildImage(opts)).ToNot(HaveOccurred())
Expect(b.ExportImage(opts)).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(tmpdir2, "output1.tar"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(tmpdir2, "output1.tar"))).To(BeTrue())
err = lspec.WriteStepImageDefinition(lspec.Image, filepath.Join(tmpdir, "LuetDockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err = helpers.Read(filepath.Join(tmpdir, "LuetDockerfile"))
dockerfile, err = fileHelper.Read(filepath.Join(tmpdir, "LuetDockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM luet/base
@@ -103,7 +103,7 @@ RUN echo bar > /test2`))
Expect(b.BuildImage(opts2)).ToNot(HaveOccurred())
Expect(b.ExportImage(opts2)).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(tmpdir, "output2.tar"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(tmpdir, "output2.tar"))).To(BeTrue())
artifacts := []artifact.ArtifactNode{{
Name: "/luetbuild/LuetDockerfile",
@@ -132,7 +132,7 @@ RUN echo bar > /test2`))
}
Expect(b.ImageDefinitionToTar(opts2)).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(tmpdir, "output3.tar"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(tmpdir, "output3.tar"))).To(BeTrue())
Expect(b.ImageExists(opts2.ImageName)).To(BeFalse())
})

View File

@@ -16,7 +16,9 @@
package compiler
import (
"crypto/md5"
"fmt"
"io"
"io/ioutil"
"os"
"path"
@@ -34,6 +36,7 @@ import (
"github.com/mudler/luet/pkg/compiler/types/options"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/solver"
@@ -89,7 +92,7 @@ func (cs *LuetCompiler) compilerWorker(i int, wg *sync.WaitGroup, cspecs chan *c
defer wg.Done()
for s := range cspecs {
ar, err := cs.compile(concurrency, keepPermissions, nil, s)
ar, err := cs.compile(concurrency, keepPermissions, nil, nil, s)
if err != nil {
errors <- err
}
@@ -317,7 +320,7 @@ func (cs *LuetCompiler) buildPackageImage(image, buildertaggedImage, packageImag
defer os.RemoveAll(buildDir) // clean up
// First we copy the source definitions into the output - we create a copy which the builds will need (we need to cache this phase somehow)
err = helpers.CopyDir(p.GetPackage().GetPath(), buildDir)
err = fileHelper.CopyDir(p.GetPackage().GetPath(), buildDir)
if err != nil {
return builderOpts, runnerOpts, errors.Wrap(err, "Could not copy package sources")
}
@@ -506,7 +509,7 @@ func oneOfImagesAvailable(images []string, b CompilerBackend) (bool, string) {
return false, ""
}
func (cs *LuetCompiler) resolveExistingImageHash(imageHash string, p *compilerspec.LuetCompilationSpec) string {
func (cs *LuetCompiler) findImageHash(imageHash string, p *compilerspec.LuetCompilationSpec) string {
var resolvedImage string
Debug("Resolving image hash for", p.Package.HumanReadableString(), "hash", imageHash, "Pull repositories", p.BuildOptions.PullImageRepository)
toChecklist := append([]string{fmt.Sprintf("%s:%s", cs.Options.PushImageRepository, imageHash)},
@@ -519,6 +522,11 @@ func (cs *LuetCompiler) resolveExistingImageHash(imageHash string, p *compilersp
resolvedImage = which
}
}
return resolvedImage
}
func (cs *LuetCompiler) resolveExistingImageHash(imageHash string, p *compilerspec.LuetCompilationSpec) string {
resolvedImage := cs.findImageHash(imageHash, p)
if resolvedImage == "" {
resolvedImage = fmt.Sprintf("%s:%s", cs.Options.PushImageRepository, imageHash)
@@ -716,7 +724,7 @@ func (cs *LuetCompiler) ComputeMinimumCompilableSet(p ...*compilerspec.LuetCompi
// Compile is a non-parallel version of CompileParallel. It builds the compilation specs and generates
// an artifact
func (cs *LuetCompiler) Compile(keepPermissions bool, p *compilerspec.LuetCompilationSpec) (*artifact.PackageArtifact, error) {
return cs.compile(cs.Options.Concurrency, keepPermissions, nil, p)
return cs.compile(cs.Options.Concurrency, keepPermissions, nil, nil, p)
}
func genImageList(refs []string, hash string) []string {
@@ -745,21 +753,163 @@ func (cs *LuetCompiler) inheritSpecBuildOptions(p *compilerspec.LuetCompilationS
Debug(p.GetPackage().HumanReadableString(), "Build options after inherit", p.BuildOptions)
}
func (cs *LuetCompiler) getSpecHash(pkgs pkg.DefaultPackages, salt string) (string, error) {
ht := NewHashTree(cs.Database)
overallFp := ""
for _, p := range pkgs {
compileSpec, err := cs.FromPackage(p)
if err != nil {
return "", errors.Wrap(err, "Error while generating compilespec for "+p.GetName())
}
packageHashTree, err := ht.Query(cs, compileSpec)
if err != nil {
return "nil", errors.Wrap(err, "failed querying hashtree")
}
overallFp = overallFp + packageHashTree.Target.Hash.PackageHash + p.GetFingerPrint()
}
h := md5.New()
io.WriteString(h, fmt.Sprintf("%s-%s", overallFp, salt))
return fmt.Sprintf("%x", h.Sum(nil)), nil
}
func (cs *LuetCompiler) resolveFinalImages(concurrency int, keepPermissions bool, p *compilerspec.LuetCompilationSpec) error {
joinTag := ">:loop: final images<"
var fromPackages pkg.DefaultPackages
if len(p.Join) > 0 {
fromPackages = p.Join
Warning(joinTag, `
Attention! the 'join' keyword is going to be deprecated in Luet >=0.18.x.
Use 'requires_final_images: true' instead in the build.yaml file`)
} else if p.RequiresFinalImages {
Info(joinTag, "Generating a parent image from final packages")
fromPackages = p.Package.GetRequires()
} else {
// No source image to resolve
return nil
}
// First compute a hash and check if image is available. if it is, then directly consume that
overallFp, err := cs.getSpecHash(fromPackages, "join")
if err != nil {
return errors.Wrap(err, "could not generate image hash")
}
Info(joinTag, "Searching existing image with hash", overallFp)
image := cs.findImageHash(overallFp, p)
if image != "" {
Info("Image already found", image)
p.SetImage(image)
return nil
}
Info(joinTag, "Image not found. Generating image join with hash ", overallFp)
// Make sure there is an output path
if err := os.MkdirAll(p.GetOutputPath(), os.ModePerm); err != nil {
return errors.Wrap(err, "while creating output path")
}
// otherwise, generate it and push it aside
joinDir, err := ioutil.TempDir(p.GetOutputPath(), "join")
if err != nil {
return errors.Wrap(err, "could not create tempdir for joining images")
}
defer os.RemoveAll(joinDir) // clean up
for _, p := range fromPackages {
Info(joinTag, ":arrow_right_hook:", p.HumanReadableString(), ":leaves:")
}
current := 0
for _, c := range fromPackages {
current++
if c != nil && c.Name != "" && c.Version != "" {
joinTag2 := fmt.Sprintf("%s %d/%d ⤑ :hammer: build %s", joinTag, current, len(p.Join), c.HumanReadableString())
Info(joinTag2, "compilation starts")
spec, err := cs.FromPackage(c)
if err != nil {
return errors.Wrap(err, "while generating images to join from")
}
wantsArtifact := true
genDepsArtifact := !cs.Options.PackageTargetOnly
spec.SetOutputPath(p.GetOutputPath())
artifact, err := cs.compile(concurrency, keepPermissions, &wantsArtifact, &genDepsArtifact, spec)
if err != nil {
return errors.Wrap(err, "failed building join image")
}
err = artifact.Unpack(joinDir, keepPermissions)
if err != nil {
return errors.Wrap(err, "failed building join image")
}
Info(joinTag2, ":white_check_mark: Done")
}
}
artifactDir, err := ioutil.TempDir(p.GetOutputPath(), "artifact")
if err != nil {
return errors.Wrap(err, "could not create tempdir for final artifact")
}
defer os.RemoveAll(joinDir) // clean up
Info(joinTag, ":droplet: generating artifact for source image of", p.GetPackage().HumanReadableString())
// After unpack, create a new artifact and a new final image from it.
// no need to compress, as we are going to toss it away.
a := artifact.NewPackageArtifact(filepath.Join(artifactDir, p.GetPackage().GetFingerPrint()+".join.tar"))
if err := a.Compress(joinDir, concurrency); err != nil {
return errors.Wrap(err, "error met while creating package archive")
}
joinImageName := fmt.Sprintf("%s:%s", cs.Options.PushImageRepository, overallFp)
Info(joinTag, ":droplet: generating image from artifact", joinImageName)
opts, err := a.GenerateFinalImage(joinImageName, cs.Backend, keepPermissions)
if err != nil {
return errors.Wrap(err, "could not create final image")
}
if cs.Options.Push {
Info(joinTag, ":droplet: pushing image from artifact", joinImageName)
if err = cs.Backend.Push(opts); err != nil {
return errors.Wrapf(err, "Could not push image: %s %s", image, opts.DockerFileName)
}
}
Info(joinTag, ":droplet: Consuming image", joinImageName)
p.SetImage(joinImageName)
return nil
}
func (cs *LuetCompiler) resolveMultiStageImages(concurrency int, keepPermissions bool, p *compilerspec.LuetCompilationSpec) error {
resolvedCopyFields := []compilerspec.CopyField{}
copyTag := ">:droplet: copy<"
if len(p.Copy) != 0 {
Info("Package has multi-stage copy, generating required images")
Info(copyTag, "Package has multi-stage copy, generating required images")
}
current := 0
// TODO: we should run this only if we are going to build the image
for _, c := range p.Copy {
current++
if c.Package != nil && c.Package.Name != "" && c.Package.Version != "" {
Info(" :droplet: generating multi-stage images for", c.Package.HumanReadableString())
copyTag2 := fmt.Sprintf("%s %d/%d ⤑ :hammer: build %s", copyTag, current, len(p.Copy), c.Package.HumanReadableString())
Info(copyTag2, "generating multi-stage images for", c.Package.HumanReadableString())
spec, err := cs.FromPackage(c.Package)
if err != nil {
return errors.Wrap(err, "while generating images to copy from")
}
noArtifact := false
artifact, err := cs.compile(concurrency, keepPermissions, &noArtifact, spec)
// If we specify --only-target package, we don't want any artifact, otherwise we do
genArtifact := !cs.Options.PackageTargetOnly
spec.SetOutputPath(p.GetOutputPath())
artifact, err := cs.compile(concurrency, keepPermissions, &genArtifact, &genArtifact, spec)
if err != nil {
return errors.Wrap(err, "failed building multi-stage image")
}
@@ -769,6 +919,7 @@ func (cs *LuetCompiler) resolveMultiStageImages(concurrency int, keepPermissions
Source: c.Source,
Destination: c.Destination,
})
Info(copyTag2, ":white_check_mark: Done")
} else {
resolvedCopyFields = append(resolvedCopyFields, c)
}
@@ -777,9 +928,15 @@ func (cs *LuetCompiler) resolveMultiStageImages(concurrency int, keepPermissions
return nil
}
func (cs *LuetCompiler) compile(concurrency int, keepPermissions bool, generateArtifact *bool, p *compilerspec.LuetCompilationSpec) (*artifact.PackageArtifact, error) {
func (cs *LuetCompiler) compile(concurrency int, keepPermissions bool, generateFinalArtifact *bool, generateDependenciesFinalArtifact *bool, p *compilerspec.LuetCompilationSpec) (*artifact.PackageArtifact, error) {
Info(":package: Compiling", p.GetPackage().HumanReadableString(), ".... :coffee:")
//Before multistage : join - same as multistage, but keep artifacts, join them, create a new one and generate a final image.
// When the image is there, use it as a source here, in place of GetImage().
if err := cs.resolveFinalImages(concurrency, keepPermissions, p); err != nil {
return nil, errors.Wrap(err, "while resolving join images")
}
if err := cs.resolveMultiStageImages(concurrency, keepPermissions, p); err != nil {
return nil, errors.Wrap(err, "while resolving multi-stage images")
}
@@ -821,14 +978,16 @@ func (cs *LuetCompiler) compile(concurrency int, keepPermissions bool, generateA
// Treat last case (easier) first. The image is provided and we just compute a plain dockerfile with the images listed as above
if p.GetImage() != "" {
localGenerateArtifact := true
if generateArtifact != nil {
localGenerateArtifact = *generateArtifact
if generateFinalArtifact != nil {
localGenerateArtifact = *generateFinalArtifact
}
a, err := cs.compileWithImage(p.GetImage(), packageHashTree.BuilderImageHash, targetAssertion.Hash.PackageHash, concurrency, keepPermissions, cs.Options.KeepImg, p, localGenerateArtifact)
if err != nil {
return nil, errors.Wrap(err, "building direct image")
}
a.SourceAssertion = p.GetSourceAssertion()
a.PackageCacheImage = targetAssertion.Hash.PackageHash
return a, nil
}
@@ -843,8 +1002,8 @@ func (cs *LuetCompiler) compile(concurrency int, keepPermissions bool, generateA
currentN := 0
packageDeps := !cs.Options.PackageTargetOnly
if generateArtifact != nil {
packageDeps = *generateArtifact
if generateDependenciesFinalArtifact != nil {
packageDeps = *generateDependenciesFinalArtifact
}
buildDeps := !cs.Options.NoDeps
@@ -878,6 +1037,14 @@ func (cs *LuetCompiler) compile(concurrency int, keepPermissions bool, generateA
Assert: assertion,
})
if err := cs.resolveFinalImages(concurrency, keepPermissions, compileSpec); err != nil {
return nil, errors.Wrap(err, "while resolving join images")
}
if err := cs.resolveMultiStageImages(concurrency, keepPermissions, compileSpec); err != nil {
return nil, errors.Wrap(err, "while resolving multi-stage images")
}
buildHash, err := packageHashTree.DependencyBuildImage(assertion.Package)
if err != nil {
return nil, errors.Wrap(err, "failed looking for dependency in hashtree")
@@ -931,8 +1098,8 @@ func (cs *LuetCompiler) compile(concurrency int, keepPermissions bool, generateA
if buildTarget {
localGenerateArtifact := true
if generateArtifact != nil {
localGenerateArtifact = *generateArtifact
if generateFinalArtifact != nil {
localGenerateArtifact = *generateFinalArtifact
}
resolvedSourceImage := cs.resolveExistingImageHash(packageHashTree.SourceHash, p)
Info(":rocket: All dependencies are satisfied, building package requested by the user", p.GetPackage().HumanReadableString())
@@ -1016,7 +1183,7 @@ func (cs *LuetCompiler) templatePackage(vals []map[string]interface{}, pack pkg.
if err != nil {
return nil, errors.Wrap(err, "while marshalling values file")
}
f := filepath.Join(valuesdir, helpers.RandStringRunes(20))
f := filepath.Join(valuesdir, fileHelper.RandStringRunes(20))
if err := ioutil.WriteFile(f, out, os.ModePerm); err != nil {
return nil, errors.Wrap(err, "while writing temporary values file")
}
@@ -1089,6 +1256,11 @@ func (cs *LuetCompiler) FromPackage(p pkg.Package) (*compilerspec.LuetCompilatio
cs.inheritSpecBuildOptions(newSpec)
// Update the package in the compiler database to catch updates from NewLuetCompilationSpec
if err := cs.Database.UpdatePackage(newSpec.Package); err != nil {
return nil, errors.Wrap(err, "failed updating new package entry in compiler database")
}
return newSpec, err
}

View File

@@ -25,6 +25,7 @@ import (
"github.com/mudler/luet/pkg/compiler/types/options"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
helpers "github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/tree"
. "github.com/onsi/ginkgo"
@@ -59,15 +60,15 @@ var _ = Describe("Compiler", func() {
artifact, err := compiler.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("test5"))
content1, err := fileHelper.Read(spec.Rel("test5"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("test6"))
content2, err := fileHelper.Read(spec.Rel("test6"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("artifact5\n"))
Expect(content2).To(Equal("artifact6\n"))
@@ -75,6 +76,68 @@ var _ = Describe("Compiler", func() {
})
})
Context("Copy and Join", func() {
It("Compiles it correctly with Copy", func() {
generalRecipe := tree.NewCompilerRecipe(pkg.NewInMemoryDatabase(false))
err := generalRecipe.Load("../../tests/fixtures/copy")
Expect(err).ToNot(HaveOccurred())
Expect(len(generalRecipe.GetDatabase().GetPackages())).To(Equal(3))
compiler := NewLuetCompiler(sd.NewSimpleDockerBackend(), generalRecipe.GetDatabase(), options.Concurrency(2))
spec, err := compiler.FromPackage(&pkg.DefaultPackage{Name: "c", Category: "test", Version: "1.2"})
Expect(err).ToNot(HaveOccurred())
Expect(spec.GetPackage().GetPath()).ToNot(Equal(""))
tmpdir, err := ioutil.TempDir("", "tree")
Expect(err).ToNot(HaveOccurred())
defer os.RemoveAll(tmpdir) // clean up
spec.SetOutputPath(tmpdir)
artifact, err := compiler.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(fileHelper.Exists(spec.Rel("result"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("bina/busybox"))).To(BeTrue())
})
It("Compiles it correctly with Join", func() {
generalRecipe := tree.NewCompilerRecipe(pkg.NewInMemoryDatabase(false))
err := generalRecipe.Load("../../tests/fixtures/join")
Expect(err).ToNot(HaveOccurred())
Expect(len(generalRecipe.GetDatabase().GetPackages())).To(Equal(3))
compiler := NewLuetCompiler(sd.NewSimpleDockerBackend(), generalRecipe.GetDatabase(), options.Concurrency(2))
spec, err := compiler.FromPackage(&pkg.DefaultPackage{Name: "c", Category: "test", Version: "1.2"})
Expect(err).ToNot(HaveOccurred())
Expect(spec.GetPackage().GetPath()).ToNot(Equal(""))
tmpdir, err := ioutil.TempDir("", "tree")
Expect(err).ToNot(HaveOccurred())
defer os.RemoveAll(tmpdir) // clean up
spec.SetOutputPath(tmpdir)
artifact, err := compiler.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(fileHelper.Exists(spec.Rel("newc"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test4"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test3"))).To(BeTrue())
})
})
Context("Simple package build definition", func() {
It("Compiles it in parallel", func() {
generalRecipe := tree.NewCompilerRecipe(pkg.NewInMemoryDatabase(false))
@@ -102,7 +165,7 @@ var _ = Describe("Compiler", func() {
artifacts, errs := compiler.CompileParallel(false, compilerspec.NewLuetCompilationspecs(spec, spec2))
Expect(errs).To(BeNil())
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
@@ -165,23 +228,23 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(3))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("test3"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test4"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test3"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test4"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("c"))
content1, err := fileHelper.Read(spec.Rel("c"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("cd"))
content2, err := fileHelper.Read(spec.Rel("cd"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("c\n"))
Expect(content2).To(Equal("c\n"))
content1, err = helpers.Read(spec.Rel("d"))
content1, err = fileHelper.Read(spec.Rel("d"))
Expect(err).ToNot(HaveOccurred())
content2, err = helpers.Read(spec.Rel("dd"))
content2, err = fileHelper.Read(spec.Rel("dd"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("s\n"))
Expect(content2).To(Equal("dd\n"))
@@ -215,17 +278,17 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts2)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
for _, artifact := range artifacts2 {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("etc/hosts"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test1"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("etc/hosts"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test1"))).To(BeTrue())
})
It("Compiles and includes ony wanted files", func() {
@@ -254,12 +317,12 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).ToNot(BeTrue())
})
It("Compiles and excludes files", func() {
@@ -288,13 +351,13 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("marvot"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvot"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
})
It("Compiles includes and excludes files", func() {
@@ -323,13 +386,13 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("marvot"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvot"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).ToNot(BeTrue())
})
It("Compiles and excludes ony wanted files also from unpacked packages", func() {
@@ -357,12 +420,12 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("marvin"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvin"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
})
It("Compiles includes and excludes ony wanted files also from unpacked packages", func() {
@@ -390,12 +453,12 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("marvin"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvin"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
})
It("Compiles and includes ony wanted files also from unpacked packages", func() {
@@ -423,16 +486,16 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Exists(spec.Rel("var/lib/udhcpd"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test5"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("test"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("test2"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("lib/firmware"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("var/lib/udhcpd"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("marvin"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test2"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("lib/firmware"))).ToNot(BeTrue())
})
It("Compiles a more complex tree", func() {
@@ -461,18 +524,18 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Untar(spec.Rel("extra-layer-0.1.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("extra-layer"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("extra-layer"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("usr/bin/pkgs-checker"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("base-layer-0.1.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("base-layer-0.1.metadata.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("extra-layer-0.1.metadata.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("extra-layer-0.1.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("usr/bin/pkgs-checker"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("base-layer-0.1.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("base-layer-0.1.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("extra-layer-0.1.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("extra-layer-0.1.package.tar"))).To(BeTrue())
})
It("Compiles with provides support", func() {
@@ -502,19 +565,19 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts[0].Dependencies)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Untar(spec.Rel("c-test-1.0.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("d"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("dd"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("c"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("cd"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("d"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("dd"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("c"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("cd"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("d-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("d-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("c-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("c-test-1.0.metadata.yaml"))).To(BeTrue())
})
It("Compiles with provides and selectors support", func() {
@@ -545,19 +608,19 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts[0].Dependencies)).To(Equal(1))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Untar(spec.Rel("c-test-1.0.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("d"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("dd"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("c"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("cd"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("d"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("dd"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("c"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("cd"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("d-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("d-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("c-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("c-test-1.0.metadata.yaml"))).To(BeTrue())
})
It("Compiles revdeps", func() {
generalRecipe := tree.NewCompilerRecipe(pkg.NewInMemoryDatabase(false))
@@ -585,16 +648,16 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(2))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Untar(spec.Rel("extra-layer-0.1.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("extra-layer"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("extra-layer"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("usr/bin/pkgs-checker"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("base-layer-0.1.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("extra-layer-0.1.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("usr/bin/pkgs-checker"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("base-layer-0.1.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("extra-layer-0.1.package.tar"))).To(BeTrue())
})
It("Compiles complex dependencies trees with best matches", func() {
@@ -623,17 +686,17 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
Expect(len(artifacts[0].Dependencies)).To(Equal(6))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
Expect(helpers.Untar(spec.Rel("vhba-sys-fs-5.4.2-20190410.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("sabayon-build-portage-layer-0.20191126.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("build-layer-0.1.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("build-sabayon-overlay-layer-0.20191212.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("build-sabayon-overlays-layer-0.1.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("linux-sabayon-sys-kernel-5.4.2.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("sabayon-sources-sys-kernel-5.4.2.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("vhba"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("sabayon-build-portage-layer-0.20191126.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("build-layer-0.1.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("build-sabayon-overlay-layer-0.20191212.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("build-sabayon-overlays-layer-0.1.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("linux-sabayon-sys-kernel-5.4.2.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("sabayon-sources-sys-kernel-5.4.2.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("vhba"))).To(BeTrue())
})
It("Compiles revdeps with seeds", func() {
@@ -658,31 +721,31 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(4))
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
}
// A deps on B, so A artifacts are here:
Expect(helpers.Exists(spec.Rel("test3"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test4"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test3"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test4"))).To(BeTrue())
// B
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("artifact42"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("artifact42"))).To(BeTrue())
// C depends on B, so B is here
content1, err := helpers.Read(spec.Rel("c"))
content1, err := fileHelper.Read(spec.Rel("c"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("cd"))
content2, err := fileHelper.Read(spec.Rel("cd"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("c\n"))
Expect(content2).To(Equal("c\n"))
// D is here as it requires C, and C was recompiled
content1, err = helpers.Read(spec.Rel("d"))
content1, err = fileHelper.Read(spec.Rel("d"))
Expect(err).ToNot(HaveOccurred())
content2, err = helpers.Read(spec.Rel("dd"))
content2, err = fileHelper.Read(spec.Rel("dd"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("s\n"))
Expect(content2).To(Equal("dd\n"))
@@ -715,19 +778,19 @@ var _ = Describe("Compiler", func() {
artifacts, errs := compiler.CompileParallel(false, compilerspec.NewLuetCompilationspecs(spec))
Expect(errs).To(BeNil())
for _, artifact := range artifacts {
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
for _, d := range artifact.Dependencies {
Expect(helpers.Exists(d.Path)).To(BeTrue())
Expect(fileHelper.Exists(d.Path)).To(BeTrue())
Expect(helpers.Untar(d.Path, tmpdir, false)).ToNot(HaveOccurred())
}
}
Expect(helpers.Exists(spec.Rel("test3"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test4"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test3"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test4"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
})
})
@@ -759,8 +822,8 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts)).To(Equal(1))
Expect(len(artifacts[0].Dependencies)).To(Equal(1))
Expect(helpers.Untar(spec.Rel("runtime-layer-0.1.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("bin/busybox"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("var"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("bin/busybox"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("var"))).ToNot(BeTrue())
})
})
@@ -807,13 +870,13 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts[0].Dependencies)).To(Equal(0))
Expect(helpers.Untar(spec.Rel("dironly-test-1.0.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test1"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test2"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test1"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test2"))).To(BeTrue())
Expect(helpers.Untar(spec2.Rel("dironly_filter-test-1.0.package.tar"), tmpdir2, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec2.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec2.Rel("test6"))).ToNot(BeTrue())
Expect(helpers.Exists(spec2.Rel("artifact42"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec2.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec2.Rel("test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec2.Rel("artifact42"))).ToNot(BeTrue())
})
})
@@ -843,12 +906,12 @@ var _ = Describe("Compiler", func() {
Expect(errs).To(BeNil())
Expect(len(artifacts)).To(Equal(1))
Expect(len(artifacts[0].Dependencies)).To(Equal(1))
Expect(helpers.Exists(spec.Rel("runtime-layer-0.1.package.tar.gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("runtime-layer-0.1.package.tar"))).To(BeFalse())
Expect(fileHelper.Exists(spec.Rel("runtime-layer-0.1.package.tar.gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("runtime-layer-0.1.package.tar"))).To(BeFalse())
Expect(artifacts[0].Unpack(tmpdir, false)).ToNot(HaveOccurred())
// Expect(helpers.Untar(spec.Rel("runtime-layer-0.1.package.tar"), tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("bin/busybox"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("var"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("bin/busybox"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("var"))).ToNot(BeTrue())
})
})
@@ -899,7 +962,7 @@ var _ = Describe("Compiler", func() {
Expect(len(artifacts[0].Dependencies)).To(Equal(1))
Expect(artifacts[0].Files).To(ContainElement("bin/busybox"))
Expect(helpers.Exists(spec.Rel("runtime-layer-0.1.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("runtime-layer-0.1.metadata.yaml"))).To(BeTrue())
art, err := LoadArtifactFromYaml(spec)
Expect(err).ToNot(HaveOccurred())

View File

@@ -25,11 +25,22 @@ import (
"github.com/pkg/errors"
)
// ImageHashTree is holding the Database
// and the options to resolve PackageImageHashTrees
// for a given specfile
// It is responsible of returning a concrete result
// which identifies a Package in a HashTree
type ImageHashTree struct {
Database pkg.PackageDatabase
SolverOptions config.LuetSolverOptions
}
// PackageImageHashTree represent the Package into a given image hash tree
// The hash tree is constructed by a set of images representing
// the package during its build stage. A Hash is assigned to each image
// from the package fingerprint, plus the SAT solver assertion result (which is hashed as well)
// and the specfile signatures. This guarantees that each image of the build stage
// is unique and can be identified later on.
type PackageImageHashTree struct {
Target *solver.PackageAssert
Dependencies solver.PackagesAssertions
@@ -53,11 +64,19 @@ func (ht *PackageImageHashTree) DependencyBuildImage(p pkg.Package) (string, err
return found, nil
}
// TODO: ___ When computing the hash per package (and evaluating the sat solver solution tree part)
// we should use the hash of each package + its fingerprint instead as a salt.
// That's because the hash will be salted with its `build.yaml`.
// In this way, we trigger recompilations if some dep of a target changes
// a build.yaml, without touching the version
func (ht *PackageImageHashTree) String() string {
return fmt.Sprintf(
"Target buildhash: %s\nTarget packagehash: %s\nBuilder Imagehash: %s\nSource Imagehash: %s\n",
ht.Target.Hash.BuildHash,
ht.Target.Hash.PackageHash,
ht.BuilderImageHash,
ht.SourceHash,
)
}
// Query takes a compiler and a compilation spec and returns a PackageImageHashTree tied to it.
// PackageImageHashTree contains all the informations to resolve the spec build images in order to
// reproducibly re-build images from packages
func (ht *ImageHashTree) Query(cs *LuetCompiler, p *compilerspec.LuetCompilationSpec) (*PackageImageHashTree, error) {
assertions, err := ht.resolve(cs, p)
if err != nil {

View File

@@ -46,14 +46,13 @@ var _ = Describe("ImageHashTree", func() {
packageHash, err := hashtree.Query(compiler, spec)
Expect(err).ToNot(HaveOccurred())
Expect(packageHash.Target.Hash.BuildHash).To(Equal("ec62e3e2cfb4c520c8b2561797c005d248c2659295f3660fa1a66582fc4dc280"))
Expect(packageHash.Target.Hash.PackageHash).To(Equal("5fa15a0eb0534eaa78ef1b4e32fe72704effaa5e54399b7cab6d630aa0aeac5c"))
Expect(packageHash.BuilderImageHash).To(Equal("builder-96e0c42b5741376ebcf0a47c8ec1c481"))
Expect(packageHash.Target.Hash.BuildHash).To(Equal("53993e5a02da4c21ad845371c872f5836fe45ff3a4e3c5ccb6296d0faee2b107"))
Expect(packageHash.Target.Hash.PackageHash).To(Equal("a786d3fd29d0b8bdfe5f304c8bf8be909d5c764cd7059c0e63294a8bff17f3ef"))
Expect(packageHash.BuilderImageHash).To(Equal("builder-0cd3c0d07fc9be568377b3bf1b699e06"))
})
})
expectedPackageHash := "bc6d354e8b9480b70c6f17eafa34cef387b8443ad150b7c9528fb7e94b764e90"
expectedPackageHash := "0d568ac04c4ca528a4e5b67978f2ad3a75d31d443ab20f9d7683b9608cc0d494"
Context("complex package definition", func() {
BeforeEach(func() {
@@ -74,24 +73,26 @@ var _ = Describe("ImageHashTree", func() {
Expect(packageHash.Dependencies[len(packageHash.Dependencies)-1].Hash.PackageHash).To(Equal(expectedPackageHash))
Expect(packageHash.SourceHash).To(Equal(expectedPackageHash))
Expect(packageHash.BuilderImageHash).To(Equal("builder-9b2bc16985446c41eca8f7922ec98078"))
Expect(packageHash.BuilderImageHash).To(Equal("builder-0f45c345f59103e84fc8bebbf02f2e2b"))
//Expect(packageHash.Target.Hash.BuildHash).To(Equal("79d7107d13d578b362e6a7bf10ec850efce26316405b8d732ce8f9e004d64281"))
Expect(packageHash.Target.Hash.PackageHash).To(Equal("bb84a30ced857725fcb575e87fe33d4aefe911abfdd5f9063bbaeb9e4b94e9e2"))
Expect(packageHash.Target.Hash.PackageHash).To(Equal("2e8159583ac825acada763358290cfbea919a33873a926cab84f4f1a67ecf111"))
a := &pkg.DefaultPackage{Name: "a", Category: "test", Version: "1.1"}
hash, err := packageHash.DependencyBuildImage(a)
Expect(err).ToNot(HaveOccurred())
Expect(hash).To(Equal("484f14294d96fd3b51cec1f2db37a269b7b903f3516b74b0cb0771b65d85b799"))
Expect(hash).To(Equal("74c6c833730e9ebd1d9fc669278152b5b58ec7ecb28fdae56658665616076adf"))
assertionA := packageHash.Dependencies.Search(a.GetFingerPrint())
Expect(assertionA.Hash.PackageHash).To(Equal(expectedPackageHash))
b := &pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}
assertionB := packageHash.Dependencies.Search(b.GetFingerPrint())
Expect(assertionB.Hash.PackageHash).To(Equal("484f14294d96fd3b51cec1f2db37a269b7b903f3516b74b0cb0771b65d85b799"))
Expect(assertionB.Hash.PackageHash).To(Equal("74c6c833730e9ebd1d9fc669278152b5b58ec7ecb28fdae56658665616076adf"))
hashB, err := packageHash.DependencyBuildImage(b)
Expect(err).ToNot(HaveOccurred())
Expect(hashB).To(Equal("828c983e755353190540565a29e71c9eb4c48d6303e1fd2c523235b7c2339c73"))
Expect(hashB).To(Equal("315075265aeb2e3c04c5428d31911f53c194ec9fa3db1421e8478f44b1e0def8"))
})
})
@@ -114,33 +115,35 @@ var _ = Describe("ImageHashTree", func() {
Expect(err).ToNot(HaveOccurred())
Expect(packageHash.Dependencies[len(packageHash.Dependencies)-1].Hash.PackageHash).ToNot(Equal(expectedPackageHash))
sourceHash := "ed1bd90e696904982a1f51998646a335067329e1a262994b5ae15c579106ac81"
sourceHash := "66ec001fe72052d0e605ca96f607ae39ea4f8b53f0b7f762e377622d9c654de3"
Expect(packageHash.Dependencies[len(packageHash.Dependencies)-1].Hash.PackageHash).To(Equal(sourceHash))
Expect(packageHash.SourceHash).To(Equal(sourceHash))
Expect(packageHash.SourceHash).ToNot(Equal(expectedPackageHash))
Expect(packageHash.BuilderImageHash).To(Equal("builder-f4b0e366e0a42774428fbdc9aa325648"))
Expect(packageHash.BuilderImageHash).To(Equal("builder-ffc02fd8aaa916d0e17249885b3226b1"))
//Expect(packageHash.Target.Hash.BuildHash).To(Equal("79d7107d13d578b362e6a7bf10ec850efce26316405b8d732ce8f9e004d64281"))
Expect(packageHash.Target.Hash.PackageHash).To(Equal("2618f12851a596f6801e2665e07147da98a0a151f44500a54ca8b76b869e378d"))
Expect(packageHash.Target.Hash.PackageHash).To(Equal("b9c0286ebf6d28be831926ec7da9cb3cda6b489722d656aefc363ebd7173f937"))
a := &pkg.DefaultPackage{Name: "a", Category: "test", Version: "1.1"}
hash, err := packageHash.DependencyBuildImage(a)
Expect(err).ToNot(HaveOccurred())
Expect(hash).To(Equal("484f14294d96fd3b51cec1f2db37a269b7b903f3516b74b0cb0771b65d85b799"))
Expect(hash).To(Equal("74c6c833730e9ebd1d9fc669278152b5b58ec7ecb28fdae56658665616076adf"))
assertionA := packageHash.Dependencies.Search(a.GetFingerPrint())
Expect(assertionA.Hash.PackageHash).To(Equal("ed1bd90e696904982a1f51998646a335067329e1a262994b5ae15c579106ac81"))
Expect(assertionA.Hash.PackageHash).To(Equal("66ec001fe72052d0e605ca96f607ae39ea4f8b53f0b7f762e377622d9c654de3"))
Expect(assertionA.Hash.PackageHash).ToNot(Equal(expectedPackageHash))
b := &pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}
assertionB := packageHash.Dependencies.Search(b.GetFingerPrint())
Expect(assertionB.Hash.PackageHash).To(Equal("484f14294d96fd3b51cec1f2db37a269b7b903f3516b74b0cb0771b65d85b799"))
Expect(assertionB.Hash.PackageHash).To(Equal("74c6c833730e9ebd1d9fc669278152b5b58ec7ecb28fdae56658665616076adf"))
hashB, err := packageHash.DependencyBuildImage(b)
Expect(err).ToNot(HaveOccurred())
Expect(hashB).To(Equal("828c983e755353190540565a29e71c9eb4c48d6303e1fd2c523235b7c2339c73"))
Expect(hashB).To(Equal("315075265aeb2e3c04c5428d31911f53c194ec9fa3db1421e8478f44b1e0def8"))
})
})

View File

@@ -42,6 +42,7 @@ import (
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
. "github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/solver"
@@ -168,7 +169,7 @@ func CreateArtifactForFile(s string, opts ...func(*PackageArtifact)) (*PackageAr
}
defer os.RemoveAll(archive) // clean up
dst := filepath.Join(archive, fileName)
if err := helpers.CopyFile(s, dst); err != nil {
if err := fileHelper.CopyFile(s, dst); err != nil {
return nil, errors.Wrapf(err, "error while copying %s to %s", s, dst)
}
@@ -209,7 +210,7 @@ func (a *PackageArtifact) GenerateFinalImage(imageName string, b ImageBuilder, k
return builderOpts, errors.Wrap(err, "error met while uncompressing artifact "+a.Path)
}
empty, err := helpers.DirectoryIsEmpty(uncompressedFiles)
empty, err := fileHelper.DirectoryIsEmpty(uncompressedFiles)
if err != nil {
return builderOpts, errors.Wrap(err, "error met while checking if directory is empty "+uncompressedFiles)
}
@@ -218,7 +219,7 @@ func (a *PackageArtifact) GenerateFinalImage(imageName string, b ImageBuilder, k
// We can't generate FROM scratch empty images. Docker will refuse to export them
// workaround: Inject a .virtual empty file
if empty {
helpers.Touch(filepath.Join(uncompressedFiles, ".virtual"))
fileHelper.Touch(filepath.Join(uncompressedFiles, ".virtual"))
}
data := a.genDockerfile()
@@ -401,12 +402,12 @@ func tarModifierWrapperFunc(dst, path string, header *tar.Header, content io.Rea
// We want to protect file only if the hash of the files are differing OR the file size are
differs := (existingHash != "" && existingHash != tarHash) || (err != nil && f != nil && header.Size != f.Size())
// Check if exists
if helpers.Exists(destPath) && differs {
if fileHelper.Exists(destPath) && differs {
for i := 1; i < 1000; i++ {
name := filepath.Join(filepath.Join(filepath.Dir(path),
fmt.Sprintf("._cfg%04d_%s", i, filepath.Base(path))))
if helpers.Exists(name) {
if fileHelper.Exists(name) {
continue
}
Info(fmt.Sprintf("Found protected file %s. Creating %s.", destPath,
@@ -452,6 +453,9 @@ func (a *PackageArtifact) GetProtectFiles() []string {
// Unpack Untar and decompress (TODO) to the given path
func (a *PackageArtifact) Unpack(dst string, keepPerms bool) error {
if !strings.HasPrefix(dst, "/") {
return errors.New("destination must be an absolute path")
}
// Create
protectedFiles := a.GetProtectFiles()
@@ -628,7 +632,7 @@ func worker(i int, wg *sync.WaitGroup, s <-chan CopyJob) {
_, err := os.Lstat(job.Dst)
if err != nil {
Debug("Copying ", job.Src)
if err := helpers.DeepCopyFile(job.Src, job.Dst); err != nil {
if err := fileHelper.DeepCopyFile(job.Src, job.Dst); err != nil {
Warning("Error copying", job, err)
}
}

View File

@@ -29,6 +29,7 @@ import (
. "github.com/mudler/luet/pkg/compiler"
helpers "github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/tree"
. "github.com/onsi/ginkgo"
@@ -71,7 +72,7 @@ var _ = Describe("Artifact", func() {
err = lspec.WriteBuildImageDefinition(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err := helpers.Read(filepath.Join(tmpdir, "Dockerfile"))
dockerfile, err := fileHelper.Read(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM alpine
@@ -89,12 +90,12 @@ ENV PACKAGE_CATEGORY=app-admin`))
}
Expect(b.BuildImage(opts)).ToNot(HaveOccurred())
Expect(b.ExportImage(opts)).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(tmpdir2, "output1.tar"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(tmpdir2, "output1.tar"))).To(BeTrue())
Expect(b.BuildImage(opts)).ToNot(HaveOccurred())
err = lspec.WriteStepImageDefinition(lspec.Image, filepath.Join(tmpdir, "LuetDockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err = helpers.Read(filepath.Join(tmpdir, "LuetDockerfile"))
dockerfile, err = fileHelper.Read(filepath.Join(tmpdir, "LuetDockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM luet/base
@@ -113,7 +114,7 @@ RUN echo bar > /test2`))
}
Expect(b.BuildImage(opts2)).ToNot(HaveOccurred())
Expect(b.ExportImage(opts2)).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(tmpdir, "output2.tar"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(tmpdir, "output2.tar"))).To(BeTrue())
diffs, err := compiler.GenerateChanges(b, opts, opts2)
Expect(err).ToNot(HaveOccurred())
@@ -140,15 +141,15 @@ RUN echo bar > /test2`))
a, err := ExtractArtifactFromDelta(rootfs, filepath.Join(tmpdir, "package.tar"), diffs, 2, false, []string{}, []string{}, compression.None)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(tmpdir, "package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(tmpdir, "package.tar"))).To(BeTrue())
err = helpers.Untar(a.Path, unpacked, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(unpacked, "test"))).To(BeTrue())
Expect(helpers.Exists(filepath.Join(unpacked, "test2"))).To(BeTrue())
content1, err := helpers.Read(filepath.Join(unpacked, "test"))
Expect(fileHelper.Exists(filepath.Join(unpacked, "test"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(unpacked, "test2"))).To(BeTrue())
content1, err := fileHelper.Read(filepath.Join(unpacked, "test"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("foo\n"))
content2, err := helpers.Read(filepath.Join(unpacked, "test2"))
content2, err := fileHelper.Read(filepath.Join(unpacked, "test2"))
Expect(err).ToNot(HaveOccurred())
Expect(content2).To(Equal("bar\n"))
@@ -156,7 +157,7 @@ RUN echo bar > /test2`))
Expect(err).ToNot(HaveOccurred())
err = a.Verify()
Expect(err).ToNot(HaveOccurred())
Expect(helpers.CopyFile(filepath.Join(tmpdir, "output2.tar"), filepath.Join(tmpdir, "package.tar"))).ToNot(HaveOccurred())
Expect(fileHelper.CopyFile(filepath.Join(tmpdir, "output2.tar"), filepath.Join(tmpdir, "package.tar"))).ToNot(HaveOccurred())
err = a.Verify()
Expect(err).To(HaveOccurred())
@@ -244,7 +245,7 @@ RUN echo bar > /test2`))
err = b.ExtractRootfs(backend.Options{ImageName: resultingImage, Destination: result}, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.DirectoryIsEmpty(result)).To(BeFalse())
Expect(fileHelper.DirectoryIsEmpty(result)).To(BeFalse())
content, err := ioutil.ReadFile(filepath.Join(result, ".virtual"))
Expect(err).ToNot(HaveOccurred())

View File

@@ -23,10 +23,11 @@ import (
"github.com/mitchellh/hashstructure/v2"
options "github.com/mudler/luet/pkg/compiler/types/options"
"github.com/ghodss/yaml"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/solver"
"github.com/otiai10/copy"
yaml "gopkg.in/yaml.v2"
dirhash "golang.org/x/mod/sumdb/dirhash"
)
type LuetCompilationspecs []LuetCompilationSpec
@@ -114,15 +115,73 @@ type LuetCompilationSpec struct {
BuildOptions *options.Compiler `json:"build_options"`
Copy []CopyField `json:"copy"`
Join pkg.DefaultPackages `json:"join"`
RequiresFinalImages bool `json:"requires_final_images" yaml:"requires_final_images"`
}
// Signature is a portion of the spec that yields a signature for the hash
type Signature struct {
Image string
Steps []string
PackageDir string
Prelude []string
Seed string
Env []string
Retrieve []string
Unpack bool
Includes []string
Excludes []string
Copy []CopyField
Join pkg.DefaultPackages
RequiresFinalImages bool
}
func (cs *LuetCompilationSpec) signature() Signature {
return Signature{
Image: cs.Image,
Steps: cs.Steps,
PackageDir: cs.PackageDir,
Prelude: cs.Prelude,
Seed: cs.Seed,
Env: cs.Env,
Retrieve: cs.Retrieve,
Unpack: cs.Unpack,
Includes: cs.Includes,
Excludes: cs.Excludes,
Copy: cs.Copy,
Join: cs.Join,
RequiresFinalImages: cs.RequiresFinalImages,
}
}
func NewLuetCompilationSpec(b []byte, p pkg.Package) (*LuetCompilationSpec, error) {
var spec LuetCompilationSpec
var packageDefinition pkg.DefaultPackage
err := yaml.Unmarshal(b, &spec)
if err != nil {
return &spec, err
}
spec.Package = p.(*pkg.DefaultPackage)
err = yaml.Unmarshal(b, &packageDefinition)
if err != nil {
return &spec, err
}
// Update requires/conflict/provides
// When we have been passed a bytes slice, parse it as a package
// and updates requires/conflicts/provides.
// This is required in order to allow manipulation of such fields with templating
copy := *p.(*pkg.DefaultPackage)
spec.Package = &copy
if len(packageDefinition.GetRequires()) != 0 {
spec.Package.Requires(packageDefinition.GetRequires())
}
if len(packageDefinition.GetConflicts()) != 0 {
spec.Package.Conflicts(packageDefinition.GetConflicts())
}
if len(packageDefinition.GetProvides()) != 0 {
spec.Package.SetProvides(packageDefinition.GetProvides())
}
return &spec, nil
}
func (cs *LuetCompilationSpec) GetSourceAssertion() solver.PackagesAssertions {
@@ -207,7 +266,7 @@ func (cs *LuetCompilationSpec) SetSeedImage(s string) {
}
func (cs *LuetCompilationSpec) EmptyPackage() bool {
return len(cs.BuildSteps()) == 0 && len(cs.GetPreBuildSteps()) == 0 && !cs.UnpackedPackage()
return len(cs.BuildSteps()) == 0 && !cs.UnpackedPackage()
}
func (cs *LuetCompilationSpec) UnpackedPackage() bool {
@@ -224,45 +283,21 @@ func (cs *LuetCompilationSpec) UnpackedPackage() bool {
// a compilation spec has an image source when it depends on other packages or have a source image
// explictly supplied
func (cs *LuetCompilationSpec) HasImageSource() bool {
return (cs.Package != nil && len(cs.GetPackage().GetRequires()) != 0) || cs.GetImage() != ""
}
// Signature is a portion of the spec that yields a signature for the hash
type Signature struct {
Image string
Steps []string
PackageDir string
Prelude []string
Seed string
Env []string
Retrieve []string
Unpack bool
Includes []string
Excludes []string
Copy []CopyField
}
func (cs *LuetCompilationSpec) signature() Signature {
return Signature{
Image: cs.Image,
Steps: cs.Steps,
PackageDir: cs.PackageDir,
Prelude: cs.Prelude,
Seed: cs.Seed,
Env: cs.Env,
Retrieve: cs.Retrieve,
Unpack: cs.Unpack,
Includes: cs.Includes,
Excludes: cs.Excludes,
Copy: cs.Copy,
}
return (cs.Package != nil && len(cs.GetPackage().GetRequires()) != 0) || cs.GetImage() != "" || len(cs.Join) != 0
}
func (cs *LuetCompilationSpec) Hash() (string, error) {
// build a signature, we want to be part of the hash only the fields that are relevant for build purposes
signature := cs.signature()
h, err := hashstructure.Hash(signature, hashstructure.FormatV2, nil)
return fmt.Sprint(h), err
if err != nil {
return "", err
}
sum, err := dirhash.HashDir(cs.Package.Path, "", dirhash.DefaultHash)
if err != nil {
return fmt.Sprint(h), err
}
return fmt.Sprint(h, sum), err
}
func (cs *LuetCompilationSpec) CopyRetrieves(dest string) error {

View File

@@ -22,9 +22,9 @@ import (
options "github.com/mudler/luet/pkg/compiler/types/options"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/compiler"
helpers "github.com/mudler/luet/pkg/helpers"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/tree"
. "github.com/onsi/ginkgo"
@@ -154,7 +154,7 @@ var _ = Describe("Spec", func() {
lspec.Env = []string{"test=1"}
err = lspec.WriteBuildImageDefinition(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err := helpers.Read(filepath.Join(tmpdir, "Dockerfile"))
dockerfile, err := fileHelper.Read(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM alpine
@@ -167,7 +167,7 @@ ENV test=1`))
err = lspec.WriteStepImageDefinition(lspec.Image, filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err = helpers.Read(filepath.Join(tmpdir, "Dockerfile"))
dockerfile, err = fileHelper.Read(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM luet/base
@@ -205,7 +205,7 @@ RUN echo bar > /test2`))
err = lspec.WriteBuildImageDefinition(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err := helpers.Read(filepath.Join(tmpdir, "Dockerfile"))
dockerfile, err := fileHelper.Read(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM alpine
@@ -222,7 +222,7 @@ ENV test=1`))
err = lspec.WriteBuildImageDefinition(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err = helpers.Read(filepath.Join(tmpdir, "Dockerfile"))
dockerfile, err = fileHelper.Read(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`
FROM alpine
@@ -237,7 +237,7 @@ ENV test=1`))
err = lspec.WriteStepImageDefinition(lspec.Image, filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
dockerfile, err = helpers.Read(filepath.Join(tmpdir, "Dockerfile"))
dockerfile, err = fileHelper.Read(filepath.Join(tmpdir, "Dockerfile"))
Expect(err).ToNot(HaveOccurred())
Expect(dockerfile).To(Equal(`

View File

@@ -27,7 +27,6 @@ import (
"strings"
"time"
"github.com/mudler/luet/pkg/helpers"
pkg "github.com/mudler/luet/pkg/package"
solver "github.com/mudler/luet/pkg/solver"
@@ -108,6 +107,19 @@ type LuetSystemConfig struct {
TmpDirBase string `yaml:"tmpdir_base" mapstructure:"tmpdir_base"`
}
func (s *LuetSystemConfig) SetRootFS(path string) error {
pathToSet := path
if !filepath.IsAbs(path) {
abs, err := filepath.Abs(path)
if err != nil {
return err
}
pathToSet = abs
}
s.Rootfs = pathToSet
return nil
}
func (sc *LuetSystemConfig) GetRepoDatabaseDirPath(name string) string {
dbpath := filepath.Join(sc.Rootfs, sc.DatabasePath)
dbpath = filepath.Join(dbpath, "repos/"+name)
@@ -406,8 +418,13 @@ system:
}
func (c *LuetSystemConfig) InitTmpDir() error {
if !helpers.Exists(c.TmpDirBase) {
return os.MkdirAll(c.TmpDirBase, os.ModePerm)
if _, err := os.Stat(c.TmpDirBase); err != nil {
if os.IsNotExist(err) {
err = os.MkdirAll(c.TmpDirBase, os.ModePerm)
if err != nil {
return err
}
}
}
return nil
}

View File

@@ -22,7 +22,7 @@ import (
"strings"
config "github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
@@ -38,7 +38,7 @@ var _ = Describe("Config", func() {
tmpDir, err := config.LuetCfg.GetSystem().TempDir("test1")
Expect(err).ToNot(HaveOccurred())
Expect(strings.HasPrefix(tmpDir, filepath.Join(os.TempDir(), "tmpluet"))).To(BeTrue())
Expect(helpers.Exists(tmpDir)).To(BeTrue())
Expect(fileHelper.Exists(tmpDir)).To(BeTrue())
defer os.RemoveAll(tmpDir)
})
@@ -49,7 +49,7 @@ var _ = Describe("Config", func() {
tmpFile, err := config.LuetCfg.GetSystem().TempFile("testfile1")
Expect(err).ToNot(HaveOccurred())
Expect(strings.HasPrefix(tmpFile.Name(), filepath.Join(os.TempDir(), "tmpluet"))).To(BeTrue())
Expect(helpers.Exists(tmpFile.Name())).To(BeTrue())
Expect(fileHelper.Exists(tmpFile.Name())).To(BeTrue())
defer os.Remove(tmpFile.Name())
})

View File

@@ -25,6 +25,8 @@ import (
"os"
"path/filepath"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
"github.com/docker/docker/pkg/archive"
. "github.com/mudler/luet/pkg/helpers"
. "github.com/onsi/ginkgo"
@@ -128,7 +130,7 @@ var _ = Describe("Helpers Archive", func() {
err = archive.Untar(replacerArchive, targetDir, opts)
Expect(err).ToNot(HaveOccurred())
Expect(Exists(filepath.Join(targetDir, "._cfg0001_file-0"))).Should(Equal(true))
Expect(fileHelper.Exists(filepath.Join(targetDir, "._cfg0001_file-0"))).Should(Equal(true))
})
})
})

View File

@@ -13,20 +13,32 @@
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package helpers
package docker
import (
"context"
"encoding/hex"
"fmt"
"os"
"strings"
"github.com/containerd/containerd/images"
"github.com/mudler/luet/pkg/helpers/imgworker"
continerdarchive "github.com/containerd/containerd/archive"
"github.com/docker/cli/cli/trust"
"github.com/docker/distribution/reference"
"github.com/docker/docker/api/types"
"github.com/docker/docker/pkg/archive"
"github.com/docker/docker/registry"
"github.com/mudler/luet/pkg/helpers/imgworker"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/mutate"
"github.com/google/go-containerregistry/pkg/v1/remote"
"github.com/mudler/luet/pkg/bus"
"github.com/opencontainers/go-digest"
specs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
"github.com/theupdateframework/notary/tuf/data"
)
@@ -94,18 +106,31 @@ func trustedResolveDigest(ctx context.Context, ref reference.NamedTagged, authCo
return reference.WithDigest(ref, dgst)
}
// DownloadAndExtractDockerImage is a re-adaption
// from genuinetools/img https://github.com/genuinetools/img/blob/54d0ca981c1260546d43961a538550eef55c87cf/pull.go
func DownloadAndExtractDockerImage(temp, image, dest string, auth *types.AuthConfig, verify bool) (*imgworker.ListedImage, error) {
type staticAuth struct {
auth *types.AuthConfig
}
if verify {
img, err := verifyImage(image, auth)
if err != nil {
return nil, errors.Wrapf(err, "failed verifying image")
}
image = img
func (s staticAuth) Authorization() (*authn.AuthConfig, error) {
if s.auth == nil {
return nil, nil
}
return &authn.AuthConfig{
Username: s.auth.Username,
Password: s.auth.Password,
Auth: s.auth.Auth,
IdentityToken: s.auth.IdentityToken,
RegistryToken: s.auth.RegistryToken,
}, nil
}
// UnpackEventData is the data structure to pass for the bus events
type UnpackEventData struct {
Image string
Dest string
}
// privilegedExtractImage uses the imgworker (which requires privileges) to extract a container image
func privilegedExtractImage(temp, image, dest string, auth *types.AuthConfig, verify bool) (*imgworker.ListedImage, error) {
defer os.RemoveAll(temp)
c, err := imgworker.New(temp, auth)
if err != nil {
@@ -116,14 +141,121 @@ func DownloadAndExtractDockerImage(temp, image, dest string, auth *types.AuthCon
listedImage, err := c.Pull(image)
if err != nil {
return nil, errors.Wrapf(err, "failed listing images")
}
os.RemoveAll(dest)
bus.Manager.Publish(bus.EventImagePreUnPack, UnpackEventData{Image: image, Dest: dest})
err = c.Unpack(image, dest)
bus.Manager.Publish(bus.EventImagePostUnPack, UnpackEventData{Image: image, Dest: dest})
return listedImage, err
}
// UnarchiveLayers extract layers with archive.Untar from docker instead of containerd
func UnarchiveLayers(temp string, img v1.Image, image, dest string, auth *types.AuthConfig, verify bool) (int64, error) {
layers, err := img.Layers()
if err != nil {
return 0, fmt.Errorf("reading layers from '%s' image failed: %v", image, err)
}
bus.Manager.Publish(bus.EventImagePreUnPack, UnpackEventData{Image: image, Dest: dest})
var size int64
for _, l := range layers {
s, err := l.Size()
if err != nil {
return 0, fmt.Errorf("reading layer size from '%s' image failed: %v", image, err)
}
size += s
layerReader, err := l.Uncompressed()
if err != nil {
return 0, fmt.Errorf("reading uncompressed layer from '%s' image failed: %v", image, err)
}
defer layerReader.Close()
// Unpack the tarfile to the rootfs path.
// FROM: https://godoc.org/github.com/moby/moby/pkg/archive#TarOptions
if err := archive.Untar(layerReader, dest, &archive.TarOptions{
NoLchown: false,
ExcludePatterns: []string{"dev/"}, // prevent 'operation not permitted'
}); err != nil {
return 0, fmt.Errorf("extracting '%s' image to directory %s failed: %v", image, dest, err)
}
}
bus.Manager.Publish(bus.EventImagePostUnPack, UnpackEventData{Image: image, Dest: dest})
return size, nil
}
// DownloadAndExtractDockerImage extracts a container image natively. It supports privileged/unprivileged mode
func DownloadAndExtractDockerImage(temp, image, dest string, auth *types.AuthConfig, verify bool) (*imgworker.ListedImage, error) {
if verify {
img, err := verifyImage(image, auth)
if err != nil {
return nil, errors.Wrapf(err, "failed verifying image")
}
image = img
}
if os.Getenv("LUET_PRIVILEGED_EXTRACT") == "true" {
return privilegedExtractImage(temp, image, dest, auth, verify)
}
ref, err := name.ParseReference(image)
if err != nil {
return nil, err
}
img, err := remote.Image(ref, remote.WithAuth(staticAuth{auth}))
if err != nil {
return nil, err
}
m, err := img.Manifest()
if err != nil {
return nil, err
}
mt, err := img.MediaType()
if err != nil {
return nil, err
}
d, err := img.Digest()
if err != nil {
return nil, err
}
reader := mutate.Extract(img)
defer reader.Close()
defer os.RemoveAll(temp)
bus.Manager.Publish(bus.EventImagePreUnPack, UnpackEventData{Image: image, Dest: dest})
c, err := continerdarchive.Apply(context.TODO(), dest, reader)
if err != nil {
return nil, err
}
bus.Manager.Publish(bus.EventImagePostUnPack, UnpackEventData{Image: image, Dest: dest})
return &imgworker.ListedImage{
Image: images.Image{
Name: image,
Labels: m.Annotations,
Target: specs.Descriptor{
MediaType: string(mt),
Digest: digest.Digest(d.String()),
Size: c,
},
},
ContentSize: c,
}, nil
}
func StripInvalidStringsFromImage(s string) string {
return strings.ReplaceAll(s, "+", "-")
}

View File

@@ -13,10 +13,10 @@
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package helpers_test
package docker_test
import (
. "github.com/mudler/luet/pkg/helpers"
"github.com/mudler/luet/pkg/helpers/docker"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
@@ -24,7 +24,7 @@ import (
var _ = Describe("StripInvalidStringsFromImage", func() {
Context("Image names", func() {
It("strips invalid chars", func() {
Expect(StripInvalidStringsFromImage("foo+bar")).To(Equal("foo-bar"))
Expect(docker.StripInvalidStringsFromImage("foo+bar")).To(Equal("foo-bar"))
})
})
})

View File

@@ -13,7 +13,7 @@
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package helpers
package file
import (
"fmt"

View File

@@ -13,14 +13,15 @@
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package helpers_test
package file_test
import (
"io/ioutil"
"os"
"path/filepath"
. "github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
@@ -28,8 +29,8 @@ import (
var _ = Describe("Helpers", func() {
Context("Exists", func() {
It("Detect existing and not-existing files", func() {
Expect(Exists("../../tests/fixtures/buildtree/app-admin/enman/1.4.0/build.yaml")).To(BeTrue())
Expect(Exists("../../tests/fixtures/buildtree/app-admin/enman/1.4.0/build.yaml.not.exists")).To(BeFalse())
Expect(fileHelper.Exists("../../tests/fixtures/buildtree/app-admin/enman/1.4.0/build.yaml")).To(BeTrue())
Expect(fileHelper.Exists("../../tests/fixtures/buildtree/app-admin/enman/1.4.0/build.yaml.not.exists")).To(BeFalse())
})
})
@@ -38,15 +39,15 @@ var _ = Describe("Helpers", func() {
testDir, err := ioutil.TempDir(os.TempDir(), "test")
Expect(err).ToNot(HaveOccurred())
defer os.RemoveAll(testDir)
Expect(DirectoryIsEmpty(testDir)).To(BeTrue())
Expect(fileHelper.DirectoryIsEmpty(testDir)).To(BeTrue())
})
It("Detects directory with files", func() {
testDir, err := ioutil.TempDir(os.TempDir(), "test")
Expect(err).ToNot(HaveOccurred())
defer os.RemoveAll(testDir)
err = Touch(filepath.Join(testDir, "foo"))
err = fileHelper.Touch(filepath.Join(testDir, "foo"))
Expect(err).ToNot(HaveOccurred())
Expect(DirectoryIsEmpty(testDir)).To(BeFalse())
Expect(fileHelper.DirectoryIsEmpty(testDir)).To(BeFalse())
})
})
@@ -72,7 +73,7 @@ var _ = Describe("Helpers", func() {
err = ioutil.WriteFile(filepath.Join(testDir, "baz2", "foo"), []byte("test\n"), 0644)
Expect(err).ToNot(HaveOccurred())
ordered, notExisting := OrderFiles(testDir, []string{"bar", "baz", "bar/foo", "baz2", "foo", "baz2/foo", "notexisting"})
ordered, notExisting := fileHelper.OrderFiles(testDir, []string{"bar", "baz", "bar/foo", "baz2", "foo", "baz2/foo", "notexisting"})
Expect(ordered).To(Equal([]string{"baz", "bar/foo", "foo", "baz2/foo", "bar", "baz2"}))
Expect(notExisting).To(Equal([]string{"notexisting"}))
@@ -96,7 +97,7 @@ var _ = Describe("Helpers", func() {
err = os.MkdirAll(filepath.Join(testDir, "foo", "baz", "fa"), os.ModePerm)
Expect(err).ToNot(HaveOccurred())
ordered, _ := OrderFiles(testDir, []string{"foo", "foo/bar", "bar", "foo/baz/fa", "foo/baz"})
ordered, _ := fileHelper.OrderFiles(testDir, []string{"foo", "foo/bar", "bar", "foo/baz/fa", "foo/baz"})
Expect(ordered).To(Equal([]string{"foo/baz/fa", "foo/bar", "foo/baz", "foo", "bar"}))
})
})

View File

@@ -3,6 +3,8 @@ package helpers
import (
"io/ioutil"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
"github.com/imdario/mergo"
"github.com/pkg/errors"
"gopkg.in/yaml.v2"
@@ -75,7 +77,7 @@ func RenderFiles(toTemplate, valuesFile string, defaultFile ...string) (string,
return "", errors.Wrap(err, "reading file "+toTemplate)
}
if !Exists(valuesFile) {
if !fileHelper.Exists(valuesFile) {
return "", errors.Wrap(err, "file not existing "+valuesFile)
}
val, err := ioutil.ReadFile(valuesFile)

View File

@@ -5,6 +5,7 @@ package imgworker
import (
"errors"
"fmt"
"github.com/mudler/luet/pkg/bus"
"os"
"github.com/containerd/containerd/content"
@@ -18,6 +19,12 @@ import (
// TODO: this requires root permissions to mount/unmount layers, althrought it shouldn't be required.
// See how backends are unpacking images without asking for root permissions.
// UnpackEventData is the data structure to pass for the bus events
type UnpackEventData struct {
Image string
Dest string
}
// Unpack exports an image to a rootfs destination directory.
func (c *Client) Unpack(image, dest string) error {
@@ -59,6 +66,8 @@ func (c *Client) Unpack(image, dest string) error {
return fmt.Errorf("getting image manifest failed: %v", err)
}
_,_ = bus.Manager.Publish(bus.EventImagePreUnPack, UnpackEventData{Image: image, Dest: dest})
for _, desc := range manifest.Layers {
logrus.Debugf("Unpacking layer %s", desc.Digest.String())
@@ -78,5 +87,7 @@ func (c *Client) Unpack(image, dest string) error {
}
}
_, _ = bus.Manager.Publish(bus.EventImagePostUnPack, UnpackEventData{Image: image, Dest: dest})
return nil
}

View File

@@ -14,7 +14,7 @@
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package helpers
package match
import (
"reflect"

10
pkg/helpers/slice.go Normal file
View File

@@ -0,0 +1,10 @@
package helpers
func Contains(s []string, e string) bool {
for _, a := range s {
if a == e {
return true
}
}
return false
}

View File

@@ -28,8 +28,8 @@ import (
"github.com/mudler/luet/pkg/compiler/types/artifact"
"github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
"github.com/mudler/luet/pkg/helpers/imgworker"
"github.com/mudler/luet/pkg/helpers/docker"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/logger"
)
@@ -64,7 +64,7 @@ func (c *DockerClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.
artifactName := path.Base(a.Path)
cacheFile := filepath.Join(config.LuetCfg.GetSystem().GetSystemPkgsCacheDirPath(), artifactName)
Debug("Cache file", cacheFile)
if err := helpers.EnsureDir(cacheFile); err != nil {
if err := fileHelper.EnsureDir(cacheFile); err != nil {
return nil, errors.Wrapf(err, "could not create cache folder %s for %s", config.LuetCfg.GetSystem().GetSystemPkgsCacheDirPath(), cacheFile)
}
ok := false
@@ -77,7 +77,7 @@ func (c *DockerClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.
// is done in such cases (see repository.go)
// Check if file is already in cache
if helpers.Exists(cacheFile) {
if fileHelper.Exists(cacheFile) {
Debug("Cache hit for artifact", artifactName)
resultingArtifact = a
resultingArtifact.Path = cacheFile
@@ -102,7 +102,7 @@ func (c *DockerClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.
}
// imageName := fmt.Sprintf("%s/%s", uri, artifact.GetCompileSpec().GetPackage().GetPackageImageName())
info, err := helpers.DownloadAndExtractDockerImage(contentstore, imageName, temp, c.auth, c.RepoData.Verify)
info, err := docker.DownloadAndExtractDockerImage(contentstore, imageName, temp, c.auth, c.RepoData.Verify)
if err != nil {
Warning(fmt.Sprintf(errImageDownloadMsg, imageName, err.Error()))
continue
@@ -140,7 +140,6 @@ func (c *DockerClient) DownloadFile(name string) (string, error) {
var file *os.File = nil
var err error
var temp, contentstore string
var info *imgworker.ListedImage
// Files should be in URI/repository:<file>
ok := false
@@ -161,10 +160,10 @@ func (c *DockerClient) DownloadFile(name string) (string, error) {
continue
}
imageName := fmt.Sprintf("%s:%s", uri, helpers.StripInvalidStringsFromImage(name))
imageName := fmt.Sprintf("%s:%s", uri, docker.StripInvalidStringsFromImage(name))
Info("Downloading", imageName)
info, err = helpers.DownloadAndExtractDockerImage(contentstore, imageName, temp, c.auth, c.RepoData.Verify)
info, err := docker.DownloadAndExtractDockerImage(contentstore, imageName, temp, c.auth, c.RepoData.Verify)
if err != nil {
Warning(fmt.Sprintf(errImageDownloadMsg, imageName, err.Error()))
continue
@@ -174,7 +173,7 @@ func (c *DockerClient) DownloadFile(name string) (string, error) {
Info(fmt.Sprintf("Size: %s", units.BytesSize(float64(info.ContentSize))))
Debug("\nCopying file ", filepath.Join(temp, name), "to", file.Name())
err = helpers.CopyFile(filepath.Join(temp, name), file.Name())
err = fileHelper.CopyFile(filepath.Join(temp, name), file.Name())
if err != nil {
continue
}

View File

@@ -22,8 +22,8 @@ import (
"github.com/mudler/luet/pkg/compiler/types/artifact"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
helpers "github.com/mudler/luet/pkg/helpers"
pkg "github.com/mudler/luet/pkg/package"
. "github.com/mudler/luet/pkg/installer/client"
@@ -32,7 +32,7 @@ import (
)
// This test expect that the repository defined in UNIT_TEST_DOCKER_IMAGE is in zstd format.
// the repository is built by the 01_simple_docker.sh integration test file.
// the repository is built by the 01_simple_docker.sh integration test fileHelper.
// This test also require root. At the moment, unpacking docker images with 'img' requires root permission to
// mount/unmount layers.
var _ = Describe("Docker client", func() {
@@ -51,7 +51,7 @@ var _ = Describe("Docker client", func() {
It("Downloads single files", func() {
f, err := c.DownloadFile("repository.yaml")
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Read(f)).To(ContainSubstring("Test Repo"))
Expect(fileHelper.Read(f)).To(ContainSubstring("Test Repo"))
os.RemoveAll(f)
})
@@ -71,8 +71,8 @@ var _ = Describe("Docker client", func() {
Expect(err).ToNot(HaveOccurred())
defer os.RemoveAll(tmpdir) // clean up
Expect(f.Unpack(tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Read(filepath.Join(tmpdir, "c"))).To(Equal("c\n"))
Expect(helpers.Read(filepath.Join(tmpdir, "cd"))).To(Equal("c\n"))
Expect(fileHelper.Read(filepath.Join(tmpdir, "c"))).To(Equal("c\n"))
Expect(fileHelper.Read(filepath.Join(tmpdir, "cd"))).To(Equal("c\n"))
os.RemoveAll(f.Path)
})
})

View File

@@ -18,19 +18,20 @@ package client
import (
"fmt"
"math"
"net/http"
"net/url"
"os"
"path"
"path/filepath"
"strconv"
"time"
"github.com/mudler/luet/pkg/compiler/types/artifact"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/logger"
"github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
"github.com/cavaliercoder/grab"
"github.com/mudler/luet/pkg/config"
"github.com/schollz/progressbar/v3"
)
@@ -43,6 +44,27 @@ func NewHttpClient(r RepoData) *HttpClient {
return &HttpClient{RepoData: r}
}
func NewGrabClient() *grab.Client {
httpTimeout := 30
timeout := os.Getenv("HTTP_TIMEOUT")
if timeout != "" {
timeoutI, err := strconv.Atoi(timeout)
if err == nil {
httpTimeout = timeoutI
}
}
return &grab.Client{
UserAgent: "grab",
HTTPClient: &http.Client{
Timeout: time.Duration(httpTimeout) * time.Second,
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
},
},
}
}
func (c *HttpClient) PrepareReq(dst, url string) (*grab.Request, error) {
req, err := grab.NewRequest(dst, url)
@@ -77,7 +99,7 @@ func (c *HttpClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.Pa
ok := false
// Check if file is already in cache
if helpers.Exists(cacheFile) {
if fileHelper.Exists(cacheFile) {
Debug("Use artifact", artifactName, "from cache.")
} else {
@@ -87,7 +109,7 @@ func (c *HttpClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.Pa
}
defer os.RemoveAll(temp)
client := grab.NewClient()
client := NewGrabClient()
for _, uri := range c.RepoData.Urls {
Debug("Downloading artifact", artifactName, "from", uri)
@@ -156,7 +178,7 @@ func (c *HttpClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.Pa
fmt.Sprintf("%.2f", (float64(resp.BytesPerSecond())/1024)/1024), "MiB/s )")
Debug("\nCopying file ", filepath.Join(temp, artifactName), "to", cacheFile)
err = helpers.CopyFile(filepath.Join(temp, artifactName), cacheFile)
err = fileHelper.CopyFile(filepath.Join(temp, artifactName), cacheFile)
bar.Finish()
ok = true
@@ -187,7 +209,7 @@ func (c *HttpClient) DownloadFile(name string) (string, error) {
return "", err
}
client := grab.NewClient()
client := NewGrabClient()
for _, uri := range c.RepoData.Urls {
@@ -218,7 +240,7 @@ func (c *HttpClient) DownloadFile(name string) (string, error) {
fmt.Sprintf("%.2f", (float64(resp.BytesComplete())/1000)/1000), "MB (",
fmt.Sprintf("%.2f", (float64(resp.BytesPerSecond())/1024)/1024), "MiB/s )")
err = helpers.CopyFile(filepath.Join(temp, name), file.Name())
err = fileHelper.CopyFile(filepath.Join(temp, name), file.Name())
if err != nil {
continue
}

View File

@@ -23,8 +23,7 @@ import (
"path/filepath"
"github.com/mudler/luet/pkg/compiler/types/artifact"
helpers "github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/installer/client"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
@@ -47,7 +46,7 @@ var _ = Describe("Http client", func() {
c := NewHttpClient(RepoData{Urls: []string{ts.URL}})
path, err := c.DownloadFile("test.txt")
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Read(path)).To(Equal("test"))
Expect(fileHelper.Read(path)).To(Equal("test"))
os.RemoveAll(path)
})
@@ -65,7 +64,7 @@ var _ = Describe("Http client", func() {
c := NewHttpClient(RepoData{Urls: []string{ts.URL}})
path, err := c.DownloadArtifact(&artifact.PackageArtifact{Path: "test.txt"})
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Read(path.Path)).To(Equal("test"))
Expect(fileHelper.Read(path.Path)).To(Equal("test"))
os.RemoveAll(path.Path)
})

View File

@@ -22,9 +22,8 @@ import (
"github.com/mudler/luet/pkg/compiler/types/artifact"
"github.com/mudler/luet/pkg/config"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/logger"
"github.com/mudler/luet/pkg/helpers"
)
type LocalClient struct {
@@ -50,7 +49,7 @@ func (c *LocalClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.P
}
// Check if file is already in cache
if helpers.Exists(cacheFile) {
if fileHelper.Exists(cacheFile) {
Debug("Use artifact", artifactName, "from cache.")
} else {
ok := false
@@ -61,7 +60,7 @@ func (c *LocalClient) DownloadArtifact(a *artifact.PackageArtifact) (*artifact.P
Info("Downloading artifact", artifactName, "from", uri)
//defer os.Remove(file.Name())
err = helpers.CopyFile(filepath.Join(uri, artifactName), cacheFile)
err = fileHelper.CopyFile(filepath.Join(uri, artifactName), cacheFile)
if err != nil {
continue
}
@@ -104,7 +103,7 @@ func (c *LocalClient) DownloadFile(name string) (string, error) {
}
//defer os.Remove(file.Name())
err = helpers.CopyFile(filepath.Join(uri, name), file.Name())
err = fileHelper.CopyFile(filepath.Join(uri, name), file.Name())
if err != nil {
continue
}

View File

@@ -21,8 +21,7 @@ import (
"path/filepath"
"github.com/mudler/luet/pkg/compiler/types/artifact"
helpers "github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/installer/client"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
@@ -42,7 +41,7 @@ var _ = Describe("Local client", func() {
c := NewLocalClient(RepoData{Urls: []string{tmpdir}})
path, err := c.DownloadFile("test.txt")
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Read(path)).To(Equal("test"))
Expect(fileHelper.Read(path)).To(Equal("test"))
os.RemoveAll(path)
})
@@ -58,7 +57,7 @@ var _ = Describe("Local client", func() {
c := NewLocalClient(RepoData{Urls: []string{tmpdir}})
path, err := c.DownloadArtifact(&artifact.PackageArtifact{Path: "test.txt"})
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Read(path.Path)).To(Equal("test"))
Expect(fileHelper.Read(path.Path)).To(Equal("test"))
os.RemoveAll(path.Path)
})

View File

@@ -24,16 +24,17 @@ import (
"strings"
"sync"
artifact "github.com/mudler/luet/pkg/compiler/types/artifact"
. "github.com/logrusorgru/aurora"
"github.com/mudler/luet/pkg/bus"
artifact "github.com/mudler/luet/pkg/compiler/types/artifact"
"github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
"github.com/mudler/luet/pkg/helpers/match"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/solver"
. "github.com/logrusorgru/aurora"
"github.com/pkg/errors"
)
@@ -271,6 +272,15 @@ func (l *LuetInstaller) swap(o Option, syncedRepos Repositories, toRemove pkg.Pa
if err := l.download(syncedRepos, match); err != nil {
return errors.Wrap(err, "Pre-downloading packages")
}
if err := l.checkFileconflicts(match, false, s); err != nil {
if !l.Options.Force {
return errors.Wrap(err, "file conflict found")
} else {
Warning("file conflict found", err.Error())
}
}
if l.Options.DownloadOnly {
return nil
}
@@ -597,7 +607,7 @@ func (l *LuetInstaller) Reclaim(s *System) error {
"from", repo.GetName(), "is installed")
FILES:
for _, f := range artefact.Files {
if helpers.Exists(filepath.Join(s.Target, f)) {
if fileHelper.Exists(filepath.Join(s.Target, f)) {
p, err := repo.GetTree().GetDatabase().FindPackage(artefact.CompileSpec.GetPackage())
if err != nil {
return err
@@ -755,12 +765,64 @@ func (l *LuetInstaller) getFinalizers(allRepos pkg.PackageDatabase, solution sol
return toFinalize, nil
}
func (l *LuetInstaller) checkFileconflicts(toInstall map[string]ArtifactMatch, checkSystem bool, s *System) error {
Info("Checking for file conflicts..")
defer s.Clean() // Release memory
filesToInstall := []string{}
for _, m := range toInstall {
a, err := l.downloadPackage(m)
if err != nil && !l.Options.Force {
return errors.Wrap(err, "Failed downloading package")
}
files, err := a.FileList()
if err != nil && !l.Options.Force {
return errors.Wrapf(err, "Could not get filelist for %s", a.CompileSpec.Package.HumanReadableString())
}
for _, f := range files {
if helpers.Contains(filesToInstall, f) {
return fmt.Errorf(
"file conflict between packages to be installed",
)
}
if checkSystem {
exists, p, err := s.ExistsPackageFile(f)
if err != nil {
return errors.Wrap(err, "failed checking into system db")
}
if exists {
return fmt.Errorf(
"file conflict between '%s' and '%s' ( file: %s )",
p.HumanReadableString(),
m.Package.HumanReadableString(),
f,
)
}
}
}
filesToInstall = append(filesToInstall, files...)
}
return nil
}
func (l *LuetInstaller) install(o Option, syncedRepos Repositories, toInstall map[string]ArtifactMatch, p pkg.Packages, solution solver.PackagesAssertions, allRepos pkg.PackageDatabase, s *System) error {
// Install packages into rootfs in parallel.
// Download packages in parallel first
if err := l.download(syncedRepos, toInstall); err != nil {
return errors.Wrap(err, "Downloading packages")
}
// Check file conflicts
if err := l.checkFileconflicts(toInstall, true, s); err != nil {
if !l.Options.Force {
return errors.Wrap(err, "file conflict found")
} else {
Warning("file conflict found", err.Error())
}
}
if l.Options.DownloadOnly {
return nil
}
@@ -915,7 +977,7 @@ func pruneEmptyFilePath(path string) {
currentPath = filepath.Join(currentPath, p)
allPaths = append(allPaths, currentPath)
}
helpers.ReverseAny(allPaths)
match.ReverseAny(allPaths)
for _, p := range allPaths {
checkAndPrunePath(p)
}
@@ -943,7 +1005,7 @@ func (l *LuetInstaller) uninstall(p pkg.Package, s *System) error {
cp.Map(files)
}
toRemove, notPresent := helpers.OrderFiles(s.Target, files)
toRemove, notPresent := fileHelper.OrderFiles(s.Target, files)
// Remove from target
for _, f := range toRemove {

View File

@@ -27,6 +27,7 @@ import (
"github.com/mudler/luet/pkg/compiler/types/options"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/installer"
pkg "github.com/mudler/luet/pkg/package"
@@ -84,34 +85,34 @@ var _ = Describe("Installer", func() {
a, err := c.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(a.Path)).To(BeTrue())
Expect(fileHelper.Exists(a.Path)).To(BeTrue())
Expect(helpers.Untar(a.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("test5"))
content1, err := fileHelper.Read(spec.Rel("test5"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("test6"))
content2, err := fileHelper.Read(spec.Rel("test6"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("artifact5\n"))
Expect(content2).To(Equal("artifact6\n"))
Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -136,8 +137,8 @@ urls:
err = inst.Install([]pkg.Package{&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}}, system)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).ToNot(HaveOccurred())
@@ -154,8 +155,8 @@ urls:
Expect(err).ToNot(HaveOccurred())
// Nothing should be there anymore (files, packagedb entry)
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).To(HaveOccurred())
@@ -199,21 +200,21 @@ urls:
artifact, err := c.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("test5"))
content1, err := fileHelper.Read(spec.Rel("test5"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("test6"))
content2, err := fileHelper.Read(spec.Rel("test6"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("artifact5\n"))
Expect(content2).To(Equal("artifact6\n"))
Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
Expect(err).ToNot(HaveOccurred())
@@ -223,15 +224,15 @@ urls:
repo.SetRepositoryFile(REPOFILE_TREE_KEY, treeFile)
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -256,8 +257,8 @@ urls:
err = inst.Install([]pkg.Package{&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}}, system)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).ToNot(HaveOccurred())
@@ -274,8 +275,8 @@ urls:
Expect(err).ToNot(HaveOccurred())
// Nothing should be there anymore (files, packagedb entry)
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).To(HaveOccurred())
@@ -319,21 +320,21 @@ urls:
artifact, err := c.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("test5"))
content1, err := fileHelper.Read(spec.Rel("test5"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("test6"))
content2, err := fileHelper.Read(spec.Rel("test6"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("artifact5\n"))
Expect(content2).To(Equal("artifact6\n"))
Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
repo, err := GenerateRepository(
"test",
@@ -345,15 +346,15 @@ urls:
pkg.NewInMemoryDatabase(false), nil, "", false, false, false, nil)
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -383,8 +384,8 @@ urls:
err = inst.Install([]pkg.Package{&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}}, system)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).ToNot(HaveOccurred())
@@ -401,8 +402,8 @@ urls:
Expect(err).ToNot(HaveOccurred())
// Nothing should be there anymore (files, packagedb entry)
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
_, err = system.Database.GetPackageFiles(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).To(HaveOccurred())
@@ -444,21 +445,21 @@ urls:
artifact, err := c.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("test5"))
content1, err := fileHelper.Read(spec.Rel("test5"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("test6"))
content2, err := fileHelper.Read(spec.Rel("test6"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("artifact5\n"))
Expect(content2).To(Equal("artifact6\n"))
Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
repo, err := GenerateRepository(
"test",
@@ -471,15 +472,15 @@ urls:
pkg.NewInMemoryDatabase(false), nil, "", false, false, false, nil)
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -531,7 +532,7 @@ urls:
artifact, err = c.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
repo, err = stubRepo(tmpdir2, "../../tests/fixtures/alpine")
Expect(err).ToNot(HaveOccurred())
@@ -604,15 +605,15 @@ urls:
repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -642,8 +643,8 @@ urls:
err = inst.Install([]pkg.Package{&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}}, system)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).ToNot(HaveOccurred())
@@ -660,11 +661,11 @@ urls:
Expect(err).ToNot(HaveOccurred())
// Nothing should be there anymore (files, packagedb entry)
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
// New version - new files
Expect(helpers.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
_, err = system.Database.GetPackageFiles(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).To(HaveOccurred())
_, err = system.Database.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
@@ -729,9 +730,9 @@ urls:
repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade_old_repo")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
@@ -774,8 +775,8 @@ urls:
err = inst.Install([]pkg.Package{&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}}, system)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).ToNot(HaveOccurred())
@@ -793,11 +794,11 @@ urls:
Expect(err).ToNot(HaveOccurred())
// Nothing should be there anymore (files, packagedb entry)
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
// New version - new files
Expect(helpers.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
_, err = system.Database.GetPackageFiles(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).To(HaveOccurred())
_, err = system.Database.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
@@ -856,17 +857,17 @@ urls:
repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar.gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.1.package.tar.gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.1.package.tar"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -896,8 +897,8 @@ urls:
err = inst.Install([]pkg.Package{&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}}, system)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeTrue())
_, err = systemDB.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).ToNot(HaveOccurred())
@@ -914,11 +915,11 @@ urls:
Expect(err).ToNot(HaveOccurred())
// Nothing should be there anymore (files, packagedb entry)
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
// New version - new files
Expect(helpers.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
_, err = system.Database.GetPackageFiles(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).To(HaveOccurred())
_, err = system.Database.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
@@ -1014,17 +1015,17 @@ urls:
repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar.gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.1.package.tar.gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.1.package.tar"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -1059,13 +1060,13 @@ urls:
Expect(err).To(HaveOccurred())
Expect(len(system.Database.World())).To(Equal(0))
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeFalse())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeFalse())
Expect(helpers.Exists(filepath.Join(fakeroot, "c"))).To(BeFalse())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeFalse())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeFalse())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "c"))).To(BeFalse())
Expect(helpers.Touch(filepath.Join(fakeroot, "test5"))).ToNot(HaveOccurred())
Expect(helpers.Touch(filepath.Join(fakeroot, "test6"))).ToNot(HaveOccurred())
Expect(helpers.Touch(filepath.Join(fakeroot, "c"))).ToNot(HaveOccurred())
Expect(fileHelper.Touch(filepath.Join(fakeroot, "test5"))).ToNot(HaveOccurred())
Expect(fileHelper.Touch(filepath.Join(fakeroot, "test6"))).ToNot(HaveOccurred())
Expect(fileHelper.Touch(filepath.Join(fakeroot, "c"))).ToNot(HaveOccurred())
err = inst.Reclaim(system)
Expect(err).ToNot(HaveOccurred())
@@ -1115,15 +1116,15 @@ urls:
repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade_old_repo")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, false)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(repo.GetUrls()[0]).To(Equal(tmpdir))
Expect(repo.GetType()).To(Equal("disk"))
@@ -1158,13 +1159,13 @@ urls:
Expect(err).To(HaveOccurred())
Expect(len(system.Database.World())).To(Equal(0))
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).To(BeFalse())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).To(BeFalse())
Expect(helpers.Exists(filepath.Join(fakeroot, "c"))).To(BeFalse())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).To(BeFalse())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).To(BeFalse())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "c"))).To(BeFalse())
Expect(helpers.Touch(filepath.Join(fakeroot, "test5"))).ToNot(HaveOccurred())
Expect(helpers.Touch(filepath.Join(fakeroot, "test6"))).ToNot(HaveOccurred())
Expect(helpers.Touch(filepath.Join(fakeroot, "c"))).ToNot(HaveOccurred())
Expect(fileHelper.Touch(filepath.Join(fakeroot, "test5"))).ToNot(HaveOccurred())
Expect(fileHelper.Touch(filepath.Join(fakeroot, "test6"))).ToNot(HaveOccurred())
Expect(fileHelper.Touch(filepath.Join(fakeroot, "c"))).ToNot(HaveOccurred())
err = inst.Reclaim(system)
Expect(err).ToNot(HaveOccurred())
@@ -1218,11 +1219,11 @@ urls:
Expect(err).ToNot(HaveOccurred())
// Nothing should be there anymore (files, packagedb entry)
Expect(helpers.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(helpers.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test5"))).ToNot(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "test6"))).ToNot(BeTrue())
// New version - new files
Expect(helpers.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
Expect(fileHelper.Exists(filepath.Join(fakeroot, "newc"))).To(BeTrue())
_, err = system.Database.GetPackageFiles(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
Expect(err).To(HaveOccurred())
_, err = system.Database.FindPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})

View File

@@ -28,11 +28,11 @@ import (
artifact "github.com/mudler/luet/pkg/compiler/types/artifact"
compression "github.com/mudler/luet/pkg/compiler/types/compression"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
"go.uber.org/multierr"
"github.com/mudler/luet/pkg/compiler"
"github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
"github.com/mudler/luet/pkg/installer/client"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
@@ -723,7 +723,7 @@ func (r *LuetSystemRepository) SyncBuildMetadata(path string) error {
if err != nil {
return errors.Wrapf(err, "while downloading metadata for %s", ai.HumanReadableString())
}
if err := helpers.Move(file, filepath.Join(path, ai.GetMetadataFilePath())); err != nil {
if err := fileHelper.Move(file, filepath.Join(path, ai.GetMetadataFilePath())); err != nil {
return err
}
}
@@ -810,7 +810,7 @@ func (r *LuetSystemRepository) Sync(force bool) (*LuetSystemRepository, error) {
if r.Cached {
// Copy updated repository.yaml file to repo dir now that the tree is synced.
err = helpers.CopyFile(file, filepath.Join(repobasedir, REPOSITORY_SPECFILE))
err = fileHelper.CopyFile(file, filepath.Join(repobasedir, REPOSITORY_SPECFILE))
if err != nil {
return nil, errors.Wrap(err, "Error on update "+REPOSITORY_SPECFILE)
}

View File

@@ -24,16 +24,16 @@ import (
"strings"
"time"
artifact "github.com/mudler/luet/pkg/compiler/types/artifact"
"github.com/mudler/luet/pkg/bus"
compiler "github.com/mudler/luet/pkg/compiler"
"github.com/mudler/luet/pkg/compiler/backend"
artifact "github.com/mudler/luet/pkg/compiler/types/artifact"
"github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
"github.com/mudler/luet/pkg/helpers/docker"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/bus"
compiler "github.com/mudler/luet/pkg/compiler"
"github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
"github.com/pkg/errors"
)
@@ -61,7 +61,7 @@ func (l *dockerRepositoryGenerator) Initialize(path string, db pkg.PackageDataba
return nil
}
if err := l.pushImageFromArtifact(artifact.NewPackageArtifact(currentpath), l.b); err != nil {
if err := l.pushImageFromArtifact(artifact.NewPackageArtifact(currentpath), l.b, true); err != nil {
return errors.Wrap(err, "while pushing metadata file associated to the artifact")
}
@@ -159,16 +159,20 @@ func (d *dockerRepositoryGenerator) pushRepoMetadata(repospec string, r *LuetSys
return nil
}
func (d *dockerRepositoryGenerator) pushImageFromArtifact(a *artifact.PackageArtifact, b compiler.CompilerBackend) error {
func (d *dockerRepositoryGenerator) pushImageFromArtifact(a *artifact.PackageArtifact, b compiler.CompilerBackend, checkIfExists bool) error {
// we generate a new archive containing the required compressed file.
// TODO: Bundle all the extra files in 1 docker image only, instead of an image for each file
treeArchive, err := artifact.CreateArtifactForFile(a.Path)
if err != nil {
return errors.Wrap(err, "failed generating checksums for tree")
}
imageTree := fmt.Sprintf("%s:%s", d.imagePrefix, helpers.StripInvalidStringsFromImage(a.GetFileName()))
return d.pushFileFromArtifact(treeArchive, imageTree)
imageTree := fmt.Sprintf("%s:%s", d.imagePrefix, docker.StripInvalidStringsFromImage(a.GetFileName()))
if checkIfExists && d.imagePush && d.b.ImageAvailable(imageTree) && !d.force {
Info("Image", imageTree, "already present, skipping. use --force-push to override")
return nil
} else {
return d.pushFileFromArtifact(treeArchive, imageTree)
}
}
// Generate creates a Docker luet repository
@@ -225,7 +229,7 @@ func (d *dockerRepositoryGenerator) Generate(r *LuetSystemRepository, imagePrefi
// we generate a new archive containing the required compressed file.
// TODO: Bundle all the extra files in 1 docker image only, instead of an image for each file
if err := d.pushImageFromArtifact(a, d.b); err != nil {
if err := d.pushImageFromArtifact(a, d.b, false); err != nil {
return errors.Wrap(err, "error met while pushing runtime tree")
}
@@ -235,7 +239,7 @@ func (d *dockerRepositoryGenerator) Generate(r *LuetSystemRepository, imagePrefi
}
// we generate a new archive containing the required compressed file.
// TODO: Bundle all the extra files in 1 docker image only, instead of an image for each file
if err := d.pushImageFromArtifact(a, d.b); err != nil {
if err := d.pushImageFromArtifact(a, d.b, false); err != nil {
return errors.Wrap(err, "error met while pushing compiler tree")
}
@@ -251,7 +255,7 @@ func (d *dockerRepositoryGenerator) Generate(r *LuetSystemRepository, imagePrefi
return errors.Wrap(err, "failed adding Metadata file to repository")
}
if err := d.pushImageFromArtifact(a, d.b); err != nil {
if err := d.pushImageFromArtifact(a, d.b, false); err != nil {
return errors.Wrap(err, "error met while pushing docker image from artifact")
}

View File

@@ -26,14 +26,15 @@ import (
"github.com/mudler/luet/pkg/compiler"
backend "github.com/mudler/luet/pkg/compiler/backend"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
artifact "github.com/mudler/luet/pkg/compiler/types/artifact"
compilerspec "github.com/mudler/luet/pkg/compiler/types/spec"
config "github.com/mudler/luet/pkg/config"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/installer"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/tree"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
@@ -83,34 +84,34 @@ var _ = Describe("Repository", func() {
artifact, err := compiler.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("test5"))
content1, err := fileHelper.Read(spec.Rel("test5"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("test6"))
content2, err := fileHelper.Read(spec.Rel("test6"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("artifact5\n"))
Expect(content2).To(Equal("artifact6\n"))
Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, true)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_SPECFILE))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
})
It("Generate repository metadata of files ONLY referenced in a tree", func() {
@@ -156,41 +157,41 @@ var _ = Describe("Repository", func() {
artifact, err := compiler.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact.Path)).To(BeTrue())
Expect(helpers.Untar(artifact.Path, tmpdir, false)).ToNot(HaveOccurred())
artifact2, err := compiler2.Compile(false, spec2)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(artifact2.Path)).To(BeTrue())
Expect(fileHelper.Exists(artifact2.Path)).To(BeTrue())
Expect(helpers.Untar(artifact2.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
content1, err := helpers.Read(spec.Rel("test5"))
content1, err := fileHelper.Read(spec.Rel("test5"))
Expect(err).ToNot(HaveOccurred())
content2, err := helpers.Read(spec.Rel("test6"))
content2, err := fileHelper.Read(spec.Rel("test6"))
Expect(err).ToNot(HaveOccurred())
Expect(content1).To(Equal("artifact5\n"))
Expect(content2).To(Equal("artifact6\n"))
Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(helpers.Exists(spec2.Rel("alpine-seed-1.0.package.tar"))).To(BeTrue())
Expect(helpers.Exists(spec2.Rel("alpine-seed-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
Expect(fileHelper.Exists(spec2.Rel("alpine-seed-1.0.package.tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec2.Rel("alpine-seed-1.0.metadata.yaml"))).To(BeTrue())
repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(tmpdir, false, true)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_SPECFILE))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).To(BeTrue())
// We check now that the artifact not referenced in the tree
// (spec2) is not indexed in the repository
@@ -267,18 +268,18 @@ urls:
a, err := localcompiler.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(a.Path)).To(BeTrue())
Expect(fileHelper.Exists(a.Path)).To(BeTrue())
Expect(helpers.Untar(a.Path, tmpdir, false)).ToNot(HaveOccurred())
Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test5"))).To(BeTrue())
Expect(fileHelper.Exists(spec.Rel("test6"))).To(BeTrue())
repo, err := dockerStubRepo(tmpdir, "../../tests/fixtures/buildable", repoImage, true, true)
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(repoImage, false, true)
Expect(err).ToNot(HaveOccurred())
@@ -295,7 +296,7 @@ urls:
f, err := c.DownloadFile("repository.yaml")
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Read(f)).To(ContainSubstring("name: test"))
Expect(fileHelper.Read(f)).To(ContainSubstring("name: test"))
a, err = c.DownloadArtifact(&artifact.PackageArtifact{
Path: "test.tar",
@@ -310,7 +311,7 @@ urls:
Expect(err).ToNot(HaveOccurred())
Expect(a.Unpack(extracted, false)).ToNot(HaveOccurred())
Expect(helpers.Read(filepath.Join(extracted, "test6"))).To(Equal("artifact6\n"))
Expect(fileHelper.Read(filepath.Join(extracted, "test6"))).To(Equal("artifact6\n"))
})
It("generates images of virtual packages", func() {
@@ -341,15 +342,15 @@ urls:
a, err := localcompiler.Compile(false, spec)
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Exists(a.Path)).To(BeTrue())
Expect(fileHelper.Exists(a.Path)).To(BeTrue())
Expect(helpers.Untar(a.Path, tmpdir, false)).ToNot(HaveOccurred())
repo, err := dockerStubRepo(tmpdir, "../../tests/fixtures/virtuals", repoImage, true, true)
Expect(err).ToNot(HaveOccurred())
Expect(repo.GetName()).To(Equal("test"))
Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
Expect(fileHelper.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
err = repo.Write(repoImage, false, true)
Expect(err).ToNot(HaveOccurred())
@@ -366,7 +367,7 @@ urls:
f, err := c.DownloadFile("repository.yaml")
Expect(err).ToNot(HaveOccurred())
Expect(helpers.Read(f)).To(ContainSubstring("name: test"))
Expect(fileHelper.Read(f)).To(ContainSubstring("name: test"))
a, err = c.DownloadArtifact(&artifact.PackageArtifact{
Path: "test.tar",
@@ -382,7 +383,7 @@ urls:
Expect(a.Unpack(extracted, false)).ToNot(HaveOccurred())
Expect(helpers.DirectoryIsEmpty(extracted)).To(BeFalse())
Expect(fileHelper.DirectoryIsEmpty(extracted)).To(BeFalse())
content, err := ioutil.ReadFile(filepath.Join(extracted, ".virtual"))
Expect(err).ToNot(HaveOccurred())

View File

@@ -1,16 +1,21 @@
package installer
import (
"sync"
"github.com/hashicorp/go-multierror"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
. "github.com/mudler/luet/pkg/logger"
pkg "github.com/mudler/luet/pkg/package"
"github.com/mudler/luet/pkg/tree"
)
type System struct {
Database pkg.PackageDatabase
Target string
Database pkg.PackageDatabase
Target string
fileIndex map[string]pkg.Package
sync.Mutex
}
func (s *System) World() (pkg.Packages, error) {
@@ -23,7 +28,7 @@ func (s *System) ExecuteFinalizers(packs []pkg.Package) error {
var errs error
executedFinalizer := map[string]bool{}
for _, p := range packs {
if helpers.Exists(p.Rel(tree.FinalizerFile)) {
if fileHelper.Exists(p.Rel(tree.FinalizerFile)) {
out, err := helpers.RenderFiles(p.Rel(tree.FinalizerFile), p.Rel(tree.DefinitionFile))
if err != nil {
Warning("Failed rendering finalizer for ", p.HumanReadableString(), err.Error())
@@ -51,3 +56,38 @@ func (s *System) ExecuteFinalizers(packs []pkg.Package) error {
}
return errs
}
func (s *System) buildFileIndex() {
s.Lock()
defer s.Unlock()
// Check if cache is empty or if it got modified
if s.fileIndex == nil { //|| len(s.Database.GetPackages()) != len(s.fileIndex) {
s.fileIndex = make(map[string]pkg.Package)
for _, p := range s.Database.World() {
files, _ := s.Database.GetPackageFiles(p)
for _, f := range files {
s.fileIndex[f] = p
}
}
}
}
func (s *System) Clean() {
s.Lock()
defer s.Unlock()
s.fileIndex = nil
}
func (s *System) ExistsPackageFile(file string) (bool, pkg.Package, error) {
Debug("Checking if file ", file, "belongs to any package")
s.buildFileIndex()
s.Lock()
defer s.Unlock()
if p, exists := s.fileIndex[file]; exists {
Debug(file, "belongs already to", p.HumanReadableString())
return exists, p, nil
}
Debug(file, "doesn't belong to any package")
return false, nil, nil
}

View File

@@ -0,0 +1,70 @@
// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License along
// with this program; if not, see <http://www.gnu.org/licenses/>.
package installer_test
import (
// . "github.com/mudler/luet/pkg/installer"
. "github.com/mudler/luet/pkg/installer"
pkg "github.com/mudler/luet/pkg/package"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
var _ = Describe("System", func() {
Context("Files", func() {
var s *System
var db pkg.PackageDatabase
var a, b *pkg.DefaultPackage
BeforeEach(func() {
db = pkg.NewInMemoryDatabase(false)
s = &System{Database: db}
a = &pkg.DefaultPackage{Name: "test", Version: "1", Category: "t"}
db.CreatePackage(a)
db.SetPackageFiles(&pkg.PackageFile{PackageFingerprint: a.GetFingerPrint(), Files: []string{"foo", "f"}})
b = &pkg.DefaultPackage{Name: "test2", Version: "1", Category: "t"}
db.CreatePackage(b)
db.SetPackageFiles(&pkg.PackageFile{PackageFingerprint: b.GetFingerPrint(), Files: []string{"barz", "f"}})
})
It("detects when are already shipped by other packages", func() {
r, p, err := s.ExistsPackageFile("foo")
Expect(r).To(BeTrue())
Expect(err).ToNot(HaveOccurred())
Expect(p).To(Equal(a))
r, p, err = s.ExistsPackageFile("baz")
Expect(r).To(BeFalse())
Expect(err).ToNot(HaveOccurred())
Expect(p).To(BeNil())
r, p, err = s.ExistsPackageFile("f")
Expect(r).To(BeTrue())
Expect(err).ToNot(HaveOccurred())
Expect(p).To(Equal(b))
r, p, err = s.ExistsPackageFile("barz")
Expect(r).To(BeTrue())
Expect(err).ToNot(HaveOccurred())
Expect(p).To(Equal(b))
})
})
})

View File

@@ -26,7 +26,8 @@ import (
"strconv"
"strings"
"github.com/mudler/luet/pkg/helpers"
"github.com/mudler/luet/pkg/helpers/docker"
"github.com/mudler/luet/pkg/helpers/match"
version "github.com/mudler/luet/pkg/versioner"
gentoo "github.com/Sabayon/pkgs-checker/pkg/gentoo"
@@ -133,6 +134,19 @@ type Tree interface {
type Packages []Package
type DefaultPackages []*DefaultPackage
func (d DefaultPackages) Hash(salt string) string {
overallFp := ""
for _, c := range d {
overallFp = overallFp + c.HashFingerprint("join")
}
h := md5.New()
io.WriteString(h, fmt.Sprintf("%s-%s", overallFp, salt))
return fmt.Sprintf("%x", h.Sum(nil))
}
// >> Unmarshallers
// DefaultPackageFromYaml decodes a package from yaml bytes
func DefaultPackageFromYaml(yml []byte) (DefaultPackage, error) {
@@ -311,7 +325,7 @@ func (p *DefaultPackage) GetPackageName() string {
}
func (p *DefaultPackage) ImageID() string {
return helpers.StripInvalidStringsFromImage(p.GetFingerPrint())
return docker.StripInvalidStringsFromImage(p.GetFingerPrint())
}
// GetBuildTimestamp returns the package build timestamp
@@ -346,19 +360,19 @@ func (p *DefaultPackage) IsHidden() bool {
}
func (p *DefaultPackage) HasLabel(label string) bool {
return helpers.MapHasKey(&p.Labels, label)
return match.MapHasKey(&p.Labels, label)
}
func (p *DefaultPackage) MatchLabel(r *regexp.Regexp) bool {
return helpers.MapMatchRegex(&p.Labels, r)
return match.MapMatchRegex(&p.Labels, r)
}
func (p *DefaultPackage) HasAnnotation(label string) bool {
return helpers.MapHasKey(&p.Annotations, label)
return match.MapHasKey(&p.Annotations, label)
}
func (p *DefaultPackage) MatchAnnotation(r *regexp.Regexp) bool {
return helpers.MapMatchRegex(&p.Annotations, r)
return match.MapMatchRegex(&p.Annotations, r)
}
// AddUse adds a use to a package

View File

@@ -26,6 +26,7 @@ import (
"path/filepath"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
pkg "github.com/mudler/luet/pkg/package"
"github.com/pkg/errors"
)
@@ -61,7 +62,7 @@ type CompilerRecipe struct {
// and the build context required for reproducible builds
func (r *CompilerRecipe) Save(path string) error {
for _, p := range r.SourcePath {
if err := helpers.CopyDir(p, filepath.Join(path, filepath.Base(p))); err != nil {
if err := fileHelper.CopyDir(p, filepath.Join(path, filepath.Base(p))); err != nil {
return errors.Wrap(err, "while copying source tree")
}
}
@@ -102,7 +103,7 @@ func (r *CompilerRecipe) Load(path string) error {
// Instead of rdeps, have a different tree for build deps.
compileDefPath := pack.Rel(CompilerDefinitionFile)
if helpers.Exists(compileDefPath) {
if fileHelper.Exists(compileDefPath) {
dat, err := helpers.RenderFiles(compileDefPath, currentpath)
if err != nil {
@@ -149,7 +150,7 @@ func (r *CompilerRecipe) Load(path string) error {
// Instead of rdeps, have a different tree for build deps.
compileDefPath := pack.Rel(CompilerDefinitionFile)
if helpers.Exists(compileDefPath) {
if fileHelper.Exists(compileDefPath) {
raw := packsRaw.Find(pack.GetName(), pack.GetCategory(), pack.GetVersion())
buildyaml, err := ioutil.ReadFile(compileDefPath)

View File

@@ -26,8 +26,9 @@ import (
"os"
"path/filepath"
"github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
pkg "github.com/mudler/luet/pkg/package"
"github.com/pkg/errors"
)
@@ -61,8 +62,8 @@ func (r *InstallerRecipe) Save(path string) error {
}
// Instead of rdeps, have a different tree for build deps.
finalizerPath := p.Rel(FinalizerFile)
if helpers.Exists(finalizerPath) { // copy finalizer file from the source tree
helpers.CopyFile(finalizerPath, filepath.Join(dir, FinalizerFile))
if fileHelper.Exists(finalizerPath) { // copy finalizer file from the source tree
fileHelper.CopyFile(finalizerPath, filepath.Join(dir, FinalizerFile))
}
}
@@ -71,7 +72,7 @@ func (r *InstallerRecipe) Save(path string) error {
func (r *InstallerRecipe) Load(path string) error {
if !helpers.Exists(path) {
if !fileHelper.Exists(path) {
return errors.New(fmt.Sprintf(
"Path %s doesn't exit.", path,
))

View File

@@ -26,9 +26,10 @@ import (
"os"
"path/filepath"
helpers "github.com/mudler/luet/pkg/helpers"
fileHelper "github.com/mudler/luet/pkg/helpers/file"
pkg "github.com/mudler/luet/pkg/package"
spectooling "github.com/mudler/luet/pkg/spectooling"
"github.com/pkg/errors"
)
@@ -77,7 +78,7 @@ func (r *Recipe) Load(path string) error {
// if err != nil {
// return err
// }
if !helpers.Exists(path) {
if !fileHelper.Exists(path) {
return errors.New(fmt.Sprintf(
"Path %s doesn't exit.", path,
))

View File

@@ -0,0 +1,6 @@
image: "alpine"
prelude:
- mkdir /foo
steps:
- echo conflict > /foo/test1
package_dir: /foo

View File

@@ -0,0 +1,7 @@
packages:
- category: "test1"
name: "conflict"
version: "1.0"
- category: "test2"
name: "conflict"
version: "1.0"

View File

@@ -0,0 +1,6 @@
image: "alpine"
prelude:
- mkdir /foo
steps:
- echo conflict > /foo/test1
package_dir: /foo

View File

@@ -0,0 +1,13 @@
packages:
- category: "test1"
name: "conflict"
version: "1.0"
- category: "test2"
name: "conflict"
version: "1.0"
- category: "test1"
name: "conflict"
version: "1.1"
- category: "test2"
name: "conflict"
version: "1.1"

8
tests/fixtures/join/c/a/build.yaml vendored Normal file
View File

@@ -0,0 +1,8 @@
join:
- name: "a"
category: "test"
version: ">=0"
- name: "b"
category: "test"
version: ">=0"
unpack: true

View File

@@ -0,0 +1,3 @@
category: "test"
name: "c"
version: "1.2"

View File

@@ -0,0 +1,7 @@
image: "alpine"
prelude:
- echo foo > /test
- echo bar > /test2
steps:
- echo artifact3 > /test3
- echo artifact4 > /test4

View File

@@ -0,0 +1,3 @@
category: "test"
name: "a"
version: "1.2"

View File

@@ -0,0 +1,13 @@
requires:
- category: "test"
name: "a"
version: ">=0"
prelude:
- echo foo > /test
- echo bar > /test2
steps:
- echo artifact5 > /newc
- echo artifact6 > /newnewc
- chmod +x generate.sh
- ./generate.sh

View File

@@ -0,0 +1,3 @@
category: "test"
name: "b"
version: "1.1"

View File

@@ -0,0 +1 @@
echo generated > /sonewc

View File

@@ -0,0 +1,7 @@
join:
- name: "a"
category: "test"
version: ">=0"
- name: "b"
category: "test"
version: ">=0"

View File

@@ -0,0 +1,3 @@
category: "test"
name: "c"
version: "1.2"

View File

@@ -0,0 +1,8 @@
join:
- name: "a"
category: "test"
version: ">=0"
- name: "b"
category: "test"
version: ">=0"
unpack: true

View File

@@ -0,0 +1,3 @@
category: "test"
name: "f"
version: "1.2"

View File

@@ -0,0 +1,2 @@
image: "alpine"
unpack: true

View File

@@ -0,0 +1,3 @@
category: "test"
name: "a"
version: "1.2"

View File

@@ -0,0 +1,13 @@
requires:
- category: "test"
name: "a"
version: ">=0"
prelude:
- echo foo > /test
- echo bar > /test2
steps:
- echo artifact5 > /newc
- echo artifact6 > /newnewc
- chmod +x generate.sh
- ./generate.sh

View File

@@ -0,0 +1,3 @@
category: "test"
name: "b"
version: "1.1"

View File

@@ -0,0 +1 @@
echo generated > /sonewc

View File

@@ -0,0 +1,10 @@
join:
- category: "test"
name: "f"
version: ">=0"
# this is actually a virtual, no files shipped in c
- category: "test"
name: "c"
version: ">=0"
steps:
- mv /newnewc /x

View File

@@ -0,0 +1,3 @@
name: "x"
category: "test"
version: "0.1"

View File

@@ -0,0 +1,6 @@
requires:
- category: "test"
name: "c"
version: ">=0"
steps:
- mv /newnewc /z

View File

@@ -0,0 +1,3 @@
name: "z"
category: "test"
version: "0.1"

View File

@@ -92,7 +92,7 @@ testUnInstall() {
assertTrue 'package uninstalled' "[ ! -e '$tmpdir/testrootfs/c' ]"
# TODO: we need remove it or not??
assertTrue 'config protect created' "[ -e '$tmpdir/testrootfs/etc/a/._cfg0001_conf' ]"
assertTrue 'config protect maintains the protected files' "[ -e '$tmpdir/testrootfs/etc/a/conf' ]"
assertTrue 'config protect maintains the protected files' "[ ! -e '$tmpdir/testrootfs/etc/a/conf' ]"
}

View File

@@ -87,7 +87,7 @@ testInstall() {
testUnInstall() {
luet uninstall -y --full --config $tmpdir/luet.yaml test/a
luet uninstall -y --full --keep-protected-files --config $tmpdir/luet.yaml test/a
installst=$?
assertEquals 'uninstall test successfully' "$installst" "0"
assertTrue 'package uninstalled' "[ ! -e '$tmpdir/testrootfs/c' ]"

View File

@@ -86,7 +86,7 @@ testInstall() {
testUnInstall() {
luet uninstall -y --full --config $tmpdir/luet.yaml test/a
luet uninstall -y --full --keep-protected-files --config $tmpdir/luet.yaml test/a
installst=$?
assertEquals 'uninstall test successfully' "$installst" "0"
assertTrue 'package uninstalled' "[ ! -e '$tmpdir/testrootfs/c' ]"

View File

@@ -51,9 +51,9 @@ testBuild() {
assertTrue 'create package' "[ -e '$tmpdir/testbuild/c-test-1.0.package.tar.zst' ]"
assertTrue 'create package Z' "[ -e '$tmpdir/testbuild/z-test-1.0+2.package.tar.zst' ]"
assertTrue 'create package interpolated' "[ -e '$tmpdir/testbuild/interpolated-test-1.0+2.package.tar.zst' ]"
assertContains 'Does use the upstream cache without specifying it test/c' "$build_output" "Images available remotely for test/c-1.0 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:7dd6062f45e78c1fe36d0f48fc21bc8c8219edfc9759f117527677a15ae42717"
assertContains 'Does use the upstream cache without specifying it test/z' "$build_output" "Images available remotely for test/z-1.0+2 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:2338dc4dc4b3d9657eb26597ad79569fe60f5529dd05676df266ad982ba2b1ba"
assertContains 'Does use the upstream cache without specifying it test/interpolated' "$build_output" "Images available remotely for test/interpolated-1.0+2 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:2edc4b6f8fc08a0fc958132dfcf2813e1ab220c2bbf60b988f1039082d61ff3e"
assertContains 'Does use the upstream cache without specifying it test/c' "$build_output" "Images available remotely for test/c-1.0 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:7a46e5c8893b574d9f984eaade75b0e68cc891bb6d01860ba91a62f9e4de2b62"
assertContains 'Does use the upstream cache without specifying it test/z' "$build_output" "Images available remotely for test/z-1.0+2 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:4277934ece4de17856b87294301cdfcdf3c6f956e682ff64697be67178b8a4b1"
assertContains 'Does use the upstream cache without specifying it test/interpolated' "$build_output" "Images available remotely for test/interpolated-1.0+2 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:132c097844b4e809efa8ae234452893b0214a1c860371e21564d89655ab10a56"
}
testRepo() {

View File

@@ -56,8 +56,8 @@ EOF
assertTrue 'create package' "[ -e '$tmpdir/testbuild/c-test-1.0.package.tar.zst' ]"
assertTrue 'create package Z' "[ -e '$tmpdir/testbuild/z-test-1.0+2.package.tar.zst' ]"
assertTrue 'create package interpolated' "[ -e '$tmpdir/testbuild/interpolated-test-1.0+2.package.tar.zst' ]"
assertNotContains 'Does NOT use the upstream cache without specifying it' "$build_output" "Images available remotely for test/interpolated-1.0+2 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:2edc4b6f8fc08a0fc958132dfcf2813e1ab220c2bbf60b988f1039082d61ff3e"
assertContains 'Does generate a new hash as values changed build.yaml' "$build_output" "Building image luet/cache:a249d16764168baf32b592dc05c33e499fa3685edda72dfb188a51191709de5a done"
assertNotContains 'Does NOT use the upstream cache without specifying it' "$build_output" "Images available remotely for test/interpolated-1.0+2 generating artifact from remote images: quay.io/mocaccinoos/integration-test-cache:132c097844b4e809efa8ae234452893b0214a1c860371e21564d89655ab10a56"
assertContains 'Does generate a new hash as values changed build.yaml for test/interpolated-1.0+2 package image' "$build_output" "Building image luet/cache:a7edf5c9ab219e406b64b2692ce08d56a8bcd212a43bb70df3737f13a008dfd4 done"
}
testRepo() {

View File

@@ -56,7 +56,7 @@ EOF
assertTrue 'create package' "[ -e '$tmpdir/testbuild/c-test-1.0.package.tar.zst' ]"
assertTrue 'create package Z' "[ -e '$tmpdir/testbuild/z-test-1.0+2.package.tar.zst' ]"
assertTrue 'create package interpolated' "[ -e '$tmpdir/testbuild/interpolated-test-1.0+2.package.tar.zst' ]"
assertContains 'Does use the upstream cache without specifying it' "$build_output" "Downloading image quay.io/mocaccinoos/integration-test-cache:ec62e3e2cfb4c520c8b2561797c005d248c2659295f3660fa1a66582fc4dc280"
assertContains 'Does use the upstream cache without specifying it (test/c-1.0)' "$build_output" "quay.io/mocaccinoos/integration-test-cache:a786d3fd29d0b8bdfe5f304c8bf8be909d5c764cd7059c0e63294a8bff17f3ef"
}
testRepo() {

39
tests/integration/31_join.sh Executable file
View File

@@ -0,0 +1,39 @@
#!/bin/bash
export LUET_NOLOCK=true
oneTimeSetUp() {
export tmpdir="$(mktemp -d)"
docker images --filter='reference=luet/cache' --format='{{.Repository}}:{{.Tag}}' | xargs -r docker rmi
}
oneTimeTearDown() {
rm -rf "$tmpdir"
docker images --filter='reference=luet/cache' --format='{{.Repository}}:{{.Tag}}' | xargs -r docker rmi
}
testBuild() {
[ "$LUET_BACKEND" == "img" ] && startSkipping
cat <<EOF > $tmpdir/default.yaml
extra: "bar"
foo: "baz"
EOF
mkdir $tmpdir/testbuild
luet build --tree "$ROOT_DIR/tests/fixtures/join" \
--destination $tmpdir/testbuild --concurrency 1 \
--compression gzip --values $tmpdir/default.yaml \
test/c
buildst=$?
assertEquals 'builds successfully' "$buildst" "0"
assertTrue 'create package c' "[ -e '$tmpdir/testbuild/c-test-1.2.package.tar.gz' ]"
mkdir $tmpdir/extract
tar -xvf $tmpdir/testbuild/c-test-1.2.package.tar.gz -C $tmpdir/extract
assertTrue 'create result from join' "[ -e '$tmpdir/extract/test3' ]"
assertTrue 'create result from join' "[ -f '$tmpdir/extract/newc' ]"
assertTrue 'create result from join' "[ -e '$tmpdir/extract/test4' ]"
}
# Load shUnit2.
. "$ROOT_DIR/tests/integration/shunit2"/shunit2

View File

@@ -0,0 +1,37 @@
#!/bin/bash
export LUET_NOLOCK=true
oneTimeSetUp() {
export tmpdir="$(mktemp -d)"
docker images --filter='reference=luet/cache' --format='{{.Repository}}:{{.Tag}}' | xargs -r docker rmi
}
oneTimeTearDown() {
rm -rf "$tmpdir"
docker images --filter='reference=luet/cache' --format='{{.Repository}}:{{.Tag}}' | xargs -r docker rmi
}
testBuild() {
[ "$LUET_BACKEND" == "img" ] && startSkipping
mkdir $tmpdir/testbuild
luet build --tree "$ROOT_DIR/tests/fixtures/join_complex" \
--destination $tmpdir/testbuild --concurrency 1 \
--compression gzip \
test/z test/x
buildst=$?
assertEquals 'builds successfully' "$buildst" "0"
assertTrue 'create package z' "[ -e '$tmpdir/testbuild/z-test-0.1.package.tar.gz' ]"
assertTrue 'create package z' "[ -e '$tmpdir/testbuild/x-test-0.1.package.tar.gz' ]"
mkdir $tmpdir/extract
tar -xvf $tmpdir/testbuild/x-test-0.1.package.tar.gz -C $tmpdir/extract
tar -xvf $tmpdir/testbuild/z-test-0.1.package.tar.gz -C $tmpdir/extract
assertTrue 'create result from a package that requires a join' "[ -e '$tmpdir/extract/z' ]"
assertTrue 'create result from join of a join' "[ -e '$tmpdir/extract/x' ]"
}
# Load shUnit2.
. "$ROOT_DIR/tests/integration/shunit2"/shunit2

View File

@@ -0,0 +1,84 @@
#!/bin/bash
export LUET_NOLOCK=true
oneTimeSetUp() {
export tmpdir="$(mktemp -d)"
}
oneTimeTearDown() {
rm -rf "$tmpdir"
}
testBuild() {
mkdir $tmpdir/testbuild
luet build --tree "$ROOT_DIR/tests/fixtures/fileconflicts" --destination $tmpdir/testbuild --compression gzip --all
buildst=$?
assertEquals 'builds successfully' "$buildst" "0"
assertTrue 'create packages' "[ -e '$tmpdir/testbuild/conflict-test1-1.0.package.tar.gz' ]"
assertTrue 'create packages' "[ -e '$tmpdir/testbuild/conflict-test2-1.0.package.tar.gz' ]"
}
testRepo() {
assertTrue 'no repository' "[ ! -e '$tmpdir/testbuild/repository.yaml' ]"
luet create-repo --tree "$ROOT_DIR/tests/fixtures/fileconflicts" \
--output $tmpdir/testbuild \
--packages $tmpdir/testbuild \
--name "test" \
--descr "Test Repo" \
--urls $tmpdir/testrootfs \
--type disk > /dev/null
createst=$?
assertEquals 'create repo successfully' "$createst" "0"
assertTrue 'create repository' "[ -e '$tmpdir/testbuild/repository.yaml' ]"
}
testConfig() {
mkdir $tmpdir/testrootfs
cat <<EOF > $tmpdir/luet.yaml
general:
debug: true
system:
rootfs: $tmpdir/testrootfs
database_path: "/"
database_engine: "boltdb"
config_from_host: true
repositories:
- name: "main"
type: "disk"
enable: true
urls:
- "$tmpdir/testbuild"
EOF
luet config --config $tmpdir/luet.yaml
res=$?
assertEquals 'config test successfully' "$res" "0"
}
testInstall() {
luet install -y --config $tmpdir/luet.yaml test1/conflict test2/conflict
#luet install -y --config $tmpdir/luet.yaml test/c@1.0 > /dev/null
installst=$?
assertEquals 'install test failed' "$installst" "1"
#assertTrue 'package installed' "[ -e '$tmpdir/testrootfs/c' ]"
}
testReInstall() {
luet install -y --config $tmpdir/luet.yaml test1/conflict
#luet install -y --config $tmpdir/luet.yaml test/c@1.0 > /dev/null
installst=$?
assertEquals 'install test succeeded' "$installst" "0"
luet install -y --config $tmpdir/luet.yaml test2/conflict
#luet install -y --config $tmpdir/luet.yaml test/c@1.0 > /dev/null
installst=$?
assertEquals 'install test succeeded' "$installst" "1"
luet install -y --force --config $tmpdir/luet.yaml test2/conflict
#luet install -y --config $tmpdir/luet.yaml test/c@1.0 > /dev/null
installst=$?
assertEquals 'install test succeeded' "$installst" "0"
}
# Load shUnit2.
. "$ROOT_DIR/tests/integration/shunit2"/shunit2

View File

@@ -0,0 +1,81 @@
#!/bin/bash
export LUET_NOLOCK=true
oneTimeSetUp() {
export tmpdir="$(mktemp -d)"
}
oneTimeTearDown() {
rm -rf "$tmpdir"
}
testBuild() {
mkdir $tmpdir/testbuild
luet build --tree "$ROOT_DIR/tests/fixtures/fileconflicts_upgrade" --destination $tmpdir/testbuild --compression gzip --all
buildst=$?
assertEquals 'builds successfully' "$buildst" "0"
assertTrue 'create packages' "[ -e '$tmpdir/testbuild/conflict-test1-1.0.package.tar.gz' ]"
assertTrue 'create packages' "[ -e '$tmpdir/testbuild/conflict-test2-1.0.package.tar.gz' ]"
}
testRepo() {
assertTrue 'no repository' "[ ! -e '$tmpdir/testbuild/repository.yaml' ]"
luet create-repo --tree "$ROOT_DIR/tests/fixtures/fileconflicts_upgrade" \
--output $tmpdir/testbuild \
--packages $tmpdir/testbuild \
--name "test" \
--descr "Test Repo" \
--urls $tmpdir/testrootfs \
--type disk > /dev/null
createst=$?
assertEquals 'create repo successfully' "$createst" "0"
assertTrue 'create repository' "[ -e '$tmpdir/testbuild/repository.yaml' ]"
}
testConfig() {
mkdir $tmpdir/testrootfs
cat <<EOF > $tmpdir/luet.yaml
general:
debug: true
system:
rootfs: $tmpdir/testrootfs
database_path: "/"
database_engine: "boltdb"
config_from_host: true
repositories:
- name: "main"
type: "disk"
enable: true
urls:
- "$tmpdir/testbuild"
EOF
luet config --config $tmpdir/luet.yaml
res=$?
assertEquals 'config test successfully' "$res" "0"
}
testInstall() {
luet install -y --force --config $tmpdir/luet.yaml test1/conflict@1.0 test2/conflict@1.0
#luet install -y --config $tmpdir/luet.yaml test/c@1.0 > /dev/null
installst=$?
assertEquals 'install test succeded' "$installst" "0"
#assertTrue 'package installed' "[ -e '$tmpdir/testrootfs/c' ]"
}
testUpgrade() {
out=$(luet upgrade -y --config $tmpdir/luet.yaml)
installst=$?
assertEquals 'install test succeeded' "$installst" "1"
assertContains 'does find conflicts' "$out" "Error: file conflict found: file conflict between packages to be installed"
luet upgrade -y --config $tmpdir/luet.yaml --force
#luet install -y --config $tmpdir/luet.yaml test/c@1.0 > /dev/null
installst=$?
assertEquals 'install test succeeded' "$installst" "0"
}
# Load shUnit2.
. "$ROOT_DIR/tests/integration/shunit2"/shunit2

View File

@@ -0,0 +1,70 @@
#!/bin/bash
export LUET_NOLOCK=true
oneTimeSetUp() {
export tmpdir="$(mktemp -d)"
}
oneTimeTearDown() {
rm -rf "$tmpdir"
}
testBuild() {
mkdir $tmpdir/testbuild
luet build --tree "$ROOT_DIR/tests/fixtures/fileconflicts" --destination $tmpdir/testbuild --compression gzip --all
buildst=$?
assertEquals 'builds successfully' "$buildst" "0"
assertTrue 'create packages' "[ -e '$tmpdir/testbuild/conflict-test1-1.0.package.tar.gz' ]"
assertTrue 'create packages' "[ -e '$tmpdir/testbuild/conflict-test2-1.0.package.tar.gz' ]"
}
testRepo() {
assertTrue 'no repository' "[ ! -e '$tmpdir/testbuild/repository.yaml' ]"
luet create-repo --tree "$ROOT_DIR/tests/fixtures/fileconflicts" \
--output $tmpdir/testbuild \
--packages $tmpdir/testbuild \
--name "test" \
--descr "Test Repo" \
--urls $tmpdir/testrootfs \
--type disk > /dev/null
createst=$?
assertEquals 'create repo successfully' "$createst" "0"
assertTrue 'create repository' "[ -e '$tmpdir/testbuild/repository.yaml' ]"
}
testConfig() {
mkdir $tmpdir/testrootfs
cat <<EOF > $tmpdir/luet.yaml
general:
debug: true
system:
rootfs: $tmpdir/testrootfs
database_path: "/"
database_engine: "boltdb"
config_from_host: true
repositories:
- name: "main"
type: "disk"
enable: true
urls:
- "$tmpdir/testbuild"
EOF
luet config --config $tmpdir/luet.yaml
res=$?
assertEquals 'config test successfully' "$res" "0"
}
testReInstall() {
luet install -y --config $tmpdir/luet.yaml test1/conflict
installst=$?
assertEquals 'install test succeeded' "$installst" "0"
luet reinstall -y --config $tmpdir/luet.yaml test1/conflict
installst=$?
assertEquals 'reinstall test succeeded' "$installst" "0"
}
# Load shUnit2.
. "$ROOT_DIR/tests/integration/shunit2"/shunit2

58
vendor/github.com/docker/cli/AUTHORS generated vendored
View File

@@ -8,6 +8,7 @@ Aaron.L.Xu <likexu@harmonycloud.cn>
Abdur Rehman <abdur_rehman@mentor.com>
Abhinandan Prativadi <abhi@docker.com>
Abin Shahab <ashahab@altiscale.com>
Abreto FU <public@abreto.email>
Ace Tang <aceapril@126.com>
Addam Hardy <addam.hardy@gmail.com>
Adolfo Ochagavía <aochagavia92@gmail.com>
@@ -17,12 +18,15 @@ Adrien Folie <folie.adrien@gmail.com>
Ahmet Alp Balkan <ahmetb@microsoft.com>
Aidan Feldman <aidan.feldman@gmail.com>
Aidan Hobson Sayers <aidanhs@cantab.net>
AJ Bowen <aj@gandi.net>
Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
AJ Bowen <aj@soulshake.net>
Akhil Mohan <akhil.mohan@mayadata.io>
Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Akim Demaille <akim.demaille@docker.com>
Alan Thompson <cloojure@gmail.com>
Albert Callarisa <shark234@gmail.com>
Albin Kerouanton <albin@akerouanton.name>
Aleksa Sarai <asarai@suse.de>
Aleksander Piotrowski <apiotrowski312@gmail.com>
Alessandro Boch <aboch@tetrationanalytics.com>
Alex Mavrogiannis <alex.mavrogiannis@docker.com>
Alex Mayer <amayer5125@gmail.com>
@@ -40,6 +44,7 @@ Amir Goldstein <amir73il@aquasec.com>
Amit Krishnan <amit.krishnan@oracle.com>
Amit Shukla <amit.shukla@docker.com>
Amy Lindburg <amy.lindburg@docker.com>
Anca Iordache <anca.iordache@docker.com>
Anda Xu <anda.xu@docker.com>
Andrea Luzzardi <aluzzardi@gmail.com>
Andreas Köhler <andi5.py@gmx.net>
@@ -49,6 +54,7 @@ Andrew Macpherson <hopscotch23@gmail.com>
Andrew McDonnell <bugs@andrewmcdonnell.net>
Andrew Po <absourd.noise@gmail.com>
Andrey Petrov <andrey.petrov@shazow.net>
Andrii Berehuliak <berkusandrew@gmail.com>
André Martins <aanm90@gmail.com>
Andy Goldstein <agoldste@redhat.com>
Andy Rothfusz <github@developersupport.net>
@@ -61,7 +67,9 @@ Antonis Kalipetis <akalipetis@gmail.com>
Anusha Ragunathan <anusha.ragunathan@docker.com>
Ao Li <la9249@163.com>
Arash Deshmeh <adeshmeh@ca.ibm.com>
Arko Dasgupta <arko.dasgupta@docker.com>
Arnaud Porterie <arnaud.porterie@docker.com>
Arthur Peka <arthur.peka@outlook.com>
Ashwini Oruganti <ashwini.oruganti@gmail.com>
Azat Khuyiyakhmetov <shadow_uz@mail.ru>
Bardia Keyoumarsi <bkeyouma@ucsc.edu>
@@ -87,6 +95,7 @@ Brent Salisbury <brent.salisbury@docker.com>
Bret Fisher <bret@bretfisher.com>
Brian (bex) Exelbierd <bexelbie@redhat.com>
Brian Goff <cpuguy83@gmail.com>
Brian Wieder <brian@4wieders.com>
Bryan Bess <squarejaw@bsbess.com>
Bryan Boreham <bjboreham@gmail.com>
Bryan Murphy <bmurphy1976@gmail.com>
@@ -95,6 +104,7 @@ Cameron Spear <cameronspear@gmail.com>
Cao Weiwei <cao.weiwei30@zte.com.cn>
Carlo Mion <mion00@gmail.com>
Carlos Alexandro Becker <caarlos0@gmail.com>
Carlos de Paula <me@carlosedp.com>
Ce Gao <ce.gao@outlook.com>
Cedric Davies <cedricda@microsoft.com>
Cezar Sa Espinola <cezarsa@gmail.com>
@@ -128,26 +138,31 @@ Coenraad Loubser <coenraad@wish.org.za>
Colin Hebert <hebert.colin@gmail.com>
Collin Guarino <collin.guarino@gmail.com>
Colm Hally <colmhally@gmail.com>
Comical Derskeal <27731088+derskeal@users.noreply.github.com>
Corey Farrell <git@cfware.com>
Corey Quon <corey.quon@docker.com>
Craig Wilhite <crwilhit@microsoft.com>
Cristian Staretu <cristian.staretu@gmail.com>
Daehyeok Mun <daehyeok@gmail.com>
Dafydd Crosby <dtcrsby@gmail.com>
Daisuke Ito <itodaisuke00@gmail.com>
dalanlan <dalanlan925@gmail.com>
Damien Nadé <github@livna.org>
Dan Cotora <dan@bluevision.ro>
Daniel Artine <daniel.artine@ufrj.br>
Daniel Cassidy <mail@danielcassidy.me.uk>
Daniel Dao <dqminh@cloudflare.com>
Daniel Farrell <dfarrell@redhat.com>
Daniel Gasienica <daniel@gasienica.ch>
Daniel Goosen <daniel.goosen@surveysampling.com>
Daniel Helfand <dhelfand@redhat.com>
Daniel Hiltgen <daniel.hiltgen@docker.com>
Daniel J Walsh <dwalsh@redhat.com>
Daniel Nephin <dnephin@docker.com>
Daniel Norberg <dano@spotify.com>
Daniel Watkins <daniel@daniel-watkins.co.uk>
Daniel Zhang <jmzwcn@gmail.com>
Daniil Nikolenko <qoo2p5@gmail.com>
Danny Berger <dpb587@gmail.com>
Darren Shepherd <darren.s.shepherd@gmail.com>
Darren Stahl <darst@microsoft.com>
@@ -180,13 +195,15 @@ Dima Stopel <dima@twistlock.com>
Dimitry Andric <d.andric@activevideo.com>
Ding Fei <dingfei@stars.org.cn>
Diogo Monica <diogo@docker.com>
Djordje Lukic <djordje.lukic@docker.com>
Dmitry Gusev <dmitry.gusev@gmail.com>
Dmitry Smirnov <onlyjob@member.fsf.org>
Dmitry V. Krivenok <krivenok.dmitry@gmail.com>
Dominik Braun <dominik.braun@nbsp.de>
Don Kjer <don.kjer@gmail.com>
Dong Chen <dongluo.chen@docker.com>
Doug Davis <dug@us.ibm.com>
Drew Erny <drew.erny@docker.com>
Drew Erny <derny@mirantis.com>
Ed Costello <epc@epcostello.com>
Elango Sivanandam <elango.siva@docker.com>
Eli Uriegas <eli.uriegas@docker.com>
@@ -249,6 +266,7 @@ Harald Albers <github@albersweb.de>
Harold Cooper <hrldcpr@gmail.com>
Harry Zhang <harryz@hyper.sh>
He Simei <hesimei@zju.edu.cn>
Hector S <hfsam88@gmail.com>
Helen Xie <chenjg@harmonycloud.cn>
Henning Sprang <henning.sprang@gmail.com>
Henry N <henrynmail-github@yahoo.de>
@@ -256,6 +274,7 @@ Hernan Garcia <hernandanielg@gmail.com>
Hongbin Lu <hongbin034@gmail.com>
Hu Keping <hukeping@huawei.com>
Huayi Zhang <irachex@gmail.com>
Hugo Gabriel Eyherabide <hugogabriel.eyherabide@gmail.com>
huqun <huqun@zju.edu.cn>
Huu Nguyen <huu@prismskylabs.com>
Hyzhou Zhy <hyzhou.zhy@alibaba-inc.com>
@@ -297,7 +316,7 @@ Jeremy Unruh <jeremybunruh@gmail.com>
Jeremy Yallop <yallop@docker.com>
Jeroen Franse <jeroenfranse@gmail.com>
Jesse Adametz <jesseadametz@gmail.com>
Jessica Frazelle <jessfraz@google.com>
Jessica Frazelle <jess@oxide.computer>
Jezeniel Zapanta <jpzapanta22@gmail.com>
Jian Zhang <zhangjian.fnst@cn.fujitsu.com>
Jie Luo <luo612@zju.edu.cn>
@@ -308,6 +327,7 @@ Jimmy Song <rootsongjc@gmail.com>
jimmyxian <jimmyxian2004@yahoo.com.cn>
Jintao Zhang <zhangjintao9020@gmail.com>
Joao Fernandes <joao.fernandes@docker.com>
Joe Abbey <joe.abbey@gmail.com>
Joe Doliner <jdoliner@pachyderm.io>
Joe Gordon <joe.gordon0@gmail.com>
Joel Handwell <joelhandwell@gmail.com>
@@ -317,7 +337,7 @@ Johan Euphrosine <proppy@google.com>
Johannes 'fish' Ziemke <github@freigeist.org>
John Feminella <jxf@jxf.me>
John Harris <john@johnharris.io>
John Howard (VM) <John.Howard@microsoft.com>
John Howard <github@lowenna.com>
John Laswell <john.n.laswell@gmail.com>
John Maguire <jmaguire@duosecurity.com>
John Mulhausen <john@docker.com>
@@ -326,12 +346,15 @@ John Stephens <johnstep@docker.com>
John Tims <john.k.tims@gmail.com>
John V. Martinez <jvmatl@gmail.com>
John Willis <john.willis@docker.com>
Jon Johnson <jonjohnson@google.com>
Jonatas Baldin <jonatas.baldin@gmail.com>
Jonathan Boulle <jonathanboulle@gmail.com>
Jonathan Lee <jonjohn1232009@gmail.com>
Jonathan Lomas <jonathan@floatinglomas.ca>
Jonathan McCrohan <jmccrohan@gmail.com>
Jonh Wendell <jonh.wendell@redhat.com>
Jordan Jennings <jjn2009@gmail.com>
Jose J. Escobar <53836904+jescobar-docker@users.noreply.github.com>
Joseph Kern <jkern@semafour.net>
Josh Bodah <jb3689@yahoo.com>
Josh Chorlton <jchorlton@gmail.com>
@@ -369,6 +392,7 @@ Kevin Kern <kaiwentan@harmonycloud.cn>
Kevin Kirsche <Kev.Kirsche+GitHub@gmail.com>
Kevin Meredith <kevin.m.meredith@gmail.com>
Kevin Richardson <kevin@kevinrichardson.co>
Kevin Woblick <mail@kovah.de>
khaled souf <khaled.souf@gmail.com>
Kim Eik <kim@heldig.org>
Kir Kolyshkin <kolyshkin@gmail.com>
@@ -406,13 +430,16 @@ Luca Favatella <luca.favatella@erlang-solutions.com>
Luca Marturana <lucamarturana@gmail.com>
Lucas Chan <lucas-github@lucaschan.com>
Luka Hartwig <mail@lukahartwig.de>
Lukas Heeren <lukas-heeren@hotmail.com>
Lukasz Zajaczkowski <Lukasz.Zajaczkowski@ts.fujitsu.com>
Lydell Manganti <LydellManganti@users.noreply.github.com>
Lénaïc Huard <lhuard@amadeus.com>
Ma Shimiao <mashimiao.fnst@cn.fujitsu.com>
Mabin <bin.ma@huawei.com>
Maciej Kalisz <maciej.d.kalisz@gmail.com>
Madhav Puri <madhav.puri@gmail.com>
Madhu Venugopal <madhu@socketplane.io>
Madhur Batra <madhurbatra097@gmail.com>
Malte Janduda <mail@janduda.net>
Manjunath A Kumatagi <mkumatag@in.ibm.com>
Mansi Nahar <mmn4185@rit.edu>
@@ -422,6 +449,7 @@ Marco Mariani <marco.mariani@alterway.fr>
Marco Vedovati <mvedovati@suse.com>
Marcus Martins <marcus@docker.com>
Marianna Tessel <mtesselh@gmail.com>
Marius Ileana <marius.ileana@gmail.com>
Marius Sturm <marius@graylog.com>
Mark Oates <fl0yd@me.com>
Marsh Macy <marsma@microsoft.com>
@@ -467,12 +495,14 @@ mikelinjie <294893458@qq.com>
Mikhail Vasin <vasin@cloud-tv.ru>
Milind Chawre <milindchawre@gmail.com>
Mindaugas Rukas <momomg@gmail.com>
Miroslav Gula <miroslav.gula@naytrolabs.com>
Misty Stanley-Jones <misty@docker.com>
Mohammad Banikazemi <mb@us.ibm.com>
Mohammed Aaqib Ansari <maaquib@gmail.com>
Mohini Anne Dsouza <mohini3917@gmail.com>
Moorthy RS <rsmoorthy@gmail.com>
Morgan Bauer <mbauer@us.ibm.com>
Morten Hekkvang <morten.hekkvang@sbab.se>
Moysés Borges <moysesb@gmail.com>
Mrunal Patel <mrunalp@gmail.com>
muicoder <muicoder@gmail.com>
@@ -503,9 +533,11 @@ Nishant Totla <nishanttotla@gmail.com>
NIWA Hideyuki <niwa.niwa@nifty.ne.jp>
Noah Treuhaft <noah.treuhaft@docker.com>
O.S. Tezer <ostezer@gmail.com>
Odin Ugedal <odin@ugedal.com>
ohmystack <jun.jiang02@ele.me>
Olle Jonsson <olle.jonsson@gmail.com>
Olli Janatuinen <olli.janatuinen@gmail.com>
Oscar Wieman <oscrx@icloud.com>
Otto Kekäläinen <otto@seravo.fi>
Ovidio Mallo <ovidio.mallo@gmail.com>
Pascal Borreli <pascal@borreli.com>
@@ -515,6 +547,7 @@ Patrick Lang <plang@microsoft.com>
Paul <paul9869@gmail.com>
Paul Kehrer <paul.l.kehrer@gmail.com>
Paul Lietar <paul@lietar.net>
Paul Mulders <justinkb@gmail.com>
Paul Weaver <pauweave@cisco.com>
Pavel Pospisil <pospispa@gmail.com>
Paweł Szczekutowicz <pszczekutowicz@gmail.com>
@@ -541,6 +574,7 @@ Qiang Huang <h.huangqiang@huawei.com>
Qinglan Peng <qinglanpeng@zju.edu.cn>
qudongfang <qudongfang@gmail.com>
Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>
Rahul Zoldyck <rahulzoldyck@gmail.com>
Ravi Shekhar Jethani <rsjethani@gmail.com>
Ray Tsang <rayt@google.com>
Reficul <xuzhenglun@gmail.com>
@@ -553,6 +587,7 @@ Richard Scothern <richard.scothern@gmail.com>
Rick Wieman <git@rickw.nl>
Ritesh H Shukla <sritesh@vmware.com>
Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
Rob Gulewich <rgulewich@netflix.com>
Robert Wallis <smilingrob@gmail.com>
Robin Naundorf <r.naundorf@fh-muenster.de>
Robin Speekenbrink <robin@kingsquare.nl>
@@ -574,10 +609,14 @@ Sainath Grandhi <sainath.grandhi@intel.com>
Sakeven Jiang <jc5930@sina.cn>
Sally O'Malley <somalley@redhat.com>
Sam Neirinck <sam@samneirinck.com>
Samarth Shah <samashah@microsoft.com>
Sambuddha Basu <sambuddhabasu1@gmail.com>
Sami Tabet <salph.tabet@gmail.com>
Samuel Cochran <sj26@sj26.com>
Samuel Karp <skarp@amazon.com>
Santhosh Manohar <santhosh@docker.com>
Sargun Dhillon <sargun@netflix.com>
Saswat Bhattacharya <sas.saswat@gmail.com>
Scott Brenner <scott@scottbrenner.me>
Scott Collier <emailscottcollier@gmail.com>
Sean Christopherson <sean.j.christopherson@intel.com>
@@ -598,6 +637,7 @@ sidharthamani <sid@rancher.com>
Silvin Lubecki <silvin.lubecki@docker.com>
Simei He <hesimei@zju.edu.cn>
Simon Ferquel <simon.ferquel@docker.com>
Simon Heimberg <simon.heimberg@heimberg-ea.ch>
Sindhu S <sindhus@live.in>
Slava Semushin <semushin@redhat.com>
Solomon Hykes <solomon@docker.com>
@@ -627,7 +667,10 @@ TAGOMORI Satoshi <tagomoris@gmail.com>
taiji-tech <csuhqg@foxmail.com>
Taylor Jones <monitorjbl@gmail.com>
Tejaswini Duggaraju <naduggar@microsoft.com>
Tengfei Wang <tfwang@alauda.io>
Teppei Fukuda <knqyf263@gmail.com>
Thatcher Peskens <thatcher@docker.com>
Thibault Coupin <thibault.coupin@gmail.com>
Thomas Gazagnaire <thomas@gazagnaire.org>
Thomas Krzero <thomas.kovatchitch@gmail.com>
Thomas Leonard <thomas.leonard@docker.com>
@@ -639,6 +682,7 @@ Tianyi Wang <capkurmagati@gmail.com>
Tibor Vass <teabee89@gmail.com>
Tim Dettrick <t.dettrick@uq.edu.au>
Tim Hockin <thockin@google.com>
Tim Sampson <tim@sampson.fi>
Tim Smith <timbot@google.com>
Tim Waugh <twaugh@redhat.com>
Tim Wraight <tim.wraight@tangentlabs.co.uk>
@@ -663,9 +707,11 @@ Tristan Carel <tristan@cogniteev.com>
Tycho Andersen <tycho@docker.com>
Tycho Andersen <tycho@tycho.ws>
uhayate <uhayate.gong@daocloud.io>
Ulrich Bareth <ulrich.bareth@gmail.com>
Ulysses Souza <ulysses.souza@docker.com>
Umesh Yadav <umesh4257@gmail.com>
Valentin Lorentz <progval+git@progval.net>
Venkateswara Reddy Bukkasamudram <bukkasamudram@outlook.com>
Veres Lajos <vlajos@gmail.com>
Victor Vieux <victor.vieux@docker.com>
Victoria Bialas <victoria.bialas@docker.com>
@@ -683,6 +729,7 @@ Wang Long <long.wanglong@huawei.com>
Wang Ping <present.wp@icloud.com>
Wang Xing <hzwangxing@corp.netease.com>
Wang Yuexiao <wang.yuexiao@zte.com.cn>
Wang Yumu <37442693@qq.com>
Wataru Ishida <ishida.wataru@lab.ntt.co.jp>
Wayne Song <wsong@docker.com>
Wen Cheng Ma <wenchma@cn.ibm.com>
@@ -691,6 +738,7 @@ Wes Morgan <cap10morgan@gmail.com>
Wewang Xiaorenfine <wang.xiaoren@zte.com.cn>
William Henry <whenry@redhat.com>
Xianglin Gao <xlgao@zju.edu.cn>
Xiaodong Liu <liuxiaodong@loongson.cn>
Xiaodong Zhang <a4012017@sina.com>
Xiaoxi He <xxhe@alauda.io>
Xinbo Weng <xihuanbo_0521@zju.edu.cn>

Some files were not shown because too many files have changed in this diff Show More