Tag 0.10.0

Grab latest version, if not get a default since luet auto updates.
Check is ran as normal user or sudo/root.
Autoinstall repos -y.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: mudler
+
+---
+
+<!-- Thanks for helping us to improve Luet! We welcome all bug reports. Please fill out each area of the template so we can better help you. Comments like this will be hidden when you post but you can delete them if you wish. -->
+
+**Luet version:**
+<!-- Provide the output from "luet --version" -->
+
+**CPU architecture, OS, and Version:**
+<!-- Provide the output from "uname -a"  -->
+
+**Describe the bug**
+<!-- A clear and concise description of what the bug is. -->
+
+**To Reproduce**
+<!-- Steps to reproduce the behavior, including the luet command used -->
+
+**Expected behavior**
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Logs**
+<!-- If applicable, add logs with the "--debug" flag enabled to help explain your problem. -->
+
+**Additional context**
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,22 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: mudler
+
+---
+
+<!-- Thanks for helping us to improve Luet! We welcome all feature requests. Please fill out each area of the template so we can better help you. Comments like this will be hidden when you post but you can delete them if you wish. -->
+
+**Is your feature request related to a problem? Please describe.**
+<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]  -->
+
+**Describe the solution you'd like**
+<!-- A clear and concise description of what you want to happen.  -->
+
+**Describe alternatives you've considered**
+<!-- A clear and concise description of any alternative solutions or features you've considered.  -->
+
+**Additional context**
+<!-- Add any other context or screenshots about the feature request here. -->

.github/workflows/release.yml

@@ -11,10 +11,22 @@ jobs:
           go-version: 1.14.x
       - name: Checkout code
         uses: actions/checkout@v2
+      - name: setup-docker
+        uses: docker-practice/actions-setup-docker@0.0.1
+      - name: Login to quay
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Install deps
+        run: sudo apt-get install -y upx
       - name: Tests
-        run: sudo -E env "PATH=$PATH" make deps multiarch-build test-integration test-coverage
+        run: |
+              sudo -E \
+                    env "PATH=$PATH" \
+                    env "TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                    env "UNIT_TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                    env "UNIT_TEST_DOCKER_IMAGE_REPOSITORY=${{ secrets.DOCKER_TESTING_UNIT_TEST_IMAGE }}" \
+                    make deps multiarch-build-small test-integration test-coverage
       - name: Build
-        run: sudo -E env "PATH=$PATH" make multiarch-build && sudo chmod -R 777 release/
+        run: sudo -E env "PATH=$PATH" make multiarch-build-small && sudo chmod -R 777 release/
       - name: Release
         uses: fnkr/github-action-ghr@v1
         if: startsWith(github.ref, 'refs/tags/')

.github/workflows/test.yml

@@ -17,5 +17,7 @@ jobs:
       uses: actions/checkout@v2
     - name: setup-docker
       uses: docker-practice/actions-setup-docker@0.0.1
+    - name: Install deps
+      run: sudo apt-get install -y upx
     - name: Tests
-      run: sudo -E env "PATH=$PATH" make deps multiarch-build test-integration test-coverage
+      run: sudo -E env "PATH=$PATH" make deps multiarch-build-small test-integration test-coverage

Makefile

@@ -88,6 +88,11 @@ test-docker:
 				--workdir /go/src/github.com/mudler/luet -ti golang:latest \
 				bash -c "make test"
 
-.PHONY: multiarch-build
 multiarch-build:
 	CGO_ENABLED=0 gox $(BUILD_PLATFORMS) -ldflags '$(LDFLAGS)' -output="release/$(NAME)-$(VERSION)-{{.OS}}-{{.Arch}}"
+
+multiarch-build-small:
+	@$(MAKE) LDFLAGS+="-s -w" multiarch-build
+	for file in $(ROOT_DIR)/release/* ; do \
+		upx --brute -1 $${file} ; \
+	done

README.md

@@ -8,7 +8,7 @@
 
 Luet is a multi-platform Package Manager based off from containers - it uses Docker (and others) to build packages. It has zero dependencies and it is well suitable for "from scratch" environments. It can also version entire rootfs and enables delivery of OTA-alike updates, making it a perfect fit for the Edge computing era and IoT embedded devices.
 
-It offers a simple [specfile format](https://luet-lab.github.io/docs/docs/concepts/specfile/) in YAML notation to define both packages and rootfs. As it is based on containers, it can be also used to build stages for Linux From Scratch installations and it can build and track updates for those systems.
+It offers a simple [specfile format](https://luet-lab.github.io/docs/docs/concepts/packages/specfile/) in YAML notation to define both [packages](https://luet-lab.github.io/docs/docs/concepts/packages/) and [rootfs](https://luet-lab.github.io/docs/docs/concepts/packages/#package-layers). As it is based on containers, it can be also used to build stages for Linux From Scratch installations and it can build and track updates for those systems.
 
 It is written entirely in Golang and where used as package manager, it can run in from scratch environment, with zero dependencies.
 
@@ -18,10 +18,11 @@ It is written entirely in Golang and where used as package manager, it can run i
 - It builds, installs, uninstalls and perform upgrades on machines
 - Installer doesn't depend on anything ( 0 dep installer !), statically built
 - You can install it aside also with your current distro package manager, and start building and distributing your packages
-- Support for packages as "layers"
-- [It uses SAT solving techniques to solve the deptree](https://luet-lab.github.io/docs/docs/concepts/constraints/) ( Inspired by [OPIUM](https://ranjitjhala.github.io/static/opium.pdf) )
-- Support for collections and templated package definitions
-- [Can be extended with Plugins and Extensions](https://luet-lab.github.io/docs/docs/plugins-and-extensions/)
+- [Support for packages as "layers"](https://luet-lab.github.io/docs/docs/concepts/packages/specfile/#building-strategies)
+- [It uses SAT solving techniques to solve the deptree](https://luet-lab.github.io/docs/docs/concepts/overview/constraints/) ( Inspired by [OPIUM](https://ranjitjhala.github.io/static/opium.pdf) )
+- Support for [collections](https://luet-lab.github.io/docs/docs/concepts/packages/collections/) and [templated package definitions](https://luet-lab.github.io/docs/docs/concepts/packages/templates/)
+- [Can be extended with Plugins and Extensions](https://luet-lab.github.io/docs/docs/concepts/plugins-and-extensions/)
+- [Can build packages in Kubernetes (experimental)](https://github.com/mudler/luet-k8s)
 
 ## Install
 
@@ -50,7 +51,7 @@ run `luet --help`,  any subcommand is documented as well, try e.g.: `luet build
 # Dependency solving
 
 Luet uses SAT and Reinforcement learning engine for dependency solving.
-It encodes the package requirements into a SAT problem, using gophersat to solve the dependency tree and give a concrete model as result.
+It encodes the package requirements into a SAT problem, using [gophersat](https://github.com/crillab/gophersat) to solve the dependency tree and give a concrete model as result.
 
 ## SAT encoding
 

cmd/build.go

@@ -18,6 +18,7 @@ import (
 	"fmt"
 	"io/ioutil"
 	"os"
+	"path/filepath"
 
 	"github.com/ghodss/yaml"
 	helpers "github.com/mudler/luet/cmd/helpers"
@@ -60,7 +61,6 @@ Build packages specifying multiple definition trees:
 
 	$ luet build --tree overlay/path --tree overlay/path2 utils/yq ...
 `, PreRun: func(cmd *cobra.Command, args []string) {
-		viper.BindPFlag("clean", cmd.Flags().Lookup("clean"))
 		viper.BindPFlag("tree", cmd.Flags().Lookup("tree"))
 		viper.BindPFlag("destination", cmd.Flags().Lookup("destination"))
 		viper.BindPFlag("backend", cmd.Flags().Lookup("backend"))
@@ -76,6 +76,7 @@ Build packages specifying multiple definition trees:
 		viper.BindPFlag("image-repository", cmd.Flags().Lookup("image-repository"))
 		viper.BindPFlag("push", cmd.Flags().Lookup("push"))
 		viper.BindPFlag("pull", cmd.Flags().Lookup("pull"))
+		viper.BindPFlag("wait", cmd.Flags().Lookup("wait"))
 		viper.BindPFlag("keep-images", cmd.Flags().Lookup("keep-images"))
 
 		LuetCfg.Viper.BindPFlag("keep-exported-images", cmd.Flags().Lookup("keep-exported-images"))
@@ -87,7 +88,6 @@ Build packages specifying multiple definition trees:
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 
-		clean := viper.GetBool("clean")
 		treePaths := viper.GetStringSlice("tree")
 		dst := viper.GetString("destination")
 		concurrency := LuetCfg.GetGeneral().Concurrency
@@ -99,7 +99,7 @@ Build packages specifying multiple definition trees:
 		compressionType := viper.GetString("compression")
 		imageRepository := viper.GetString("image-repository")
 		values := viper.GetString("values")
-
+		wait := viper.GetBool("wait")
 		push := viper.GetBool("push")
 		pull := viper.GetBool("pull")
 		keepImages := viper.GetBool("keep-images")
@@ -108,7 +108,6 @@ Build packages specifying multiple definition trees:
 		keepExportedImages := viper.GetBool("keep-exported-images")
 		onlyTarget, _ := cmd.Flags().GetBool("only-target-package")
 		full, _ := cmd.Flags().GetBool("full")
-		skip, _ := cmd.Flags().GetBool("skip-if-metadata-exists")
 		concurrent, _ := cmd.Flags().GetBool("solver-concurrent")
 		var results Results
 
@@ -118,14 +117,9 @@ Build packages specifying multiple definition trees:
 		}
 		pretend, _ := cmd.Flags().GetBool("pretend")
 		compilerSpecs := compiler.NewLuetCompilationspecs()
-		var compilerBackend compiler.CompilerBackend
 		var db pkg.PackageDatabase
-		switch backendType {
-		case "img":
-			compilerBackend = backend.NewSimpleImgBackend()
-		case "docker":
-			compilerBackend = backend.NewSimpleDockerBackend()
-		}
+
+		compilerBackend := backend.NewBackend(backendType)
 
 		switch databaseType {
 		case "memory":
@@ -143,10 +137,6 @@ Build packages specifying multiple definition trees:
 
 		generalRecipe := tree.NewCompilerRecipe(db)
 
-		if len(treePaths) <= 0 {
-			Fatal("No tree path supplied!")
-		}
-
 		for _, src := range treePaths {
 			Info("Loading tree", src)
 
@@ -173,14 +163,13 @@ Build packages specifying multiple definition trees:
 		opts := compiler.NewDefaultCompilerOptions()
 		opts.SolverOptions = *LuetCfg.GetSolverOptions()
 		opts.ImageRepository = imageRepository
-		opts.Clean = clean
 		opts.PullFirst = pull
 		opts.KeepImg = keepImages
 		opts.Push = push
 		opts.OnlyDeps = onlydeps
 		opts.NoDeps = nodeps
+		opts.Wait = wait
 		opts.KeepImageExport = keepExportedImages
-		opts.SkipIfMetadataExists = skip
 		opts.PackageTargetOnly = onlyTarget
 		opts.BuildValuesFile = values
 		var solverOpts solver.Options
@@ -308,8 +297,7 @@ func init() {
 	if err != nil {
 		Fatal(err)
 	}
-	buildCmd.Flags().Bool("clean", true, "Build all packages without considering the packages present in the build directory")
-	buildCmd.Flags().StringSliceP("tree", "t", []string{}, "Path of the tree to use.")
+	buildCmd.Flags().StringSliceP("tree", "t", []string{path}, "Path of the tree to use.")
 	buildCmd.Flags().String("backend", "docker", "backend used (docker,img)")
 	buildCmd.Flags().Bool("privileged", false, "Privileged (Keep permissions)")
 	buildCmd.Flags().String("database", "memory", "database used for solving (memory,boltdb)")
@@ -318,16 +306,16 @@ func init() {
 	buildCmd.Flags().Bool("full", false, "Build all packages (optimized)")
 	buildCmd.Flags().String("values", "", "Build values file to interpolate with each package")
 
-	buildCmd.Flags().String("destination", path, "Destination folder")
-	buildCmd.Flags().String("compression", "none", "Compression alg: none, gzip")
+	buildCmd.Flags().String("destination", filepath.Join(path, "build"), "Destination folder")
+	buildCmd.Flags().String("compression", "none", "Compression alg: none, gzip, zstd")
 	buildCmd.Flags().String("image-repository", "luet/cache", "Default base image string for generated image")
 	buildCmd.Flags().Bool("push", false, "Push images to a hub")
 	buildCmd.Flags().Bool("pull", false, "Pull images from a hub")
+	buildCmd.Flags().Bool("wait", false, "Don't build all intermediate images, but wait for them until they are available")
 	buildCmd.Flags().Bool("keep-images", true, "Keep built docker images in the host")
 	buildCmd.Flags().Bool("nodeps", false, "Build only the target packages, skipping deps (it works only if you already built the deps locally, or by using --pull) ")
 	buildCmd.Flags().Bool("onlydeps", false, "Build only package dependencies")
 	buildCmd.Flags().Bool("keep-exported-images", false, "Keep exported images used during building")
-	buildCmd.Flags().Bool("skip-if-metadata-exists", false, "Skip package if metadata exists")
 	buildCmd.Flags().Bool("only-target-package", false, "Build packages of only the required target. Otherwise builds all the necessary ones not present in the destination")
 	buildCmd.Flags().String("solver-type", "", "Solver strategy")
 	buildCmd.Flags().Float32("solver-rate", 0.7, "Solver learning rate")

cmd/create-repo.go

@@ -16,8 +16,10 @@ package cmd
 
 import (
 	"os"
+	"path/filepath"
 
 	"github.com/mudler/luet/pkg/compiler"
+	"github.com/mudler/luet/pkg/compiler/backend"
 	. "github.com/mudler/luet/pkg/config"
 	installer "github.com/mudler/luet/pkg/installer"
 	. "github.com/mudler/luet/pkg/logger"
@@ -54,6 +56,7 @@ Create a repository from the metadata description defined in the luet.yaml confi
 		viper.BindPFlag("packages", cmd.Flags().Lookup("packages"))
 		viper.BindPFlag("tree", cmd.Flags().Lookup("tree"))
 		viper.BindPFlag("output", cmd.Flags().Lookup("output"))
+		viper.BindPFlag("backend", cmd.Flags().Lookup("backend"))
 		viper.BindPFlag("name", cmd.Flags().Lookup("name"))
 		viper.BindPFlag("descr", cmd.Flags().Lookup("descr"))
 		viper.BindPFlag("urls", cmd.Flags().Lookup("urls"))
@@ -64,6 +67,10 @@ Create a repository from the metadata description defined in the luet.yaml confi
 		viper.BindPFlag("meta-filename", cmd.Flags().Lookup("meta-filename"))
 		viper.BindPFlag("reset-revision", cmd.Flags().Lookup("reset-revision"))
 		viper.BindPFlag("repo", cmd.Flags().Lookup("repo"))
+
+		viper.BindPFlag("force-push", cmd.Flags().Lookup("force-push"))
+		viper.BindPFlag("push-images", cmd.Flags().Lookup("push-images"))
+
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 		var err error
@@ -82,9 +89,13 @@ Create a repository from the metadata description defined in the luet.yaml confi
 		metatype := viper.GetString("meta-compression")
 		metaName := viper.GetString("meta-filename")
 		source_repo := viper.GetString("repo")
+		backendType := viper.GetString("backend")
 
 		treeFile := installer.NewDefaultTreeRepositoryFile()
 		metaFile := installer.NewDefaultMetaRepositoryFile()
+		compilerBackend := backend.NewBackend(backendType)
+		force := viper.GetBool("force-push")
+		imagePush := viper.GetBool("push-images")
 
 		if source_repo != "" {
 			// Search for system repository
@@ -107,11 +118,11 @@ Create a repository from the metadata description defined in the luet.yaml confi
 				lrepo.Priority,
 				packages,
 				treePaths,
-				pkg.NewInMemoryDatabase(false))
+				pkg.NewInMemoryDatabase(false), compilerBackend, dst, imagePush, force)
 
 		} else {
 			repo, err = installer.GenerateRepository(name, descr, t, urls, 1, packages,
-				treePaths, pkg.NewInMemoryDatabase(false))
+				treePaths, pkg.NewInMemoryDatabase(false), compilerBackend, dst, imagePush, force)
 		}
 
 		if err != nil {
@@ -137,7 +148,7 @@ Create a repository from the metadata description defined in the luet.yaml confi
 		repo.SetRepositoryFile(installer.REPOFILE_TREE_KEY, treeFile)
 		repo.SetRepositoryFile(installer.REPOFILE_META_KEY, metaFile)
 
-		err = repo.Write(dst, reset)
+		err = repo.Write(dst, reset, true)
 		if err != nil {
 			Fatal("Error: " + err.Error())
 		}
@@ -149,19 +160,23 @@ func init() {
 	if err != nil {
 		Fatal(err)
 	}
-	createrepoCmd.Flags().String("packages", path, "Packages folder (output from build)")
-	createrepoCmd.Flags().StringSliceP("tree", "t", []string{}, "Path of the source trees to use.")
-	createrepoCmd.Flags().String("output", path, "Destination folder")
+	createrepoCmd.Flags().String("packages", filepath.Join(path, "build"), "Packages folder (output from build)")
+	createrepoCmd.Flags().StringSliceP("tree", "t", []string{path}, "Path of the source trees to use.")
+	createrepoCmd.Flags().String("output", filepath.Join(path, "build"), "Destination for generated archives. With 'docker' repository type, it should be an image reference (e.g 'foo/bar')")
 	createrepoCmd.Flags().String("name", "luet", "Repository name")
 	createrepoCmd.Flags().String("descr", "luet", "Repository description")
 	createrepoCmd.Flags().StringSlice("urls", []string{}, "Repository URLs")
-	createrepoCmd.Flags().String("type", "disk", "Repository type (disk)")
+	createrepoCmd.Flags().String("type", "disk", "Repository type (disk, http, docker)")
 	createrepoCmd.Flags().Bool("reset-revision", false, "Reset repository revision.")
 	createrepoCmd.Flags().String("repo", "", "Use repository defined in configuration.")
+	createrepoCmd.Flags().String("backend", "docker", "backend used (docker,img)")
 
-	createrepoCmd.Flags().String("tree-compression", "gzip", "Compression alg: none, gzip")
+	createrepoCmd.Flags().Bool("force-push", false, "Force overwrite of docker images if already present online")
+	createrepoCmd.Flags().Bool("push-images", false, "Enable/Disable docker image push for docker repositories")
+
+	createrepoCmd.Flags().String("tree-compression", "gzip", "Compression alg: none, gzip, zstd")
 	createrepoCmd.Flags().String("tree-filename", installer.TREE_TARBALL, "Repository tree filename")
-	createrepoCmd.Flags().String("meta-compression", "none", "Compression alg: none, gzip")
+	createrepoCmd.Flags().String("meta-compression", "none", "Compression alg: none, gzip, zstd")
 	createrepoCmd.Flags().String("meta-filename", installer.REPOSITORY_METAFILE+".tar", "Repository metadata filename")
 
 	RootCmd.AddCommand(createrepoCmd)

cmd/database/create.go

@@ -17,7 +17,6 @@ package cmd_database
 
 import (
 	"io/ioutil"
-	"path/filepath"
 
 	"github.com/mudler/luet/pkg/compiler"
 	. "github.com/mudler/luet/pkg/logger"
@@ -51,7 +50,7 @@ For reference, inspect a "metadata.yaml" file generated while running "luet buil
 		},
 		Run: func(cmd *cobra.Command, args []string) {
 
-			var systemDB pkg.PackageDatabase
+			systemDB := LuetCfg.GetSystemDB()
 
 			for _, a := range args {
 				dat, err := ioutil.ReadFile(a)
@@ -63,13 +62,6 @@ For reference, inspect a "metadata.yaml" file generated while running "luet buil
 					Fatal("Failed reading yaml ", a, ": ", err.Error())
 				}
 
-				if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-					systemDB = pkg.NewBoltDatabase(
-						filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-				} else {
-					systemDB = pkg.NewInMemoryDatabase(true)
-				}
-
 				files := art.GetFiles()
 
 				if _, err := systemDB.CreatePackage(art.GetCompileSpec().GetPackage()); err != nil {

cmd/database/remove.go

@@ -16,13 +16,10 @@
 package cmd_database
 
 import (
-	"path/filepath"
-
 	. "github.com/mudler/luet/pkg/logger"
 
 	helpers "github.com/mudler/luet/cmd/helpers"
 	. "github.com/mudler/luet/pkg/config"
-	pkg "github.com/mudler/luet/pkg/package"
 
 	"github.com/spf13/cobra"
 )
@@ -44,7 +41,7 @@ This commands takes multiple packages as arguments and prunes their entries from
 
 		},
 		Run: func(cmd *cobra.Command, args []string) {
-			var systemDB pkg.PackageDatabase
+			systemDB := LuetCfg.GetSystemDB()
 
 			for _, a := range args {
 				pack, err := helpers.ParsePackageStr(a)
@@ -52,13 +49,6 @@ This commands takes multiple packages as arguments and prunes their entries from
 					Fatal("Invalid package string ", a, ": ", err.Error())
 				}
 
-				if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-					systemDB = pkg.NewBoltDatabase(
-						filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-				} else {
-					systemDB = pkg.NewInMemoryDatabase(true)
-				}
-
 				if err := systemDB.RemovePackage(pack); err != nil {
 					Fatal("Failed removing ", a, ": ", err.Error())
 				}

cmd/install.go

@@ -16,7 +16,6 @@ package cmd
 
 import (
 	"os"
-	"path/filepath"
 
 	installer "github.com/mudler/luet/pkg/installer"
 	"github.com/mudler/luet/pkg/solver"
@@ -63,7 +62,6 @@ To force install a package:
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 		var toInstall pkg.Packages
-		var systemDB pkg.PackageDatabase
 
 		for _, a := range args {
 			pack, err := helpers.ParsePackageStr(a)
@@ -120,13 +118,7 @@ To force install a package:
 		})
 		inst.Repositories(repos)
 
-		if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-			systemDB = pkg.NewBoltDatabase(
-				filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-		} else {
-			systemDB = pkg.NewInMemoryDatabase(true)
-		}
-		system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
+		system := &installer.System{Database: LuetCfg.GetSystemDB(), Target: LuetCfg.GetSystem().Rootfs}
 		err := inst.Install(toInstall, system)
 		if err != nil {
 			Fatal("Error: " + err.Error())

cmd/reclaim.go

@@ -16,13 +16,11 @@ package cmd
 
 import (
 	"os"
-	"path/filepath"
 
 	installer "github.com/mudler/luet/pkg/installer"
 
 	. "github.com/mudler/luet/pkg/config"
 	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
 
 	"github.com/spf13/cobra"
 )
@@ -42,7 +40,6 @@ var reclaimCmd = &cobra.Command{
 It scans the target file system, and if finds a match with a package available in the repositories, it marks as installed in the system database.
 `,
 	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
 
 		// This shouldn't be necessary, but we need to unmarshal the repositories to a concrete struct, thus we need to port them back to the Repositories type
 		repos := installer.Repositories{}
@@ -65,13 +62,7 @@ It scans the target file system, and if finds a match with a package available i
 		})
 		inst.Repositories(repos)
 
-		if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-			systemDB = pkg.NewBoltDatabase(
-				filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-		} else {
-			systemDB = pkg.NewInMemoryDatabase(true)
-		}
-		system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
+		system := &installer.System{Database: LuetCfg.GetSystemDB(), Target: LuetCfg.GetSystem().Rootfs}
 		err := inst.Reclaim(system)
 		if err != nil {
 			Fatal("Error: " + err.Error())

cmd/replace.go

@@ -16,7 +16,6 @@ package cmd
 
 import (
 	"os"
-	"path/filepath"
 
 	installer "github.com/mudler/luet/pkg/installer"
 	"github.com/mudler/luet/pkg/solver"
@@ -54,7 +53,6 @@ var replaceCmd = &cobra.Command{
 	Run: func(cmd *cobra.Command, args []string) {
 		var toUninstall pkg.Packages
 		var toAdd pkg.Packages
-		var systemDB pkg.PackageDatabase
 
 		f := LuetCfg.Viper.GetStringSlice("for")
 		stype := LuetCfg.Viper.GetString("solver.type")
@@ -120,13 +118,7 @@ var replaceCmd = &cobra.Command{
 		})
 		inst.Repositories(repos)
 
-		if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-			systemDB = pkg.NewBoltDatabase(
-				filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-		} else {
-			systemDB = pkg.NewInMemoryDatabase(true)
-		}
-		system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
+		system := &installer.System{Database: LuetCfg.GetSystemDB(), Target: LuetCfg.GetSystem().Rootfs}
 		err := inst.Swap(toUninstall, toAdd, system)
 		if err != nil {
 			Fatal("Error: " + err.Error())

cmd/root.go

@@ -40,7 +40,7 @@ var Verbose bool
 var LockedCommands = []string{"install", "uninstall", "upgrade"}
 
 const (
-	LuetCLIVersion = "0.9.12"
+	LuetCLIVersion = "0.10.0"
 	LuetEnvPrefix  = "LUET"
 )
 

cmd/search.go

@@ -17,7 +17,6 @@ package cmd
 import (
 	"fmt"
 	"os"
-	"path/filepath"
 	"strings"
 
 	"github.com/ghodss/yaml"
@@ -112,7 +111,6 @@ Search can also return results in the terminal in different ways: as terminal ou
 		LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
 	},
 	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
 		var results Results
 		if len(args) > 1 {
 			Fatal("Wrong number of arguments (expected 1)")
@@ -213,13 +211,7 @@ Search can also return results in the terminal in different ways: as terminal ou
 			}
 		} else {
 
-			if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-				systemDB = pkg.NewBoltDatabase(
-					filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-			} else {
-				systemDB = pkg.NewInMemoryDatabase(true)
-			}
-			system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
+			system := &installer.System{Database: LuetCfg.GetSystemDB(), Target: LuetCfg.GetSystem().Rootfs}
 
 			var err error
 			iMatches := pkg.Packages{}

cmd/tree/images.go

@@ -18,6 +18,7 @@ package cmd_tree
 
 import (
 	"fmt"
+	"os"
 
 	//. "github.com/mudler/luet/pkg/config"
 	"github.com/ghodss/yaml"
@@ -127,9 +128,12 @@ func NewTreeImageCommand() *cobra.Command {
 			}
 		},
 	}
-
+	path, err := os.Getwd()
+	if err != nil {
+		Fatal(err)
+	}
 	ans.Flags().StringP("output", "o", "terminal", "Output format ( Defaults: terminal, available: json,yaml )")
-	ans.Flags().StringArrayP("tree", "t", []string{}, "Path of the tree to use.")
+	ans.Flags().StringArrayP("tree", "t", []string{path}, "Path of the tree to use.")
 	ans.Flags().String("image-repository", "luet/cache", "Default base image string for generated image")
 
 	return ans

cmd/tree/pkglist.go

@@ -18,6 +18,7 @@ package cmd_tree
 
 import (
 	"fmt"
+	"os"
 	"sort"
 
 	//. "github.com/mudler/luet/pkg/config"
@@ -266,7 +267,10 @@ func NewTreePkglistCommand() *cobra.Command {
 
 		},
 	}
-
+	path, err := os.Getwd()
+	if err != nil {
+		Fatal(err)
+	}
 	ans.Flags().BoolP("buildtime", "b", false, "Build time match")
 	ans.Flags().StringP("output", "o", "terminal", "Output format ( Defaults: terminal, available: json,yaml )")
 	ans.Flags().Bool("revdeps", false, "Search package reverse dependencies")
@@ -274,7 +278,7 @@ func NewTreePkglistCommand() *cobra.Command {
 
 	ans.Flags().BoolP("verbose", "v", false, "Add package version")
 	ans.Flags().BoolP("full", "f", false, "Show package detail")
-	ans.Flags().StringArrayP("tree", "t", []string{}, "Path of the tree to use.")
+	ans.Flags().StringArrayP("tree", "t", []string{path}, "Path of the tree to use.")
 	ans.Flags().StringSliceVarP(&matches, "matches", "m", []string{},
 		"Include only matched packages from list. (Use string as regex).")
 	ans.Flags().StringSliceVarP(&excludes, "exclude", "e", []string{},

cmd/tree/validate.go

@@ -458,12 +458,15 @@ func NewTreeValidateCommand() *cobra.Command {
 			}
 		},
 	}
-
+	path, err := os.Getwd()
+	if err != nil {
+		Fatal(err)
+	}
 	ans.Flags().Bool("only-runtime", false, "Check only runtime dependencies.")
 	ans.Flags().Bool("only-buildtime", false, "Check only buildtime dependencies.")
 	ans.Flags().BoolP("with-solver", "s", false,
 		"Enable check of requires also with solver.")
-	ans.Flags().StringSliceVarP(&treePaths, "tree", "t", []string{},
+	ans.Flags().StringSliceVarP(&treePaths, "tree", "t", []string{path},
 		"Path of the tree to use.")
 	ans.Flags().StringSliceVarP(&excludes, "exclude", "e", []string{},
 		"Exclude matched packages from analysis. (Use string as regex).")

cmd/uninstall.go

@@ -16,7 +16,6 @@ package cmd
 
 import (
 	"os"
-	"path/filepath"
 
 	helpers "github.com/mudler/luet/cmd/helpers"
 	. "github.com/mudler/luet/pkg/config"
@@ -45,63 +44,58 @@ var uninstallCmd = &cobra.Command{
 		LuetCfg.Viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
 	},
 	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
-
+		toRemove := []pkg.Package{}
 		for _, a := range args {
 
 			pack, err := helpers.ParsePackageStr(a)
 			if err != nil {
 				Fatal("Invalid package string ", a, ": ", err.Error())
 			}
+			toRemove = append(toRemove, pack)
+		}
 
-			stype := LuetCfg.Viper.GetString("solver.type")
-			discount := LuetCfg.Viper.GetFloat64("solver.discount")
-			rate := LuetCfg.Viper.GetFloat64("solver.rate")
-			attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
-			force := LuetCfg.Viper.GetBool("force")
-			nodeps, _ := cmd.Flags().GetBool("nodeps")
-			full, _ := cmd.Flags().GetBool("full")
-			checkconflicts, _ := cmd.Flags().GetBool("conflictscheck")
-			fullClean, _ := cmd.Flags().GetBool("full-clean")
-			concurrent, _ := cmd.Flags().GetBool("solver-concurrent")
-			yes := LuetCfg.Viper.GetBool("yes")
+		stype := LuetCfg.Viper.GetString("solver.type")
+		discount := LuetCfg.Viper.GetFloat64("solver.discount")
+		rate := LuetCfg.Viper.GetFloat64("solver.rate")
+		attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
+		force := LuetCfg.Viper.GetBool("force")
+		nodeps, _ := cmd.Flags().GetBool("nodeps")
+		full, _ := cmd.Flags().GetBool("full")
+		checkconflicts, _ := cmd.Flags().GetBool("conflictscheck")
+		fullClean, _ := cmd.Flags().GetBool("full-clean")
+		concurrent, _ := cmd.Flags().GetBool("solver-concurrent")
+		yes := LuetCfg.Viper.GetBool("yes")
 
-			LuetCfg.GetSolverOptions().Type = stype
-			LuetCfg.GetSolverOptions().LearnRate = float32(rate)
-			LuetCfg.GetSolverOptions().Discount = float32(discount)
-			LuetCfg.GetSolverOptions().MaxAttempts = attempts
-			if concurrent {
-				LuetCfg.GetSolverOptions().Implementation = solver.ParallelSimple
-			} else {
-				LuetCfg.GetSolverOptions().Implementation = solver.SingleCoreSimple
-			}
-			Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
+		LuetCfg.GetSolverOptions().Type = stype
+		LuetCfg.GetSolverOptions().LearnRate = float32(rate)
+		LuetCfg.GetSolverOptions().Discount = float32(discount)
+		LuetCfg.GetSolverOptions().MaxAttempts = attempts
+		if concurrent {
+			LuetCfg.GetSolverOptions().Implementation = solver.ParallelSimple
+		} else {
+			LuetCfg.GetSolverOptions().Implementation = solver.SingleCoreSimple
+		}
+		Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
 
-			// Load config protect configs
-			installer.LoadConfigProtectConfs(LuetCfg)
+		// Load config protect configs
+		installer.LoadConfigProtectConfs(LuetCfg)
 
-			inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
-				Concurrency:        LuetCfg.GetGeneral().Concurrency,
-				SolverOptions:      *LuetCfg.GetSolverOptions(),
-				NoDeps:             nodeps,
-				Force:              force,
-				FullUninstall:      full,
-				FullCleanUninstall: fullClean,
-				CheckConflicts:     checkconflicts,
-				Ask:                !yes,
-			})
+		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
+			Concurrency:                 LuetCfg.GetGeneral().Concurrency,
+			SolverOptions:               *LuetCfg.GetSolverOptions(),
+			NoDeps:                      nodeps,
+			Force:                       force,
+			FullUninstall:               full,
+			FullCleanUninstall:          fullClean,
+			CheckConflicts:              checkconflicts,
+			Ask:                         !yes,
+			PreserveSystemEssentialData: true,
+		})
 
-			if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-				systemDB = pkg.NewBoltDatabase(
-					filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-			} else {
-				systemDB = pkg.NewInMemoryDatabase(true)
-			}
-			system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
-			err = inst.Uninstall(pack, system)
-			if err != nil {
-				Fatal("Error: " + err.Error())
-			}
+		system := &installer.System{Database: LuetCfg.GetSystemDB(), Target: LuetCfg.GetSystem().Rootfs}
+
+		if err := inst.Uninstall(system, toRemove...); err != nil {
+			Fatal("Error: " + err.Error())
 		}
 	},
 }

cmd/upgrade.go

@@ -16,12 +16,10 @@ package cmd
 
 import (
 	"os"
-	"path/filepath"
 
 	. "github.com/mudler/luet/pkg/config"
 	installer "github.com/mudler/luet/pkg/installer"
 	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
 	"github.com/mudler/luet/pkg/solver"
 
 	"github.com/spf13/cobra"
@@ -43,7 +41,6 @@ var upgradeCmd = &cobra.Command{
 	},
 	Long: `Upgrades packages in parallel`,
 	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
 
 		repos := installer.Repositories{}
 		for _, repo := range LuetCfg.SystemRepositories {
@@ -84,26 +81,20 @@ var upgradeCmd = &cobra.Command{
 		installer.LoadConfigProtectConfs(LuetCfg)
 
 		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
-			Concurrency:                LuetCfg.GetGeneral().Concurrency,
-			SolverOptions:              *LuetCfg.GetSolverOptions(),
-			Force:                      force,
-			FullUninstall:              full,
-			NoDeps:                     nodeps,
-			SolverUpgrade:              universe,
-			RemoveUnavailableOnUpgrade: clean,
-			UpgradeNewRevisions:        sync,
-			Ask:                        !yes,
+			Concurrency:                 LuetCfg.GetGeneral().Concurrency,
+			SolverOptions:               *LuetCfg.GetSolverOptions(),
+			Force:                       force,
+			FullUninstall:               full,
+			NoDeps:                      nodeps,
+			SolverUpgrade:               universe,
+			RemoveUnavailableOnUpgrade:  clean,
+			UpgradeNewRevisions:         sync,
+			PreserveSystemEssentialData: true,
+			Ask:                         !yes,
 		})
 		inst.Repositories(repos)
 
-		if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-			systemDB = pkg.NewBoltDatabase(
-				filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-		} else {
-			systemDB = pkg.NewInMemoryDatabase(true)
-		}
-		system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
-
+		system := &installer.System{Database: LuetCfg.GetSystemDB(), Target: LuetCfg.GetSystem().Rootfs}
 		if err := inst.Upgrade(system); err != nil {
 			Fatal("Error: " + err.Error())
 		}

contrib/config/get_luet_root.sh

@@ -1,14 +1,19 @@
 #!/bin/bash
+if [ $(id -u) -ne 0 ]
+  then echo "Please run the installer with sudo/as root"
+  exit
+fi
+
 set -ex
 export LUET_NOLOCK=true
 
-LUET_VERSION=0.8.6
+LUET_VERSION=$(curl -s https://api.github.com/repos/mudler/luet/releases/latest | ( grep -oP '"tag_name": "\K(.*)(?=")' || echo "0.9.24" ))
 LUET_ROOTFS=${LUET_ROOTFS:-/}
 LUET_DATABASE_PATH=${LUET_DATABASE_PATH:-/var/luet/db}
 LUET_DATABASE_ENGINE=${LUET_DATABASE_ENGINE:-boltdb}
 LUET_CONFIG_PROTECT=${LUET_CONFIG_PROTECT:-1}
 
-wget -q https://github.com/mudler/luet/releases/download/0.8.6/luet-0.8.6-linux-amd64 -O luet
+curl -L https://github.com/mudler/luet/releases/download/${LUET_VERSION}/luet-${LUET_VERSION}-linux-amd64 --output luet
 chmod +x luet
 
 mkdir -p /etc/luet/repos.conf.d || true
@@ -17,9 +22,9 @@ mkdir -p /var/tmp/luet || true
 
 if [ "${LUET_CONFIG_PROTECT}" = "1" ] ; then
   mkdir -p /etc/luet/config.protect.d || true
-  wget -q  https://raw.githubusercontent.com/mudler/luet/master/contrib/config/config.protect.d/01_etc.yml.example -O /etc/luet/config.protect.d/01_etc.yml
+  curl -L https://raw.githubusercontent.com/mudler/luet/master/contrib/config/config.protect.d/01_etc.yml.example --output /etc/luet/config.protect.d/01_etc.yml
 fi
-wget -q https://raw.githubusercontent.com/mocaccinoOS/repository-index/master/packages/mocaccino-repository-index/mocaccino-repository-index.yml -O /etc/luet/repos.conf.d/mocaccino-repository-index.yml
+curl -L https://raw.githubusercontent.com/mocaccinoOS/repository-index/master/packages/mocaccino-repository-index.yml --output /etc/luet/repos.conf.d/mocaccino-repository-index.yml
 
 cat > /etc/luet/luet.yaml <<EOF
 general:
@@ -31,7 +36,8 @@ system:
   tmpdir_base: "/var/tmp/luet"
 EOF
 
-./luet install repository/luet repository/mocaccino-repository-index
-./luet install system/luet system/luet-extensions
+./luet install -y repository/luet repository/mocaccino-repository-index
+./luet install -y system/luet system/luet-extensions
 
 rm -rf luet
+

go.mod

@@ -1,6 +1,6 @@
 module github.com/mudler/luet
 
-go 1.12
+go 1.14
 
 require (
 	github.com/DataDog/zstd v1.4.4 // indirect
@@ -8,44 +8,58 @@ require (
 	github.com/asdine/storm v0.0.0-20190418133842-e0f77eada154
 	github.com/briandowns/spinner v1.7.0
 	github.com/cavaliercoder/grab v1.0.1-0.20201108051000-98a5bfe305ec
+	github.com/containerd/containerd v1.4.1-0.20201117152358-0edc412565dc
 	github.com/crillab/gophersat v1.3.2-0.20201023142334-3fc2ac466765
-	github.com/docker/docker v17.12.0-ce-rc1.0.20200417035958-130b0bc6032c+incompatible
-	github.com/docker/go-events v0.0.0-20190806004212-e31b211e4f1c // indirect
+	github.com/docker/distribution v2.7.1+incompatible
+	github.com/docker/docker v20.10.0-beta1.0.20201110211921-af34b94a78a1+incompatible
+	github.com/docker/go-units v0.4.0
 	github.com/ecooper/qlearning v0.0.0-20160612200101-3075011a69fd
 	github.com/fsouza/go-dockerclient v1.6.4
+	github.com/genuinetools/img v0.5.11
 	github.com/ghodss/yaml v1.0.0
+	github.com/google/go-containerregistry v0.2.1
+	github.com/hashicorp/go-multierror v1.0.0
 	github.com/hashicorp/go-version v1.2.0
 	github.com/jedib0t/go-pretty v4.3.0+incompatible
 	github.com/jedib0t/go-pretty/v6 v6.0.5
 	github.com/jinzhu/copier v0.0.0-20180308034124-7e38e58719c3
+	github.com/klauspost/compress v1.8.3
 	github.com/klauspost/pgzip v1.2.1
 	github.com/knqyf263/go-deb-version v0.0.0-20190517075300-09fca494f03d
-	github.com/kr/pretty v0.2.0 // indirect
 	github.com/kyokomi/emoji v2.1.0+incompatible
 	github.com/logrusorgru/aurora v0.0.0-20190417123914-21d75270181e
 	github.com/marcsauter/single v0.0.0-20181104081128-f8bf46f26ec0
-	github.com/moby/sys/mount v0.1.1-0.20200320164225-6154f11e6840 // indirect
+	github.com/moby/buildkit v0.7.2
+	github.com/moby/sys/mount v0.2.0 // indirect
 	github.com/mudler/cobra-extensions v0.0.0-20200612154940-31a47105fe3d
 	github.com/mudler/docker-companion v0.4.6-0.20200418093252-41846f112d87
 	github.com/mudler/go-pluggable v0.0.0-20201113184918-d36448fc8f82
 	github.com/mudler/topsort v0.0.0-20201103161459-db5c7901c290
 	github.com/onsi/ginkgo v1.14.2
 	github.com/onsi/gomega v1.10.3
+	github.com/opencontainers/image-spec v1.0.1
 	github.com/otiai10/copy v1.2.1-0.20200916181228-26f84a0b1578
-	github.com/pelletier/go-toml v1.6.0 // indirect
 	github.com/philopon/go-toposort v0.0.0-20170620085441-9be86dbd762f
 	github.com/pkg/errors v0.9.1
 	github.com/schollz/progressbar/v3 v3.7.1
+	github.com/sirupsen/logrus v1.6.0
 	github.com/spf13/cobra v1.0.0
-	github.com/spf13/viper v1.6.3
-	go.etcd.io/bbolt v1.3.4
+	github.com/spf13/viper v1.7.0
+	go.etcd.io/bbolt v1.3.5
 	go.uber.org/atomic v1.5.1 // indirect
 	go.uber.org/multierr v1.4.0 // indirect
 	go.uber.org/zap v1.13.0
-	golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f // indirect
 	gopkg.in/yaml.v2 v2.3.0
-	gotest.tools/v3 v3.0.2 // indirect
 	helm.sh/helm/v3 v3.3.4
+
 )
 
 replace github.com/docker/docker => github.com/Luet-lab/moby v17.12.0-ce-rc1.0.20200605210607-749178b8f80d+incompatible
+
+replace github.com/containerd/containerd => github.com/containerd/containerd v1.3.1-0.20200227195959-4d242818bf55
+
+replace github.com/hashicorp/go-immutable-radix => github.com/tonistiigi/go-immutable-radix v0.0.0-20170803185627-826af9ccf0fe
+
+replace github.com/jaguilar/vt100 => github.com/tonistiigi/vt100 v0.0.0-20190402012908-ad4c4a574305
+
+replace github.com/opencontainers/runc => github.com/opencontainers/runc v1.0.0-rc9.0.20200221051241-688cf6d43cc4

pkg/bus/events.go

@@ -16,9 +16,9 @@ var (
 
 	// EventPackagePreBuild is the event fired before a package is being built
 	EventPackagePreBuild pluggable.EventType = "package.pre.build"
-	// EventPackagePreBuildArtifact is the event fired before a package artifact is being built
+	// EventPackagePreBuildArtifact is the event fired before a package tarball is being generated
 	EventPackagePreBuildArtifact pluggable.EventType = "package.pre.build_artifact"
-	// EventPackagePostBuildArtifact is the event fired after a package artifact was built
+	// EventPackagePostBuildArtifact is the event fired after a package tarball is generated
 	EventPackagePostBuildArtifact pluggable.EventType = "package.post.build_artifact"
 	// EventPackagePostBuild is the event fired after a package was built
 	EventPackagePostBuild pluggable.EventType = "package.post.build"

pkg/compiler/artifact.go

@@ -28,6 +28,7 @@ import (
 	"regexp"
 
 	system "github.com/docker/docker/pkg/system"
+	zstd "github.com/klauspost/compress/zstd"
 	gzip "github.com/klauspost/pgzip"
 
 	//"strconv"
@@ -47,8 +48,9 @@ import (
 type CompressionImplementation string
 
 const (
-	None CompressionImplementation = "none" // e.g. tar for standard packages
-	GZip CompressionImplementation = "gzip"
+	None      CompressionImplementation = "none" // e.g. tar for standard packages
+	GZip      CompressionImplementation = "gzip"
+	Zstandard CompressionImplementation = "zstd"
 )
 
 type ArtifactIndex []Artifact
@@ -142,14 +144,14 @@ func (a *PackageArtifact) Hash() error {
 
 func (a *PackageArtifact) Verify() error {
 	sum := Checksums{}
-	err := sum.Generate(a)
-	if err != nil {
+	if err := sum.Generate(a); err != nil {
 		return err
 	}
-	err = sum.Compare(a.Checksums)
-	if err != nil {
+
+	if err := sum.Compare(a.Checksums); err != nil {
 		return err
 	}
+
 	return nil
 }
 
@@ -235,13 +237,121 @@ func (a *PackageArtifact) GetPath() string {
 	return a.Path
 }
 
+func (a *PackageArtifact) GetFileName() string {
+	return path.Base(a.GetPath())
+}
+
 func (a *PackageArtifact) SetPath(p string) {
 	a.Path = p
 }
 
-// Compress Archives and compress (TODO) to the artifact path
+func (a *PackageArtifact) genDockerfile() string {
+	return `
+FROM scratch
+COPY * /`
+}
+
+// CreateArtifactForFile creates a new artifact from the given file
+func CreateArtifactForFile(s string, opts ...func(*PackageArtifact)) (*PackageArtifact, error) {
+
+	fileName := path.Base(s)
+	archive, err := LuetCfg.GetSystem().TempDir("archive")
+	if err != nil {
+		return nil, errors.Wrap(err, "error met while creating tempdir for "+s)
+	}
+	defer os.RemoveAll(archive) // clean up
+	helpers.CopyFile(s, filepath.Join(archive, fileName))
+	artifact, err := LuetCfg.GetSystem().TempDir("artifact")
+	if err != nil {
+		return nil, errors.Wrap(err, "error met while creating tempdir for "+s)
+	}
+	a := &PackageArtifact{Path: filepath.Join(artifact, fileName)}
+
+	for _, o := range opts {
+		o(a)
+	}
+
+	return a, a.Compress(archive, 1)
+}
+
+// GenerateFinalImage takes an artifact and builds a Docker image with its content
+func (a *PackageArtifact) GenerateFinalImage(imageName string, b CompilerBackend, keepPerms bool) (CompilerBackendOptions, error) {
+	builderOpts := CompilerBackendOptions{}
+	archive, err := LuetCfg.GetSystem().TempDir("archive")
+	if err != nil {
+		return builderOpts, errors.Wrap(err, "error met while creating tempdir for "+a.Path)
+	}
+	defer os.RemoveAll(archive) // clean up
+
+	uncompressedFiles := filepath.Join(archive, "files")
+	dockerFile := filepath.Join(archive, "Dockerfile")
+
+	if err := os.MkdirAll(uncompressedFiles, os.ModePerm); err != nil {
+		return builderOpts, errors.Wrap(err, "error met while creating tempdir for "+a.Path)
+	}
+
+	data := a.genDockerfile()
+	if err := ioutil.WriteFile(dockerFile, []byte(data), 0644); err != nil {
+		return builderOpts, errors.Wrap(err, "error met while rendering artifact dockerfile "+a.Path)
+	}
+
+	if err := a.Unpack(uncompressedFiles, keepPerms); err != nil {
+		return builderOpts, errors.Wrap(err, "error met while uncompressing artifact "+a.Path)
+	}
+
+	builderOpts = CompilerBackendOptions{
+		ImageName:      imageName,
+		SourcePath:     archive,
+		DockerFileName: dockerFile,
+		Context:        uncompressedFiles,
+	}
+	return builderOpts, b.BuildImage(builderOpts)
+}
+
+// Compress is responsible to archive and compress to the artifact Path.
+// It accepts a source path, which is the content to be archived/compressed
+// and a concurrency parameter.
 func (a *PackageArtifact) Compress(src string, concurrency int) error {
 	switch a.CompressionType {
+
+	case Zstandard:
+		err := helpers.Tar(src, a.Path)
+		if err != nil {
+			return err
+		}
+		original, err := os.Open(a.Path)
+		if err != nil {
+			return err
+		}
+		defer original.Close()
+
+		zstdFile := a.getCompressedName()
+		bufferedReader := bufio.NewReader(original)
+
+		// Open a file for writing.
+		dst, err := os.Create(zstdFile)
+		if err != nil {
+			return err
+		}
+
+		enc, err := zstd.NewWriter(dst)
+		if err != nil {
+			return err
+		}
+		_, err = io.Copy(enc, bufferedReader)
+		if err != nil {
+			enc.Close()
+			return err
+		}
+		if err := enc.Close(); err != nil {
+			return err
+		}
+
+		os.RemoveAll(a.Path) // Remove original
+		Debug("Removed artifact", a.Path)
+
+		a.Path = zstdFile
+		return nil
 	case GZip:
 		err := helpers.Tar(src, a.Path)
 		if err != nil {
@@ -253,7 +363,7 @@ func (a *PackageArtifact) Compress(src string, concurrency int) error {
 		}
 		defer original.Close()
 
-		gzipfile := a.Path + ".gz"
+		gzipfile := a.getCompressedName()
 		bufferedReader := bufio.NewReader(original)
 
 		// Open a file for writing.
@@ -272,6 +382,7 @@ func (a *PackageArtifact) Compress(src string, concurrency int) error {
 		}
 		w.Close()
 		os.RemoveAll(a.Path) // Remove original
+		Debug("Removed artifact", a.Path)
 		//	a.CompressedPath = gzipfile
 		a.Path = gzipfile
 		return nil
@@ -279,12 +390,31 @@ func (a *PackageArtifact) Compress(src string, concurrency int) error {
 
 	// Defaults to tar only (covers when "none" is supplied)
 	default:
-		return helpers.Tar(src, a.Path)
-
+		return helpers.Tar(src, a.getCompressedName())
 	}
 	return errors.New("Compression type must be supplied")
 }
 
+func (a *PackageArtifact) getCompressedName() string {
+	switch a.CompressionType {
+	case Zstandard:
+		return a.Path + ".zst"
+
+	case GZip:
+		return a.Path + ".gz"
+	}
+	return a.Path
+}
+
+// GetUncompressedName returns the artifact path without the extension suffix
+func (a *PackageArtifact) GetUncompressedName() string {
+	switch a.CompressionType {
+	case Zstandard, GZip:
+		return strings.TrimSuffix(a.Path, filepath.Ext(a.Path))
+	}
+	return a.Path
+}
+
 func tarModifierWrapperFunc(dst, path string, header *tar.Header, content io.Reader) (*tar.Header, []byte, error) {
 	// If the destination path already exists I rename target file name with postfix.
 	var destPath string
@@ -367,6 +497,40 @@ func (a *PackageArtifact) Unpack(dst string, keepPerms bool) error {
 	tarModifier := helpers.NewTarModifierWrapper(dst, tarModifierWrapperFunc)
 
 	switch a.CompressionType {
+	case Zstandard:
+		// Create the uncompressed archive
+		archive, err := os.Create(a.GetPath() + ".uncompressed")
+		if err != nil {
+			return err
+		}
+		defer os.RemoveAll(a.GetPath() + ".uncompressed")
+		defer archive.Close()
+
+		original, err := os.Open(a.Path)
+		if err != nil {
+			return errors.Wrap(err, "Cannot open "+a.Path)
+		}
+		defer original.Close()
+
+		bufferedReader := bufio.NewReader(original)
+
+		d, err := zstd.NewReader(bufferedReader)
+		if err != nil {
+			return err
+		}
+		defer d.Close()
+
+		_, err = io.Copy(archive, d)
+		if err != nil {
+			return errors.Wrap(err, "Cannot copy to "+a.GetPath()+".uncompressed")
+		}
+
+		err = helpers.UntarProtect(a.GetPath()+".uncompressed", dst,
+			LuetCfg.GetGeneral().SameOwner, protectedFiles, tarModifier)
+		if err != nil {
+			return err
+		}
+		return nil
 	case GZip:
 		// Create the uncompressed archive
 		archive, err := os.Create(a.GetPath() + ".uncompressed")
@@ -412,6 +576,27 @@ func (a *PackageArtifact) Unpack(dst string, keepPerms bool) error {
 func (a *PackageArtifact) FileList() ([]string, error) {
 	var tr *tar.Reader
 	switch a.CompressionType {
+	case Zstandard:
+		archive, err := os.Create(a.GetPath() + ".uncompressed")
+		if err != nil {
+			return []string{}, err
+		}
+		defer os.RemoveAll(a.GetPath() + ".uncompressed")
+		defer archive.Close()
+
+		original, err := os.Open(a.Path)
+		if err != nil {
+			return []string{}, errors.Wrap(err, "Cannot open "+a.Path)
+		}
+		defer original.Close()
+
+		bufferedReader := bufio.NewReader(original)
+		r, err := zstd.NewReader(bufferedReader)
+		if err != nil {
+			return []string{}, err
+		}
+		defer r.Close()
+		tr = tar.NewReader(r)
 	case GZip:
 		// Create the uncompressed archive
 		archive, err := os.Create(a.GetPath() + ".uncompressed")

pkg/compiler/artifact_test.go

@@ -20,6 +20,7 @@ import (
 	"os"
 	"path/filepath"
 
+	"github.com/mudler/luet/pkg/compiler"
 	. "github.com/mudler/luet/pkg/compiler/backend"
 	"github.com/mudler/luet/pkg/solver"
 
@@ -96,24 +97,28 @@ ENV PACKAGE_CATEGORY=app-admin`))
 			Expect(err).ToNot(HaveOccurred())
 			Expect(dockerfile).To(Equal(`
 FROM luet/base
+COPY . /luetbuild
 WORKDIR /luetbuild
 ENV PACKAGE_NAME=enman
 ENV PACKAGE_VERSION=1.4.0
 ENV PACKAGE_CATEGORY=app-admin
 RUN echo foo > /test
 RUN echo bar > /test2`))
-			opts = CompilerBackendOptions{
+			opts2 := CompilerBackendOptions{
 				ImageName:      "test",
 				SourcePath:     tmpdir,
 				DockerFileName: "LuetDockerfile",
 				Destination:    filepath.Join(tmpdir, "output2.tar"),
 			}
-			Expect(b.ImageDefinitionToTar(opts)).ToNot(HaveOccurred())
+			Expect(b.ImageDefinitionToTar(opts2)).ToNot(HaveOccurred())
 			Expect(helpers.Exists(filepath.Join(tmpdir, "output2.tar"))).To(BeTrue())
-			diffs, err := b.Changes(filepath.Join(tmpdir2, "output1.tar"), filepath.Join(tmpdir, "output2.tar"))
+			diffs, err := b.Changes(opts, opts2)
 			Expect(err).ToNot(HaveOccurred())
 
-			artifacts := []ArtifactNode{}
+			artifacts := []ArtifactNode{{
+				Name: "/luetbuild/LuetDockerfile",
+				Size: 175,
+			}}
 			if os.Getenv("DOCKER_BUILDKIT") == "1" {
 				artifacts = append(artifacts, ArtifactNode{Name: "/etc/resolv.conf", Size: 0})
 			}
@@ -155,5 +160,59 @@ RUN echo bar > /test2`))
 			Expect(err).To(HaveOccurred())
 		})
 
+		It("Generates packages images", func() {
+			b := NewSimpleDockerBackend()
+			imageprefix := "foo/"
+			testString := []byte(`funky test data`)
+
+			tmpdir, err := ioutil.TempDir(os.TempDir(), "artifact")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpdir) // clean up
+
+			tmpWork, err := ioutil.TempDir(os.TempDir(), "artifact2")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpWork) // clean up
+
+			err = ioutil.WriteFile(filepath.Join(tmpdir, "test"), testString, 0644)
+			Expect(err).ToNot(HaveOccurred())
+
+			artifact := NewPackageArtifact(filepath.Join(tmpWork, "fake.tar"))
+			artifact.SetCompileSpec(&LuetCompilationSpec{Package: &pkg.DefaultPackage{Name: "foo", Version: "1.0"}})
+
+			err = artifact.Compress(tmpdir, 1)
+			Expect(err).ToNot(HaveOccurred())
+			resultingImage := imageprefix + "foo--1.0"
+			opts, err := artifact.GenerateFinalImage(resultingImage, b, false)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(opts.ImageName).To(Equal(resultingImage))
+
+			Expect(b.ImageExists(resultingImage)).To(BeTrue())
+
+			result, err := ioutil.TempDir(os.TempDir(), "result")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(result) // clean up
+
+			err = b.ExtractRootfs(CompilerBackendOptions{ImageName: resultingImage, Destination: result}, false)
+			Expect(err).ToNot(HaveOccurred())
+
+			content, err := ioutil.ReadFile(filepath.Join(result, "test"))
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(content).To(Equal(testString))
+		})
+
+		It("Retrieves uncompressed name", func() {
+			a := NewPackageArtifact("foo.tar.gz")
+			a.SetCompressionType(compiler.GZip)
+			Expect(a.GetUncompressedName()).To(Equal("foo.tar"))
+
+			a = NewPackageArtifact("foo.tar.zst")
+			a.SetCompressionType(compiler.Zstandard)
+			Expect(a.GetUncompressedName()).To(Equal("foo.tar"))
+
+			a = NewPackageArtifact("foo.tar")
+			a.SetCompressionType(compiler.None)
+			Expect(a.GetUncompressedName()).To(Equal("foo.tar"))
+		})
 	})
 })

pkg/compiler/backend/common.go

@@ -0,0 +1,43 @@
+// Copyright © 2020 Ettore Di Giacinto <mudler@gentoo.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package backend
+
+import (
+	"github.com/google/go-containerregistry/pkg/crane"
+	"github.com/mudler/luet/pkg/compiler"
+)
+
+const (
+	ImgBackend    = "img"
+	DockerBackend = "docker"
+)
+
+func imageAvailable(image string) bool {
+	_, err := crane.Digest(image)
+	return err == nil
+}
+
+func NewBackend(s string) compiler.CompilerBackend {
+	var compilerBackend compiler.CompilerBackend
+
+	switch s {
+	case ImgBackend:
+		compilerBackend = NewSimpleImgBackend()
+	case DockerBackend:
+		compilerBackend = NewSimpleDockerBackend()
+	}
+	return compilerBackend
+}

pkg/compiler/backend/diff.go

@@ -52,7 +52,10 @@ import (
 //     }
 //   }
 // ]
-func GenerateChanges(b compiler.CompilerBackend, srcImage, dstImage string) ([]compiler.ArtifactLayer, error) {
+func GenerateChanges(b compiler.CompilerBackend, fromImage, toImage compiler.CompilerBackendOptions) ([]compiler.ArtifactLayer, error) {
+
+	srcImage := fromImage.Destination
+	dstImage := toImage.Destination
 
 	res := compiler.ArtifactLayer{FromImage: srcImage, ToImage: dstImage}
 
@@ -95,6 +98,7 @@ func GenerateChanges(b compiler.CompilerBackend, srcImage, dstImage string) ([]c
 
 	srcImageExtract := compiler.CompilerBackendOptions{
 		SourcePath:  srcImage,
+		ImageName:   fromImage.ImageName,
 		Destination: srcRootFS,
 	}
 	err = b.ExtractRootfs(srcImageExtract, false) // No need to keep permissions as we just collect file diffs
@@ -123,6 +127,7 @@ func GenerateChanges(b compiler.CompilerBackend, srcImage, dstImage string) ([]c
 
 	dstImageExtract := compiler.CompilerBackendOptions{
 		SourcePath:  dstImage,
+		ImageName:   toImage.ImageName,
 		Destination: dstRootFS,
 	}
 	err = b.ExtractRootfs(dstImageExtract, false)

pkg/compiler/backend/diff_test.go

@@ -32,12 +32,14 @@ var _ = Describe("Docker image diffs", func() {
 
 	Context("Generate diffs from docker images", func() {
 		It("Detect no changes", func() {
-			err := b.DownloadImage(compiler.CompilerBackendOptions{
-				ImageName: "alpine:latest",
-			})
+			opts := compiler.CompilerBackendOptions{
+				ImageName:   "alpine:latest",
+				Destination: "alpine:latest",
+			}
+			err := b.DownloadImage(opts)
 			Expect(err).ToNot(HaveOccurred())
 
-			layers, err := GenerateChanges(b, "alpine:latest", "alpine:latest")
+			layers, err := GenerateChanges(b, opts, opts)
 			Expect(err).ToNot(HaveOccurred())
 			Expect(len(layers)).To(Equal(1))
 			Expect(len(layers[0].Diffs.Additions)).To(Equal(0))
@@ -55,7 +57,13 @@ var _ = Describe("Docker image diffs", func() {
 			})
 			Expect(err).ToNot(HaveOccurred())
 
-			layers, err := GenerateChanges(b, "quay.io/mocaccino/micro", "quay.io/mocaccino/extra")
+			layers, err := GenerateChanges(b, compiler.CompilerBackendOptions{
+				ImageName:   "quay.io/mocaccino/micro",
+				Destination: "quay.io/mocaccino/micro",
+			}, compiler.CompilerBackendOptions{
+				ImageName:   "quay.io/mocaccino/extra",
+				Destination: "quay.io/mocaccino/extra",
+			})
 			Expect(err).ToNot(HaveOccurred())
 			Expect(len(layers)).To(Equal(1))
 

pkg/compiler/backend/simpledocker.go

@@ -46,8 +46,12 @@ func (*SimpleDocker) BuildImage(opts compiler.CompilerBackendOptions) error {
 	name := opts.ImageName
 	path := opts.SourcePath
 	dockerfileName := opts.DockerFileName
+	context := opts.Context
 
-	buildarg := []string{"build", "-f", dockerfileName, "-t", name, "."}
+	if context == "" {
+		context = "."
+	}
+	buildarg := []string{"build", "-f", dockerfileName, "-t", name, context}
 
 	Debug(":whale2: Building image " + name)
 	cmd := exec.Command("docker", buildarg...)
@@ -119,6 +123,10 @@ func (*SimpleDocker) ImageExists(imagename string) bool {
 	return true
 }
 
+func (*SimpleDocker) ImageAvailable(imagename string) bool {
+	return imageAvailable(imagename)
+}
+
 func (*SimpleDocker) RemoveImage(opts compiler.CompilerBackendOptions) error {
 	name := opts.ImageName
 	buildarg := []string{"rmi", name}
@@ -175,10 +183,23 @@ type ManifestEntry struct {
 	Layers []string `json:"Layers"`
 }
 
-func (*SimpleDocker) ExtractRootfs(opts compiler.CompilerBackendOptions, keepPerms bool) error {
+func (b *SimpleDocker) ExtractRootfs(opts compiler.CompilerBackendOptions, keepPerms bool) error {
 	src := opts.SourcePath
 	dst := opts.Destination
 
+	if src == "" && opts.ImageName != "" {
+		tempUnpack, err := ioutil.TempDir(dst, "tempUnpack")
+		if err != nil {
+			return errors.Wrap(err, "Error met while creating tempdir for rootfs")
+		}
+		defer os.RemoveAll(tempUnpack) // clean up
+		imageExport := filepath.Join(tempUnpack, "image.tar")
+		if err := b.ExportImage(compiler.CompilerBackendOptions{ImageName: opts.ImageName, Destination: imageExport}); err != nil {
+			return errors.Wrap(err, "while exporting image before extraction")
+		}
+		src = imageExport
+	}
+
 	rootfs, err := ioutil.TempDir(dst, "tmprootfs")
 	if err != nil {
 		return errors.Wrap(err, "Error met while creating tempdir for rootfs")
@@ -219,7 +240,7 @@ func (*SimpleDocker) ExtractRootfs(opts compiler.CompilerBackendOptions, keepPer
 			}
 		}
 	}
-
+	// TODO: Drop capi in favor of the img approach already used in pkg/installer/repository
 	export, err := capi.CreateExport(rootfs)
 	if err != nil {
 		return err
@@ -239,7 +260,7 @@ func (*SimpleDocker) ExtractRootfs(opts compiler.CompilerBackendOptions, keepPer
 }
 
 // Changes retrieves changes between image layers
-func (d *SimpleDocker) Changes(fromImage, toImage string) ([]compiler.ArtifactLayer, error) {
+func (d *SimpleDocker) Changes(fromImage, toImage compiler.CompilerBackendOptions) ([]compiler.ArtifactLayer, error) {
 	diffs, err := GenerateChanges(d, fromImage, toImage)
 
 	if config.LuetCfg.GetGeneral().Debug {

pkg/compiler/backend/simpledocker_test.go

@@ -87,29 +87,33 @@ ENV PACKAGE_CATEGORY=app-admin`))
 			Expect(err).ToNot(HaveOccurred())
 			Expect(dockerfile).To(Equal(`
 FROM luet/base
+COPY . /luetbuild
 WORKDIR /luetbuild
 ENV PACKAGE_NAME=enman
 ENV PACKAGE_VERSION=1.4.0
 ENV PACKAGE_CATEGORY=app-admin
 RUN echo foo > /test
 RUN echo bar > /test2`))
-			opts = CompilerBackendOptions{
+			opts2 := CompilerBackendOptions{
 				ImageName:      "test",
 				SourcePath:     tmpdir,
 				DockerFileName: "LuetDockerfile",
 				Destination:    filepath.Join(tmpdir, "output2.tar"),
 			}
-			Expect(b.ImageDefinitionToTar(opts)).ToNot(HaveOccurred())
+			Expect(b.ImageDefinitionToTar(opts2)).ToNot(HaveOccurred())
 			Expect(helpers.Exists(filepath.Join(tmpdir, "output2.tar"))).To(BeTrue())
 
-			artifacts := []ArtifactNode{}
+			artifacts := []ArtifactNode{{
+				Name: "/luetbuild/LuetDockerfile",
+				Size: 175,
+			}}
 			if os.Getenv("DOCKER_BUILDKIT") == "1" {
 				artifacts = append(artifacts, ArtifactNode{Name: "/etc/resolv.conf", Size: 0})
 			}
 			artifacts = append(artifacts, ArtifactNode{Name: "/test", Size: 4})
 			artifacts = append(artifacts, ArtifactNode{Name: "/test2", Size: 4})
 
-			Expect(b.Changes(filepath.Join(tmpdir2, "output1.tar"), filepath.Join(tmpdir, "output2.tar"))).To(Equal(
+			Expect(b.Changes(opts, opts2)).To(Equal(
 				[]ArtifactLayer{{
 					FromImage: filepath.Join(tmpdir2, "output1.tar"),
 					ToImage:   filepath.Join(tmpdir, "output2.tar"),
@@ -120,5 +124,11 @@ RUN echo bar > /test2`))
 
 		})
 
+		It("Detects available images", func() {
+			b := NewSimpleDockerBackend()
+			Expect(b.ImageAvailable("quay.io/mocaccino/extra")).To(BeTrue())
+			Expect(b.ImageAvailable("ubuntu:20.10")).To(BeTrue())
+			Expect(b.ImageAvailable("igjo5ijgo25nho52")).To(BeFalse())
+		})
 	})
 })

pkg/compiler/backend/simpleimg.go

@@ -35,9 +35,14 @@ func NewSimpleImgBackend() compiler.CompilerBackend {
 func (*SimpleImg) BuildImage(opts compiler.CompilerBackendOptions) error {
 	name := opts.ImageName
 	path := opts.SourcePath
+	context := opts.Context
+
+	if context == "" {
+		context = "."
+	}
 	dockerfileName := opts.DockerFileName
 
-	buildarg := []string{"build", "-f", dockerfileName, "-t", name, "."}
+	buildarg := []string{"build", "-f", dockerfileName, "-t", name, context}
 	Spinner(22)
 	defer SpinnerStop()
 	Debug(":tea: Building image " + name)
@@ -59,7 +64,7 @@ func (*SimpleImg) RemoveImage(opts compiler.CompilerBackendOptions) error {
 	defer SpinnerStop()
 	out, err := exec.Command("img", buildarg...).CombinedOutput()
 	if err != nil {
-		return errors.Wrap(err, "Failed building image: "+string(out))
+		return errors.Wrap(err, "Failed removing image: "+string(out))
 	}
 
 	Info(":tea: Image " + name + " removed")
@@ -75,7 +80,7 @@ func (*SimpleImg) DownloadImage(opts compiler.CompilerBackendOptions) error {
 	cmd := exec.Command("img", buildarg...)
 	out, err := cmd.CombinedOutput()
 	if err != nil {
-		return errors.Wrap(err, "Failed building image: "+string(out))
+		return errors.Wrap(err, "Failed downloading image: "+string(out))
 	}
 
 	Info(":tea: Image " + name + " downloaded")
@@ -97,6 +102,10 @@ func (*SimpleImg) CopyImage(src, dst string) error {
 	return nil
 }
 
+func (*SimpleImg) ImageAvailable(imagename string) bool {
+	return imageAvailable(imagename)
+}
+
 func (*SimpleImg) ImageExists(imagename string) bool {
 	// NOOP: not implemented
 	// TODO: Since img doesn't have an inspect command,
@@ -124,13 +133,13 @@ func (*SimpleImg) ExportImage(opts compiler.CompilerBackendOptions) error {
 	Debug(":tea: Saving image " + name)
 	out, err := exec.Command("img", buildarg...).CombinedOutput()
 	if err != nil {
-		return errors.Wrap(err, "Failed building image: "+string(out))
+		return errors.Wrap(err, "Failed exporting image: "+string(out))
 	}
 	Info(":tea: Image " + name + " saved")
 	return nil
 }
 
-// TODO: Dup in docker, refactor common code in helpers for shared parts
+// ExtractRootfs extracts the docker image content inside the destination
 func (*SimpleImg) ExtractRootfs(opts compiler.CompilerBackendOptions, keepPerms bool) error {
 	name := opts.ImageName
 	path := opts.Destination
@@ -144,12 +153,11 @@ func (*SimpleImg) ExtractRootfs(opts compiler.CompilerBackendOptions, keepPerms
 	}
 	Info(":tea: Image " + name + " extracted")
 	return nil
-	//return NewSimpleDockerBackend().ExtractRootfs(opts, keepPerms)
 }
 
 // TODO: Use container-diff (https://github.com/GoogleContainerTools/container-diff) for checking out layer diffs
 // Changes uses container-diff (https://github.com/GoogleContainerTools/container-diff) for retrieving out layer diffs
-func (i *SimpleImg) Changes(fromImage, toImage string) ([]compiler.ArtifactLayer, error) {
+func (i *SimpleImg) Changes(fromImage, toImage compiler.CompilerBackendOptions) ([]compiler.ArtifactLayer, error) {
 	return GenerateChanges(i, fromImage, toImage)
 }
 

pkg/compiler/compiler.go

@@ -16,7 +16,6 @@
 package compiler
 
 import (
-	"archive/tar"
 	"fmt"
 	"io/ioutil"
 	"os"
@@ -67,7 +66,6 @@ func NewLuetCompiler(backend CompilerBackend, db pkg.PackageDatabase, opt *Compi
 		CompressionType: opt.CompressionType,
 		KeepImg:         opt.KeepImg,
 		Concurrency:     opt.Concurrency,
-		Clean:           opt.Clean,
 		Options:         *opt,
 		SolverOptions:   solvopts,
 	}
@@ -265,14 +263,22 @@ func (cs *LuetCompiler) unpackFs(rootfs string, concurrency int, p CompilationSp
 
 func (cs *LuetCompiler) unpackDelta(rootfs string, concurrency int, keepPermissions bool, p CompilationSpec, builderOpts, runnerOpts CompilerBackendOptions) (Artifact, error) {
 	pkgTag := ":package: " + p.GetPackage().HumanReadableString()
+	if cs.Options.PullFirst && !cs.Backend.ImageExists(builderOpts.ImageName) && cs.Backend.ImageAvailable(builderOpts.ImageName) {
+		err := cs.Backend.DownloadImage(builderOpts)
+		if err != nil {
+			return nil, errors.Wrap(err, "Could not pull image")
+		}
+	} else if !cs.Backend.ImageExists(builderOpts.ImageName) {
+		return nil, errors.New("No image found for " + builderOpts.ImageName)
+	}
 	if err := cs.Backend.ExportImage(builderOpts); err != nil {
-		return nil, errors.Wrap(err, "Could not export image")
+		return nil, errors.Wrap(err, "Could not export image"+builderOpts.ImageName)
 	}
 	if !cs.Options.KeepImageExport {
 		defer os.Remove(builderOpts.Destination)
 	}
 	Info(pkgTag, ":hammer: Generating delta")
-	diffs, err := cs.Backend.Changes(builderOpts.Destination, runnerOpts.Destination)
+	diffs, err := cs.Backend.Changes(builderOpts, runnerOpts)
 	if err != nil {
 		return nil, errors.Wrap(err, "Could not generate changes from layers")
 	}
@@ -420,29 +426,33 @@ func (cs *LuetCompiler) genArtifact(p CompilationSpec, builderOpts, runnerOpts C
 	var artifact Artifact
 	var rootfs string
 	var err error
-	unpack := p.ImageUnpack()
 	pkgTag := ":package: " + p.GetPackage().HumanReadableString()
 
-	// If package_dir was specified in the spec, we want to treat the content of the directory
-	// as the root of our archive.  ImageUnpack is implied to be true. override it
-	if p.GetPackageDir() != "" {
-		unpack = true
-	}
-
-	if len(p.BuildSteps()) == 0 && len(p.GetPreBuildSteps()) == 0 && !unpack {
+	// We can't generate delta in this case. It implies the package is a virtual, and nothing has to be done really
+	if p.EmptyPackage() {
 		fakePackage := p.Rel(p.GetPackage().GetFingerPrint() + ".package.tar")
-		// We can't generate delta in this case. It implies the package is a virtual, and nothing has to be done really
 
-		file, err := os.Create(fakePackage)
+		rootfs, err = ioutil.TempDir(p.GetOutputPath(), "rootfs")
 		if err != nil {
-			return nil, errors.Wrap(err, "Failed creating virtual package")
+			return nil, errors.Wrap(err, "Could not create tempdir")
 		}
-		defer file.Close()
-		tw := tar.NewWriter(file)
-		defer tw.Close()
+		defer os.RemoveAll(rootfs) // clean up
 
 		artifact := NewPackageArtifact(fakePackage)
 		artifact.SetCompressionType(cs.CompressionType)
+
+		if err := artifact.Compress(rootfs, concurrency); err != nil {
+			return nil, errors.Wrap(err, "Error met while creating package archive")
+		}
+
+		artifact.SetCompileSpec(p)
+		artifact.GetCompileSpec().GetPackage().SetBuildTimestamp(time.Now().String())
+
+		err = artifact.WriteYaml(p.GetOutputPath())
+		if err != nil {
+			return artifact, errors.Wrap(err, "Failed while writing metadata file")
+		}
+		Info(pkgTag, "   :white_check_mark: done (empty virtual package)")
 		return artifact, nil
 	}
 
@@ -469,17 +479,17 @@ func (cs *LuetCompiler) genArtifact(p CompilationSpec, builderOpts, runnerOpts C
 		return nil, errors.Wrap(err, "Could not extract rootfs")
 	}
 
-	if unpack {
+	if p.UnpackedPackage() {
 		// Take content of container as a base for our package files
 		artifact, err = cs.unpackFs(rootfs, concurrency, p)
 		if err != nil {
-			return nil, errors.Wrap(err, "Error met while creating package archive")
+			return nil, errors.Wrap(err, "Error met while extracting image")
 		}
 	} else {
 		// Generate delta between the two images
 		artifact, err = cs.unpackDelta(rootfs, concurrency, keepPermissions, p, builderOpts, runnerOpts)
 		if err != nil {
-			return nil, errors.Wrap(err, "Error met while creating package archive")
+			return nil, errors.Wrap(err, "Error met while generating delta")
 		}
 	}
 
@@ -500,17 +510,40 @@ func (cs *LuetCompiler) genArtifact(p CompilationSpec, builderOpts, runnerOpts C
 	return artifact, nil
 }
 
+func (cs *LuetCompiler) waitForImage(image string) {
+	if cs.Options.PullFirst && cs.Options.Wait && !cs.Backend.ImageAvailable(image) {
+		Info(fmt.Sprintf("Waiting for image %s to be available... :zzz:", image))
+		Spinner(22)
+		defer SpinnerStop()
+		for !cs.Backend.ImageAvailable(image) {
+			Info(fmt.Sprintf("Image %s not available yet, sleeping", image))
+			time.Sleep(5 * time.Second)
+		}
+	}
+}
+
 func (cs *LuetCompiler) compileWithImage(image, buildertaggedImage, packageImage string,
 	concurrency int,
 	keepPermissions, keepImg bool,
 	p CompilationSpec, generateArtifact bool) (Artifact, error) {
 
-	if !cs.Clean {
-		exists := cs.Backend.ImageExists(buildertaggedImage) && cs.Backend.ImageExists(packageImage)
-		if art, err := LoadArtifactFromYaml(p); err == nil && (cs.Options.SkipIfMetadataExists || exists) {
-			Debug("Artifact reloaded. Skipping build")
+	// If it is a virtual, check if we have to generate an empty artifact or not.
+	if generateArtifact && p.EmptyPackage() && !p.HasImageSource() {
+		return cs.genArtifact(p, CompilerBackendOptions{}, CompilerBackendOptions{}, concurrency, keepPermissions)
+	} else if p.EmptyPackage() && !p.HasImageSource() {
+		return &PackageArtifact{}, nil
+	}
+
+	if !generateArtifact {
+		exists := cs.Backend.ImageExists(packageImage)
+		if art, err := LoadArtifactFromYaml(p); err == nil && exists { // If YAML is correctly loaded, and both images exists, no reason to rebuild.
+			Debug("Artifact reloaded from YAML. Skipping build")
 			return art, err
 		}
+		cs.waitForImage(packageImage)
+		if cs.Options.PullFirst && cs.Backend.ImageAvailable(packageImage) {
+			return &PackageArtifact{}, nil
+		}
 	}
 
 	builderOpts, runnerOpts, err := cs.buildPackageImage(image, buildertaggedImage, packageImage, concurrency, keepPermissions, p)
@@ -626,10 +659,13 @@ func (cs *LuetCompiler) Compile(keepPermissions bool, p CompilationSpec) (Artifa
 func (cs *LuetCompiler) compile(concurrency int, keepPermissions bool, p CompilationSpec) (Artifact, error) {
 	Info(":package: Compiling", p.GetPackage().HumanReadableString(), ".... :coffee:")
 
-	if len(p.GetPackage().GetRequires()) == 0 && p.GetImage() == "" {
-		Error("Package with no deps and no seed image supplied, bailing out")
-		return nil, errors.New("Package " + p.GetPackage().GetFingerPrint() +
-			" with no deps and no seed image supplied, bailing out")
+	Debug(fmt.Sprintf("%s: has images %t, empty package: %t", p.GetPackage().HumanReadableString(), p.HasImageSource(), p.EmptyPackage()))
+	if !p.HasImageSource() && !p.EmptyPackage() {
+		return nil,
+			fmt.Errorf(
+				"%s is invalid: package has no dependencies and no seed image supplied while it has steps defined",
+				p.GetPackage().GetFingerPrint(),
+			)
 	}
 
 	targetAssertion := p.GetSourceAssertion().Search(p.GetPackage().GetFingerPrint())

pkg/compiler/compiler_test.go

@@ -50,8 +50,8 @@ var _ = Describe("Compiler", func() {
 			Expect(err).ToNot(HaveOccurred())
 			defer os.RemoveAll(tmpdir) // clean up
 
-			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "./generate.sh"}))
-			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2", "chmod +x generate.sh"}))
+			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "chmod +x generate.sh", "./generate.sh"}))
+			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2"}))
 
 			spec.SetOutputPath(tmpdir)
 			compiler.SetConcurrency(2)

pkg/compiler/interface.go

@@ -42,6 +42,7 @@ type CompilerBackendOptions struct {
 	SourcePath     string
 	DockerFileName string
 	Destination    string
+	Context        string
 }
 
 type CompilerOptions struct {
@@ -49,14 +50,13 @@ type CompilerOptions struct {
 	PullFirst, KeepImg, Push bool
 	Concurrency              int
 	CompressionType          CompressionImplementation
-	Clean                    bool
 	KeepImageExport          bool
 
-	OnlyDeps             bool
-	NoDeps               bool
-	SolverOptions        config.LuetSolverOptions
-	SkipIfMetadataExists bool
-	BuildValuesFile      string
+	Wait            bool
+	OnlyDeps        bool
+	NoDeps          bool
+	SolverOptions   config.LuetSolverOptions
+	BuildValuesFile string
 
 	PackageTargetOnly bool
 }
@@ -69,7 +69,6 @@ func NewDefaultCompilerOptions() *CompilerOptions {
 		CompressionType: None,
 		KeepImg:         true,
 		Concurrency:     runtime.NumCPU(),
-		Clean:           true,
 		OnlyDeps:        false,
 		NoDeps:          false,
 	}
@@ -79,7 +78,7 @@ type CompilerBackend interface {
 	BuildImage(CompilerBackendOptions) error
 	ExportImage(CompilerBackendOptions) error
 	RemoveImage(CompilerBackendOptions) error
-	Changes(fromImage, toImage string) ([]ArtifactLayer, error)
+	Changes(fromImage, toImage CompilerBackendOptions) ([]ArtifactLayer, error)
 	ImageDefinitionToTar(CompilerBackendOptions) error
 	ExtractRootfs(opts CompilerBackendOptions, keepPerms bool) error
 
@@ -87,6 +86,7 @@ type CompilerBackend interface {
 	DownloadImage(opts CompilerBackendOptions) error
 
 	Push(opts CompilerBackendOptions) error
+	ImageAvailable(string) bool
 
 	ImageExists(string) bool
 }
@@ -111,9 +111,13 @@ type Artifact interface {
 
 	SetFiles(f []string)
 	GetFiles() []string
+	GetFileName() string
 
 	GetChecksums() Checksums
 	SetChecksums(c Checksums)
+
+	GenerateFinalImage(string, CompilerBackend, bool) (CompilerBackendOptions, error)
+	GetUncompressedName() string
 }
 
 type ArtifactNode struct {
@@ -179,6 +183,10 @@ type CompilationSpec interface {
 
 	SetPackageDir(string)
 	GetPackageDir() string
+
+	EmptyPackage() bool
+	UnpackedPackage() bool
+	HasImageSource() bool
 }
 
 type CompilationSpecs interface {

pkg/compiler/spec.go

@@ -185,6 +185,24 @@ func (cs *LuetCompilationSpec) SetSeedImage(s string) {
 	cs.Seed = s
 }
 
+func (cs *LuetCompilationSpec) EmptyPackage() bool {
+	return len(cs.BuildSteps()) == 0 && len(cs.GetPreBuildSteps()) == 0 && !cs.UnpackedPackage()
+}
+
+func (cs *LuetCompilationSpec) UnpackedPackage() bool {
+	// If package_dir was specified in the spec, we want to treat the content of the directory
+	// as the root of our archive.  ImageUnpack is implied to be true. override it
+	unpack := cs.ImageUnpack()
+	if cs.GetPackageDir() != "" {
+		unpack = true
+	}
+	return unpack
+}
+
+func (cs *LuetCompilationSpec) HasImageSource() bool {
+	return len(cs.GetPackage().GetRequires()) != 0 || cs.GetImage() != ""
+}
+
 func (cs *LuetCompilationSpec) CopyRetrieves(dest string) error {
 	var err error
 	if len(cs.Retrieve) > 0 {
@@ -203,10 +221,9 @@ func (cs *LuetCompilationSpec) CopyRetrieves(dest string) error {
 	return err
 }
 
-// TODO: docker build image first. Then a backend can be used to actually spin up a container with it and run the steps within
-func (cs *LuetCompilationSpec) RenderBuildImage() (string, error) {
+func (cs *LuetCompilationSpec) genDockerfile(image string, steps []string) string {
 	spec := `
-FROM ` + cs.GetSeedImage() + `
+FROM ` + image + `
 COPY . /luetbuild
 WORKDIR /luetbuild
 ENV PACKAGE_NAME=` + cs.Package.GetName() + `
@@ -231,45 +248,22 @@ ADD ` + s + ` /luetbuild/`
 ENV ` + s
 	}
 
-	for _, s := range cs.GetPreBuildSteps() {
+	for _, s := range steps {
 		spec = spec + `
 RUN ` + s
 	}
-	return spec, nil
+	return spec
 }
 
-// TODO: docker build image first. Then a backend can be used to actually spin up a container with it and run the steps within
+// RenderBuildImage renders the dockerfile of the image used as a pre-build step
+func (cs *LuetCompilationSpec) RenderBuildImage() (string, error) {
+	return cs.genDockerfile(cs.GetSeedImage(), cs.GetPreBuildSteps()), nil
+
+}
+
+// RenderStepImage renders the dockerfile used for the image used for building the package
 func (cs *LuetCompilationSpec) RenderStepImage(image string) (string, error) {
-	spec := `
-FROM ` + image + `
-WORKDIR /luetbuild
-ENV PACKAGE_NAME=` + cs.Package.GetName() + `
-ENV PACKAGE_VERSION=` + cs.Package.GetVersion() + `
-ENV PACKAGE_CATEGORY=` + cs.Package.GetCategory()
-
-	if len(cs.Retrieve) > 0 {
-		for _, s := range cs.Retrieve {
-			//var file string
-			// if helpers.IsValidUrl(s) {
-			// 	file = s
-			// } else {
-			// 	file = cs.Rel(s)
-			// }
-			spec = spec + `
-ADD ` + s + ` /luetbuild/`
-		}
-	}
-
-	for _, s := range cs.Env {
-		spec = spec + `
-ENV ` + s
-	}
-	for _, s := range cs.BuildSteps() {
-		spec = spec + `
-RUN ` + s
-	}
-
-	return spec, nil
+	return cs.genDockerfile(image, cs.BuildSteps()), nil
 }
 
 func (cs *LuetCompilationSpec) WriteBuildImageDefinition(path string) error {

pkg/compiler/spec_test.go

@@ -96,6 +96,7 @@ ENV test=1`))
 			Expect(err).ToNot(HaveOccurred())
 			Expect(dockerfile).To(Equal(`
 FROM luet/base
+COPY . /luetbuild
 WORKDIR /luetbuild
 ENV PACKAGE_NAME=enman
 ENV PACKAGE_VERSION=1.4.0
@@ -169,6 +170,7 @@ ENV test=1`))
 
 		Expect(dockerfile).To(Equal(`
 FROM luet/base
+COPY . /luetbuild
 WORKDIR /luetbuild
 ENV PACKAGE_NAME=a
 ENV PACKAGE_VERSION=1.0

pkg/config/config.go

@@ -28,6 +28,7 @@ import (
 	"time"
 
 	"github.com/mudler/luet/pkg/helpers"
+	pkg "github.com/mudler/luet/pkg/package"
 	solver "github.com/mudler/luet/pkg/solver"
 
 	v "github.com/spf13/viper"
@@ -276,6 +277,16 @@ func GenDefault(viper *v.Viper) {
 	viper.SetDefault("solver.max_attempts", 9000)
 }
 
+func (c *LuetConfig) GetSystemDB() pkg.PackageDatabase {
+	switch LuetCfg.GetSystem().DatabaseEngine {
+	case "boltdb":
+		return pkg.NewBoltDatabase(
+			filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
+	default:
+		return pkg.NewInMemoryDatabase(true)
+	}
+}
+
 func (c *LuetConfig) AddSystemRepository(r LuetRepository) {
 	c.SystemRepositories = append(c.SystemRepositories, r)
 }

pkg/helpers/file.go

@@ -117,14 +117,15 @@ func Read(file string) (string, error) {
 	return string(dat), nil
 }
 
-func ensureDir(fileName string) {
+func EnsureDir(fileName string) error {
 	dirName := filepath.Dir(fileName)
 	if _, serr := os.Stat(dirName); serr != nil {
 		merr := os.MkdirAll(dirName, os.ModePerm) // FIXME: It should preserve permissions from src to dst instead
 		if merr != nil {
-			panic(merr)
+			return merr
 		}
 	}
+	return nil
 }
 
 // CopyFile copies the contents of the file named src to the file named

pkg/installer/client/docker.go

@@ -0,0 +1,186 @@
+// Copyright © 2020 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package client
+
+import (
+	"fmt"
+	"os"
+	"path"
+	"path/filepath"
+
+	"github.com/docker/go-units"
+	"github.com/pkg/errors"
+
+	imgworker "github.com/mudler/luet/pkg/installer/client/imgworker"
+
+	"github.com/mudler/luet/pkg/compiler"
+	"github.com/mudler/luet/pkg/config"
+	"github.com/mudler/luet/pkg/helpers"
+	. "github.com/mudler/luet/pkg/logger"
+)
+
+type DockerClient struct {
+	RepoData RepoData
+}
+
+func NewDockerClient(r RepoData) *DockerClient {
+	return &DockerClient{RepoData: r}
+}
+
+func downloadAndExtractDockerImage(image, dest string) error {
+	temp, err := config.LuetCfg.GetSystem().TempDir("contentstore")
+	if err != nil {
+		return err
+	}
+	defer os.RemoveAll(temp)
+	Debug("Temporary directory", temp)
+	c, err := imgworker.New(temp)
+	if err != nil {
+		return errors.Wrapf(err, "failed creating client")
+	}
+	defer c.Close()
+
+	// FROM Slightly adapted from genuinetools/img https://github.com/genuinetools/img/blob/54d0ca981c1260546d43961a538550eef55c87cf/pull.go
+	Debug("Pulling image", image)
+	listedImage, err := c.Pull(image)
+	if err != nil {
+		return errors.Wrapf(err, "failed listing images")
+
+	}
+	Debug("Pulled:", listedImage.Target.Digest)
+	Debug("Size:", units.BytesSize(float64(listedImage.ContentSize)))
+	Debug("Unpacking", image, "to", dest)
+	os.RemoveAll(dest)
+	return c.Unpack(image, dest)
+}
+
+func (c *DockerClient) DownloadArtifact(artifact compiler.Artifact) (compiler.Artifact, error) {
+	//var u *url.URL = nil
+	var err error
+	var temp string
+	var resultingArtifact compiler.Artifact
+	artifactName := path.Base(artifact.GetPath())
+	cacheFile := filepath.Join(config.LuetCfg.GetSystem().GetSystemPkgsCacheDirPath(), artifactName)
+	Debug("Cache file", cacheFile)
+	if err := helpers.EnsureDir(cacheFile); err != nil {
+		return nil, errors.Wrapf(err, "could not create cache folder %s for %s", config.LuetCfg.GetSystem().GetSystemPkgsCacheDirPath(), cacheFile)
+	}
+	ok := false
+
+	// TODO:
+	// Files are in URI/packagename:version (GetPackageImageName() method)
+	// use downloadAndExtract .. and egenrate an archive to consume. Checksum should be already checked while downloading the image
+	// with the above functions, because Docker images already contain such metadata
+	// - Check how verification is done when calling DownloadArtifact outside, similarly we need to check DownloadFile, and how verification
+	// is done in such cases (see repository.go)
+
+	// Check if file is already in cache
+	if helpers.Exists(cacheFile) {
+		Debug("Cache hit for artifact", artifactName)
+		resultingArtifact = artifact
+		resultingArtifact.SetPath(cacheFile)
+		resultingArtifact.SetChecksums(compiler.Checksums{})
+	} else {
+
+		temp, err = config.LuetCfg.GetSystem().TempDir("tree")
+		if err != nil {
+			return nil, err
+		}
+		defer os.RemoveAll(temp)
+
+		for _, uri := range c.RepoData.Urls {
+			Debug("Downloading artifact", artifactName, "from", uri)
+
+			imageName := fmt.Sprintf("%s:%s", uri, artifact.GetCompileSpec().GetPackage().GetFingerPrint())
+
+			// imageName := fmt.Sprintf("%s/%s", uri, artifact.GetCompileSpec().GetPackage().GetPackageImageName())
+			err = downloadAndExtractDockerImage(imageName, temp)
+			if err != nil {
+				Debug("Failed download of image", imageName)
+				continue
+			}
+			Debug("\nCompressing result ", filepath.Join(temp), "to", cacheFile)
+
+			newart := artifact
+			// We discard checksum, that are checked while during pull and unpack
+			newart.SetChecksums(compiler.Checksums{})
+			newart.SetPath(cacheFile)                    // First set to cache file
+			newart.SetPath(newart.GetUncompressedName()) // Calculate the real path from cacheFile
+			err = newart.Compress(temp, 1)
+			if err != nil {
+				Error(fmt.Sprintf("Failed compressing package %s: %s", imageName, err.Error()))
+				continue
+			}
+			resultingArtifact = newart
+
+			ok = true
+			break
+		}
+
+		if !ok {
+			return nil, err
+		}
+	}
+
+	return resultingArtifact, nil
+}
+
+func (c *DockerClient) DownloadFile(name string) (string, error) {
+	var file *os.File = nil
+	var err error
+	var temp string
+
+	// Files should be in URI/repository:<file>
+	ok := false
+
+	temp, err = config.LuetCfg.GetSystem().TempDir("tree")
+	if err != nil {
+		return "", err
+	}
+
+	for _, uri := range c.RepoData.Urls {
+
+		file, err = config.LuetCfg.GetSystem().TempFile("DockerClient")
+		if err != nil {
+			continue
+		}
+
+		Debug("Downloading file", name, "from", uri)
+
+		imageName := fmt.Sprintf("%s:%s", uri, name)
+		//imageName := fmt.Sprintf("%s/%s:%s", uri, "repository", name)
+		err = downloadAndExtractDockerImage(imageName, temp)
+		if err != nil {
+			Debug("Failed download of image", imageName)
+			continue
+		}
+
+		Debug("\nCopying file ", filepath.Join(temp, name), "to", file.Name())
+		err = helpers.CopyFile(filepath.Join(temp, name), file.Name())
+
+		if err != nil {
+			continue
+		}
+		ok = true
+		break
+	}
+
+	if !ok {
+		return "", err
+	}
+
+	return file.Name(), err
+}

pkg/installer/client/docker_test.go

@@ -0,0 +1,77 @@
+// Copyright © 2019 Ettore Di Giacinto <mudler@gentoo.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package client_test
+
+import (
+	"io/ioutil"
+	"os"
+	"path/filepath"
+
+	compiler "github.com/mudler/luet/pkg/compiler"
+	helpers "github.com/mudler/luet/pkg/helpers"
+	pkg "github.com/mudler/luet/pkg/package"
+
+	. "github.com/mudler/luet/pkg/installer/client"
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+)
+
+// This test expect that the repository defined in UNIT_TEST_DOCKER_IMAGE is in zstd format.
+// the repository is built by the 01_simple_docker.sh integration test file.
+// This test also require root. At the moment, unpacking docker images with 'img' requires root permission to
+// mount/unmount layers.
+var _ = Describe("Docker client", func() {
+	Context("With repository", func() {
+		repoImage := os.Getenv("UNIT_TEST_DOCKER_IMAGE")
+		var repoURL []string
+		var c *DockerClient
+		BeforeEach(func() {
+			if repoImage == "" {
+				Skip("UNIT_TEST_DOCKER_IMAGE not specified")
+			}
+			repoURL = []string{repoImage}
+			c = NewDockerClient(RepoData{Urls: repoURL})
+		})
+
+		It("Downloads single files", func() {
+			f, err := c.DownloadFile("repository.yaml")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(helpers.Read(f)).To(ContainSubstring("Test Repo"))
+			os.RemoveAll(f)
+		})
+
+		It("Downloads artifacts", func() {
+			f, err := c.DownloadArtifact(&compiler.PackageArtifact{
+				Path: "test.tar",
+				CompileSpec: &compiler.LuetCompilationSpec{
+					Package: &pkg.DefaultPackage{
+						Name:     "c",
+						Category: "test",
+						Version:  "1.0",
+					},
+				},
+			})
+			Expect(err).ToNot(HaveOccurred())
+			tmpdir, err := ioutil.TempDir("", "test")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpdir) // clean up
+			Expect(f.Unpack(tmpdir, false)).ToNot(HaveOccurred())
+			Expect(helpers.Read(filepath.Join(tmpdir, "c"))).To(Equal("c\n"))
+			Expect(helpers.Read(filepath.Join(tmpdir, "cd"))).To(Equal("c\n"))
+			os.RemoveAll(f.GetPath())
+		})
+	})
+})

pkg/installer/client/http.go

@@ -78,7 +78,7 @@ func (c *HttpClient) DownloadArtifact(artifact compiler.Artifact) (compiler.Arti
 
 	// Check if file is already in cache
 	if helpers.Exists(cacheFile) {
-		Info("Use artifact", artifactName, "from cache.")
+		Debug("Use artifact", artifactName, "from cache.")
 	} else {
 
 		temp, err = config.LuetCfg.GetSystem().TempDir("tree")
@@ -90,7 +90,7 @@ func (c *HttpClient) DownloadArtifact(artifact compiler.Artifact) (compiler.Arti
 		client := grab.NewClient()
 
 		for _, uri := range c.RepoData.Urls {
-			Info("Downloading artifact", artifactName, "from", uri)
+			Debug("Downloading artifact", artifactName, "from", uri)
 
 			u, err = url.Parse(uri)
 			if err != nil {
@@ -202,7 +202,7 @@ func (c *HttpClient) DownloadFile(name string) (string, error) {
 		}
 		u.Path = path.Join(u.Path, name)
 
-		Info("Downloading", u.String())
+		Debug("Downloading", u.String())
 
 		req, err = c.PrepareReq(temp, u.String())
 		if err != nil {

pkg/installer/client/imgworker/client.go

@@ -0,0 +1,78 @@
+package imgworker
+
+// FROM Slightly adapted from genuinetools/img worker
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+
+	"github.com/containerd/containerd/namespaces"
+	"github.com/genuinetools/img/types"
+	"github.com/moby/buildkit/control"
+	"github.com/moby/buildkit/session"
+	"github.com/moby/buildkit/util/appcontext"
+	"github.com/moby/buildkit/worker/base"
+	"github.com/pkg/errors"
+)
+
+// Client holds the information for the client we will use for communicating
+// with the buildkit controller.
+type Client struct {
+	backend   string
+	localDirs map[string]string
+	root      string
+
+	sessionManager *session.Manager
+	controller     *control.Controller
+	opts           *base.WorkerOpt
+
+	sess *session.Session
+	ctx  context.Context
+}
+
+// New returns a new client for communicating with the buildkit controller.
+func New(root string) (*Client, error) {
+	// Native backend is fine, our images have just one layer. No need to depend on anything
+	backend := types.NativeBackend
+
+	// Create the root/
+	root = filepath.Join(root, "runc", backend)
+	if err := os.MkdirAll(root, 0700); err != nil {
+		return nil, err
+	}
+	c := &Client{
+		backend:   types.NativeBackend,
+		root:      root,
+		localDirs: nil,
+	}
+
+	if err := c.prepare(); err != nil {
+		return nil, errors.Wrapf(err, "failed preparing client")
+	}
+
+	// Create the start of the client.
+	return c, nil
+}
+
+func (c *Client) Close() {
+	c.sess.Close()
+}
+
+func (c *Client) prepare() error {
+	ctx := appcontext.Context()
+	sess, sessDialer, err := c.Session(ctx)
+	if err != nil {
+		return errors.Wrapf(err, "failed creating Session")
+	}
+	ctx = session.NewContext(ctx, sess.ID())
+	ctx = namespaces.WithNamespace(ctx, "buildkit")
+
+	c.ctx = ctx
+	c.sess = sess
+
+	go func() {
+		sess.Run(ctx, sessDialer)
+	}()
+	return nil
+}

pkg/installer/client/imgworker/pull.go

@@ -0,0 +1,129 @@
+package imgworker
+
+// FROM Slightly adapted from genuinetools/img worker
+
+import (
+	"fmt"
+
+	"github.com/containerd/containerd/images"
+	"github.com/containerd/containerd/platforms"
+	"github.com/docker/distribution/reference"
+	"github.com/moby/buildkit/cache"
+	"github.com/moby/buildkit/exporter"
+	imageexporter "github.com/moby/buildkit/exporter/containerimage"
+	"github.com/moby/buildkit/source"
+	"github.com/moby/buildkit/source/containerimage"
+)
+
+// ListedImage represents an image structure returuned from ListImages.
+// It extends containerd/images.Image with extra fields.
+type ListedImage struct {
+	images.Image
+	ContentSize int64
+}
+
+// Pull retrieves an image from a remote registry.
+func (c *Client) Pull(image string) (*ListedImage, error) {
+
+	ctx := c.ctx
+
+	sm, err := c.getSessionManager()
+	if err != nil {
+		return nil, err
+	}
+	// Parse the image name and tag.
+	named, err := reference.ParseNormalizedNamed(image)
+	if err != nil {
+		return nil, fmt.Errorf("parsing image name %q failed: %v", image, err)
+	}
+	// Add the latest lag if they did not provide one.
+	named = reference.TagNameOnly(named)
+	image = named.String()
+
+	// Get the identifier for the image.
+	identifier, err := source.NewImageIdentifier(image)
+	if err != nil {
+		return nil, err
+	}
+
+	// Create the worker opts.
+	opt, err := c.createWorkerOpt()
+	if err != nil {
+		return nil, fmt.Errorf("creating worker opt failed: %v", err)
+	}
+
+	cm, err := cache.NewManager(cache.ManagerOpt{
+		Snapshotter:    opt.Snapshotter,
+		MetadataStore:  opt.MetadataStore,
+		ContentStore:   opt.ContentStore,
+		LeaseManager:   opt.LeaseManager,
+		GarbageCollect: opt.GarbageCollect,
+		Applier:        opt.Applier,
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	// Create the source for the pull.
+	srcOpt := containerimage.SourceOpt{
+		Snapshotter:   opt.Snapshotter,
+		ContentStore:  opt.ContentStore,
+		Applier:       opt.Applier,
+		CacheAccessor: cm,
+		ImageStore:    opt.ImageStore,
+		RegistryHosts: opt.RegistryHosts,
+		LeaseManager:  opt.LeaseManager,
+	}
+	src, err := containerimage.NewSource(srcOpt)
+	if err != nil {
+		return nil, err
+	}
+	s, err := src.Resolve(ctx, identifier, sm)
+	if err != nil {
+		return nil, err
+	}
+	ref, err := s.Snapshot(ctx)
+	if err != nil {
+		return nil, err
+	}
+
+	// Create the exporter for the pull.
+	iw, err := imageexporter.NewImageWriter(imageexporter.WriterOpt{
+		Snapshotter:  opt.Snapshotter,
+		ContentStore: opt.ContentStore,
+		Differ:       opt.Differ,
+	})
+	if err != nil {
+		return nil, err
+	}
+	expOpt := imageexporter.Opt{
+		SessionManager: sm,
+		ImageWriter:    iw,
+		Images:         opt.ImageStore,
+		RegistryHosts:  opt.RegistryHosts,
+		LeaseManager:   opt.LeaseManager,
+	}
+	exp, err := imageexporter.New(expOpt)
+	if err != nil {
+		return nil, err
+	}
+	e, err := exp.Resolve(ctx, map[string]string{"name": image})
+	if err != nil {
+		return nil, err
+	}
+	if _, err := e.Export(ctx, exporter.Source{Ref: ref}); err != nil {
+		return nil, err
+	}
+
+	// Get the image.
+	img, err := opt.ImageStore.Get(ctx, image)
+	if err != nil {
+		return nil, fmt.Errorf("getting image %s from image store failed: %v", image, err)
+	}
+	size, err := img.Size(ctx, opt.ContentStore, platforms.Default())
+	if err != nil {
+		return nil, fmt.Errorf("calculating size of image %s failed: %v", img.Name, err)
+	}
+
+	return &ListedImage{Image: img, ContentSize: size}, nil
+}

pkg/installer/client/imgworker/session.go

@@ -0,0 +1,51 @@
+package imgworker
+
+// FROM Slightly adapted from genuinetools/img worker
+
+import (
+	"context"
+	"os"
+
+	"github.com/moby/buildkit/session"
+	"github.com/moby/buildkit/session/auth/authprovider"
+	"github.com/moby/buildkit/session/filesync"
+	"github.com/moby/buildkit/session/testutil"
+	"github.com/pkg/errors"
+)
+
+func (c *Client) getSessionManager() (*session.Manager, error) {
+	if c.sessionManager == nil {
+		var err error
+		c.sessionManager, err = session.NewManager()
+		if err != nil {
+			return nil, err
+		}
+	}
+	return c.sessionManager, nil
+}
+
+// Session creates the session manager and returns the session and it's
+// dialer.
+func (c *Client) Session(ctx context.Context) (*session.Session, session.Dialer, error) {
+	m, err := c.getSessionManager()
+	if err != nil {
+		return nil, nil, errors.Wrap(err, "failed to create session manager")
+	}
+	sessionName := "img"
+	s, err := session.NewSession(ctx, sessionName, "")
+	if err != nil {
+		return nil, nil, errors.Wrap(err, "failed to create session")
+	}
+	syncedDirs := make([]filesync.SyncedDir, 0, len(c.localDirs))
+	for name, d := range c.localDirs {
+		syncedDirs = append(syncedDirs, filesync.SyncedDir{Name: name, Dir: d})
+	}
+	s.Allow(filesync.NewFSSyncProvider(syncedDirs))
+	s.Allow(authprovider.NewDockerAuthProvider(os.Stderr))
+	return s, sessionDialer(s, m), err
+}
+
+func sessionDialer(s *session.Session, m *session.Manager) session.Dialer {
+	// FIXME: rename testutil
+	return session.Dialer(testutil.TestStream(testutil.Handler(m.HandleConn)))
+}

pkg/installer/client/imgworker/unpack.go

@@ -0,0 +1,82 @@
+package imgworker
+
+// FROM Slightly adapted from genuinetools/img worker
+
+import (
+	"errors"
+	"fmt"
+	"os"
+
+	"github.com/containerd/containerd/content"
+	"github.com/containerd/containerd/images"
+	"github.com/containerd/containerd/platforms"
+	"github.com/docker/distribution/reference"
+	"github.com/docker/docker/pkg/archive"
+	"github.com/sirupsen/logrus"
+)
+
+// TODO: this requires root permissions to mount/unmount layers, althrought it shouldn't be required.
+// See how backends are unpacking images without asking for root permissions.
+
+// Unpack exports an image to a rootfs destination directory.
+func (c *Client) Unpack(image, dest string) error {
+
+	ctx := c.ctx
+	if len(dest) < 1 {
+		return errors.New("destination directory for rootfs cannot be empty")
+	}
+
+	if _, err := os.Stat(dest); err == nil {
+		return fmt.Errorf("destination directory already exists: %s", dest)
+	}
+
+	// Parse the image name and tag.
+	named, err := reference.ParseNormalizedNamed(image)
+	if err != nil {
+		return fmt.Errorf("parsing image name %q failed: %v", image, err)
+	}
+	// Add the latest lag if they did not provide one.
+	named = reference.TagNameOnly(named)
+	image = named.String()
+
+	// Create the worker opts.
+	opt, err := c.createWorkerOpt()
+	if err != nil {
+		return fmt.Errorf("creating worker opt failed: %v", err)
+	}
+
+	if opt.ImageStore == nil {
+		return errors.New("image store is nil")
+	}
+
+	img, err := opt.ImageStore.Get(ctx, image)
+	if err != nil {
+		return fmt.Errorf("getting image %s from image store failed: %v", image, err)
+	}
+
+	manifest, err := images.Manifest(ctx, opt.ContentStore, img.Target, platforms.Default())
+	if err != nil {
+		return fmt.Errorf("getting image manifest failed: %v", err)
+	}
+
+	for _, desc := range manifest.Layers {
+		logrus.Debugf("Unpacking layer %s", desc.Digest.String())
+
+		// Read the blob from the content store.
+		layer, err := opt.ContentStore.ReaderAt(ctx, desc)
+		if err != nil {
+			return fmt.Errorf("getting reader for digest %s failed: %v", desc.Digest.String(), err)
+		}
+
+		// Unpack the tarfile to the rootfs path.
+		// FROM: https://godoc.org/github.com/moby/moby/pkg/archive#TarOptions
+		if err := archive.Untar(content.NewReader(layer), dest, &archive.TarOptions{
+			NoLchown:        true,
+			ExcludePatterns: []string{"dev/"}, // prevent 'operation not permitted'
+		}); err != nil {
+			return fmt.Errorf("extracting tar for %s to directory %s failed: %v", desc.Digest.String(), dest, err)
+		}
+	}
+
+	return nil
+}

pkg/installer/client/imgworker/workeropts.go

@@ -0,0 +1,106 @@
+package imgworker
+
+// FROM Slightly adapted from genuinetools/img worker
+
+import (
+	"context"
+	"fmt"
+	"path/filepath"
+
+	"github.com/containerd/containerd/content/local"
+	"github.com/containerd/containerd/diff/apply"
+	"github.com/containerd/containerd/diff/walking"
+	ctdmetadata "github.com/containerd/containerd/metadata"
+	"github.com/containerd/containerd/platforms"
+	"github.com/containerd/containerd/remotes/docker"
+	ctdsnapshot "github.com/containerd/containerd/snapshots"
+	"github.com/containerd/containerd/snapshots/native"
+	"github.com/moby/buildkit/cache/metadata"
+	containerdsnapshot "github.com/moby/buildkit/snapshot/containerd"
+	"github.com/moby/buildkit/util/binfmt_misc"
+	"github.com/moby/buildkit/util/leaseutil"
+	"github.com/moby/buildkit/worker/base"
+	specs "github.com/opencontainers/image-spec/specs-go/v1"
+	bolt "go.etcd.io/bbolt"
+)
+
+// createWorkerOpt creates a base.WorkerOpt to be used for a new worker.
+func (c *Client) createWorkerOpt() (opt base.WorkerOpt, err error) {
+
+	if c.opts != nil {
+		return *c.opts, nil
+	}
+
+	// Create the metadata store.
+	md, err := metadata.NewStore(filepath.Join(c.root, "metadata.db"))
+	if err != nil {
+		return opt, err
+	}
+
+	snapshotRoot := filepath.Join(c.root, "snapshots")
+
+	s, err := native.NewSnapshotter(snapshotRoot)
+	if err != nil {
+		return opt, fmt.Errorf("creating %s snapshotter failed: %v", c.backend, err)
+	}
+
+	// Create the content store locally.
+	contentStore, err := local.NewStore(filepath.Join(c.root, "content"))
+	if err != nil {
+		return opt, err
+	}
+
+	// Open the bolt database for metadata.
+	db, err := bolt.Open(filepath.Join(c.root, "containerdmeta.db"), 0644, nil)
+	if err != nil {
+		return opt, err
+	}
+
+	// Create the new database for metadata.
+	mdb := ctdmetadata.NewDB(db, contentStore, map[string]ctdsnapshot.Snapshotter{
+		c.backend: s,
+	})
+	if err := mdb.Init(context.TODO()); err != nil {
+		return opt, err
+	}
+
+	// Create the image store.
+	imageStore := ctdmetadata.NewImageStore(mdb)
+
+	contentStore = containerdsnapshot.NewContentStore(mdb.ContentStore(), "buildkit")
+
+	id, err := base.ID(c.root)
+	if err != nil {
+		return opt, err
+	}
+
+	xlabels := base.Labels("oci", c.backend)
+
+	var supportedPlatforms []specs.Platform
+	for _, p := range binfmt_misc.SupportedPlatforms(false) {
+		parsed, err := platforms.Parse(p)
+		if err != nil {
+			return opt, err
+		}
+		supportedPlatforms = append(supportedPlatforms, platforms.Normalize(parsed))
+	}
+
+	opt = base.WorkerOpt{
+		ID:             id,
+		Labels:         xlabels,
+		MetadataStore:  md,
+		Snapshotter:    containerdsnapshot.NewSnapshotter(c.backend, mdb.Snapshotter(c.backend), "buildkit", nil),
+		ContentStore:   contentStore,
+		Applier:        apply.NewFileSystemApplier(contentStore),
+		Differ:         walking.NewWalkingDiff(contentStore),
+		ImageStore:     imageStore,
+		Platforms:      supportedPlatforms,
+		RegistryHosts:  docker.ConfigureDefaultRegistries(),
+		LeaseManager:   leaseutil.WithNamespace(ctdmetadata.NewLeaseManager(mdb), "buildkit"),
+		GarbageCollect: mdb.GarbageCollect,
+	}
+
+	c.opts = &opt
+
+	return opt, err
+}

pkg/installer/client/local.go

@@ -51,7 +51,7 @@ func (c *LocalClient) DownloadArtifact(artifact compiler.Artifact) (compiler.Art
 
 	// Check if file is already in cache
 	if helpers.Exists(cacheFile) {
-		Info("Use artifact", artifactName, "from cache.")
+		Debug("Use artifact", artifactName, "from cache.")
 	} else {
 		ok := false
 		for _, uri := range c.RepoData.Urls {

pkg/installer/installer.go

@@ -175,7 +175,7 @@ func (l *LuetInstaller) Upgrade(s *System) error {
 		Info("By going forward, you are also accepting the licenses of the packages that you are going to install in your system.")
 		if Ask() {
 			l.Options.Ask = false // Don't prompt anymore
-			return l.swap(syncedRepos, uninstall, toInstall, s)
+			return l.swap(syncedRepos, uninstall, toInstall, s, true)
 		} else {
 			return errors.New("Aborted by user")
 		}
@@ -183,7 +183,7 @@ func (l *LuetInstaller) Upgrade(s *System) error {
 
 	Spinner(32)
 	defer SpinnerStop()
-	return l.swap(syncedRepos, uninstall, toInstall, s)
+	return l.swap(syncedRepos, uninstall, toInstall, s, true)
 }
 
 func (l *LuetInstaller) SyncRepositories(inMemory bool) (Repositories, error) {
@@ -214,33 +214,58 @@ func (l *LuetInstaller) Swap(toRemove pkg.Packages, toInstall pkg.Packages, s *S
 		return err
 	}
 
-	if len(toRemove) > 0 {
-		Info(":recycle: Packages that are going to be removed from the system:\n ", Yellow(packsToList(toRemove)).BgBlack().String())
-	}
-
-	if len(toInstall) > 0 {
-		Info(":zap:Packages that are going to be installed in the system:\n ", Green(packsToList(toInstall)).BgBlack().String())
-	}
-
-	if l.Options.Ask {
-		Info("By going forward, you are also accepting the licenses of the packages that you are going to install in your system.")
-		if Ask() {
-			l.Options.Ask = false // Don't prompt anymore
-			return l.swap(syncedRepos, toRemove, toInstall, s)
-		} else {
-			return errors.New("Aborted by user")
+	toRemoveFinal := pkg.Packages{}
+	for _, p := range toRemove {
+		packs, _ := s.Database.FindPackages(p)
+		if len(packs) == 0 {
+			return errors.New("Package " + p.HumanReadableString() + " not found in the system")
+		}
+		for _, pp := range packs {
+			toRemoveFinal = append(toRemoveFinal, pp)
 		}
 	}
 
-	return l.swap(syncedRepos, toRemove, toInstall, s)
+	return l.swap(syncedRepos, toRemoveFinal, toInstall, s, false)
 }
 
-func (l *LuetInstaller) swap(syncedRepos Repositories, toRemove pkg.Packages, toInstall pkg.Packages, s *System) error {
-	// First match packages against repositories by priority
+func (l *LuetInstaller) computeSwap(syncedRepos Repositories, toRemove pkg.Packages, toInstall pkg.Packages, s *System) (map[string]ArtifactMatch, pkg.Packages, solver.PackagesAssertions, pkg.PackageDatabase, error) {
+
 	allRepos := pkg.NewInMemoryDatabase(false)
 	syncedRepos.SyncDatabase(allRepos)
+
 	toInstall = syncedRepos.ResolveSelectors(toInstall)
 
+	// First check what would have been done
+	installedtmp, err := s.Database.Copy()
+	if err != nil {
+		return nil, nil, nil, nil, errors.Wrap(err, "Failed create temporary in-memory db")
+	}
+
+	systemAfterChanges := &System{Database: installedtmp}
+
+	packs, err := l.computeUninstall(systemAfterChanges, toRemove...)
+	if err != nil && !l.Options.Force {
+		Error("Failed computing uninstall for ", packsToList(toRemove))
+		return nil, nil, nil, nil, errors.Wrap(err, "computing uninstall "+packsToList(toRemove))
+	}
+	for _, p := range packs {
+		err = systemAfterChanges.Database.RemovePackage(p)
+		if err != nil {
+			return nil, nil, nil, nil, errors.Wrap(err, "Failed removing package from database")
+		}
+	}
+
+	match, packages, assertions, allRepos, err := l.computeInstall(syncedRepos, toInstall, systemAfterChanges)
+	for _, p := range toInstall {
+		assertions = append(assertions, solver.PackageAssert{Package: p.(*pkg.DefaultPackage), Value: true})
+	}
+	return match, packages, assertions, allRepos, err
+}
+
+func (l *LuetInstaller) swap(syncedRepos Repositories, toRemove pkg.Packages, toInstall pkg.Packages, s *System, forceNodeps bool) error {
+	forced := l.Options.Force
+	nodeps := l.Options.NoDeps
+
 	// We don't want any conflict with the installed to raise during the upgrade.
 	// In this way we both force uninstalls and we avoid to check with conflicts
 	// against the current system state which is pending to deletion
@@ -248,51 +273,42 @@ func (l *LuetInstaller) swap(syncedRepos Repositories, toRemove pkg.Packages, to
 	// if the old A results installed in the system. This is due to the fact that
 	// now the solver enforces the constraints and explictly denies two packages
 	// of the same version installed.
-	forced := l.Options.Force
-	nodeps := l.Options.NoDeps
 	l.Options.Force = true
-	l.Options.NoDeps = true
-
-	// First check what would have been done
-	installedtmp := pkg.NewInMemoryDatabase(false)
-
-	for _, i := range s.Database.World() {
-		_, err := installedtmp.CreatePackage(i)
-		if err != nil {
-			return errors.Wrap(err, "Failed create temporary in-memory db")
-		}
-	}
-	systemAfterChanges := &System{Database: installedtmp}
-
-	for _, u := range toRemove {
-		packs, err := l.computeUninstall(u, systemAfterChanges)
-		if err != nil && !l.Options.Force {
-			Error("Failed computing uninstall for ", u.HumanReadableString())
-			return errors.Wrap(err, "computing uninstall "+u.HumanReadableString())
-		}
-		for _, p := range packs {
-			err = systemAfterChanges.Database.RemovePackage(p)
-			if err != nil {
-				return errors.Wrap(err, "Failed removing package from database")
-			}
-		}
+	if forceNodeps {
+		l.Options.NoDeps = true
 	}
 
-	match, packages, assertions, allRepos, err := l.computeInstall(syncedRepos, toInstall, systemAfterChanges)
+	match, packages, assertions, allRepos, err := l.computeSwap(syncedRepos, toRemove, toInstall, s)
 	if err != nil {
-		return errors.Wrap(err, "computing installation")
+		return errors.Wrap(err, "failed computing package replacement")
 	}
 
+	if l.Options.Ask {
+		if len(toRemove) > 0 {
+			Info(":recycle: Packages that are going to be removed from the system:\n ", Yellow(packsToList(toRemove)).BgBlack().String())
+		}
+
+		if len(match) > 0 {
+			Info("Packages that are going to be installed in the system: \n ", Green(matchesToList(match)).BgBlack().String())
+		}
+
+		Info("By going forward, you are also accepting the licenses of the packages that you are going to install in your system.")
+		if Ask() {
+			l.Options.Ask = false // Don't prompt anymore
+		} else {
+			return errors.New("Aborted by user")
+		}
+	}
+
+	// First match packages against repositories by priority
 	if err := l.download(syncedRepos, match); err != nil {
 		return errors.Wrap(err, "Pre-downloading packages")
 	}
 
-	for _, u := range toRemove {
-		err := l.Uninstall(u, s)
-		if err != nil && !l.Options.Force {
-			Error("Failed uninstall for ", u.HumanReadableString())
-			return errors.Wrap(err, "uninstalling "+u.HumanReadableString())
-		}
+	err = l.Uninstall(s, toRemove...)
+	if err != nil && !l.Options.Force {
+		Error("Failed uninstall for ", packsToList(toRemove))
+		return errors.Wrap(err, "uninstalling "+packsToList(toRemove))
 	}
 
 	l.Options.Force = forced
@@ -337,7 +353,6 @@ func (l *LuetInstaller) Install(cp pkg.Packages, s *System) error {
 			}
 		}
 	}
-
 	Info("Packages that are going to be installed in the system: \n ", Green(matchesToList(match)).BgBlack().String())
 
 	if l.Options.Ask {
@@ -576,7 +591,7 @@ func (l *LuetInstaller) install(syncedRepos Repositories, toInstall map[string]A
 		}
 	}
 
-	return s.ExecuteFinalizers(toFinalize, l.Options.Force)
+	return s.ExecuteFinalizers(toFinalize)
 }
 
 func (l *LuetInstaller) downloadPackage(a ArtifactMatch) (compiler.Artifact, error) {
@@ -587,7 +602,7 @@ func (l *LuetInstaller) downloadPackage(a ArtifactMatch) (compiler.Artifact, err
 	}
 
 	err = artifact.Verify()
-	if err != nil && !l.Options.Force {
+	if err != nil {
 		return nil, errors.Wrap(err, "Artifact integrity check failure")
 	}
 	return artifact, nil
@@ -621,15 +636,11 @@ func (l *LuetInstaller) downloadWorker(i int, wg *sync.WaitGroup, c <-chan Artif
 	for p := range c {
 		// TODO: Keep trace of what was added from the tar, and save it into system
 		_, err := l.downloadPackage(p)
-		if err != nil && !l.Options.Force {
-			//TODO: Uninstall, rollback.
-			Fatal("Failed installing package "+p.Package.GetName(), err.Error())
-			return errors.Wrap(err, "Failed installing package "+p.Package.GetName())
-		}
-		if err == nil {
-			Info("\n:package: Package ", p.Package.HumanReadableString(), "downloaded")
-		} else if err != nil && l.Options.Force {
-			Info("\n:package: ", p.Package.HumanReadableString(), "downloaded with failures (force download)")
+		if err != nil {
+			Fatal("Failed downloading package "+p.Package.GetName(), err.Error())
+			return errors.Wrap(err, "Failed downloading package "+p.Package.GetName())
+		} else {
+			Info(":package: Package ", p.Package.HumanReadableString(), "downloaded")
 		}
 	}
 
@@ -748,7 +759,7 @@ func (l *LuetInstaller) uninstall(p pkg.Package, s *System) error {
 	return nil
 }
 
-func (l *LuetInstaller) computeUninstall(p pkg.Package, s *System) (pkg.Packages, error) {
+func (l *LuetInstaller) computeUninstall(s *System, packs ...pkg.Package) (pkg.Packages, error) {
 
 	var toUninstall pkg.Packages
 	// compute uninstall from all world - remove packages in parallel - run uninstall finalizer (in order) TODO - mark the uninstallation in db
@@ -762,13 +773,10 @@ func (l *LuetInstaller) computeUninstall(p pkg.Package, s *System) (pkg.Packages
 
 	// Create a temporary DB with the installed packages
 	// so the solver is much faster finding the deptree
-	installedtmp := pkg.NewInMemoryDatabase(false)
-
-	for _, i := range s.Database.World() {
-		_, err := installedtmp.CreatePackage(i)
-		if err != nil {
-			return toUninstall, errors.Wrap(err, "Failed create temporary in-memory db")
-		}
+	// First check what would have been done
+	installedtmp, err := s.Database.Copy()
+	if err != nil {
+		return toUninstall, errors.Wrap(err, "Failed create temporary in-memory db")
 	}
 
 	if !l.Options.NoDeps {
@@ -776,12 +784,12 @@ func (l *LuetInstaller) computeUninstall(p pkg.Package, s *System) (pkg.Packages
 		var solution pkg.Packages
 		var err error
 		if l.Options.FullCleanUninstall {
-			solution, err = solv.UninstallUniverse(pkg.Packages{p})
+			solution, err = solv.UninstallUniverse(packs)
 			if err != nil {
 				return toUninstall, errors.Wrap(err, "Could not solve the uninstall constraints. Tip: try with --solver-type qlearning or with --force, or by removing packages excluding their dependencies with --nodeps")
 			}
 		} else {
-			solution, err = solv.Uninstall(checkConflicts, full, p)
+			solution, err = solv.Uninstall(checkConflicts, full, packs...)
 			if err != nil && !l.Options.Force {
 				return toUninstall, errors.Wrap(err, "Could not solve the uninstall constraints. Tip: try with --solver-type qlearning or with --force, or by removing packages excluding their dependencies with --nodeps")
 			}
@@ -791,25 +799,22 @@ func (l *LuetInstaller) computeUninstall(p pkg.Package, s *System) (pkg.Packages
 			toUninstall = append(toUninstall, p)
 		}
 	} else {
-		toUninstall = append(toUninstall, p)
-
+		toUninstall = append(toUninstall, packs...)
 	}
 
 	return toUninstall, nil
 }
-func (l *LuetInstaller) Uninstall(p pkg.Package, s *System) error {
-	if p.IsSelector() {
+func (l *LuetInstaller) Uninstall(s *System, packs ...pkg.Package) error {
+
+	for _, p := range packs {
 		if packs, _ := s.Database.FindPackages(p); len(packs) == 0 {
 			return errors.New("Package not found in the system")
 		}
-	} else {
-		if _, err := s.Database.FindPackage(p); err != nil {
-			return errors.Wrap(err, "package not found in the system")
-		}
+
 	}
 
 	Spinner(32)
-	toUninstall, err := l.computeUninstall(p, s)
+	toUninstall, err := l.computeUninstall(s, packs...)
 	if err != nil {
 		return errors.Wrap(err, "while computing uninstall")
 	}
@@ -830,10 +835,8 @@ func (l *LuetInstaller) Uninstall(p pkg.Package, s *System) error {
 		return nil
 	}
 
-	Info(":recycle: Packages that are going to be removed from the system:\n   ", Yellow(packsToList(toUninstall)).BgBlack().String())
-
 	if l.Options.Ask {
-		Info("By going forward, you are also accepting the licenses of the packages that you are going to install in your system.")
+		Info(":recycle: Packages that are going to be removed from the system:\n   ", Yellow(packsToList(toUninstall)).BgBlack().String())
 		if Ask() {
 			l.Options.Ask = false // Don't prompt anymore
 			return uninstall()

pkg/installer/installer_test.go

@@ -24,6 +24,7 @@ import (
 	compiler "github.com/mudler/luet/pkg/compiler"
 	backend "github.com/mudler/luet/pkg/compiler/backend"
 	"github.com/mudler/luet/pkg/helpers"
+	"github.com/mudler/luet/pkg/installer"
 	solver "github.com/mudler/luet/pkg/solver"
 
 	. "github.com/mudler/luet/pkg/installer"
@@ -33,6 +34,18 @@ import (
 	. "github.com/onsi/gomega"
 )
 
+func stubRepo(tmpdir, tree string) (installer.Repository, error) {
+	return GenerateRepository(
+		"test",
+		"description",
+		"disk",
+		[]string{tmpdir},
+		1,
+		tmpdir,
+		[]string{tree},
+		pkg.NewInMemoryDatabase(false), nil, "", false, false)
+}
+
 var _ = Describe("Installer", func() {
 	Context("Writes a repository definition", func() {
 		It("Writes a repo and can install packages from it", func() {
@@ -60,8 +73,8 @@ var _ = Describe("Installer", func() {
 			Expect(err).ToNot(HaveOccurred())
 			defer os.RemoveAll(tmpdir) // clean up
 
-			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "./generate.sh"}))
-			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2", "chmod +x generate.sh"}))
+			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "chmod +x generate.sh", "./generate.sh"}))
+			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2"}))
 
 			spec.SetOutputPath(tmpdir)
 			c.SetConcurrency(2)
@@ -84,13 +97,13 @@ var _ = Describe("Installer", func() {
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/buildable"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
@@ -134,7 +147,7 @@ urls:
 			Expect(err).ToNot(HaveOccurred())
 			Expect(p.GetName()).To(Equal("b"))
 
-			err = inst.Uninstall(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}, system)
+			err = inst.Uninstall(system, &pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
 			Expect(err).ToNot(HaveOccurred())
 
 			// Nothing should be there anymore (files, packagedb entry)
@@ -176,8 +189,8 @@ urls:
 			Expect(err).ToNot(HaveOccurred())
 			defer os.RemoveAll(tmpdir) // clean up
 
-			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "./generate.sh"}))
-			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2", "chmod +x generate.sh"}))
+			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "chmod +x generate.sh", "./generate.sh"}))
+			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2"}))
 
 			spec.SetOutputPath(tmpdir)
 			c.SetConcurrency(2)
@@ -200,7 +213,9 @@ urls:
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/buildable"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
+			Expect(err).ToNot(HaveOccurred())
+
 			treeFile := NewDefaultTreeRepositoryFile()
 			treeFile.SetCompressionType(compiler.None)
 			repo.SetRepositoryFile(REPOFILE_TREE_KEY, treeFile)
@@ -209,7 +224,7 @@ urls:
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
@@ -253,7 +268,7 @@ urls:
 			Expect(err).ToNot(HaveOccurred())
 			Expect(p.GetName()).To(Equal("b"))
 
-			err = inst.Uninstall(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}, system)
+			err = inst.Uninstall(system, &pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
 			Expect(err).ToNot(HaveOccurred())
 
 			// Nothing should be there anymore (files, packagedb entry)
@@ -294,8 +309,8 @@ urls:
 			Expect(err).ToNot(HaveOccurred())
 			defer os.RemoveAll(tmpdir) // clean up
 
-			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "./generate.sh"}))
-			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2", "chmod +x generate.sh"}))
+			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "chmod +x generate.sh", "./generate.sh"}))
+			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2"}))
 
 			spec.SetOutputPath(tmpdir)
 			c.SetConcurrency(2)
@@ -318,13 +333,20 @@ urls:
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/buildable"}, pkg.NewInMemoryDatabase(false))
+			repo, err := GenerateRepository(
+				"test",
+				"description",
+				"disk",
+				[]string{tmpdir}, 1,
+				tmpdir,
+				[]string{"../../tests/fixtures/buildable"},
+				pkg.NewInMemoryDatabase(false), nil, "", false, false)
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
@@ -373,7 +395,7 @@ urls:
 			Expect(files).To(Equal([]string{"artifact42", "test5", "test6"}))
 			Expect(err).ToNot(HaveOccurred())
 
-			err = inst.Uninstall(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"}, system)
+			err = inst.Uninstall(system, &pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
 			Expect(err).ToNot(HaveOccurred())
 
 			// Nothing should be there anymore (files, packagedb entry)
@@ -412,8 +434,8 @@ urls:
 			Expect(err).ToNot(HaveOccurred())
 			defer os.RemoveAll(tmpdir) // clean up
 
-			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "./generate.sh"}))
-			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2", "chmod +x generate.sh"}))
+			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "chmod +x generate.sh", "./generate.sh"}))
+			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2"}))
 
 			spec.SetOutputPath(tmpdir)
 			c.SetConcurrency(2)
@@ -436,13 +458,21 @@ urls:
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/buildable"}, pkg.NewInMemoryDatabase(false))
+			repo, err := GenerateRepository(
+				"test",
+				"description",
+				"disk",
+				[]string{tmpdir},
+				1,
+				tmpdir,
+				[]string{"../../tests/fixtures/buildable"},
+				pkg.NewInMemoryDatabase(false), nil, "", false, false)
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
@@ -502,9 +532,9 @@ urls:
 			Expect(err).ToNot(HaveOccurred())
 			Expect(helpers.Exists(artifact.GetPath())).To(BeTrue())
 
-			repo, err = GenerateRepository("test", "description", "disk", []string{tmpdir2}, 1, tmpdir2, []string{"../../tests/fixtures/alpine"}, pkg.NewInMemoryDatabase(false))
+			repo, err = stubRepo(tmpdir2, "../../tests/fixtures/alpine")
 			Expect(err).ToNot(HaveOccurred())
-			err = repo.Write(tmpdir2, false)
+			err = repo.Write(tmpdir2, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			fakeroot, err = ioutil.TempDir("", "fakeroot")
@@ -571,13 +601,13 @@ urls:
 
 			Expect(errs).To(BeEmpty())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/upgrade"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
@@ -697,18 +727,18 @@ urls:
 			_, errs = c2.CompileParallel(false, compiler.NewLuetCompilationspecs(spec2))
 			Expect(errs).To(BeEmpty())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/upgrade_old_repo"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade_old_repo")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
-			repoupgrade, err := GenerateRepository("test", "description", "disk", []string{tmpdirnewrepo}, 1, tmpdirnewrepo, []string{"../../tests/fixtures/upgrade_new_repo"}, pkg.NewInMemoryDatabase(false))
+			repoupgrade, err := stubRepo(tmpdirnewrepo, "../../tests/fixtures/upgrade_new_repo")
 			Expect(err).ToNot(HaveOccurred())
-			err = repoupgrade.Write(tmpdirnewrepo, false)
+			err = repoupgrade.Write(tmpdirnewrepo, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			fakeroot, err := ioutil.TempDir("", "fakeroot")
@@ -820,13 +850,13 @@ urls:
 
 			Expect(errs).To(BeEmpty())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/upgrade"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 			Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar.gz"))).To(BeTrue())
 			Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar"))).ToNot(BeTrue())
@@ -899,6 +929,47 @@ urls:
 
 	})
 
+	Context("Uninstallation", func() {
+		It("fails if package is required by others which are installed", func() {
+
+			fakeroot, err := ioutil.TempDir("", "fakeroot")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(fakeroot) // clean up
+			bolt, err := ioutil.TempDir("", "db")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(bolt) // clean up
+
+			systemDB := pkg.NewBoltDatabase(filepath.Join(bolt, "db.db"))
+			system := &System{Database: systemDB, Target: fakeroot}
+
+			inst := NewLuetInstaller(LuetInstallerOptions{Concurrency: 1, CheckConflicts: true})
+
+			D := pkg.NewPackage("D", "", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
+			B := pkg.NewPackage("calamares", "", []*pkg.DefaultPackage{D}, []*pkg.DefaultPackage{})
+			C := pkg.NewPackage("kpmcore", "", []*pkg.DefaultPackage{B}, []*pkg.DefaultPackage{})
+			A := pkg.NewPackage("A", "", []*pkg.DefaultPackage{B}, []*pkg.DefaultPackage{})
+			Z := pkg.NewPackage("chromium", "", []*pkg.DefaultPackage{A}, []*pkg.DefaultPackage{})
+			F := pkg.NewPackage("F", "", []*pkg.DefaultPackage{Z, B}, []*pkg.DefaultPackage{})
+
+			Z.SetVersion("86.0.4240.193+2")
+			Z.SetCategory("www-client")
+			B.SetVersion("3.2.32.1+5")
+			B.SetCategory("app-admin")
+			C.SetVersion("4.2.0+2")
+			C.SetCategory("sys-libs-5")
+			D.SetVersion("5.19.5+9")
+			D.SetCategory("layers")
+
+			for _, p := range []pkg.Package{A, B, C, D, Z, F} {
+				_, err := systemDB.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+
+			err = inst.Uninstall(system, D)
+			Expect(err).To(HaveOccurred())
+		})
+	})
+
 	Context("Existing files", func() {
 		It("Reclaims them", func() {
 			//repo:=NewLuetSystemRepository()
@@ -937,13 +1008,13 @@ urls:
 
 			Expect(errs).To(BeEmpty())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/upgrade"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 			Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar.gz"))).To(BeTrue())
 			Expect(helpers.Exists(spec.Rel("b-test-1.1.package.tar"))).ToNot(BeTrue())
@@ -1039,13 +1110,13 @@ urls:
 
 			Expect(errs).To(BeEmpty())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/upgrade_old_repo"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/upgrade_old_repo")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel("repository.yaml"))).To(BeTrue())
@@ -1125,10 +1196,10 @@ urls:
 
 			Expect(errs).To(BeEmpty())
 
-			repo, err = GenerateRepository("test", "description", "disk", []string{tmpdir2}, 1, tmpdir2, []string{"../../tests/fixtures/upgrade_new_repo"}, pkg.NewInMemoryDatabase(false))
+			repo, err = stubRepo(tmpdir2, "../../tests/fixtures/upgrade_new_repo")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
-			err = repo.Write(tmpdir2, false)
+			err = repo.Write(tmpdir2, false, false)
 			Expect(err).ToNot(HaveOccurred())
 
 			inst = NewLuetInstaller(LuetInstallerOptions{Concurrency: 1})

pkg/installer/interface.go

@@ -24,7 +24,7 @@ import (
 
 type Installer interface {
 	Install(pkg.Packages, *System) error
-	Uninstall(pkg.Package, *System) error
+	Uninstall(*System, ...pkg.Package) error
 	Upgrade(s *System) error
 	Reclaim(s *System) error
 
@@ -51,7 +51,7 @@ type Repository interface {
 	SetIndex(i compiler.ArtifactIndex)
 	GetTree() tree.Builder
 	SetTree(tree.Builder)
-	Write(path string, resetRevision bool) error
+	Write(path string, resetRevision, force bool) error
 	Sync(bool) (Repository, error)
 	GetTreePath() string
 	SetTreePath(string)
@@ -72,4 +72,6 @@ type Repository interface {
 	SetRepositoryFile(string, LuetRepositoryFile)
 	SetName(p string)
 	Serialize() (*LuetSystemRepositoryMetadata, LuetSystemRepositorySerialized)
+	GetBackend() compiler.CompilerBackend
+	SetBackend(b compiler.CompilerBackend)
 }

pkg/installer/repository.go

@@ -46,6 +46,10 @@ const (
 
 	REPOFILE_TREE_KEY = "tree"
 	REPOFILE_META_KEY = "meta"
+
+	DiskRepositoryType   = "disk"
+	HttpRepositoryType   = "http"
+	DockerRepositoryType = "docker"
 )
 
 type LuetRepositoryFile struct {
@@ -60,6 +64,9 @@ type LuetSystemRepository struct {
 	Index           compiler.ArtifactIndex        `json:"index"`
 	Tree            tree.Builder                  `json:"-"`
 	RepositoryFiles map[string]LuetRepositoryFile `json:"repo_files"`
+	Backend         compiler.CompilerBackend      `json:"-"`
+	PushImages      bool                          `json:"-"`
+	ForcePush       bool                          `json:"-"`
 }
 
 type LuetSystemRepositorySerialized struct {
@@ -157,27 +164,54 @@ func NewDefaultMetaRepositoryFile() LuetRepositoryFile {
 	}
 }
 
+// SetFileName sets the name of the repository file.
+// Each repository can ship arbitrary file that will be downloaded by the client
+// in case of need, this set the filename that the client will pull
 func (f *LuetRepositoryFile) SetFileName(n string) {
 	f.FileName = n
 }
 
+// GetFileName returns the name of the repository file.
+// Each repository can ship arbitrary file that will be downloaded by the client
+// in case of need, this gets the filename that the client will pull
 func (f *LuetRepositoryFile) GetFileName() string {
 	return f.FileName
 }
+
+// SetCompressionType sets the compression type of the repository file.
+// Each repository can ship arbitrary file that will be downloaded by the client
+// in case of need, this sets the compression type that the client will use to uncompress the artifact
 func (f *LuetRepositoryFile) SetCompressionType(c compiler.CompressionImplementation) {
 	f.CompressionType = c
 }
+
+// GetCompressionType gets the compression type of the repository file.
+// Each repository can ship arbitrary file that will be downloaded by the client
+// in case of need, this gets the compression type that the client will use to uncompress the artifact
 func (f *LuetRepositoryFile) GetCompressionType() compiler.CompressionImplementation {
 	return f.CompressionType
 }
+
+// SetChecksums sets the checksum of the repository file.
+// Each repository can ship arbitrary file that will be downloaded by the client
+// in case of need, this sets the checksums that the client will use to verify the artifact
 func (f *LuetRepositoryFile) SetChecksums(c compiler.Checksums) {
 	f.Checksums = c
 }
+
+// GetChecksums gets the checksum of the repository file.
+// Each repository can ship arbitrary file that will be downloaded by the client
+// in case of need, this gets the checksums that the client will use to verify the artifact
 func (f *LuetRepositoryFile) GetChecksums() compiler.Checksums {
 	return f.Checksums
 }
 
-func GenerateRepository(name, descr, t string, urls []string, priority int, src string, treesDir []string, db pkg.PackageDatabase) (Repository, error) {
+// GenerateRepository generates a new repository from the given argument.
+// If the repository is of the docker type, it will also push the package images.
+// In case the repository is local, it will build the package Index
+func GenerateRepository(name, descr, t string, urls []string,
+	priority int, src string, treesDir []string, db pkg.PackageDatabase,
+	b compiler.CompilerBackend, imagePrefix string, pushImages, force bool) (Repository, error) {
 
 	tr := tree.NewInstallerRecipe(db)
 
@@ -188,14 +222,31 @@ func GenerateRepository(name, descr, t string, urls []string, priority int, src
 		}
 	}
 
-	art, err := buildPackageIndex(src, tr.GetDatabase())
-	if err != nil {
-		return nil, err
+	// if !strings.HasSuffix(imagePrefix, "/") {
+	// 	imagePrefix = imagePrefix + "/"
+	// }
+
+	var art []compiler.Artifact
+	var err error
+	switch t {
+	case DiskRepositoryType, HttpRepositoryType:
+		art, err = buildPackageIndex(src, tr.GetDatabase())
+		if err != nil {
+			return nil, err
+		}
+
+	case DockerRepositoryType:
+		art, err = generatePackageImages(b, imagePrefix, src, tr.GetDatabase(), pushImages, force)
+		if err != nil {
+			return nil, err
+		}
 	}
 
-	return NewLuetSystemRepository(
+	repo := NewLuetSystemRepository(
 		config.NewLuetRepository(name, t, descr, urls, priority, true, false),
-		art, tr), nil
+		art, tr, pushImages, force)
+	repo.SetBackend(b)
+	return repo, nil
 }
 
 func NewSystemRepository(repo config.LuetRepository) Repository {
@@ -205,12 +256,14 @@ func NewSystemRepository(repo config.LuetRepository) Repository {
 	}
 }
 
-func NewLuetSystemRepository(repo *config.LuetRepository, art []compiler.Artifact, builder tree.Builder) Repository {
+func NewLuetSystemRepository(repo *config.LuetRepository, art []compiler.Artifact, builder tree.Builder, pushImages, force bool) Repository {
 	return &LuetSystemRepository{
 		LuetRepository:  repo,
 		Index:           art,
 		Tree:            builder,
 		RepositoryFiles: map[string]LuetRepositoryFile{},
+		PushImages:      pushImages,
+		ForcePush:       force,
 	}
 }
 
@@ -244,6 +297,71 @@ func NewLuetSystemRepositoryFromYaml(data []byte, db pkg.PackageDatabase) (Repos
 	return r, err
 }
 
+func pushImage(b compiler.CompilerBackend, image string, force bool) error {
+	if b.ImageAvailable(image) && !force {
+		Debug("Image", image, "already present, skipping")
+		return nil
+	}
+	return b.Push(compiler.CompilerBackendOptions{ImageName: image})
+}
+
+func generatePackageImages(b compiler.CompilerBackend, imagePrefix, path string, db pkg.PackageDatabase, imagePush, force bool) ([]compiler.Artifact, error) {
+	Info("Generating docker images for packages in", imagePrefix)
+	var art []compiler.Artifact
+	var ff = func(currentpath string, info os.FileInfo, err error) error {
+
+		if !strings.HasSuffix(info.Name(), ".metadata.yaml") {
+			return nil // Skip with no errors
+		}
+
+		dat, err := ioutil.ReadFile(currentpath)
+		if err != nil {
+			return errors.Wrap(err, "Error reading file "+currentpath)
+		}
+
+		artifact, err := compiler.NewPackageArtifactFromYaml(dat)
+		if err != nil {
+			return errors.Wrap(err, "Error reading yaml "+currentpath)
+		}
+
+		// We want to include packages that are ONLY referenced in the tree.
+		// the ones which aren't should be deleted. (TODO: by another cli command?)
+		if _, notfound := db.FindPackage(artifact.GetCompileSpec().GetPackage()); notfound != nil {
+			Debug(fmt.Sprintf("Package %s not found in tree. Ignoring it.",
+				artifact.GetCompileSpec().GetPackage().HumanReadableString()))
+			return nil
+		}
+
+		packageImage := fmt.Sprintf("%s:%s", imagePrefix, artifact.GetCompileSpec().GetPackage().GetFingerPrint())
+
+		if imagePush && b.ImageAvailable(packageImage) && !force {
+			Info("Image", packageImage, "already present, skipping. use --force-push to override")
+		} else {
+			Info("Generating final image", packageImage,
+				"for package ", artifact.GetCompileSpec().GetPackage().HumanReadableString())
+			if opts, err := artifact.GenerateFinalImage(packageImage, b, true); err != nil {
+				return errors.Wrap(err, "Failed generating metadata tree"+opts.ImageName)
+			}
+		}
+		if imagePush {
+			if err := pushImage(b, packageImage, force); err != nil {
+				return errors.Wrapf(err, "Failed while pushing image: '%s'", packageImage)
+			}
+		}
+
+		art = append(art, artifact)
+
+		return nil
+	}
+
+	err := filepath.Walk(path, ff)
+	if err != nil {
+		return nil, err
+
+	}
+	return art, nil
+}
+
 func buildPackageIndex(path string, db pkg.PackageDatabase) ([]compiler.Artifact, error) {
 
 	var art []compiler.Artifact
@@ -266,7 +384,7 @@ func buildPackageIndex(path string, db pkg.PackageDatabase) ([]compiler.Artifact
 		// We want to include packages that are ONLY referenced in the tree.
 		// the ones which aren't should be deleted. (TODO: by another cli command?)
 		if _, notfound := db.FindPackage(artifact.GetCompileSpec().GetPackage()); notfound != nil {
-			Info(fmt.Sprintf("Package %s not found in tree. Ignoring it.",
+			Debug(fmt.Sprintf("Package %s not found in tree. Ignoring it.",
 				artifact.GetCompileSpec().GetPackage().HumanReadableString()))
 			return nil
 		}
@@ -306,6 +424,13 @@ func (r *LuetSystemRepository) SetType(p string) {
 	r.LuetRepository.Type = p
 }
 
+func (r *LuetSystemRepository) GetBackend() compiler.CompilerBackend {
+	return r.Backend
+}
+func (r *LuetSystemRepository) SetBackend(b compiler.CompilerBackend) {
+	r.Backend = b
+}
+
 func (r *LuetSystemRepository) SetName(p string) {
 	r.LuetRepository.Name = p
 }
@@ -400,7 +525,7 @@ func (r *LuetSystemRepository) ReadSpecFile(file string, removeFile bool) (Repos
 	return repo, err
 }
 
-func (r *LuetSystemRepository) Write(dst string, resetRevision bool) error {
+func (r *LuetSystemRepository) genLocalRepo(dst string, resetRevision bool) error {
 	err := os.MkdirAll(dst, os.ModePerm)
 	if err != nil {
 		return err
@@ -522,24 +647,240 @@ func (r *LuetSystemRepository) Write(dst string, resetRevision bool) error {
 		Repo: *r,
 		Path: dst,
 	})
-
 	return nil
 }
 
+func (r *LuetSystemRepository) genDockerRepo(imagePrefix string, resetRevision, force bool) error {
+	// - Iterate over meta, build final images, push them if necessary
+	//   - while pushing, check if image already exists, and if exist push them only if --force is supplied
+	// - Generate final images for metadata and push
+
+	imageRepository := fmt.Sprintf("%s:%s", imagePrefix, REPOSITORY_SPECFILE)
+
+	r.LastUpdate = strconv.FormatInt(time.Now().Unix(), 10)
+
+	repoTemp, err := config.LuetCfg.GetSystem().TempDir("repo")
+	if err != nil {
+		return errors.Wrap(err, "Error met while creating tempdir for repository")
+	}
+	defer os.RemoveAll(repoTemp) // clean up
+
+	if r.GetBackend().ImageAvailable(imageRepository) {
+		if err := r.GetBackend().DownloadImage(compiler.CompilerBackendOptions{ImageName: imageRepository}); err != nil {
+			return errors.Wrapf(err, "while downloading '%s'", imageRepository)
+		}
+
+		if err := r.GetBackend().ExtractRootfs(compiler.CompilerBackendOptions{ImageName: imageRepository, Destination: repoTemp}, false); err != nil {
+			return errors.Wrapf(err, "while extracting '%s'", imageRepository)
+		}
+	}
+
+	repospec := filepath.Join(repoTemp, REPOSITORY_SPECFILE)
+	if resetRevision {
+		r.Revision = 0
+	} else {
+		if _, err := os.Stat(repospec); !os.IsNotExist(err) {
+			// Read existing file for retrieve revision
+			spec, err := r.ReadSpecFile(repospec, false)
+			if err != nil {
+				return err
+			}
+			r.Revision = spec.GetRevision()
+		}
+	}
+	r.Revision++
+
+	Info(fmt.Sprintf(
+		"For repository %s creating revision %d and last update %s...",
+		r.Name, r.Revision, r.LastUpdate,
+	))
+
+	bus.Manager.Publish(bus.EventRepositoryPreBuild, struct {
+		Repo LuetSystemRepository
+		Path string
+	}{
+		Repo: *r,
+		Path: imageRepository,
+	})
+
+	// Create tree and repository file
+	archive, err := config.LuetCfg.GetSystem().TempDir("archive")
+	if err != nil {
+		return errors.Wrap(err, "Error met while creating tempdir for archive")
+	}
+	defer os.RemoveAll(archive) // clean up
+	err = r.GetTree().Save(archive)
+	if err != nil {
+		return errors.Wrap(err, "Error met while saving the tree")
+	}
+
+	treeFile, err := r.GetRepositoryFile(REPOFILE_TREE_KEY)
+	if err != nil {
+		treeFile = NewDefaultTreeRepositoryFile()
+		r.SetRepositoryFile(REPOFILE_TREE_KEY, treeFile)
+	}
+
+	a := compiler.NewPackageArtifact(filepath.Join(repoTemp, treeFile.GetFileName()))
+	a.SetCompressionType(treeFile.GetCompressionType())
+	err = a.Compress(archive, 1)
+	if err != nil {
+		return errors.Wrap(err, "Error met while creating package archive")
+	}
+
+	// Update the tree name with the name created by compression selected.
+	treeFile.SetFileName(a.GetFileName())
+	err = a.Hash()
+	if err != nil {
+		return errors.Wrap(err, "Failed generating checksums for tree")
+	}
+	treeFile.SetChecksums(a.GetChecksums())
+	r.SetRepositoryFile(REPOFILE_TREE_KEY, treeFile)
+
+	// we generate a new archive containing the required compressed file.
+	// TODO: Bundle all the extra files in 1 docker image only, instead of an image for each file
+	treeArchive, err := compiler.CreateArtifactForFile(a.GetPath())
+	if err != nil {
+		return errors.Wrap(err, "Failed generating checksums for tree")
+	}
+	imageTree := fmt.Sprintf("%s:%s", imagePrefix, a.GetFileName())
+	Debug("Generating image", imageTree)
+	if opts, err := treeArchive.GenerateFinalImage(imageTree, r.GetBackend(), false); err != nil {
+		return errors.Wrap(err, "Failed generating metadata tree "+opts.ImageName)
+	}
+	if r.PushImages {
+		if err := pushImage(r.GetBackend(), imageTree, true); err != nil {
+			return errors.Wrapf(err, "Failed while pushing image: '%s'", imageTree)
+		}
+	}
+
+	// Create Metadata struct and serialized repository
+	meta, serialized := r.Serialize()
+
+	// Create metadata file and repository file
+	metaTmpDir, err := config.LuetCfg.GetSystem().TempDir("metadata")
+	if err != nil {
+		return errors.Wrap(err, "Error met while creating tempdir for metadata")
+	}
+	defer os.RemoveAll(metaTmpDir) // clean up
+
+	metaFile, err := r.GetRepositoryFile(REPOFILE_META_KEY)
+	if err != nil {
+		metaFile = NewDefaultMetaRepositoryFile()
+		r.SetRepositoryFile(REPOFILE_META_KEY, metaFile)
+	}
+
+	repoMetaSpec := filepath.Join(metaTmpDir, REPOSITORY_METAFILE)
+	// Create repository.meta.yaml file
+	err = meta.WriteFile(repoMetaSpec)
+	if err != nil {
+		return err
+	}
+
+	// create temp dir for metafile
+	metaDir, err := config.LuetCfg.GetSystem().TempDir("metadata")
+	if err != nil {
+		return errors.Wrap(err, "Error met while creating tempdir for metadata")
+	}
+	defer os.RemoveAll(metaDir) // clean up
+
+	a = compiler.NewPackageArtifact(filepath.Join(metaDir, metaFile.GetFileName()))
+	a.SetCompressionType(metaFile.GetCompressionType())
+	err = a.Compress(metaTmpDir, 1)
+	if err != nil {
+		return errors.Wrap(err, "Error met while archiving repository metadata")
+	}
+
+	metaFile.SetFileName(a.GetFileName())
+	r.SetRepositoryFile(REPOFILE_META_KEY, metaFile)
+	err = a.Hash()
+	if err != nil {
+		return errors.Wrap(err, "Failed generating checksums for metadata")
+	}
+	metaFile.SetChecksums(a.GetChecksums())
+
+	// Files are downloaded as-is from docker images
+	// we generate a new archive containing the required compressed file.
+	// TODO: Bundle all the extra files in 1 docker image only, instead of an image for each file
+	metaArchive, err := compiler.CreateArtifactForFile(a.GetPath())
+	if err != nil {
+		return errors.Wrap(err, "Failed generating checksums for tree")
+	}
+	imageMetaTree := fmt.Sprintf("%s:%s", imagePrefix, a.GetFileName())
+	if opts, err := metaArchive.GenerateFinalImage(imageMetaTree, r.GetBackend(), false); err != nil {
+		return errors.Wrap(err, "Failed generating metadata tree"+opts.ImageName)
+	}
+	if r.PushImages {
+		if err := pushImage(r.GetBackend(), imageMetaTree, true); err != nil {
+			return errors.Wrapf(err, "Failed while pushing image: '%s'", imageMetaTree)
+		}
+	}
+	data, err := yaml.Marshal(serialized)
+	if err != nil {
+		return err
+	}
+	err = ioutil.WriteFile(repospec, data, os.ModePerm)
+	if err != nil {
+		return err
+	}
+
+	tempRepoFile := filepath.Join(metaDir, REPOSITORY_SPECFILE+".tar")
+	if err := helpers.Tar(repospec, tempRepoFile); err != nil {
+		return errors.Wrap(err, "Error met while archiving repository file")
+	}
+
+	a = compiler.NewPackageArtifact(tempRepoFile)
+	imageRepo := fmt.Sprintf("%s:%s", imagePrefix, REPOSITORY_SPECFILE)
+	if opts, err := a.GenerateFinalImage(imageRepo, r.GetBackend(), false); err != nil {
+		return errors.Wrap(err, "Failed generating repository image"+opts.ImageName)
+	}
+	if r.PushImages {
+		if err := pushImage(r.GetBackend(), imageRepo, true); err != nil {
+			return errors.Wrapf(err, "Failed while pushing image: '%s'", imageRepo)
+		}
+	}
+
+	bus.Manager.Publish(bus.EventRepositoryPostBuild, struct {
+		Repo LuetSystemRepository
+		Path string
+	}{
+		Repo: *r,
+		Path: imagePrefix,
+	})
+	return nil
+}
+
+// Write writes the repository metadata to the supplied destination
+func (r *LuetSystemRepository) Write(dst string, resetRevision, force bool) error {
+	switch r.GetType() {
+	case DiskRepositoryType, HttpRepositoryType:
+		return r.genLocalRepo(dst, resetRevision)
+	case DockerRepositoryType:
+		return r.genDockerRepo(dst, resetRevision, force)
+	}
+	return errors.New("invalid repository type")
+}
+
 func (r *LuetSystemRepository) Client() Client {
 	switch r.GetType() {
-	case "disk":
+	case DiskRepositoryType:
 		return client.NewLocalClient(client.RepoData{Urls: r.GetUrls()})
-	case "http":
+	case HttpRepositoryType:
 		return client.NewHttpClient(
 			client.RepoData{
 				Urls:           r.GetUrls(),
 				Authentication: r.GetAuthentication(),
 			})
-	}
 
+	case DockerRepositoryType:
+		return client.NewDockerClient(
+			client.RepoData{
+				Urls:           r.GetUrls(),
+				Authentication: r.GetAuthentication(),
+			})
+	}
 	return nil
 }
+
 func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 	var repoUpdated bool = false
 	var treefs, metafs string
@@ -548,7 +889,7 @@ func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 	Debug("Sync of the repository", r.Name, "in progress...")
 	c := r.Client()
 	if c == nil {
-		return nil, errors.New("No client could be generated from repository.")
+		return nil, errors.New("no client could be generated from repository")
 	}
 
 	// Retrieve remote repository.yaml for retrieve revision and date
@@ -562,7 +903,7 @@ func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 	if err != nil {
 		return nil, err
 	}
-	// Remove temporary file that contains repository.html.
+	// Remove temporary file that contains repository.yaml
 	// Example: /tmp/HttpClient236052003
 	defer os.RemoveAll(file)
 
@@ -592,15 +933,10 @@ func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 		if err != nil {
 			return nil, errors.Wrap(err, "Error met while creating tempdir for rootfs")
 		}
-		// Note: If we always remove them, later on, no other structure can access
-		// to the tree for e.g. to retrieve finalizers
-
 		metafs, err = config.LuetCfg.GetSystem().TempDir("metafs")
 		if err != nil {
 			return nil, errors.Wrap(err, "Error met whilte creating tempdir for metafs")
 		}
-		//defer os.RemoveAll(metafs)
-
 	}
 
 	// POST: treeFile and metaFile are present. I check this inside
@@ -611,17 +947,18 @@ func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 	if !repoUpdated {
 
 		// Get Tree
-		a := compiler.NewPackageArtifact(treeFile.GetFileName())
-		artifactTree, err := c.DownloadArtifact(a)
+		downloadedTreeFile, err := c.DownloadFile(treeFile.GetFileName())
 		if err != nil {
 			return nil, errors.Wrap(err, "While downloading "+treeFile.GetFileName())
 		}
-		defer os.Remove(artifactTree.GetPath())
+		defer os.Remove(downloadedTreeFile)
 
-		artifactTree.SetChecksums(treeFile.GetChecksums())
-		artifactTree.SetCompressionType(treeFile.GetCompressionType())
+		// Treat the file as artifact, in order to verify it
+		treeFileArtifact := compiler.NewPackageArtifact(downloadedTreeFile)
+		treeFileArtifact.SetChecksums(treeFile.GetChecksums())
+		treeFileArtifact.SetCompressionType(treeFile.GetCompressionType())
 
-		err = artifactTree.Verify()
+		err = treeFileArtifact.Verify()
 		if err != nil {
 			return nil, errors.Wrap(err, "Tree integrity check failure")
 		}
@@ -629,17 +966,17 @@ func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 		Debug("Tree tarball for the repository " + r.GetName() + " downloaded correctly.")
 
 		// Get Repository Metadata
-		a = compiler.NewPackageArtifact(metaFile.GetFileName())
-		artifactMeta, err := c.DownloadArtifact(a)
+		downloadedMeta, err := c.DownloadFile(metaFile.GetFileName())
 		if err != nil {
 			return nil, errors.Wrap(err, "While downloading "+metaFile.GetFileName())
 		}
-		defer os.Remove(artifactMeta.GetPath())
+		defer os.Remove(downloadedMeta)
 
-		artifactMeta.SetChecksums(metaFile.GetChecksums())
-		artifactMeta.SetCompressionType(metaFile.GetCompressionType())
+		metaFileArtifact := compiler.NewPackageArtifact(downloadedMeta)
+		metaFileArtifact.SetChecksums(metaFile.GetChecksums())
+		metaFileArtifact.SetCompressionType(metaFile.GetCompressionType())
 
-		err = artifactMeta.Verify()
+		err = metaFileArtifact.Verify()
 		if err != nil {
 			return nil, errors.Wrap(err, "Metadata integrity check failure")
 		}
@@ -659,7 +996,7 @@ func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 		}
 		Debug("Decompress tree of the repository " + r.Name + "...")
 
-		err = artifactTree.Unpack(treefs, true)
+		err = treeFileArtifact.Unpack(treefs, true)
 		if err != nil {
 			return nil, errors.Wrap(err, "Error met while unpacking tree")
 		}
@@ -667,7 +1004,7 @@ func (r *LuetSystemRepository) Sync(force bool) (Repository, error) {
 		// FIXME: It seems that tar with only one file doesn't create destination
 		//       directory. I create directory directly for now.
 		os.MkdirAll(metafs, os.ModePerm)
-		err = artifactMeta.Unpack(metafs, true)
+		err = metaFileArtifact.Unpack(metafs, true)
 		if err != nil {
 			return nil, errors.Wrap(err, "Error met while unpacking metadata")
 		}
@@ -829,13 +1166,11 @@ PACKAGE:
 				c, err := r.GetTree().GetDatabase().FindPackageCandidate(pack)
 				// If FindPackageCandidate returns the same package, it means it couldn't find one.
 				// Skip this repository and keep looking.
-				if c.String() == pack.String() {
+				if err != nil { //c.String() == pack.String() {
 					continue REPOSITORY
 				}
-				if err == nil {
-					matches = append(matches, c)
-					continue PACKAGE
-				}
+				matches = append(matches, c)
+				continue PACKAGE
 			} else {
 				// If it's not a selector, just append it
 				matches = append(matches, pack)

pkg/installer/repository_test.go

@@ -19,13 +19,16 @@ import (
 
 	//	. "github.com/mudler/luet/pkg/installer"
 
+	"fmt"
 	"io/ioutil"
 	"os"
+	"path/filepath"
 
 	"github.com/mudler/luet/pkg/compiler"
 	backend "github.com/mudler/luet/pkg/compiler/backend"
 	config "github.com/mudler/luet/pkg/config"
 	"github.com/mudler/luet/pkg/helpers"
+	"github.com/mudler/luet/pkg/installer"
 	. "github.com/mudler/luet/pkg/installer"
 	pkg "github.com/mudler/luet/pkg/package"
 	"github.com/mudler/luet/pkg/solver"
@@ -34,6 +37,18 @@ import (
 	. "github.com/onsi/gomega"
 )
 
+func dockerStubRepo(tmpdir, tree, image string, push, force bool) (installer.Repository, error) {
+	return GenerateRepository(
+		"test",
+		"description",
+		"docker",
+		[]string{image},
+		1,
+		tmpdir,
+		[]string{tree},
+		pkg.NewInMemoryDatabase(false), backend.NewSimpleDockerBackend(), image, push, force)
+}
+
 var _ = Describe("Repository", func() {
 	Context("Generation", func() {
 		It("Generate repository metadata", func() {
@@ -60,8 +75,8 @@ var _ = Describe("Repository", func() {
 			Expect(err).ToNot(HaveOccurred())
 			defer os.RemoveAll(tmpdir) // clean up
 
-			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "./generate.sh"}))
-			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2", "chmod +x generate.sh"}))
+			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "chmod +x generate.sh", "./generate.sh"}))
+			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2"}))
 
 			spec.SetOutputPath(tmpdir)
 			compiler.SetConcurrency(1)
@@ -84,13 +99,13 @@ var _ = Describe("Repository", func() {
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.package.tar"))).To(BeTrue())
 			Expect(helpers.Exists(spec.Rel("b-test-1.0.metadata.yaml"))).To(BeTrue())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/buildable"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, true)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).To(BeTrue())
@@ -133,8 +148,8 @@ var _ = Describe("Repository", func() {
 			Expect(err).ToNot(HaveOccurred())
 			defer os.RemoveAll(tmpdir) // clean up
 
-			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "./generate.sh"}))
-			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2", "chmod +x generate.sh"}))
+			Expect(spec.BuildSteps()).To(Equal([]string{"echo artifact5 > /test5", "echo artifact6 > /test6", "chmod +x generate.sh", "./generate.sh"}))
+			Expect(spec.GetPreBuildSteps()).To(Equal([]string{"echo foo > /test", "echo bar > /test2"}))
 
 			spec.SetOutputPath(tmpdir)
 			spec2.SetOutputPath(tmpdir)
@@ -166,13 +181,13 @@ var _ = Describe("Repository", func() {
 			Expect(helpers.Exists(spec2.Rel("alpine-seed-1.0.package.tar"))).To(BeTrue())
 			Expect(helpers.Exists(spec2.Rel("alpine-seed-1.0.metadata.yaml"))).To(BeTrue())
 
-			repo, err := GenerateRepository("test", "description", "disk", []string{tmpdir}, 1, tmpdir, []string{"../../tests/fixtures/buildable"}, pkg.NewInMemoryDatabase(false))
+			repo, err := stubRepo(tmpdir, "../../tests/fixtures/buildable")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(repo.GetName()).To(Equal("test"))
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
-			err = repo.Write(tmpdir, false)
+			err = repo.Write(tmpdir, false, true)
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).To(BeTrue())
@@ -216,6 +231,89 @@ urls:
 			Expect(matches).To(Equal([]PackageMatch{{Repo: repo1, Package: package1}}))
 
 		})
+	})
+	Context("Docker repository", func() {
+		repoImage := os.Getenv("UNIT_TEST_DOCKER_IMAGE_REPOSITORY")
 
+		BeforeEach(func() {
+			if repoImage == "" {
+				Skip("UNIT_TEST_DOCKER_IMAGE_REPOSITORY not specified")
+			}
+		})
+
+		It("generates images", func() {
+			b := backend.NewSimpleDockerBackend()
+			tmpdir, err := ioutil.TempDir("", "tree")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpdir) // clean up
+
+			generalRecipe := tree.NewCompilerRecipe(pkg.NewInMemoryDatabase(false))
+
+			err = generalRecipe.Load("../../tests/fixtures/buildable")
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(len(generalRecipe.GetDatabase().GetPackages())).To(Equal(3))
+
+			localcompiler := compiler.NewLuetCompiler(backend.NewSimpleDockerBackend(), generalRecipe.GetDatabase(), compiler.NewDefaultCompilerOptions(), solver.Options{Type: solver.SingleCoreSimple})
+
+			spec, err := localcompiler.FromPackage(&pkg.DefaultPackage{Name: "b", Category: "test", Version: "1.0"})
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(spec.GetPackage().GetPath()).ToNot(Equal(""))
+
+			tmpdir, err = ioutil.TempDir("", "tree")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(tmpdir) // clean up
+
+			spec.SetOutputPath(tmpdir)
+			localcompiler.SetConcurrency(1)
+
+			artifact, err := localcompiler.Compile(false, spec)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(helpers.Exists(artifact.GetPath())).To(BeTrue())
+			Expect(helpers.Untar(artifact.GetPath(), tmpdir, false)).ToNot(HaveOccurred())
+
+			Expect(helpers.Exists(spec.Rel("test5"))).To(BeTrue())
+			Expect(helpers.Exists(spec.Rel("test6"))).To(BeTrue())
+
+			repo, err := dockerStubRepo(tmpdir, "../../tests/fixtures/buildable", repoImage, true, true)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(repo.GetName()).To(Equal("test"))
+			Expect(helpers.Exists(spec.Rel(REPOSITORY_SPECFILE))).ToNot(BeTrue())
+			Expect(helpers.Exists(spec.Rel(TREE_TARBALL + ".gz"))).ToNot(BeTrue())
+			Expect(helpers.Exists(spec.Rel(REPOSITORY_METAFILE + ".tar"))).ToNot(BeTrue())
+			err = repo.Write(repoImage, false, true)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(b.ImageAvailable(fmt.Sprintf("%s:%s", repoImage, "tree.tar.gz"))).To(BeTrue())
+			Expect(b.ImageAvailable(fmt.Sprintf("%s:%s", repoImage, "repository.meta.yaml.tar"))).To(BeTrue())
+			Expect(b.ImageAvailable(fmt.Sprintf("%s:%s", repoImage, "repository.yaml"))).To(BeTrue())
+			Expect(b.ImageAvailable(fmt.Sprintf("%s:%s", repoImage, "b-test-1.0"))).To(BeTrue())
+
+			extracted, err := ioutil.TempDir("", "extracted")
+			Expect(err).ToNot(HaveOccurred())
+			defer os.RemoveAll(extracted) // clean up
+
+			c := repo.Client()
+
+			f, err := c.DownloadFile("repository.yaml")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(helpers.Read(f)).To(ContainSubstring("name: test"))
+
+			a, err := c.DownloadArtifact(&compiler.PackageArtifact{
+				Path: "test.tar",
+				CompileSpec: &compiler.LuetCompilationSpec{
+					Package: &pkg.DefaultPackage{
+						Name:     "b",
+						Category: "test",
+						Version:  "1.0",
+					},
+				},
+			})
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(a.Unpack(extracted, false)).ToNot(HaveOccurred())
+			Expect(helpers.Read(filepath.Join(extracted, "test6"))).To(Equal("artifact6\n"))
+		})
 	})
 })

pkg/installer/system.go

@@ -1,12 +1,11 @@
 package installer
 
 import (
-	. "github.com/mudler/luet/pkg/logger"
-
+	"github.com/hashicorp/go-multierror"
 	"github.com/mudler/luet/pkg/helpers"
+	. "github.com/mudler/luet/pkg/logger"
 	pkg "github.com/mudler/luet/pkg/package"
 	"github.com/mudler/luet/pkg/tree"
-	"github.com/pkg/errors"
 )
 
 type System struct {
@@ -20,28 +19,35 @@ func (s *System) World() (pkg.Packages, error) {
 
 type templatedata map[string]interface{}
 
-func (s *System) ExecuteFinalizers(packs []pkg.Package, force bool) error {
+func (s *System) ExecuteFinalizers(packs []pkg.Package) error {
+	var errs error
 	executedFinalizer := map[string]bool{}
 	for _, p := range packs {
 		if helpers.Exists(p.Rel(tree.FinalizerFile)) {
 			out, err := helpers.RenderFiles(p.Rel(tree.FinalizerFile), p.Rel(tree.DefinitionFile), "")
-			if err != nil && !force {
-				return errors.Wrap(err, "reading file "+p.Rel(tree.FinalizerFile))
+			if err != nil {
+				Warning("Failed rendering finalizer for ", p.HumanReadableString(), err.Error())
+				errs = multierror.Append(errs, err)
+				continue
 			}
 
 			if _, exists := executedFinalizer[p.GetFingerPrint()]; !exists {
+				executedFinalizer[p.GetFingerPrint()] = true
 				Info("Executing finalizer for " + p.HumanReadableString())
 				finalizer, err := NewLuetFinalizerFromYaml([]byte(out))
-				if err != nil && !force {
-					return errors.Wrap(err, "Error reading finalizer "+p.Rel(tree.FinalizerFile))
+				if err != nil {
+					Warning("Failed reading finalizer for ", p.HumanReadableString(), err.Error())
+					errs = multierror.Append(errs, err)
+					continue
 				}
 				err = finalizer.RunInstall(s)
-				if err != nil && !force {
-					return errors.Wrap(err, "Error executing install finalizer "+p.Rel(tree.FinalizerFile))
+				if err != nil {
+					Warning("Failed running finalizer for ", p.HumanReadableString(), err.Error())
+					errs = multierror.Append(errs, err)
+					continue
 				}
-				executedFinalizer[p.GetFingerPrint()] = true
 			}
 		}
 	}
-	return nil
+	return errs
 }

pkg/logger/logger.go

@@ -3,7 +3,9 @@ package logger
 import (
 	"fmt"
 	"os"
+	"path"
 	"regexp"
+	"runtime"
 	"strings"
 
 	. "github.com/mudler/luet/pkg/config"
@@ -196,7 +198,7 @@ func msg(level string, withoutColor bool, msg ...interface{}) {
 	} else {
 		switch level {
 		case "warning":
-			levelMsg = Bold(Yellow(":construction: " + message)).BgBlack().String()
+			levelMsg = Yellow(":construction: warning" + message).BgBlack().String()
 		case "debug":
 			levelMsg = White(message).BgBlack().String()
 		case "info":
@@ -228,6 +230,11 @@ func Warning(mess ...interface{}) {
 }
 
 func Debug(mess ...interface{}) {
+	pc, file, line, ok := runtime.Caller(1)
+	if ok {
+		mess = append([]interface{}{fmt.Sprintf("DEBUG (%s:#%d:%v)",
+			path.Base(file), line, runtime.FuncForPC(pc).Name())}, mess...)
+	}
 	msg("debug", false, mess...)
 }
 

pkg/package/database.go

@@ -28,6 +28,9 @@ type PackageDatabase interface {
 }
 
 type PackageSet interface {
+	Clone(PackageDatabase) error
+	Copy() (PackageDatabase, error)
+
 	GetRevdeps(p Package) (Packages, error)
 	GetPackages() []string //Ids
 	CreatePackage(pkg Package) (string, error)

pkg/package/database_boltdb.go

@@ -47,6 +47,14 @@ func NewBoltDatabase(path string) PackageDatabase {
 	return &BoltDatabase{Path: path, ProvidesDatabase: map[string]map[string]Package{}}
 }
 
+func (db *BoltDatabase) Clone(to PackageDatabase) error {
+	return clone(db, to)
+}
+
+func (db *BoltDatabase) Copy() (PackageDatabase, error) {
+	return copy(db)
+}
+
 func (db *BoltDatabase) Get(s string) (string, error) {
 	bolt, err := storm.Open(db.Path, storm.BoltOptions(0600, &bbolt.Options{Timeout: 30 * time.Second}))
 	if err != nil {
@@ -90,9 +98,9 @@ func (db *BoltDatabase) Retrieve(ID string) ([]byte, error) {
 // TODO: Have a memory instance for boltdb, so we don't compute each time we get called
 // as this is REALLY expensive. But we don't perform usually those operations in a file db.
 func (db *BoltDatabase) GetRevdeps(p Package) (Packages, error) {
-	memory := NewInMemoryDatabase(false)
-	for _, p := range db.World() {
-		memory.CreatePackage(p)
+	memory, err := db.Copy()
+	if err != nil {
+		return nil, errors.New("Failed copying bolt db to memory")
 	}
 	return memory.GetRevdeps(p)
 }
@@ -341,17 +349,18 @@ func (db *BoltDatabase) FindPackageCandidate(p Package) (Package, error) {
 
 	required, err := db.FindPackage(p)
 	if err != nil {
-
+		err = nil
 		//	return nil, errors.Wrap(err, "Couldn't find required package in db definition")
 		packages, err := p.Expand(db)
 		//	Info("Expanded", packages, err)
 		if err != nil || len(packages) == 0 {
 			required = p
+			err = errors.Wrap(err, "Candidate not found")
 		} else {
 			required = packages.Best(nil)
 
 		}
-		return required, nil
+		return required, err
 		//required = &DefaultPackage{Name: "test"}
 	}
 
@@ -362,10 +371,22 @@ func (db *BoltDatabase) FindPackageCandidate(p Package) (Package, error) {
 // FindPackages return the list of the packages beloging to cat/name  (any versions in requested range)
 // FIXME: Optimize, see inmemorydb
 func (db *BoltDatabase) FindPackages(p Package) (Packages, error) {
+	if !p.IsSelector() {
+		pack, err := db.FindPackage(p)
+		if err != nil {
+			return []Package{}, err
+		}
+		return []Package{pack}, nil
+	}
+
 	// Provides: Treat as the replaced package here
 	if provided, err := db.getProvide(p); err == nil {
 		p = provided
+		if !provided.IsSelector() {
+			return Packages{provided}, nil
+		}
 	}
+
 	var versionsInWorld []Package
 	for _, w := range db.World() {
 		if w.GetName() != p.GetName() || w.GetCategory() != p.GetCategory() {

pkg/package/database_boltdb_test.go

@@ -81,7 +81,6 @@ var _ = Describe("BoltDB Database", func() {
 		})
 
 		It("Find best package candidate", func() {
-			db := NewInMemoryDatabase(false)
 			a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
 			a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
 			a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
@@ -102,7 +101,6 @@ var _ = Describe("BoltDB Database", func() {
 		})
 
 		It("Find specific package candidate", func() {
-			db := NewInMemoryDatabase(false)
 			a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
 			a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
 			a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
@@ -123,7 +121,6 @@ var _ = Describe("BoltDB Database", func() {
 		})
 
 		It("Provides replaces definitions", func() {
-			db := NewInMemoryDatabase(false)
 			a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
 			a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
 			a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
@@ -147,6 +144,144 @@ var _ = Describe("BoltDB Database", func() {
 			Expect(pack).To(Equal(a3))
 		})
 
+		Context("Provides", func() {
+
+			It("replaces definitions", func() {
+				a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				a3.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}})
+				Expect(a3.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}}))
+
+				_, err := db.CreatePackage(a)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a3)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				pack, err := db.FindPackage(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(pack).To(Equal(a3))
+			})
+
+			It("replaces definitions", func() {
+				a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				a3.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}})
+				Expect(a3.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}}))
+
+				_, err := db.CreatePackage(a)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a3)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(a3))
+			})
+
+			It("replaces definitions", func() {
+				a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+				Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+				_, err := db.CreatePackage(a)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(z)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(z))
+			})
+
+			It("replaces definitions of unexisting packages", func() {
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+				Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+				_, err := db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(z)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(z))
+			})
+
+			It("replaces definitions of a required package", func() {
+
+				c := NewPackage("C", "1.1", []*DefaultPackage{{Name: "A", Category: "", Version: ">=0"}}, []*DefaultPackage{})
+				z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+				Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+				_, err := db.CreatePackage(z)
+				Expect(err).ToNot(HaveOccurred())
+				_, err = db.CreatePackage(c)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(z))
+			})
+
+			When("Searching with selectors", func() {
+				It("replaces definitions of a required package", func() {
+
+					c := NewPackage("C", "1.1", []*DefaultPackage{{Name: "A", Category: "", Version: ">=0"}}, []*DefaultPackage{})
+					z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+					z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+					Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+					_, err := db.CreatePackage(z)
+					Expect(err).ToNot(HaveOccurred())
+					_, err = db.CreatePackage(c)
+					Expect(err).ToNot(HaveOccurred())
+
+					s := NewPackage("A", ">=1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+					packs, err := db.FindPackages(s)
+					Expect(err).ToNot(HaveOccurred())
+					Expect(packs).To(ContainElement(z))
+				})
+			})
+
+		})
+
 	})
 
 })

pkg/package/database_common.go

@@ -0,0 +1,38 @@
+// Copyright © 2020 Ettore Di Giacinto <mudler@gentoo.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package pkg
+
+import "github.com/pkg/errors"
+
+func clone(src, dst PackageDatabase) error {
+	for _, i := range src.World() {
+		_, err := dst.CreatePackage(i)
+		if err != nil {
+			return errors.Wrap(err, "Failed create package "+i.HumanReadableString())
+		}
+	}
+	return nil
+}
+
+func copy(src PackageDatabase) (PackageDatabase, error) {
+	dst := NewInMemoryDatabase(false)
+
+	if err := clone(src, dst); err != nil {
+		return dst, errors.Wrap(err, "Failed create temporary in-memory db")
+	}
+
+	return dst, nil
+}

pkg/package/database_mem.go

@@ -41,6 +41,7 @@ type InMemoryDatabase struct {
 	CacheNoVersion   map[string]map[string]interface{}
 	ProvidesDatabase map[string]map[string]Package
 	RevDepsDatabase  map[string]map[string]Package
+	cached           map[string]interface{}
 }
 
 func NewInMemoryDatabase(singleton bool) PackageDatabase {
@@ -53,6 +54,7 @@ func NewInMemoryDatabase(singleton bool) PackageDatabase {
 			CacheNoVersion:   map[string]map[string]interface{}{},
 			ProvidesDatabase: map[string]map[string]Package{},
 			RevDepsDatabase:  map[string]map[string]Package{},
+			cached:           map[string]interface{}{},
 		}
 	}
 	return DBInMemoryInstance
@@ -197,6 +199,11 @@ func (db *InMemoryDatabase) populateCaches(p Package) {
 
 	// Create extra cache between package -> []versions
 	db.Lock()
+	if _, ok := db.cached[p.GetFingerPrint()]; ok {
+		db.Unlock()
+		return
+	}
+	db.cached[p.GetFingerPrint()] = nil
 
 	// Provides: Store package provides, we will reuse this when walking deps
 	for _, provide := range pd.Provides {
@@ -213,6 +220,25 @@ func (db *InMemoryDatabase) populateCaches(p Package) {
 		db.CacheNoVersion[p.GetPackageName()] = make(map[string]interface{})
 	}
 	db.CacheNoVersion[p.GetPackageName()][p.GetVersion()] = nil
+
+	// Updating Revdeps
+	// Given that when we populate the cache we don't have the full db at hand
+	// We cycle over reverse dependency of a package to update their entry if they are matching
+	// the version selector
+	toUpdate, ok := db.RevDepsDatabase[pd.GetPackageName()]
+	if ok {
+		for _, pp := range toUpdate {
+			for _, re := range pp.GetRequires() {
+				if match, _ := pd.VersionMatchSelector(re.GetVersion(), nil); match {
+					_, ok = db.RevDepsDatabase[pd.GetFingerPrint()]
+					if !ok {
+						db.RevDepsDatabase[pd.GetFingerPrint()] = make(map[string]Package)
+					}
+					db.RevDepsDatabase[pd.GetFingerPrint()][pp.GetFingerPrint()] = pp
+				}
+			}
+		}
+	}
 	db.Unlock()
 
 	for _, re := range pd.GetRequires() {
@@ -225,12 +251,23 @@ func (db *InMemoryDatabase) populateCaches(p Package) {
 				db.RevDepsDatabase[pa.GetFingerPrint()] = make(map[string]Package)
 			}
 			db.RevDepsDatabase[pa.GetFingerPrint()][pd.GetFingerPrint()] = pd
+			_, ok = db.RevDepsDatabase[pa.GetPackageName()]
+			if !ok {
+				db.RevDepsDatabase[pa.GetPackageName()] = make(map[string]Package)
+			}
+			db.RevDepsDatabase[pa.GetPackageName()][pd.GetPackageName()] = pd
 		}
 		_, ok := db.RevDepsDatabase[re.GetFingerPrint()]
 		if !ok {
 			db.RevDepsDatabase[re.GetFingerPrint()] = make(map[string]Package)
 		}
 		db.RevDepsDatabase[re.GetFingerPrint()][pd.GetFingerPrint()] = pd
+		_, ok = db.RevDepsDatabase[re.GetPackageName()]
+		if !ok {
+			db.RevDepsDatabase[re.GetPackageName()] = make(map[string]Package)
+		}
+		db.RevDepsDatabase[re.GetPackageName()][pd.GetPackageName()] = pd
+
 		db.Unlock()
 	}
 }
@@ -270,6 +307,14 @@ func (db *InMemoryDatabase) getProvide(p Package) (Package, error) {
 	return db.FindPackage(pa)
 }
 
+func (db *InMemoryDatabase) Clone(to PackageDatabase) error {
+	return clone(db, to)
+}
+
+func (db *InMemoryDatabase) Copy() (PackageDatabase, error) {
+	return copy(db)
+}
+
 func (db *InMemoryDatabase) encodePackage(p Package) (string, string, error) {
 	pd, ok := p.(*DefaultPackage)
 	if !ok {
@@ -330,6 +375,9 @@ func (db *InMemoryDatabase) FindPackages(p Package) (Packages, error) {
 	// Provides: Treat as the replaced package here
 	if provided, err := db.getProvide(p); err == nil {
 		p = provided
+		if !provided.IsSelector() {
+			return Packages{provided}, nil
+		}
 	}
 
 	db.Lock()
@@ -431,16 +479,17 @@ func (db *InMemoryDatabase) FindPackageCandidate(p Package) (Package, error) {
 
 	required, err := db.FindPackage(p)
 	if err != nil {
+		err = nil
 		//	return nil, errors.Wrap(err, "Couldn't find required package in db definition")
 		packages, err := p.Expand(db)
 		//	Info("Expanded", packages, err)
 		if err != nil || len(packages) == 0 {
 			required = p
+			err = errors.Wrap(err, "Candidate not found")
 		} else {
 			required = packages.Best(nil)
-
 		}
-		return required, nil
+		return required, err
 		//required = &DefaultPackage{Name: "test"}
 	}
 

pkg/package/database_mem_test.go

@@ -98,30 +98,150 @@ var _ = Describe("Database", func() {
 
 		})
 
-		It("Provides replaces definitions", func() {
-			db := NewInMemoryDatabase(false)
-			a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
-			a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
-			a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+		Context("Provides", func() {
 
-			a3.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}})
-			Expect(a3.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}}))
+			It("replaces definitions", func() {
+				db := NewInMemoryDatabase(false)
+				a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
 
-			_, err := db.CreatePackage(a)
-			Expect(err).ToNot(HaveOccurred())
+				a3.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}})
+				Expect(a3.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}}))
 
-			_, err = db.CreatePackage(a1)
-			Expect(err).ToNot(HaveOccurred())
+				_, err := db.CreatePackage(a)
+				Expect(err).ToNot(HaveOccurred())
 
-			_, err = db.CreatePackage(a3)
-			Expect(err).ToNot(HaveOccurred())
+				_, err = db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
 
-			s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+				_, err = db.CreatePackage(a3)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				pack, err := db.FindPackage(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(pack).To(Equal(a3))
+			})
+
+			It("replaces definitions", func() {
+				db := NewInMemoryDatabase(false)
+				a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				a3 := NewPackage("A", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				a3.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}})
+				Expect(a3.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: "1.0"}}))
+
+				_, err := db.CreatePackage(a)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a3)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(a3))
+			})
+
+			It("replaces definitions", func() {
+				db := NewInMemoryDatabase(false)
+				a := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+				Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+				_, err := db.CreatePackage(a)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(z)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(z))
+			})
+
+			It("replaces definitions of unexisting packages", func() {
+				db := NewInMemoryDatabase(false)
+				a1 := NewPackage("A", "1.1", []*DefaultPackage{}, []*DefaultPackage{})
+				z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+				Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+				_, err := db.CreatePackage(a1)
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = db.CreatePackage(z)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(z))
+			})
+
+			It("replaces definitions of a required package", func() {
+				db := NewInMemoryDatabase(false)
+
+				c := NewPackage("C", "1.1", []*DefaultPackage{{Name: "A", Category: "", Version: ">=0"}}, []*DefaultPackage{})
+				z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+				z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+				Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+				_, err := db.CreatePackage(z)
+				Expect(err).ToNot(HaveOccurred())
+				_, err = db.CreatePackage(c)
+				Expect(err).ToNot(HaveOccurred())
+
+				s := NewPackage("A", "1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+				packs, err := db.FindPackages(s)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(packs).To(ContainElement(z))
+			})
+
+			When("Searching with selectors", func() {
+				It("replaces definitions of a required package", func() {
+					db := NewInMemoryDatabase(false)
+
+					c := NewPackage("C", "1.1", []*DefaultPackage{{Name: "A", Category: "", Version: ">=0"}}, []*DefaultPackage{})
+					z := NewPackage("Z", "1.3", []*DefaultPackage{}, []*DefaultPackage{})
+
+					z.SetProvides([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}})
+					Expect(z.GetProvides()).To(Equal([]*DefaultPackage{{Name: "A", Category: "", Version: ">=1.0"}}))
+
+					_, err := db.CreatePackage(z)
+					Expect(err).ToNot(HaveOccurred())
+					_, err = db.CreatePackage(c)
+					Expect(err).ToNot(HaveOccurred())
+
+					s := NewPackage("A", ">=1.0", []*DefaultPackage{}, []*DefaultPackage{})
+
+					packs, err := db.FindPackages(s)
+					Expect(err).ToNot(HaveOccurred())
+					Expect(packs).To(ContainElement(z))
+				})
+			})
 
-			pack, err := db.FindPackage(s)
-			Expect(err).ToNot(HaveOccurred())
-			Expect(pack).To(Equal(a3))
 		})
+
 	})
 
 })

pkg/package/package.go

@@ -46,6 +46,7 @@ type Package interface {
 
 	GetFingerPrint() string
 	GetPackageName() string
+	GetPackageImageName() string
 	Requires([]*DefaultPackage) Package
 	Conflicts([]*DefaultPackage) Package
 	Revdeps(PackageDatabase) Packages
@@ -288,6 +289,10 @@ func (p *DefaultPackage) GetPackageName() string {
 	return fmt.Sprintf("%s-%s", p.Name, p.Category)
 }
 
+func (p *DefaultPackage) GetPackageImageName() string {
+	return fmt.Sprintf("%s-%s:%s", p.Name, p.Category, p.Version)
+}
+
 // GetBuildTimestamp returns the package build timestamp
 func (p *DefaultPackage) GetBuildTimestamp() string {
 	return p.BuildTimestamp

pkg/solver/parallel.go

@@ -76,7 +76,7 @@ func (s *Parallel) noRulesInstalled() bool {
 	return true
 }
 
-func (s *Parallel) buildParallelFormula(formulas []bf.Formula, packages pkg.Packages) (bf.Formula, error) {
+func (s *Parallel) buildParallelFormula(db pkg.PackageDatabase, formulas []bf.Formula, packages pkg.Packages) (bf.Formula, error) {
 	var wg = new(sync.WaitGroup)
 	var wg2 = new(sync.WaitGroup)
 
@@ -87,7 +87,7 @@ func (s *Parallel) buildParallelFormula(formulas []bf.Formula, packages pkg.Pack
 		go func(wg *sync.WaitGroup, c <-chan pkg.Package) {
 			defer wg.Done()
 			for p := range c {
-				solvable, err := p.BuildFormula(s.DefinitionDatabase, s.ParallelDatabase)
+				solvable, err := p.BuildFormula(db, s.ParallelDatabase)
 				if err != nil {
 					panic(err)
 				}
@@ -126,13 +126,13 @@ func (s *Parallel) BuildInstalled() (bf.Formula, error) {
 	var packages pkg.Packages
 	for _, p := range s.Installed() {
 		packages = append(packages, p)
-		for _, dep := range p.Related(s.DefinitionDatabase) {
+		for _, dep := range p.Related(s.InstalledDatabase) {
 			packages = append(packages, dep)
 		}
 
 	}
 
-	return s.buildParallelFormula(formulas, packages)
+	return s.buildParallelFormula(s.InstalledDatabase, formulas, packages)
 }
 
 // BuildWorld builds the formula which olds the requirements from the package definitions
@@ -148,7 +148,7 @@ func (s *Parallel) BuildWorld(includeInstalled bool) (bf.Formula, error) {
 		//f = bf.And(f, solvable)
 		formulas = append(formulas, solvable)
 	}
-	return s.buildParallelFormula(formulas, s.World())
+	return s.buildParallelFormula(s.DefinitionDatabase, formulas, s.World())
 }
 
 // BuildWorld builds the formula which olds the requirements from the package definitions
@@ -200,7 +200,7 @@ func (s *Parallel) BuildPartialWorld(includeInstalled bool) (bf.Formula, error)
 	close(results)
 	wg2.Wait()
 
-	return s.buildParallelFormula(formulas, packages)
+	return s.buildParallelFormula(s.DefinitionDatabase, formulas, packages)
 
 	//return s.buildParallelFormula(formulas, s.World())
 }
@@ -446,10 +446,9 @@ func (s *Parallel) UpgradeUniverse(dropremoved bool) (pkg.Packages, PackagesAsse
 	removed := pkg.Packages{}
 
 	// TODO: this is memory expensive, we need to optimize this
-	universe := pkg.NewInMemoryDatabase(false)
-
-	for _, p := range s.DefinitionDatabase.World() {
-		universe.CreatePackage(p)
+	universe, err := s.DefinitionDatabase.Copy()
+	if err != nil {
+		return nil, nil, errors.Wrap(err, "couldn't build world copy")
 	}
 	for _, p := range s.Installed() {
 		universe.CreatePackage(p)
@@ -471,23 +470,22 @@ func (s *Parallel) UpgradeUniverse(dropremoved bool) (pkg.Packages, PackagesAsse
 		go func(wg *sync.WaitGroup, c <-chan pkg.Package) {
 			defer wg.Done()
 			for p := range c {
-				available, err := universe.FindPackageVersions(p)
-				if err != nil {
-					removed = append(removed, p) /// FIXME: Racy
-				}
-				if len(available) == 0 {
+				available, err := s.DefinitionDatabase.FindPackageVersions(p)
+				if len(available) == 0 || err != nil {
+					removed = append(removed, p)
 					continue
 				}
 
 				bestmatch := available.Best(nil)
 				// Found a better version available
 				if !bestmatch.Matches(p) {
-					encodedP, _ := p.Encode(universe)
-					P := bf.Var(encodedP)
-					results <- bf.And(bf.Not(P), r)
-					encodedP, _ = bestmatch.Encode(universe)
-					P = bf.Var(encodedP)
-					results <- bf.And(P, r)
+					oldP, _ := p.Encode(universe)
+					toreplaceP := bf.Var(oldP)
+					best, _ := bestmatch.Encode(universe)
+					toUpgrade := bf.Var(best)
+
+					solvablenew, _ := bestmatch.BuildFormula(s.DefinitionDatabase, s.ParallelDatabase)
+					results <- bf.And(bf.Not(toreplaceP), bf.And(append(solvablenew, toUpgrade)...))
 				}
 			}
 		}(wg, all)
@@ -514,8 +512,7 @@ func (s *Parallel) UpgradeUniverse(dropremoved bool) (pkg.Packages, PackagesAsse
 	// Treat removed packages from universe as marked for deletion
 	if dropremoved {
 
-		// SAT encode the clauses against the world
-		for _, p := range removed {
+		for _, p := range removed.Unique() {
 			encodedP, err := p.Encode(universe)
 			if err != nil {
 				return nil, nil, errors.Wrap(err, "couldn't encode package")
@@ -560,9 +557,9 @@ func (s *Parallel) Upgrade(checkconflicts, full bool) (pkg.Packages, PackagesAss
 	toInstall := pkg.Packages{}
 
 	// we do this in memory so we take into account of provides
-	universe := pkg.NewInMemoryDatabase(false)
-	for _, p := range s.DefinitionDatabase.World() {
-		universe.CreatePackage(p)
+	universe, err := s.DefinitionDatabase.Copy()
+	if err != nil {
+		return nil, nil, errors.Wrap(err, "Could not copy def db")
 	}
 
 	installedcopy := pkg.NewInMemoryDatabase(false)
@@ -699,7 +696,7 @@ func (s *Parallel) Uninstall(checkconflicts, full bool, packs ...pkg.Package) (p
 		}
 	}
 
-	s2 := &Parallel{Concurrency: s.Concurrency, InstalledDatabase: pkg.NewInMemoryDatabase(false), DefinitionDatabase: s.DefinitionDatabase, ParallelDatabase: pkg.NewInMemoryDatabase(false)}
+	s2 := &Parallel{Concurrency: s.Concurrency, InstalledDatabase: pkg.NewInMemoryDatabase(false), DefinitionDatabase: s.InstalledDatabase, ParallelDatabase: pkg.NewInMemoryDatabase(false)}
 	s2.SetResolver(s.Resolver)
 	// Get the requirements to install the candidate
 	asserts, err := s2.Install(toRemove)
@@ -744,7 +741,7 @@ func (s *Parallel) Uninstall(checkconflicts, full bool, packs ...pkg.Package) (p
 func (s *Parallel) BuildFormula() (bf.Formula, error) {
 	var formulas []bf.Formula
 
-	r, err := s.BuildPartialWorld(false)
+	r, err := s.BuildWorld(false)
 	if err != nil {
 		return nil, err
 	}

pkg/solver/parallel_test.go

@@ -111,7 +111,7 @@ var _ = Describe("Parallel", func() {
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: B, Value: true}))
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: D, Value: true}))
 
-			Expect(len(solution)).To(Equal(3))
+			Expect(len(solution)).To(Equal(5))
 		})
 
 		It("Solves correctly if the selected package to install has requirements", func() {
@@ -401,7 +401,7 @@ var _ = Describe("Parallel", func() {
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: D, Value: false}))
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: E, Value: true}))
 
-			Expect(len(solution)).To(Equal(3))
+			Expect(len(solution)).To(Equal(4))
 			Expect(err).ToNot(HaveOccurred())
 		})
 
@@ -529,7 +529,7 @@ var _ = Describe("Parallel", func() {
 			Expect(solution).To(ContainElement(PackageAssert{Package: D1, Value: false}))
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: E, Value: true}))
 
-			Expect(len(solution)).To(Equal(5))
+			Expect(len(solution)).To(Equal(6))
 			Expect(err).ToNot(HaveOccurred())
 		})
 
@@ -570,7 +570,7 @@ var _ = Describe("Parallel", func() {
 			Expect(solution).To(ContainElement(PackageAssert{Package: D1, Value: false}))
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: E, Value: true}))
 
-			Expect(len(solution)).To(Equal(5))
+			Expect(len(solution)).To(Equal(6))
 			Expect(err).ToNot(HaveOccurred())
 		})
 
@@ -671,8 +671,9 @@ var _ = Describe("Parallel", func() {
 				Expect(err).ToNot(HaveOccurred())
 
 				Expect(solution).To(ContainElement(A))
+				Expect(solution).To(ContainElement(B))
 
-				Expect(len(solution)).To(Equal(1))
+				Expect(len(solution)).To(Equal(2))
 			})
 
 			It("Uninstalls complex packages correctly, even if shared deps are required by system packages", func() {
@@ -1285,12 +1286,172 @@ var _ = Describe("Parallel", func() {
 			Expect(uninstall[0].GetName()).To(Equal("a"))
 			Expect(uninstall[0].GetVersion()).To(Equal("1.1"))
 
-			Expect(solution).To(ContainElement(PackageAssert{Package: A1, Value: true}))
-			Expect(solution).To(ContainElement(PackageAssert{Package: B, Value: true}))
-			Expect(solution).To(ContainElement(PackageAssert{Package: C, Value: false}))
-			Expect(solution).To(ContainElement(PackageAssert{Package: A, Value: false}))
+			Expect(solution).ToNot(ContainElement(PackageAssert{Package: C, Value: true}))
+			Expect(solution).ToNot(ContainElement(PackageAssert{Package: A, Value: true}))
 
-			Expect(len(solution)).To(Equal(4))
+			Expect(len(solution)).To(Equal(3))
+		})
+
+		It("UpgradeUniverse upgrades correctly", func() {
+
+			D := pkg.NewPackage("d", "1.5", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			D.SetCategory("test")
+			C = pkg.NewPackage("c", "1.5", []*pkg.DefaultPackage{
+				&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"},
+				&pkg.DefaultPackage{Name: "d", Version: ">=1.0", Category: "test"},
+			}, []*pkg.DefaultPackage{})
+			C.SetCategory("test")
+			C1 := pkg.NewPackage("c", "1.6", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			C1.SetCategory("test")
+			B = pkg.NewPackage("b", "1.0", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
+			B.SetCategory("test")
+
+			B1 := pkg.NewPackage("b", "1.1", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "c", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			B1.SetCategory("test")
+			A = pkg.NewPackage("a", "1.1", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "b", Version: "1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			A.SetCategory("test")
+			A1 = pkg.NewPackage("a", "1.2", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "b", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			A1.SetCategory("test")
+
+			for _, p := range []pkg.Package{A1, B, B1, C, C1, D} {
+				_, err := dbDefinitions.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+
+			for _, p := range []pkg.Package{A, B, D} {
+				_, err := dbInstalled.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+			uninstall, solution, err := s.UpgradeUniverse(false)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(len(uninstall)).To(Equal(2))
+			Expect(uninstall).To(ContainElement(A))
+			Expect(uninstall).To(ContainElement(B))
+
+			Expect(solution).To(ContainElement(PackageAssert{Package: A1, Value: true}))
+			Expect(solution).To(ContainElement(PackageAssert{Package: B1, Value: true}))
+			Expect(solution).To(ContainElement(PackageAssert{Package: C1, Value: true}))
+
+			Expect(len(solution)).To(Equal(6))
+		})
+
+		It("UpgradeUniverse upgrades correctly", func() {
+
+			D := pkg.NewPackage("d", "1.5", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			D.SetCategory("test")
+			D1 := pkg.NewPackage("d", "1.6", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			D1.SetCategory("test")
+			C = pkg.NewPackage("c", "1.5", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
+			C.SetCategory("test")
+			B = pkg.NewPackage("b", "1.0", []*pkg.DefaultPackage{
+				&pkg.DefaultPackage{Name: "c", Version: ">=1.0", Category: "test"},
+			}, []*pkg.DefaultPackage{})
+			B.SetCategory("test")
+
+			A = pkg.NewPackage("a", "1.1", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "b", Version: "1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			A.SetCategory("test")
+
+			for _, p := range []pkg.Package{A, B, C, D, D1} {
+				_, err := dbDefinitions.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+
+			for _, p := range []pkg.Package{A, B, C, D} {
+				_, err := dbInstalled.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+			uninstall, solution, err := s.UpgradeUniverse(false)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(len(uninstall)).To(Equal(1))
+			Expect(uninstall[0].GetName()).To(Equal("d"))
+			Expect(uninstall[0].GetVersion()).To(Equal("1.5"))
+
+			Expect(solution).To(ContainElement(PackageAssert{Package: D1, Value: true}))
+
+			Expect(len(solution)).To(Equal(3))
+		})
+
+		It("Upgrade upgrades correctly", func() {
+
+			D := pkg.NewPackage("d", "1.5", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			D.SetCategory("test")
+			C = pkg.NewPackage("c", "1.5", []*pkg.DefaultPackage{
+				&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"},
+				&pkg.DefaultPackage{Name: "d", Version: ">=1.0", Category: "test"},
+			}, []*pkg.DefaultPackage{})
+			C.SetCategory("test")
+			C1 := pkg.NewPackage("c", "1.6", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			C1.SetCategory("test")
+			B = pkg.NewPackage("b", "1.0", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
+			B.SetCategory("test")
+
+			B1 := pkg.NewPackage("b", "1.1", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "c", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			B1.SetCategory("test")
+			A = pkg.NewPackage("a", "1.1", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "b", Version: "1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			A.SetCategory("test")
+			A1 = pkg.NewPackage("a", "1.2", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "b", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			A1.SetCategory("test")
+
+			for _, p := range []pkg.Package{A1, B, B1, C, C1, D} {
+				_, err := dbDefinitions.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+
+			for _, p := range []pkg.Package{A, B, D} {
+				_, err := dbInstalled.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+			uninstall, solution, err := s.Upgrade(false, false)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(len(uninstall)).To(Equal(2))
+			Expect(uninstall).To(ContainElement(A))
+			Expect(uninstall).To(ContainElement(B))
+
+			Expect(solution).To(ContainElement(PackageAssert{Package: A1, Value: true}))
+			Expect(solution).To(ContainElement(PackageAssert{Package: B1, Value: true}))
+			Expect(solution).To(ContainElement(PackageAssert{Package: C1, Value: true}))
+
+			Expect(len(solution)).To(Equal(6))
+		})
+
+		It("Upgrade upgrades correctly", func() {
+
+			D := pkg.NewPackage("d", "1.5", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			D.SetCategory("test")
+			D1 := pkg.NewPackage("d", "1.6", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			D1.SetCategory("test")
+			C = pkg.NewPackage("c", "1.5", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
+			C.SetCategory("test")
+			B = pkg.NewPackage("b", "1.0", []*pkg.DefaultPackage{
+				&pkg.DefaultPackage{Name: "c", Version: ">=1.0", Category: "test"},
+			}, []*pkg.DefaultPackage{})
+			B.SetCategory("test")
+
+			A = pkg.NewPackage("a", "1.1", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "b", Version: "1.0", Category: "test"}}, []*pkg.DefaultPackage{})
+			A.SetCategory("test")
+
+			for _, p := range []pkg.Package{A, B, C, D, D1} {
+				_, err := dbDefinitions.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+
+			for _, p := range []pkg.Package{A, B, C, D} {
+				_, err := dbInstalled.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+			uninstall, solution, err := s.Upgrade(false, false)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(len(uninstall)).To(Equal(1))
+			Expect(uninstall[0].GetName()).To(Equal("d"))
+			Expect(uninstall[0].GetVersion()).To(Equal("1.5"))
+
+			Expect(solution).To(ContainElement(PackageAssert{Package: D1, Value: true}))
+
+			Expect(len(solution)).To(Equal(5))
 		})
 	})
 })

pkg/solver/resolver_test.go

@@ -79,7 +79,7 @@ var _ = Describe("Resolver", func() {
 				solution, err := s.Install([]pkg.Package{D, F}) // D and F should go as they have no deps. A/E should be filtered by QLearn
 				Expect(err).ToNot(HaveOccurred())
 
-				Expect(len(solution)).To(Equal(3))
+				Expect(len(solution)).To(Equal(6))
 
 				Expect(solution).ToNot(ContainElement(PackageAssert{Package: A, Value: true}))
 				Expect(solution).ToNot(ContainElement(PackageAssert{Package: B, Value: true}))
@@ -117,7 +117,7 @@ var _ = Describe("Resolver", func() {
 				Expect(solution).To(ContainElement(PackageAssert{Package: C, Value: true}))
 				Expect(solution).To(ContainElement(PackageAssert{Package: D, Value: true}))
 
-				Expect(len(solution)).To(Equal(2))
+				Expect(len(solution)).To(Equal(4))
 			})
 
 			It("will find out that we can install D and F by ignoring E and A", func() {
@@ -148,8 +148,7 @@ var _ = Describe("Resolver", func() {
 				Expect(solution).To(ContainElement(PackageAssert{Package: D, Value: true}))
 				Expect(solution).ToNot(ContainElement(PackageAssert{Package: E, Value: true}))
 				Expect(solution).To(ContainElement(PackageAssert{Package: F, Value: true}))
-				Expect(len(solution)).To(Equal(3))
-
+				Expect(len(solution)).To(Equal(6))
 			})
 		})
 

pkg/solver/solver.go

@@ -134,13 +134,13 @@ func (s *Solver) BuildInstalled() (bf.Formula, error) {
 	var packages pkg.Packages
 	for _, p := range s.Installed() {
 		packages = append(packages, p)
-		for _, dep := range p.Related(s.DefinitionDatabase) {
+		for _, dep := range p.Related(s.InstalledDatabase) {
 			packages = append(packages, dep)
 		}
 	}
 
 	for _, p := range packages {
-		solvable, err := p.BuildFormula(s.DefinitionDatabase, s.SolverDatabase)
+		solvable, err := p.BuildFormula(s.InstalledDatabase, s.SolverDatabase)
 		if err != nil {
 			return nil, err
 		}
@@ -167,7 +167,6 @@ func (s *Solver) BuildWorld(includeInstalled bool) (bf.Formula, error) {
 	}
 
 	for _, p := range s.World() {
-
 		solvable, err := p.BuildFormula(s.DefinitionDatabase, s.SolverDatabase)
 		if err != nil {
 			return nil, err
@@ -266,7 +265,7 @@ func (s *Solver) Conflicts(pack pkg.Package, lsp pkg.Packages) (bool, error) {
 		if revdepsErr == nil {
 			revdepsErr = errors.New("")
 		}
-		revdepsErr = errors.New(fmt.Sprintf("%s\n%s", revdepsErr.Error(), r.HumanReadableString()))
+		revdepsErr = fmt.Errorf("%s\n%s", revdepsErr.Error(), r.HumanReadableString())
 	}
 
 	return len(revdeps) != 0, revdepsErr
@@ -278,7 +277,6 @@ func (s *Solver) ConflictsWith(pack pkg.Package, lsp pkg.Packages) (bool, error)
 	p, err := s.DefinitionDatabase.FindPackage(pack)
 	if err != nil {
 		p = pack //Relax search, otherwise we cannot compute solutions for packages not in definitions
-
 		//	return false, errors.Wrap(err, "Package not found in definition db")
 	}
 
@@ -399,23 +397,23 @@ func (s *Solver) UpgradeUniverse(dropremoved bool) (pkg.Packages, PackagesAssert
 	notUptodate := pkg.Packages{}
 	removed := pkg.Packages{}
 	toUpgrade := pkg.Packages{}
+	replacements := map[pkg.Package]pkg.Package{}
 
 	// TODO: this is memory expensive, we need to optimize this
-	universe := pkg.NewInMemoryDatabase(false)
-	for _, p := range s.DefinitionDatabase.World() {
-		universe.CreatePackage(p)
+	universe, err := s.DefinitionDatabase.Copy()
+	if err != nil {
+		return nil, nil, errors.Wrap(err, "Failed copying db")
 	}
+
 	for _, p := range s.Installed() {
 		universe.CreatePackage(p)
 	}
 
 	// Grab all the installed ones, see if they are eligible for update
 	for _, p := range s.Installed() {
-		available, err := universe.FindPackageVersions(p)
-		if err != nil {
+		available, err := s.DefinitionDatabase.FindPackageVersions(p)
+		if len(available) == 0 || err != nil {
 			removed = append(removed, p)
-		}
-		if len(available) == 0 {
 			continue
 		}
 
@@ -424,6 +422,7 @@ func (s *Solver) UpgradeUniverse(dropremoved bool) (pkg.Packages, PackagesAssert
 		if !bestmatch.Matches(p) {
 			notUptodate = append(notUptodate, p)
 			toUpgrade = append(toUpgrade, bestmatch)
+			replacements[p] = bestmatch
 		}
 	}
 
@@ -437,28 +436,37 @@ func (s *Solver) UpgradeUniverse(dropremoved bool) (pkg.Packages, PackagesAssert
 
 	// Treat removed packages from universe as marked for deletion
 	if dropremoved {
-		notUptodate = append(notUptodate, removed...)
+		// SAT encode the clauses against the world
+		for _, p := range removed.Unique() {
+			encodedP, err := p.Encode(universe)
+			if err != nil {
+				return nil, nil, errors.Wrap(err, "couldn't encode package")
+			}
+			P := bf.Var(encodedP)
+			formulas = append(formulas, bf.And(bf.Not(P), r))
+		}
 	}
 
-	// SAT encode the clauses against the world
-	for _, p := range notUptodate.Unique() {
-		encodedP, err := p.Encode(universe)
+	for old, new := range replacements {
+		oldP, err := old.Encode(universe)
 		if err != nil {
 			return nil, nil, errors.Wrap(err, "couldn't encode package")
 		}
-		P := bf.Var(encodedP)
-		formulas = append(formulas, bf.And(bf.Not(P), r))
-	}
-
-	for _, p := range toUpgrade {
-		encodedP, err := p.Encode(universe)
+		oldencodedP := bf.Var(oldP)
+		newP, err := new.Encode(universe)
 		if err != nil {
 			return nil, nil, errors.Wrap(err, "couldn't encode package")
 		}
-		P := bf.Var(encodedP)
-		formulas = append(formulas, bf.And(P, r))
+		newEncodedP := bf.Var(newP)
+
+		//solvable, err := old.BuildFormula(s.DefinitionDatabase, s.SolverDatabase)
+		solvablenew, err := new.BuildFormula(s.DefinitionDatabase, s.SolverDatabase)
+
+		formulas = append(formulas, bf.And(bf.Not(oldencodedP), bf.And(append(solvablenew, newEncodedP)...)))
 	}
 
+	//formulas = append(formulas, r)
+
 	markedForRemoval := pkg.Packages{}
 
 	if len(formulas) == 0 {
@@ -492,10 +500,10 @@ func (s *Solver) Upgrade(checkconflicts, full bool) (pkg.Packages, PackagesAsser
 	toUninstall := pkg.Packages{}
 	toInstall := pkg.Packages{}
 
-	// we do this in memory so we take into account of provides
-	universe := pkg.NewInMemoryDatabase(false)
-	for _, p := range s.DefinitionDatabase.World() {
-		universe.CreatePackage(p)
+	// we do this in memory so we take into account of provides, and its faster
+	universe, err := s.DefinitionDatabase.Copy()
+	if err != nil {
+		return nil, nil, errors.Wrap(err, "failed creating db copy")
 	}
 
 	installedcopy := pkg.NewInMemoryDatabase(false)
@@ -581,7 +589,7 @@ func (s *Solver) Uninstall(checkconflicts, full bool, packs ...pkg.Package) (pkg
 	if !full && checkconflicts {
 		for _, candidate := range toRemove {
 			if conflicts, err := s.Conflicts(candidate, s.Installed()); conflicts {
-				return nil, err
+				return nil, errors.Wrap(err, "while searching for "+candidate.HumanReadableString()+" conflicts")
 			}
 		}
 		return toRemove, nil
@@ -607,7 +615,7 @@ func (s *Solver) Uninstall(checkconflicts, full bool, packs ...pkg.Package) (pkg
 		}
 	}
 
-	s2 := NewSolver(Options{Type: SingleCoreSimple}, pkg.NewInMemoryDatabase(false), s.DefinitionDatabase, pkg.NewInMemoryDatabase(false))
+	s2 := NewSolver(Options{Type: SingleCoreSimple}, pkg.NewInMemoryDatabase(false), s.InstalledDatabase, pkg.NewInMemoryDatabase(false))
 	s2.SetResolver(s.Resolver)
 
 	// Get the requirements to install the candidate
@@ -652,7 +660,7 @@ func (s *Solver) Uninstall(checkconflicts, full bool, packs ...pkg.Package) (pkg
 // BuildFormula builds the main solving formula that is evaluated by the sat solver.
 func (s *Solver) BuildFormula() (bf.Formula, error) {
 	var formulas []bf.Formula
-	r, err := s.BuildPartialWorld(false)
+	r, err := s.BuildWorld(false)
 	if err != nil {
 		return nil, err
 	}

pkg/solver/solver_test.go

@@ -111,7 +111,7 @@ var _ = Describe("Solver", func() {
 			//	Expect(solution).To(ContainElement(PackageAssert{Package: B, Value: false}))
 			//Expect(solution).To(ContainElement(PackageAssert{Package: D, Value: false}))
 
-			Expect(len(solution)).To(Equal(3))
+			Expect(len(solution)).To(Equal(5))
 		})
 
 		It("Solves correctly if the selected package to install has requirements", func() {
@@ -401,7 +401,7 @@ var _ = Describe("Solver", func() {
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: D, Value: false}))
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: E, Value: true}))
 
-			Expect(len(solution)).To(Equal(3))
+			Expect(len(solution)).To(Equal(4))
 			Expect(err).ToNot(HaveOccurred())
 		})
 
@@ -529,7 +529,7 @@ var _ = Describe("Solver", func() {
 			Expect(solution).To(ContainElement(PackageAssert{Package: D1, Value: false}))
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: E, Value: true}))
 
-			Expect(len(solution)).To(Equal(5))
+			Expect(len(solution)).To(Equal(6))
 			Expect(err).ToNot(HaveOccurred())
 		})
 
@@ -570,7 +570,7 @@ var _ = Describe("Solver", func() {
 			Expect(solution).To(ContainElement(PackageAssert{Package: D1, Value: false}))
 			Expect(solution).ToNot(ContainElement(PackageAssert{Package: E, Value: true}))
 
-			Expect(len(solution)).To(Equal(5))
+			Expect(len(solution)).To(Equal(6))
 			Expect(err).ToNot(HaveOccurred())
 		})
 
@@ -671,8 +671,9 @@ var _ = Describe("Solver", func() {
 				Expect(err).ToNot(HaveOccurred())
 
 				Expect(solution).To(ContainElement(A))
+				Expect(solution).To(ContainElement(B))
 
-				Expect(len(solution)).To(Equal(1))
+				Expect(len(solution)).To(Equal(2))
 			})
 
 			It("Uninstalls complex packages correctly, even if shared deps are required by system packages", func() {
@@ -752,6 +753,33 @@ var _ = Describe("Solver", func() {
 
 			})
 
+			It("Fails to uninstall if a package is required", func() {
+				D := pkg.NewPackage("D", "", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
+				B := pkg.NewPackage("B", "", []*pkg.DefaultPackage{D}, []*pkg.DefaultPackage{})
+				C := pkg.NewPackage("C", "", []*pkg.DefaultPackage{B}, []*pkg.DefaultPackage{})
+				A := pkg.NewPackage("A", "", []*pkg.DefaultPackage{B}, []*pkg.DefaultPackage{})
+				Z := pkg.NewPackage("Z", "", []*pkg.DefaultPackage{A}, []*pkg.DefaultPackage{})
+				F := pkg.NewPackage("F", "", []*pkg.DefaultPackage{Z, B}, []*pkg.DefaultPackage{})
+
+				Z.SetVersion("1.4101.dvw.dqc.")
+				B.SetVersion("1.4101qe.eq.ff..dvw.dqc.")
+				C.SetVersion("1.aaaa.eq.ff..dvw.dqc.")
+
+				for _, p := range []pkg.Package{A, B, C, D, Z, F} {
+					_, err := dbDefinitions.CreatePackage(p)
+					Expect(err).ToNot(HaveOccurred())
+				}
+
+				for _, p := range []pkg.Package{A, B, C, D, Z, F} {
+					_, err := dbInstalled.CreatePackage(p)
+					Expect(err).ToNot(HaveOccurred())
+				}
+
+				solution, err := s.Uninstall(true, false, B)
+				Expect(err).To(HaveOccurred())
+				Expect(len(solution)).To(Equal(0))
+			})
+
 			It("UninstallUniverse simple package correctly", func() {
 
 				C := pkg.NewPackage("C", "", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
@@ -1209,6 +1237,8 @@ var _ = Describe("Solver", func() {
 		})
 	})
 	Context("Upgrades", func() {
+		E := pkg.NewPackage("e", "1.5", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
+		E.SetCategory("test")
 		C := pkg.NewPackage("c", "1.5", []*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=1.0", Category: "test"}}, []*pkg.DefaultPackage{})
 		C.SetCategory("test")
 		B := pkg.NewPackage("b", "1.0", []*pkg.DefaultPackage{}, []*pkg.DefaultPackage{})
@@ -1253,7 +1283,37 @@ var _ = Describe("Solver", func() {
 		})
 
 		It("upgrades correctly with provides", func() {
-			B.SetProvides([]*pkg.DefaultPackage{&pkg.DefaultPackage{Name: "a", Version: ">=0", Category: "test"}, &pkg.DefaultPackage{Name: "c", Version: ">=0", Category: "test"}})
+			B.SetProvides([]*pkg.DefaultPackage{
+				&pkg.DefaultPackage{Name: "a", Version: ">=0", Category: "test"},
+				&pkg.DefaultPackage{Name: "c", Version: ">=0", Category: "test"},
+			})
+
+			for _, p := range []pkg.Package{B} {
+				_, err := dbDefinitions.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+
+			for _, p := range []pkg.Package{A, C} {
+				_, err := dbInstalled.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+			uninstall, solution, err := s.Upgrade(true, true)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(len(uninstall)).To(Equal(2))
+
+			Expect(uninstall).To(ContainElement(C))
+			Expect(uninstall).To(ContainElement(A))
+
+			Expect(solution).To(ContainElement(PackageAssert{Package: B, Value: true}))
+			Expect(len(solution)).To(Equal(1))
+		})
+
+		PIt("upgrades correctly with provides, also if definitiondb contains both a provide, and the package to be provided", func() {
+			B.SetProvides([]*pkg.DefaultPackage{
+				&pkg.DefaultPackage{Name: "a", Version: ">=0", Category: "test"},
+				&pkg.DefaultPackage{Name: "c", Version: ">=0", Category: "test"},
+			})
 
 			for _, p := range []pkg.Package{A1, B} {
 				_, err := dbDefinitions.CreatePackage(p)
@@ -1268,10 +1328,9 @@ var _ = Describe("Solver", func() {
 			Expect(err).ToNot(HaveOccurred())
 
 			Expect(len(uninstall)).To(Equal(2))
-			Expect(uninstall[1].GetName()).To(Equal("c"))
-			Expect(uninstall[1].GetVersion()).To(Equal("1.5"))
-			Expect(uninstall[0].GetName()).To(Equal("a"))
-			Expect(uninstall[0].GetVersion()).To(Equal("1.1"))
+
+			Expect(uninstall).To(ContainElement(C))
+			Expect(uninstall).To(ContainElement(A))
 
 			Expect(solution).To(ContainElement(PackageAssert{Package: B, Value: true}))
 			Expect(len(solution)).To(Equal(1))
@@ -1296,10 +1355,37 @@ var _ = Describe("Solver", func() {
 
 			Expect(solution).To(ContainElement(PackageAssert{Package: A1, Value: true}))
 			Expect(solution).To(ContainElement(PackageAssert{Package: B, Value: true}))
-			Expect(solution).To(ContainElement(PackageAssert{Package: C, Value: false}))
 			Expect(solution).To(ContainElement(PackageAssert{Package: A, Value: false}))
 
-			Expect(len(solution)).To(Equal(4))
+			Expect(solution).ToNot(ContainElement(PackageAssert{Package: C, Value: true}))
+			Expect(solution).ToNot(ContainElement(PackageAssert{Package: A, Value: true}))
+
+			Expect(len(solution)).To(Equal(3))
+		})
+
+		It("Suggests to remove untracked packages", func() {
+			for _, p := range []pkg.Package{E} {
+				_, err := dbDefinitions.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+
+			for _, p := range []pkg.Package{A, B, C, E} {
+				_, err := dbInstalled.CreatePackage(p)
+				Expect(err).ToNot(HaveOccurred())
+			}
+			uninstall, solution, err := s.UpgradeUniverse(true)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(len(uninstall)).To(Equal(3))
+
+			Expect(uninstall).To(ContainElement(B))
+			Expect(uninstall).To(ContainElement(A))
+			Expect(uninstall).To(ContainElement(C))
+
+			Expect(solution).To(ContainElement(PackageAssert{Package: C, Value: false}))
+			Expect(solution).To(ContainElement(PackageAssert{Package: B, Value: false}))
+			Expect(solution).To(ContainElement(PackageAssert{Package: A, Value: false}))
+
+			Expect(len(solution)).To(Equal(3))
 		})
 	})
 })

tests/fixtures/buildable/cat/b/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /test5
   - echo artifact6 > /test6
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/buildableseed/cat/b/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /test5
   - echo artifact6 > /test6
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/complex/selection/0.1/build.yaml

@@ -1,2 +1,4 @@
 
 image: "alpine:3.5"
+steps:
+- echo "foo"

tests/fixtures/complex/selection/0.20191212/build.yaml

@@ -15,3 +15,6 @@ requires:
 - category: "layer"
   name: "build-sabayon-overlay"
   version: "0.20191212"
+
+steps:
+- echo "foo"

tests/fixtures/complex/selection/kernel/build.yaml

@@ -5,3 +5,6 @@ requires:
 
 includes:
 - /usr/portage/packages/.*
+
+steps:
+- echo "foo"

tests/fixtures/complex/selection/meta/build.yaml

@@ -16,3 +16,6 @@ requires:
 - category: "layer"
   name: "build-sabayon-overlay"
   version: ">=0.1"
+
+steps:
+- echo "foo"

tests/fixtures/complex/selection/sabayon-build-overlays/portage/0.20191126/build.yaml

@@ -2,3 +2,5 @@ requires:
 - category: "layer"
   name: "build"
   version: ">=0.1"
+steps:
+- echo "foo"

tests/fixtures/complex/selection/sabayon-build-overlays/portage/0.20191212/build.yaml

@@ -2,3 +2,5 @@ requires:
 - category: "layer"
   version:  "0.1"
   name:     "build"
+steps:
+- echo "foo"

tests/fixtures/complex/selection/sabayon-build-overlays/sabayon-overlay/0.20191205/build.yaml

@@ -5,3 +5,5 @@ requires:
 - category: "layer"
   name: "sabayon-build-portage"
   version: ">=0.1"
+steps:
+- echo "foo"

tests/fixtures/complex/selection/sabayon-build-overlays/sabayon-overlay/0.20191205/definition.yaml

@@ -1,3 +1,5 @@
 category: "layer"
 name: "build-sabayon-overlay"
 version: "0.20191205"
+steps:
+- echo "foo"

tests/fixtures/complex/selection/sabayon-build-overlays/sabayon-overlay/0.20191212/build.yaml

@@ -5,3 +5,5 @@ requires:
 - category: "layer"
   name: "sabayon-build-portage"
   version: ">=0.1"
+steps:
+- echo "foo"

tests/fixtures/complex/selection/sources5.4.2/build.yaml

@@ -2,6 +2,7 @@ requires:
 - category: "layer"
   version:  "0.1"
   name:     "build-sabayon-overlays"
-
+steps:
+- echo "foo"
 includes:
 - /usr/portage/packages/.*

tests/fixtures/expansion/cat/b/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /test5
   - echo artifact6 > /test6
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/finalizers_upgrade/alpine old/build.yaml

@@ -0,0 +1,6 @@
+image: "alpine"
+unpack: true
+excludes:
+  - /sys/.*
+  - /proc/.*
+  - /etc/.*

tests/fixtures/finalizers_upgrade/alpine old/definition.yaml

@@ -0,0 +1,3 @@
+category: "seed"
+name: "alpine"
+version: "0.9"

tests/fixtures/finalizers_upgrade/alpine/build.yaml

@@ -0,0 +1,6 @@
+image: "alpine"
+unpack: true
+excludes:
+  - /sys/.*
+  - /proc/.*
+  - /etc/.*

tests/fixtures/finalizers_upgrade/alpine/definition.yaml

@@ -0,0 +1,3 @@
+category: "seed"
+name: "alpine"
+version: "1.0"

tests/fixtures/finalizers_upgrade/alpine/finalize.yaml

@@ -0,0 +1,2 @@
+install:
+- echo "$0" > /tmp/foo

tests/fixtures/package_dir/dironly_filter/build.yaml

@@ -1,10 +1,10 @@
 image: "alpine"
 prelude:
   - mkdir /foo
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /foo/test5
   - echo artifact6 > /foo/test6
+  - chmod +x generate.sh
   - ./generate.sh
 package_dir: /foo
 includes:

tests/fixtures/simple_dep/a/build.yaml

@@ -0,0 +1,5 @@
+image: "alpine"
+prelude:
+  - echo a > /a-prelude
+steps:
+  - echo a > /a

tests/fixtures/simple_dep/a/definition.yaml

@@ -0,0 +1,3 @@
+category: "test"
+name: "a"
+version: "1.2"

tests/fixtures/simple_dep/b/build.yaml

@@ -0,0 +1,9 @@
+requires:
+- category: "test"
+  name: "a"
+  version: ">=0"
+
+prelude:
+  - echo b > /b-prelude
+steps:
+  - echo b > /b

tests/fixtures/simple_dep/b/definition.yaml

@@ -0,0 +1,3 @@
+category: "test"
+name: "b"
+version: "1.1"

tests/fixtures/simple_dep/c/build.yaml

@@ -0,0 +1,7 @@
+requires:
+- category: "test"
+  name: "a"
+  version: ">=0"
+steps:
+  - echo c > /c
+

tests/fixtures/simple_dep/c/definition.yaml

@@ -0,0 +1,9 @@
+category: "test"
+name: "c"
+version: "1.0"
+# Boom?
+
+requires:
+- category: "test"
+  name: "a"
+  version: ">=0.1"

tests/fixtures/upgrade/cat/b-1.1/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /newc
   - echo artifact6 > /newnewc
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/upgrade/cat/b/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /test5
   - echo artifact6 > /test6
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/upgrade_integration/cat/b-1.1/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /newc
   - echo artifact6 > /newnewc
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/upgrade_integration/cat/b/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /test5
   - echo artifact6 > /test6
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/upgrade_new_repo/cat/b-1.1/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /newc
   - echo artifact6 > /newnewc
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/upgrade_old_repo/cat/b/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /test5
   - echo artifact6 > /test6
+  - chmod +x generate.sh
   - ./generate.sh

tests/fixtures/upgrade_old_repo_revision/cat/b-1.1/build.yaml

@@ -2,8 +2,8 @@ image: "alpine"
 prelude:
   - echo foo > /test
   - echo bar > /test2
-  - chmod +x generate.sh
 steps:
   - echo artifact5 > /newc
   - echo artifact6 > /newnewc
+  - chmod +x generate.sh
   - ./generate.sh