Run go mod tidy and fix casing (#377)

* Run `go mod tidy` and fix casing

Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>

* Use go 1.24 in workflows

Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>

---------

Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: mudler
+
+---
+
+<!-- Thanks for helping us to improve Luet! We welcome all bug reports. Please fill out each area of the template so we can better help you. Comments like this will be hidden when you post but you can delete them if you wish. -->
+
+**Luet version:**
+<!-- Provide the output from "luet --version" -->
+
+**CPU architecture, OS, and Version:**
+<!-- Provide the output from "uname -a"  -->
+
+**Describe the bug**
+<!-- A clear and concise description of what the bug is. -->
+
+**To Reproduce**
+<!-- Steps to reproduce the behavior, including the luet command used -->
+
+**Expected behavior**
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Logs**
+<!-- If applicable, add logs with the "--debug" flag enabled to help explain your problem. -->
+
+**Additional context**
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,22 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: mudler
+
+---
+
+<!-- Thanks for helping us to improve Luet! We welcome all feature requests. Please fill out each area of the template so we can better help you. Comments like this will be hidden when you post but you can delete them if you wish. -->
+
+**Is your feature request related to a problem? Please describe.**
+<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]  -->
+
+**Describe the solution you'd like**
+<!-- A clear and concise description of what you want to happen.  -->
+
+**Describe alternatives you've considered**
+<!-- A clear and concise description of any alternative solutions or features you've considered.  -->
+
+**Additional context**
+<!-- Add any other context or screenshots about the feature request here. -->

.github/workflows/dependabot.yml

@@ -0,0 +1,42 @@
+name: Dependabot auto-merge
+on:
+- pull_request_target
+
+permissions:
+  contents: write
+  pull-requests: write
+  packages: read
+
+jobs:
+  dependabot:
+    runs-on: ubuntu-latest
+    if: ${{ github.actor == 'dependabot[bot]' }}
+    steps:
+      - name: Dependabot metadata
+        id: metadata
+        uses: dependabot/fetch-metadata@v2.0.0
+        with:
+          github-token: "${{ secrets.GITHUB_TOKEN }}"
+          skip-commit-verification: true
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Approve a PR if not already approved
+        run: |
+          gh pr checkout "$PR_URL"
+            if [ "$(gh pr status --json reviewDecision -q .currentBranch.reviewDecision)" != "APPROVED" ];
+          then
+            gh pr review --approve "$PR_URL"
+          else
+            echo "PR already approved.";
+          fi
+        env:
+          PR_URL: ${{github.event.pull_request.html_url}}
+          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
+
+      - name: Enable auto-merge for Dependabot PRs
+        run: gh pr merge --auto --squash "$PR_URL"
+        env:
+          PR_URL: ${{github.event.pull_request.html_url}}
+          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

.github/workflows/image.yml

@@ -0,0 +1,67 @@
+---
+name: 'build container images'
+
+on:
+  push:
+    branches:
+      - master
+    tags:
+      - '*'
+
+jobs:
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Prepare
+        id: prep
+        run: |
+          DOCKER_IMAGE=quay.io/luet/base
+          VERSION=latest
+          SHORTREF=${GITHUB_SHA::8}
+
+          # If this is git tag, use the tag name as a docker tag
+          if [[ $GITHUB_REF == refs/tags/* ]]; then
+            VERSION=${GITHUB_REF#refs/tags/}
+          fi
+          TAGS="${DOCKER_IMAGE}:${VERSION},${DOCKER_IMAGE}:${SHORTREF}"
+
+          # If the VERSION looks like a version number, assume that
+          # this is the most recent version of the image and also
+          # tag it 'latest'.
+          if [[ $VERSION =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
+            TAGS="$TAGS,${DOCKER_IMAGE}:latest"
+          fi
+
+          # Set output parameters.
+          echo ::set-output name=tags::${TAGS}
+          echo ::set-output name=docker_image::${DOCKER_IMAGE}
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@master
+        with:
+          platforms: all
+
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Login to DockerHub
+        if: github.event_name != 'pull_request'
+        uses: docker/login-action@v1
+        with:
+          registry: quay.io
+          username: ${{ secrets.DOCKER_RELEASE_USERNAME }}
+          password: ${{ secrets.DOCKER_RELEASE_PASSWORD }}
+
+      - name: Build
+        uses: docker/build-push-action@v2
+        with:
+          builder: ${{ steps.buildx.outputs.name }}
+          context: .
+          file: ./Dockerfile
+          platforms: linux/amd64,linux/arm64,linux/arm
+          push: true
+          tags: ${{ steps.prep.outputs.tags }}

.github/workflows/pages.yml

@@ -0,0 +1,23 @@
+name: Github Pages
+on:
+  push:
+    branches: [ master, docs ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        submodules: 'recursive'
+    - name: Build 🔧
+      run: |
+           cd docs && make build
+           mv public ../
+    - name: Deploy 🚀
+      if: github.ref == 'refs/heads/master' && github.event_name == 'push'
+      uses: JamesIves/github-pages-deploy-action@releases/v3
+      with:
+        ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
+        BRANCH: gh-pages
+        FOLDER: public

.github/workflows/pr.yml

@@ -0,0 +1,81 @@
+
+on: pull_request
+name: Build and Test
+jobs:
+  tests-integration-img:
+    strategy:
+      matrix:
+        go-version: [1.24.x]
+        platform: [ubuntu-latest]
+    runs-on: ${{ matrix.platform }}
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: ${{ matrix.go-version }}
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: setup-docker
+      uses: docker-practice/actions-setup-docker@0.0.1
+    - name: Install deps
+      run: |
+            sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+            sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
+            sudo chmod a+x "/usr/bin/img"
+    - name: Tests with Img backend
+      run: sudo -E env "PATH=$PATH" env "LUET_BACKEND=img" make test-integration
+
+  tests-integration:
+    strategy:
+      matrix:
+        go-version: [1.24.x]
+        platform: [ubuntu-latest]
+    runs-on: ${{ matrix.platform }}
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: ${{ matrix.go-version }}
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: setup-docker
+      uses: docker-practice/actions-setup-docker@0.0.1
+    - name: Install deps
+      run: |
+            sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+    - name: Tests
+      run: sudo -E env "PATH=$PATH" make test-integration
+  tests-unit:
+    strategy:
+      matrix:
+        go-version: [1.24.x]
+        platform: [ubuntu-latest]
+    runs-on: ${{ matrix.platform }}
+    steps:
+    - name: Install Go
+      uses: actions/setup-go@v2
+      with:
+        go-version: ${{ matrix.go-version }}
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: setup-docker
+      uses: docker-practice/actions-setup-docker@0.0.1
+    - name: Install deps
+      run: |
+            sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+            sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
+            sudo chmod a+x "/usr/bin/img"
+            sudo make build
+            sudo cp -rf luet /usr/bin/luet
+    - name: Install GoReleaser
+      uses: goreleaser/goreleaser-action@v2
+      with:
+        install-only: true
+    - name: Build
+      run: sudo -E env "PATH=$PATH" make multiarch-build-small
+    - name: Tests
+      run: sudo -E env "PATH=$PATH" make coverage
+    - name: Codecov
+      uses: codecov/codecov-action@v2.1.0
+      with:
+        file: coverage.txt

.github/workflows/push.yml

@@ -0,0 +1,92 @@
+on: push
+concurrency:
+  group: registries-tests
+
+name: Build on push
+jobs:
+  tests-integration-img:
+    name: Integration tests with img
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.24.x
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Login to quay
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Install deps
+        run: |
+            sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+            sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
+            sudo chmod a+x "/usr/bin/img"
+      - name: Login to quay with img
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo img login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Tests with Img backend
+        run: |
+            sudo -E env "PATH=$PATH" \
+                    env "LUET_BACKEND=img" \
+                    make test-integration
+  tests-integration:
+    name: Integration tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.24.x
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Login to quay
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Install deps
+        run: |
+            sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+      - name: Tests
+        run: |
+              sudo -E \
+                    env "PATH=$PATH" \
+                    env "TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                    env "UNIT_TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                    env "UNIT_TEST_DOCKER_IMAGE_REPOSITORY=${{ secrets.DOCKER_TESTING_UNIT_TEST_IMAGE }}" \
+                    make test-integration
+
+  tests-unit:
+    name: Unit tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.24.x
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Login to quay
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Install deps
+        run: |
+            sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+            sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
+            sudo chmod a+x "/usr/bin/img"
+            sudo make build
+            sudo cp -rf luet /usr/bin/luet
+      - name: Install GoReleaser
+        uses: goreleaser/goreleaser-action@v2
+        with:
+          install-only: true
+      - name: Build test
+        run: sudo -E env "PATH=$PATH" make multiarch-build-small
+      - name: Tests
+        run: |
+              sudo -E \
+                    env "PATH=$PATH" \
+                    env "TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                    env "UNIT_TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                    env "UNIT_TEST_DOCKER_IMAGE_REPOSITORY=${{ secrets.DOCKER_TESTING_UNIT_TEST_IMAGE }}" \
+                    make coverage
+      - name: Codecov
+        uses: codecov/codecov-action@v2.1.0
+        with:
+          file: coverage.txt
+

.github/workflows/release.yml

@@ -1,23 +1,115 @@
-on: push
-name: Build and release on push
+on:
+  push:
+    tags:
+      - '*'  # only test and release when a tag is pushed
+concurrency:
+  group: registries-tests
+
+name: Test and Release on tag
 jobs:
-  release:
-    name: Test and Release
+  tests-integration-img:
+    name: Integration tests
     runs-on: ubuntu-latest
     steps:
       - name: Install Go
         uses: actions/setup-go@v2
         with:
-          go-version: 1.14.x
+          go-version: 1.24.x
       - name: Checkout code
         uses: actions/checkout@v2
+      - name: setup-docker
+        uses: docker-practice/actions-setup-docker@0.0.1
+      - name: Login to quay
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Install deps
+        run: |
+          sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+          sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
+          sudo chmod a+x "/usr/bin/img"
+      - name: Login to quay with img
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo img login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Tests with Img backend
+        run: |
+          sudo -E env "PATH=$PATH" \
+                  env "LUET_BACKEND=img" \
+                  make test-integration
+  tests-integration:
+    name: Integration tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.24.x
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: setup-docker
+        uses: docker-practice/actions-setup-docker@0.0.1
+      - name: Login to quay
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Install deps
+        run: |
+          sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
       - name: Tests
-        run: sudo -E env "PATH=$PATH" make deps multiarch-build test-integration test-coverage
-      - name: Build
-        run: sudo -E env "PATH=$PATH" make multiarch-build && sudo chmod -R 777 release/
-      - name: Release
-        uses: fnkr/github-action-ghr@v1
-        if: startsWith(github.ref, 'refs/tags/')
+        run: |
+          sudo -E \
+                env "PATH=$PATH" \
+                env "TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                env "UNIT_TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                env "UNIT_TEST_DOCKER_IMAGE_REPOSITORY=${{ secrets.DOCKER_TESTING_UNIT_TEST_IMAGE }}" \
+                make test-integration
+
+  tests-unit:
+    name: Unit tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.24.x
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: setup-docker
+        uses: docker-practice/actions-setup-docker@0.0.1
+      - name: Login to quay
+        run: echo ${{ secrets.DOCKER_TESTING_PASSWORD }} | sudo -E docker login -u ${{ secrets.DOCKER_TESTING_USERNAME }} --password-stdin quay.io
+      - name: Install deps
+        run: |
+          sudo apt-get update && sudo apt-get install -y upx && sudo -E env "PATH=$PATH" make deps
+          sudo curl -fSL "https://github.com/genuinetools/img/releases/download/v0.5.11/img-linux-amd64" -o "/usr/bin/img"
+          sudo chmod a+x "/usr/bin/img"
+          sudo make build
+          sudo cp -rf luet /usr/bin/luet
+      - name: Tests
+        run: |
+          sudo -E \
+                env "PATH=$PATH" \
+                env "TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                env "UNIT_TEST_DOCKER_IMAGE=${{ secrets.DOCKER_TESTING_IMAGE }}" \
+                env "UNIT_TEST_DOCKER_IMAGE_REPOSITORY=${{ secrets.DOCKER_TESTING_UNIT_TEST_IMAGE }}" \
+                make coverage
+      - name: Codecov
+        uses: codecov/codecov-action@v2.1.0
+        with:
+          file: coverage.txt
+
+  release:
+    name: Release
+    runs-on: ubuntu-latest
+    needs: [ "tests-integration-img", "tests-integration","tests-unit" ]
+    steps:
+      - name: Install Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.24.x
+      - name: Checkout code
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+      - name: Run GoReleaser
+        uses: goreleaser/goreleaser-action@v2
+        with:
+          version: latest
+          args: release --clean
         env:
-          GHR_PATH: release/
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/test.yml

@@ -1,21 +0,0 @@
-
-on: pull_request
-name: Build and Test
-jobs:
-  test:
-    strategy:
-      matrix:
-        go-version: [1.14.x]
-        platform: [ubuntu-latest]
-    runs-on: ${{ matrix.platform }}
-    steps:
-    - name: Install Go
-      uses: actions/setup-go@v2
-      with:
-        go-version: ${{ matrix.go-version }}
-    - name: Checkout code
-      uses: actions/checkout@v2
-    - name: setup-docker
-      uses: docker-practice/actions-setup-docker@0.0.1
-    - name: Tests
-      run: sudo -E env "PATH=$PATH" make deps multiarch-build test-integration test-coverage

.gitignore

@@ -1,4 +1,12 @@
 *.swp
+.idea/
 luet
 tests/integration/shunit2
-tests/integration/bin
+tests/integration/bin
+release/
+/docs/bin/
+/docs/public/
+/docs/resources/
+/docs/node_modules/
+/docs/package-lock.json
+/docs/package.json

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "docs/themes/docsy"]
+	path = docs/themes/docsy
+	url = https://github.com/google/docsy.git

.goreleaser.yml

@@ -0,0 +1,43 @@
+before:
+  hooks:
+    - go mod tidy
+dist: release
+source:
+  enabled: true
+  name_template: '{{ .ProjectName }}-{{ .Tag }}-source'
+checksum:
+  name_template: '{{ .ProjectName }}-{{ .Tag }}-checksums.txt'
+builds:
+  -
+    env:
+      - CGO_ENABLED=0
+    ldflags:
+      - -s -w
+      - -X "github.com/mudler/luet/cmd.Version={{ .Summary }}"
+      - -X "github.com/mudler/luet/cmd.BuildTime={{ time "2006-01-02 15:04:05 MST" }}"
+    goos:
+      - linux
+    goarch:
+      - amd64
+      - arm
+      - arm64
+      - 386
+    goarm:
+      - 6
+      - 7
+archives:
+  - format: binary # this removes the tar of the archives, leaving the binaries alone
+    name_template: luet-{{ .Tag }}-{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}
+changelog:
+  sort: asc
+  filters:
+    exclude:
+      - '^docs:'
+      - '^test:'
+      - '^Merge pull request'
+release:
+  header: |
+    Luet is a multi-platform Package Manager based off from containers - it uses Docker (and others) to build packages.
+
+    It has zero dependencies and it is well suitable for "from scratch" environments.
+    It can also version entire rootfs and enables delivery of OTA-alike updates, making it a perfect fit for the Edge computing era and IoT embedded device

.travis.yml

@@ -1,19 +0,0 @@
-language: go
-services:
-  - docker
-go:
-  - "1.14"
-env:
-  - "GO15VENDOREXPERIMENT=1"
-before_install:
-  - sudo -E env "PATH=$PATH" apt-get install -y libcap2-bin
-  - sudo -E env "PATH=$PATH" make deps
-script:
-  - sudo -E env "PATH=$PATH" make multiarch-build test-integration test-coverage
-#after_success:
-#  - |
-#      if [ -n "$TRAVIS_TAG" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ]; then
-#        sudo -E env "PATH=$PATH" git config --global user.name "Deployer" && git config --global user.email foo@bar.com
-#        sudo -E env "PATH=$PATH" go get github.com/tcnksm/ghr
-#        sudo -E env "PATH=$PATH" ghr -u mudler -r luet --replace $TRAVIS_TAG release/
-#      fi

CONTRIBUTING.md

@@ -0,0 +1,53 @@
+
+We love your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+- Becoming a maintainer
+
+## We Develop with Github
+We use github to host code, to track issues and feature requests, as well as accept pull requests.
+
+## Stay in touch
+
+Join us in [slack](https://luet.slack.com/join/shared_invite/enQtOTQxMjcyNDQ0MDUxLWQ5ODVlNTI1MTYzNDRkYzkyYmM1YWE5YjM0NTliNDEzNmQwMTkxNDRhNDIzM2Y5NDBlOTZjZTYxYWQyNDE4YzY#/) and hang out with the community! It will be much easier to get started and do your first steps in contributing to the project.
+
+## All Code Changes Happen Through Pull Requests
+Pull requests are the best way to propose changes to the codebase. We actively welcome your pull requests:
+
+1. Fork the repo you want to contribute to and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the [documentation](https://github.com/Luet-lab/docs).
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Issue that pull request!
+
+## Any contributions you make will be under the Software License of the repository
+In short, when you submit code changes, your submissions are understood to be under the same License that covers the project. Feel free to contact the maintainers if that's a concern.
+
+## Report bugs using Github's [issues](https://github.com/mudler/luet/issues)
+We use GitHub issues to track public bugs. Report a bug by [opening a new issue](https://github.com/mudler/luet/issues/new); it's that easy!
+
+## Write bug reports with detail, background, and sample code
+Try to be as more descriptive as possible. When opening a new issue you will be prompted to choose between a bug or a feature request, with a small template to fill details with. Be specific!
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+  - Be specific!
+  - Give sample code if you can. 
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, or stuff you tried that didn't work)
+
+People *love* thorough bug reports.
+
+
+## License
+By contributing, you agree that your contributions will be licensed under the project Licenses.
+
+## References
+This document was adapted from the open-source contribution guidelines from https://gist.github.com/briandk/3d2e8b3ec8daf5a27a62

Dockerfile

@@ -1,7 +1,7 @@
-FROM golang as builder
+FROM golang:bullseye AS builder
 RUN apt-get update && apt-get install -y upx
 ADD . /luet
-RUN cd /luet && make build-small
+RUN cd /luet && make build
 
 FROM scratch
 ENV LUET_NOLOCK=true

Makefile

@@ -1,15 +1,11 @@
 
 # go tool nm ./luet | grep Commit
 override LDFLAGS += -X "github.com/mudler/luet/cmd.BuildTime=$(shell date -u '+%Y-%m-%d %I:%M:%S %Z')"
-override LDFLAGS += -X "github.com/mudler/luet/cmd.BuildCommit=$(shell git rev-parse HEAD)"
+override LDFLAGS += -X "github.com/mudler/luet/cmd.Version=$(shell git describe --dirty --always --tags)"
 
 NAME ?= luet
 PACKAGE_NAME ?= $(NAME)
 PACKAGE_CONFLICT ?= $(PACKAGE_NAME)-beta
-REVISION := $(shell git rev-parse --short HEAD || echo dev)
-VERSION := $(shell git describe --tags || echo $(REVISION))
-VERSION := $(shell echo $(VERSION) | sed -e 's/^v//g')
-BUILD_PLATFORMS ?= -osarch="linux/amd64" -osarch="linux/386" -osarch="linux/arm"
 ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
 
 .PHONY: all
@@ -21,9 +17,7 @@ fmt:
 
 .PHONY: test
 test:
-	GO111MODULE=off go get github.com/onsi/ginkgo/ginkgo
-	GO111MODULE=off go get github.com/onsi/gomega/...
-	ginkgo -race -r -flakeAttempts 3 ./...
+	go run github.com/onsi/ginkgo/v2/ginkgo -r --flake-attempts=3 ./...
 
 .PHONY: test-integration
 test-integration:
@@ -31,11 +25,7 @@ test-integration:
 
 .PHONY: coverage
 coverage:
-	go test ./... -race -coverprofile=coverage.txt -covermode=atomic
-
-.PHONY: test-coverage
-test-coverage:
-	scripts/ginkgo.coverage.sh --codecov
+	go run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts=3 --fail-fast -cover -covermode=atomic -coverprofile=coverage.txt -r .
 
 .PHONY: help
 help:
@@ -55,11 +45,8 @@ clean:
 deps:
 	go env
 	# Installing dependencies...
-	GO111MODULE=off go get golang.org/x/lint/golint
-	GO111MODULE=off go get github.com/mitchellh/gox
-	GO111MODULE=off go get golang.org/x/tools/cmd/cover
-	GO111MODULE=off go get github.com/onsi/ginkgo/ginkgo
-	GO111MODULE=off go get github.com/onsi/gomega/...
+	go get golang.org/x/lint/golint
+	go get github.com/mitchellh/gox
 
 .PHONY: build
 build:
@@ -88,6 +75,17 @@ test-docker:
 				--workdir /go/src/github.com/mudler/luet -ti golang:latest \
 				bash -c "make test"
 
-.PHONY: multiarch-build
+.PHONY: test-integration-docker
+test-integration-docker:
+	docker run -v $(ROOT_DIR):/go/src/github.com/mudler/luet -v /var/run/docker.sock:/var/run/docker.sock \
+				--workdir /go/src/github.com/mudler/luet -ti golang:latest \
+				bash -c "apt-get update && apt-get install docker.io && make test-integration"
+
 multiarch-build:
-	CGO_ENABLED=0 gox $(BUILD_PLATFORMS) -ldflags '$(LDFLAGS)' -output="release/$(NAME)-$(VERSION)-{{.OS}}-{{.Arch}}"
+	goreleaser build --snapshot --clean
+
+multiarch-build-small:
+	@$(MAKE) multiarch-build
+	for file in $(ROOT_DIR)/release/**/* ; do \
+		upx --brute -1 $${file} ; \
+	done

README.md

@@ -1,40 +1,68 @@
+
+<p align="center">
+  <img width=150 height=150 src="https://user-images.githubusercontent.com/2420543/119691600-0293d700-be4b-11eb-827f-49ff1174a07a.png">
+</p>
+
 # luet - Container-based Package manager
+
 [![Docker Repository on Quay](https://quay.io/repository/luet/base/status "Docker Repository on Quay")](https://quay.io/repository/luet/base)
-[![Go Report Card](https://goreportcard.com/badge/github.com/mudler/luet)](https://goreportcard.com/report/github.com/mudler/luet)
-[![Build Status](https://travis-ci.org/mudler/luet.svg?branch=master)](https://travis-ci.org/mudler/luet)
+[![Build and release on push](https://github.com/mudler/luet/actions/workflows/release.yml/badge.svg)](https://github.com/mudler/luet/actions/workflows/release.yml)
 [![GoDoc](https://godoc.org/github.com/mudler/luet?status.svg)](https://godoc.org/github.com/mudler/luet)
 [![codecov](https://codecov.io/gh/mudler/luet/branch/master/graph/badge.svg)](https://codecov.io/gh/mudler/luet)
 
-Luet is a multi-platform Package Manager based off from containers - it uses Docker (and other tech) to sandbox your builds and generate packages from them. It has zero dependencies and it is well suitable for "from scratch" environments. It can also version entire rootfs and enables delivery of OTA-alike updates, making it a perfect fit for the Edge computing era and IoT embedded devices.
+Luet is a multi-platform Package Manager based off from containers - it uses Docker (and others) to build packages. It has zero dependencies and it is well suitable for "from scratch" environments. It can also version entire rootfs and enables delivery of OTA-alike updates, making it a perfect fit for the Edge computing era and IoT embedded devices.
 
-It offers a simple [specfile format](https://luet-lab.github.io/docs/docs/concepts/specfile/) in YAML notation to define both packages and rootfs. As it is based on containers, it can be used to build seed stages for Linux From Scratch installations and it can build and track updates for those systems.
+It offers a simple [specfile format](https://luet.io/docs/concepts/packages/specfile/) in YAML notation to define both [packages](https://luet.io/docs/concepts/packages/) and [rootfs](https://luet.io/docs/concepts/packages/#package-layers). As it is based on containers, it can be also used to build stages for Linux From Scratch installations and it can build and track updates for those systems.
 
 It is written entirely in Golang and where used as package manager, it can run in from scratch environment, with zero dependencies.
 
+[![asciicast](https://asciinema.org/a/388348.svg)](https://asciinema.org/a/388348)
+
+
 ## In a glance
 
 - Luet can reuse Gentoo's portage tree hierarchy, and it is heavily inspired from it.
-- It builds, installs, uninstalls and perform upgrades on machines
+- It builds from containers, but installs, uninstalls and perform upgrades on machines
 - Installer doesn't depend on anything ( 0 dep installer !), statically built
-- Support for packages as "layers"
-- It uses SAT solving techniques to solve the deptree ( Inspired by [OPIUM](https://ranjitjhala.github.io/static/opium.pdf) )
+- You can install it aside also with your current distro package manager, and start building and distributing your packages
+- [Support for packages as "layers"](https://luet.io/docs/concepts/packages/specfile/#building-strategies)
+- [It uses SAT solving techniques to solve the deptree](https://luet.io/docs/concepts/overview/constraints/) ( Inspired by [OPIUM](https://ranjitjhala.github.io/static/opium.pdf) )
+- Support for [collections](https://luet.io/docs/concepts/packages/collections/) and [templated package definitions](https://luet.io/docs/concepts/packages/templates/)
+- [Can be extended with Plugins and Extensions](https://luet.io/docs/concepts/plugins-and-extensions/)
+- [Can build packages in Kubernetes (experimental)](https://github.com/mudler/luet-k8s)
+- Uses containerd/go-containerregistry to manipulate images - works also daemonless with the img backend
 
 ## Install
 
-To install luet, you can grab a release on the [Release page](https://github.com/mudler/luet/releases) or compile it in your machine (requires Golang installed):
+- **Using official installer script:**
 
-    $ git clone https://github.com/mudler/luet.git
-    $ cd luet
-    $ make build
+    ```console
+    $ curl https://luet.io/install.sh | sudo sh
+    ```
 
-## Status
+- **Or using [`bin`](https://github.com/marcosnils/bin)**
 
-Luet is not feature-complete yet, it can build, install/uninstall/upgrade packages - but it doesn't support yet all the features you would normally expect from a Package Manager nowadays.
+    ```console
+    $ bin i github.com/mudler/luet
+    ```
+
+### Test installation:
+
+```console
+$ luet --help
+$ luet search ...
+$ luet install ..
+```
+
+## Documentation
+
+[Documentation](https://luet.io/) is available, or
+run `luet --help`,  any subcommand is documented as well, try e.g.: `luet build --help`.
 
 # Dependency solving
 
 Luet uses SAT and Reinforcement learning engine for dependency solving.
-It encodes the package requirements into a SAT problem, using gophersat to solve the dependency tree and give a concrete model as result.
+It encodes the package requirements into a SAT problem, using [gophersat](https://github.com/crillab/gophersat) to solve the dependency tree and give a concrete model as result.
 
 ## SAT encoding
 
@@ -48,10 +76,6 @@ when they arises while trying to validate your queries against the system model.
 
 To leverage it, simply pass ```--solver-type qlearning``` to the subcommands that supports it ( you can check out by invoking ```--help``` ).
 
-## Documentation
-
-[Documentation](https://luet-lab.github.io/docs) is available, or
-run `luet --help`,  any subcommand is documented as well, try e.g.: `luet build --help`.
 
 ## Authors
 

cmd/box/exec.go

@@ -21,8 +21,8 @@ import (
 
 	b64 "encoding/base64"
 
+	"github.com/mudler/luet/cmd/util"
 	"github.com/mudler/luet/pkg/box"
-	. "github.com/mudler/luet/pkg/logger"
 
 	"github.com/spf13/cobra"
 )
@@ -54,18 +54,18 @@ func NewBoxExecCommand() *cobra.Command {
 
 				args = ss
 			}
-			Info("Executing", args, "in", rootfs)
+			util.DefaultContext.Info("Executing", args, "in", rootfs)
 
 			b := box.NewBox(entrypoint, args, mounts, envs, rootfs, stdin, stdout, stderr)
 			err := b.Run()
 			if err != nil {
-				Fatal(err)
+				util.DefaultContext.Fatal(err)
 			}
 		},
 	}
 	path, err := os.Getwd()
 	if err != nil {
-		Fatal(err)
+		util.DefaultContext.Fatal(err)
 	}
 	ans.Flags().String("rootfs", path, "Rootfs path")
 	ans.Flags().Bool("stdin", false, "Attach to stdin")

cmd/build.go

@@ -15,15 +15,20 @@
 package cmd
 
 import (
-	"io/ioutil"
+	"fmt"
 	"os"
+	"path/filepath"
 
+	"github.com/ghodss/yaml"
 	helpers "github.com/mudler/luet/cmd/helpers"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
+	"github.com/mudler/luet/pkg/api/core/types/artifact"
 	"github.com/mudler/luet/pkg/compiler"
-	"github.com/mudler/luet/pkg/compiler/backend"
-	. "github.com/mudler/luet/pkg/config"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
+	"github.com/mudler/luet/pkg/installer"
+
+	pkg "github.com/mudler/luet/pkg/database"
+	fileHelpers "github.com/mudler/luet/pkg/helpers/file"
 	tree "github.com/mudler/luet/pkg/tree"
 
 	"github.com/spf13/cobra"
@@ -31,147 +36,206 @@ import (
 )
 
 var buildCmd = &cobra.Command{
+	// Skip processing output
+	Annotations: map[string]string{
+		util.CommandProcessOutput: "",
+	},
 	Use:   "build <package name> <package name> <package name> ...",
 	Short: "build a package or a tree",
-	Long:  `build packages or trees from luet tree definitions. Packages are in [category]/[name]-[version] form`,
-	PreRun: func(cmd *cobra.Command, args []string) {
-		viper.BindPFlag("clean", cmd.Flags().Lookup("clean"))
+	Long: `Builds one or more packages from a tree (current directory is implied):
+
+	$ luet build utils/busybox utils/yq ...
+
+Builds all packages
+
+	$ luet build --all
+
+Builds only the leaf packages:
+
+	$ luet build --full
+
+Build package revdeps:
+
+	$ luet build --revdeps utils/yq
+
+Build package without dependencies (needs the images already in the host, or either need to be available online):
+
+	$ luet build --nodeps utils/yq ...
+
+Build packages specifying multiple definition trees:
+
+	$ luet build --tree overlay/path --tree overlay/path2 utils/yq ...
+`, PreRun: func(cmd *cobra.Command, args []string) {
 		viper.BindPFlag("tree", cmd.Flags().Lookup("tree"))
 		viper.BindPFlag("destination", cmd.Flags().Lookup("destination"))
 		viper.BindPFlag("backend", cmd.Flags().Lookup("backend"))
 		viper.BindPFlag("privileged", cmd.Flags().Lookup("privileged"))
-		viper.BindPFlag("database", cmd.Flags().Lookup("database"))
 		viper.BindPFlag("revdeps", cmd.Flags().Lookup("revdeps"))
 		viper.BindPFlag("all", cmd.Flags().Lookup("all"))
 		viper.BindPFlag("compression", cmd.Flags().Lookup("compression"))
 		viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
 		viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
+		util.BindValuesFlags(cmd)
+		viper.BindPFlag("backend-args", cmd.Flags().Lookup("backend-args"))
 
 		viper.BindPFlag("image-repository", cmd.Flags().Lookup("image-repository"))
 		viper.BindPFlag("push", cmd.Flags().Lookup("push"))
 		viper.BindPFlag("pull", cmd.Flags().Lookup("pull"))
+		viper.BindPFlag("wait", cmd.Flags().Lookup("wait"))
 		viper.BindPFlag("keep-images", cmd.Flags().Lookup("keep-images"))
 
-		LuetCfg.Viper.BindPFlag("keep-exported-images", cmd.Flags().Lookup("keep-exported-images"))
+		viper.BindPFlag("backend-args", cmd.Flags().Lookup("backend-args"))
 
-		LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
-		LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
-		LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
-		LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 
-		clean := viper.GetBool("clean")
 		treePaths := viper.GetStringSlice("tree")
 		dst := viper.GetString("destination")
-		concurrency := LuetCfg.GetGeneral().Concurrency
+		concurrency := util.DefaultContext.Config.General.Concurrency
 		backendType := viper.GetString("backend")
 		privileged := viper.GetBool("privileged")
 		revdeps := viper.GetBool("revdeps")
 		all := viper.GetBool("all")
-		databaseType := viper.GetString("database")
 		compressionType := viper.GetString("compression")
 		imageRepository := viper.GetString("image-repository")
+		values := util.ValuesFlags()
+		wait := viper.GetBool("wait")
 		push := viper.GetBool("push")
 		pull := viper.GetBool("pull")
 		keepImages := viper.GetBool("keep-images")
 		nodeps := viper.GetBool("nodeps")
 		onlydeps := viper.GetBool("onlydeps")
-		keepExportedImages := viper.GetBool("keep-exported-images")
 		onlyTarget, _ := cmd.Flags().GetBool("only-target-package")
 		full, _ := cmd.Flags().GetBool("full")
-		skip, _ := cmd.Flags().GetBool("skip-if-metadata-exists")
+		rebuild, _ := cmd.Flags().GetBool("rebuild")
+		pushFinalImages, _ := cmd.Flags().GetBool("push-final-images")
+		pushFinalImagesRepository, _ := cmd.Flags().GetString("push-final-images-repository")
+		pushFinalImagesForce, _ := cmd.Flags().GetBool("push-final-images-force")
+		generateImages, _ := cmd.Flags().GetBool("generate-final-images")
 
-		compilerSpecs := compiler.NewLuetCompilationspecs()
-		var compilerBackend compiler.CompilerBackend
-		var db pkg.PackageDatabase
-		switch backendType {
-		case "img":
-			compilerBackend = backend.NewSimpleImgBackend()
-		case "docker":
-			compilerBackend = backend.NewSimpleDockerBackend()
-		}
+		backendArgs := viper.GetStringSlice("backend-args")
+		out, _ := cmd.Flags().GetString("output")
+		pretend, _ := cmd.Flags().GetBool("pretend")
+		fromRepo, _ := cmd.Flags().GetBool("from-repositories")
+		fromDockerfiles, _ := cmd.Flags().GetBool("dockerfiles")
 
-		switch databaseType {
-		case "memory":
-			db = pkg.NewInMemoryDatabase(false)
+		compilerSpecs := types.NewLuetCompilationspecs()
 
-		case "boltdb":
-			tmpdir, err := ioutil.TempDir("", "package")
-			if err != nil {
-				Fatal(err)
-			}
-			db = pkg.NewBoltDatabase(tmpdir)
+		var db types.PackageDatabase
+		var results Results
+		var templateFolders []string
 
-		}
+		compilerBackend, err := compiler.NewBackend(util.DefaultContext, backendType)
+		helpers.CheckErr(err)
+
+		db = pkg.NewInMemoryDatabase(false)
 		defer db.Clean()
 
-		generalRecipe := tree.NewCompilerRecipe(db)
+		runtimeDB := pkg.NewInMemoryDatabase(false)
+		defer runtimeDB.Clean()
 
-		if len(treePaths) <= 0 {
-			Fatal("No tree path supplied!")
+		installerRecipeParsers := tree.DefaultInstallerParsers
+		generalRecipeParsers := tree.DefaultCompilerParsers
+
+		if fromDockerfiles {
+			installerRecipeParsers = append(installerRecipeParsers, tree.RuntimeDockerfileParser)
+			generalRecipeParsers = append(generalRecipeParsers, tree.BuildDockerfileParser)
 		}
 
-		for _, src := range treePaths {
-			Info("Loading tree", src)
+		installerRecipe := tree.NewInstallerRecipe(runtimeDB, installerRecipeParsers...)
+		generalRecipe := tree.NewCompilerRecipe(db, generalRecipeParsers...)
 
-			err := generalRecipe.Load(src)
+		for _, src := range treePaths {
+			util.DefaultContext.Info("Loading tree", src)
+			helpers.CheckErr(generalRecipe.Load(src))
+			helpers.CheckErr(installerRecipe.Load(src))
+		}
+
+		if fromRepo {
+			bt, err := installer.LoadBuildTree(generalRecipe, db, util.DefaultContext)
 			if err != nil {
-				Fatal("Error: " + err.Error())
+				util.DefaultContext.Warning("errors while loading trees from repositories", err.Error())
+			}
+
+			for _, r := range bt.RepoDir {
+				helpers.CheckErr(installerRecipe.Load(r))
+			}
+
+			templateFolders = util.TemplateFolders(util.DefaultContext, bt, treePaths)
+		} else {
+			templateFolders = util.TemplateFolders(util.DefaultContext, installer.BuildTreeResult{}, treePaths)
+		}
+
+		util.DefaultContext.Info("Building in", dst)
+
+		if !fileHelpers.Exists(dst) {
+			os.MkdirAll(dst, 0600)
+			util.DefaultContext.Debug("Creating destination folder", dst)
+		}
+
+		opts := util.DefaultContext.GetConfig().Solver
+		pullRepo, _ := cmd.Flags().GetStringArray("pull-repository")
+
+		util.DefaultContext.Debug("Solver", opts.CompactString())
+
+		compileropts := []types.CompilerOption{compiler.NoDeps(nodeps),
+			compiler.WithBackendType(backendType),
+			compiler.PushImages(push),
+			compiler.WithBuildValues(values),
+			compiler.WithPullRepositories(pullRepo),
+			compiler.WithPushRepository(imageRepository),
+			compiler.Rebuild(rebuild),
+			compiler.WithTemplateFolder(templateFolders),
+			compiler.WithSolverOptions(opts),
+			compiler.Wait(wait),
+			compiler.WithRuntimeDatabase(installerRecipe.GetDatabase()),
+			compiler.OnlyTarget(onlyTarget),
+			compiler.PullFirst(pull),
+			compiler.KeepImg(keepImages),
+			compiler.OnlyDeps(onlydeps),
+			compiler.WithContext(util.DefaultContext),
+			compiler.BackendArgs(backendArgs),
+			compiler.Concurrency(concurrency),
+			compiler.WithCompressionType(types.CompressionImplementation(compressionType))}
+
+		if pushFinalImages {
+			compileropts = append(compileropts, compiler.EnablePushFinalImages)
+			if pushFinalImagesForce {
+				compileropts = append(compileropts, compiler.ForcePushFinalImages)
+			}
+			if pushFinalImagesRepository != "" {
+				compileropts = append(compileropts, compiler.WithFinalRepository(pushFinalImagesRepository))
+			} else if imageRepository != "" {
+				compileropts = append(compileropts, compiler.WithFinalRepository(imageRepository))
 			}
 		}
 
-		Info("Building in", dst)
+		if generateImages {
+			compileropts = append(compileropts, compiler.EnableGenerateFinalImages)
+		}
 
-		stype := LuetCfg.Viper.GetString("solver.type")
-		discount := LuetCfg.Viper.GetFloat64("solver.discount")
-		rate := LuetCfg.Viper.GetFloat64("solver.rate")
-		attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
+		luetCompiler := compiler.NewLuetCompiler(compilerBackend, generalRecipe.GetDatabase(), compileropts...)
 
-		LuetCfg.GetSolverOptions().Type = stype
-		LuetCfg.GetSolverOptions().LearnRate = float32(rate)
-		LuetCfg.GetSolverOptions().Discount = float32(discount)
-		LuetCfg.GetSolverOptions().MaxAttempts = attempts
-
-		Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
-
-		opts := compiler.NewDefaultCompilerOptions()
-		opts.SolverOptions = *LuetCfg.GetSolverOptions()
-		opts.ImageRepository = imageRepository
-		opts.Clean = clean
-		opts.PullFirst = pull
-		opts.KeepImg = keepImages
-		opts.Push = push
-		opts.OnlyDeps = onlydeps
-		opts.NoDeps = nodeps
-		opts.KeepImageExport = keepExportedImages
-		opts.SkipIfMetadataExists = skip
-		opts.PackageTargetOnly = onlyTarget
-
-		luetCompiler := compiler.NewLuetCompiler(compilerBackend, generalRecipe.GetDatabase(), opts)
-		luetCompiler.SetConcurrency(concurrency)
-		luetCompiler.SetCompressionType(compiler.CompressionImplementation(compressionType))
 		if full {
 			specs, err := luetCompiler.FromDatabase(generalRecipe.GetDatabase(), true, dst)
 			if err != nil {
-				Fatal(err.Error())
+				util.DefaultContext.Fatal(err.Error())
 			}
 			for _, spec := range specs {
-				Info(":package: Selecting ", spec.GetPackage().GetName(), spec.GetPackage().GetVersion())
+				util.DefaultContext.Info(":package: Selecting ", spec.GetPackage().GetName(), spec.GetPackage().GetVersion())
 
 				compilerSpecs.Add(spec)
 			}
 		} else if !all {
 			for _, a := range args {
-
 				pack, err := helpers.ParsePackageStr(a)
 				if err != nil {
-					Fatal("Invalid package string ", a, ": ", err.Error())
+					util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
 				}
 
 				spec, err := luetCompiler.FromPackage(pack)
 				if err != nil {
-					Fatal("Error: " + err.Error())
+					util.DefaultContext.Fatal("Error: " + err.Error())
 				}
 
 				spec.SetOutputPath(dst)
@@ -183,31 +247,82 @@ var buildCmd = &cobra.Command{
 			for _, p := range w {
 				spec, err := luetCompiler.FromPackage(p)
 				if err != nil {
-					Fatal("Error: " + err.Error())
+					util.DefaultContext.Fatal("Error: " + err.Error())
 				}
-				Info(":package: Selecting ", p.GetName(), p.GetVersion())
+				util.DefaultContext.Info(":package: Selecting ", p.GetName(), p.GetVersion())
 				spec.SetOutputPath(dst)
 				compilerSpecs.Add(spec)
 			}
 		}
 
-		var artifact []compiler.Artifact
+		var artifact []*artifact.PackageArtifact
 		var errs []error
 		if revdeps {
 			artifact, errs = luetCompiler.CompileWithReverseDeps(privileged, compilerSpecs)
 
-		} else {
-			artifact, errs = luetCompiler.CompileParallel(privileged, compilerSpecs)
+		} else if pretend {
+			var toCalculate []*types.LuetCompilationSpec
+			if full {
+				var err error
+				toCalculate, err = luetCompiler.ComputeMinimumCompilableSet(compilerSpecs.All()...)
+				if err != nil {
+					errs = append(errs, err)
+				}
+			} else {
+				toCalculate = compilerSpecs.All()
+			}
 
+			for _, sp := range toCalculate {
+				ht := compiler.NewHashTree(generalRecipe.GetDatabase())
+				hashTree, err := ht.Query(luetCompiler, sp)
+				if err != nil {
+					errs = append(errs, err)
+				}
+				for _, p := range hashTree.Dependencies {
+					results.Packages = append(results.Packages,
+						PackageResult{
+							Name:       p.Package.GetName(),
+							Version:    p.Package.GetVersion(),
+							Category:   p.Package.GetCategory(),
+							Repository: "",
+							Hidden:     p.Package.IsHidden(),
+							Target:     sp.GetPackage().HumanReadableString(),
+						})
+				}
+			}
+
+			y, err := yaml.Marshal(results)
+			if err != nil {
+				fmt.Printf("err: %v\n", err)
+				return
+			}
+			switch out {
+			case "yaml":
+				fmt.Println(string(y))
+			case "json":
+				j2, err := yaml.YAMLToJSON(y)
+				if err != nil {
+					fmt.Printf("err: %v\n", err)
+					return
+				}
+				fmt.Println(string(j2))
+			case "terminal":
+				for _, p := range results.Packages {
+					util.DefaultContext.Info(p.String())
+				}
+			}
+		} else {
+
+			artifact, errs = luetCompiler.CompileParallel(privileged, compilerSpecs)
 		}
 		if len(errs) != 0 {
 			for _, e := range errs {
-				Error("Error: " + e.Error())
+				util.DefaultContext.Error("Error: " + e.Error())
 			}
-			Fatal("Bailing out")
+			util.DefaultContext.Fatal("Bailing out")
 		}
 		for _, a := range artifact {
-			Info("Artifact generated:", a.GetPath())
+			util.DefaultContext.Info("Artifact generated:", a.Path)
 		}
 	},
 }
@@ -215,32 +330,45 @@ var buildCmd = &cobra.Command{
 func init() {
 	path, err := os.Getwd()
 	if err != nil {
-		Fatal(err)
+		util.DefaultContext.Fatal(err)
 	}
-	buildCmd.Flags().Bool("clean", true, "Build all packages without considering the packages present in the build directory")
-	buildCmd.Flags().StringSliceP("tree", "t", []string{}, "Path of the tree to use.")
+
+	buildCmd.Flags().StringSliceP("tree", "t", []string{path}, "Path of the tree to use.")
 	buildCmd.Flags().String("backend", "docker", "backend used (docker,img)")
-	buildCmd.Flags().Bool("privileged", false, "Privileged (Keep permissions)")
-	buildCmd.Flags().String("database", "memory", "database used for solving (memory,boltdb)")
+	buildCmd.Flags().Bool("privileged", true, "Privileged (Keep permissions)")
 	buildCmd.Flags().Bool("revdeps", false, "Build with revdeps")
 	buildCmd.Flags().Bool("all", false, "Build all specfiles in the tree")
-	buildCmd.Flags().Bool("full", false, "Build all packages (optimized)")
 
-	buildCmd.Flags().String("destination", path, "Destination folder")
-	buildCmd.Flags().String("compression", "none", "Compression alg: none, gzip")
+	buildCmd.Flags().Bool("generate-final-images", false, "Generate final images while building")
+	buildCmd.Flags().Bool("push-final-images", false, "Push final images while building")
+	buildCmd.Flags().Bool("push-final-images-force", false, "Override existing images")
+	buildCmd.Flags().String("push-final-images-repository", "", "Repository where to push final images to")
+	buildCmd.Flags().Bool("dockerfiles", false, "Source packages also from dockerfiles")
+	buildCmd.Flags().Bool("full", false, "Build all packages (optimized)")
+	buildCmd.Flags().StringSlice("values", []string{}, "Build values file to interpolate with each package")
+	buildCmd.Flags().StringSliceP("backend-args", "a", []string{}, "Backend args")
+
+	buildCmd.Flags().String("destination", filepath.Join(path, "build"), "Destination folder")
+	buildCmd.Flags().String("compression", "none", "Compression alg: none, gzip, zstd")
 	buildCmd.Flags().String("image-repository", "luet/cache", "Default base image string for generated image")
 	buildCmd.Flags().Bool("push", false, "Push images to a hub")
 	buildCmd.Flags().Bool("pull", false, "Pull images from a hub")
+	buildCmd.Flags().Bool("wait", false, "Don't build all intermediate images, but wait for them until they are available")
 	buildCmd.Flags().Bool("keep-images", true, "Keep built docker images in the host")
 	buildCmd.Flags().Bool("nodeps", false, "Build only the target packages, skipping deps (it works only if you already built the deps locally, or by using --pull) ")
 	buildCmd.Flags().Bool("onlydeps", false, "Build only package dependencies")
-	buildCmd.Flags().Bool("keep-exported-images", false, "Keep exported images used during building")
-	buildCmd.Flags().Bool("skip-if-metadata-exists", false, "Skip package if metadata exists")
 	buildCmd.Flags().Bool("only-target-package", false, "Build packages of only the required target. Otherwise builds all the necessary ones not present in the destination")
 	buildCmd.Flags().String("solver-type", "", "Solver strategy")
 	buildCmd.Flags().Float32("solver-rate", 0.7, "Solver learning rate")
 	buildCmd.Flags().Float32("solver-discount", 1.0, "Solver discount rate")
 	buildCmd.Flags().Int("solver-attempts", 9000, "Solver maximum attempts")
+	buildCmd.Flags().Bool("solver-concurrent", false, "Use concurrent solver (experimental)")
+	buildCmd.Flags().Bool("from-repositories", false, "Consume the user-defined repositories to pull specfiles from")
+	buildCmd.Flags().Bool("rebuild", false, "To combine with --pull. Allows to rebuild the target package even if an image is available, against a local values file")
+	buildCmd.Flags().Bool("pretend", false, "Just print what packages will be compiled")
+	buildCmd.Flags().StringArrayP("pull-repository", "p", []string{}, "A list of repositories to pull the cache from")
+
+	buildCmd.Flags().StringP("output", "o", "terminal", "Output format ( Defaults: terminal, available: json,yaml )")
 
 	RootCmd.AddCommand(buildCmd)
 }

cmd/cleanup.go

@@ -16,13 +16,13 @@
 package cmd
 
 import (
+	"fmt"
 	"io/ioutil"
 	"os"
 	"path/filepath"
 
-	config "github.com/mudler/luet/pkg/config"
-	"github.com/mudler/luet/pkg/helpers"
-	. "github.com/mudler/luet/pkg/logger"
+	"github.com/mudler/luet/cmd/util"
+	fileHelper "github.com/mudler/luet/pkg/helpers/file"
 
 	"github.com/spf13/cobra"
 )
@@ -31,36 +31,31 @@ var cleanupCmd = &cobra.Command{
 	Use:   "cleanup",
 	Short: "Clean packages cache.",
 	Long:  `remove downloaded packages tarballs and clean cache directory`,
+
 	Run: func(cmd *cobra.Command, args []string) {
 		var cleaned int = 0
-
 		// Check if cache dir exists
-		if helpers.Exists(config.LuetCfg.GetSystem().GetSystemPkgsCacheDirPath()) {
+		if fileHelper.Exists(util.DefaultContext.Config.System.PkgsCachePath) {
 
-			files, err := ioutil.ReadDir(config.LuetCfg.GetSystem().GetSystemPkgsCacheDirPath())
+			files, err := ioutil.ReadDir(util.DefaultContext.Config.System.PkgsCachePath)
 			if err != nil {
-				Fatal("Error on read cachedir ", err.Error())
+				util.DefaultContext.Fatal("Error on read cachedir ", err.Error())
 			}
 
 			for _, file := range files {
-				if file.IsDir() {
-					continue
-				}
 
-				if config.LuetCfg.GetGeneral().Debug {
-					Info("Removing ", file.Name())
-				}
+				util.DefaultContext.Debug("Removing ", file.Name())
 
 				err := os.RemoveAll(
-					filepath.Join(config.LuetCfg.GetSystem().GetSystemPkgsCacheDirPath(), file.Name()))
+					filepath.Join(util.DefaultContext.Config.System.PkgsCachePath, file.Name()))
 				if err != nil {
-					Fatal("Error on removing", file.Name())
+					util.DefaultContext.Fatal("Error on removing", file.Name())
 				}
 				cleaned++
 			}
 		}
 
-		Info("Cleaned: ", cleaned, "packages.")
+		util.DefaultContext.Info(fmt.Sprintf("Cleaned: %d files from %s", cleaned, util.DefaultContext.Config.System.PkgsCachePath))
 
 	},
 }

cmd/config.go

@@ -18,8 +18,7 @@ package cmd
 import (
 	"fmt"
 
-	config "github.com/mudler/luet/pkg/config"
-	installer "github.com/mudler/luet/pkg/installer"
+	"github.com/mudler/luet/cmd/util"
 
 	"github.com/spf13/cobra"
 )
@@ -30,46 +29,12 @@ var configCmd = &cobra.Command{
 	Long:    `Show luet configuration`,
 	Aliases: []string{"c"},
 	Run: func(cmd *cobra.Command, args []string) {
-		fmt.Println(config.LuetCfg.GetLogging())
-		fmt.Println(config.LuetCfg.GetGeneral())
-		fmt.Println(config.LuetCfg.GetSystem())
-		if len(config.LuetCfg.CacheRepositories) > 0 {
-			fmt.Println("repetitors:")
-			for _, r := range config.LuetCfg.CacheRepositories {
-				fmt.Println("  - ", r.String())
-			}
-		}
-		if len(config.LuetCfg.SystemRepositories) > 0 {
-			fmt.Println("repositories:")
-			for _, r := range config.LuetCfg.SystemRepositories {
-				fmt.Println("  - ", r.String())
-			}
+		data, err := util.DefaultContext.Config.YAML()
+		if err != nil {
+			util.DefaultContext.Fatal(err.Error())
 		}
 
-		if len(config.LuetCfg.RepositoriesConfDir) > 0 {
-			fmt.Println("repos_confdir:")
-			for _, dir := range config.LuetCfg.RepositoriesConfDir {
-				fmt.Println("  - ", dir)
-			}
-		}
-
-		if len(config.LuetCfg.ConfigProtectConfDir) > 0 {
-
-			// Load config protect configs
-			installer.LoadConfigProtectConfs(config.LuetCfg)
-
-			fmt.Println("config_protect_confdir:")
-			for _, dir := range config.LuetCfg.ConfigProtectConfDir {
-				fmt.Println("  - ", dir)
-			}
-
-			if len(config.LuetCfg.GetConfigProtectConfFiles()) > 0 {
-				fmt.Println("protect_conf_files:")
-				for _, file := range config.LuetCfg.GetConfigProtectConfFiles() {
-					fmt.Println("  - ", file.String())
-				}
-			}
-		}
+		fmt.Println(string(data))
 	},
 }
 

cmd/convert.go

@@ -1,101 +0,0 @@
-// Copyright © 2019 Ettore Di Giacinto <mudler@gentoo.org>
-//
-// This program is free software; you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation; either version 2 of the License, or
-// (at your option) any later version.
-//
-// This program is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-//
-// You should have received a copy of the GNU General Public License along
-// with this program; if not, see <http://www.gnu.org/licenses/>.
-package cmd
-
-import (
-	"io/ioutil"
-
-	. "github.com/mudler/luet/pkg/config"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
-	tree "github.com/mudler/luet/pkg/tree"
-
-	"github.com/mudler/luet/pkg/tree/builder/gentoo"
-	"github.com/spf13/cobra"
-	"github.com/spf13/viper"
-)
-
-var convertCmd = &cobra.Command{
-	Use:   "convert",
-	Short: "convert other package manager tree into luet",
-	Long:  `Parses external PM and produces a luet parsable tree`,
-	PreRun: func(cmd *cobra.Command, args []string) {
-		viper.BindPFlag("type", cmd.Flags().Lookup("type"))
-		viper.BindPFlag("database", cmd.Flags().Lookup("database"))
-	},
-	Run: func(cmd *cobra.Command, args []string) {
-
-		t := viper.GetString("type")
-		databaseType := viper.GetString("database")
-		var db pkg.PackageDatabase
-
-		if len(args) != 2 {
-			Fatal("Incorrect number of arguments")
-		}
-
-		input := args[0]
-		output := args[1]
-		Info("Converting trees from " + input + " [" + t + "]")
-
-		var builder tree.Parser
-		switch t {
-		case "gentoo":
-			builder = gentoo.NewGentooBuilder(
-				&gentoo.SimpleEbuildParser{},
-				LuetCfg.GetGeneral().Concurrency,
-				gentoo.InMemory)
-		default: // dup
-			builder = gentoo.NewGentooBuilder(
-				&gentoo.SimpleEbuildParser{},
-				LuetCfg.GetGeneral().Concurrency,
-				gentoo.InMemory)
-		}
-
-		switch databaseType {
-		case "memory":
-			db = pkg.NewInMemoryDatabase(false)
-		case "boltdb":
-			tmpdir, err := ioutil.TempDir("", "package")
-			if err != nil {
-				Fatal(err)
-			}
-			db = pkg.NewBoltDatabase(tmpdir)
-		}
-		defer db.Clean()
-
-		packageTree, err := builder.Generate(input)
-		if err != nil {
-			Fatal("Error: " + err.Error())
-		}
-
-		defer packageTree.Clean()
-		Info("Tree generated")
-
-		generalRecipe := tree.NewGeneralRecipe(packageTree)
-		Info("Saving generated tree to " + output)
-
-		err = generalRecipe.Save(output)
-		if err != nil {
-			Fatal("Error: " + err.Error())
-		}
-	},
-}
-
-func init() {
-	convertCmd.Flags().String("type", "gentoo", "source type")
-	convertCmd.Flags().String("database", "memory", "database used for solving (memory,boltdb)")
-
-	RootCmd.AddCommand(convertCmd)
-}

cmd/create-repo.go

@@ -16,12 +16,17 @@ package cmd
 
 import (
 	"os"
+	"path/filepath"
 
+	helpers "github.com/mudler/luet/cmd/helpers"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
 	"github.com/mudler/luet/pkg/compiler"
-	. "github.com/mudler/luet/pkg/config"
 	installer "github.com/mudler/luet/pkg/installer"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
+	"github.com/mudler/luet/pkg/tree"
+
+	//	. "github.com/mudler/luet/pkg/logger"
+	pkg "github.com/mudler/luet/pkg/database"
 
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
@@ -30,11 +35,31 @@ import (
 var createrepoCmd = &cobra.Command{
 	Use:   "create-repo",
 	Short: "Create a luet repository from a build",
-	Long:  `Generate and renew repository metadata`,
+	Long: `Builds tree metadata from a set of packages and a tree definition:
+
+	$ luet create-repo
+
+Provide specific paths for packages, tree, and metadata output which is generated:
+
+	$ luet create-repo --packages my/packages/path --tree my/tree/path --output my/packages/path ...
+
+Provide name and description of the repository:
+
+	$ luet create-repo --name "foo" --description "bar" ...
+
+Change compression method:
+	
+	$ luet create-repo --tree-compression gzip --meta-compression gzip
+
+Create a repository from the metadata description defined in the luet.yaml config file:
+
+	$ luet create-repo --repo repository1
+`,
 	PreRun: func(cmd *cobra.Command, args []string) {
 		viper.BindPFlag("packages", cmd.Flags().Lookup("packages"))
 		viper.BindPFlag("tree", cmd.Flags().Lookup("tree"))
 		viper.BindPFlag("output", cmd.Flags().Lookup("output"))
+		viper.BindPFlag("backend", cmd.Flags().Lookup("backend"))
 		viper.BindPFlag("name", cmd.Flags().Lookup("name"))
 		viper.BindPFlag("descr", cmd.Flags().Lookup("descr"))
 		viper.BindPFlag("urls", cmd.Flags().Lookup("urls"))
@@ -45,14 +70,18 @@ var createrepoCmd = &cobra.Command{
 		viper.BindPFlag("meta-filename", cmd.Flags().Lookup("meta-filename"))
 		viper.BindPFlag("reset-revision", cmd.Flags().Lookup("reset-revision"))
 		viper.BindPFlag("repo", cmd.Flags().Lookup("repo"))
+		viper.BindPFlag("from-metadata", cmd.Flags().Lookup("from-metadata"))
+		viper.BindPFlag("force-push", cmd.Flags().Lookup("force-push"))
+		viper.BindPFlag("push-images", cmd.Flags().Lookup("push-images"))
+
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 		var err error
-		var repo installer.Repository
+		var repo *installer.LuetSystemRepository
 
 		treePaths := viper.GetStringSlice("tree")
 		dst := viper.GetString("output")
-		packages := viper.GetString("packages")
+
 		name := viper.GetString("name")
 		descr := viper.GetString("descr")
 		urls := viper.GetStringSlice("urls")
@@ -63,16 +92,39 @@ var createrepoCmd = &cobra.Command{
 		metatype := viper.GetString("meta-compression")
 		metaName := viper.GetString("meta-filename")
 		source_repo := viper.GetString("repo")
+		backendType := viper.GetString("backend")
+		fromRepo, _ := cmd.Flags().GetBool("from-repositories")
+		dockerFiles, _ := cmd.Flags().GetBool("dockerfiles")
 
 		treeFile := installer.NewDefaultTreeRepositoryFile()
 		metaFile := installer.NewDefaultMetaRepositoryFile()
+		compilerBackend, err := compiler.NewBackend(util.DefaultContext, backendType)
+		helpers.CheckErr(err)
+		force := viper.GetBool("force-push")
+		imagePush := viper.GetBool("push-images")
+		snapshotID, _ := cmd.Flags().GetString("snapshot-id")
+
+		opts := []installer.RepositoryOption{
+			installer.WithSource(viper.GetString("packages")),
+			installer.WithPushImages(imagePush),
+			installer.WithForce(force),
+			installer.FromRepository(fromRepo),
+			installer.WithImagePrefix(dst),
+			installer.WithDatabase(pkg.NewInMemoryDatabase(false)),
+			installer.WithCompilerBackend(compilerBackend),
+			installer.FromMetadata(viper.GetBool("from-metadata")),
+			installer.WithContext(util.DefaultContext),
+		}
+
+		if dockerFiles {
+			opts = append(opts, installer.WithCompilerParser(append(tree.DefaultCompilerParsers, tree.BuildDockerfileParser)...))
+			opts = append(opts, installer.WithRuntimeParser(append(tree.DefaultInstallerParsers, tree.RuntimeDockerfileParser)...))
+		}
 
 		if source_repo != "" {
 			// Search for system repository
-			lrepo, err := LuetCfg.GetSystemRepository(source_repo)
-			if err != nil {
-				Fatal("Error: " + err.Error())
-			}
+			lrepo, err := util.DefaultContext.Config.GetSystemRepository(source_repo)
+			helpers.CheckErr(err)
 
 			if len(treePaths) <= 0 {
 				treePaths = []string{lrepo.TreePath}
@@ -82,25 +134,30 @@ var createrepoCmd = &cobra.Command{
 				t = lrepo.Type
 			}
 
-			repo, err = installer.GenerateRepository(lrepo.Name,
-				lrepo.Description, t,
-				lrepo.Urls,
-				lrepo.Priority,
-				packages,
-				treePaths,
-				pkg.NewInMemoryDatabase(false))
+			opts = append(opts,
+				installer.WithName(lrepo.Name),
+				installer.WithDescription(lrepo.Description),
+				installer.WithType(t),
+				installer.WithUrls(lrepo.Urls...),
+				installer.WithPriority(lrepo.Priority),
+				installer.WithTree(treePaths...),
+			)
 
 		} else {
-			repo, err = installer.GenerateRepository(name, descr, t, urls, 1, packages,
-				treePaths, pkg.NewInMemoryDatabase(false))
+			opts = append(opts,
+				installer.WithName(name),
+				installer.WithDescription(descr),
+				installer.WithType(t),
+				installer.WithUrls(urls...),
+				installer.WithTree(treePaths...),
+			)
 		}
 
-		if err != nil {
-			Fatal("Error: " + err.Error())
-		}
+		repo, err = installer.GenerateRepository(opts...)
+		helpers.CheckErr(err)
 
 		if treetype != "" {
-			treeFile.SetCompressionType(compiler.CompressionImplementation(treetype))
+			treeFile.SetCompressionType(types.CompressionImplementation(treetype))
 		}
 
 		if treeName != "" {
@@ -108,42 +165,48 @@ var createrepoCmd = &cobra.Command{
 		}
 
 		if metatype != "" {
-			metaFile.SetCompressionType(compiler.CompressionImplementation(metatype))
+			metaFile.SetCompressionType(types.CompressionImplementation(metatype))
 		}
 
 		if metaName != "" {
 			metaFile.SetFileName(metaName)
 		}
-
+		repo.SetSnapshotID(snapshotID)
 		repo.SetRepositoryFile(installer.REPOFILE_TREE_KEY, treeFile)
 		repo.SetRepositoryFile(installer.REPOFILE_META_KEY, metaFile)
 
-		err = repo.Write(dst, reset)
-		if err != nil {
-			Fatal("Error: " + err.Error())
-		}
+		err = repo.Write(util.DefaultContext, dst, reset, true)
+		helpers.CheckErr(err)
+
 	},
 }
 
 func init() {
 	path, err := os.Getwd()
-	if err != nil {
-		Fatal(err)
-	}
-	createrepoCmd.Flags().String("packages", path, "Packages folder (output from build)")
-	createrepoCmd.Flags().StringSliceP("tree", "t", []string{}, "Path of the source trees to use.")
-	createrepoCmd.Flags().String("output", path, "Destination folder")
+	helpers.CheckErr(err)
+
+	createrepoCmd.Flags().String("packages", filepath.Join(path, "build"), "Packages folder (output from build)")
+	createrepoCmd.Flags().StringSliceP("tree", "t", []string{path}, "Path of the source trees to use.")
+	createrepoCmd.Flags().String("output", filepath.Join(path, "build"), "Destination for generated archives. With 'docker' repository type, it should be an image reference (e.g 'foo/bar')")
 	createrepoCmd.Flags().String("name", "luet", "Repository name")
 	createrepoCmd.Flags().String("descr", "luet", "Repository description")
 	createrepoCmd.Flags().StringSlice("urls", []string{}, "Repository URLs")
-	createrepoCmd.Flags().String("type", "disk", "Repository type (disk)")
+	createrepoCmd.Flags().String("type", "disk", "Repository type (disk, http, docker)")
 	createrepoCmd.Flags().Bool("reset-revision", false, "Reset repository revision.")
 	createrepoCmd.Flags().String("repo", "", "Use repository defined in configuration.")
+	createrepoCmd.Flags().String("backend", "docker", "backend used (docker,img)")
+	createrepoCmd.Flags().Bool("dockerfiles", false, "Read dockerfiles in tree as packages.")
 
-	createrepoCmd.Flags().String("tree-compression", "gzip", "Compression alg: none, gzip")
+	createrepoCmd.Flags().Bool("force-push", false, "Force overwrite of docker images if already present online")
+	createrepoCmd.Flags().Bool("push-images", false, "Enable/Disable docker image push for docker repositories")
+	createrepoCmd.Flags().Bool("from-metadata", false, "Consider metadata files from the packages folder while indexing the new tree")
+
+	createrepoCmd.Flags().String("tree-compression", "gzip", "Compression alg: none, gzip, zstd")
 	createrepoCmd.Flags().String("tree-filename", installer.TREE_TARBALL, "Repository tree filename")
-	createrepoCmd.Flags().String("meta-compression", "none", "Compression alg: none, gzip")
+	createrepoCmd.Flags().String("meta-compression", "none", "Compression alg: none, gzip, zstd")
 	createrepoCmd.Flags().String("meta-filename", installer.REPOSITORY_METAFILE+".tar", "Repository metadata filename")
+	createrepoCmd.Flags().Bool("from-repositories", false, "Consume the user-defined repositories to pull specfiles from")
+	createrepoCmd.Flags().String("snapshot-id", "", "Unique ID to use when creating repository snapshots")
 
 	RootCmd.AddCommand(createrepoCmd)
 }

cmd/database.go

@@ -25,6 +25,10 @@ import (
 var databaseGroupCmd = &cobra.Command{
 	Use:   "database [command] [OPTIONS]",
 	Short: "Manage system database (dangerous commands ahead!)",
+	Long: `Allows to manipulate Luet internal database of installed packages. Use with caution!
+
+Removing packages by hand from the database can result in a broken system, and thus it's not reccomended.
+`,
 }
 
 func init() {
@@ -32,6 +36,8 @@ func init() {
 
 	databaseGroupCmd.AddCommand(
 		NewDatabaseCreateCommand(),
+		NewDatabaseGetCommand(),
 		NewDatabaseRemoveCommand(),
+		NewDatabaseShowAllCommand(),
 	)
 }

cmd/database/create.go

@@ -17,65 +17,63 @@ package cmd_database
 
 import (
 	"io/ioutil"
-	"path/filepath"
 
-	"github.com/mudler/luet/pkg/compiler"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
-
-	. "github.com/mudler/luet/pkg/config"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
+	artifact "github.com/mudler/luet/pkg/api/core/types/artifact"
 
 	"github.com/spf13/cobra"
 )
 
 func NewDatabaseCreateCommand() *cobra.Command {
-	var ans = &cobra.Command{
+	return &cobra.Command{
 		Use:   "create <artifact_metadata1.yaml> <artifact_metadata1.yaml>",
 		Short: "Insert a package in the system DB",
-		Args:  cobra.OnlyValidArgs,
-		PreRun: func(cmd *cobra.Command, args []string) {
-			LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
-			LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
+		Long: `Inserts a package in the system database:
 
-		},
+		$ luet database create foo.yaml
+
+"luet database create" injects a package in the system database without actually installing it, use it with caution.
+
+This commands takes multiple yaml input file representing package artifacts, that are usually generated while building packages.
+
+The yaml must contain the package definition, and the file list at least.
+
+For reference, inspect a "metadata.yaml" file generated while running "luet build"`,
+		Args: cobra.OnlyValidArgs,
 		Run: func(cmd *cobra.Command, args []string) {
 
-			var systemDB pkg.PackageDatabase
+			systemDB := util.SystemDB(util.DefaultContext.Config)
 
 			for _, a := range args {
 				dat, err := ioutil.ReadFile(a)
 				if err != nil {
-					Fatal("Failed reading ", a, ": ", err.Error())
+					util.DefaultContext.Fatal("Failed reading ", a, ": ", err.Error())
 				}
-				art, err := compiler.NewPackageArtifactFromYaml(dat)
+				art, err := artifact.NewPackageArtifactFromYaml(dat)
 				if err != nil {
-					Fatal("Failed reading yaml ", a, ": ", err.Error())
+					util.DefaultContext.Fatal("Failed reading yaml ", a, ": ", err.Error())
 				}
 
-				if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-					systemDB = pkg.NewBoltDatabase(
-						filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-				} else {
-					systemDB = pkg.NewInMemoryDatabase(true)
+				files := art.Files
+
+				// Check if the package is already present
+				if p, err := systemDB.FindPackage(art.CompileSpec.GetPackage()); err == nil && p.GetName() != "" {
+					util.DefaultContext.Fatal("Package", art.CompileSpec.GetPackage().HumanReadableString(),
+						" already present.")
 				}
 
-				files, err := art.FileList()
-				if err != nil {
-					Fatal("Failed getting file list for ", a, ": ", err.Error())
+				if _, err := systemDB.CreatePackage(art.CompileSpec.GetPackage()); err != nil {
+					util.DefaultContext.Fatal("Failed to create ", a, ": ", err.Error())
+				}
+				if err := systemDB.SetPackageFiles(&types.PackageFile{PackageFingerprint: art.CompileSpec.GetPackage().GetFingerPrint(), Files: files}); err != nil {
+					util.DefaultContext.Fatal("Failed setting package files for ", a, ": ", err.Error())
 				}
 
-				if _, err := systemDB.CreatePackage(art.GetCompileSpec().GetPackage()); err != nil {
-					Fatal("Failed to create ", a, ": ", err.Error())
-				}
-				if err := systemDB.SetPackageFiles(&pkg.PackageFile{PackageFingerprint: art.GetCompileSpec().GetPackage().GetFingerPrint(), Files: files}); err != nil {
-					Fatal("Failed setting package files for ", a, ": ", err.Error())
-				}
-
-				Info(art.GetCompileSpec().GetPackage().HumanReadableString(), " created")
+				util.DefaultContext.Info(art.CompileSpec.GetPackage().HumanReadableString(), " created")
 			}
 
 		},
 	}
 
-	return ans
 }

cmd/database/get.go

@@ -0,0 +1,79 @@
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd_database
+
+import (
+	"fmt"
+
+	helpers "github.com/mudler/luet/cmd/helpers"
+	"github.com/mudler/luet/cmd/util"
+	"gopkg.in/yaml.v2"
+
+	"github.com/spf13/cobra"
+)
+
+func NewDatabaseGetCommand() *cobra.Command {
+	var c = &cobra.Command{
+		Use:   "get <package>",
+		Short: "Get a package in the system DB as yaml",
+		Long: `Get a package in the system database in the YAML format:
+
+		$ luet database get system/foo
+
+To return also files:
+		$ luet database get --files system/foo`,
+		Args: cobra.OnlyValidArgs,
+
+		Run: func(cmd *cobra.Command, args []string) {
+			showFiles, _ := cmd.Flags().GetBool("files")
+
+			systemDB := util.SystemDB(util.DefaultContext.Config)
+
+			for _, a := range args {
+				pack, err := helpers.ParsePackageStr(a)
+				if err != nil {
+					continue
+				}
+
+				ps, err := systemDB.FindPackages(pack)
+				if err != nil {
+					continue
+				}
+				for _, p := range ps {
+					y, err := p.Yaml()
+					if err != nil {
+						continue
+					}
+					fmt.Println(string(y))
+					if showFiles {
+						files, err := systemDB.GetPackageFiles(p)
+						if err != nil {
+							continue
+						}
+						b, err := yaml.Marshal(files)
+						if err != nil {
+							continue
+						}
+						fmt.Println("files:\n" + string(b))
+					}
+				}
+			}
+		},
+	}
+	c.Flags().Bool("files", false, "Show package files.")
+
+	return c
+}

cmd/database/remove.go

@@ -16,54 +16,44 @@
 package cmd_database
 
 import (
-	"path/filepath"
-
-	. "github.com/mudler/luet/pkg/logger"
-
 	helpers "github.com/mudler/luet/cmd/helpers"
-	. "github.com/mudler/luet/pkg/config"
-	pkg "github.com/mudler/luet/pkg/package"
+	"github.com/mudler/luet/cmd/util"
 
 	"github.com/spf13/cobra"
 )
 
 func NewDatabaseRemoveCommand() *cobra.Command {
-	var ans = &cobra.Command{
+	return &cobra.Command{
 		Use:   "remove [package1] [package2] ...",
 		Short: "Remove a package from the system DB (forcefully - you normally don't want to do that)",
-		Args:  cobra.OnlyValidArgs,
-		PreRun: func(cmd *cobra.Command, args []string) {
-			LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
-			LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
+		Long: `Removes a package in the system database without actually uninstalling it:
+
+		$ luet database remove foo/bar
+
+This commands takes multiple packages as arguments and prunes their entries from the system database.
+`,
+		Args: cobra.OnlyValidArgs,
 
-		},
 		Run: func(cmd *cobra.Command, args []string) {
-			var systemDB pkg.PackageDatabase
+
+			systemDB := util.SystemDB(util.DefaultContext.Config)
 
 			for _, a := range args {
 				pack, err := helpers.ParsePackageStr(a)
 				if err != nil {
-					Fatal("Invalid package string ", a, ": ", err.Error())
-				}
-
-				if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-					systemDB = pkg.NewBoltDatabase(
-						filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-				} else {
-					systemDB = pkg.NewInMemoryDatabase(true)
+					util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
 				}
 
 				if err := systemDB.RemovePackage(pack); err != nil {
-					Fatal("Failed removing ", a, ": ", err.Error())
+					util.DefaultContext.Fatal("Failed removing ", a, ": ", err.Error())
 				}
 
 				if err := systemDB.RemovePackageFiles(pack); err != nil {
-					Fatal("Failed removing files for ", a, ": ", err.Error())
+					util.DefaultContext.Fatal("Failed removing files for ", a, ": ", err.Error())
 				}
 			}
 
 		},
 	}
 
-	return ans
 }

cmd/database/show_all.go

@@ -0,0 +1,68 @@
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd_database
+
+import (
+	"fmt"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
+	"github.com/spf13/cobra"
+	"gopkg.in/yaml.v2"
+	"os"
+)
+
+func NewDatabaseShowAllCommand() *cobra.Command {
+	var c = &cobra.Command{
+		Use:   "get-all-installed",
+		Short: "Show all installed packages in the system DB as yaml",
+		Args:  cobra.NoArgs,
+
+		Run: func(cmd *cobra.Command, args []string) {
+			systemDB := util.SystemDB(util.DefaultContext.Config)
+			var packages []*types.Package
+
+			packs := systemDB.GetPackages()
+			for _, p := range packs {
+				pack, _ := systemDB.GetPackage(p)
+				packages = append(packages, pack)
+			}
+			marshal, err := yaml.Marshal(packages)
+			if err != nil {
+				return
+			}
+			fmt.Println(string(marshal))
+			output, _ := cmd.Flags().GetString("output")
+			f, err := os.Create(output)
+			if err != nil {
+				fmt.Printf("Error creating file: %s\n", err)
+				return
+			}
+			_, err = f.WriteString(string(marshal))
+			if err != nil {
+				fmt.Printf("Error writing file: %s\n", err)
+				return
+			}
+			err = f.Close()
+			if err != nil {
+				fmt.Printf("Error closing file: %s\n", err)
+				return
+			}
+		},
+	}
+
+	c.Flags().String("output", "", "Save output to given file.")
+	return c
+}

cmd/exec.go

@@ -20,8 +20,8 @@ import (
 
 	b64 "encoding/base64"
 
+	"github.com/mudler/luet/cmd/util"
 	"github.com/mudler/luet/pkg/box"
-	. "github.com/mudler/luet/pkg/logger"
 	"github.com/pkg/errors"
 	"github.com/spf13/cobra"
 )
@@ -55,12 +55,12 @@ var execCmd = &cobra.Command{
 
 			args = ss
 		}
-		Info("Executing", args, "in", rootfs)
+		util.DefaultContext.Info("Executing", args, "in", rootfs)
 
 		b := box.NewBox(entrypoint, args, mounts, envs, rootfs, stdin, stdout, stderr)
 		err := b.Exec()
 		if err != nil {
-			Fatal(errors.Wrap(err, fmt.Sprintf("entrypoint: %s rootfs: %s", entrypoint, rootfs)))
+			util.DefaultContext.Fatal(errors.Wrap(err, fmt.Sprintf("entrypoint: %s rootfs: %s", entrypoint, rootfs)))
 		}
 	},
 }
@@ -68,7 +68,7 @@ var execCmd = &cobra.Command{
 func init() {
 	path, err := os.Getwd()
 	if err != nil {
-		Fatal(err)
+		util.DefaultContext.Fatal(err)
 	}
 	execCmd.Hidden = true
 	execCmd.Flags().String("rootfs", path, "Rootfs path")

cmd/helpers/cli.go

@@ -20,10 +20,12 @@ import (
 	"errors"
 	"fmt"
 	"regexp"
+	"strings"
+
+	"github.com/mudler/luet/pkg/api/core/types"
 
 	_gentoo "github.com/Sabayon/pkgs-checker/pkg/gentoo"
-	pkg "github.com/mudler/luet/pkg/package"
-	version "github.com/mudler/luet/pkg/versioner"
+	"github.com/mudler/luet/cmd/util"
 )
 
 func CreateRegexArray(rgx []string) ([]*regexp.Regexp, error) {
@@ -41,38 +43,93 @@ func CreateRegexArray(rgx []string) ([]*regexp.Regexp, error) {
 	return ans, nil
 }
 
-func ParsePackageStr(p string) (*pkg.DefaultPackage, error) {
-	gp, err := _gentoo.ParsePackageStr(p)
-	if err != nil {
-		return nil, err
+func packageData(p string) (string, string) {
+	cat := ""
+	name := ""
+	if strings.Contains(p, "/") {
+		packagedata := strings.Split(p, "/")
+		cat = packagedata[0]
+		name = packagedata[1]
+	} else {
+		name = p
 	}
-	if gp.Version == "" {
-		gp.Version = "0"
-		gp.Condition = _gentoo.PkgCondGreaterEqual
+	return cat, name
+}
+
+func packageHasGentooSelector(v string) bool {
+	return (strings.HasPrefix(v, "=") || strings.HasPrefix(v, ">") ||
+		strings.HasPrefix(v, "<"))
+}
+
+func gentooVersion(gp *_gentoo.GentooPackage) string {
+
+	condition := gp.Condition.String()
+	if condition == "=" {
+		condition = ""
 	}
 
-	pkgVersion := ""
+	pkgVersion := fmt.Sprintf("%s%s%s",
+		condition,
+		gp.Version,
+		gp.VersionSuffix,
+	)
 	if gp.VersionBuild != "" {
 		pkgVersion = fmt.Sprintf("%s%s%s+%s",
-			version.PkgSelectorConditionFromInt(gp.Condition.Int()).String(),
+			condition,
 			gp.Version,
 			gp.VersionSuffix,
 			gp.VersionBuild,
 		)
-	} else {
-		pkgVersion = fmt.Sprintf("%s%s%s",
-			version.PkgSelectorConditionFromInt(gp.Condition.Int()).String(),
-			gp.Version,
-			gp.VersionSuffix,
-		)
 	}
-
-	pack := &pkg.DefaultPackage{
-		Name:     gp.Name,
-		Category: gp.Category,
-		Version:  pkgVersion,
-		Uri:      make([]string, 0),
-	}
-
-	return pack, nil
+	return pkgVersion
+}
+
+func ParsePackageStr(p string) (*types.Package, error) {
+
+	if packageHasGentooSelector(p) {
+		gp, err := _gentoo.ParsePackageStr(p)
+		if err != nil {
+			return nil, err
+		}
+		if gp.Version == "" {
+			gp.Version = "0"
+			gp.Condition = _gentoo.PkgCondGreaterEqual
+		}
+
+		return &types.Package{
+			Name:     gp.Name,
+			Category: gp.Category,
+			Version:  gentooVersion(gp),
+			Uri:      make([]string, 0),
+		}, nil
+	}
+
+	ver := ""
+	cat := ""
+	name := ""
+
+	if strings.Contains(p, "@") {
+		packageinfo := strings.Split(p, "@")
+		ver = packageinfo[1]
+		cat, name = packageData(packageinfo[0])
+	} else {
+		cat, name = packageData(p)
+	}
+
+	if (cat != "") && ver == "" {
+		ver = ">=0"
+	}
+
+	return &types.Package{
+		Name:     name,
+		Category: cat,
+		Version:  ver,
+		Uri:      make([]string, 0),
+	}, nil
+}
+
+func CheckErr(err error) {
+	if err != nil {
+		util.DefaultContext.Fatal(err)
+	}
 }

cmd/helpers/cli_suite_test.go

@@ -0,0 +1,28 @@
+// Copyright © 2019 Ettore Di Giacinto <mudler@gentoo.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd_helpers_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestSolver(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "CLI helpers test Suite")
+}

cmd/helpers/cli_test.go

@@ -0,0 +1,102 @@
+// Copyright © 2019 Ettore Di Giacinto <mudler@gentoo.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd_helpers_test
+
+import (
+	. "github.com/mudler/luet/cmd/helpers"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("CLI Helpers", func() {
+	Context("Can parse package strings correctly", func() {
+		It("accept single package names", func() {
+			pack, err := ParsePackageStr("foo")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal(""))
+			Expect(pack.GetVersion()).To(Equal(""))
+		})
+		It("accept unversioned packages with category", func() {
+			pack, err := ParsePackageStr("cat/foo")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal(">=0"))
+		})
+		It("accept versioned packages with category", func() {
+			pack, err := ParsePackageStr("cat/foo@1.1")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal("1.1"))
+		})
+		It("accept versioned ranges with category", func() {
+			pack, err := ParsePackageStr("cat/foo@>=1.1")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal(">=1.1"))
+		})
+		It("accept gentoo regex parsing without versions", func() {
+			pack, err := ParsePackageStr("=cat/foo")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal(">=0"))
+		})
+		It("accept gentoo regex parsing with versions", func() {
+			pack, err := ParsePackageStr("=cat/foo-1.2")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal("1.2"))
+		})
+
+		It("accept gentoo regex parsing with with condition", func() {
+			pack, err := ParsePackageStr(">=cat/foo-1.2")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal(">=1.2"))
+		})
+
+		It("accept gentoo regex parsing with with condition2", func() {
+			pack, err := ParsePackageStr("<cat/foo-1.2")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal("<1.2"))
+		})
+
+		It("accept gentoo regex parsing with with condition3", func() {
+			pack, err := ParsePackageStr(">cat/foo-1.2")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal(">1.2"))
+		})
+
+		It("accept gentoo regex parsing with with condition4", func() {
+			pack, err := ParsePackageStr("<=cat/foo-1.2")
+			Expect(err).ToNot(HaveOccurred())
+			Expect(pack.GetName()).To(Equal("foo"))
+			Expect(pack.GetCategory()).To(Equal("cat"))
+			Expect(pack.GetVersion()).To(Equal("<=1.2"))
+		})
+	})
+})

cmd/install.go

@@ -15,112 +15,99 @@
 package cmd
 
 import (
-	"os"
-	"path/filepath"
-
+	"github.com/mudler/luet/pkg/api/core/types"
 	installer "github.com/mudler/luet/pkg/installer"
 
 	helpers "github.com/mudler/luet/cmd/helpers"
-	. "github.com/mudler/luet/pkg/config"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
+	"github.com/mudler/luet/cmd/util"
 
 	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
 )
 
 var installCmd = &cobra.Command{
-	Use:     "install <pkg1> <pkg2> ...",
-	Short:   "Install a package",
+	Use:   "install <pkg1> <pkg2> ...",
+	Short: "Install a package",
+	Long: `Installs one or more packages without asking questions:
+
+	$ luet install -y utils/busybox utils/yq ...
+	
+To install only deps of a package:
+	
+	$ luet install --onlydeps utils/busybox ...
+	
+To not install deps of a package:
+	
+	$ luet install --nodeps utils/busybox ...
+
+To force install a package:
+	
+	$ luet install --force utils/busybox ...
+`,
 	Aliases: []string{"i"},
 	PreRun: func(cmd *cobra.Command, args []string) {
-		LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
-		LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
-		LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
-		LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
-		LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
-		LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
-		LuetCfg.Viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
-		LuetCfg.Viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
-		LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
+		viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
+		viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
 	},
-	Long: `Install packages in parallel`,
 	Run: func(cmd *cobra.Command, args []string) {
-		var toInstall pkg.Packages
-		var systemDB pkg.PackageDatabase
+		var toInstall types.Packages
 
 		for _, a := range args {
 			pack, err := helpers.ParsePackageStr(a)
 			if err != nil {
-				Fatal("Invalid package string ", a, ": ", err.Error())
+				util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
 			}
 			toInstall = append(toInstall, pack)
 		}
 
-		// This shouldn't be necessary, but we need to unmarshal the repositories to a concrete struct, thus we need to port them back to the Repositories type
-		repos := installer.Repositories{}
-		for _, repo := range LuetCfg.SystemRepositories {
-			if !repo.Enable {
-				continue
-			}
-			r := installer.NewSystemRepository(repo)
-			repos = append(repos, r)
-		}
+		force := viper.GetBool("force")
+		nodeps := viper.GetBool("nodeps")
+		onlydeps := viper.GetBool("onlydeps")
+		yes := viper.GetBool("yes")
+		downloadOnly, _ := cmd.Flags().GetBool("download-only")
+		relax, _ := cmd.Flags().GetBool("relax")
 
-		stype := LuetCfg.Viper.GetString("solver.type")
-		discount := LuetCfg.Viper.GetFloat64("solver.discount")
-		rate := LuetCfg.Viper.GetFloat64("solver.rate")
-		attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
-		force := LuetCfg.Viper.GetBool("force")
-		nodeps := LuetCfg.Viper.GetBool("nodeps")
-		onlydeps := LuetCfg.Viper.GetBool("onlydeps")
-		LuetCfg.GetSolverOptions().Type = stype
-		LuetCfg.GetSolverOptions().LearnRate = float32(rate)
-		LuetCfg.GetSolverOptions().Discount = float32(discount)
-		LuetCfg.GetSolverOptions().MaxAttempts = attempts
-
-		Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
-
-		// Load config protect configs
-		installer.LoadConfigProtectConfs(LuetCfg)
+		util.DefaultContext.Debug("Solver", util.DefaultContext.Config.Solver.CompactString())
 
 		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
-			Concurrency:                 LuetCfg.GetGeneral().Concurrency,
-			SolverOptions:               *LuetCfg.GetSolverOptions(),
+			Concurrency:                 util.DefaultContext.Config.General.Concurrency,
+			SolverOptions:               util.DefaultContext.Config.Solver,
 			NoDeps:                      nodeps,
 			Force:                       force,
 			OnlyDeps:                    onlydeps,
 			PreserveSystemEssentialData: true,
+			DownloadOnly:                downloadOnly,
+			Ask:                         !yes,
+			Relaxed:                     relax,
+			PackageRepositories:         util.DefaultContext.Config.SystemRepositories,
+			Context:                     util.DefaultContext,
 		})
-		inst.Repositories(repos)
 
-		if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-			systemDB = pkg.NewBoltDatabase(
-				filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-		} else {
-			systemDB = pkg.NewInMemoryDatabase(true)
+		system := &installer.System{
+			Database: util.SystemDB(util.DefaultContext.Config),
+			Target:   util.DefaultContext.Config.System.Rootfs,
 		}
-		system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
 		err := inst.Install(toInstall, system)
 		if err != nil {
-			Fatal("Error: " + err.Error())
+			util.DefaultContext.Fatal("Error: " + err.Error())
 		}
 	},
 }
 
 func init() {
-	path, err := os.Getwd()
-	if err != nil {
-		Fatal(err)
-	}
-	installCmd.Flags().String("system-dbpath", path, "System db path")
-	installCmd.Flags().String("system-target", path, "System rootpath")
-	installCmd.Flags().String("solver-type", "", "Solver strategy ( Defaults none, available: "+AvailableResolvers+" )")
-	installCmd.Flags().Float32("solver-rate", 0.7, "Solver learning rate")
-	installCmd.Flags().Float32("solver-discount", 1.0, "Solver discount rate")
-	installCmd.Flags().Int("solver-attempts", 9000, "Solver maximum attempts")
+
 	installCmd.Flags().Bool("nodeps", false, "Don't consider package dependencies (harmful!)")
+	installCmd.Flags().Bool("relax", false, "Relax installation constraints")
+
 	installCmd.Flags().Bool("onlydeps", false, "Consider **only** package dependencies")
 	installCmd.Flags().Bool("force", false, "Skip errors and keep going (potentially harmful)")
+	installCmd.Flags().Bool("solver-concurrent", false, "Use concurrent solver (experimental)")
+	installCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
+	installCmd.Flags().Bool("download-only", false, "Download only")
+	installCmd.Flags().StringArray("finalizer-env", []string{},
+		"Set finalizer environment in the format key=value.")
 
 	RootCmd.AddCommand(installCmd)
 }

cmd/oscheck.go

@@ -0,0 +1,124 @@
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+package cmd
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/mudler/luet/pkg/api/core/types"
+	installer "github.com/mudler/luet/pkg/installer"
+
+	"github.com/mudler/luet/cmd/util"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+var osCheckCmd = &cobra.Command{
+	Use:   "oscheck",
+	Short: "Checks packages integrity",
+	Long: `List packages that are installed in the system which files are missing in the system.
+
+	$ luet oscheck
+	
+To reinstall packages in the list:
+	
+	$ luet oscheck --reinstall
+`,
+	Aliases: []string{"i"},
+	PreRun: func(cmd *cobra.Command, args []string) {
+		viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
+		viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
+		viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
+	},
+	Run: func(cmd *cobra.Command, args []string) {
+
+		force := viper.GetBool("force")
+		onlydeps := viper.GetBool("onlydeps")
+		yes := viper.GetBool("yes")
+
+		downloadOnly, _ := cmd.Flags().GetBool("download-only")
+
+		system := &installer.System{
+			Database: util.SystemDB(util.DefaultContext.Config),
+			Target:   util.DefaultContext.Config.System.Rootfs,
+		}
+		packs := system.OSCheck(util.DefaultContext)
+		if !util.DefaultContext.Config.General.Quiet {
+			if len(packs) == 0 {
+				util.DefaultContext.Success("All good!")
+				os.Exit(0)
+			} else {
+				util.DefaultContext.Info("Following packages are missing files or are incomplete:")
+				for _, p := range packs {
+					util.DefaultContext.Info(p.HumanReadableString())
+				}
+			}
+		} else {
+			var s []string
+			for _, p := range packs {
+				s = append(s, p.HumanReadableString())
+			}
+			fmt.Println(strings.Join(s, " "))
+		}
+
+		reinstall, _ := cmd.Flags().GetBool("reinstall")
+		if reinstall {
+
+			// Strip version for reinstall
+			toInstall := types.Packages{}
+			for _, p := range packs {
+				new := p.Clone()
+				new.SetVersion(">=0")
+				toInstall = append(toInstall, new)
+			}
+
+			util.DefaultContext.Debug("Solver", util.DefaultContext.Config.Solver.CompactString())
+
+			inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
+				Concurrency:                 util.DefaultContext.Config.General.Concurrency,
+				SolverOptions:               util.DefaultContext.Config.Solver,
+				NoDeps:                      true,
+				Force:                       force,
+				OnlyDeps:                    onlydeps,
+				PreserveSystemEssentialData: true,
+				Ask:                         !yes,
+				DownloadOnly:                downloadOnly,
+				Context:                     util.DefaultContext,
+				PackageRepositories:         util.DefaultContext.Config.SystemRepositories,
+			})
+
+			err := inst.Swap(packs, toInstall, system)
+			if err != nil {
+				util.DefaultContext.Fatal("Error: " + err.Error())
+			}
+		}
+	},
+}
+
+func init() {
+
+	osCheckCmd.Flags().Bool("reinstall", false, "reinstall")
+
+	osCheckCmd.Flags().Bool("onlydeps", false, "Consider **only** package dependencies")
+	osCheckCmd.Flags().Bool("force", false, "Skip errors and keep going (potentially harmful)")
+	osCheckCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
+	osCheckCmd.Flags().Bool("download-only", false, "Download only")
+
+	RootCmd.AddCommand(osCheckCmd)
+}

cmd/pack.go

@@ -0,0 +1,98 @@
+// Copyright © 2019 Ettore Di Giacinto <mudler@gentoo.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+package cmd
+
+import (
+	"os"
+	"path/filepath"
+	"time"
+
+	helpers "github.com/mudler/luet/cmd/helpers"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
+	"github.com/mudler/luet/pkg/api/core/types/artifact"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+var packCmd = &cobra.Command{
+	Use:   "pack <package name>",
+	Short: "pack a custom package",
+	Long: `Pack creates a package from a directory, generating the metadata required from a tree to generate a repository.
+
+Pack can be used to manually replace what "luet build" does automatically by reading the packages build.yaml files.
+
+	$ mkdir -p output/etc/foo
+	$ echo "my config" > output/etc/foo
+	$ luet pack foo/bar@1.1 --source output
+
+Afterwards, you can use the content generated and associate it with a tree and a corresponding definition.yaml file with "luet create-repo".
+`,
+	PreRun: func(cmd *cobra.Command, args []string) {
+		viper.BindPFlag("destination", cmd.Flags().Lookup("destination"))
+		viper.BindPFlag("compression", cmd.Flags().Lookup("compression"))
+		viper.BindPFlag("source", cmd.Flags().Lookup("source"))
+	},
+	Run: func(cmd *cobra.Command, args []string) {
+		sourcePath := viper.GetString("source")
+
+		dst := viper.GetString("destination")
+		compressionType := viper.GetString("compression")
+		concurrency := util.DefaultContext.Config.General.Concurrency
+
+		if len(args) != 1 {
+			util.DefaultContext.Fatal("You must specify a package name")
+		}
+
+		packageName := args[0]
+
+		p, err := helpers.ParsePackageStr(packageName)
+		if err != nil {
+			util.DefaultContext.Fatal("Invalid package string ", packageName, ": ", err.Error())
+		}
+
+		spec := &types.LuetCompilationSpec{Package: p}
+		a := artifact.NewPackageArtifact(filepath.Join(dst, p.GetFingerPrint()+".package.tar"))
+		a.CompressionType = types.CompressionImplementation(compressionType)
+		err = a.Compress(sourcePath, concurrency)
+		if err != nil {
+			util.DefaultContext.Fatal("failed compressing ", packageName, ": ", err.Error())
+		}
+		a.CompileSpec = spec
+		filelist, err := a.FileList()
+		if err != nil {
+			util.DefaultContext.Fatal("failed generating file list for ", packageName, ": ", err.Error())
+		}
+		a.Files = filelist
+		a.CompileSpec.GetPackage().SetBuildTimestamp(time.Now().String())
+		err = a.WriteYAML(dst)
+		if err != nil {
+			util.DefaultContext.Fatal("failed writing metadata yaml file for ", packageName, ": ", err.Error())
+		}
+	},
+}
+
+func init() {
+	path, err := os.Getwd()
+	if err != nil {
+		util.DefaultContext.Fatal(err)
+	}
+	packCmd.Flags().String("source", path, "Source folder")
+	packCmd.Flags().String("destination", path, "Destination folder")
+	packCmd.Flags().String("compression", "gzip", "Compression alg: none, gzip")
+
+	RootCmd.AddCommand(packCmd)
+}

cmd/reclaim.go

@@ -15,73 +15,52 @@
 package cmd
 
 import (
-	"os"
-	"path/filepath"
-
+	"github.com/mudler/luet/cmd/util"
 	installer "github.com/mudler/luet/pkg/installer"
 
-	. "github.com/mudler/luet/pkg/config"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
-
 	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
 )
 
 var reclaimCmd = &cobra.Command{
 	Use:   "reclaim",
 	Short: "Reclaim packages to Luet database from available repositories",
 	PreRun: func(cmd *cobra.Command, args []string) {
-		LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
-		LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
-		LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("force", cmd.Flags().Lookup("force"))
 	},
-	Long: `Add packages to the systemdb if files belonging to packages 
-in available repositories exists in the target root.`,
+	Long: `Reclaim tries to find association between packages in the online repositories and the system one.
+
+	$ luet reclaim
+
+It scans the target file system, and if finds a match with a package available in the repositories, it marks as installed in the system database.
+`,
 	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
 
-		// This shouldn't be necessary, but we need to unmarshal the repositories to a concrete struct, thus we need to port them back to the Repositories type
-		repos := installer.Repositories{}
-		for _, repo := range LuetCfg.SystemRepositories {
-			if !repo.Enable {
-				continue
-			}
-			r := installer.NewSystemRepository(repo)
-			repos = append(repos, r)
-		}
+		force := viper.GetBool("force")
 
-		force := LuetCfg.Viper.GetBool("force")
-
-		Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
+		util.DefaultContext.Debug("Solver", util.DefaultContext.Config.Solver.CompactString())
 
 		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
-			Concurrency:                 LuetCfg.GetGeneral().Concurrency,
+			Concurrency:                 util.DefaultContext.Config.General.Concurrency,
 			Force:                       force,
 			PreserveSystemEssentialData: true,
+			PackageRepositories:         util.DefaultContext.Config.SystemRepositories,
+			Context:                     util.DefaultContext,
 		})
-		inst.Repositories(repos)
 
-		if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-			systemDB = pkg.NewBoltDatabase(
-				filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-		} else {
-			systemDB = pkg.NewInMemoryDatabase(true)
+		system := &installer.System{
+			Database: util.SystemDB(util.DefaultContext.Config),
+			Target:   util.DefaultContext.Config.System.Rootfs,
 		}
-		system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
 		err := inst.Reclaim(system)
 		if err != nil {
-			Fatal("Error: " + err.Error())
+			util.DefaultContext.Fatal("Error: " + err.Error())
 		}
 	},
 }
 
 func init() {
-	path, err := os.Getwd()
-	if err != nil {
-		Fatal(err)
-	}
-	reclaimCmd.Flags().String("system-dbpath", path, "System db path")
-	reclaimCmd.Flags().String("system-target", path, "System rootpath")
+
 	reclaimCmd.Flags().Bool("force", false, "Skip errors and keep going (potentially harmful)")
 
 	RootCmd.AddCommand(reclaimCmd)

cmd/reinstall.go

@@ -0,0 +1,103 @@
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+package cmd
+
+import (
+	"github.com/mudler/luet/pkg/api/core/types"
+	installer "github.com/mudler/luet/pkg/installer"
+
+	helpers "github.com/mudler/luet/cmd/helpers"
+	"github.com/mudler/luet/cmd/util"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+var reinstallCmd = &cobra.Command{
+	Use:   "reinstall <pkg1> <pkg2> <pkg3>",
+	Short: "reinstall a set of packages",
+	Long: `Reinstall a group of packages in the system:
+
+	$ luet reinstall -y system/busybox shells/bash system/coreutils ...
+`,
+	PreRun: func(cmd *cobra.Command, args []string) {
+		viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
+		viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("for", cmd.Flags().Lookup("for"))
+
+		viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
+	},
+	Run: func(cmd *cobra.Command, args []string) {
+		var toUninstall types.Packages
+		var toAdd types.Packages
+
+		force := viper.GetBool("force")
+		onlydeps := viper.GetBool("onlydeps")
+		yes := viper.GetBool("yes")
+
+		downloadOnly, _ := cmd.Flags().GetBool("download-only")
+		installed, _ := cmd.Flags().GetBool("installed")
+
+		util.DefaultContext.Debug("Solver", util.DefaultContext.Config.Solver.CompactString())
+
+		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
+			Concurrency:                 util.DefaultContext.Config.General.Concurrency,
+			SolverOptions:               util.DefaultContext.Config.Solver,
+			NoDeps:                      true,
+			Force:                       force,
+			OnlyDeps:                    onlydeps,
+			PreserveSystemEssentialData: true,
+			Ask:                         !yes,
+			DownloadOnly:                downloadOnly,
+			Context:                     util.DefaultContext,
+			PackageRepositories:         util.DefaultContext.Config.SystemRepositories,
+		})
+
+		system := &installer.System{Database: util.SystemDB(util.DefaultContext.Config), Target: util.DefaultContext.Config.System.Rootfs}
+
+		if installed {
+			for _, p := range system.Database.World() {
+				toUninstall = append(toUninstall, p)
+				c := p.Clone()
+				c.SetVersion(">=0")
+				toAdd = append(toAdd, c)
+			}
+		} else {
+			for _, a := range args {
+				pack, err := helpers.ParsePackageStr(a)
+				if err != nil {
+					util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
+				}
+				toUninstall = append(toUninstall, pack)
+				toAdd = append(toAdd, pack)
+			}
+		}
+
+		err := inst.Swap(toUninstall, toAdd, system)
+		if err != nil {
+			util.DefaultContext.Fatal("Error: " + err.Error())
+		}
+	},
+}
+
+func init() {
+	reinstallCmd.Flags().Bool("onlydeps", false, "Consider **only** package dependencies")
+	reinstallCmd.Flags().Bool("force", false, "Skip errors and keep going (potentially harmful)")
+	reinstallCmd.Flags().Bool("installed", false, "Reinstall installed packages")
+	reinstallCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
+	reinstallCmd.Flags().Bool("download-only", false, "Download only")
+
+	RootCmd.AddCommand(reinstallCmd)
+}

cmd/replace.go

@@ -0,0 +1,106 @@
+// Copyright © 2020 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+package cmd
+
+import (
+	"github.com/mudler/luet/pkg/api/core/types"
+	installer "github.com/mudler/luet/pkg/installer"
+
+	helpers "github.com/mudler/luet/cmd/helpers"
+	"github.com/mudler/luet/cmd/util"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+var replaceCmd = &cobra.Command{
+	Use:     "replace <pkg1> <pkg2> --for <pkg3> --for <pkg4> ...",
+	Short:   "replace a set of packages",
+	Aliases: []string{"r"},
+	Long: `Replaces one or a group of packages without asking questions:
+
+	$ luet replace -y system/busybox ... --for shells/bash --for system/coreutils ...
+`,
+	PreRun: func(cmd *cobra.Command, args []string) {
+		viper.BindPFlag("onlydeps", cmd.Flags().Lookup("onlydeps"))
+		viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
+		viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("for", cmd.Flags().Lookup("for"))
+
+		viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
+	},
+	Run: func(cmd *cobra.Command, args []string) {
+		var toUninstall types.Packages
+		var toAdd types.Packages
+
+		f := viper.GetStringSlice("for")
+		force := viper.GetBool("force")
+		nodeps := viper.GetBool("nodeps")
+		onlydeps := viper.GetBool("onlydeps")
+		yes := viper.GetBool("yes")
+		downloadOnly, _ := cmd.Flags().GetBool("download-only")
+
+		for _, a := range args {
+			pack, err := helpers.ParsePackageStr(a)
+			if err != nil {
+				util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
+			}
+			toUninstall = append(toUninstall, pack)
+		}
+
+		for _, a := range f {
+			pack, err := helpers.ParsePackageStr(a)
+			if err != nil {
+				util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
+			}
+			toAdd = append(toAdd, pack)
+		}
+
+		util.DefaultContext.Config.Solver.Implementation = types.SolverSingleCoreSimple
+
+		util.DefaultContext.Debug("Solver", util.DefaultContext.Config.Solver.CompactString())
+
+		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
+			Concurrency:                 util.DefaultContext.Config.General.Concurrency,
+			SolverOptions:               util.DefaultContext.Config.Solver,
+			NoDeps:                      nodeps,
+			Force:                       force,
+			OnlyDeps:                    onlydeps,
+			PreserveSystemEssentialData: true,
+			Ask:                         !yes,
+			DownloadOnly:                downloadOnly,
+			PackageRepositories:         util.DefaultContext.Config.SystemRepositories,
+			Context:                     util.DefaultContext,
+		})
+
+		system := &installer.System{Database: util.SystemDB(util.DefaultContext.Config), Target: util.DefaultContext.Config.System.Rootfs}
+		err := inst.Swap(toUninstall, toAdd, system)
+		if err != nil {
+			util.DefaultContext.Fatal("Error: " + err.Error())
+		}
+	},
+}
+
+func init() {
+
+	replaceCmd.Flags().Bool("nodeps", false, "Don't consider package dependencies (harmful!)")
+	replaceCmd.Flags().Bool("onlydeps", false, "Consider **only** package dependencies")
+	replaceCmd.Flags().Bool("force", false, "Skip errors and keep going (potentially harmful)")
+	replaceCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
+	replaceCmd.Flags().StringSlice("for", []string{}, "Packages that has to be installed in place of others")
+	replaceCmd.Flags().Bool("download-only", false, "Download only")
+
+	RootCmd.AddCommand(replaceCmd)
+}

cmd/repo.go

@@ -31,6 +31,8 @@ func init() {
 	RootCmd.AddCommand(repoGroupCmd)
 
 	repoGroupCmd.AddCommand(
+		NewRepoAddCommand(),
+		NewRepoGetCommand(),
 		NewRepoListCommand(),
 		NewRepoUpdateCommand(),
 	)

cmd/repo/add.go

@@ -0,0 +1,151 @@
+// Copyright © 2022 Ettore Di Giacinto <mudler@luet.io>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd_repo
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+
+	"github.com/ghodss/yaml"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
+	"github.com/mudler/luet/pkg/helpers"
+
+	"github.com/spf13/cobra"
+)
+
+func NewRepoAddCommand() *cobra.Command {
+	cmd := &cobra.Command{
+		Use:   "add [OPTIONS] https://..../something.yaml /local/file.yaml",
+		Short: "Add a repository to the system",
+		Args:  cobra.ExactArgs(1),
+		Long: `
+Adds a repository to the system. URLs, local files or inline repo can be specified, examples:
+
+# URL/File:
+
+ luet repo add /path/to/file
+
+ luet repo add https://....
+
+ luet repo add ... --name "foo"
+
+# Inline (provided you have $PASSWORD environent variable set):
+
+ luet repo add testfo --description "Bar" --url "FOZZ" --type "ff" --username "user" --passwd $(echo "$PASSWORD")
+
+ `,
+		Run: func(cmd *cobra.Command, args []string) {
+
+			uri := args[0]
+			d, _ := cmd.Flags().GetString("dir")
+			yes, _ := cmd.Flags().GetBool("yes")
+
+			desc, _ := cmd.Flags().GetString("description")
+			t, _ := cmd.Flags().GetString("type")
+			url, _ := cmd.Flags().GetString("url")
+			ref, _ := cmd.Flags().GetString("reference")
+			prio, _ := cmd.Flags().GetInt("priority")
+			username, _ := cmd.Flags().GetString("username")
+			passwd, _ := cmd.Flags().GetString("passwd")
+
+			if len(util.DefaultContext.Config.RepositoriesConfDir) == 0 && d == "" {
+				util.DefaultContext.Fatal("No repository dirs defined")
+				return
+			}
+			if d == "" {
+				d = util.DefaultContext.Config.RepositoriesConfDir[0]
+			}
+
+			var r *types.LuetRepository
+			str, err := helpers.GetURI(uri)
+			if err != nil {
+				r = &types.LuetRepository{
+					Enable:      true,
+					Cached:      true,
+					Name:        uri,
+					Description: desc,
+					ReferenceID: ref,
+					Type:        t,
+					Urls:        []string{url},
+					Priority:    prio,
+					Authentication: map[string]string{
+						"username": username,
+						"password": passwd,
+					},
+				}
+			} else {
+				r, err = types.LoadRepository([]byte(str))
+				if err != nil {
+					util.DefaultContext.Fatal(err)
+				}
+				if desc != "" {
+					r.Description = desc
+				}
+				if ref != "" {
+					r.ReferenceID = ref
+				}
+				if t != "" {
+					r.Type = t
+				}
+				if url != "" {
+					r.Urls = []string{url}
+				}
+				if prio != 0 {
+					r.Priority = prio
+				}
+				if username != "" && passwd != "" {
+					r.Authentication = map[string]string{
+						"username": username,
+						"password": passwd,
+					}
+				}
+			}
+
+			file := filepath.Join(util.DefaultContext.Config.System.Rootfs, d, fmt.Sprintf("%s.yaml", r.Name))
+
+			b, err := yaml.Marshal(r)
+			if err != nil {
+				util.DefaultContext.Fatal(err)
+			}
+
+			util.DefaultContext.Infof("Adding repository to the sytem as %s", file)
+			fmt.Println(string(b))
+			util.DefaultContext.Info(r.String())
+
+			if !yes && !util.DefaultContext.Ask() {
+				util.DefaultContext.Info("Aborted by user")
+				return
+			}
+
+			if err := ioutil.WriteFile(file, b, os.ModePerm); err != nil {
+				util.DefaultContext.Fatal(err)
+			}
+		},
+	}
+	cmd.Flags().BoolP("yes", "y", false, "Assume yes to questions")
+	cmd.Flags().StringP("dir", "o", "", "Folder to write to")
+	cmd.Flags().String("description", "", "Repository description")
+	cmd.Flags().String("type", "", "Repository type")
+	cmd.Flags().String("url", "", "Repository URL")
+	cmd.Flags().String("reference", "", "Repository Reference ID")
+	cmd.Flags().IntP("priority", "p", 99, "repository prio")
+	cmd.Flags().String("username", "", "repository username")
+	cmd.Flags().String("passwd", "", "repository password")
+	return cmd
+}

cmd/repo/get.go

@@ -0,0 +1,62 @@
+// Copyright © 2022 Ettore Di Giacinto <mudler@luet.io>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd_repo
+
+import (
+	"encoding/json"
+	"fmt"
+	"strings"
+
+	"github.com/ghodss/yaml"
+	"github.com/mudler/luet/cmd/util"
+
+	"github.com/spf13/cobra"
+)
+
+func NewRepoGetCommand() *cobra.Command {
+	var ans = &cobra.Command{
+		Use:   "get [OPTIONS] name",
+		Short: "get repository in the system",
+		Args:  cobra.ExactArgs(1),
+		PreRun: func(cmd *cobra.Command, args []string) {
+		},
+		Run: func(cmd *cobra.Command, args []string) {
+			o, _ := cmd.Flags().GetString("output")
+
+			for _, repo := range util.DefaultContext.Config.SystemRepositories {
+				if repo.Name != args[0] {
+					continue
+				}
+
+				switch strings.ToLower(o) {
+				case "json":
+					b, _ := json.Marshal(repo)
+					fmt.Println(string(b))
+				case "yaml":
+					b, _ := yaml.Marshal(repo)
+					fmt.Println(string(b))
+				default:
+					fmt.Println(repo)
+				}
+				break
+			}
+		},
+	}
+
+	ans.Flags().StringP("output", "o", "", "output format (json, yaml, text)")
+
+	return ans
+}

cmd/repo/list.go

@@ -22,10 +22,10 @@ import (
 	"strconv"
 	"time"
 
-	. "github.com/mudler/luet/pkg/config"
+	"github.com/mudler/luet/cmd/util"
 	installer "github.com/mudler/luet/pkg/installer"
+	"github.com/pterm/pterm"
 
-	. "github.com/logrusorgru/aurora"
 	"github.com/spf13/cobra"
 )
 
@@ -43,7 +43,7 @@ func NewRepoListCommand() *cobra.Command {
 			quiet, _ := cmd.Flags().GetBool("quiet")
 			repoType, _ := cmd.Flags().GetString("type")
 
-			for _, repo := range LuetCfg.SystemRepositories {
+			for _, repo := range util.DefaultContext.Config.SystemRepositories {
 				if enable && !repo.Enable {
 					continue
 				}
@@ -58,26 +58,26 @@ func NewRepoListCommand() *cobra.Command {
 					fmt.Println(repo.Name)
 				} else {
 					if repo.Enable {
-						repoColor = Bold(BrightGreen(repo.Name)).String()
+						repoColor = pterm.LightGreen(repo.Name)
 					} else {
-						repoColor = Bold(BrightRed(repo.Name)).String()
+						repoColor = pterm.LightRed(repo.Name)
 					}
 					if repo.Description != "" {
-						repoText = Yellow(repo.Description).String()
+						repoText = pterm.LightYellow(repo.Description)
 					} else {
-						repoText = Yellow(repo.Urls[0]).String()
+						repoText = pterm.LightYellow(repo.Urls[0])
 					}
 
-					repobasedir := LuetCfg.GetSystem().GetRepoDatabaseDirPath(repo.Name)
+					repobasedir := util.DefaultContext.Config.System.GetRepoDatabaseDirPath(repo.Name)
 					if repo.Cached {
 
 						r := installer.NewSystemRepository(repo)
-						localRepo, _ := r.(*installer.LuetSystemRepository).ReadSpecFile(filepath.Join(repobasedir,
-							installer.REPOSITORY_SPECFILE), false)
+						localRepo, _ := r.ReadSpecFile(filepath.Join(repobasedir,
+							installer.REPOSITORY_SPECFILE))
 						if localRepo != nil {
 							tsec, _ := strconv.ParseInt(localRepo.GetLastUpdate(), 10, 64)
-							repoRevision = Bold(Red(localRepo.GetRevision())).String() +
-								" - " + Bold(Green(time.Unix(tsec, 0).String())).String()
+							repoRevision = pterm.LightRed(localRepo.GetRevision()) +
+								" - " + pterm.LightGreen(time.Unix(tsec, 0).String())
 						}
 					}
 

cmd/repo/update.go

@@ -1,5 +1,6 @@
 // Copyright © 2019 Ettore Di Giacinto <mudler@gentoo.org>
 //                  Daniele Rondina <geaaru@sabayonlinux.org>
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
 //
 // This program is free software; you can redistribute it and/or modify
 // it under the terms of the GNU General Public License as published by
@@ -17,15 +18,14 @@
 package cmd_repo
 
 import (
-	. "github.com/mudler/luet/pkg/config"
+	"github.com/mudler/luet/cmd/util"
 	installer "github.com/mudler/luet/pkg/installer"
-	. "github.com/mudler/luet/pkg/logger"
 
 	"github.com/spf13/cobra"
 )
 
 func NewRepoUpdateCommand() *cobra.Command {
-	var ans = &cobra.Command{
+	var repoUpdate = &cobra.Command{
 		Use:   "update [repo1] [repo2] [OPTIONS]",
 		Short: "Update a specific cached repository or all cached repositories.",
 		Example: `
@@ -45,40 +45,36 @@ $> luet repo update repo1 repo2
 
 			if len(args) > 0 {
 				for _, rname := range args {
-					repo, err := LuetCfg.GetSystemRepository(rname)
+					repo, err := util.DefaultContext.Config.GetSystemRepository(rname)
 					if err != nil && !ignore {
-						Fatal(err.Error())
+						util.DefaultContext.Fatal(err.Error())
 					} else if err != nil {
 						continue
 					}
 
 					r := installer.NewSystemRepository(*repo)
-					Spinner(32)
-					_, err = r.Sync(force)
+					_, err = r.Sync(util.DefaultContext, force)
 					if err != nil && !ignore {
-						Fatal("Error on sync repository " + rname + ": " + err.Error())
+						util.DefaultContext.Fatal("Error on sync repository " + rname + ": " + err.Error())
 					}
-					SpinnerStop()
 				}
 
 			} else {
-				for _, repo := range LuetCfg.SystemRepositories {
+				for _, repo := range util.DefaultContext.Config.SystemRepositories {
 					if repo.Cached && repo.Enable {
 						r := installer.NewSystemRepository(repo)
-						Spinner(32)
-						_, err := r.Sync(force)
+						_, err := r.Sync(util.DefaultContext, force)
 						if err != nil && !ignore {
-							Fatal("Error on sync repository " + r.GetName() + ": " + err.Error())
+							util.DefaultContext.Fatal("Error on sync repository " + r.GetName() + ": " + err.Error())
 						}
-						SpinnerStop()
 					}
 				}
 			}
 		},
 	}
 
-	ans.Flags().BoolP("ignore-errors", "i", false, "Ignore errors on sync repositories.")
-	ans.Flags().BoolP("force", "f", false, "Force resync.")
+	repoUpdate.Flags().BoolP("ignore-errors", "i", false, "Ignore errors on sync repositories.")
+	repoUpdate.Flags().BoolP("force", "f", true, "Force resync.")
 
-	return ans
+	return repoUpdate
 }

cmd/root.go

@@ -18,123 +18,107 @@ package cmd
 import (
 	"fmt"
 	"os"
-	"os/user"
-	"path/filepath"
-	"runtime"
-	"strings"
 
-	"github.com/marcsauter/single"
-	extensions "github.com/mudler/cobra-extensions"
-	config "github.com/mudler/luet/pkg/config"
-	helpers "github.com/mudler/luet/pkg/helpers"
-	. "github.com/mudler/luet/pkg/logger"
-	repo "github.com/mudler/luet/pkg/repository"
+	"github.com/mudler/luet/cmd/util"
+	bus "github.com/mudler/luet/pkg/api/core/bus"
+
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 )
 
 var cfgFile string
 var Verbose bool
-var LockedCommands = []string{"install", "uninstall", "upgrade"}
 
 const (
-	LuetCLIVersion = "0.8.7"
-	LuetEnvPrefix  = "LUET"
+	LuetEnvPrefix = "LUET"
 )
 
+var license = []string{
+	"Copyright (C) 2019-2022 Ettore Di Giacinto",
+	"This program comes with ABSOLUTELY NO WARRANTY.",
+	"This is free software, and you are welcome to redistribute it under certain conditions.",
+	"For documentation, visit https://luet.io.",
+}
+
 // Build time and commit information.
 //
 // ⚠️ WARNING: should only be set by "-ldflags".
 var (
-	BuildTime   string
-	BuildCommit string
+	BuildTime string
+	Version   = "0.0.0"
 )
 
+func version() string {
+	return fmt.Sprintf("%s (Build time: %s)", Version, BuildTime)
+}
+
 // RootCmd represents the base command when called without any subcommands
 var RootCmd = &cobra.Command{
-	Use:     "luet",
-	Short:   "Package manager for the XXth century!",
-	Long:    `Package manager which uses containers to build packages`,
-	Version: fmt.Sprintf("%s-g%s %s", LuetCLIVersion, BuildCommit, BuildTime),
+	Use:   "luet",
+	Short: "Container based package manager",
+	Long: `Luet is a single-binary package manager based on containers to build packages. 
+
+For documentation, visit https://luet.io.
+	
+To install a package:
+
+	$ luet install package
+
+To search for a package in the repositories:
+
+$ luet search package
+
+To list all packages installed in the system:
+
+	$ luet search --installed .
+
+To show hidden packages:
+
+	$ luet search --hidden package
+
+To build a package, from a tree definition:
+
+	$ luet build --tree tree/path package
+	
+`,
+	Version: version(),
 	PersistentPreRun: func(cmd *cobra.Command, args []string) {
-		err := LoadConfig(config.LuetCfg)
+		ctx, err := util.InitContext(cmd)
 		if err != nil {
-			Fatal("failed to load configuration:", err.Error())
+			fmt.Println("failed to load configuration:", err.Error())
+			os.Exit(1)
 		}
-		// Initialize tmpdir prefix. TODO: Move this with LoadConfig
-		// directly on sub command to ensure the creation only when it's
-		// needed.
-		err = config.LuetCfg.GetSystem().InitTmpDir()
-		if err != nil {
-			Fatal("failed on init tmp basedir:", err.Error())
+
+		util.DefaultContext = ctx
+
+		util.DisplayVersionBanner(util.DefaultContext, version, license)
+
+		viper.BindPFlag("plugin", cmd.Flags().Lookup("plugin"))
+
+		plugin := viper.GetStringSlice("plugin")
+
+		bus.Manager.Initialize(util.DefaultContext, plugin...)
+		if len(bus.Manager.Plugins) != 0 {
+			util.DefaultContext.Info(":lollipop:Enabled plugins:")
+			for _, p := range bus.Manager.Plugins {
+				util.DefaultContext.Info(fmt.Sprintf("\t:arrow_right: %s (at %s)", p.Name, p.Executable))
+			}
 		}
 	},
 	PersistentPostRun: func(cmd *cobra.Command, args []string) {
 		// Cleanup all tmp directories used by luet
-		err := config.LuetCfg.GetSystem().CleanupTmpDir()
+		err := util.DefaultContext.Clean()
 		if err != nil {
-			Warning("failed on cleanup tmpdir:", err.Error())
+			util.DefaultContext.Warning("failed on cleanup tmpdir:", err.Error())
 		}
 	},
 	SilenceErrors: true,
 }
 
-func LoadConfig(c *config.LuetConfig) error {
-	// If a config file is found, read it in.
-	c.Viper.ReadInConfig()
-
-	err := c.Viper.Unmarshal(&config.LuetCfg)
-	if err != nil {
-		return err
-	}
-
-	noSpinner := c.Viper.GetBool("no_spinner")
-
-	InitAurora()
-	if !noSpinner {
-		NewSpinner()
-	}
-
-	Debug("Using config file:", c.Viper.ConfigFileUsed())
-
-	if c.GetLogging().EnableLogFile && c.GetLogging().Path != "" {
-		// Init zap logger
-		err = ZapLogger()
-		if err != nil {
-			return err
-		}
-	}
-
-	// Load repositories
-	err = repo.LoadRepositories(c)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
 // Execute adds all child commands to the root command sets flags appropriately.
 // This is called by main.main(). It only needs to happen once to the rootCmd.
 func Execute() {
-
-	if os.Getenv("LUET_NOLOCK") != "true" {
-		if len(os.Args) > 1 {
-			for _, lockedCmd := range LockedCommands {
-				if os.Args[1] == lockedCmd {
-					s := single.New("luet")
-					if err := s.CheckLock(); err != nil && err == single.ErrAlreadyRunning {
-						Fatal("another instance of the app is already running, exiting")
-					} else if err != nil {
-						// Another error occurred, might be worth handling it as well
-						Fatal("failed to acquire exclusive app lock:", err.Error())
-					}
-					defer s.TryUnlock()
-					break
-				}
-			}
-		}
-	}
+	util.HandleLock()
 
 	if err := RootCmd.Execute(); err != nil {
 		fmt.Println(err)
@@ -143,92 +127,5 @@ func Execute() {
 }
 
 func init() {
-	cobra.OnInitialize(initConfig)
-	pflags := RootCmd.PersistentFlags()
-	pflags.StringVar(&cfgFile, "config", "", "config file (default is $HOME/.luet.yaml)")
-	pflags.BoolP("debug", "d", false, "verbose output")
-	pflags.Bool("fatal", false, "Enables Warnings to exit")
-	pflags.Bool("enable-logfile", false, "Enable log to file")
-	pflags.Bool("no-spinner", false, "Disable spinner.")
-	pflags.Bool("color", config.LuetCfg.GetLogging().Color, "Enable/Disable color.")
-	pflags.Bool("emoji", config.LuetCfg.GetLogging().EnableEmoji, "Enable/Disable emoji.")
-	pflags.Bool("skip-config-protect", config.LuetCfg.ConfigProtectSkip,
-		"Disable config protect analysis.")
-	pflags.StringP("logfile", "l", config.LuetCfg.GetLogging().Path,
-		"Logfile path. Empty value disable log to file.")
-
-	// os/user doesn't work in from scratch environments.
-	// Check if i can retrieve user informations.
-	_, err := user.Current()
-	if err != nil {
-		Warning("failed to retrieve user identity:", err.Error())
-	}
-	pflags.Bool("same-owner", config.LuetCfg.GetGeneral().SameOwner, "Maintain same owner on uncompress.")
-	pflags.Int("concurrency", runtime.NumCPU(), "Concurrency")
-
-	config.LuetCfg.Viper.BindPFlag("logging.color", pflags.Lookup("color"))
-	config.LuetCfg.Viper.BindPFlag("logging.enable_emoji", pflags.Lookup("emoji"))
-	config.LuetCfg.Viper.BindPFlag("logging.enable_logfile", pflags.Lookup("enable-logfile"))
-	config.LuetCfg.Viper.BindPFlag("logging.path", pflags.Lookup("logfile"))
-
-	config.LuetCfg.Viper.BindPFlag("general.concurrency", pflags.Lookup("concurrency"))
-	config.LuetCfg.Viper.BindPFlag("general.debug", pflags.Lookup("debug"))
-	config.LuetCfg.Viper.BindPFlag("general.fatal_warnings", pflags.Lookup("fatal"))
-	config.LuetCfg.Viper.BindPFlag("general.same_owner", pflags.Lookup("same-owner"))
-	// Currently I maintain this only from cli.
-	config.LuetCfg.Viper.BindPFlag("no_spinner", pflags.Lookup("no-spinner"))
-	config.LuetCfg.Viper.BindPFlag("config_protect_skip", pflags.Lookup("skip-config-protect"))
-
-	// Extensions must be binary with the "luet-" prefix to be able to be shown in the help.
-	// we also accept extensions in the relative path where luet is being started, "extensions/"
-	exts := extensions.Discover("luet", "extensions")
-	for _, ex := range exts {
-		cobraCmd := ex.CobraCommand()
-		RootCmd.AddCommand(cobraCmd)
-	}
-
-}
-
-// initConfig reads in config file and ENV variables if set.
-func initConfig() {
-	// Luet support these priorities on read configuration file:
-	// - command line option (if available)
-	// - $PWD/.luet.yaml
-	// - $HOME/.luet.yaml
-	// - /etc/luet/luet.yaml
-	//
-	// Note: currently a single viper instance support only one config name.
-
-	viper.SetEnvPrefix(LuetEnvPrefix)
-	viper.SetConfigType("yaml")
-
-	if cfgFile != "" { // enable ability to specify config file via flag
-		viper.SetConfigFile(cfgFile)
-	} else {
-		// Retrieve pwd directory
-		pwdDir, err := os.Getwd()
-		if err != nil {
-			Error(err)
-			os.Exit(1)
-		}
-		homeDir := helpers.GetHomeDir()
-
-		if helpers.Exists(filepath.Join(pwdDir, ".luet.yaml")) || (homeDir != "" && helpers.Exists(filepath.Join(homeDir, ".luet.yaml"))) {
-			viper.AddConfigPath(".")
-			if homeDir != "" {
-				viper.AddConfigPath(homeDir)
-			}
-			viper.SetConfigName(".luet")
-		} else {
-			viper.SetConfigName("luet")
-			viper.AddConfigPath("/etc/luet")
-		}
-	}
-
-	viper.AutomaticEnv() // read in environment variables that match
-
-	// Create EnvKey Replacer for handle complex structure
-	replacer := strings.NewReplacer(".", "__")
-	viper.SetEnvKeyReplacer(replacer)
-	viper.SetTypeByDefaultValue(true)
+	util.InitViper(RootCmd)
 }

cmd/search.go

@@ -16,192 +16,368 @@ package cmd
 
 import (
 	"fmt"
-	"os"
-	"path/filepath"
+	"strings"
 
 	"github.com/ghodss/yaml"
-	. "github.com/mudler/luet/pkg/config"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
 	installer "github.com/mudler/luet/pkg/installer"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
+	"github.com/pterm/pterm"
 	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
 )
 
 type PackageResult struct {
-	Name       string `json:"name"`
-	Category   string `json:"category"`
-	Version    string `json:"version"`
-	Repository string `json:"repository"`
-	Hidden     bool   `json:"hidden"`
+	Name       string   `json:"name"`
+	Category   string   `json:"category"`
+	Version    string   `json:"version"`
+	Repository string   `json:"repository"`
+	Target     string   `json:"target"`
+	Hidden     bool     `json:"hidden"`
+	Files      []string `json:"files"`
+	Installed  bool     `json:"installed"`
 }
 
 type Results struct {
 	Packages []PackageResult `json:"packages"`
 }
 
-var searchCmd = &cobra.Command{
-	Use:     "search <term>",
-	Short:   "Search packages",
-	Long:    `Search for installed and available packages`,
-	Aliases: []string{"s"},
-	PreRun: func(cmd *cobra.Command, args []string) {
-		LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
-		LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
-		LuetCfg.Viper.BindPFlag("installed", cmd.Flags().Lookup("installed"))
-		LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
-		LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
-		LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
-		LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
-	},
-	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
-		var results Results
-		if len(args) != 1 {
-			Fatal("Wrong number of arguments (expected 1)")
-		}
+func (r PackageResult) String() string {
+	return fmt.Sprintf("%s/%s-%s required for %s", r.Category, r.Name, r.Version, r.Target)
+}
 
-		hidden, _ := cmd.Flags().GetBool("hidden")
+var rows []string = []string{"Package", "Category", "Name", "Version", "Repository", "License", "Installed"}
 
-		installed := LuetCfg.Viper.GetBool("installed")
-		stype := LuetCfg.Viper.GetString("solver.type")
-		discount := LuetCfg.Viper.GetFloat64("solver.discount")
-		rate := LuetCfg.Viper.GetFloat64("solver.rate")
-		attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
-		searchWithLabel, _ := cmd.Flags().GetBool("by-label")
-		searchWithLabelMatch, _ := cmd.Flags().GetBool("by-label-regex")
-		revdeps, _ := cmd.Flags().GetBool("revdeps")
+func packageToRow(repo string, p *types.Package, installed bool) []string {
+	return []string{p.HumanReadableString(), p.GetCategory(), p.GetName(), p.GetVersion(), repo, p.GetLicense(), fmt.Sprintf("%t", installed)}
+}
 
-		out, _ := cmd.Flags().GetString("output")
-		if out != "terminal" {
-			LuetCfg.GetLogging().SetLogLevel("error")
-		}
+func packageToList(l *util.ListWriter, repo string, p *types.Package, installed bool) {
+	l.AppendItem(pterm.BulletListItem{
+		Level: 0, Text: p.HumanReadableString(),
+		TextStyle: pterm.NewStyle(pterm.FgCyan), Bullet: ">", BulletStyle: pterm.NewStyle(pterm.FgYellow),
+	})
+	l.AppendItem(pterm.BulletListItem{
+		Level: 1, Text: fmt.Sprintf("Category: %s", p.GetCategory()),
+		Bullet: "->", BulletStyle: pterm.NewStyle(pterm.FgDarkGray),
+	})
+	l.AppendItem(pterm.BulletListItem{
+		Level: 1, Text: fmt.Sprintf("Name: %s", p.GetName()),
+		Bullet: "->", BulletStyle: pterm.NewStyle(pterm.FgDarkGray),
+	})
+	l.AppendItem(pterm.BulletListItem{
+		Level: 1, Text: fmt.Sprintf("Version: %s", p.GetVersion()),
+		Bullet: "->", BulletStyle: pterm.NewStyle(pterm.FgDarkGray),
+	})
+	l.AppendItem(pterm.BulletListItem{
+		Level: 1, Text: fmt.Sprintf("Description: %s", p.GetDescription()),
+		Bullet: "->", BulletStyle: pterm.NewStyle(pterm.FgDarkGray),
+	})
+	l.AppendItem(pterm.BulletListItem{
+		Level: 1, Text: fmt.Sprintf("Repository: %s ", repo),
+		Bullet: "->", BulletStyle: pterm.NewStyle(pterm.FgDarkGray),
+	})
+	l.AppendItem(pterm.BulletListItem{
+		Level: 1, Text: fmt.Sprintf("Uri: %s ", strings.Join(p.GetURI(), " ")),
+		Bullet: "->", BulletStyle: pterm.NewStyle(pterm.FgDarkGray),
+	})
+	l.AppendItem(pterm.BulletListItem{
+		Level: 1, Text: fmt.Sprintf("Installed: %t ", installed),
+		Bullet: "->", BulletStyle: pterm.NewStyle(pterm.FgDarkGray),
+	})
+}
 
-		LuetCfg.GetSolverOptions().Type = stype
-		LuetCfg.GetSolverOptions().LearnRate = float32(rate)
-		LuetCfg.GetSolverOptions().Discount = float32(discount)
-		LuetCfg.GetSolverOptions().MaxAttempts = attempts
+var s *installer.System
 
-		Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
+func sys() *installer.System {
+	if s != nil {
+		return s
+	}
+	s = &installer.System{Database: util.SystemDB(util.DefaultContext.Config), Target: util.DefaultContext.Config.System.Rootfs}
+	return s
+}
 
-		if !installed {
+func installed(p *types.Package) bool {
+	s := sys()
+	_, err := s.Database.FindPackage(p)
+	return err == nil
+}
 
-			repos := installer.Repositories{}
-			for _, repo := range LuetCfg.SystemRepositories {
-				if !repo.Enable {
-					continue
-				}
-				r := installer.NewSystemRepository(repo)
-				repos = append(repos, r)
-			}
+func searchLocally(term string, l *util.ListWriter, t *util.TableWriter, label, labelMatch, revdeps, hidden bool) Results {
+	var results Results
 
-			inst := installer.NewLuetInstaller(
-				installer.LuetInstallerOptions{
-					Concurrency:   LuetCfg.GetGeneral().Concurrency,
-					SolverOptions: *LuetCfg.GetSolverOptions(),
-				},
-			)
-			inst.Repositories(repos)
-			synced, err := inst.SyncRepositories(false)
-			if err != nil {
-				Fatal("Error: " + err.Error())
-			}
+	system := sys()
 
-			Info("--- Search results (" + args[0] + "): ---")
+	var err error
+	iMatches := types.Packages{}
+	if label {
+		iMatches, err = system.Database.FindPackageLabel(term)
+	} else if labelMatch {
+		iMatches, err = system.Database.FindPackageLabelMatch(term)
+	} else {
+		iMatches, err = system.Database.FindPackageMatch(term)
+	}
 
-			matches := []installer.PackageMatch{}
-			if searchWithLabel {
-				matches = synced.SearchLabel(args[0])
-			} else if searchWithLabelMatch {
-				matches = synced.SearchLabelMatch(args[0])
-			} else {
-				matches = synced.Search(args[0])
-			}
-			for _, m := range matches {
-				if !revdeps {
-					if !m.Package.IsHidden() || m.Package.IsHidden() && hidden {
-						Info(fmt.Sprintf(":file_folder:%s", m.Repo.GetName()), fmt.Sprintf(":package:%s", m.Package.HumanReadableString()))
-						results.Packages = append(results.Packages,
-							PackageResult{
-								Name:       m.Package.GetName(),
-								Version:    m.Package.GetVersion(),
-								Category:   m.Package.GetCategory(),
-								Repository: m.Repo.GetName(),
-								Hidden:     m.Package.IsHidden(),
-							})
-					}
-				} else {
-					visited := make(map[string]interface{})
-					for _, revdep := range m.Package.ExpandedRevdeps(m.Repo.GetTree().GetDatabase(), visited) {
-						if !revdep.IsHidden() || revdep.IsHidden() && hidden {
-							Info(fmt.Sprintf(":file_folder:%s", m.Repo.GetName()), fmt.Sprintf(":package:%s", revdep.HumanReadableString()))
-							results.Packages = append(results.Packages,
-								PackageResult{
-									Name:       revdep.GetName(),
-									Version:    revdep.GetVersion(),
-									Category:   revdep.GetCategory(),
-									Repository: m.Repo.GetName(),
-									Hidden:     revdep.IsHidden(),
-								})
-						}
-					}
-				}
+	if err != nil {
+		util.DefaultContext.Fatal("Error: " + err.Error())
+	}
+
+	for _, pack := range iMatches {
+		if !revdeps {
+			if !pack.IsHidden() || pack.IsHidden() && hidden {
+				t.AppendRow(packageToRow("system", pack, true))
+				packageToList(l, "system", pack, true)
+				f, _ := system.Database.GetPackageFiles(pack)
+				results.Packages = append(results.Packages,
+					PackageResult{
+						Name:       pack.GetName(),
+						Version:    pack.GetVersion(),
+						Category:   pack.GetCategory(),
+						Repository: "system",
+						Hidden:     pack.IsHidden(),
+						Files:      f,
+						Installed:  true,
+					})
 			}
 		} else {
 
-			if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-				systemDB = pkg.NewBoltDatabase(
-					filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-			} else {
-				systemDB = pkg.NewInMemoryDatabase(true)
-			}
-			system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
-
-			var err error
-			iMatches := pkg.Packages{}
-			if searchWithLabel {
-				iMatches, err = system.Database.FindPackageLabel(args[0])
-			} else if searchWithLabelMatch {
-				iMatches, err = system.Database.FindPackageLabelMatch(args[0])
-			} else {
-				iMatches, err = system.Database.FindPackageMatch(args[0])
-			}
-
-			if err != nil {
-				Fatal("Error: " + err.Error())
-			}
-
-			for _, pack := range iMatches {
-				if !revdeps {
-					if !pack.IsHidden() || pack.IsHidden() && hidden {
-						Info(fmt.Sprintf(":package:%s", pack.HumanReadableString()))
-						results.Packages = append(results.Packages,
-							PackageResult{
-								Name:       pack.GetName(),
-								Version:    pack.GetVersion(),
-								Category:   pack.GetCategory(),
-								Repository: "system",
-								Hidden:     pack.IsHidden(),
-							})
-					}
-				} else {
-					visited := make(map[string]interface{})
-
-					for _, revdep := range pack.ExpandedRevdeps(system.Database, visited) {
-						if !revdep.IsHidden() || revdep.IsHidden() && hidden {
-							Info(fmt.Sprintf(":package:%s", pack.HumanReadableString()))
-							results.Packages = append(results.Packages,
-								PackageResult{
-									Name:       revdep.GetName(),
-									Version:    revdep.GetVersion(),
-									Category:   revdep.GetCategory(),
-									Repository: "system",
-									Hidden:     revdep.IsHidden(),
-								})
-						}
-					}
+			packs, _ := system.Database.GetRevdeps(pack)
+			for _, revdep := range packs {
+				if !revdep.IsHidden() || revdep.IsHidden() && hidden {
+					i := installed(pack)
+					t.AppendRow(packageToRow("system", pack, i))
+					packageToList(l, "system", pack, i)
+					f, _ := system.Database.GetPackageFiles(revdep)
+					results.Packages = append(results.Packages,
+						PackageResult{
+							Name:       revdep.GetName(),
+							Version:    revdep.GetVersion(),
+							Category:   revdep.GetCategory(),
+							Repository: "system",
+							Hidden:     revdep.IsHidden(),
+							Files:      f,
+							Installed:  i,
+						})
 				}
 			}
 		}
+	}
+	return results
+
+}
+func searchOnline(term string, l *util.ListWriter, t *util.TableWriter, label, labelMatch, revdeps, hidden bool) Results {
+	var results Results
+
+	inst := installer.NewLuetInstaller(
+		installer.LuetInstallerOptions{
+			Concurrency:         util.DefaultContext.Config.General.Concurrency,
+			SolverOptions:       util.DefaultContext.Config.Solver,
+			PackageRepositories: util.DefaultContext.Config.SystemRepositories,
+			Context:             util.DefaultContext,
+		},
+	)
+	synced, err := inst.SyncRepositories()
+	if err != nil {
+		util.DefaultContext.Fatal("Error: " + err.Error())
+	}
+
+	util.DefaultContext.Info("--- Search results (" + term + "): ---")
+
+	matches := []installer.PackageMatch{}
+	if label {
+		matches = synced.SearchLabel(term)
+	} else if labelMatch {
+		matches = synced.SearchLabelMatch(term)
+	} else {
+		matches = synced.Search(term)
+	}
+
+	for _, m := range matches {
+		if !revdeps {
+			if !m.Package.IsHidden() || m.Package.IsHidden() && hidden {
+				i := installed(m.Package)
+				t.AppendRow(packageToRow(m.Repo.GetName(), m.Package, i))
+				packageToList(l, m.Repo.GetName(), m.Package, i)
+				r := &PackageResult{
+					Name:       m.Package.GetName(),
+					Version:    m.Package.GetVersion(),
+					Category:   m.Package.GetCategory(),
+					Repository: m.Repo.GetName(),
+					Hidden:     m.Package.IsHidden(),
+					Installed:  i,
+				}
+				if m.Artifact != nil {
+					r.Files = m.Artifact.Files
+				}
+				results.Packages = append(results.Packages, *r)
+			}
+		} else {
+			packs, _ := m.Repo.GetTree().GetDatabase().GetRevdeps(m.Package)
+			for _, revdep := range packs {
+				if !revdep.IsHidden() || revdep.IsHidden() && hidden {
+					i := installed(revdep)
+					t.AppendRow(packageToRow(m.Repo.GetName(), revdep, i))
+					packageToList(l, m.Repo.GetName(), revdep, i)
+					r := &PackageResult{
+						Name:       revdep.GetName(),
+						Installed:  i,
+						Version:    revdep.GetVersion(),
+						Category:   revdep.GetCategory(),
+						Repository: m.Repo.GetName(),
+						Hidden:     revdep.IsHidden(),
+					}
+					if m.Artifact != nil {
+						r.Files = m.Artifact.Files
+					}
+					results.Packages = append(results.Packages, *r)
+				}
+			}
+		}
+	}
+	return results
+}
+func searchLocalFiles(term string, l *util.ListWriter, t *util.TableWriter) Results {
+	var results Results
+	util.DefaultContext.Info("--- Search results (" + term + "): ---")
+
+	matches, _ := util.SystemDB(util.DefaultContext.Config).FindPackageByFile(term)
+	for _, pack := range matches {
+		i := installed(pack)
+		t.AppendRow(packageToRow("system", pack, i))
+		packageToList(l, "system", pack, i)
+		f, _ := util.SystemDB(util.DefaultContext.Config).GetPackageFiles(pack)
+		results.Packages = append(results.Packages,
+			PackageResult{
+				Name:       pack.GetName(),
+				Version:    pack.GetVersion(),
+				Category:   pack.GetCategory(),
+				Repository: "system",
+				Hidden:     pack.IsHidden(),
+				Files:      f,
+				Installed:  i,
+			})
+	}
+
+	return results
+}
+
+func searchFiles(term string, l *util.ListWriter, t *util.TableWriter) Results {
+	var results Results
+
+	inst := installer.NewLuetInstaller(
+		installer.LuetInstallerOptions{
+			Concurrency:         util.DefaultContext.Config.General.Concurrency,
+			SolverOptions:       util.DefaultContext.Config.Solver,
+			PackageRepositories: util.DefaultContext.Config.SystemRepositories,
+			Context:             util.DefaultContext,
+		},
+	)
+	synced, err := inst.SyncRepositories()
+	if err != nil {
+		util.DefaultContext.Fatal("Error: " + err.Error())
+	}
+
+	util.DefaultContext.Info("--- Search results (" + term + "): ---")
+
+	matches := []installer.PackageMatch{}
+
+	matches = synced.SearchPackages(term, installer.FileSearch)
+
+	for _, m := range matches {
+		i := installed(m.Package)
+		t.AppendRow(packageToRow(m.Repo.GetName(), m.Package, i))
+		packageToList(l, m.Repo.GetName(), m.Package, i)
+		results.Packages = append(results.Packages,
+			PackageResult{
+				Name:       m.Package.GetName(),
+				Version:    m.Package.GetVersion(),
+				Category:   m.Package.GetCategory(),
+				Repository: m.Repo.GetName(),
+				Hidden:     m.Package.IsHidden(),
+				Files:      m.Artifact.Files,
+				Installed:  i,
+			})
+	}
+	return results
+}
+
+var searchCmd = &cobra.Command{
+	Use: "search <term>",
+	// Skip processing output
+	Annotations: map[string]string{
+		util.CommandProcessOutput: "",
+	},
+	Short: "Search packages",
+	Long: `Search for installed and available packages
+	
+To search a package in the repositories:
+
+	$ luet search <regex>
+
+To search a package and display results in a table (wide screens):
+
+	$ luet search --table <regex>
+
+To look into the installed packages:
+
+	$ luet search --installed <regex>
+
+Note: the regex argument is optional, if omitted implies "all"
+
+To search a package by label:
+
+	$ luet search --by-label <label>
+
+or by regex against the label:
+
+	$ luet search --by-label-regex <label>
+
+It can also show a package revdeps by:
+
+	$ luet search --revdeps <regex>
+
+Search can also return results in the terminal in different ways: as terminal output, as json or as yaml.
+
+	$ luet search --json <regex> # JSON output
+	$ luet search --yaml <regex> # YAML output
+`,
+	Aliases: []string{"s"},
+	PreRun: func(cmd *cobra.Command, args []string) {
+
+		viper.BindPFlag("installed", cmd.Flags().Lookup("installed"))
+	},
+	Run: func(cmd *cobra.Command, args []string) {
+		var results Results
+		if len(args) > 1 {
+			util.DefaultContext.Fatal("Wrong number of arguments (expected 1)")
+		} else if len(args) == 0 {
+			args = []string{"."}
+		}
+		hidden, _ := cmd.Flags().GetBool("hidden")
+
+		installed := viper.GetBool("installed")
+		searchWithLabel, _ := cmd.Flags().GetBool("by-label")
+		searchWithLabelMatch, _ := cmd.Flags().GetBool("by-label-regex")
+		revdeps, _ := cmd.Flags().GetBool("revdeps")
+		tableMode, _ := cmd.Flags().GetBool("table")
+		files, _ := cmd.Flags().GetBool("files")
+
+		out, _ := cmd.Flags().GetString("output")
+		l := &util.ListWriter{}
+		t := &util.TableWriter{}
+		t.AppendRow(rows)
+		util.DefaultContext.Debug("Solver", util.DefaultContext.Config.Solver.CompactString())
+
+		switch {
+		case files && installed:
+			results = searchLocalFiles(args[0], l, t)
+		case files && !installed:
+			results = searchFiles(args[0], l, t)
+		case !installed:
+			results = searchOnline(args[0], l, t, searchWithLabel, searchWithLabelMatch, revdeps, hidden)
+		default:
+			results = searchLocally(args[0], l, t, searchWithLabel, searchWithLabelMatch, revdeps, hidden)
+		}
 
 		y, err := yaml.Marshal(results)
 		if err != nil {
@@ -218,28 +394,30 @@ var searchCmd = &cobra.Command{
 				return
 			}
 			fmt.Println(string(j2))
+		default:
+			if tableMode {
+				t.Render()
+			} else if util.DefaultContext.Config.General.Quiet {
+				for _, tt := range results.Packages {
+					fmt.Printf("%s/%s-%s\n", tt.Category, tt.Name, tt.Version)
+				}
+			} else {
+				l.Render()
+			}
 		}
-
 	},
 }
 
 func init() {
-	path, err := os.Getwd()
-	if err != nil {
-		Fatal(err)
-	}
-	searchCmd.Flags().String("system-dbpath", path, "System db path")
-	searchCmd.Flags().String("system-target", path, "System rootpath")
+
 	searchCmd.Flags().Bool("installed", false, "Search between system packages")
-	searchCmd.Flags().String("solver-type", "", "Solver strategy ( Defaults none, available: "+AvailableResolvers+" )")
 	searchCmd.Flags().StringP("output", "o", "terminal", "Output format ( Defaults: terminal, available: json,yaml )")
-	searchCmd.Flags().Float32("solver-rate", 0.7, "Solver learning rate")
-	searchCmd.Flags().Float32("solver-discount", 1.0, "Solver discount rate")
-	searchCmd.Flags().Int("solver-attempts", 9000, "Solver maximum attempts")
 	searchCmd.Flags().Bool("by-label", false, "Search packages through label")
 	searchCmd.Flags().Bool("by-label-regex", false, "Search packages through label regex")
 	searchCmd.Flags().Bool("revdeps", false, "Search package reverse dependencies")
 	searchCmd.Flags().Bool("hidden", false, "Include hidden packages")
+	searchCmd.Flags().Bool("table", false, "show output in a table (wider screens)")
+	searchCmd.Flags().Bool("files", false, "Search between packages files")
 
 	RootCmd.AddCommand(searchCmd)
 }

cmd/serve-repo.go

@@ -18,7 +18,7 @@ import (
 	"net/http"
 	"os"
 
-	. "github.com/mudler/luet/pkg/logger"
+	"github.com/mudler/luet/cmd/util"
 
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
@@ -41,15 +41,15 @@ var serverepoCmd = &cobra.Command{
 
 		http.Handle("/", http.FileServer(http.Dir(dir)))
 
-		Info("Serving ", dir, " on HTTP port: ", port)
-		Fatal(http.ListenAndServe(address+":"+port, nil))
+		util.DefaultContext.Info("Serving ", dir, " on HTTP port: ", port)
+		util.DefaultContext.Fatal(http.ListenAndServe(address+":"+port, nil))
 	},
 }
 
 func init() {
 	path, err := os.Getwd()
 	if err != nil {
-		Fatal(err)
+		util.DefaultContext.Fatal(err)
 	}
 	serverepoCmd.Flags().String("dir", path, "Packages folder (output from build)")
 	serverepoCmd.Flags().String("port", "9090", "Listening port")

cmd/tree.go

@@ -34,5 +34,6 @@ func init() {
 		NewTreePkglistCommand(),
 		NewTreeValidateCommand(),
 		NewTreeBumpCommand(),
+		NewTreeImageCommand(),
 	)
 }

cmd/tree/bump.go

@@ -19,7 +19,7 @@ package cmd_tree
 import (
 	"fmt"
 
-	. "github.com/mudler/luet/pkg/logger"
+	"github.com/mudler/luet/cmd/util"
 	spectooling "github.com/mudler/luet/pkg/spectooling"
 	tree "github.com/mudler/luet/pkg/tree"
 	version "github.com/mudler/luet/pkg/versioner"
@@ -36,7 +36,7 @@ func NewTreeBumpCommand() *cobra.Command {
 		PreRun: func(cmd *cobra.Command, args []string) {
 			df, _ := cmd.Flags().GetString("definition-file")
 			if df == "" {
-				Fatal("Mandatory definition.yaml path missing.")
+				util.DefaultContext.Fatal("Mandatory definition.yaml path missing.")
 			}
 		},
 		Run: func(cmd *cobra.Command, args []string) {
@@ -45,34 +45,34 @@ func NewTreeBumpCommand() *cobra.Command {
 			pkgVersion, _ := cmd.Flags().GetString("pkg-version")
 			pack, err := tree.ReadDefinitionFile(spec)
 			if err != nil {
-				Fatal(err.Error())
+				util.DefaultContext.Fatal(err.Error())
 			}
 
 			if pkgVersion != "" {
 				validator := &version.WrappedVersioner{}
 				err := validator.Validate(pkgVersion)
 				if err != nil {
-					Fatal("Invalid version string: " + err.Error())
+					util.DefaultContext.Fatal("Invalid version string: " + err.Error())
 				}
 				pack.SetVersion(pkgVersion)
 			} else {
 				// Retrieve version build section with Gentoo parser
 				err = pack.BumpBuildVersion()
 				if err != nil {
-					Fatal("Error on increment build version: " + err.Error())
+					util.DefaultContext.Fatal("Error on increment build version: " + err.Error())
 				}
 			}
 			if toStdout {
 				data, err := spectooling.NewDefaultPackageSanitized(&pack).Yaml()
 				if err != nil {
-					Fatal("Error on yaml conversion: " + err.Error())
+					util.DefaultContext.Fatal("Error on yaml conversion: " + err.Error())
 				}
 				fmt.Println(string(data))
 			} else {
 
 				err = tree.WriteDefinitionFile(&pack, spec)
 				if err != nil {
-					Fatal("Error on write definition file: " + err.Error())
+					util.DefaultContext.Fatal("Error on write definition file: " + err.Error())
 				}
 
 				fmt.Printf("Bumped package %s/%s-%s.\n", pack.Category, pack.Name, pack.Version)

cmd/tree/images.go

@@ -0,0 +1,159 @@
+// Copyright © 2020 Ettore Di Giacinto <mudler@gentoo.org>
+//                  Daniele Rondina <geaaru@sabayonlinux.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd_tree
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/ghodss/yaml"
+	helpers "github.com/mudler/luet/cmd/helpers"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
+	"github.com/mudler/luet/pkg/compiler"
+	"github.com/mudler/luet/pkg/compiler/backend"
+	"github.com/mudler/luet/pkg/installer"
+
+	pkg "github.com/mudler/luet/pkg/database"
+	tree "github.com/mudler/luet/pkg/tree"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+func NewTreeImageCommand() *cobra.Command {
+
+	var ans = &cobra.Command{
+		Use: "images [OPTIONS]",
+		// Skip processing output
+		Annotations: map[string]string{
+			util.CommandProcessOutput: "",
+		},
+		Short: "List of the images of a package",
+		PreRun: func(cmd *cobra.Command, args []string) {
+			t, _ := cmd.Flags().GetStringArray("tree")
+			if len(t) == 0 {
+				util.DefaultContext.Fatal("Mandatory tree param missing.")
+			}
+
+			if len(args) != 1 {
+				util.DefaultContext.Fatal("Expects one package as parameter")
+			}
+			util.BindValuesFlags(cmd)
+			viper.BindPFlag("image-repository", cmd.Flags().Lookup("image-repository"))
+
+		},
+		Run: func(cmd *cobra.Command, args []string) {
+			var results TreeResults
+
+			treePath, _ := cmd.Flags().GetStringArray("tree")
+			imageRepository := viper.GetString("image-repository")
+			pullRepo, _ := cmd.Flags().GetStringArray("pull-repository")
+			values := util.ValuesFlags()
+			out, _ := cmd.Flags().GetString("output")
+			reciper := tree.NewCompilerRecipe(pkg.NewInMemoryDatabase(false))
+
+			for _, t := range treePath {
+				err := reciper.Load(t)
+				if err != nil {
+					util.DefaultContext.Fatal("Error on load tree ", err)
+				}
+			}
+			compilerBackend := backend.NewSimpleDockerBackend(util.DefaultContext)
+
+			opts := util.DefaultContext.Config.Solver
+			opts.SolverOptions = types.SolverOptions{Type: types.SolverSingleCoreSimple, Concurrency: 1}
+			luetCompiler := compiler.NewLuetCompiler(
+				compilerBackend,
+				reciper.GetDatabase(),
+				compiler.WithBuildValues(values),
+				compiler.WithContext(util.DefaultContext),
+				compiler.WithPushRepository(imageRepository),
+				compiler.WithPullRepositories(pullRepo),
+				compiler.WithTemplateFolder(util.TemplateFolders(util.DefaultContext, installer.BuildTreeResult{}, treePath)),
+				compiler.WithSolverOptions(opts),
+			)
+
+			a := args[0]
+
+			pack, err := helpers.ParsePackageStr(a)
+			if err != nil {
+				util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
+			}
+
+			spec, err := luetCompiler.FromPackage(pack)
+			if err != nil {
+				util.DefaultContext.Fatal("Error: " + err.Error())
+			}
+
+			ht := compiler.NewHashTree(reciper.GetDatabase())
+
+			copy, err := compiler.CompilerFinalImages(luetCompiler)
+			if err != nil {
+				util.DefaultContext.Fatal("Error: " + err.Error())
+			}
+			hashtree, err := ht.Query(copy, spec)
+			if err != nil {
+				util.DefaultContext.Fatal("Error: " + err.Error())
+			}
+
+			for _, assertion := range hashtree.Solution { //highly dependent on the order
+
+				//buildImageHash := imageRepository + ":" + assertion.Hash.BuildHash
+				currentPackageImageHash := imageRepository + ":" + assertion.Hash.PackageHash
+
+				results.Packages = append(results.Packages, TreePackageResult{
+					Name:     assertion.Package.GetName(),
+					Version:  assertion.Package.GetVersion(),
+					Category: assertion.Package.GetCategory(),
+					Image:    currentPackageImageHash,
+				})
+			}
+
+			y, err := yaml.Marshal(results)
+			if err != nil {
+				fmt.Printf("err: %v\n", err)
+				return
+			}
+			switch out {
+			case "yaml":
+				fmt.Println(string(y))
+			case "json":
+				j2, err := yaml.YAMLToJSON(y)
+				if err != nil {
+					fmt.Printf("err: %v\n", err)
+					return
+				}
+				fmt.Println(string(j2))
+			default:
+				for _, p := range results.Packages {
+					fmt.Println(fmt.Sprintf("%s/%s-%s: %s", p.Category, p.Name, p.Version, p.Image))
+				}
+			}
+		},
+	}
+	path, err := os.Getwd()
+	if err != nil {
+		util.DefaultContext.Fatal(err)
+	}
+	ans.Flags().StringP("output", "o", "terminal", "Output format ( Defaults: terminal, available: json,yaml )")
+	ans.Flags().StringArrayP("tree", "t", []string{path}, "Path of the tree to use.")
+	ans.Flags().String("image-repository", "luet/cache", "Default base image string for generated image")
+	ans.Flags().StringArrayP("pull-repository", "p", []string{}, "A list of repositories to pull the cache from")
+
+	return ans
+}

cmd/tree/pkglist.go

@@ -18,14 +18,14 @@ package cmd_tree
 
 import (
 	"fmt"
+	"os"
 	"sort"
 
-	//. "github.com/mudler/luet/pkg/config"
 	"github.com/ghodss/yaml"
 	helpers "github.com/mudler/luet/cmd/helpers"
-	. "github.com/mudler/luet/pkg/config"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
+	pkg "github.com/mudler/luet/pkg/database"
 	"github.com/mudler/luet/pkg/solver"
 	tree "github.com/mudler/luet/pkg/tree"
 
@@ -37,13 +37,14 @@ type TreePackageResult struct {
 	Category string `json:"category"`
 	Version  string `json:"version"`
 	Path     string `json:"path"`
+	Image    string `json:"image"`
 }
 
 type TreeResults struct {
 	Packages []TreePackageResult `json:"packages"`
 }
 
-func pkgDetail(pkg pkg.Package) string {
+func pkgDetail(pkg *types.Package) string {
 	ans := fmt.Sprintf(`
   @@ Package: %s/%s-%s
      Description: %s
@@ -67,25 +68,29 @@ func NewTreePkglistCommand() *cobra.Command {
 	var matches []string
 
 	var ans = &cobra.Command{
+		// Skip processing output
+		Annotations: map[string]string{
+			util.CommandProcessOutput: "",
+		},
 		Use:   "pkglist [OPTIONS]",
 		Short: "List of the packages found in tree.",
 		Args:  cobra.NoArgs,
 		PreRun: func(cmd *cobra.Command, args []string) {
 			t, _ := cmd.Flags().GetStringArray("tree")
 			if len(t) == 0 {
-				Fatal("Mandatory tree param missing.")
+				util.DefaultContext.Fatal("Mandatory tree param missing.")
 			}
 
 			revdeps, _ := cmd.Flags().GetBool("revdeps")
 			deps, _ := cmd.Flags().GetBool("deps")
 			if revdeps && deps {
-				Fatal("Both revdeps and deps option used. Choice only one.")
+				util.DefaultContext.Fatal("Both revdeps and deps option used. Choice only one.")
 			}
 
 		},
 		Run: func(cmd *cobra.Command, args []string) {
 			var results TreeResults
-			var depSolver solver.PackageSolver
+			var depSolver types.PackageSolver
 
 			treePath, _ := cmd.Flags().GetStringArray("tree")
 			verbose, _ := cmd.Flags().GetBool("verbose")
@@ -95,9 +100,6 @@ func NewTreePkglistCommand() *cobra.Command {
 			deps, _ := cmd.Flags().GetBool("deps")
 
 			out, _ := cmd.Flags().GetString("output")
-			if out != "terminal" {
-				LuetCfg.GetLogging().SetLogLevel("error")
-			}
 
 			var reciper tree.Builder
 			if buildtime {
@@ -109,14 +111,14 @@ func NewTreePkglistCommand() *cobra.Command {
 			for _, t := range treePath {
 				err := reciper.Load(t)
 				if err != nil {
-					Fatal("Error on load tree ", err)
+					util.DefaultContext.Fatal("Error on load tree ", err)
 				}
 			}
 
 			if deps {
 				emptyInstallationDb := pkg.NewInMemoryDatabase(false)
 
-				depSolver = solver.NewSolver(pkg.NewInMemoryDatabase(false),
+				depSolver = solver.NewSolver(types.SolverOptions{Type: types.SolverSingleCoreSimple}, pkg.NewInMemoryDatabase(false),
 					reciper.GetDatabase(),
 					emptyInstallationDb)
 
@@ -124,11 +126,11 @@ func NewTreePkglistCommand() *cobra.Command {
 
 			regExcludes, err := helpers.CreateRegexArray(excludes)
 			if err != nil {
-				Fatal(err.Error())
+				util.DefaultContext.Fatal(err.Error())
 			}
 			regMatches, err := helpers.CreateRegexArray(matches)
 			if err != nil {
-				Fatal(err.Error())
+				util.DefaultContext.Fatal(err.Error())
 			}
 
 			plist := make([]string, 0)
@@ -165,79 +167,79 @@ func NewTreePkglistCommand() *cobra.Command {
 					}
 				}
 
-				if addPkg {
-					if revdeps {
+				if !addPkg {
+					continue
+				}
+
+				if revdeps {
+					packs, _ := reciper.GetDatabase().GetRevdeps(p)
+					for i := range packs {
+						revdep := packs[i]
+						if full {
+							pkgstr = pkgDetail(revdep)
+						} else if verbose {
+							pkgstr = revdep.HumanReadableString()
+						} else {
+							pkgstr = fmt.Sprintf("%s/%s", revdep.GetCategory(), revdep.GetName())
+						}
+						plist = append(plist, pkgstr)
+						results.Packages = append(results.Packages, TreePackageResult{
+							Name:     revdep.GetName(),
+							Version:  revdep.GetVersion(),
+							Category: revdep.GetCategory(),
+							Path:     revdep.GetPath(),
+						})
+					}
+				} else if deps {
+
+					solution, err := depSolver.Install(types.Packages{p})
+					if err != nil {
+						util.DefaultContext.Fatal(err.Error())
+					}
+					ass := solution.SearchByName(p.GetPackageName())
+					solution, err = solution.Order(reciper.GetDatabase(), ass.Package.GetFingerPrint())
+					if err != nil {
+						util.DefaultContext.Fatal(err.Error())
+					}
+
+					for _, pa := range solution {
+
+						if pa.Value {
+							// Exclude itself
+							if pa.Package.GetName() == p.GetName() && pa.Package.GetCategory() == p.GetCategory() {
+								continue
+							}
 
-						visited := make(map[string]interface{})
-						for _, revdep := range p.ExpandedRevdeps(reciper.GetDatabase(), visited) {
 							if full {
-								pkgstr = pkgDetail(revdep)
+								pkgstr = pkgDetail(pa.Package)
 							} else if verbose {
-								pkgstr = revdep.HumanReadableString()
+								pkgstr = pa.Package.HumanReadableString()
 							} else {
-								pkgstr = fmt.Sprintf("%s/%s", revdep.GetCategory(), revdep.GetName())
+								pkgstr = fmt.Sprintf("%s/%s", pa.Package.GetCategory(), pa.Package.GetName())
 							}
 							plist = append(plist, pkgstr)
 							results.Packages = append(results.Packages, TreePackageResult{
-								Name:     revdep.GetName(),
-								Version:  revdep.GetVersion(),
-								Category: revdep.GetCategory(),
-								Path:     revdep.GetPath(),
+								Name:     pa.Package.GetName(),
+								Version:  pa.Package.GetVersion(),
+								Category: pa.Package.GetCategory(),
+								Path:     pa.Package.GetPath(),
 							})
 						}
-					} else if deps {
-
-						Spinner(32)
-						solution, err := depSolver.Install(pkg.Packages{p})
-						if err != nil {
-							Fatal(err.Error())
-						}
-						ass := solution.SearchByName(p.GetPackageName())
-						solution, err = solution.Order(reciper.GetDatabase(), ass.Package.GetFingerPrint())
-						if err != nil {
-							Fatal(err.Error())
-						}
-						SpinnerStop()
-
-						for _, pa := range solution {
-
-							if pa.Value {
-								// Exclude itself
-								if pa.Package.GetName() == p.GetName() && pa.Package.GetCategory() == p.GetCategory() {
-									continue
-								}
-
-								if full {
-									pkgstr = pkgDetail(pa.Package)
-								} else if verbose {
-									pkgstr = pa.Package.HumanReadableString()
-								} else {
-									pkgstr = fmt.Sprintf("%s/%s", pa.Package.GetCategory(), pa.Package.GetName())
-								}
-								plist = append(plist, pkgstr)
-								results.Packages = append(results.Packages, TreePackageResult{
-									Name:     pa.Package.GetName(),
-									Version:  pa.Package.GetVersion(),
-									Category: pa.Package.GetCategory(),
-									Path:     pa.Package.GetPath(),
-								})
-							}
-
-						}
-
-					} else {
-
-						plist = append(plist, pkgstr)
-						results.Packages = append(results.Packages, TreePackageResult{
-							Name:     p.GetName(),
-							Version:  p.GetVersion(),
-							Category: p.GetCategory(),
-							Path:     p.GetPath(),
-						})
 
 					}
 
+				} else {
+
+					plist = append(plist, pkgstr)
+					results.Packages = append(results.Packages, TreePackageResult{
+						Name:     p.GetName(),
+						Version:  p.GetVersion(),
+						Category: p.GetCategory(),
+						Path:     p.GetPath(),
+					})
+
 				}
+
 			}
 
 			y, err := yaml.Marshal(results)
@@ -266,7 +268,10 @@ func NewTreePkglistCommand() *cobra.Command {
 
 		},
 	}
-
+	path, err := os.Getwd()
+	if err != nil {
+		util.DefaultContext.Fatal(err)
+	}
 	ans.Flags().BoolP("buildtime", "b", false, "Build time match")
 	ans.Flags().StringP("output", "o", "terminal", "Output format ( Defaults: terminal, available: json,yaml )")
 	ans.Flags().Bool("revdeps", false, "Search package reverse dependencies")
@@ -274,7 +279,7 @@ func NewTreePkglistCommand() *cobra.Command {
 
 	ans.Flags().BoolP("verbose", "v", false, "Add package version")
 	ans.Flags().BoolP("full", "f", false, "Show package detail")
-	ans.Flags().StringArrayP("tree", "t", []string{}, "Path of the tree to use.")
+	ans.Flags().StringArrayP("tree", "t", []string{path}, "Path of the tree to use.")
 	ans.Flags().StringSliceVarP(&matches, "matches", "m", []string{},
 		"Include only matched packages from list. (Use string as regex).")
 	ans.Flags().StringSliceVarP(&excludes, "exclude", "e", []string{},

cmd/tree/validate.go

@@ -25,10 +25,12 @@ import (
 	"strconv"
 	"sync"
 
+	"github.com/mudler/luet/pkg/api/core/types"
+
 	helpers "github.com/mudler/luet/cmd/helpers"
-	. "github.com/mudler/luet/pkg/config"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
+	"github.com/mudler/luet/cmd/util"
+
+	pkg "github.com/mudler/luet/pkg/database"
 	"github.com/mudler/luet/pkg/solver"
 	tree "github.com/mudler/luet/pkg/tree"
 
@@ -77,21 +79,21 @@ func (o *ValidateOpts) AddError(err error) {
 	o.Errors = append(o.Errors, err)
 }
 
-func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, reciper tree.Builder, cacheDeps *pkg.InMemoryDatabase) error {
+func validatePackage(p *types.Package, checkType string, opts *ValidateOpts, reciper tree.Builder, cacheDeps *pkg.InMemoryDatabase) error {
 	var errstr string
 	var ans error
 
-	var depSolver solver.PackageSolver
+	var depSolver types.PackageSolver
 
 	if opts.WithSolver {
 		emptyInstallationDb := pkg.NewInMemoryDatabase(false)
-		depSolver = solver.NewSolver(pkg.NewInMemoryDatabase(false),
+		depSolver = solver.NewSolver(types.SolverOptions{Type: types.SolverSingleCoreSimple}, pkg.NewInMemoryDatabase(false),
 			reciper.GetDatabase(),
 			emptyInstallationDb)
 	}
 
 	found, err := reciper.GetDatabase().FindPackages(
-		&pkg.DefaultPackage{
+		&types.Package{
 			Name:     p.GetName(),
 			Category: p.GetCategory(),
 			Version:  ">=0",
@@ -104,7 +106,7 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 		} else {
 			errstr = "No packages"
 		}
-		Error(fmt.Sprintf("[%9s] %s/%s-%s: Broken. No versions could be found by database %s",
+		util.DefaultContext.Error(fmt.Sprintf("[%9s] %s/%s-%s: Broken. No versions could be found by database %s",
 			checkType,
 			p.GetCategory(), p.GetName(), p.GetVersion(),
 			errstr,
@@ -122,7 +124,7 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 
 	// Ensure that we use the right package from right recipier for deps
 	pReciper, err := reciper.GetDatabase().FindPackage(
-		&pkg.DefaultPackage{
+		&types.Package{
 			Name:     p.GetName(),
 			Category: p.GetCategory(),
 			Version:  p.GetVersion(),
@@ -134,7 +136,7 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 			p.GetCategory(), p.GetName(), p.GetVersion(),
 			err.Error(),
 		)
-		Error(errstr)
+		util.DefaultContext.Error(errstr)
 
 		return errors.New(errstr)
 	}
@@ -173,7 +175,7 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 		}
 	}
 
-	Info(fmt.Sprintf("[%9s] Checking package ", checkType)+
+	util.DefaultContext.Info(fmt.Sprintf("[%9s] Checking package ", checkType)+
 		fmt.Sprintf("%s/%s-%s", p.GetCategory(), p.GetName(), p.GetVersion()),
 		"with", len(p.GetRequires()), "dependencies and", len(p.GetConflicts()), "conflicts.")
 
@@ -181,11 +183,11 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 	all = append(all, p.GetConflicts()...)
 	for idx, r := range all {
 
-		var deps pkg.Packages
+		var deps types.Packages
 		var err error
 		if r.IsSelector() {
 			deps, err = reciper.GetDatabase().FindPackages(
-				&pkg.DefaultPackage{
+				&types.Package{
 					Name:     r.GetName(),
 					Category: r.GetCategory(),
 					Version:  r.GetVersion(),
@@ -201,7 +203,7 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 			} else {
 				errstr = "No packages"
 			}
-			Error(fmt.Sprintf("[%9s] %s/%s-%s: Broken Dep %s/%s-%s - %s",
+			util.DefaultContext.Error(fmt.Sprintf("[%9s] %s/%s-%s: Broken Dep %s/%s-%s - %s",
 				checkType,
 				p.GetCategory(), p.GetName(), p.GetVersion(),
 				r.GetCategory(), r.GetName(), r.GetVersion(),
@@ -221,12 +223,12 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 
 		} else {
 
-			Debug(fmt.Sprintf("[%9s] Find packages for dep", checkType),
+			util.DefaultContext.Debug(fmt.Sprintf("[%9s] Find packages for dep", checkType),
 				fmt.Sprintf("%s/%s-%s", r.GetCategory(), r.GetName(), r.GetVersion()))
 
 			if opts.WithSolver {
 
-				Info(fmt.Sprintf("[%9s]  :soap: [%2d/%2d] %s/%s-%s: %s/%s-%s",
+				util.DefaultContext.Info(fmt.Sprintf("[%9s]  :soap: [%2d/%2d] %s/%s-%s: %s/%s-%s",
 					checkType,
 					idx+1, len(all),
 					p.GetCategory(), p.GetName(), p.GetVersion(),
@@ -236,22 +238,45 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 				// Check if the solver is already been done for the deep
 				_, err := cacheDeps.Get(r.HashFingerprint(""))
 				if err == nil {
-					Debug(fmt.Sprintf("[%9s]  :direct_hit: Cache Hit for dep", checkType),
+					util.DefaultContext.Debug(fmt.Sprintf("[%9s]  :direct_hit: Cache Hit for dep", checkType),
 						fmt.Sprintf("%s/%s-%s", r.GetCategory(), r.GetName(), r.GetVersion()))
 					continue
 				}
 
-				Spinner(32)
-				solution, err := depSolver.Install(pkg.Packages{r})
+				util.DefaultContext.Spinner()
+				solution, err := depSolver.Install(types.Packages{r})
 				ass := solution.SearchByName(r.GetPackageName())
+				util.DefaultContext.SpinnerStop()
 				if err == nil {
-					_, err = solution.Order(reciper.GetDatabase(), ass.Package.GetFingerPrint())
+					if ass == nil {
+
+						ans = errors.New(
+							fmt.Sprintf("[%9s] %s/%s-%s: solution doesn't retrieve package %s/%s-%s.",
+								checkType,
+								p.GetCategory(), p.GetName(), p.GetVersion(),
+								r.GetCategory(), r.GetName(), r.GetVersion(),
+							))
+
+						if util.DefaultContext.Config.General.Debug {
+							for idx, pa := range solution {
+								fmt.Println(fmt.Sprintf("[%9s] %s/%s-%s: solution %d: %s",
+									checkType,
+									p.GetCategory(), p.GetName(), p.GetVersion(), idx,
+									pa.Package.GetPackageName()))
+							}
+						}
+
+						util.DefaultContext.Error(ans.Error())
+						opts.IncrBrokenDeps()
+						validpkg = false
+					} else {
+						_, err = solution.Order(reciper.GetDatabase(), ass.Package.GetFingerPrint())
+					}
 				}
-				SpinnerStop()
 
 				if err != nil {
 
-					Error(fmt.Sprintf("[%9s] %s/%s-%s: solver broken for dep %s/%s-%s - %s",
+					util.DefaultContext.Error(fmt.Sprintf("[%9s] %s/%s-%s: solver broken for dep %s/%s-%s - %s",
 						checkType,
 						p.GetCategory(), p.GetName(), p.GetVersion(),
 						r.GetCategory(), r.GetName(), r.GetVersion(),
@@ -286,7 +311,7 @@ func validatePackage(p pkg.Package, checkType string, opts *ValidateOpts, recipe
 
 func validateWorker(i int,
 	wg *sync.WaitGroup,
-	c <-chan pkg.Package,
+	c <-chan *types.Package,
 	opts *ValidateOpts) {
 
 	defer wg.Done()
@@ -353,28 +378,28 @@ func initOpts(opts *ValidateOpts, onlyRuntime, onlyBuildtime, withSolver bool, t
 	opts.BuildtimeCacheDeps = pkg.NewInMemoryDatabase(false).(*pkg.InMemoryDatabase)
 
 	for _, treePath := range treePaths {
-		Info(fmt.Sprintf("Loading :deciduous_tree: %s...", treePath))
+		util.DefaultContext.Info(fmt.Sprintf("Loading :deciduous_tree: %s...", treePath))
 		if opts.BuildtimeReciper != nil {
 			err = opts.BuildtimeReciper.Load(treePath)
 			if err != nil {
-				Fatal("Error on load tree ", err)
+				util.DefaultContext.Fatal("Error on load tree ", err)
 			}
 		}
 		if opts.RuntimeReciper != nil {
 			err = opts.RuntimeReciper.Load(treePath)
 			if err != nil {
-				Fatal("Error on load tree ", err)
+				util.DefaultContext.Fatal("Error on load tree ", err)
 			}
 		}
 	}
 
 	opts.RegExcludes, err = helpers.CreateRegexArray(opts.Excludes)
 	if err != nil {
-		Fatal(err.Error())
+		util.DefaultContext.Fatal(err.Error())
 	}
 	opts.RegMatches, err = helpers.CreateRegexArray(opts.Matches)
 	if err != nil {
-		Fatal(err.Error())
+		util.DefaultContext.Fatal(err.Error())
 	}
 
 }
@@ -394,16 +419,16 @@ func NewTreeValidateCommand() *cobra.Command {
 			onlyBuildtime, _ := cmd.Flags().GetBool("only-buildtime")
 
 			if len(treePaths) < 1 {
-				Fatal("Mandatory tree param missing.")
+				util.DefaultContext.Fatal("Mandatory tree param missing.")
 			}
 			if onlyRuntime && onlyBuildtime {
-				Fatal("Both --only-runtime and --only-buildtime options are not possibile.")
+				util.DefaultContext.Fatal("Both --only-runtime and --only-buildtime options are not possibile.")
 			}
 		},
 		Run: func(cmd *cobra.Command, args []string) {
 			var reciper tree.Builder
 
-			concurrency := LuetCfg.GetGeneral().Concurrency
+			concurrency := util.DefaultContext.Config.General.Concurrency
 
 			withSolver, _ := cmd.Flags().GetBool("with-solver")
 			onlyRuntime, _ := cmd.Flags().GetBool("only-runtime")
@@ -420,7 +445,7 @@ func NewTreeValidateCommand() *cobra.Command {
 				reciper = opts.RuntimeReciper
 			}
 
-			all := make(chan pkg.Package)
+			all := make(chan *types.Package)
 
 			var wg = new(sync.WaitGroup)
 
@@ -449,21 +474,24 @@ func NewTreeValidateCommand() *cobra.Command {
 
 			// fmt.Println("Broken packages:", brokenPkgs, "(", brokenDeps, "deps ).")
 			if len(stringerrs) != 0 {
-				Error(fmt.Sprintf("Found %d broken packages and %d broken deps.",
+				util.DefaultContext.Error(fmt.Sprintf("Found %d broken packages and %d broken deps.",
 					opts.BrokenPkgs, opts.BrokenDeps))
-				Fatal("Errors: " + strconv.Itoa(len(stringerrs)))
+				util.DefaultContext.Fatal("Errors: " + strconv.Itoa(len(stringerrs)))
 			} else {
-				Info("All good! :white_check_mark:")
+				util.DefaultContext.Info("All good! :white_check_mark:")
 				os.Exit(0)
 			}
 		},
 	}
-
+	path, err := os.Getwd()
+	if err != nil {
+		util.DefaultContext.Fatal(err)
+	}
 	ans.Flags().Bool("only-runtime", false, "Check only runtime dependencies.")
 	ans.Flags().Bool("only-buildtime", false, "Check only buildtime dependencies.")
 	ans.Flags().BoolP("with-solver", "s", false,
 		"Enable check of requires also with solver.")
-	ans.Flags().StringSliceVarP(&treePaths, "tree", "t", []string{},
+	ans.Flags().StringSliceVarP(&treePaths, "tree", "t", []string{path},
 		"Path of the tree to use.")
 	ans.Flags().StringSliceVarP(&excludes, "exclude", "e", []string{},
 		"Exclude matched packages from analysis. (Use string as regex).")

cmd/uninstall.go

@@ -15,16 +15,13 @@
 package cmd
 
 import (
-	"os"
-	"path/filepath"
-
 	helpers "github.com/mudler/luet/cmd/helpers"
-	. "github.com/mudler/luet/pkg/config"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
 	installer "github.com/mudler/luet/pkg/installer"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
 
 	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
 )
 
 var uninstallCmd = &cobra.Command{
@@ -33,83 +30,67 @@ var uninstallCmd = &cobra.Command{
 	Long:    `Uninstall packages`,
 	Aliases: []string{"rm", "un"},
 	PreRun: func(cmd *cobra.Command, args []string) {
-		LuetCfg.Viper.BindPFlag("system.database_path", cmd.Flags().Lookup("system-dbpath"))
-		LuetCfg.Viper.BindPFlag("system.rootfs", cmd.Flags().Lookup("system-target"))
-		LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
-		LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
-		LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
-		LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
-		LuetCfg.Viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
-		LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+
+		viper.BindPFlag("nodeps", cmd.Flags().Lookup("nodeps"))
+		viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
 	},
 	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
-
+		toRemove := []*types.Package{}
 		for _, a := range args {
 
 			pack, err := helpers.ParsePackageStr(a)
 			if err != nil {
-				Fatal("Invalid package string ", a, ": ", err.Error())
+				util.DefaultContext.Fatal("Invalid package string ", a, ": ", err.Error())
 			}
+			toRemove = append(toRemove, pack)
+		}
 
-			stype := LuetCfg.Viper.GetString("solver.type")
-			discount := LuetCfg.Viper.GetFloat64("solver.discount")
-			rate := LuetCfg.Viper.GetFloat64("solver.rate")
-			attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
-			force := LuetCfg.Viper.GetBool("force")
-			nodeps, _ := cmd.Flags().GetBool("nodeps")
-			full, _ := cmd.Flags().GetBool("full")
-			checkconflicts, _ := cmd.Flags().GetBool("conflictscheck")
-			fullClean, _ := cmd.Flags().GetBool("full-clean")
+		force := viper.GetBool("force")
+		nodeps, _ := cmd.Flags().GetBool("nodeps")
+		full, _ := cmd.Flags().GetBool("full")
+		checkconflicts, _ := cmd.Flags().GetBool("conflictscheck")
+		fullClean, _ := cmd.Flags().GetBool("full-clean")
+		yes := viper.GetBool("yes")
+		keepProtected, _ := cmd.Flags().GetBool("keep-protected-files")
 
-			LuetCfg.GetSolverOptions().Type = stype
-			LuetCfg.GetSolverOptions().LearnRate = float32(rate)
-			LuetCfg.GetSolverOptions().Discount = float32(discount)
-			LuetCfg.GetSolverOptions().MaxAttempts = attempts
+		util.DefaultContext.Config.ConfigProtectSkip = !keepProtected
 
-			Debug("Solver", LuetCfg.GetSolverOptions().CompactString())
+		util.DefaultContext.Config.Solver.Implementation = types.SolverSingleCoreSimple
 
-			inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
-				Concurrency:        LuetCfg.GetGeneral().Concurrency,
-				SolverOptions:      *LuetCfg.GetSolverOptions(),
-				NoDeps:             nodeps,
-				Force:              force,
-				FullUninstall:      full,
-				FullCleanUninstall: fullClean,
-				CheckConflicts:     checkconflicts,
-			})
+		util.DefaultContext.Debug("Solver", util.DefaultContext.Config.Solver.CompactString())
 
-			if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-				systemDB = pkg.NewBoltDatabase(
-					filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-			} else {
-				systemDB = pkg.NewInMemoryDatabase(true)
-			}
-			system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
-			err = inst.Uninstall(pack, system)
-			if err != nil {
-				Fatal("Error: " + err.Error())
-			}
+		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
+			Concurrency:                 util.DefaultContext.Config.General.Concurrency,
+			SolverOptions:               util.DefaultContext.Config.Solver,
+			NoDeps:                      nodeps,
+			Force:                       force,
+			FullUninstall:               full,
+			FullCleanUninstall:          fullClean,
+			CheckConflicts:              checkconflicts,
+			Ask:                         !yes,
+			PreserveSystemEssentialData: true,
+			Context:                     util.DefaultContext,
+		})
+
+		system := &installer.System{Database: util.SystemDB(util.DefaultContext.Config), Target: util.DefaultContext.Config.System.Rootfs}
+
+		if err := inst.Uninstall(system, toRemove...); err != nil {
+			util.DefaultContext.Fatal("Error: " + err.Error())
 		}
 	},
 }
 
 func init() {
-	path, err := os.Getwd()
-	if err != nil {
-		Fatal(err)
-	}
-	uninstallCmd.Flags().String("system-dbpath", path, "System db path")
-	uninstallCmd.Flags().String("system-target", path, "System rootpath")
-	uninstallCmd.Flags().String("solver-type", "", "Solver strategy ( Defaults none, available: "+AvailableResolvers+" )")
-	uninstallCmd.Flags().Float32("solver-rate", 0.7, "Solver learning rate")
-	uninstallCmd.Flags().Float32("solver-discount", 1.0, "Solver discount rate")
-	uninstallCmd.Flags().Int("solver-attempts", 9000, "Solver maximum attempts")
+
 	uninstallCmd.Flags().Bool("nodeps", false, "Don't consider package dependencies (harmful! overrides checkconflicts and full!)")
 	uninstallCmd.Flags().Bool("force", false, "Force uninstall")
 	uninstallCmd.Flags().Bool("full", false, "Attempts to remove as much packages as possible which aren't required (slow)")
 	uninstallCmd.Flags().Bool("conflictscheck", true, "Check if the package marked for deletion is required by other packages")
 	uninstallCmd.Flags().Bool("full-clean", false, "(experimental) Uninstall packages and all the other deps/revdeps of it.")
+	uninstallCmd.Flags().Bool("solver-concurrent", false, "Use concurrent solver (experimental)")
+	uninstallCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
+	uninstallCmd.Flags().BoolP("keep-protected-files", "k", false, "Keep package protected files around")
 
 	RootCmd.AddCommand(uninstallCmd)
 }

cmd/upgrade.go

@@ -15,15 +15,12 @@
 package cmd
 
 import (
-	"os"
-	"path/filepath"
-
-	. "github.com/mudler/luet/pkg/config"
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/types"
 	installer "github.com/mudler/luet/pkg/installer"
-	. "github.com/mudler/luet/pkg/logger"
-	pkg "github.com/mudler/luet/pkg/package"
 
 	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
 )
 
 var upgradeCmd = &cobra.Command{
@@ -31,93 +28,63 @@ var upgradeCmd = &cobra.Command{
 	Short:   "Upgrades the system",
 	Aliases: []string{"u"},
 	PreRun: func(cmd *cobra.Command, args []string) {
-		LuetCfg.Viper.BindPFlag("system.database_path", installCmd.Flags().Lookup("system-dbpath"))
-		LuetCfg.Viper.BindPFlag("system.rootfs", installCmd.Flags().Lookup("system-target"))
-		LuetCfg.Viper.BindPFlag("solver.type", cmd.Flags().Lookup("solver-type"))
-		LuetCfg.Viper.BindPFlag("solver.discount", cmd.Flags().Lookup("solver-discount"))
-		LuetCfg.Viper.BindPFlag("solver.rate", cmd.Flags().Lookup("solver-rate"))
-		LuetCfg.Viper.BindPFlag("solver.max_attempts", cmd.Flags().Lookup("solver-attempts"))
-		LuetCfg.Viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+
+		viper.BindPFlag("force", cmd.Flags().Lookup("force"))
+		viper.BindPFlag("yes", cmd.Flags().Lookup("yes"))
 	},
 	Long: `Upgrades packages in parallel`,
 	Run: func(cmd *cobra.Command, args []string) {
-		var systemDB pkg.PackageDatabase
 
-		repos := installer.Repositories{}
-		for _, repo := range LuetCfg.SystemRepositories {
-			if !repo.Enable {
-				continue
-			}
-
-			r := installer.NewSystemRepository(repo)
-			repos = append(repos, r)
-		}
-
-		stype := LuetCfg.Viper.GetString("solver.type")
-		discount := LuetCfg.Viper.GetFloat64("solver.discount")
-		rate := LuetCfg.Viper.GetFloat64("solver.rate")
-		attempts := LuetCfg.Viper.GetInt("solver.max_attempts")
-		force := LuetCfg.Viper.GetBool("force")
+		force := viper.GetBool("force")
 		nodeps, _ := cmd.Flags().GetBool("nodeps")
 		full, _ := cmd.Flags().GetBool("full")
 		universe, _ := cmd.Flags().GetBool("universe")
 		clean, _ := cmd.Flags().GetBool("clean")
 		sync, _ := cmd.Flags().GetBool("sync")
+		osCheck, _ := cmd.Flags().GetBool("oscheck")
 
-		LuetCfg.GetSolverOptions().Type = stype
-		LuetCfg.GetSolverOptions().LearnRate = float32(rate)
-		LuetCfg.GetSolverOptions().Discount = float32(discount)
-		LuetCfg.GetSolverOptions().MaxAttempts = attempts
+		yes := viper.GetBool("yes")
+		downloadOnly, _ := cmd.Flags().GetBool("download-only")
 
-		Debug("Solver", LuetCfg.GetSolverOptions().String())
+		util.DefaultContext.Config.Solver.Implementation = types.SolverSingleCoreSimple
+
+		util.DefaultContext.Debug("Solver", util.DefaultContext.GetConfig().Solver)
 
 		inst := installer.NewLuetInstaller(installer.LuetInstallerOptions{
-			Concurrency:                LuetCfg.GetGeneral().Concurrency,
-			SolverOptions:              *LuetCfg.GetSolverOptions(),
-			Force:                      force,
-			FullUninstall:              full,
-			NoDeps:                     nodeps,
-			SolverUpgrade:              universe,
-			RemoveUnavailableOnUpgrade: clean,
-			UpgradeNewRevisions:        sync,
+			Concurrency:                 util.DefaultContext.Config.General.Concurrency,
+			SolverOptions:               util.DefaultContext.Config.Solver,
+			Force:                       force,
+			FullUninstall:               full,
+			NoDeps:                      nodeps,
+			SolverUpgrade:               universe,
+			RemoveUnavailableOnUpgrade:  clean,
+			UpgradeNewRevisions:         sync,
+			PreserveSystemEssentialData: true,
+			Ask:                         !yes,
+			AutoOSCheck:                 osCheck,
+			DownloadOnly:                downloadOnly,
+			PackageRepositories:         util.DefaultContext.Config.SystemRepositories,
+			Context:                     util.DefaultContext,
 		})
-		inst.Repositories(repos)
-		_, err := inst.SyncRepositories(false)
-		if err != nil {
-			Fatal("Error: " + err.Error())
-		}
 
-		if LuetCfg.GetSystem().DatabaseEngine == "boltdb" {
-			systemDB = pkg.NewBoltDatabase(
-				filepath.Join(LuetCfg.GetSystem().GetSystemRepoDatabaseDirPath(), "luet.db"))
-		} else {
-			systemDB = pkg.NewInMemoryDatabase(true)
-		}
-		system := &installer.System{Database: systemDB, Target: LuetCfg.GetSystem().Rootfs}
-		err = inst.Upgrade(system)
-		if err != nil {
-			Fatal("Error: " + err.Error())
+		system := &installer.System{Database: util.SystemDB(util.DefaultContext.Config), Target: util.DefaultContext.Config.System.Rootfs}
+		if err := inst.Upgrade(system); err != nil {
+			util.DefaultContext.Fatal("Error: " + err.Error())
 		}
 	},
 }
 
 func init() {
-	path, err := os.Getwd()
-	if err != nil {
-		Fatal(err)
-	}
-	upgradeCmd.Flags().String("system-dbpath", path, "System db path")
-	upgradeCmd.Flags().String("system-target", path, "System rootpath")
-	upgradeCmd.Flags().String("solver-type", "", "Solver strategy ( Defaults none, available: "+AvailableResolvers+" )")
-	upgradeCmd.Flags().Float32("solver-rate", 0.7, "Solver learning rate")
-	upgradeCmd.Flags().Float32("solver-discount", 1.0, "Solver discount rate")
-	upgradeCmd.Flags().Int("solver-attempts", 9000, "Solver maximum attempts")
 	upgradeCmd.Flags().Bool("force", false, "Force upgrade by ignoring errors")
 	upgradeCmd.Flags().Bool("nodeps", false, "Don't consider package dependencies (harmful! overrides checkconflicts and full!)")
-	upgradeCmd.Flags().Bool("full", true, "Attempts to remove as much packages as possible which aren't required (slow)")
+	upgradeCmd.Flags().Bool("full", false, "Attempts to remove as much packages as possible which aren't required (slow)")
 	upgradeCmd.Flags().Bool("universe", false, "Use ONLY the SAT solver to compute upgrades (experimental)")
 	upgradeCmd.Flags().Bool("clean", false, "Try to drop removed packages (experimental, only when --universe is enabled)")
 	upgradeCmd.Flags().Bool("sync", false, "Upgrade packages with new revisions (experimental)")
+	upgradeCmd.Flags().Bool("solver-concurrent", false, "Use concurrent solver (experimental)")
+	upgradeCmd.Flags().BoolP("yes", "y", false, "Don't ask questions")
+	upgradeCmd.Flags().Bool("download-only", false, "Download only")
+	upgradeCmd.Flags().Bool("oscheck", false, "Perform automatically oschecks after upgrades")
 
 	RootCmd.AddCommand(upgradeCmd)
 }

cmd/util.go

@@ -0,0 +1,202 @@
+// Copyright © 2020-2021 Ettore Di Giacinto <mudler@gentoo.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package cmd
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+	"strings"
+
+	registrytypes "github.com/docker/docker/api/types/registry"
+	"github.com/docker/go-units"
+	"github.com/mudler/luet/pkg/api/core/image"
+	fileHelper "github.com/mudler/luet/pkg/helpers/file"
+	"github.com/pkg/errors"
+
+	"github.com/mudler/luet/cmd/util"
+	"github.com/mudler/luet/pkg/api/core/context"
+	"github.com/mudler/luet/pkg/helpers/docker"
+
+	"github.com/spf13/cobra"
+)
+
+const (
+	filePrefix = "file://"
+)
+
+func pack(ctx *context.Context, p, dst, imageName, arch, OS string) error {
+
+	tempimage, err := ctx.TempFile("tempimage")
+	if err != nil {
+		return errors.Wrap(err, "error met while creating tempdir for "+p)
+	}
+	defer os.RemoveAll(tempimage.Name()) // clean up
+
+	if err := image.CreateTar(p, tempimage.Name(), imageName, arch, OS); err != nil {
+		return errors.Wrap(err, "could not create image from tar")
+	}
+
+	return fileHelper.CopyFile(tempimage.Name(), dst)
+}
+
+func NewPackCommand() *cobra.Command {
+
+	c := &cobra.Command{
+		Use:   "pack image src.tar dst.tar",
+		Short: "Pack a standard tar archive as a container image",
+		Long: `Pack creates a tar which can be loaded as an image from a standard flat tar archive, for e.g. with docker load. 
+It doesn't need the docker daemon to run, and allows to override default os/arch:
+		
+	luet util pack --os arm64 image:tag src.tar dst.tar
+`,
+		Args: cobra.MinimumNArgs(3),
+		Run: func(cmd *cobra.Command, args []string) {
+
+			image := args[0]
+			src := args[1]
+			dst := args[2]
+
+			arch, _ := cmd.Flags().GetString("arch")
+			os, _ := cmd.Flags().GetString("os")
+
+			err := pack(util.DefaultContext, src, dst, image, arch, os)
+			if err != nil {
+				util.DefaultContext.Fatal(err.Error())
+			}
+			util.DefaultContext.Info("Image packed as", image)
+		},
+	}
+
+	c.Flags().String("arch", runtime.GOARCH, "Image architecture")
+	c.Flags().String("os", runtime.GOOS, "Image OS")
+
+	return c
+}
+
+func NewUnpackCommand() *cobra.Command {
+
+	c := &cobra.Command{
+		Use:   "unpack image path",
+		Short: "Unpack a docker image natively",
+		Long: `unpack doesn't need the docker daemon to run, and unpacks a docker image in the specified directory:
+
+	luet util unpack golang:alpine /alpine
+`,
+		PreRun: func(cmd *cobra.Command, args []string) {
+
+			if len(args) != 2 {
+				util.DefaultContext.Fatal("Expects an image and a path")
+			}
+
+		},
+		Run: func(cmd *cobra.Command, args []string) {
+
+			image := args[0]
+			destination, err := filepath.Abs(args[1])
+			if err != nil {
+				util.DefaultContext.Error("Invalid path %s", destination)
+				os.Exit(1)
+			}
+			local, _ := cmd.Flags().GetBool("local")
+			verify, _ := cmd.Flags().GetBool("verify")
+			user, _ := cmd.Flags().GetString("auth-username")
+			pass, _ := cmd.Flags().GetString("auth-password")
+			authType, _ := cmd.Flags().GetString("auth-type")
+			server, _ := cmd.Flags().GetString("auth-server-address")
+			identity, _ := cmd.Flags().GetString("auth-identity-token")
+			registryToken, _ := cmd.Flags().GetString("auth-registry-token")
+
+			util.DefaultContext.Info("Downloading", image, "to", destination)
+			auth := &registrytypes.AuthConfig{
+				Username:      user,
+				Password:      pass,
+				ServerAddress: server,
+				Auth:          authType,
+				IdentityToken: identity,
+				RegistryToken: registryToken,
+			}
+
+			if !local && !strings.HasPrefix(image, filePrefix) {
+				info, err := docker.DownloadAndExtractDockerImage(util.DefaultContext, image, destination, auth, verify)
+				if err != nil {
+					util.DefaultContext.Error(err.Error())
+					os.Exit(1)
+				}
+				util.DefaultContext.Info(fmt.Sprintf("Pulled: %s %s", info.Target.Digest, info.Name))
+				util.DefaultContext.Info(fmt.Sprintf("Size: %s", units.BytesSize(float64(info.Target.Size))))
+			} else {
+				info, err := docker.ExtractDockerImage(util.DefaultContext, image, destination)
+				if err != nil {
+					util.DefaultContext.Error(err.Error())
+					os.Exit(1)
+				}
+				util.DefaultContext.Info(fmt.Sprintf("Size: %s", units.BytesSize(float64(info.Target.Size))))
+			}
+		},
+	}
+
+	c.Flags().String("auth-username", "", "Username to authenticate to registry/notary")
+	c.Flags().String("auth-password", "", "Password to authenticate to registry")
+	c.Flags().String("auth-type", "", "Auth type")
+	c.Flags().String("auth-server-address", "", "Authentication server address")
+	c.Flags().String("auth-identity-token", "", "Authentication identity token")
+	c.Flags().String("auth-registry-token", "", "Authentication registry token")
+	c.Flags().Bool("verify", false, "Verify signed images to notary before to pull")
+	c.Flags().Bool("local", false, "Unpack local image")
+	return c
+}
+
+func NewExistCommand() *cobra.Command {
+
+	c := &cobra.Command{
+		Use:   "image-exist image path",
+		Short: "Check if an image exist",
+		Long:  `Exits 0 if the image exist, otherwise exits with 1`,
+		PreRun: func(cmd *cobra.Command, args []string) {
+
+			if len(args) != 1 {
+				util.DefaultContext.Fatal("Expects an image")
+			}
+
+		},
+		Run: func(cmd *cobra.Command, args []string) {
+			if image.Available(args[0]) {
+				os.Exit(0)
+			} else {
+				os.Exit(1)
+			}
+		},
+	}
+
+	return c
+}
+
+var utilGroup = &cobra.Command{
+	Use:   "util [command] [OPTIONS]",
+	Short: "General luet internal utilities exposed",
+}
+
+func init() {
+	RootCmd.AddCommand(utilGroup)
+
+	utilGroup.AddCommand(
+		NewUnpackCommand(),
+		NewPackCommand(),
+		NewExistCommand(),
+	)
+}

cmd/util/cli.go

@@ -0,0 +1,92 @@
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package util
+
+import (
+	"fmt"
+	"os"
+	"strings"
+
+	"github.com/marcsauter/single"
+	"github.com/pterm/pterm"
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+
+	"github.com/mudler/luet/pkg/api/core/context"
+	"github.com/mudler/luet/pkg/api/core/template"
+	"github.com/mudler/luet/pkg/installer"
+)
+
+var lockedCommands = []string{"install", "uninstall", "upgrade"}
+var bannerCommands = []string{"install", "build", "uninstall", "upgrade"}
+
+func BindValuesFlags(cmd *cobra.Command) {
+	viper.BindPFlag("values", cmd.Flags().Lookup("values"))
+}
+
+func ValuesFlags() []string {
+	return viper.GetStringSlice("values")
+}
+
+// TemplateFolders returns the default folders which holds shared template between packages in a given tree path
+func TemplateFolders(ctx *context.Context, i installer.BuildTreeResult, treePaths []string) []string {
+	templateFolders := []string{}
+	for _, t := range treePaths {
+		templateFolders = append(templateFolders, template.FindPossibleTemplatesDir(t)...)
+	}
+	for _, r := range i.TemplatesDir {
+		templateFolders = append(templateFolders, r...)
+	}
+
+	return templateFolders
+}
+
+func HandleLock() {
+	if os.Getenv("LUET_NOLOCK") == "true" || len(os.Args) < 2 {
+		return
+	}
+
+	for _, lockedCmd := range lockedCommands {
+		if os.Args[1] == lockedCmd {
+			s := single.New("luet")
+			if err := s.CheckLock(); err != nil && err == single.ErrAlreadyRunning {
+				fmt.Println("another instance of the app is already running, exiting")
+				os.Exit(1)
+			} else if err != nil {
+				// Another error occurred, might be worth handling it as well
+				fmt.Println("failed to acquire exclusive app lock:", err.Error())
+				os.Exit(1)
+			}
+			defer s.TryUnlock()
+			break
+		}
+	}
+}
+
+func DisplayVersionBanner(c *context.Context, version func() string, license []string) {
+	display := false
+	if len(os.Args) > 1 {
+		for _, c := range bannerCommands {
+			if os.Args[1] == c {
+				display = true
+			}
+		}
+	}
+	if display {
+		pterm.Info.Printf("Luet %s\n", version())
+		pterm.Info.Println(strings.Join(license, "\n"))
+	}
+}

cmd/util/config.go

@@ -0,0 +1,320 @@
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package util
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"os/user"
+	"path/filepath"
+	"runtime"
+	"strings"
+
+	"github.com/ipfs/go-log/v2"
+	extensions "github.com/mudler/cobra-extensions"
+	"github.com/mudler/luet/pkg/api/core/context"
+	gc "github.com/mudler/luet/pkg/api/core/garbagecollector"
+	"github.com/mudler/luet/pkg/api/core/logger"
+	"github.com/mudler/luet/pkg/api/core/types"
+	"github.com/mudler/luet/pkg/solver"
+	"github.com/pterm/pterm"
+	"go.uber.org/zap/zapcore"
+
+	helpers "github.com/mudler/luet/pkg/helpers"
+	fileHelper "github.com/mudler/luet/pkg/helpers/file"
+
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+)
+
+const (
+	LuetEnvPrefix = "LUET"
+)
+
+var cfgFile string
+
+// initConfig reads in config file and ENV variables if set.
+func initConfig() {
+	setDefaults(viper.GetViper())
+	// Luet support these priorities on read configuration file:
+	// - command line option (if available)
+	// - $PWD/.luet.yaml
+	// - $HOME/.luet.yaml
+	// - /etc/luet/luet.yaml
+	//
+	// Note: currently a single viper instance support only one config name.
+
+	viper.SetEnvPrefix(LuetEnvPrefix)
+	viper.SetConfigType("yaml")
+
+	if cfgFile != "" { // enable ability to specify config file via flag
+		viper.SetConfigFile(cfgFile)
+	} else {
+		// Retrieve pwd directory
+		pwdDir, err := os.Getwd()
+		if err != nil {
+			fmt.Println(err)
+			os.Exit(1)
+		}
+		homeDir := helpers.GetHomeDir()
+
+		if fileHelper.Exists(filepath.Join(pwdDir, ".luet.yaml")) || (homeDir != "" && fileHelper.Exists(filepath.Join(homeDir, ".luet.yaml"))) {
+			viper.AddConfigPath(".")
+			if homeDir != "" {
+				viper.AddConfigPath(homeDir)
+			}
+			viper.SetConfigName(".luet")
+		} else {
+			viper.SetConfigName("luet")
+			viper.AddConfigPath("/etc/luet")
+		}
+	}
+
+	viper.AutomaticEnv() // read in environment variables that match
+
+	// Create EnvKey Replacer for handle complex structure
+	replacer := strings.NewReplacer(".", "__")
+	viper.SetEnvKeyReplacer(replacer)
+	viper.SetTypeByDefaultValue(true)
+	// If a config file is found, read it in.
+	viper.ReadInConfig()
+
+}
+
+var DefaultContext *context.Context
+
+// InitContext inits the context by parsing the configurations from viper
+// this is meant to be run before each command to be able to parse any override from
+// the CLI/ENV
+func InitContext(cmd *cobra.Command) (ctx *context.Context, err error) {
+
+	c := &types.LuetConfig{}
+	err = viper.Unmarshal(c)
+	if err != nil {
+		return
+	}
+
+	// Converts user-defined config into paths
+	// and creates the required directory on the system if necessary
+	c.Init()
+
+	finalizerEnvs, _ := cmd.Flags().GetStringArray("finalizer-env")
+	setCliFinalizerEnvs(c, finalizerEnvs)
+
+	c.Solver.SolverOptions = types.SolverOptions{Type: types.SolverSingleCoreSimple, Concurrency: c.General.Concurrency}
+
+	ctx = context.NewContext(
+		context.WithConfig(c),
+		context.WithGarbageCollector(gc.GarbageCollector(c.System.TmpDirBase)),
+	)
+
+	// Inits the context with the configurations loaded
+	// It reads system repositories, sets logging, and all the
+	// context which is required to perform luet actions
+	return ctx, initContext(cmd, ctx)
+}
+
+func setCliFinalizerEnvs(c *types.LuetConfig, finalizerEnvs []string) error {
+	if len(finalizerEnvs) > 0 {
+		for _, v := range finalizerEnvs {
+			idx := strings.Index(v, "=")
+			if idx < 0 {
+				return errors.New("Found invalid runtime finalizer environment: " + v)
+			}
+
+			c.SetFinalizerEnv(v[0:idx], v[idx+1:])
+		}
+	}
+
+	return nil
+}
+
+const (
+	CommandProcessOutput = "command.process.output"
+)
+
+func initContext(cmd *cobra.Command, c *context.Context) (err error) {
+	if logger.IsTerminal() {
+		if !c.Config.Logging.Color {
+			pterm.DisableColor()
+		}
+	} else {
+		pterm.DisableColor()
+		c.Debug("Not a terminal, colors disabled")
+	}
+
+	if c.Config.General.Quiet {
+		pterm.DisableColor()
+		pterm.DisableStyling()
+	}
+
+	level := c.Config.Logging.Level
+	if c.Config.General.Debug {
+		level = "debug"
+	}
+
+	if _, ok := cmd.Annotations[CommandProcessOutput]; ok {
+		// Note: create-repo output is different, so we annotate in the cmd of create-repo CommandNoProcess
+		// to avoid
+		out, _ := cmd.Flags().GetString("output")
+		if out != "terminal" {
+			level = zapcore.Level(log.LevelFatal).String()
+		}
+	}
+
+	// Init logging
+	opts := []logger.LoggerOptions{
+		logger.WithLevel(level),
+	}
+
+	if c.Config.Logging.NoSpinner {
+		opts = append(opts, logger.NoSpinner)
+	}
+
+	if c.Config.Logging.EnableLogFile && c.Config.Logging.Path != "" {
+		f := "console"
+		if c.Config.Logging.JSONFormat {
+			f = "json"
+		}
+		opts = append(opts, logger.WithFileLogging(c.Config.Logging.Path, f))
+	}
+
+	if c.Config.Logging.EnableEmoji {
+		opts = append(opts, logger.EnableEmoji())
+	}
+
+	l, err := logger.New(opts...)
+
+	c.Logger = l
+
+	c.Debug("System rootfs:", c.Config.System.Rootfs)
+	c.Debug("Colors", c.Config.Logging.Color)
+	c.Debug("Logging level", c.Config.Logging.Level)
+	c.Debug("Debug mode", c.Config.General.Debug)
+
+	return
+}
+
+func setDefaults(viper *viper.Viper) {
+	viper.SetDefault("logging.level", "info")
+	viper.SetDefault("logging.enable_logfile", false)
+	viper.SetDefault("logging.path", "/var/log/luet.log")
+	viper.SetDefault("logging.json_format", false)
+	viper.SetDefault("logging.enable_emoji", true)
+	viper.SetDefault("logging.color", true)
+
+	viper.SetDefault("general.concurrency", runtime.NumCPU())
+	viper.SetDefault("general.debug", false)
+	viper.SetDefault("general.quiet", false)
+	viper.SetDefault("general.show_build_output", true)
+	viper.SetDefault("general.fatal_warnings", false)
+	viper.SetDefault("general.http_timeout", 360)
+
+	u, err := user.Current()
+	// os/user doesn't work in from scratch environments
+	if err != nil || (u != nil && u.Uid == "0") {
+		viper.SetDefault("general.same_owner", true)
+	} else {
+		viper.SetDefault("general.same_owner", false)
+	}
+
+	viper.SetDefault("system.database_engine", "boltdb")
+	viper.SetDefault("system.database_path", "/var/cache/luet")
+	viper.SetDefault("system.rootfs", "/")
+	viper.SetDefault("system.tmpdir_base", filepath.Join(os.TempDir(), "tmpluet"))
+	viper.SetDefault("system.pkgs_cache_path", "packages")
+
+	viper.SetDefault("repos_confdir", []string{"/etc/luet/repos.conf.d"})
+	viper.SetDefault("config_protect_confdir", []string{"/etc/luet/config.protect.d"})
+	viper.SetDefault("config_protect_skip", false)
+	// TODO: Set default to false when we are ready for migration.
+	viper.SetDefault("config_from_host", true)
+	viper.SetDefault("cache_repositories", []string{})
+	viper.SetDefault("system_repositories", []string{})
+	viper.SetDefault("finalizer_envs", make(map[string]string))
+
+	viper.SetDefault("solver.type", "")
+	viper.SetDefault("solver.rate", 0.7)
+	viper.SetDefault("solver.discount", 1.0)
+	viper.SetDefault("solver.max_attempts", 9000)
+}
+
+// InitViper inits a new viper
+// this is meant to be run just once at beginning to setup the root command
+func InitViper(RootCmd *cobra.Command) {
+	cobra.OnInitialize(initConfig)
+	pflags := RootCmd.PersistentFlags()
+	pflags.StringVar(&cfgFile, "config", "", "config file (default is $HOME/.luet.yaml)")
+	pflags.BoolP("debug", "d", false, "debug output")
+	pflags.BoolP("quiet", "q", false, "quiet output")
+	pflags.Bool("fatal", false, "Enables Warnings to exit")
+	pflags.Bool("enable-logfile", false, "Enable log to file")
+	pflags.Bool("no-spinner", false, "Disable spinner.")
+	pflags.Bool("color", true, "Enable/Disable color.")
+	pflags.Bool("emoji", true, "Enable/Disable emoji.")
+	pflags.Bool("skip-config-protect", true, "Disable config protect analysis.")
+	pflags.StringP("logfile", "l", "", "Logfile path. Empty value disable log to file.")
+	pflags.StringSlice("plugin", []string{}, "A list of runtime plugins to load")
+
+	pflags.String("system-dbpath", "", "System db path")
+	pflags.String("system-target", "", "System rootpath")
+	pflags.String("system-engine", "", "System DB engine")
+
+	pflags.String("solver-type", "", "Solver strategy ( Defaults none, available: "+solver.AvailableResolvers+" )")
+	pflags.Float32("solver-rate", 0.7, "Solver learning rate")
+	pflags.Float32("solver-discount", 1.0, "Solver discount rate")
+	pflags.Int("solver-attempts", 9000, "Solver maximum attempts")
+	pflags.Bool("live-output", true, "Show live output during build")
+
+	pflags.Bool("same-owner", true, "Maintain same owner on uncompress.")
+	pflags.Int("concurrency", runtime.NumCPU(), "Concurrency")
+	pflags.Int("http-timeout", 360, "Default timeout for http(s) requests")
+
+	viper.BindPFlag("system.database_path", pflags.Lookup("system-dbpath"))
+	viper.BindPFlag("system.rootfs", pflags.Lookup("system-target"))
+	viper.BindPFlag("system.database_engine", pflags.Lookup("system-engine"))
+	viper.BindPFlag("solver.type", pflags.Lookup("solver-type"))
+	viper.BindPFlag("solver.discount", pflags.Lookup("solver-discount"))
+	viper.BindPFlag("solver.rate", pflags.Lookup("solver-rate"))
+	viper.BindPFlag("solver.max_attempts", pflags.Lookup("solver-attempts"))
+
+	viper.BindPFlag("logging.color", pflags.Lookup("color"))
+	viper.BindPFlag("logging.enable_emoji", pflags.Lookup("emoji"))
+	viper.BindPFlag("logging.enable_logfile", pflags.Lookup("enable-logfile"))
+	viper.BindPFlag("logging.path", pflags.Lookup("logfile"))
+	viper.BindPFlag("logging.no_spinner", pflags.Lookup("no-spinner"))
+	viper.BindPFlag("general.concurrency", pflags.Lookup("concurrency"))
+	viper.BindPFlag("general.debug", pflags.Lookup("debug"))
+	viper.BindPFlag("general.quiet", pflags.Lookup("quiet"))
+	viper.BindPFlag("general.fatal_warnings", pflags.Lookup("fatal"))
+	viper.BindPFlag("general.same_owner", pflags.Lookup("same-owner"))
+	viper.BindPFlag("plugin", pflags.Lookup("plugin"))
+	viper.BindPFlag("general.http_timeout", pflags.Lookup("http-timeout"))
+	viper.BindPFlag("general.show_build_output", pflags.Lookup("live-output"))
+
+	// Currently I maintain this only from cli.
+	viper.BindPFlag("no_spinner", pflags.Lookup("no-spinner"))
+	viper.BindPFlag("config_protect_skip", pflags.Lookup("skip-config-protect"))
+
+	// Extensions must be binary with the "luet-" prefix to be able to be shown in the help.
+	// we also accept extensions in the relative path where luet is being started, "extensions/"
+	exts := extensions.Discover("luet", "extensions")
+	for _, ex := range exts {
+		cobraCmd := ex.CobraCommand()
+		RootCmd.AddCommand(cobraCmd)
+	}
+}

cmd/util/search.go

@@ -0,0 +1,42 @@
+// Copyright © 2021 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package util
+
+import "github.com/pterm/pterm"
+
+type TableWriter struct {
+	td pterm.TableData
+}
+
+func (l *TableWriter) AppendRow(item []string) {
+	l.td = append(l.td, item)
+}
+
+func (l *TableWriter) Render() {
+	pterm.DefaultTable.WithHasHeader().WithData(l.td).Render()
+}
+
+type ListWriter struct {
+	bb []pterm.BulletListItem
+}
+
+func (l *ListWriter) AppendItem(item pterm.BulletListItem) {
+	l.bb = append(l.bb, item)
+}
+
+func (l *ListWriter) Render() {
+	pterm.DefaultBulletList.WithItems(l.bb).Render()
+}

cmd/util/system.go

@@ -0,0 +1,33 @@
+// Copyright © 2022 Ettore Di Giacinto <mudler@mocaccino.org>
+//
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License along
+// with this program; if not, see <http://www.gnu.org/licenses/>.
+
+package util
+
+import (
+	"path/filepath"
+
+	"github.com/mudler/luet/pkg/api/core/types"
+	pkg "github.com/mudler/luet/pkg/database"
+)
+
+func SystemDB(c *types.LuetConfig) types.PackageDatabase {
+	switch c.System.DatabaseEngine {
+	case "boltdb":
+		return pkg.NewBoltDatabase(
+			filepath.Join(c.System.DatabasePath, "luet.db"))
+	default:
+		return pkg.NewInMemoryDatabase(true)
+	}
+}

cmd/version.go

@@ -0,0 +1,19 @@
+package cmd
+
+import (
+	"fmt"
+
+	"github.com/spf13/cobra"
+)
+
+func init() {
+	RootCmd.AddCommand(versionCmd)
+}
+
+var versionCmd = &cobra.Command{
+	Use:   "version",
+	Short: "Print the version of luet",
+	Run: func(cmd *cobra.Command, args []string) {
+		fmt.Println(Version)
+	},
+}

contrib/config/get_luet_root.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+if [ $(id -u) -ne 0 ]
+  then echo "Please run the installer with sudo/as root"
+  exit
+fi
+
+set -ex
+export LUET_NOLOCK=true
+
+LUET_VERSION=$(curl -s https://api.github.com/repos/mudler/luet/releases/latest | grep tag_name | awk '{ print $2 }' | sed -e 's/\"//g' -e 's/,//g' || echo "0.9.24" )
+LUET_ROOTFS=${LUET_ROOTFS:-/}
+LUET_DATABASE_PATH=${LUET_DATABASE_PATH:-/var/luet/db}
+LUET_DATABASE_ENGINE=${LUET_DATABASE_ENGINE:-boltdb}
+LUET_CONFIG_PROTECT=${LUET_CONFIG_PROTECT:-1}
+
+curl -L https://github.com/mudler/luet/releases/download/${LUET_VERSION}/luet-${LUET_VERSION}-linux-amd64 --output luet
+chmod +x luet
+
+mkdir -p /etc/luet/repos.conf.d || true
+mkdir -p $LUET_DATABASE_PATH || true
+mkdir -p /var/tmp/luet || true
+
+if [ "${LUET_CONFIG_PROTECT}" = "1" ] ; then
+  mkdir -p /etc/luet/config.protect.d || true
+  curl -L https://raw.githubusercontent.com/mudler/luet/master/contrib/config/config.protect.d/01_etc.yml.example --output /etc/luet/config.protect.d/01_etc.yml
+fi
+curl -L https://raw.githubusercontent.com/mocaccinoOS/repository-index/master/packages/mocaccino-repository-index.yml --output /etc/luet/repos.conf.d/mocaccino-repository-index.yml
+
+cat > /etc/luet/luet.yaml <<EOF
+general:
+  debug: false
+system:
+  rootfs: ${LUET_ROOTFS}
+  database_path: "${LUET_DATABASE_PATH}"
+  database_engine: "${LUET_DATABASE_ENGINE}"
+  tmpdir_base: "/var/tmp/luet"
+EOF
+
+./luet install -y repository/luet repository/mocaccino-repository-index
+./luet install -y system/luet system/luet-extensions
+
+rm -rf luet
+

contrib/config/luet.yaml

@@ -69,6 +69,7 @@
 #   Default $TMPDIR/tmpluet
 #   tmpdir_base: "/tmp/tmpluet"
 #
+#
 # ---------------------------------------------
 # Repositories configurations directories.
 # ---------------------------------------------
@@ -93,6 +94,19 @@
 # annotation.
 # config_protect_skip: false
 #
+# The paths used for load repositories and config
+# protects are based on host rootfs.
+# If set to false rootfs path is used as prefix.
+# config_from_host: true
+#
+#
+# ------------------------------------------------
+# Finalizer Environment Variables
+# -----------------------------------------------
+# finalizer_envs:
+#  - key: "BUILD_ISO"
+#    value: "1"
+#
 # System repositories
 # ---------------------------------------------
 # In alternative to define repositories files

docs/.github/workflows/pages.yml

@@ -0,0 +1,22 @@
+name: Github Pages
+on:
+  push:
+    branches: [ master ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        submodules: 'recursive'
+    - name: Build 🔧
+      run: |
+           make build
+    - name: Deploy 🚀
+      if: github.ref == 'refs/heads/master' && github.event_name == 'push'
+      uses: JamesIves/github-pages-deploy-action@releases/v3
+      with:
+        ACCESS_TOKEN: ${{ secrets.ACCESS_TOKEN }}
+        BRANCH: gh-pages
+        FOLDER: public

docs/.gitignore

@@ -0,0 +1,6 @@
+bin/
+public/
+resources/
+node_modules/
+tech-doc-hugo
+

docs/.gitmodules

@@ -0,0 +1,4 @@
+
+[submodule "themes/docsy"]
+	path = themes/docsy
+	url = https://github.com/google/docsy

docs/CNAME

@@ -0,0 +1 @@
+luet.io

docs/Makefile

@@ -0,0 +1,18 @@
+export HUGO_VERSION?=0.91.2
+export HUGO_PLATFORM?=Linux-64bit
+
+export ROOT_DIR:=$(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
+
+.DEFAULT_GOAL := build
+
+.PHONY: build
+build:
+	scripts/build.sh
+
+.PHONY: serve
+serve:
+	scripts/serve.sh
+
+.PHONY: publish
+publish:
+	scripts/publish.sh

docs/assets/scss/_variables_project.scss

@@ -0,0 +1,62 @@
+/*
+
+Add styles or override variables from the theme here.
+
+*/
+
+$enable-gradients: true;
+$enable-rounded: false;
+$enable-shadows: true;
+$google_font_name: "Alata";
+$google_font_family: "Alata:300,300i,400,400i,700,700i";
+
+
+.td-box--primary .td-arrow-down::before, .td-box--1 .td-arrow-down::before{
+    border-color: #2F3061 transparent transparent transparent !important;
+
+}
+.td-box--primary {
+    background-color: #2F3061 !important;
+}
+
+.td-box--1 {     background-color: #2F3061 !important; }
+.td-box--2 {     background-color: #201A23 !important; }
+
+
+.td-box--dark {
+    background-color: #343434 !important;
+
+}
+
+.td-box--3 {
+    color: #fff !important;
+    background-color: #5F5980 !important;
+
+}
+
+.td-navbar {
+    background: #5F5980 !important;
+}
+
+.navbar-bg-onscroll {
+    background: #5F5980 !important;
+
+}
+.td-box--4 {
+    background-color: #DFDFDF !important;
+
+}
+
+
+
+.btn-primary {
+    background: #5F5980 !important;
+    border-color:  #5F5980 !important;
+
+}
+
+.btn-secondary {
+    background: #657DC2 !important;
+    border-color:  #657DC2 !important;
+
+}

docs/config.toml

@@ -0,0 +1,165 @@
+baseURL = "https://www.luet.io"
+title = "Luet"
+
+enableRobotsTXT = true
+
+# Hugo allows theme composition (and inheritance). The precedence is from left to right.
+theme = ["docsy"]
+
+# Will give values to .Lastmod etc.
+enableGitInfo = true
+
+# Language settings
+contentDir = "content/en"
+defaultContentLanguage = "en"
+defaultContentLanguageInSubdir = false
+# Useful when translating.
+enableMissingTranslationPlaceholders = true
+
+disableKinds = ["taxonomy", "taxonomyTerm"]
+
+# Highlighting config
+pygmentsCodeFences = true
+pygmentsUseClasses = false
+# Use the new Chroma Go highlighter in Hugo.
+pygmentsUseClassic = false
+#pygmentsOptions = "linenos=table"
+# See https://help.farbox.com/pygments.html
+pygmentsStyle = "tango"
+
+# Configure how URLs look like per section.
+[permalinks]
+blog = "/:section/:year/:month/:day/:slug/"
+
+## Configuration for BlackFriday markdown parser: https://github.com/russross/blackfriday
+[blackfriday]
+plainIDAnchors = true
+hrefTargetBlank = true
+angledQuotes = false
+latexDashes = true
+
+# Image processing configuration.
+[imaging]
+resampleFilter = "CatmullRom"
+quality = 75
+anchor = "smart"
+
+[services]
+[services.googleAnalytics]
+# Comment out the next line to disable GA tracking. Also disables the feature described in [params.ui.feedback].
+id = "UA-00000000-0"
+
+# Language configuration
+
+[languages]
+[languages.en]
+title = "Luet"
+description = "Package manager built from containers"
+languageName ="English"
+# Weight used for sorting.
+weight = 1
+#[languages.it]
+#title = "Luet"
+#description = "Gestore di pacchetti basato su containers"
+#languageName ="Italian"
+#contentDir = "content/it"
+#time_format_default = "02.01.2006"
+#time_format_blog = "02.01.2006"
+[[menu.main]]
+    name = "Contribution guidelines"
+    weight = 50
+    url = "https://github.com/mudler/luet/contribute"
+    pre = "<i class='fab fa-github'></i>"
+    post = "" 
+[markup]
+  [markup.goldmark]
+    [markup.goldmark.renderer]
+      unsafe = true
+
+# Everything below this are Site Params
+
+[params]
+copyright = "Ettore Di Giacinto"
+privacy_policy = "https://policies.google.com/privacy"
+
+# First one is picked as the Twitter card image if not set on page.
+# images = ["images/project-illustration.png"]
+
+# Menu title if your navbar has a versions selector to access old versions of your site.
+# This menu appears only if you have at least one [params.versions] set.
+version_menu = "Releases"
+
+# Repository configuration (URLs for in-page links to opening issues and suggesting changes)
+github_repo = "https://github.com/mudler/luet"
+# An optional link to a related project repo. For example, the sibling repository where your product code lives.
+github_project_repo = "https://github.com/mudler/luet"
+
+# Specify a value here if your content directory is not in your repo's root directory
+github_subdir = "docs"
+
+# Google Custom Search Engine ID. Remove or comment out to disable search.
+#gcs_engine_id = "011737558837375720776:fsdu1nryfng"
+
+# Enable Algolia DocSearch
+algolia_docsearch = false
+
+# Enable Lunr.js offline search
+offlineSearch = true
+
+# User interface configuration
+[params.ui]
+# Enable to show the side bar menu in its compact state.
+sidebar_menu_compact = false
+#  Set to true to disable breadcrumb navigation.
+breadcrumb_disable = true
+#  Set to true to hide the sidebar search box (the top nav search box will still be displayed if search is enabled)
+sidebar_search_disable = false
+#  Set to false if you don't want to display a logo (/assets/icons/logo.svg) in the top nav bar
+navbar_logo = true
+# Set to true to disable the About link in the site footer
+footer_about_disable = false
+
+# Adds a H2 section titled "Feedback" to the bottom of each doc. The responses are sent to Google Analytics as events.
+# This feature depends on [services.googleAnalytics] and will be disabled if "services.googleAnalytics.id" is not set.
+# If you want this feature, but occasionally need to remove the "Feedback" section from a single page,
+# add "hide_feedback: true" to the page's front matter.
+[params.ui.feedback]
+enable = true
+# The responses that the user sees after clicking "yes" (the page was helpful) or "no" (the page was not helpful).
+yes = 'Glad to hear it! Please <a href="https://github.com/Luet-lab/docs/issues/new">tell us how we can improve</a>.'
+no = 'Sorry to hear that. Please <a href="https://github.com/Luet-lab/docs/issues/new">tell us how we can improve</a>.'
+
+[params.links]
+# End user relevant links. These will show up on left side of footer and in the community page if you have one.
+# [[params.links.user]]
+# 	name = "User mailing list"
+# 	url = "https://example.org/mail"
+# 	icon = "fa fa-envelope"
+#         desc = "Discussion and help from your fellow users"
+# [[params.links.user]]
+# 	name ="Twitter"
+# 	url = "https://example.org/twitter"
+# 	icon = "fab fa-twitter"
+#         desc = "Follow us on Twitter to get the latest news!"
+# [[params.links.user]]
+# 	name = "Stack Overflow"
+# 	url = "https://example.org/stack"
+# 	icon = "fab fa-stack-overflow"
+#         desc = "Practical questions and curated answers"
+# Developer relevant links. These will show up on right side of footer and in the community page if you have one.
+[[params.links.developer]]
+	name = "GitHub"
+	url = "https://github.com/mudler/luet"
+	icon = "fab fa-github"
+        desc = "Development takes place here!"
+[[params.links.developer]]
+	name = "Slack"
+	url = "https://join.slack.com/t/luet/shared_invite/enQtOTQxMjcyNDQ0MDUxLWQ5ODVlNTI1MTYzNDRkYzkyYmM1YWE5YjM0NTliNDEzNmQwMTkxNDRhNDIzM2Y5NDBlOTZjZTYxYWQyNDE4YzY"
+	icon = "fab fa-slack"
+        desc = "Join us on Slack!"
+# [[params.links.developer]]
+# 	name = "Developer mailing list"
+# 	url = "https://example.org/mail"
+# 	icon = "fa fa-envelope"
+#         desc = "Discuss development issues around the project"
+

docs/content/en/_index.html

@@ -0,0 +1,81 @@
++++
+title = "Luet docs"
+linkTitle = "luetdocs"
+
++++
+
+{{< blocks/cover title="Luet" image_anchor="top" height="full" color="orange" >}}
+<div class="mx-auto">
+	<a class="btn btn-lg btn-primary mr-3 mb-4" href="{{< relref "/docs" >}}">
+		Documentation <i class="fas fa-arrow-alt-circle-right ml-2"></i>
+	</a>
+	<a class="btn btn-lg btn-secondary mr-3 mb-4" href="https://github.com/mudler/luet/releases">
+		Download <i class="fab fa-github ml-2 "></i>
+	</a>
+	<p class="lead mt-5">Container Package Manager</p>
+	<div class="mx-auto mt-5">
+		{{< blocks/link-down color="info" >}}
+	</div>
+</div>
+{{< /blocks/cover >}}
+
+
+{{% blocks/lead color="primary" %}}
+
+Luet uses Container runtimes to build packages in a reproducible manner.
+It provides an abstraction over the Dockerfile format introducing relation and versioning of images.
+
+{{% /blocks/lead %}}
+
+{{< blocks/section color="dark" >}}
+{{% blocks/feature icon="fa-tree" title="SAT Solver" %}}
+Luet uses SAT Solving techniques to compute the dependencies graph.
+This allows to refer to docker images by using semver constraints.
+
+No Relational db is involved.
+{{% /blocks/feature %}}
+
+
+{{% blocks/feature icon="fab fa-github" title="Contributions welcome!" url="https://github.com/mudler/luet" %}}
+If you like to play with code, check out our issues that are marked as ["good first issue"](https://github.com/mudler/luet/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) and open a [Pull Request](https://github.com/mudler/luet/pulls) on **GitHub**. 
+New users are always welcome, and have fun!
+{{% /blocks/feature %}}
+
+
+{{% blocks/feature icon="fa-terminal" title="Container-based, reproducible builds" %}}
+Use container abstraction to define package repositories. Intermediate build images are pushed along to guarantee reproducible builds.
+{{% /blocks/feature %}}
+
+{{< /blocks/section >}}
+
+
+{{< blocks/section >}}
+<div class="col">
+<h1 class="text-center">Releasing is not anymore a nightmare</h1>
+<center>You can carefully pick now dependencies to make your release - release composed of container images or either of a set of packages
+</center>
+</div>
+
+{{< /blocks/section >}}
+
+
+{{< blocks/section >}}
+{{% blocks/feature icon="fa-truck" title="Package Management" %}}
+Build, ship and delivery your software. Faster
+{{% /blocks/feature %}}
+
+
+{{% blocks/feature icon="fa-heartbeat" title="0 dep installer" %}}
+When Luet is used as installer, it has zero dependencies. 
+
+Your system can't break anymore
+{{% /blocks/feature %}}
+
+
+{{% blocks/feature icon="fa-magic" title="Reconstruct images" %}}
+Thanks to its SAT core, Luet can reconstruct images defined by dependencies and version constraints. 
+
+Building a de-facto tree of container images
+{{% /blocks/feature %}}
+
+{{< /blocks/section >}}

docs/content/en/about/_index.html

@@ -0,0 +1,41 @@
+---
+title: About Luet
+linkTitle: About
+menu:
+  main:
+    weight: 10
+
+---
+
+
+{{< blocks/cover title="About Luet" image_anchor="bottom" height="min" >}}
+
+<p class="lead mt-5">Luet is a Package Manager which uses Containers technologies.
+</p>
+
+{{< /blocks/cover >}}
+
+{{% blocks/lead %}}
+Luet uses Container technologies ( Docker, img ) to build packages.
+It provides an abstraction over the Dockerfile format introducing relation and versioning of images.
+{{% /blocks/lead %}}
+
+
+{{< blocks/section >}}
+<div class="col-12">
+<h1 class="text-center">Zero-deps installer</h1>
+<center>The installer can run in "from scratch" environment - your system will be always recoverable - everything which was built from containers can be installed locally.
+</center>
+</div>
+
+{{< /blocks/section >}}
+
+
+
+{{< blocks/section >}}
+
+<div class="col-12">
+<h1 class="text-center">In few commands it allows you to switch between multiple Linux Distributions, in runtime!</h1>
+</div>
+
+{{< /blocks/section >}}

docs/content/en/blog/_index.md

@@ -0,0 +1,13 @@
+---
+title: "Luet Blog"
+linkTitle: "Blog"
+menu:
+  main:
+    weight: 30
+---
+
+
+This is the **blog** section. It has two categories: News and Releases.
+
+Files in these directories will be listed in reverse chronological order.
+

docs/content/en/blog/news/_index.md

@@ -0,0 +1,8 @@
+
+---
+title: "News About Luet"
+linkTitle: "News"
+weight: 20
+---
+
+

docs/content/en/blog/news/website.md

@@ -0,0 +1,10 @@
+
+---
+title: "Website is up"
+linkTitle: "Website is up"
+date: 2019-12-23
+description: >
+  Website is up
+---
+
+Finally the website is up! Docs are a work in progress. Stay tuned for more upcoming updates

docs/content/en/blog/releases/0.3.md

@@ -0,0 +1,14 @@
+
+---
+title: "0.3 Release"
+linkTitle: "0.3"
+date: 2019-12-23
+description: >
+  X-Mas release!
+---
+
+This release comes with a lot of bugfixes and enhancement to the SAT solver core:
+
+- Add support for `provides`. They allow to have `virtual` packages which can be replaced during solving by other drop-in packages.
+- Tons of fixes
+- Preparation for upcoming compression support

docs/content/en/blog/releases/_index.md

@@ -0,0 +1,8 @@
+
+---
+title: "New Releases"
+linkTitle: "Releases"
+weight: 20
+---
+
+

docs/content/en/community/_index.md

@@ -0,0 +1,8 @@
+---
+title: Community
+menu:
+  main:
+    weight: 40
+---
+
+<!--add blocks of content here to add more sections to the community page -->

docs/content/en/docs/Concepts/Overview/_index.md

@@ -0,0 +1,33 @@
+---
+title: "Overview"
+linkTitle: "Overview"
+weight: 1
+description: >
+  See Luet in action.
+---
+
+
+Luet provides an abstraction layer on top of the container image layer to make the package a first class construct. A package definition and all its dependencies are translated by Luet to Dockerfiles which can then be built anywhere that docker runs.
+
+Luet is written entirely in Go and comes as a single static binary. This has a few advantages:
+
+- Easy to recover. You can use luet to bootstrap the system entirely from the ground-up.
+- Package manager has no dependencies on the packages that it installs. There is no chance of breaking the package manager by installing a conflicting package, or uninstalling one.
+- Portable - it can run on any architecture
+
+Luet brings the containers ecosystem to standard software package management and delivery. It is fully built around the container concept, and leverages the huge catalog already present in the wild. It lets you use container images from [Docker Hub](https://hub.docker.com/), or from private registries to build packages, and helps you to redistribute them.
+
+Systems that are using luet as a package manager can consume Luet repositories with only luet itself. No dependency is required by the Package manager, giving you the full control on what you install or not in the system. It can be used to generate *Linux from Scratch* distributions,  also to build Docker images, or to simply build standalone packages that you might want to redistribute.
+
+The syntax proposed aims to be [KISS](https://en.wikipedia.org/wiki/KISS_principle) - you define a set of steps that need to be run to build your image, and a set of constraints denoting the requirements or conflicts of your package.
+
+## Why another Package manager?
+
+There is no known package manager with 0-dependency that fully leverages the container ecosystem. This gap forces current package managers to depend on a specific system layout as base of the building process and the corresponding depencies. This can cause situations leading to a broken system. We want to fix that by empowering the user, by building their own packages, and redistribute them. 
+Luet allows also to create packages entirely from Docker images content. In this way the user can actually bundle all the files of an image into a package and deliver part of it, or entirely as a layer. All of that, without the package manager depending on a single bit from it.
+
+## Package definitions
+
+Luet uses [YAML](https://en.wikipedia.org/wiki/YAML) for the package specification format, Luet parses the [requirements](/docs/concepts/overview/constraints) to build [packages](/docs/concepts/packages), so Luet can consume them.
+
+Below you can find links to tutorials on how to build packages, images and repositories.

docs/content/en/docs/Concepts/Overview/build_packages.md

@@ -0,0 +1,222 @@
+---
+title: "Building packages"
+linkTitle: "Building packages"
+weight: 1
+date: 2017-01-05
+description: >
+  How to build packages with Luet
+---
+
+
+## Prerequisistes
+
+Luet currently supports [Docker](https://www.docker.com/) and [Img](https://github.com/genuinetools/img) as backends to build packages. Both of them can be used and switched in runtime with the ```--backend``` option, so either one of them must be present in the host system.
+
+### Docker
+
+Docker is the (less) experimental Luet engine supported. Be sure to have Docker installed and the daemon running. The user running `luet` commands needs the corresponding permissions to run the `docker` executable, and to connect to a `docker` daemon. The only feature needed by the daemon is the ability to build images, so it fully supports remote daemon as well (this can be specified with the `DOCKER_HOST` environment variable, that is respected by `luet`)
+
+### Img
+
+Luet supports [Img](https://github.com/genuinetools/img). To use it, simply install it in your system, and while running `luet build`, you can switch the backend by providing it as a parameter: `luet build --backend img`. For small packages it is particularly powerful, as it doesn't require any docker daemon running in the host.
+
+### Building packages on Kubernetes
+
+Luet and img can be used together to orchestrate package builds also on kubernetes. There is available an experimental [Kubernetes CRD for Luet](https://github.com/mudler/luet-k8s) which allows to build packages seamelessly in Kubernetes and push package artifacts to an S3 Compatible object storage (e.g. Minio).
+
+## Building packages
+
+![Build packages](/docs/tree.jpg)
+
+Luet provides an abstraction layer on top of the container image layer to make the package a first class construct. A package definition and all its dependencies are translated by Luet to Dockerfiles which can then be built anywhere that docker runs.
+
+To resolve the dependency tree Luet uses a SAT solver and no database. It is responsible for calculating the dependencies of a package and to prevent conflicts. The Luet core is still young, but it has a comprehensive test suite that we use to validate any future changes.
+
+Building a package with Luet requires only a [definition](/docs/concepts/packages/specfile). This definition can be self-contained and be only composed of one [specfile](/docs/concepts/packages/specfile), or a group of them, forming a Luet tree. For more complex use-cases, see [collections](/docs/concepts/packages/collections). Luet also supports building packages from standard `Dockerfile` directly.
+
+Run `luet build --help` to get more help for each parameter.
+
+Build accepts a list of packages to build, which syntax is in the `category/name-version` notation. See also [specfile documentation page](/docs/concepts/packages/specfile/#refering-to-packages-from-the-cli) to see how to express packages from the CLI.
+
+## Reproducible builds
+
+Pinning a container build is not easy - there are always so many moving pieces, and sometimes just set `FROM` an image tag might not be enough.
+
+Luet while building a package generates intermediate images that are stored and can be optionally pushed in a registry. Those images can be re-used by Luet if building again the same tree to guarantuee highly reproducible builds.
+
+## Environmental variables
+
+Luet builds passes its environment variable at the engine which is called during build, so for example the environment variable `DOCKER_HOST` or `DOCKER_BUILDKIT` can be setted.
+
+Every argument from the CLI can be setted via environment variable too with a `LUET_` prefix, for instance the flag `--clean`, can be setted via environment with `LUET_CLEAN`, `--privileged` can be enabled with `LUET_PRIVILEGED` and so on.
+
+## Supported compression format
+
+At the moment, `luet` can compress packages and tree with `zstd` and `gzip`. For example: 
+
+```bash
+luet build --compression zstd ...
+```
+
+Will output package compressed in the zstd format.
+
+See the `--help` of `create-repo` and `build` to learn all the available options.
+
+## Example
+
+
+Luet can seamlessly build packages also from Dockerfiles (_since luet>=0.32.0_), consider the following example, that will generate a `curl` package from an `alpine` image:
+
+```bash
+$> # put yourself in some workdir
+
+$~/workdir> mkdir curl
+
+$~/workdir> cat <<EOF > curl/Dockerfile
+FROM alpine
+apk add curl
+EOF
+
+$~/workdir> luet build --all
+```
+
+However, `luet` supports an extended syntax that allows to define packages with a more fine-grained control, templating support, and several other features that makes creation batch images much faster.
+
+### The extended syntax
+
+A [package definition](/docs/concepts/packages/specfile) is composed of a `build.yaml` and a sibiling `definition.yaml`.
+
+In the following example, we are creating a dummy package (`bar/foo`). Which ships one file only, `/foo`
+
+```bash
+$> # put yourself in some workdir
+
+$~/workdir> mkdir package
+
+$~/workdir> cat <<EOF > package/build.yaml
+image: busybox
+steps:
+- echo "foo=bar" > /foo
+EOF
+
+$~/workdir> cat <<EOF > package/definition.yaml
+name: "foo"
+version: "0.1"
+category: "bar"
+EOF
+
+```
+
+To build it, simply run `luet build bar/foo` or `luet build --all` to build all the packages in the current directory:
+
+```bash
+$> luet build --all
+
+📦  Selecting  foo 0.1
+📦  Compiling foo version 0.1 .... ☕
+🐋  Downloading image luet/cache-foo-bar-0.1-builder
+🐋  Downloading image luet/cache-foo-bar-0.1
+📦   foo Generating 🐋  definition for builder image from busybox
+🐋  Building image luet/cache-foo-bar-0.1-builder
+🐋  Building image luet/cache-foo-bar-0.1-builder done
+ Sending build context to Docker daemon  4.096kB
+ ...
+
+```
+
+Luet "trees" are just a group of specfiles, in the above example, our tree was the current directory. You can also specify a directory with the `--tree` option. Luet doesn't enforce any tree layout, so they can be nested at any level. The only rule of thumb is that a `build.yaml` file needs to have either a `definition.yaml` or a `collection.yaml` file next to it.
+
+## Nesting dependencies
+
+In the example above we have created a package from a `delta`. Luet by default creates packages by analyzing the differences between the generated containers, and extracts the differences as archive, the resulting files then are compressed and can be consumed later on by `luet install`.
+
+Luet can create packages from different [building strategies](/docs/docs/concepts/packages/specfile/#building-strategies): by delta, by taking a whole container content, or by considering a single directory in the build container.
+
+Besides that, [a package can reference a strict dependency on others](/docs/docs/concepts/packages/specfile/#build-time-dependencies).
+
+### Example
+
+Let's extend the above example with two packages which depends on it during the build phase.
+
+```bash
+
+$~/workdir> mkdir package2
+
+$~/workdir> cat <<EOF > package2/build.yaml
+requires:
+- name: "foo"
+  category: "bar"
+  version: ">=0"
+
+steps:
+- source /foo && echo "$foo" > /bar
+EOF
+
+$~/workdir> cat <<EOF > package2/definition.yaml
+name: "ineedfoo"
+version: "0.1"
+category: "bar"
+EOF
+
+
+$~/workdir> mkdir package3
+
+$~/workdir> cat <<EOF > package3/build.yaml
+requires:
+- name: "foo"
+  category: "bar"
+  version: ">=0"
+- name: "ineedfoo"
+  category: "bar"
+  version: ">=0"
+
+steps:
+- source /foo && echo "$foo" > /ineedboth
+- cat /bar > /bar
+
+EOF
+
+$~/workdir> cat <<EOF > package3/definition.yaml
+name: "ineedfooandbar"
+version: "0.1"
+category: "bar"
+EOF
+
+```
+
+To build, run again: 
+
+```bash
+$> luet build --all
+```
+
+As we can see, now Luet generated 3 packages, `bar/foo`, `bar/ineedfoo` and `bar/ineedfooandbar`. They aren't doing anything special than just shipping text files, this is an illustrative example on how build requirements can be combined to form new packages:
+
+ `bar/ineedfooandbar` depends on both `bar/ineedfoo` and `bar/foo` during build-time, while `bar/foo` uses a docker image as a build base.
+
+See the [package definition documentation page](/docs/docs/concepts/packages/specfile/#building-strategies) for more details on how to instruct the Luet compiler to build packages with different strategies.
+
+## Caching docker images
+
+Luet can push and pull the docker images that are being generated during the build process. A tree is represented by a single docker image, and each package can have one or more tags attached to it.
+
+To push automatically docker images that are built, use the `--push` option, to pull, use the `--pull` option. An image repository can be specified with `--image-repository` flag, and can include also the remote registries where the images are pushed to. 
+
+Luet doesn't handle login to registries, so that has to be handled separately with `docker login` or `img login` before the build process starts.
+
+### Build faster
+
+When packages are cached, for iterating locally it's particularly useful to jump straight to the image that you want to build. You can use ```--only-target-package``` to jump directly to the image you are interested in. Luet will take care of checking if the images are present in the remote registry, and would build them if any of those are missing.
+
+## Building for a different platform
+
+Sometimes you need to build a package for a different platform than the one running on your host machine. For example, you may want to build an arm64 package, but your machine is x86. To do this, all you need to do is pass the following arguments:
+
+```
+luet --backend-args --load --backend-args --platform --backend-args linux/arm64 build PACKAGE_NAME
+```
+
+## Notes
+
+- All the files which are next to a `build.yaml` are copied in the container which is running your build, so they are always accessible during build time.
+- If you notice errors about disk space, mind to set the `TMPDIR` env variable to a different folder. By default luet respects the O.S. default (which in the majority of system is `/tmp`).

docs/content/en/docs/Concepts/Overview/configuration.md

@@ -0,0 +1,177 @@
+title: "Configuration"
+linkTitle: "Configuration
+weight: 2
+description: >
+    Configuring Luet
+---
+
+### General
+
+```yaml
+general:
+  # Define max concurrency processes. Default is based of arch: runtime.NumCPU()
+  concurrency: 1
+  # Enable Debug. If debug is active spinner is disabled.
+  debug: false
+  # Show output of build execution (docker, img, etc.)
+  show_build_output: false
+  # Define spinner ms
+  spinner_ms: 200
+  # Define spinner charset. See https://github.com/briandowns/spinner
+  spinner_charset: 22
+  # Enable warnings to exit
+  fatal_warnings: false
+  # Try extracting tree/packages with the same ownership as exists in the archive (default for superuser).
+  same_owner: false
+```
+
+### Images
+
+After the building of the packages, you can apply arbitrary images on top using the `images` stanza. This is useful if you need to pin a package to a specific version.
+
+```yaml
+images:
+  - quay.io/kairos/packages:kairos-agent-system-2.1.12
+```
+
+### Logging
+
+```yaml
+logging:
+  # Enable loggging to file (if path is not empty)
+  enable_logfile: false
+  #  Leave empty to skip logging to file.
+  path: "/var/log/luet.log"
+  # Set logging level: error|warning|info|debug
+  level: "info"
+  #  Enable JSON log format instead of console mode.
+  json_format: false.
+  #  Disable/Enable color
+  color: true
+  #  Enable/Disable emoji
+  enable_emoji: true
+```
+
+### Repositories configurations directories.
+
+```yaml
+# Define the list of directories where luet
+# try for files with .yml extension that define
+# luet repository.
+repos_confdir:
+  - /etc/luet/repos.conf.d
+```
+
+### Finalizer Environment Variables
+
+```yaml
+finalizer_envs:
+ - key: "BUILD_ISO"
+   value: "1"
+```
+
+### Repositories
+
+To add repositories, you can either add a `repositories` stanza in your `/etc/luet/luet.yaml` or either add one or more yaml files in `/etc/luet/repos.conf.d/`.
+
+#### Configuring repositories in the main configuration file
+
+```yaml
+logging:
+  color: true # Enable/Disable colored output
+  enable_emoji: true # Enable/Disable emoji from output
+general:
+  debug: false # Enable/Disable debug
+system:
+  rootfs: "/" # What's our rootfs. Luet can install packages outside of "/"
+  database_path: "/var/db/luet" # Where to store DB files
+  database_engine: "boltdb"
+  tmpdir_base: "/var/tmp/luet" # The temporary directory to be used
+repositories:
+- name: "some-repository-name" # Repository name
+  description: "A beautiful description"
+  type: "http" # Repository type, disk or http are supported (disk for local path)
+  enable: true # Enable/Disable repo
+  cached: true # Enable cache for repository
+  priority: 3 # Cache priority
+  urls: # Repository URLs
+    - "...."
+```
+
+#### Using different files to configure repositories
+
+In the main configuration file you can specify the directory where all repositories are configured:
+
+```yaml
+repos_confdir:
+  - /etc/luet/repos.conf.d
+```
+ 
+Then add a file inside `/etc/luet/repos.conf.d/example.yaml` with your configuration, e.g.:
+
+```yaml
+name: "..." # Repository name
+description: "..."
+type: "http" # Repository type, disk or http are supported (disk for local path)
+enable: true # Enable/Disable repo
+cached: true # Enable cache for repository
+priority: 3 # Cache priority
+urls: # Repository URLs
+  - "..."
+```
+
+There is available a [collection of repositories](https://packages.mocaccino.org/repository-index), which is containing a list of repositories that can be installed in the system with `luet install`.
+
+If you installed Luet from the curl command, you just need to run `luet search repository` to see a list of all the available repository, and you can install them singularly by running `luet install repository/<name>`. Otherwise, add the repository stanzas you need to `/etc/luet/luet.yaml`.
+
+#### Config protect configuration files directories.
+
+```yaml
+# Define the list of directories where load
+# configuration files with the list of config
+# protect paths.
+config_protect_confdir:
+  - /etc/luet/config.protect.d
+# Ignore rules defined on
+# config protect confdir and packages
+# annotation.
+config_protect_skip: false
+# The paths used for load repositories and config
+# protects are based on host rootfs.
+# If set to false rootfs path is used as prefix.
+config_from_host: true
+```
+
+### Solver Parameter Configuration
+
+```yaml
+solver:
+  # Solver strategy to solve possible conflicts during depedency
+  # solving. Defaults to empty (none). Available: qlearning
+  type: ""
+  # Solver agent learning rate. 0.1 to 1.0
+  rate: 0.7
+  # Learning discount factor.
+  discount: 1.0
+  # Number of overall attempts that the solver has available before bailing out.
+  max_attempts: 9000
+```
+
+### System
+
+```yaml
+system:
+  # Rootfs path of the luet system. Default is /.
+  # A specific path could be used for test installation to
+  # a chroot environment.
+  rootfs: "/"
+  # Database engine used for luet database.
+  # Supported values: boltdb|memory
+  database_engine: boltdb
+  # Database path directory where store luet database.
+  # The path is appended to rootfs option path.
+  database_path: "/var/cache/luet"
+  # Define the tmpdir base directory where luet store temporary files.
+  # Default $TMPDIR/tmpluet
+  tmpdir_base: "/tmp/tmpluet"
+```

docs/content/en/docs/Concepts/Overview/constraints.md

@@ -0,0 +1,34 @@
+---
+title: "CSP, SAT && RL"
+linkTitle: "CSP, SAT && RL"
+weight: 4
+description: >
+  How Luet turns Image resolution into CSP
+---
+
+Under the hood, Luet uses boolean satisfiability problem ([SAT](https://en.wikipedia.org/wiki/Boolean_satisfiability_problem))  [reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning) techniques to solve package constraints.
+
+Luet allows you to specify 3 types of set of contraints on a [package](/docs/concepts/packages/) definition:
+
+- Requires
+- Conflicts
+- Provides
+
+The package definition in your tree definition, along with its Requires and Conflicts, are turned into Boolean formulas that are consumed by the solver to compute a solution. The solution represent the state of your system after a particular query is asked to the solver (Install, Uninstall, Upgrade).
+
+## Requires and Conflicts
+
+A list of requires and conflicts, composed of one or more [packages](/docs/concepts/packages/), becomes a SAT formula. The formula is then given to the SAT solver to compute a finite state set of packages which must be installed in the system in order to met the requirements.
+
+As Luet allows to express constraints with selectors ( e.g. `A depends on >=B-1.0`) it generates additional constraints to guarantee that at least one package and at most one is picked as dependency (*ALO* and *AMO*).
+
+## Provides
+
+Provides constraints are not encoded in a SAT formula. Instead, they are `expanded` into an in-place substitution of the packages that they have to be replaced with.
+They share the same SAT logic of expansion, allowing to swap entire version ranges (e.g. `>=1.0`), allowing to handle package rename, removals, and virtuals.
+
+## References
+
+- OPIUM (Luet is inspired by it): https://ranjitjhala.github.io/static/opium.pdf
+- FROM TRACTABLE CSP TO TRACTABLE SAT: https://www.cs.ox.ac.uk/files/4014/maxclosed_orderencoding_v16_TR.pdf
+- Solver concepts applied to packages (`zypper`): https://en.opensuse.org/openSUSE:Libzypp_satsolver_basics

docs/content/en/docs/Concepts/Overview/repositories.md

@@ -0,0 +1,148 @@
+---
+title: "Creating Luet repositories"
+linkTitle: "Creating Luet repositories"
+weight: 2
+date: 2017-01-05
+description: >
+  How to create Luet repositories
+---
+
+After a set of packages has been built, a repository must be created in order to make them accessible by Luet clients. A Repository can be served either local files or via http(s) (at the moment of writing). Luet, by default, supports multiple-repositories with priorities.
+
+
+## Repository fields
+
+```yaml
+name: "..."
+description: "..."
+type: "docker"
+cached: true
+enabled: true
+arch: "amd64"
+priority: 3
+urls:
+  - "..."
+```
+
+Repositories have the following fields, notably:
+
+- `name`: Repository name
+- `description`: Repository description
+- `cached`: Enable/disable repository cache
+- `enable`: Enable/disables the repository
+- `urls`: A List of urls where the repository is hosted from
+- `type`: Repository type ( `docker`, `disk`, `http` are currently supported )
+- `arch`:  (optional) Denotes the arch repository. If present, it will enable the repository automatically if the corresponding arch is matching with the host running `luet`. `enable: true` would override this behavior
+- `reference`: (optional) A reference to a repository index file to use to retrieve the repository metadata instead of latest. This can be used to point to a different or an older repository index to act as a "wayback machine". The client will consume the repository state from that snapshot instead of latest.
+  
+{{% alert title="Note" %}}
+The `reference` field has to be a valid tag. For example, if a repository is a docker type, browse the image tags. The repository index snapshots are prefixed with a timestamp, and ending in `repository.yaml`. For example ` 20211027153653-repository.yaml`
+{{% /alert %}}
+
+## Create a repository
+
+After issuing a `luet build`, the built packages are present in the output build directory. The `create-repo` step is needed to generate a portable tree, which is read by the clients, and a `repository.yaml` which contains the repository metadata.
+
+Note that the output of `create-repo` is *additive* so it integrates with the current build content. The repository is composed by the packages generated by the `build` command (or `pack`) and the `create-repo` generated metadata. 
+
+### Flags
+
+Some of the relevant flags for `create-repo` are:
+
+- **--descr**: Repository description
+- **--name**: Repository name
+- **--output**: Metadata output folder (while a different path can be specified, it's prefered to output the metadata files directly to the package directory).This most of the time matches the packages path for convenience.
+- **--packages**: Directory where built packages are stored. This most of the time is also the output path.
+- **--reset-revision**: Reset the repository revision number
+- **--tree-path**: Specify a custom name for the tree path. (Defaults to tree.tar)
+- **--tree-compression**: Specify a compression algorithm for the tree. (Available: gzip, Defaults: none)
+- **--tree**: Path of the tree which was used to generate the packages and holds package metadatas
+- **--type**: Repository type (http/local). It is just descriptive, the clients will be able to consume the repo in whatsoever way it is served.
+- **--urls**: List of URIS where the repository is available
+
+See `luet create-repo --help` for a full description.
+
+## Example
+
+Build a package and generate the repository metadata:
+
+```
+$> mkdir package
+
+$> cat <<EOF > package/build.yaml
+image: busybox
+steps:
+- echo "foo" > /foo
+EOF
+
+$> cat <<EOF > package/definition.yaml
+name: "foo"
+version: "0.1"
+category: "bar" # optional!
+EOF
+
+$> luet build --all --destination $PWD/out/ --tree $PWD/package
+
+📦  Selecting  foo 0.1
+📦  Compiling foo version 0.1 .... ☕
+🐋  Downloading image luet/cache-foo-bar-0.1-builder
+🐋  Downloading image luet/cache-foo-bar-0.1
+📦   foo Generating 🐋  definition for builder image from busybox
+🐋  Building image luet/cache-foo-bar-0.1-builder
+🐋  Building image luet/cache-foo-bar-0.1-builder done
+ Sending build context to Docker daemon  4.096kB
+ ...
+
+$> luet create-repo --name "test" --output $PWD/out --packages $PWD/out --tree $PWD/package
+ For repository test creating revision 1 and last update 1580641614...
+
+$> ls out
+foo-bar-0.1-builder.image.tar  foo-bar-0.1.image.tar  foo-bar-0.1.metadata.yaml  foo-bar-0.1.package.tar  repository.yaml  tree.tar
+
+```
+
+### Repositories type
+
+There are 3 types of repositories supported by luet: `disk`, `http`, `docker`.
+
+#### `disk`
+
+It is a repository which is merely a local folder in your system. When creating a repository and specifying ```--output```, `luet` expects a local path to the system where to store the generated metadata.
+
+#### `http`
+
+It is a repository type which is hosted behind a webserver. When creating a repository and specifying ```--output```, `luet` expects a local path to the system where to store the generated metadata, similarly to the `disk` repository type. Luet is not handling any file upload. The `http` repository type gains meaning when being used from the client, where the repository source must be specified
+
+#### `docker`
+
+When specifying the `docker` repository type, `luet` will generate final images from the build results and upload them to the docker reference specified with ```--output```. The images contains the artifact output from the build result, and they are tagged accordingly to their package name. A single image reference needs to be passed, all the packages will be pushed in a single image but with different tags.
+
+The login to the container registry is not handled, the daemon needs to have already proper permissions to push the image to the destination.
+
+## Repositories snapshots
+
+Luet automatically will create repository index snapshots. This allows clients to point to specific references of repositories besides the latest package set published.
+
+`luet create-repo` optionally takes a `--snapshot-id` argument to define the snapshot name, otherwise it defaults to the unix date timestamp.
+
+Combined with `--push-images` with a container repository type, it automatically tags and pushes snapshots images too.
+
+### Consuming repository snapshots
+
+A client can define a repository, with an optional `reference` keyword:
+
+```yaml
+name: "..."
+description: "..."
+type: "docker"
+priority: 3
+reference: 20220204175357-repository.yaml
+urls:
+  - "..."
+```
+
+
+## Notes
+
+- The tree of definition being used to build the repository, and the package directories must **not** be symlinks.
+- To build a repository is not required to hold the packages artifacts, only the respective `metadata.yaml` file is required.

docs/content/en/docs/Concepts/Overview/usage.md

@@ -0,0 +1,156 @@
+---
+title: "CLI usage"
+linkTitle: "CLI usage"
+weight: 3
+date: 2019-12-14
+description: >
+  How to install packages, manage repositories, ...
+---
+
+
+## Installing a package
+
+To install a package with `luet`, simply run:
+
+```bash
+
+$ luet install <package_name>
+
+```
+
+To relax dependency constraints and avoid auto-upgrades, add the `--relax` flag:
+
+```bash
+$ luet install --relax <package name>
+```
+
+To install only the package without considering the deps, add the `--nodeps` flag:
+
+```bash
+$ luet install --nodeps <package name>
+```
+
+To install only package dependencies, add the `--onlydeps` flag:
+
+```bash
+$ luet install --onlydeps <package name>
+```
+
+To only download packages, without installing them use the `--download-only` flag:
+
+```bash
+$ luet install --download-only <package name>
+```
+
+## Uninstalling a package
+
+To uninstall a package with `luet`, simply run:
+
+```bash
+
+$ luet uninstall <package_name>
+
+```
+
+## Upgrading the system
+
+To upgrade your system, simply run:
+
+```bash
+$ luet upgrade
+```
+
+## Refreshing repositories
+
+Luet automatically syncs repositories definition on the machine when necessary, but it avoids to sync up in a 24h range. In order to refresh the repositories manually, run:
+
+```bash
+$ luet repo update
+```
+
+## Searching a package
+
+To search a package:
+
+```bash
+
+$ luet search <regex>
+
+```
+
+To search a package and display results in a table:
+
+```bash
+
+$ luet search --table <regex>
+
+```
+
+To look into the installed packages:
+
+```bash
+
+$ luet search --installed <regex>
+
+```
+
+Note: the regex argument is optional
+
+## Search file belonging to packages
+
+```bash
+$ luet search --file <file_pattern>
+```
+
+## Show package files
+
+Files are displayed when visualizing output of search in json or in yaml, for instance, consider the following example:
+
+```bash
+$ luet search -o yaml system/luet-0.32.5
+packages:                                                                                           
+- category: system
+  files:
+  - usr/bin/luet
+  hidden: false
+  installed: true
+  name: luet
+  repository: luet
+  target: ""
+  version: 0.32.5
+```
+
+### Search output
+
+Search can return results in the terminal in different ways: as terminal output, as json or as yaml.
+
+#### JSON
+
+```bash
+
+$ luet search --json <regex>
+
+```
+
+#### YAML
+
+```bash
+
+$ luet search --yaml <regex>
+
+```
+
+#### Tabular
+
+
+```bash
+
+$ luet search --table <regex>
+
+```
+
+## Quiet luet output
+
+Luet output is verbose by default and colourful, however will try to adapt to the terminal, based on which environment is executed (as a service, in the terminal, etc.)
+
+You can quiet `luet` output with  the `--quiet` flag or `-q` to have a more compact output in all the commands.

docs/content/en/docs/Concepts/Packages/_index.md

@@ -0,0 +1,271 @@
+---
+title: "Packages"
+linkTitle: "Packages"
+weight: 2
+description: >
+  Package definition syntax
+---
+
+A Package in Luet is denoted by a triple (`name`, `category` and `version`), here called *package form* in a `definition.yaml` file in YAML: 
+
+```yaml
+name: "awesome"
+version: "0.1"
+category: "foo"
+```
+
+While `category` and `version` can be omitted, the name is required. Note that when refering to a package, the triplet is always present:
+
+```yaml
+requires:
+- name: "awesome"
+  version: "0.1"
+  category: "foo"
+- name: "bar"
+  version: "0.1"
+  category: "foo"
+```
+
+## Building process
+
+When a package is required to be built, Luet resolves the dependency trees and orders the spec files to satisfy the given contraints.
+
+Each package build context is where the spec files are found (`definition.yaml` and `build.yaml`). This means that in the container, which is running the build process, the resources inside the package folder are accessible, as normally in Docker.
+
+```
+❯ tree distro/raspbian/buster
+distro/raspbian/buster
+├── build.sh
+├── build.yaml
+├── definition.yaml
+└── finalize.yaml
+```
+In the example above, `build.sh` is accessible in build time and can be invoked easily in build time in `build.yaml`:
+```yaml
+steps:
+- sh build.sh
+```
+
+## Package provides
+
+Packages can specify a list of `provides`. This is a list of packages in *package form*, which indicates that the current definition *replaces* every occurrence of the packages in the list (both at *build* and *runtime*). This mechanism is particularly helpful for handling package moves or for enabling virtual packages (e.g., [gentoo virtual packages](https://packages.gentoo.org/categories/virtual)).
+
+*Note: packages in the `provides` list don't need to exist or have a valid build definition either.*
+
+## Package types
+
+By a combination of keywords in `build.yaml`, you end up with categories of packages that can be built:
+
+- Seed packages
+- Packages deltas
+- Package layers
+- Package with includes
+
+Check the [Specfile concept](/docs/concepts/packages/specfile) page for a full overview of the available keywords in the Luet specfile format.
+
+### Seed packages
+
+Seed packages denote a parent package (or root) that can be used by other packages as a dependency. Normally, seed packages include just an image (preferably tagged) used as a base for other packages to depend on.
+
+It is useful to pin to specific image versions, and to write down in a tree where packages are coming from. There can be as many seed packages as you like in a tree.
+
+A seed package `build.yaml` example is the following:
+
+```yaml
+image: "alpine:3.1"
+```
+
+Every other package that depends on it will inherit the layers from it.
+
+If you want to extract the content of the seed package in a separate packages (splitting), you can just create as many package as you wish depending on that one, and extract its content, for example:
+
+**alpine/build.yaml**
+```yaml
+image: "alpine:3.1"
+```
+
+**alpine/definition.yaml**
+```yaml
+name: "alpine"
+version: "3.1"
+category: "seed"
+```
+
+**sh/build.yaml**
+```yaml
+# List of build-time dependencies
+requires:
+- name: "alpine"
+  version: "3.1"
+  category: "seed"
+unpack: true # Tells luet to use the image content by unpacking it
+includes: 
+- /bin/sh
+```
+
+**sh/definition.yaml**
+```yaml
+name: "sh"
+category: "utils"
+version: "1.0"
+```
+
+In this example, there are two packages being specified:
+
+- One is the `seed` package, which is the base image employed to later extract packages. It has no installable content, and it is just virtually used during build phase.
+- `sh` is the package which contains `/bin/sh`, extracted from the seed image and packaged. This can be consumed by Luet clients in order to install `sh` in their system.
+
+### Packages delta
+
+Luet, by default, will try to calculate the delta of the package that is meant to be built. This means that it tracks **incrementally** the changes in the packages, to ease the build definition. Let's see an example.
+
+Given the root package:
+**alpine/build.yaml**
+```yaml
+image: "alpine:3.1"
+```
+
+**alpine/definition.yaml**
+```yaml
+name: "alpine"
+version: "3.1"
+category: "seed"
+```
+
+We can generate any file, and include it in our package by defining this simple package:
+
+**foo/build.yaml**
+```yaml
+# List of build-time dependencies
+requires:
+- name: "alpine"
+  version: "3.1"
+  category: "seed"
+steps:
+- echo "Awesome" > /foo
+```
+
+**foo/definition.yaml**
+```yaml
+name: "foo"
+category: "utils"
+version: "1.0"
+```
+
+By analyzing the difference between the two packages, Luet will automatically track and package `/foo` as part of the `foo` package.
+
+To allow operations that must not be accounted in to the final package, you can use the `prelude` keyword:
+
+**foo/build.yaml**
+```yaml
+# List of build-time dependencies
+requires:
+- name: "alpine"
+  version: "3.1"
+  category: "seed"
+prelude:
+- echo "Not packaged" > /invisible
+steps:
+- echo "Awesome" > /foo
+```
+
+**foo/definition.yaml**
+```yaml
+name: "foo"
+category: "utils"
+version: "1.0"
+```
+
+The list of commands inside `prelude` that would produce artifacts, are not accounted to the final package. In this example, only `/foo` would be packaged (which output is equivalent to the example above).
+
+This can be used, for instance, to fetch sources that must not be part of the package.
+
+You can apply restrictions anytime and use the `includes` keyword to specifically pin to the files you wish in your package.
+
+### Package layers
+
+Luet can be used to track entire layers and make them installable by Luet clients. 
+
+Given the examples above:
+
+**alpine/build.yaml**
+```yaml
+image: "alpine:3.1"
+```
+
+**alpine/definition.yaml**
+```yaml
+name: "alpine"
+version: "3.1"
+category: "seed"
+```
+
+An installable package derived by the seed, with the actual full content of the layer can be composed as follows:
+
+**foo/build.yaml**
+```yaml
+# List of build-time dependencies
+requires:
+- name: "alpine"
+  version: "3.1"
+  category: "seed"
+unpack: true # It advertize Luet to consume the package as is
+```
+
+**foo/definition.yaml**
+```yaml
+name: "foo"
+category: "utils"
+version: "1.0"
+```
+
+This can be combined with other keywords to manipulate the resulting package (layer), for example:
+
+**foo/build.yaml**
+```yaml
+# List of build-time dependencies
+requires:
+- name: "alpine"
+  version: "3.1"
+  category: "seed"
+unpack: true # It advertize Luet to consume the package as is
+steps:
+- apk update
+- apk add git
+- apk add ..
+```
+
+**foo/definition.yaml**
+```yaml
+name: "foo"
+category: "utils"
+version: "1.0"
+```
+
+### Package includes
+
+In addition, the `includes` keyword can be set in order to extract portions from the package image.
+
+**git/build.yaml**
+```yaml
+# List of build-time dependencies
+requires:
+- name: "alpine"
+  version: "3.1"
+  category: "seed"
+unpack: true # It advertize Luet to consume the package as is
+steps:
+- apk update
+- apk add git
+includes:
+- /usr/bin/git
+```
+
+**foo/definition.yaml**
+```yaml
+name: "git"
+category: "utils"
+version: "1.0"
+```
+
+As a reminder, the `includes` keywords accepts regular expressions in the Golang format. Any criteria expressed by means of Golang regular expressions, and matching the file name (absolute path), will be part of the final package.

docs/content/en/docs/Concepts/Packages/collections.md

@@ -0,0 +1,33 @@
+---
+title: "Collections"
+linkTitle: "Collections"
+weight: 4
+description: >
+  Group a set of package build spec with templating
+---
+
+`Collections` are a special superset of packages. To define a collection, instead of using a `definition.yaml` file, create a `collection.yaml` file with a list of packages:
+
+{{<githubembed repo="mudler/luet" file="tests/fixtures/shared_templates/test/collection.yaml" lang="yaml">}}
+
+Packages under a collection shares the same `build.yaml` and `finalize.yaml`, so a typical package layout can be:
+
+```
+collection/
+    collection.yaml
+    build.yaml
+    finalize.yaml
+    ... additional files in the build context
+```
+
+Luet during the build phase, will treat packages of a collection individually. A collection is a way to share the same build process across different packages.
+
+## Templating
+
+[The templating mechanism](/docs/concepts/packages/templates) can be used in collections as well, and each stanza in `packages` is used to interpolate each single package. 
+
+## Examples
+
+- https://github.com/mocaccinoOS/mocaccino-musl-universe/tree/master/multi-arch/packages/entities
+- https://github.com/mocaccinoOS/portage-tree/tree/master/multi-arch/packages/groups
+- https://github.com/mocaccinoOS/mocaccino-musl-universe/tree/master/multi-arch/packages/X

docs/content/en/docs/Concepts/Packages/specfile.md

@@ -0,0 +1,658 @@
+---
+title: "Specfile"
+linkTitle: "Specfile"
+weight: 2
+description: >
+  Luet specfile syntax
+---
+
+
+# Specfiles
+
+Luet [packages](/docs/concepts/packages/) are defined by specfiles. Specfiles define the runtime and builtime requirements of a package.  There is an hard distinction between runtime and buildtime. A spec is composed at least by the runtime (`definition.yaml` or a `collection.yaml`) and the buildtime specification (`build.yaml`).
+
+Luet identifies the package definition by looking at directories that contains a `build.yaml` and a `definition.yaml` (or `collection.yaml`) files. A Luet tree is merely a composition of directories that follows this convention. There is no constriction on either folder naming or hierarchy.
+
+*Example of a [tree folder hierarchy](https://github.com/Luet-lab/luet-embedded/tree/master/distro)*
+```bash
+tree distro                                                      
+distro
+├── funtoo              
+│   ├── 1.4
+│   │   ├── build.sh        
+│   │   ├── build.yaml                                             
+│   │   ├── definition.yaml
+│   │   └── finalize.yaml
+│   ├── docker
+│   │   ├── build.yaml
+│   │   ├── definition.yaml
+│   │   └── finalize.yaml
+│   └── meta
+│       └── rpi
+│           └── 0.1
+│               ├── build.yaml
+│               └── definition.yaml
+├── packages
+│   ├── container-diff
+│   │   └── 0.15.0
+│   │       ├── build.yaml
+│   │       └── definition.yaml
+│   └── luet
+│       ├── build.yaml
+│       └── definition.yaml
+├── raspbian
+│   ├── buster
+│   │   ├── build.sh
+│   │   ├── build.yaml
+│   │   ├── definition.yaml
+│   │   └── finalize.yaml
+│   ├── buster-boot
+│   │   ├── build.sh
+│   │   ├── build.yaml
+│   │   ├── definition.yaml
+│   │   └── finalize.yaml
+```
+
+## Build specs
+
+Build specs are defined in `build.yaml` files. They denote the build-time `dependencies` and `conflicts`, together with a definition of the content of the package.
+
+*Example of a `build.yaml` file*:
+```yaml
+steps:
+- echo "Luet is awesome" > /awesome
+prelude:
+- echo "nooops!"
+requires:
+- name: "echo"
+  version: ">=1.0"
+conflicts:
+- name: "foo"
+  version: ">=1.0"
+provides:
+- name: "bar"
+  version: ">=1.0"
+env:
+- FOO=bar
+includes:
+- /awesome
+
+unpack: true
+```
+
+### Building strategies
+
+Luet can create packages with different strategies: 
+
+- by delta. Luet will analyze the containers differencies to find out which files got **added**. 
+  You can use the `prelude` section to exclude certains file during analysis.
+- by taking a whole container content
+- by considering a single directory in the build container.
+
+#### Package by delta
+
+By default Luet will analyze the container content and extract any file that gets **added** to it. The difference is calculated by using the container which is depending on, or either by the container which is created by running the steps in the `prelude` section of the package build spec:
+
+```yaml
+prelude:
+- # do something...
+steps:
+- # real work that should be calculated delta over
+```
+
+By omitting the `prelude` keyword, the delta will be calculated from the parent container where the build will start from.
+
+#### Package by container content
+
+Luet can also generate a package content from a container. This is really useful when creating packages that are entire versioned `rootfs`. To enable this behavior, simply add `unpack: true` to the `build.yaml`. This enables the Luet unpacking features, which will extract all the files contained in the container which is built from the `prelude` and `steps` fields.
+
+To include/exclude single files from it, use the `includes` and `excludes` directives.
+
+#### Package by a folder in the final container
+
+Similarly, you can tell Luet to create a package from a folder in the build container. To enable this behavior, simply add `package_dir: "/path/to/final/dir"`.
+The directory must represent exactly how the files will be ultimately installed from clients, and they will show up in the same layout in the final archive.
+
+So for example, to create a package which ships `/usr/bin/mybin`, we could write:
+```yaml
+package_dir: "/output"
+steps:
+- mkdir -p /output/usr/bin/
+- echo "fancy stuff" > /output/usr/bin/mybin && chmod +x /output/usr/bin/mybin
+```
+
+### Build time dependencies
+
+A package build spec defines how a package is built. In order to do this, Luet needs to know where to start. Hence a package must declare at least either one of the following:
+
+- an `image` keyword which tells which Docker image to use as base, or
+- a list of `requires`, which are references to other packages available in the tree.
+
+They can't be both present in the same specfile. 
+
+To note, it's not possible to mix package build definitions from different `image` sources. They must form a unique sub-graph in the build dependency tree. 
+
+On the other hand it's possible to have multiple packages depending on a combination of different `requires`, given they are coming from the same `image` parent.
+
+### Excluding/including files explictly
+
+Luet can also *exclude* and *include* single files or folders from a package by using the `excludes` and `includes` keyword respecitvely. 
+
+Both of them are parsed as a list of Golang regex expressions, and they can be combined together to fine-grainly decide which files should be inside the final artifact. You can refer to the files as they were in the resulting package. So if a package produces a `/foo` file,  and you want to exclude it, you can add it to `excludes` as `/foo`.
+
+### Package source image
+
+Luet needs an image to kick-off the build process for each package. This image is being used to run the commands in the `steps` and `prelude`, and then the image is processed by the **building strategies** explained above. 
+
+The image can be resolved either by: 
+
+1) providing a specific image name with `image` 
+2) providing a set of package requirements with `requires` which will be constructed a new image from. The resulting image is an image linked between each other with the `FROM` field in the Dockerfile following the SAT solver ordering.
+3) providing a set of packages to squash their result from `requires` and by specifying `requires_final_images: true`. 
+
+{{< alert color="info" title="Note" >}}
+The above keywords cannot be present in the same spec **at the same time**, or they cannot be combined. But you are free to create further intermediate specs to achieve the desired image.
+{{< /alert >}}
+
+#### Difference between `requires` and `requires` with `requires_final_images: true`
+
+`requires` generates a graph from all the `images` of the specfile referenced inside the list. This means it builds a chain of images that are used to build the packages, e.g.: `packageA(image: busybox) -> packageB (requires: A) -> packageC (requires: C)`. The container which is running your build then **inherits** it's parents from a chain of order resolution, provided by the SAT solver.
+
+When specifying `requires_final_images: true` luet builds an artifact for each of the packages listed from their compilation specs and it will later *squash* them together in a new container image which is then used in the build process to create an artifact.
+
+The key difference is about *where* your build is going to run from. By specifying `requires_final_images` it will be constructed a new image with the content of each package specified and its dependencies - while if setting it to false, it will order the images appropriately and link them together with the Dockerfile `FROM` field. That allows to reuse the same images used to build the packages in the require section - or - create a new one from the result of each package compilation.
+
+## Keywords
+
+Here is a list of the full keyword refereces for the `build.yaml` file.
+
+### `conflicts`
+
+(optional) List of packages which it conflicts with in *build time*. In the same form of `requires` it is a list of packages that the current one is conflicting with.
+
+```yaml
+conflicts:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+```
+
+See [Package concepts](/docs/concepts/packages) for more information on how to represent a package in a Luet tree.
+
+### `copy`
+
+_since luet>=0.15.0_
+
+(optional) A list of packages/images where to copy files from. It is the [Docker multi-stage build](https://docs.docker.com/develop/develop-images/multistage-build/) equivalent but enhanced with tree hashing resolution.
+
+To copy a specific file from a package *build* container:
+
+```yaml
+steps:
+- ...
+prelude:
+- ...
+copy:
+- package: 
+    category: "foo"
+    name: "bar"
+    version: ">=0"
+  source: "/foo"
+  destination: "/bar"
+```
+
+Any package that is listed in the section will be compiled beforeahead the package, and the file is available both in `prelude` and `steps`.
+
+Internally, it's rendered as `COPY --from=package/image:sha /foo /bar`
+
+To copy a specific file from an external image:
+
+```yaml
+steps:
+- ...
+prelude:
+- ...
+copy:
+- image: "buxybox:latest"
+  source: "/foo"
+  destination: "/bar"
+```
+
+### `env`
+
+(optional) A list of environment variables ( in `NAME=value` format ) that are expanded in `step` and in `prelude`. ( e.g. `${NAME}` ).
+
+```yaml
+env:
+- PATH=$PATH:/usr/local/go/bin
+- GOPATH=/luetbuild/go
+- GO111MODULE=on
+- CGO_ENABLED=0
+- LDFLAGS="-s -w"
+```
+
+### `excludes`
+
+(optional) List of golang regexes. They are in full path form (e.g. `^/usr/bin/foo` ) and indicates that the files listed shouldn't be part of the final artifact
+
+Wildcards and golang regular expressions are supported. If specified, files which are not matching any of the regular expressions in the list will be excluded in the final package.
+
+```yaml
+excludes:
+- ^/etc/shadow
+- ^/etc/os-release
+- ^/etc/gshadow
+```
+
+By combining `excludes` with `includes`, it's possible to include certain files while excluding explicitly some others (`excludes` takes precedence over `includes`).
+
+
+### `image`
+
+(optional/required) Docker image to be used to build the package.
+
+```yaml
+image: "busybox"
+```
+
+It might be omitted in place of `requires`, and indicates the image used to build the package. The image will be pulled and used to build the package.
+
+### `includes`
+
+(optional)  List of regular expressions to match files in the resulting package. The path is absolute as it would refer directly to the artifact content.
+
+Wildcards and golang regular expressions are supported. If specified, files which are not matching any of the regular expressions in the list will be excluded in the final package.
+
+```yaml
+includes:
+- /etc$
+- /etc/lvm$
+- /etc/lvm/.*
+- /usr$
+- /usr/bin$
+- /usr/bin/cc.*
+- /usr/bin/c\+\+.*
+- /usr/bin/cpp.*
+- /usr/bin/g\+\+.*
+```
+
+__Note__: Directories are treated as standard entries, so to include a single file, you need also to explictly include also it's directory. Consider this example to include `/etc/lvm/lvm.conf`:
+```yaml
+includes:
+- /etc$
+- /etc/lvm$
+- /etc/lvm/lvm.conf
+```
+
+### `join`
+
+_since luet>=0.16.0_
+_to be deprecated in luet>=0.18.0 in favor of `requires_final_images`_
+
+(optional/required) List of packages which are used to generate a parent image from.
+
+It might be omitted in place of `image` or `requires`, and will generate an image which will be used as source of the package from the final packages in the above list. The new image is used to run eventually the package building process and a new artifact can be generated out of it.
+
+
+```yaml
+join:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+```
+
+See [Package concepts](/docs/concepts/packages) for more information on how to represent a package in a Luet tree.
+
+#### Examples
+
+- https://github.com/mocaccinoOS/mocaccino-stage3/blob/278e3637cf65761bf01a22c891135e237e4717ad/packages/system/stage3/build.yaml
+
+### `package_dir`
+
+(optional) A path relative to the build container where to create the package from.
+
+Similarly to `unpack`, changes the building strategy.
+
+
+```yaml
+steps:
+- mkdir -p /foo/bar/etc/myapp
+- touch /foo/bar/etc/myapp/config
+package_dir: /foo/bar
+```
+
+### `prelude`
+
+(optional) A list of commands to perform in the build container before building.
+
+```yaml
+prelude:
+- |
+   PACKAGE_VERSION=${PACKAGE_VERSION%\+*} && \
+   git clone https://github.com/mudler/yip && cd yip && git checkout "${PACKAGE_VERSION}" -b build
+```
+
+### `requires`
+
+(optional/required) List of packages which it depends on.
+
+A list of packages that the current package depends on in *build time*. It might be omitted in place of `image`, and determines the resolution tree of the package itself. A new image is composed from the packages listed in this section in order to build the package
+
+```yaml
+requires:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+```
+
+See [Package concepts](/docs/concepts/packages) for more information on how to represent a package in a Luet tree.
+
+### `requires_final_images`
+
+_since luet>=0.17.0_
+
+(optional) A boolean flag which instruct luet to use the final images in the `requires` field.
+
+By setting `requires_final_images: true` in the compilation spec, packages in the `requires` section and its dependencies will be fetched if available or compiled, and afterwards the result is squashed together in a new image that will be used as a source of the build process of the package.
+
+```yaml
+requires:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+
+requires_final_images: true
+```
+
+`requires_final_images` replaces the use of `join`, which will be deprecated in luet `>=0.18.0`.
+
+### `step`
+
+(optional) List of commands to perform in the build container.
+
+```yaml
+steps:
+- |
+   cd yip && make build-small && mv yip /usr/bin/yip
+```
+
+### `unpack`
+
+(optional) Boolean flag. It indicates to use the unpacking strategy while building a package
+
+```yaml
+unpack: true
+```
+
+It indicates that the package content **is** the whole container content.
+
+### `subpackages`
+
+_since luet>=0.32.0_
+
+(optional) A list of packages to create with the result of the current package definition.
+
+```yaml
+subpackages:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+  includes: 
+  - ...
+  excludes:
+  - ...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+```
+
+After generating the main package, `luet` will create split packages from the resulting one from the list.
+
+Every subpackage stanza supports `excludes` and `includes` to selectively exclude and include files in every resulting package.
+
+Note, subpackages support is available for collection, standard packages and templated packages.
+
+See [Package concepts](/docs/concepts/packages) for more information on how to represent a package in a Luet tree.
+
+## Rutime specs
+
+Runtime specification are denoted in a `definition.yaml` or a `collection.yaml` sibiling file. It identifies the package and the runtime contraints attached to it.
+
+*definition.yaml*:
+```yaml
+name: "awesome"
+version: "0.1"
+category: "foo"
+requires:
+- name: "echo"
+  version: ">=1.0"
+  category: "bar"
+conflicts:
+- name: "foo"
+  version: "1.0"
+provides:
+- name: "bar"
+  version: "<1.0"
+```
+
+A `collection.yaml` can be used in place of a `definition.yaml` to identify a **set** of packages that instead shares a common `build.yaml`:
+
+*collection.yaml*:
+```yaml
+packages:
+- name: "awesome"
+  version: "0.1"
+  category: "foo"
+  requires:
+  - name: "echo"
+    version: ">=1.0"
+    category: "bar"
+  conflicts:
+  - name: "foo"
+    version: "1.0"
+  provides:
+  - name: "bar"
+    version: "<1.0"
+- name: "awesome"
+  version: "0.2"
+  category: "foo"
+  requires:
+  - name: "echo"
+    version: ">=1.0"
+    category: "bar"
+  conflicts:
+  - name: "foo"
+    version: "1.0"
+  provides:
+  - name: "bar"
+    version: "<1.0"
+...
+```
+
+All the fields (also the ones which are not part of the spec) in the `definition.yaml` file are available as templating values when rendering the `build.yaml` file. When running [finalizers](/docs/concepts/packages/specfile/#finalizers) instead only the fields belonging to the specs are available.
+
+### Keywords
+
+Here is a list of the full keyword refereces
+
+
+### `annotations`
+
+(optional) A map of freeform package annotations:
+
+```yaml
+annotations:
+  foo: "bar"
+  baz: "test"
+```
+
+#### `category`
+
+(optional) A string containing the category of the package
+
+```yaml
+category: "system"
+```
+
+### `conflicts`
+
+(optional) List of packages which it conflicts with in *runtime*. In the same form of `requires` it is a list of packages that the current one is conflicting with.
+
+```yaml
+conflicts:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+```
+
+See [Package concepts](/docs/concepts/packages) for more information on how to represent a package in a Luet tree.
+
+### `description`
+
+(optional) A string indicating the package description
+
+```yaml
+name: "foo"
+description: "foo is capable of..."
+```
+
+
+### `hidden`
+
+(optional) A boolean indicating whether the package has to be shown or not in the search results (`luet search...`)
+
+```yaml
+hidden: true
+```
+
+### `labels`
+
+(optional) A map of freeform package labels:
+
+```yaml
+labels:
+  foo: "bar"
+  baz: "test"
+```
+
+Labels can be used in `luet search` to find packages by labels, e.g.:
+
+```bash
+$> luet search --by-label foo
+```
+
+### `license`
+
+(optional) A string indicating the package license type.
+
+```yaml
+license: "GPL-3"
+```
+
+#### `name`
+
+(required) A string containing the name of the package
+
+```yaml
+name: "foo"
+```
+
+### `provides`
+
+(optional) List of packages which the current package is providing.
+
+```yaml
+conflicts:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+```
+
+See [Package concepts](/docs/concepts/packages) for more information on how to represent a package in a Luet tree.
+
+### `requires`
+
+(optional) List of packages which it depends on in runtime.
+
+A list of packages that the current package depends on in *runtime*. The determines the resolution tree of the package itself.
+
+```yaml
+requires:
+- name: "foo"
+  category: "bar"
+  version: "1.0"
+...
+- name: "baz"
+  category: "bar"
+  version: "1.0"
+```
+
+See [Package concepts](/docs/concepts/packages) for more information on how to represent a package in a Luet tree.
+
+### `uri`
+
+(optional) A list of URI relative to the package ( e.g. the official project pages, wikis, README, etc )
+
+```yaml
+uri:
+- "http://www.mocaccino.org"
+- ...
+```
+
+#### `version`
+
+(required) A string containing the version of the package
+
+```yaml
+version: "1.0"
+```
+
+
+
+
+## Refering to packages from the CLI
+
+All the `luet` commands which takes a package as argument, respect the following syntax notation:
+
+- `cat/name`: will default to selecting any available package
+- `=cat/name`: will default to gentoo parsing with regexp so also `=cat/name-1.1` works
+- `cat/name@version`: will select the specific version wanted ( e.g. `cat/name@1.1` ) but can also include ranges as well `cat/name@>=1.1`
+- `name`: just name, category is omitted and considered empty
+
+## Finalizers
+
+Finalizers are denoted in a `finalize.yaml` file, which is a sibiling of `definition.yaml` and `build.yaml` file. It contains a list of commands that finalize the package when it is installed in the machine.
+
+*finalize.yaml*:
+```yaml
+install:
+- rc-update add docker default
+```
+
+### Keywords
+
+- `install`: List of commands to run in the host machine. Failures are eventually ignored, but will be reported and luet will exit non-zero in such case.

docs/content/en/docs/Concepts/Packages/templates.md

@@ -0,0 +1,95 @@
+---
+title: "Templated packages"
+linkTitle: "Templated packages"
+weight: 3
+description: >
+  Use templates to fine tune build specs
+---
+
+Luet supports the [`sprig` rendering engine template, like helm](http://masterminds.github.io/sprig/). It's being used to interpolate `build.yaml` and `finalize.yaml` files before their execution. The following document assumes you are familiar with the `helm` templating.
+
+The `build.yaml` and `finalize.yaml` files are rendered during build time, and it's possible to use the `helm` templating syntax inside such files. The `definition.yaml` file will be used to interpolate templating values available in `build.yaml`
+
+Given the following `definition.yaml`:
+
+```yaml
+name: "test"
+category: "foo"
+version: "1.1"
+
+additional_field: "baz"
+```
+
+A `build.yaml` can look like the following, and interpolates it's values during build time:
+
+```yaml
+image: ...
+
+steps:
+- echo {{.Values.name}} > /package_name
+- echo {{.Values.additional_field}} > /extra
+
+```
+
+Which would be for example automatically rendered by luet like the following:
+
+```yaml
+
+image: ...
+
+steps:
+- echo test > /package_name
+- echo baz > /extra
+
+```
+
+This mechanism can be used in collections as well, and each stanza in `packages` is used to interpolate each single package.
+
+## Interpolating globally
+
+It's possible to interpolate during build phase all the package specs targeted for build with the ```--values``` flag, which takes a yaml file of an arbitrary format, if variables are clashing, the yaml supplied in `--values` takes precedence and overwrite the values of each single `definition.yaml` file.
+
+## Shared templates
+
+Since luet `0.17.5` it is possible to share templates across different packages. All templates blocks found inside the `templates` folder inside the root `luet tree` of a repository gets templated and shared across all the packages while rendering each compilation spec of the given tree.
+
+Consider the following:
+
+```
+shared_templates
+├── templates
+│   └── writefile.yaml
+└── test
+    ├── build.yaml
+    └── collection.yaml
+```
+#### `collection.yaml`
+We define here two packages with a collection. They will share the same compilation spec to generate two different packages
+{{<githubembed repo="mudler/luet" file="tests/fixtures/shared_templates/test/collection.yaml" lang="yaml">}}
+
+
+#### `writefile.yaml`
+
+All the files in the `templates` folder will get rendered by the template for each package in the tree. We define here a simple block to write out a file from the context which is passed by:
+{{<githubembed repo="mudler/luet" file="tests/fixtures/shared_templates/templates/writefile.yaml" lang="yaml">}}
+
+
+#### `build.yaml`
+Finally the build spec consumes the template block we declared above, passing by the name of the package:
+{{<githubembed repo="mudler/luet" file="tests/fixtures/shared_templates/test/build.yaml" lang="yaml">}}
+
+
+## Limitations
+
+The `finalize.yaml` file has access only to the package fields during templating. Extra fields that are present in the `definition.yaml` file are *not* accessible during rendering in the `finalize.yaml` file, but only the package fields (`name`, `version`, `labels`, `annotations`, ...)
+
+## References
+
+- [Sprig docs](http://masterminds.github.io/sprig/)
+- [Helm Templating functions](https://helm.sh/docs/chart_template_guide/function_list/)
+- [Helm Templating variable](https://helm.sh/docs/chart_template_guide/variables/)
+
+## Examples
+
+- https://github.com/mocaccinoOS/mocaccino-musl-universe/tree/master/multi-arch/packages/tar
+

docs/content/en/docs/Concepts/Plugins and Extensions/_index.md

@@ -0,0 +1,153 @@
+
+---
+title: "Plugins and Extensions"
+linkTitle: "Plugins and Extensions"
+weight: 3
+description: >
+  Extend luet with plugins and extensions
+---
+
+Luet can be extended in 2 ways by extensions and plugins.
+
+# Before you begin
+
+You need to have a working `luet` binary installed.
+
+## Extensions
+
+Extensions expand Luet featureset horizontally, so for example, “luet geniso” will allow you to build an iso using luet, without this needing to be part of the luet core.
+
+An Extension is nothing more than a standalone executable file, whose name begins with `luet-`. To install an extension, simply move its executable file to anywhere on your system `PATH`. 
+
+All the plugins will be accessible to luet as `luet pluginname`
+
+### Writing an Extension 
+
+You can write an extension in any programming language or script that allows you to write command-line commands.
+
+Executables receive the inherited environment from luet. An extension determines which command path it wishes to implement based on its name. For example, a plugin wanting to provide a new command luet foo, would simply be named luet-foo, and live somewhere in your PATH.
+
+#### Example Extension
+
+```bash
+#!/bin/bash
+
+if [[ "$1" == "help" ]]
+then
+    echo "Extension help"
+    exit 0
+fi
+
+if [[ "$1" == "run" ]]
+then
+    # do something interesting
+fi
+
+echo "I am an Extension named luet-foo"
+
+```
+### Using an Extension
+
+To use the above extension, simply make it executable:
+
+```bash
+$ sudo chmod +x ./luet-foo
+```
+
+and place it anywhere in your PATH:
+
+```bash
+$ sudo mv ./luet-foo /usr/local/bin
+```
+
+You may now invoke your extension as a luet command:
+
+```bash
+$ luet foo
+I am an Extension named luet-foo
+```
+
+All args and flags are passed as-is to the executable:
+
+```bash
+$ luet foo help
+
+Extension help
+```
+## Plugins
+
+Plugins instead are expanding Luet vertically by hooking into internal events. Plugins and Extensions can be written in any language, bash included! Luet uses [go-pluggable](https://github.com/mudler/go-pluggable) so it can dispatch events to external binaries.
+
+Similarly to **Extensions**, a **Plugin** is nothing more than a standalone executable file, but without any special prefix. To install a plugin, simply move its executable file to anywhere on your system `PATH`. 
+
+Differently from **Extensions**, they are not available from the **CLI** and cannot be invoked directly by the user, instead they are called by Luet during its lifecycle.
+
+### Writing a Plugin
+
+You can write a plugin in any programming language or script.
+
+The first argument that is passed to a plugin will always be the event that was emitted by Luet in its lifecycle. You can see all the [events available here](https://github.com/mudler/luet/blob/master/pkg/bus/events.go). The second argument, is a `JSON` encoded payload of the object that Luet is emitting with the event. The object(s) may vary depending on the emitted event.
+
+The output of the plugin (`stdout`) will be parsed as JSON. Every plugin must return a valid JSON at the end of its execution, or it will be marked as failed and stops `luet` further execution. [See also the go-pluggable README](https://github.com/mudler/go-pluggable#plugin-processed-data).
+
+The returning payload should be in the following form:
+
+```json
+{ "state": "", "data": "data", "error": ""}
+```
+
+By returning a json with the error field not empty, it will make fail the overall execution.
+
+
+#### Example Plugin
+
+```bash
+#!/bin/bash
+echo "$1" >> /tmp/event.txt
+echo "$2" >> /tmp/payload.txt
+
+echo "{}"
+
+```
+### Using a plugin
+
+To use the above plugin, simply make it executable:
+
+```bash
+$ sudo chmod +x ./test-foo
+```
+
+and place it anywhere in your PATH:
+
+```bash
+$ sudo mv ./test-foo /usr/local/bin
+```
+
+Now, when running luet, add ```--plugin test-foo```:
+
+```bash
+
+$ luet --plugin test-foo install -y foopackage
+
+```
+
+And check `/tmp/event.txt` to see the event fired and `/tmp/payload.txt` to check the payloads that were emitted by Luet.
+
+### Concrete example
+
+A plugin that prints the images that are being built in `/tmp/exec.log`:
+
+```bash
+#!/bin/bash
+exec >> /tmp/exec.log
+exec 2>&1
+event="$1"
+payload="$2"
+if [ "$event" == "image.post.build" ]; then
+  image=$(echo "$payload" | jq -r .data | jq -r .ImageName )
+    echo "{ \"data\": \"$image built\" }"
+else
+    echo "{}"
+fi
+
+```

docs/content/en/docs/Concepts/_index.md

@@ -0,0 +1,7 @@
+---
+title: "Concepts"
+linkTitle: "Concepts"
+weight: 2
+description: >
+  Documentation references
+---

docs/content/en/docs/Getting started/_index.md

@@ -0,0 +1,99 @@
+---
+title: "Getting Started"
+linkTitle: "Getting Started"
+weight: 1
+description: >
+  First steps with Luet
+---
+
+
+## Prerequisites
+
+No dependencies. For building packages [see the Build Packages section](/docs/concepts/overview/build_packages/)
+
+## Get Luet  
+
+### From release
+
+Just grab a release from [the release page on GitHub](https://github.com/mudler/luet/releases). The binaries are statically compiled.
+
+Or you can install Luet also with a single command:
+
+```bash
+curl https://luet.io/install.sh | sudo sh
+``` 
+
+### Building Luet from source
+
+Requirements:
+
+- [Golang](https://golang.org/) installed in your system.
+- make
+
+
+```bash
+$> git clone https://github.com/mudler/luet
+$> cd luet
+$> make build # or just go build
+```
+
+## Install it as a system package
+
+In the following section we will see how to install luet with luet itself. We will use a transient luet version that we are going to throw away right after we install it in the system.
+
+```bash
+# Get a luet release. It will be used to install luet in your system
+wget https://github.com/mudler/luet/releases/download/0.8.3/luet-0.8.3-linux-amd64 -O luet
+chmod +x luet
+
+# Creates the luet configuration file and add the luet-index repository.
+# The luet-index repository contains a collection of repositories which are 
+# installable and tracked in your system as standard packages.
+cat > .luet.yaml <<EOF
+repositories:
+- name: "mocaccino-repository-index"
+  description: "MocaccinoOS Repository index"
+  type: "http"
+  enable: true
+  cached: true
+  priority: 1
+  urls:
+  - "https://raw.githubusercontent.com/mocaccinoOS/repository-index/gh-pages"
+EOF
+
+# Install the official luet repository to get always the latest luet version
+./luet install repository/luet
+
+# Install luet (with luet) in your system
+./luet install system/luet
+
+# Remove the temporary luet used for bootstrapping
+rm -rf luet
+
+# Copy over the config file to your system
+mkdir -p /etc/luet
+mv .luet.yaml /etc/luet/luet.yaml
+```
+
+## Configuration
+
+Luet stores its configuration files in `/etc/luet`. If you wish to override its default settings, create a file `/etc/luet/luet.yaml`.
+
+An example of a configuration file can be found [here](https://github.com/mudler/luet/blob/master/contrib/config/luet.yaml).
+
+There are a bunch of configuration settings available, but the most relevant are:
+
+```yaml
+logging:
+  color: true # Enable/Disable colored output
+  enable_emoji: true # Enable/Disable emoji from output
+general:
+  debug: false # Enable/Disable debug
+system:
+  rootfs: "/" # What's our rootfs. Luet can install packages outside of "/"
+  database_path: "/var/db/luet" # Where to store DB files
+  database_engine: "boltdb"
+  tmpdir_base: "/var/tmp/luet" # The temporary directory to be used
+```
+
+To learn more about how to configure luet, [see the configuration section](/docs/concepts/overview/configuration/)

docs/content/en/docs/Resources/_index.md

@@ -0,0 +1,10 @@
+
+---
+title: "Resources"
+linkTitle: "Resources"
+weight: 20
+date: 2017-01-05
+description: >
+  Luet examples, resources, API reference
+---
+

docs/content/en/docs/Resources/arm.md

@@ -0,0 +1,66 @@
+---
+title: "ARM images"
+linkTitle: "ARM images"
+weight: 4
+description: >
+  Use Luet to build, track, and release OTA update for your embedded devices.
+---
+
+{{< alert color="warning" title="Warning" >}}
+This article is outdated.
+Please refer to the ["Hello World"](../../tutorials/hello_world/) tutorial instead.
+{{< /alert >}}
+
+Here we show an example on how to build "burnable" SD images for Raspberry Pi with Luet. This approach lets you describe and version OTA upgrades for your embedded devices, delivering upgrades as layer upgrades on the Pi.
+
+The other good side of the medal is that you can build a Luet package repository with multiple distributions (e.g. `Raspbian`, `OpenSUSE`, `Gentoo`, ... ) and switch among them in runtime. In the above example `Raspbian` and `Funtoo` (at the time of writing) are available.
+
+## Prerequisites
+
+You have to run the following steps inside an ARM board to produce arm-compatible binaries. Any distribution with Docker will work. Note that the same steps could be done in a cross-compilation approach, or with qemu-binfmt in a amd64 host. 
+
+You will also need in your host:
+
+- Docker
+- Luet installed (+container-diff) in `/usr/bin/luet` (arm build)
+- make
+
+## Build the packages
+
+Clone the repository https://github.com/Luet-lab/luet-embedded
+
+    $> git clone https://github.com/Luet-lab/luet-embedded
+    $> cd luet-embedded
+    $> sudo make build-all
+    ...
+
+If a rebuild is needed, just do `sudo make rebuild-all` after applying the changes.
+
+## Create the repository
+
+    $> sudo make create-repo
+    ...
+
+## Serve the repo locally
+
+    $> make serve-repo
+    ...
+
+## Create the flashable image
+
+### Funtoo based system
+
+    $> sudo LUET_PACKAGES='distro/funtoo-1.4 distro/raspbian-boot-0.20191208 system/luet-develop-0.5' make image
+    ...
+
+### Raspbian based system
+
+    $> sudo LUET_PACKAGES='distro/raspbian-0.20191208 distro/raspbian-boot-0.20191208 system/luet-develop-0.5' make image
+    ...
+
+
+At the end of the process, a file `luet_os.img`, ready to be flashed to an SD card, should be present in the current directory.
+
+## Add packages
+
+In order to build and add [packages](/docs/concepts/packages/) to the exiting repository, simply add or edit the [specfiles](/docs/concepts/packages/specfile) under the `distro` folder. When doing ```make rebuild-all``` the packages will be automatically compiled and made available to the local repository.

docs/content/en/docs/Resources/building.md

@@ -0,0 +1,74 @@
+---
+title: "Building"
+linkTitle: "Building"
+weight: 4
+description: >
+  Examples to build with Luet
+---
+
+## Simple package build
+
+Creating and building a simple [package](/docs/concepts/packages/):
+
+```
+$> mkdir package
+
+$> cat <<EOF > package/build.yaml
+image: busybox
+steps:
+- echo "foo" > /foo
+EOF
+
+$> cat <<EOF > package/definition.yaml
+name: "foo"
+version: "0.1"
+EOF
+
+$> luet build --all
+
+📦  Selecting  foo 0.1
+📦  Compiling foo version 0.1 .... ☕
+🐋  Downloading image luet/cache-foo-bar-0.1-builder
+🐋  Downloading image luet/cache-foo-bar-0.1
+📦   foo Generating 🐋  definition for builder image from busybox
+🐋  Building image luet/cache-foo-bar-0.1-builder
+🐋  Building image luet/cache-foo-bar-0.1-builder done
+ Sending build context to Docker daemon  4.096kB
+ ...
+
+```
+
+### Build packages
+
+In order to build a specific version, a full [package](/docs/concepts/packages/) definition (triple of `category`, `name` and `version`) has to be specified.
+In this example we will also enable package compression (gzip).
+
+```
+$> mkdir package
+
+$> cat <<EOF > package/build.yaml
+image: busybox
+steps:
+- echo "foo" > /foo
+EOF
+
+$> cat <<EOF > package/definition.yaml
+name: "foo"
+version: "0.1"
+category: "bar"
+EOF
+
+$> luet build bar/foo-0.1 --compression gzip
+
+📦  Selecting  foo 0.1
+📦  Compiling foo version 0.1 .... ☕
+🐋  Downloading image luet/cache-foo-bar-0.1-builder
+🐋  Downloading image luet/cache-foo-bar-0.1
+📦   foo Generating 🐋  definition for builder image from busybox
+🐋  Building image luet/cache-foo-bar-0.1-builder
+🐋  Building image luet/cache-foo-bar-0.1-builder done
+ Sending build context to Docker daemon  4.096kB
+ ...
+
+```
+

docs/content/en/docs/Resources/faq.md

@@ -0,0 +1,30 @@
+---
+title: "Frequently Asked Questions"
+linkTitle: "FAQ"
+weight: 4
+description: >
+  FAQ
+---
+
+## Can't build packages
+
+There might be several reasons why packages fails to build, for example, if your build fails like this:
+
+```
+$ luet build ...
+
+ INFO   Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
+  ERROR    Error: Failed compiling development/toolchain-go-0.6: failed building package image: Could not push image: quay.io/mocaccino/micro-toolchain:latest toolchain-go-development-0.6-builder.dockerfile: Could not build image: quay.io/mocaccino/micro-toolchain:latest toolchain-go-development-0.6-builder.dockerfile: Failed running command: : exit status 1
+  ERROR    Bailing out
+```
+
+means the user you are running the build command can't either connect to docker or `docker` is not started.
+
+Check if the user you are running the build is in the `docker` group, or if the `docker` daemon is started.
+
+Luet by default if run with multiple packages summarize errors and can be difficult to navigate to logs, but if you think you might have found a bug, run the build with `--debug` before opening an issue.
+
+## Why the name `luet`?
+
+Well, I have the idea that programs should be small, so they are not difficult to type and easy to remember, and easy to stick in. `luet` is really a combination of the first letters of my fiancee name (Lucia) and my name (Ettore) `lu+et = luet`! and besides, happen to be also a [small bridge](http://www.comuniterrae.it/punto/ponte-luet/) in Italy ;)
+

docs/content/en/docs/Resources/references.md

@@ -0,0 +1,22 @@
+---
+title: "References"
+linkTitle: "References"
+weight: 100
+description: >-
+     References to various resources related to luet
+---
+
+## References
+
+Here is a list of references to projects that are related to Luet (open up a PR to add yours to the list!):
+
+| Description                                                                                                                                                    | URL                                                |
+|----------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------|
+| Official Luet repository                                                                                                                                       | https://github.com/Luet-lab/luet-repo              |
+| Example repository to host package browser websites on gh-pages. It uses the package-browser extension to generate HTML pages from a list of luet repositories | https://github.com/Luet-lab/package-browser-sample |
+| LineageOS builds with luet                                                                                                                                     | https://github.com/mudler/android-builds           |
+| Example repository template to build packages on github actions and push packages on a container registry                                                                                                                                     | https://github.com/Luet-lab/github-repository           |
+| Immutable container OS toolkit                                                                                                                                 | https://github.com/rancher-sandbox/cOS-toolkit     |
+| mocaccinoOS desktop                                                                                                                                 | https://github.com/mocaccinoOS/desktop     |
+| mocaccinoOS extra                                                                                                                                 | https://github.com/mocaccinoOS/mocaccino-extra     |
+| agregOS built with luet/mocaccino | https://interne.agreg.org/agregOS/ |

docs/content/en/docs/Resources/scratch.md

@@ -0,0 +1,38 @@
+---
+title: "Images from scratch"
+linkTitle: "ScratchImages"
+weight: 4
+description: >
+  Using Luet to compose images from scratch
+---
+
+The Docker image `quay.io/luet/base` is a `scratch` Docker image always kept up-to-date with the latest luet version. That image can be used to bootstrap new images with Luet repositories with the packages you want, from the repositories you prefer.
+
+For example we can mount a config file, and later on install a package:
+
+```bash
+cat <<EOF > $PWD/luet.yaml  
+repositories: 
+  - name: "micro-stable"
+    enable: true
+    cached: true
+    priority: 1
+    type: "http"
+    urls: 
+    - "https://get.mocaccino.org/mocaccino-micro-stable"
+EOF
+
+docker rm luet-runtime-test || true
+docker run --name luet-runtime-test \
+       -ti -v /tmp:/tmp \
+       -v $PWD/luet.yaml:/etc/luet/luet.yaml:ro \
+       quay.io/luet/base install shells/bash
+ 
+docker commit luet-runtime-test luet-runtime-test-image
+
+# Try your new image!
+
+docker run -ti --entrypoint /bin/bash --rm luet-runtime-test-image
+```
+
+In this way we will create a new image, with only `luet` and `bash`, and nothing else from a scratch image.

docs/content/en/docs/Tutorials/_index.md

@@ -0,0 +1,9 @@
+
+---
+title: "Tutorials"
+linkTitle: "Tutorials"
+weight: 8
+date: 2017-01-04
+description: >
+  Howtos, Cookbooks 
+---

docs/content/en/docs/Tutorials/build_package.md

@@ -0,0 +1,246 @@
+---
+title: "Build a package"
+linkTitle: "Build a package"
+weight: 3
+date: 2017-01-04
+description: >
+  Example on how to build a package and run it locally with luet box
+---
+
+{{< alert color="warning" title="Warning" >}}
+This article contains references to Luet repositories that were deprecated, and needs to be updated.
+Please refer to the ["Hello World"](../../tutorials/hello_world/) tutorial instead.
+{{< /alert >}}
+
+![catclock](https://github.com/BarkyTheDog/catclock/raw/master/catclock.gif)
+
+# Catclock example
+
+In this example, we will build the awesome [CatClock](https://github.com/BarkyTheDog/catclock) on containers we will run it locally in a Luet box. 
+
+We will do this experiment to prove two things: 
+1) how we can build a package with Luet and 
+2) two packages from different distributions can (sometime) work together.
+
+## Prerequisites
+
+To build packages with Luet, you must have installed Docker and container-diff, follow our [setup guide](../../getting-started).
+
+
+## 1) Create the package
+
+To prove our point, we will build our package from an OpenSUSE image, and later on we will consume
+entropy repositories for runtime dependencies. To note, this is not the main focus of Luet, and this is a restricted example on its features on build-time resolution.  For more syntax examples, see also [Build specs](../../concepts/specfile/#build-specs) and [Package types](../../concepts/packages/#package-types).
+
+
+Run this commands in any directory you choose to be your workspace:
+
+```bash
+
+# Let's create a directory to store our package spec:
+mkdir -p tree/misc/catclock/
+```
+
+### 1.1) Build spec
+
+Now, let's generate our **build** spec:
+
+```bash
+# Create a build file. We use here opensuse/leap to build the package, as an example
+cat <<EOF > tree/misc/catclock/build.yaml
+image: opensuse/leap
+
+# Preparation phase
+prelude:
+- zypper in -y git make libXt-devel xmh gcc motif-devel libXext-devel libpulse-devel libaubio-devel
+- git clone https://github.com/BarkyTheDog/catclock
+
+# Here we define the steps that Luet will follow
+steps:
+- cd catclock && make DEFINES="-Wno-incompatible-pointer-types"
+- mv catclock/xclock /usr/bin/xclock
+
+# (optional) File list that will be included in the final package
+# Luet will filter out files that won't match any entry in the list (regex syntax IS supported)
+includes:
+- /usr/bin/xclock
+EOF
+
+```
+
+`build.yaml` is what an ebuild is for Gentoo and for e.g. what PKGBUILD is for Arch.
+
+- *image: opensuse/leap* tells luet to use opensuse/leap as a build image. We collect the build time dependencies with `zypper` (the openSUSE package manager), and the [CatClock](https://github.com/BarkyTheDog/catclock) with `git`. When we declare an `image` keyword in a spec, it becomes a *seed* package ( [Package types](../../concepts/packages/#package-types) ) as doesn't depend on any package in build time, we will cover more use cases in other examples.
+- *prelude* is a list of commands that will happen during the build phase.
+They might generate binaries, or download sources, but those are not took into consideration when generating the final package.
+- *steps* is a list of commands that will happen during the build phase.
+Luet will execute those commands and all the binaries generated from them become part of the final package
+- *includes* is a (optional) list of regex that tells to Luet what files to filter out from the final artifact.
+
+### 1.2) Runtime spec
+
+Now we generate the runtime spec, it's the part about the binary end which will be installed in the system. It also holds the metadata relative to the package definition (`name`, `category`, `version`).
+
+```bash
+# Create a runtime definition.
+# We will leverage packages already present on Sabayon Entropy repositories
+# the end-system needs to have the Luet Sabayon Entropy repositories enabled.
+cat <<EOF > tree/misc/catclock/definition.yaml
+category: "misc"
+name: "catclock"
+version: "0.20200318"
+requires:
+- category: meta
+  name: users
+  version: ">=0"
+- category: x11-libs
+  name: motif
+  version: ">=0.1"
+- category: media-libs
+  name: libjpeg-turbo
+  version: ">=0.1"
+EOF
+```
+
+- *category*, *name*, and *version*: identifies the package in a Luet tree. This is the unique identifier for a package.
+- *requires* it's a list of packages which our **catclock** depends on during runtime (when we will execute catclock inside a small-container!). To find out what's required by your binaries it can be a try-learn-fail effort. If the package you wish to build is specifying the deps it requires, and those are available in a Luet repository, you are all set, just point them there. Otherwise you have to figure out after you build the binary the first time (for example, with `ldd`) to which libraries it depends on.
+In this example we consume the dependencies from the [Luet Entropy Repo](https://github.com/Luet-lab/luet-entropy-repo), that we will enable on the following steps.
+
+## 2) Build it!
+
+```bash
+sudo /usr/bin/luet build \
+--tree=$PWD/tree misc/catclock \
+--destination $PWD/build \
+--compression gzip
+
+sudo chown -R $USER $PWD/build # So later on, we can access to the repository with our user
+```
+
+We are building the specs in this step.
+
+- *tree*: is the path where our specs are, in our case it's `tree`.
+- *destination*: is the path where our packages will be stored, in our case this is `build`.
+- *compression*: is the compression algorithm used to compress the final artifacts
+
+Note, we need *sudo* to keep the permissions properly mapped in the artifact which is produced
+this is not always the case. Depends on the package content.
+
+
+## 3) Create a local repository
+
+We will generate now our repository metadata:
+```bash
+/usr/bin/luet create-repo --tree "tree" \
+--output $PWD/build \
+--packages $PWD/build \
+--name "test repo" \
+--descr "Test Repo" \
+--tree-compression gzip \
+--meta-compression gzip
+```
+
+Creating a repository in Luet is about adding metadata and make our spec tree available to other systems running Luet to intall the package. 
+
+- **output**: a path which is where Luet will store the repository metadata.
+- **packages**: a path containing the packages that were built during the build step
+- **name**: Repository name
+- **descr**: Repository description 
+- **tree-compression**: optional, algorithm to use when compression the tree metadata
+- **meta-compression**: optional, algorithm to use when compression the repository metadata
+
+## 4) Let's test it!
+
+Now we are all set. We have the packages compiled, and we are ready to consume them. We don't want to break our host system, and we want to test this from our user.
+
+Let's create a directory, we will try to setup a full running system, and install everything there.
+
+```bash
+# Let's create a directory for our "fake" rootfilesystem
+# it will be populated with a minimal set of packages needed to run 
+# our amazing catclock
+mkdir -p $PWD/rootfs
+```
+
+```bash
+# Let's also create a directory to store our config files
+mkdir -p $PWD/conf
+```
+
+We will generate now a Luet config. The Luet config is used to read where install things from, and in which directory.
+It also lists the repositories that are used by the client to retrieve packages remotely.
+
+```bash
+# We create here a config file which references the rootfs.
+# In this way, luet instead installing packages to your host system, will populate the rootfs
+# (note, all the steps are run by a user here, no root required!)
+cat <<EOF > conf/luet-dso-local.yaml
+system:
+  rootfs: $PWD/rootfs # our "fake" rootfs that we created before
+  database_path: "/" # this is where our Luet DB will live
+  database_engine: "boltdb" # this is the Luet DB engine
+repositories:
+   - name: "main"
+     type: "disk"
+     priority: 3
+     enable: true
+     urls:
+       - "$PWD/build" # This is the repository we have created before!
+   - name: "sabayonlinux.org"
+     description: "Sabayon Linux Repository"
+     type: "http"
+     enable: true
+     cached: true
+     priority: 2
+     urls:
+       - "https://dispatcher.sabayon.org/sbi/namespace/luet-entropy-repo"
+   - name: "luet-repo"
+     description: "Luet Official Repository"
+     type: "http"
+     enable: true
+     cached: true
+     priority: 1
+     urls:
+       - "https://raw.githubusercontent.com/Luet-lab/luet-repo/gh-pages"
+EOF
+# we have specified an additional repository, one that is luet-entropy-repo (which contains
+# the runtime dependencies we specified in our package)
+```
+
+```bash
+# Let's populate our rootfs with some minimal things: base-gcc, and bash
+# meta/users is a meta package providing minimal base to run things with a full
+# user-level support.
+export LUET_NOLOCK=true
+luet install \
+--config $PWD/conf/luet-dso-local.yaml \
+meta/users
+```
+
+```bash
+# catclock is a X11 app! we want to be able to play with it locally from our host :)
+# Let's copy the .Xauthority file to allow the X app to communicate with our X server
+# Note: This can be achieved in other ways (set up a tcp X server, and so on)
+cp -rfv $HOME/.Xauthority $PWD/rootfs/                                                        
+```
+
+```bash
+luet install \
+--config $PWD/conf/luet-dso-local.yaml \
+misc/catclock
+```
+
+```bash
+# Let's run our beautiful catclock :)
+luet box exec --rootfs $PWD/rootfs \
+--stdin --stdout --stderr --env DISPLAY=$DISPLAY \
+--env XAUTHORITY=/.Xauthority --mount /tmp --entrypoint /usr/bin/xclock
+```
+
+Spawn a bash shell inside our box (with permission to access to our running X):
+
+```bash
+luet box exec --rootfs $PWD/rootfs \
+--stdin --stdout --stderr --env DISPLAY=$DISPLAY \
+--env XAUTHORITY=/.Xauthority --mount /tmp --entrypoint /bin/bash
+```

docs/content/en/docs/Tutorials/hello_world.md

@@ -0,0 +1,147 @@
+---
+title: "Hello world!"
+linkTitle: "Hello world!"
+weight: 1
+date: 2017-01-04
+description: >
+  Everything starts from an "Hello!"
+---
+
+This article will guide you to build your first package with Luet! 
+For this purpose, we have picked a real-world example: [gogs](https://github.com/gogs/gogs) which is a "painless self-hosted Git service", an open-source alternative to Github.
+
+Gogs is written in Golang, and we need a working Golang version in order to build it.
+
+Here you can see a live recorded session of this tutorial:
+<script id="asciicast-388348" src="https://asciinema.org/a/388348.js" data-autoplay="true" data-size="small" data-cols="120" data-rows="40" async></script>
+
+
+# Define a Luet tree
+
+Everything starts from a Luet tree. A Luet tree is just a directory containing one (or more) Luet specfile, here on we assume that you are working in a dedicated folder (e.g. `~/demo`) in your system.
+
+Let's create then a package that will be our base to build other packages from now on, we have picked `busybox` here - it is really small and enough for our purpose.
+
+## busybox
+
+```bash
+mkdir busybox
+```
+
+Let's now write the build specification, which is just containing the image tag that we are referencing to
+
+```bash
+cat <<EOF > busybox/build.yaml
+image: "busybox:{{.Values.version}}-glibc"
+EOF
+```
+
+Now, lets write the `definition.yaml`, which contains the metadata information about our package ( e.g. how we refer to it with luet, the version, and so on )
+
+```bash
+cat <<EOF > busybox/definition.yaml
+category: "distro"
+name: "busybox"
+version: "1.33.0"
+EOF
+```
+
+## golang
+
+We need now golang in order to build `gogs`. Let's declare then a golang package:
+
+```bash
+mkdir golang
+```
+
+And a build specfile, which is simply fetch golang from https://golang.org and installing it in the busybox container:
+```bash
+cat <<EOF > golang/build.yaml
+requires:
+- category: "distro"
+  name: "busybox"
+  version: ">=0"
+
+prelude:
+- wget https://golang.org/dl/go{{.Values.version}}.linux-{{.Values.arch}}.tar.gz -O golang.tar.gz
+- mkdir /usr/local
+steps:
+- tar -C /usr/local -xzf golang.tar.gz
+EOF
+```
+
+Note how we `require` busybox. The Golang container will now be based from busybox, and the `prelude` and `steps` fields will be executed in that context.
+
+And finally let's write the golang metadata files, so we can refer to it from other packages
+```bash
+cat <<EOF > golang/definition.yaml
+name: "go"
+category: "dev-lang"
+version: "1.15.6"
+arch: "amd64"
+EOF
+```
+
+## gogs
+
+Finally we can write the gogs package definition!
+
+```bash
+mkdir gogs
+```
+
+The build specfile, will just fetch the `gogs` sources at a given version (specified in the `definition.yaml`) and build the sources with go:
+
+```bash
+cat <<'EOF' > gogs/build.yaml
+requires:
+- category: "dev-lang"
+  name: "go"
+  version: ">=0"
+env:
+- GOPATH="/go"
+- GOGSPATH="$GOPATH/src/github.com/gogs/gogs"
+- PATH=$PATH:/usr/local/go/bin
+- CGO_ENABLED=0
+prelude:
+- mkdir -p $GOPATH/src/github.com/gogs
+- wget https://github.com/gogs/gogs/archive/v{{.Values.version}}.tar.gz -O - | tar -xzf - -C ./  && mv gogs-{{.Values.version}} $GOGSPATH
+steps:
+- mkdir /usr/bin
+- cd $GOGSPATH && go build && mv gogs /usr/bin/gogs
+excludes:
+# Cache generated by Golang
+- ^/root
+EOF
+```
+
+And the metadata, in this way we can refer to gogs in a Luet tree:
+```bash
+cat <<EOF > gogs/definition.yaml
+category: "dev-vcs"
+name: "gogs"
+version: "0.11.91"
+EOF
+```
+
+# Build packages
+
+The simplest and mostly immediate way to build packages, is running `luet build <packagename>` in the same folder you have your Luet tree.
+
+In this case, to build gogs and its deps, we can do:
+
+```bash
+luet build dev-vcs/gogs
+```
+
+And that's it! you will find the package archives in `build/` in the same folder where you started the command.
+
+You will see that Luet generates not only archives with the file resulting to your builds, but it will also generate metadata files (ending with `.metadata.yaml`) that contains additional metadata information about your build and the package itself (e.g. checksums).
+
+You can use tools like [yq](https://github.com/mikefarah/yq) to inspect those:
+
+```bash
+yq r build/gogs-dev-vcs-0.11.91.metadata.yaml checksums
+```
+
+Now if you want to consume the artifacts just built with `luet install`, you can create a repository with `luet create-repo`.