1
0
mirror of https://github.com/containers/skopeo.git synced 2025-05-11 01:15:29 +00:00

Multi-arch github-action workflow unification

This is a port from the podman and buildah repository workflows.
It's purpose is to build and push multi-arch images containing the
latest upstream, testing, and stable versions of skopeo.  It fully
replaces the last remaining use of Travis in this repo, for
substantially the same purpose.

In a future commit, I intend to de-duplicate this workflow from
podman and buildah, such that all three share a common set of details.
Until then, any changes will need to be manually duplicated across
all three repos.

Signed-off-by: Chris Evich <cevich@redhat.com>
This commit is contained in:
Chris Evich 2021-05-17 11:51:55 -04:00
parent 5af5f8a0e7
commit 6452a9b6f6
No known key found for this signature in database
GPG Key ID: 03EDC70FD578067F
5 changed files with 231 additions and 178 deletions
.github/workflow
.travis.yml
contrib/skopeoimage
release

206
.github/workflow/multi-arch-build.yaml vendored Normal file
View File

@ -0,0 +1,206 @@
---
# Please see contrib/skopeoimage/README.md for details on the intentions
# of this workflow.
#
# BIG FAT WARNING: This workflow is duplicated across containers/skopeo,
# containers/buildah, and containers/podman. ANY AND
# ALL CHANGES MADE HERE MUST BE MANUALLY DUPLICATED
# TO THE OTHER REPOS.
name: build multi-arch images
on:
# Upstream skopeo tends to be very active, with many merges per day.
# Only run this daily via cron schedule, or manually, not by branch push.
schedule:
- cron: '0 8 * * *'
# allows to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
multi:
name: multi-arch Skopeo build
env:
SKOPEO_QUAY_REGISTRY: quay.io/skopeo
CONTAINERS_QUAY_REGISTRY: quay.io/containers
# list of architectures for build
PLATFORMS: linux/amd64,linux/s390x,linux/ppc64le,linux/arm64
# build several images (upstream, testing, stable) in parallel
strategy:
# By default, failure of one matrix item cancels all others
fail-fast: false
matrix:
# Builds are located under contrib/skopeoimage/<source> directory
source:
- upstream
- testing
- stable
runs-on: ubuntu-latest
# internal registry caches build for inspection before push
services:
registry:
image: quay.io/libpod/registry:2
ports:
- 5000:5000
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
with:
driver-opts: network=host
install: true
- name: Build and locally push Skopeo
uses: docker/build-push-action@v2
with:
context: contrib/skopeoimage/${{ matrix.source }}
file: ./contrib/skopeoimage/${{ matrix.source }}/Dockerfile
platforms: ${{ env.PLATFORMS }}
push: true
tags: localhost:5000/skopeo/${{ matrix.source }}
# Simple verification that stable images work, and
# also grab version number use in forming the FQIN.
- name: amd64 container sniff test
if: matrix.source == 'stable'
id: sniff_test
run: |
VERSION_OUTPUT="$(docker run localhost:5000/skopeo/${{ matrix.source }} \
skopeo --storage-driver=vfs version)"
echo "$VERSION_OUTPUT"
VERSION=$(grep -Em1 '^Version: ' <<<"$VERSION_OUTPUT" | awk '{print $2}')
test -n "$VERSION"
echo "::set-output name=version::${VERSION}"
- name: Generate skopeo reg. image FQIN(s)
id: skopeo_reg
run: |
if [[ "${{ matrix.source }}" == 'stable' ]]; then
# The `skopeo version` in image just built
VERSION='v${{ steps.sniff_test.outputs.version }}'
# workaround vim syntax-highlight bug: '
# Image tags previously pushed to quay
ALLTAGS=$(skopeo list-tags \
docker://$SKOPEO_QUAY_REGISTRY/stable | \
jq -r '.Tags[]')
# New version? Push quay.io/skopeo/stable:vX.X.X and :latest
if ! fgrep -qx "$VERSION" <<<"$ALLTAGS"; then
# Assume version-tag is also the most up to date (i.e. "latest")
FQIN="$SKOPEO_QUAY_REGISTRY/stable:$VERSION,$SKOPEO_QUAY_REGISTRY/stable:latest"
else # Not a new version-tagged image
# Assume other contents changed, so this is the "new" latest.
FQIN="$SKOPEO_QUAY_REGISTRY/stable:latest"
fi
elif [[ "${{ matrix.source }}" == 'testing' ]]; then
# Assume some contents changed, always push latest testing.
FQIN="$SKOPEO_QUAY_REGISTRY/testing:latest"
elif [[ "${{ matrix.source }}" == 'upstream' ]]; then
# Assume some contents changed, always push latest upstream.
FQIN="$SKOPEO_QUAY_REGISTRY/upstream:latest"
else
echo "::error::Unknown matrix item '${{ matrix.source }}'"
exit 1
fi
echo "::warning::Pushing $FQIN"
echo "::set-output name=fqin::${FQIN}"
echo '::set-output name=push::true'
# This is substantially similar to the above logic,
# but only handles $CONTAINERS_QUAY_REGISTRY for
# the stable "latest" and named-version tagged images.
- name: Generate containers reg. image FQIN(s)
if: matrix.source == 'stable'
id: containers_reg
run: |
VERSION='v${{ steps.sniff_test.outputs.version }}'
# workaround vim syntax-highlight bug: '
ALLTAGS=$(skopeo list-tags \
docker://$CONTAINERS_QUAY_REGISTRY/skopeo | \
jq -r '.Tags[]')
# New version? Push quay.io/containers/skopeo:vX.X.X and latest
if ! fgrep -qx "$VERSION" <<<"$ALLTAGS"; then
FQIN="$CONTAINERS_QUAY_REGISTRY/skopeo:$VERSION,$CONTAINERS_QUAY_REGISTRY/skopeo:latest"
else # Not a new version-tagged image, only update latest.
FQIN="$CONTAINERS_QUAY_REGISTRY/skopeo:latest"
fi
echo "::warning::Pushing $FQIN"
echo "::set-output name=fqin::${FQIN}"
echo '::set-output name=push::true'
- name: Define LABELS multi-line env. var. value
run: |
# This is a really hacky/strange workflow idiom, required
# for setting multi-line $LABELS value for consumption in
# a future step.
# https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#multiline-strings
cat << EOF | tee -a $GITHUB_ENV
LABELS<<DELIMITER
org.opencontainers.image.source=https://github.com/${{ github.repository }}.git
org.opencontainers.image.revision=${{ github.sha }}
org.opencontainers.image.created=$(date -u --iso-8601=seconds)
DELIMITER
EOF
# Separate steps to login and push for skopeo and containers quay
# repositories are required, because 2 sets of credentials are used and `docker
# login` as well as `skopeo login` do not support having 2 different
# credential sets for 1 registry.
# At the same time reuse of non-shell steps is not supported by Github Actions
# via anchors or composite actions
# Push to 'skopeo' Quay repo for stable, testing. and upstream
- name: Login to 'skopeo' Quay registry
uses: docker/login-action@v1
if: steps.skopeo_reg.outputs.push == 'true'
with:
registry: ${{ env.SKOPEO_QUAY_REGISTRY }}
# N/B: Secrets are not passed to workflows that are triggered
# by a pull request from a fork
username: ${{ secrets.SKOPEO_QUAY_USERNAME }}
password: ${{ secrets.SKOPEO_QUAY_PASSWORD }}
- name: Push images to 'skopeo' Quay
uses: docker/build-push-action@v2
if: steps.skopeo_reg.outputs.push == 'true'
with:
cache-from: type=registry,ref=localhost:5000/skopeo/${{ matrix.source }}
cache-to: type=inline
context: contrib/skopeoimage/${{ matrix.source }}
file: ./contrib/skopeoimage/${{ matrix.source }}/Dockerfile
platforms: ${{ env.PLATFORMS }}
push: true
tags: ${{ steps.skopeo_reg.outputs.fqin }}
labels: |
${{ env.LABELS }}
# Push to 'containers' Quay repo only stable skopeo
- name: Login to 'containers' Quay registry
if: steps.containers_reg.outputs.push == 'true'
uses: docker/login-action@v1
with:
registry: ${{ env.CONTAINERS_QUAY_REGISTRY}}
username: ${{ secrets.CONTAINERS_QUAY_USERNAME }}
password: ${{ secrets.CONTAINERS_QUAY_PASSWORD }}
- name: Push images to 'containers' Quay
if: steps.containers_reg.outputs.push == 'true'
uses: docker/build-push-action@v2
with:
cache-from: type=registry,ref=localhost:5000/skopeo/${{ matrix.source }}
cache-to: type=inline
context: contrib/skopeoimage/${{ matrix.source }}
file: ./contrib/skopeoimage/${{ matrix.source }}/Dockerfile
platforms: ${{ env.PLATFORMS }}
push: true
tags: ${{ steps.containers_reg.outputs.fqin }}
labels: |
${{ env.LABELS }}

View File

@ -1,75 +0,0 @@
language: go
go:
- 1.15.x
notifications:
email: false
env:
global:
# Multiarch manifest will support architectures from this list. It should be the same architectures, as ones in image-build-push step in this Travis config
- MULTIARCH_MANIFEST_ARCHITECTURES="amd64 s390x ppc64le"
# env variables for stable image build
- STABLE_IMAGE=quay.io/skopeo/stable:v1.2.0
- EXTRA_STABLE_IMAGE=quay.io/containers/skopeo:v1.2.0
# env variable for upstream image build
- UPSTREAM_IMAGE=quay.io/skopeo/upstream:master
# Just declaration of the image-build-push step with script actions to execute
x_base_steps:
- &image-build-push
services:
- docker
os: linux
dist: focal
script:
# skopeo upstream image build
- make -f release/Makefile build-image/upstream
# Push image in case if build is started via cron job or code is pushed to master
- if [ "$TRAVIS_EVENT_TYPE" == "push" ] || [ "$TRAVIS_EVENT_TYPE" == "cron" ]; then
make -f release/Makefile push-image/upstream ;
fi
# skopeo stable image build
- make -f release/Makefile build-image/stable
# Push image in case if build is started via cron job or code is pushed to master
- if [ "$TRAVIS_EVENT_TYPE" == "push" ] || [ "$TRAVIS_EVENT_TYPE" == "cron" ]; then
make -f release/Makefile push-image/stable ;
fi
# Just declaration of stage order to run
stages:
# Build and push image for 1 architecture
- name: image-build-push
if: branch = master
# Create and push image manifest to have multiarch image on top of architecture specific images
- name: manifest-multiarch-push
if: (type IN (push, cron)) AND branch = master
# Actual execution of steps
jobs:
include:
# Run 3 image-build-push tasks in parallel for linux/amd64, linux/s390x and linux/ppc64le platforms (for upstream and stable)
- stage: image-build-push
<<: *image-build-push
name: images for amd64
arch: amd64
- stage: image-build-push
<<: *image-build-push
name: images for s390x
arch: s390x
- stage: image-build-push
<<: *image-build-push
name: images for ppc64le
arch: ppc64le
# Run final task to generate multi-arch image manifests (for upstream and stable)
- stage: manifest-multiarch-push
os: linux
dist: focal
script:
- make -f release/Makefile push-manifest-multiarch/upstream
- make -f release/Makefile push-manifest-multiarch/stable

View File

@ -6,22 +6,35 @@
## Overview
This directory contains the Dockerfiles necessary to create the three skopeoimage container
images that are housed on quay.io under the skopeo account. All three repositories where
the images live are public and can be pulled without credentials. These container images
are secured and the resulting containers can run safely. The container images are built
using the latest Fedora and then Skopeo is installed into them:
This directory contains the Dockerfiles necessary to create the skopeoimage container
images that are housed on quay.io under the skopeo account. All repositories where
the images live are public and can be pulled without credentials. These container images are secured and the
resulting containers can run safely with privileges within the container.
* quay.io/skopeo/stable - This image is built using the latest stable version of Skopeo in a Fedora based container. Built with skopeoimage/stable/Dockerfile.
* quay.io/skopeo/upstream - This image is built using the latest code found in this GitHub repository. When someone creates a commit and pushes it, the image is created. Due to that the image changes frequently and is not guaranteed to be stable. Built with skopeoimage/upstream/Dockerfile.
* quay.io/skopeo/testing - This image is built using the latest version of Skopeo that is or was in updates testing for Fedora. At times this may be the same as the stable image. This container image will primarily be used by the development teams for verification testing when a new package is created. Built with skopeoimage/testing/Dockerfile.
The container images are built using the latest Fedora and then Skopeo is installed into them.
The PATH in the container images is set to the default PATH provided by Fedora. Also, the
ENTRYPOINT and the WORKDIR variables are not set within these container images, as such they
default to `/`.
## Multiarch images
The container images are:
Multiarch images are available for Skopeo upstream and stable versions. Supported architectures are `amd64`, `s390x`, `ppc64le`.
Available images are `quay.io/skopeo/upstream:master`, `quay.io/skopeo/stable:v1.2.0`, `quay.io/containers/skopeo:v1.2.0`.
* `quay.io/containers/skopeo:<version>` and `quay.io/skopeo/stable:<version>` -
These images are built when a new Skopeo version becomes available in
Fedora. These images are intended to be unchanging and stable, they will
never be updated by automation once they've been pushed. For build details,
please [see the configuration file](stable/Dockerfile).
* `quay.io/containers/skopeo:latest` and `quay.io/skopeo/stable:latest` -
Built daily using the same Dockerfile as above. The skopeo version
will remain the "latest" available in Fedora, however the image
contents may vary compared to the version-tagged images.
* `quay.io/skopeo/testing:latest` - This image is built daily, using the
latest version of Skopeo that was in the Fedora `updates-testing` repository.
The image is Built with [the testing Dockerfile](testing/Dockerfile).
* `quay.io/skopeo/upstream:latest` - This image is built daily using the latest
code found in this GitHub repository. Due to the image changing frequently,
it's not guaranteed to be stable or even executable. The image is built with
[the upstream Dockerfile](upstream/Dockerfile).
Images can be used the same way as in a single architecture case, no extra setup is required. For samples see next chapter.
## Sample Usage

View File

@ -1,51 +0,0 @@
## This Makefile is used to publish skopeo container images with Travis CI ##
## Environment variables, used in this Makefile are specified in .travis.yml
export DOCKER_CLI_EXPERIMENTAL=enabled
GOARCH ?= $(shell go env GOARCH)
# Dereference variable $(1), return value if non-empty, otherwise raise an error.
err_if_empty = $(if $(strip $($(1))),$(strip $($(1))),$(error Required $(1) variable is undefined or empty))
# Requires two arguments: Names of the username and the password env. vars.
define quay_login
@echo "$(call err_if_empty,$(2))" | \
docker login quay.io -u "$(call err_if_empty,$(1))" --password-stdin
endef
# Build container image of skopeo upstream based on host architecture
build-image/upstream:
docker build -t "${UPSTREAM_IMAGE}-${GOARCH}" contrib/skopeoimage/upstream
# Build container image of skopeo stable based on host architecture
build-image/stable:
docker build -t "${STABLE_IMAGE}-${GOARCH}" contrib/skopeoimage/stable
# Push container image of skopeo upstream (based on host architecture) to image repository
push-image/upstream:
$(call quay_login,SKOPEO_QUAY_USERNAME,SKOPEO_QUAY_PASSWORD)
docker push "${UPSTREAM_IMAGE}-${GOARCH}"
# Push container image of skopeo stable (based on host architecture) to image default and extra repositories
push-image/stable:
$(call quay_login,SKOPEO_QUAY_USERNAME,SKOPEO_QUAY_PASSWORD)
docker push "${STABLE_IMAGE}-${GOARCH}"
docker tag "${STABLE_IMAGE}-${GOARCH}" "${EXTRA_STABLE_IMAGE}-${GOARCH}"
$(call quay_login,CONTAINERS_QUAY_USERNAME,CONTAINERS_QUAY_PASSWORD)
docker push "${EXTRA_STABLE_IMAGE}-${GOARCH}"
# Create and push multiarch image manifest of skopeo upstream
push-manifest-multiarch/upstream:
docker manifest create "${UPSTREAM_IMAGE}" $(foreach arch,${MULTIARCH_MANIFEST_ARCHITECTURES}, ${UPSTREAM_IMAGE}-${arch})
$(call quay_login,SKOPEO_QUAY_USERNAME,SKOPEO_QUAY_PASSWORD)
docker manifest push --purge "${UPSTREAM_IMAGE}"
# Create and push multiarch image manifest of skopeo stable
push-manifest-multiarch/stable:
docker manifest create "${STABLE_IMAGE}" $(foreach arch,${MULTIARCH_MANIFEST_ARCHITECTURES}, ${STABLE_IMAGE}-${arch})
$(call quay_login,SKOPEO_QUAY_USERNAME,SKOPEO_QUAY_PASSWORD)
docker manifest push --purge "${STABLE_IMAGE}"
# Push to extra repository
docker manifest create "${EXTRA_STABLE_IMAGE}" $(foreach arch,${MULTIARCH_MANIFEST_ARCHITECTURES}, ${EXTRA_STABLE_IMAGE}-${arch})
$(call quay_login,CONTAINERS_QUAY_USERNAME,CONTAINERS_QUAY_PASSWORD)
docker manifest push --purge "${EXTRA_STABLE_IMAGE}"

View File

@ -1,40 +0,0 @@
# skopeo container image build with Travis
This document describes the details and requirements to build and publish skopeo container images. The images are published as several architecture specific images and multiarch images on top for upstream and stable versions.
The Travis configuration is available at `.travis.yml`.
The code to build and publish images is available at `release/Makefile` and should be used only via Travis.
Travis workflow has 3 major pieces:
- `local-build` - build and test source code locally on osx and linux/amd64 environments, 2 jobs are running in parallel
- `image-build-push` - build and push container images with several Travis jobs running in parallel to build images for several architectures (linux/amd64, linux/s390x, linux/ppc64le). Build part is done for each PR, push part is executed only in case of cron job or master branch update.
- `manifest-multiarch-push` - create and push image manifests, which consists of architecture specific images from previous step. Executed only in case of cron job or master branch update.
## Ways to have full workflow run
- [cron job](https://docs.travis-ci.com/user/cron-jobs/#adding-cron-jobs)
- Trigger build from Travis CI
- Update code in master branch
## Environment variables
Several environment variables are used to customize image names and keep private credentials to push to quay.io repositories.
**Image tags** are specified in environment variable and should be manually updated in case of new release.
- `SKOPEO_QUAY_USERNAME` and `SKOPEO_QUAY_PASSWORD` are credentials to push images to `quay.io/skopeo/stable` and `quay.io/skopeo/upstream` repos, and require the credentials to have write permissions. These variables should be specified in [Travis](https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings).
- `CONTAINERS_QUAY_USERNAME` and `CONTAINERS_QUAY_PASSWORD` are credentials to push images to `quay.io/containers/skopeo` repos, and require the credentials to have write permissions. These variables should be specified in [Travis](https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings).
Variables in .travis.yml
- `MULTIARCH_MANIFEST_ARCHITECTURES` is a list with architecture shortnames, to appear in final multiarch manifest. The values should fit to architectures used in the `image-build-push` Travis step.
- `STABLE_IMAGE`, `EXTRA_STABLE_IMAGE` are image names to publish stable Skopeo.
- `UPSTREAM_IMAGE` is an image name to publish upstream Skopeo.
### Values for environment variables
| Env variable | Value |
| -------------------------------- |----------------------------------|
| MULTIARCH_MANIFEST_ARCHITECTURES | "amd64 s390x ppc64le" |
| STABLE_IMAGE | quay.io/skopeo/stable:v1.2.0 |
| EXTRA_STABLE_IMAGE | quay.io/containers/skopeo:v1.2.0 |
| UPSTREAM_IMAGE | quay.io/skopeo/upstream:master |