10 Commits

Author SHA1 Message Date
Avi Deitcher
dc8c6d5985 Merge pull request #4089 from deitch/tag-in-build-yml
support --tag in build.yml for packages
2024-12-23 18:00:06 +02:00
Avi Deitcher
4f765b5da0 support --tag in build.yml for packages
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-12-23 17:28:49 +02:00
Avi Deitcher
ad95c6fc2e Merge pull request #4085 from deitch/volume-image
additional volume support in building
2024-10-01 15:57:17 +03:00
Avi Deitcher
76f4802ccf additional volume support in building
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-10-01 15:27:55 +03:00
Avi Deitcher
e4d41061b6 Merge pull request #4084 from deitch/cache-platform-instead-of-arch
internal restructure to use explicit platform instead of implicit arch in cache
2024-10-01 15:14:21 +03:00
Avi Deitcher
81f0c3eff2 internal restructure to use explicit platform instead of implicit arch in cache
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-10-01 14:30:03 +03:00
Avi Deitcher
5e3f7dd9a5 Merge pull request #4083 from deitch/restructure-logging
restructure logging
2024-10-01 14:00:06 +03:00
Avi Deitcher
67e9e22a36 restructure logging
Signed-off-by: Avi Deitcher <avi@deitcher.net>
2024-10-01 12:50:43 +03:00
Avi Deitcher
8556f024ef Merge pull request #4082 from kolyshkin/moby-cap
vendor: switch to moby/sys/capability
2024-10-01 11:07:29 +03:00
Kir Kolyshkin
da3be29998 vendor: switch to moby/sys/capability
github.com/moby/sys/capability is a fork of the (no longer maintained)
github.com/syndtr/gocapability package.

For changes since the fork took place, see
https://github.com/moby/sys/blob/main/capability/CHANGELOG.md

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2024-09-30 18:10:16 -07:00
51 changed files with 1242 additions and 592 deletions

View File

@@ -122,13 +122,13 @@ jobs:
- name: Build Packages
# Skip s390x as emulation is unreliable
run: |
make OPTIONS="-v --skip-platforms linux/s390x" -C pkg build
make OPTIONS="-v 2 --skip-platforms linux/s390x" -C pkg build
- name: Build Test Packages
# ensures that the test packages are in linuxkit cache when we need them for tests later
# Skip s390x as emulation is unreliable
run: |
make OPTIONS="-v --skip-platforms linux/s390x" -C test/pkg build
make OPTIONS="-v 2 --skip-platforms linux/s390x" -C test/pkg build
- name: Check Kernel Dependencies up to date
# checks that any kernel dependencies are up to date.
@@ -145,7 +145,7 @@ jobs:
# ensures that the kernel packages are in linuxkit cache when we need them for tests later
# no need for excluding s390x, as each build.yml in the kernel explicitly lists archs
run: |
make OPTIONS="-v" -C kernel build
make OPTIONS="-v 2" -C kernel build
- name: list cache contents
run: |

View File

@@ -50,6 +50,7 @@ A package source consists of a directory containing at least two files:
- `image` _(string)_: *(mandatory)* The name of the image to build
- `org` _(string)_: The hub/registry organisation to which this package belongs
- `tag` _(string)_: The tag to use for the image, can be fixed string or template (default: `{{.Hash}}`)
- `dockerfile` _(string)_: The dockerfile to use to build this package, must be in this directory or below (default: `Dockerfile`)
- `arches` _(list of string)_: The architectures which this package should be built for (valid entries are `GOARCH` names)
- `extra-sources` _(list of strings)_: Additional sources for the package outside the package directory. The format is `src:dst`, where `src` can be relative to the package directory and `dst` is the destination in the build context. This is useful for sharing files, such as vendored go code, between packages.

View File

@@ -18,8 +18,17 @@ For private registries or private repositories on a registry credentials provide
## Sections
The configuration file is processed in the order `kernel`, `init`, `onboot`, `onshutdown`,
`services`, `files`, `volumes`. Each section adds files to the root file system. Sections may be omitted.
The configuration file is processed in the order:
1. `kernel`
1. `init`
1. `volumes`
1. `onboot`
1. `onshutdown`
1. `services`
1. `files`
Each section adds files to the root file system. Sections may be omitted.
Each container that is specified is allocated a unique `uid` and `gid` that it may use if it
wishes to run as an isolated user (or user namespace). Anywhere you specify a `uid` or `gid`
@@ -100,8 +109,13 @@ including those in `services`, `onboot` and `onshutdown`. The volumes are create
chosen by linuxkit at build-time. The volumes then can be referenced by other containers and
mounted into them.
Volumes normally are blank directories. If an image is provided, the contents of that image
will be used to populate the volume.
Volumes can be in one of several formats:
* Blank directory: This is the default, and is an empty directory that is created at build-time. It is an overlayfs mount, and can be shared among multiple containers.
* Image laid out as filesystem: The contents of the image are used to populate the volume. Default format when an image is provided.
* Image as OCI v1-layout: The image is used as an [OCI v1-layout](https://github.com/opencontainers/image-spec/blob/main/image-layout.md). Indicated by `format: oci`.
Examples of each are given later in this section.
The `volumes` section can declare a volume to be read-write or read-only. If the volume is read-write,
a volume that is mounted into a container can be mounted read-only or read-write. If the volume is read-only,
@@ -111,7 +125,36 @@ By default, volumes are created read-write, and are mounted read-write.
Volume names **must** be unique, and must contain only lower-case alphanumeric characters, hyphens, and
underscores.
Sample `volumes` section:
#### Samples of `volumes`
##### Empty directory
Yaml showing both read-only and read-write:
```yml
volumes:
- name: dira
readonly: true
- name: dirb
readonly: true
```
Contents:
```sh
$ cd dir && ls -la
drwxr-xr-x 19 root wheel 608 Sep 30 15:03 .
drwxrwxrwt 130 root wheel 4160 Sep 30 15:03 ..
```
In the above example:
* `dira` is empty and is read-only.
* `volb` is empty and is read-write.
##### Image directory
Yaml showing both read-only and read-write:
```yml
volumes:
@@ -120,8 +163,7 @@ volumes:
readonly: true
- name: volb
image: alpine:latest
readonly: false
- name: volc
format: filesystem # optional, as this is the default format
readonly: false
```
@@ -129,7 +171,56 @@ In the above example:
* `vola` is populated by the contents of `alpine:latest` and is read-only.
* `volb` is populated by the contents of `alpine:latest` and is read-write.
* `volc` is an empty volume and is read-write.
Contents:
```sh
$ cd dir && ls -la
drwxr-xr-x 19 root wheel 608 Sep 30 15:03 .
drwxrwxrwt 130 root wheel 4160 Sep 30 15:03 ..
drwxr-xr-x 84 root wheel 2688 Sep 6 14:34 bin
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 dev
drwxr-xr-x 37 root wheel 1184 Sep 6 14:34 etc
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 home
drwxr-xr-x 13 root wheel 416 Sep 6 14:34 lib
drwxr-xr-x 5 root wheel 160 Sep 6 14:34 media
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 mnt
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 opt
dr-xr-xr-x 2 root wheel 64 Sep 6 14:34 proc
drwx------ 2 root wheel 64 Sep 6 14:34 root
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 run
drwxr-xr-x 63 root wheel 2016 Sep 6 14:34 sbin
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 srv
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 sys
drwxr-xr-x 2 root wheel 64 Sep 6 14:34 tmp
drwxr-xr-x 7 root wheel 224 Sep 6 14:34 usr
drwxr-xr-x 13 root wheel 416 Sep 6 14:34 var
```
##### Image OCI Layout
Yaml showing both read-only and read-write, and both all architectures and a limited subset:
```yml
volumes:
- name: volo
image: alpine:latest
format: oci
readonly: true
- name: volp
image: alpine:latest
readonly: false
format: oci
platforms:
- linux/amd64
```
In the above example:
* `volo` is populated by the contents of `alpine:latest` as an OCI v1-layout for all architectures and is read-only.
* `volb` is populated by the contents of `alpine:latest` as an OCI v1-layout just for linux/amd64 and is read-write.
##### Volumes in `services`
Sample usage of volumes in `services` section:

View File

@@ -22,12 +22,13 @@ func createPackageResolver(baseDir string) spec.PackageResolver {
pkgValue = pkgTmpl
case strings.HasPrefix(pkgTmpl, templateFlag+templatePkg):
pkgPath := strings.TrimPrefix(pkgTmpl, templateFlag+templatePkg)
piBase := pkglib.NewPkgInfo()
var pkgs []pkglib.Pkg
pkgConfig := pkglib.PkglibConfig{
BuildYML: defaultPkgBuildYML,
HashCommit: defaultPkgCommit,
Tag: defaultPkgTag,
Tag: piBase.Tag,
}
pkgs, err = pkglib.NewFromConfig(pkgConfig, path.Join(baseDir, pkgPath))
if err != nil {

View File

@@ -8,6 +8,7 @@ import (
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/match"
"github.com/google/go-containerregistry/pkg/v1/partial"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// matchPlatformsOSArch because match.Platforms rejects it if the provided
@@ -46,7 +47,7 @@ func matchAllAnnotations(annotations map[string]string) match.Matcher {
}
}
func (p *Provider) findImage(imageName, architecture string) (v1.Image, error) {
func (p *Provider) findImage(imageName string, platform imagespec.Platform) (v1.Image, error) {
root, err := p.FindRoot(imageName)
if err != nil {
return nil, err
@@ -58,7 +59,7 @@ func (p *Provider) findImage(imageName, architecture string) (v1.Image, error) {
ii, err := root.ImageIndex()
if err == nil {
// we have the index, get the manifest that represents the manifest for the desired architecture
platform := v1.Platform{OS: "linux", Architecture: architecture}
platform := v1.Platform{OS: platform.OS, Architecture: platform.Architecture}
images, err := partial.FindImages(ii, matchPlatformsOSArch(platform))
if err != nil || len(images) < 1 {
return nil, fmt.Errorf("error retrieving image %s for platform %v from cache: %v", imageName, platform, err)

View File

@@ -10,7 +10,6 @@ import (
"github.com/containerd/containerd/reference"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
"github.com/google/go-containerregistry/pkg/v1/match"
"github.com/google/go-containerregistry/pkg/v1/mutate"
"github.com/google/go-containerregistry/pkg/v1/partial"
@@ -32,10 +31,10 @@ const (
// ImageSource a source for an image in the OCI distribution cache.
// Implements a spec.ImageSource.
type ImageSource struct {
ref *reference.Spec
provider *Provider
architecture string
descriptor *v1.Descriptor
ref *reference.Spec
provider *Provider
platform *imagespec.Platform
descriptor *v1.Descriptor
}
type spdxStatement struct {
@@ -45,12 +44,12 @@ type spdxStatement struct {
// NewSource return an ImageSource for a specific ref and architecture in the given
// cache directory.
func (p *Provider) NewSource(ref *reference.Spec, architecture string, descriptor *v1.Descriptor) lktspec.ImageSource {
func (p *Provider) NewSource(ref *reference.Spec, platform *imagespec.Platform, descriptor *v1.Descriptor) lktspec.ImageSource {
return ImageSource{
ref: ref,
provider: p,
architecture: architecture,
descriptor: descriptor,
ref: ref,
provider: p,
platform: platform,
descriptor: descriptor,
}
}
@@ -58,7 +57,7 @@ func (p *Provider) NewSource(ref *reference.Spec, architecture string, descripto
// architecture, if necessary.
func (c ImageSource) Config() (imagespec.ImageConfig, error) {
imageName := c.ref.String()
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return imagespec.ImageConfig{}, err
}
@@ -84,7 +83,7 @@ func (c ImageSource) TarReader() (io.ReadCloser, error) {
imageName := c.ref.String()
// get a reference to the image
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return nil, err
}
@@ -104,7 +103,7 @@ func (c ImageSource) V1TarReader(overrideName string) (io.ReadCloser, error) {
return nil, fmt.Errorf("error parsing image name: %v", err)
}
// get a reference to the image
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return nil, err
}
@@ -129,7 +128,7 @@ func (c ImageSource) OCITarReader(overrideName string) (io.ReadCloser, error) {
return nil, fmt.Errorf("error parsing image name: %v", err)
}
// get a reference to the image
image, err := c.provider.findImage(imageName, c.architecture)
image, err := c.provider.findImage(imageName, *c.platform)
if err != nil {
return nil, err
}
@@ -139,160 +138,37 @@ func (c ImageSource) OCITarReader(overrideName string) (io.ReadCloser, error) {
defer w.Close()
tw := tar.NewWriter(w)
defer tw.Close()
// layout file
layoutFileBytes := []byte(layoutFile)
if err := tw.WriteHeader(&tar.Header{
Name: "oci-layout",
Mode: 0644,
Size: int64(len(layoutFileBytes)),
Typeflag: tar.TypeReg,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(layoutFileBytes); err != nil {
if err := writeLayoutHeader(tw); err != nil {
_ = w.CloseWithError(err)
return
}
// make blobs directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
if err := writeLayoutImage(tw, image); err != nil {
_ = w.CloseWithError(err)
return
}
// make blobs/sha256 directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/sha256/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
_ = w.CloseWithError(err)
return
}
// write config, each layer, manifest, saving the digest for each
config, err := image.RawConfigFile()
imageDigest, err := image.Digest()
if err != nil {
_ = w.CloseWithError(err)
return
}
configDigest, configSize, err := v1.SHA256(bytes.NewReader(config))
imageSize, err := image.Size()
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", configDigest.Hex),
Mode: 0644,
Typeflag: tar.TypeReg,
Size: configSize,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(config); err != nil {
_ = w.CloseWithError(err)
return
}
layers, err := image.Layers()
if err != nil {
_ = w.CloseWithError(err)
return
}
for _, layer := range layers {
blob, err := layer.Compressed()
if err != nil {
_ = w.CloseWithError(err)
return
}
defer blob.Close()
blobDigest, err := layer.Digest()
if err != nil {
_ = w.CloseWithError(err)
return
}
blobSize, err := layer.Size()
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", blobDigest.Hex),
Mode: 0644,
Size: blobSize,
Typeflag: tar.TypeReg,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := io.Copy(tw, blob); err != nil {
_ = w.CloseWithError(err)
return
}
}
// write the manifest
manifest, err := image.RawManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
manifestDigest, manifestSize, err := v1.SHA256(bytes.NewReader(manifest))
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", manifestDigest.Hex),
Mode: 0644,
Size: int64(len(manifest)),
Typeflag: tar.TypeReg,
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(manifest); err != nil {
_ = w.CloseWithError(err)
return
}
// write the index file
desc := v1.Descriptor{
MediaType: types.OCIImageIndex,
Size: manifestSize,
Digest: manifestDigest,
Size: imageSize,
Digest: imageDigest,
Annotations: map[string]string{
imagespec.AnnotationRefName: refName.String(),
},
}
ii := empty.Index
index, err := ii.IndexManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
index.Manifests = append(index.Manifests, desc)
rawIndex, err := json.MarshalIndent(index, "", " ")
if err != nil {
_ = w.CloseWithError(err)
return
}
// write the index
if err := tw.WriteHeader(&tar.Header{
Name: "index.json",
Mode: 0644,
Size: int64(len(rawIndex)),
}); err != nil {
_ = w.CloseWithError(err)
return
}
if _, err := tw.Write(rawIndex); err != nil {
if err := writeLayoutIndex(tw, desc); err != nil {
_ = w.CloseWithError(err)
return
}
@@ -314,15 +190,15 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
}
// get the digest of the manifest that represents our targeted architecture
descs, err := partial.FindManifests(index, matchPlatformsOSArch(v1.Platform{OS: "linux", Architecture: c.architecture}))
descs, err := partial.FindManifests(index, matchPlatformsOSArch(v1.Platform{OS: c.platform.OS, Architecture: c.platform.Architecture}))
if err != nil {
return nil, err
}
if len(descs) < 1 {
return nil, fmt.Errorf("no manifest found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("no manifest found for %s platform %s", c.ref.String(), c.platform)
}
if len(descs) > 1 {
return nil, fmt.Errorf("multiple manifests found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("multiple manifests found for %s platform %s", c.ref.String(), c.platform)
}
// get the digest of the manifest that represents our targeted architecture
desc := descs[0]
@@ -336,7 +212,7 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
return nil, err
}
if len(descs) > 1 {
return nil, fmt.Errorf("multiple manifests found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("multiple manifests found for %s platform %s", c.ref.String(), c.platform)
}
if len(descs) < 1 {
return nil, nil
@@ -348,10 +224,10 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
return nil, err
}
if len(images) < 1 {
return nil, fmt.Errorf("no attestation image found for %s arch %s, even though the manifest exists", c.ref.String(), c.architecture)
return nil, fmt.Errorf("no attestation image found for %s platform %s, even though the manifest exists", c.ref.String(), c.platform)
}
if len(images) > 1 {
return nil, fmt.Errorf("multiple attestation images found for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("multiple attestation images found for %s platform %s", c.ref.String(), c.platform)
}
image := images[0]
manifest, err := image.Manifest()
@@ -363,7 +239,7 @@ func (c ImageSource) SBoMs() ([]io.ReadCloser, error) {
return nil, err
}
if len(manifest.Layers) != len(layers) {
return nil, fmt.Errorf("manifest layers and image layers do not match for the attestation for %s arch %s", c.ref.String(), c.architecture)
return nil, fmt.Errorf("manifest layers and image layers do not match for the attestation for %s platform %s", c.ref.String(), c.platform)
}
var readers []io.ReadCloser
for i, layer := range manifest.Layers {

162
src/cmd/linuxkit/cache/indexsource.go vendored Normal file
View File

@@ -0,0 +1,162 @@
package cache
import (
"archive/tar"
"bytes"
"fmt"
"io"
"github.com/containerd/containerd/reference"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/types"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// IndexSource a source for an image in the OCI distribution cache.
// Implements a spec.ImageSource.
type IndexSource struct {
ref *reference.Spec
provider *Provider
descriptor *v1.Descriptor
platforms []imagespec.Platform
}
// NewIndexSource return an IndexSource for a specific ref in the given
// cache directory.
func (p *Provider) NewIndexSource(ref *reference.Spec, descriptor *v1.Descriptor, platforms []imagespec.Platform) lktspec.IndexSource {
return IndexSource{
ref: ref,
provider: p,
descriptor: descriptor,
platforms: platforms,
}
}
// Config return the imagespec.ImageConfig for the given source. Resolves to the
// architecture, if necessary.
func (c IndexSource) Image(platform imagespec.Platform) (spec.ImageSource, error) {
imageName := c.ref.String()
index, err := c.provider.findIndex(imageName)
if err != nil {
return nil, err
}
manifests, err := index.IndexManifest()
if err != nil {
return nil, err
}
for _, manifest := range manifests.Manifests {
if manifest.Platform != nil && manifest.Platform.Architecture == platform.Architecture && manifest.Platform.OS == platform.OS {
return c.provider.NewSource(c.ref, &platform, &manifest), nil
}
}
return nil, fmt.Errorf("no manifest found for platform %q", platform)
}
// OCITarReader return an io.ReadCloser to read the image as a v1 tarball whose contents match OCI v1 layout spec
func (c IndexSource) OCITarReader(overrideName string) (io.ReadCloser, error) {
imageName := c.ref.String()
saveName := imageName
if overrideName != "" {
saveName = overrideName
}
refName, err := name.ParseReference(saveName)
if err != nil {
return nil, fmt.Errorf("error parsing image name: %v", err)
}
// get a reference to the image
index, err := c.provider.findIndex(c.ref.String())
if err != nil {
return nil, err
}
// convert the writer to a reader
r, w := io.Pipe()
go func() {
defer w.Close()
tw := tar.NewWriter(w)
defer tw.Close()
if err := writeLayoutHeader(tw); err != nil {
_ = w.CloseWithError(err)
return
}
manifests, err := index.IndexManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
// for each manifest, write the manifest blob, then go through each manifest and find the image for it
// and write its blobs
for _, manifest := range manifests.Manifests {
// if we restricted this image source to certain platforms, we should only write those
if len(c.platforms) > 0 {
found := false
for _, platform := range c.platforms {
if platform.Architecture == manifest.Platform.Architecture && platform.OS == manifest.Platform.OS &&
(platform.Variant == "" || platform.Variant == manifest.Platform.Variant) {
found = true
break
}
}
if !found {
continue
}
}
switch manifest.MediaType {
case types.OCIManifestSchema1, types.DockerManifestSchema2:
// this is an image manifest
image, err := index.Image(manifest.Digest)
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := writeLayoutImage(tw, image); err != nil {
_ = w.CloseWithError(err)
return
}
}
}
// write the index directly as a blob
indexSize, err := index.Size()
if err != nil {
_ = w.CloseWithError(err)
return
}
indexDigest, err := index.Digest()
if err != nil {
_ = w.CloseWithError(err)
return
}
indexBytes, err := index.RawManifest()
if err != nil {
_ = w.CloseWithError(err)
return
}
if err := writeLayoutBlob(tw, indexDigest.Hex, indexSize, bytes.NewReader(indexBytes)); err != nil {
_ = w.CloseWithError(err)
return
}
desc := v1.Descriptor{
MediaType: types.OCIImageIndex,
Size: indexSize,
Digest: indexDigest,
Annotations: map[string]string{
imagespec.AnnotationRefName: refName.String(),
},
}
if err := writeLayoutIndex(tw, desc); err != nil {
_ = w.CloseWithError(err)
return
}
}()
return r, nil
}
// Descriptor return the descriptor of the index.
func (c IndexSource) Descriptor() *v1.Descriptor {
return c.descriptor
}

145
src/cmd/linuxkit/cache/layout.go vendored Normal file
View File

@@ -0,0 +1,145 @@
package cache
import (
"archive/tar"
"bytes"
"encoding/json"
"fmt"
"io"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/empty"
)
func writeLayoutHeader(tw *tar.Writer) error {
// layout file
layoutFileBytes := []byte(layoutFile)
if err := tw.WriteHeader(&tar.Header{
Name: "oci-layout",
Mode: 0644,
Size: int64(len(layoutFileBytes)),
Typeflag: tar.TypeReg,
}); err != nil {
return err
}
if _, err := tw.Write(layoutFileBytes); err != nil {
return err
}
// make blobs directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
return err
}
// make blobs/sha256 directory
if err := tw.WriteHeader(&tar.Header{
Name: "blobs/sha256/",
Mode: 0755,
Typeflag: tar.TypeDir,
}); err != nil {
return err
}
return nil
}
func writeLayoutImage(tw *tar.Writer, image v1.Image) error {
// write config, each layer, manifest, saving the digest for each
manifest, err := image.Manifest()
if err != nil {
return err
}
configDesc := manifest.Config
configBytes, err := image.RawConfigFile()
if err != nil {
return err
}
if err := writeLayoutBlob(tw, configDesc.Digest.Hex, configDesc.Size, bytes.NewReader(configBytes)); err != nil {
return err
}
layers, err := image.Layers()
if err != nil {
return err
}
for _, layer := range layers {
blob, err := layer.Compressed()
if err != nil {
return err
}
defer blob.Close()
blobDigest, err := layer.Digest()
if err != nil {
return err
}
blobSize, err := layer.Size()
if err != nil {
return err
}
if err := writeLayoutBlob(tw, blobDigest.Hex, blobSize, blob); err != nil {
return err
}
}
// write the manifest
manifestSize, err := image.Size()
if err != nil {
return err
}
manifestDigest, err := image.Digest()
if err != nil {
return err
}
manifestBytes, err := image.RawManifest()
if err != nil {
return err
}
if err := writeLayoutBlob(tw, manifestDigest.Hex, manifestSize, bytes.NewReader(manifestBytes)); err != nil {
return err
}
return nil
}
func writeLayoutBlob(tw *tar.Writer, digest string, size int64, blob io.Reader) error {
if err := tw.WriteHeader(&tar.Header{
Name: fmt.Sprintf("blobs/sha256/%s", digest),
Mode: 0644,
Size: size,
Typeflag: tar.TypeReg,
}); err != nil {
return err
}
if _, err := io.Copy(tw, blob); err != nil {
return err
}
return nil
}
func writeLayoutIndex(tw *tar.Writer, desc v1.Descriptor) error {
ii := empty.Index
index, err := ii.IndexManifest()
if err != nil {
return err
}
index.Manifests = append(index.Manifests, desc)
rawIndex, err := json.MarshalIndent(index, "", " ")
if err != nil {
return err
}
// write the index
if err := tw.WriteHeader(&tar.Header{
Name: "index.json",
Mode: 0644,
Size: int64(len(rawIndex)),
}); err != nil {
return err
}
if _, err := tw.Write(rawIndex); err != nil {
return err
}
return nil
}

33
src/cmd/linuxkit/cache/platform.go vendored Normal file
View File

@@ -0,0 +1,33 @@
package cache
import (
"fmt"
"strings"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
func platformString(p imagespec.Platform) string {
parts := []string{p.OS, p.Architecture}
if p.Variant != "" {
parts = append(parts, p.Variant)
}
return strings.Join(parts, "/")
}
func platformMessageGenerator(platforms []imagespec.Platform) string {
var platformMessage string
switch {
case len(platforms) == 0:
platformMessage = "all platforms"
case len(platforms) == 1:
platformMessage = fmt.Sprintf("platform %s", platformString(platforms[0]))
default:
var platStrings []string
for _, p := range platforms {
platStrings = append(platStrings, platformString(p))
}
platformMessage = fmt.Sprintf("platforms %s", strings.Join(platStrings, ","))
}
return platformMessage
}

View File

@@ -3,6 +3,7 @@ package cache
import (
"errors"
"fmt"
"strings"
"github.com/containerd/containerd/reference"
"github.com/google/go-containerregistry/pkg/authn"
@@ -30,12 +31,13 @@ const (
// architecture, and any manifests that have no architecture at all. It will ignore manifests
// for other architectures. If no architecture is provided, it will validate all manifests.
// It also calculates the hash of each component.
func (p *Provider) ValidateImage(ref *reference.Spec, architecture string) (lktspec.ImageSource, error) {
func (p *Provider) ValidateImage(ref *reference.Spec, platforms []imagespec.Platform) (lktspec.ImageSource, error) {
var (
imageIndex v1.ImageIndex
image v1.Image
imageName = ref.String()
desc *v1.Descriptor
imageIndex v1.ImageIndex
image v1.Image
imageName = ref.String()
desc *v1.Descriptor
platformMessage = platformMessageGenerator(platforms)
)
// next try the local cache
root, err := p.FindRoot(imageName)
@@ -71,7 +73,15 @@ func (p *Provider) ValidateImage(ref *reference.Spec, architecture string) (lkts
if err != nil {
return ImageSource{}, fmt.Errorf("could not get index manifest: %w", err)
}
var architectures = make(map[string]bool)
var (
targetPlatforms = make(map[string]bool)
foundPlatforms = make(map[string]bool)
)
for _, plat := range platforms {
pString := platformString(plat)
targetPlatforms[pString] = false
foundPlatforms[pString] = false
}
// ignore only other architectures; manifest entries that have no architectures at all
// are going to be additional metadata, so we need to check them
for _, m := range im.Manifests {
@@ -80,29 +90,50 @@ func (p *Provider) ValidateImage(ref *reference.Spec, architecture string) (lkts
return ImageSource{}, fmt.Errorf("invalid image: %w", err)
}
}
if architecture != "" && m.Platform.Architecture == architecture && m.Platform.OS == linux {
if err := validateManifestContents(imageIndex, m.Digest); err != nil {
return ImageSource{}, fmt.Errorf("invalid image: %w", err)
// go through each target platform, and see if this one matched. If it did, mark the target as
for _, plat := range platforms {
if plat.Architecture == m.Platform.Architecture && plat.OS == m.Platform.OS &&
(plat.Variant == "" || plat.Variant == m.Platform.Variant) {
targetPlatforms[platformString(plat)] = true
break
}
architectures[architecture] = true
}
}
if architecture == "" || architectures[architecture] {
if len(platforms) == 0 {
return p.NewSource(
ref,
architecture,
nil,
desc,
), nil
}
return ImageSource{}, fmt.Errorf("index for %s did not contain image for platform linux/%s", imageName, architecture)
// we have cycled through all of the manifests, let's check if we have all of the platforms
var missing []string
for plat, found := range targetPlatforms {
if !found {
missing = append(missing, plat)
}
}
if len(missing) == 0 {
return p.NewSource(
ref,
nil,
desc,
), nil
}
return ImageSource{}, fmt.Errorf("index for %s did not contain image for platforms %s", imageName, strings.Join(missing, ", "))
case image != nil:
if len(platforms) > 1 {
return ImageSource{}, fmt.Errorf("image %s is not a multi-arch image, but asked for %s", imageName, platformMessage)
}
// we found a local image, make sure it is up to date
if err := validate.Image(image); err != nil {
return ImageSource{}, fmt.Errorf("invalid image, %s", err)
}
return p.NewSource(
ref,
architecture,
&platforms[0],
desc,
), nil
}
@@ -164,7 +195,7 @@ func (p *Provider) Pull(name string, withArchReferences bool) error {
if err := p.cache.WriteIndex(ii); err != nil {
return fmt.Errorf("unable to write index: %v", err)
}
if _, err := p.DescriptorWrite(&v1ref, desc.Descriptor); err != nil {
if err := p.DescriptorWrite(&v1ref, desc.Descriptor); err != nil {
return fmt.Errorf("unable to write index descriptor to cache: %v", err)
}
if withArchReferences {
@@ -179,7 +210,7 @@ func (p *Provider) Pull(name string, withArchReferences bool) error {
if err != nil {
return fmt.Errorf("unable to parse arch-specific reference %s: %v", archSpecific, err)
}
if _, err := p.DescriptorWrite(&archRef, m); err != nil {
if err := p.DescriptorWrite(&archRef, m); err != nil {
return fmt.Errorf("unable to write index descriptor to cache: %v", err)
}
}

View File

@@ -19,9 +19,7 @@ import (
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/remote"
"github.com/google/go-containerregistry/pkg/v1/types"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
lktutil "github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
log "github.com/sirupsen/logrus"
)
@@ -41,45 +39,42 @@ const (
// If you just want to check the status of a local ref, use ValidateImage.
// Note that ImagePull does try ValidateImage first, so if the image is already in the cache, it will not
// do any network activity at all.
func (p *Provider) ImagePull(ref *reference.Spec, trustedRef, architecture string, alwaysPull bool) (lktspec.ImageSource, error) {
func (p *Provider) ImagePull(ref *reference.Spec, platforms []imagespec.Platform, alwaysPull bool) error {
imageName := util.ReferenceExpand(ref.String())
canonicalRef, err := reference.Parse(imageName)
if err != nil {
return ImageSource{}, fmt.Errorf("invalid image name %s: %v", imageName, err)
return fmt.Errorf("invalid image name %s: %v", imageName, err)
}
ref = &canonicalRef
image := ref.String()
pullImageName := image
platformMessage := platformMessageGenerator(platforms)
remoteOptions := []remote.Option{remote.WithAuthFromKeychain(authn.DefaultKeychain)}
if trustedRef != "" {
pullImageName = trustedRef
}
log.Debugf("ImagePull to cache %s trusted reference %s", image, pullImageName)
// unless alwaysPull is set to true, check locally first
if alwaysPull {
log.Printf("Instructed always to pull, so pulling image %s arch %s", image, architecture)
log.Debugf("Instructed always to pull, so pulling image %s %s", image, platformMessage)
} else {
imgSrc, err := p.ValidateImage(ref, architecture)
imgSrc, err := p.ValidateImage(ref, platforms)
switch {
case err == nil && imgSrc != nil:
log.Printf("Image %s arch %s found in local cache, not pulling", image, architecture)
return imgSrc, nil
log.Debugf("Image %s %s found in local cache, not pulling", image, platformMessage)
return nil
case err != nil && errors.Is(err, &noReferenceError{}):
log.Printf("Image %s arch %s not found in local cache, pulling", image, architecture)
log.Debugf("Image %s %s not found in local cache, pulling", image, platformMessage)
default:
log.Printf("Image %s arch %s incomplete or invalid in local cache, error %v, pulling", image, architecture, err)
log.Debugf("Image %s %s incomplete or invalid in local cache, error %v, pulling", image, platformMessage, err)
}
// there was an error, so try to pull
}
remoteRef, err := name.ParseReference(pullImageName)
if err != nil {
return ImageSource{}, fmt.Errorf("invalid image name %s: %v", pullImageName, err)
return fmt.Errorf("invalid image name %s: %v", pullImageName, err)
}
desc, err := remote.Get(remoteRef, remoteOptions...)
if err != nil {
return ImageSource{}, fmt.Errorf("error getting manifest for trusted image %s: %v", pullImageName, err)
return fmt.Errorf("error getting manifest for image %s: %v", pullImageName, err)
}
// use the original image name in the annotation
@@ -90,46 +85,57 @@ func (p *Provider) ImagePull(ref *reference.Spec, trustedRef, architecture strin
// first attempt as an index
ii, err := desc.ImageIndex()
if err == nil {
log.Debugf("ImageWrite retrieved %s is index, saving, first checking if it contains target arch %s", pullImageName, architecture)
log.Debugf("ImageWrite retrieved %s is index, saving, first checking if it contains target %s", pullImageName, platformMessage)
im, err := ii.IndexManifest()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to get IndexManifest: %v", err)
return fmt.Errorf("unable to get IndexManifest: %v", err)
}
// only useful if it contains our architecture
var foundArch bool
var foundPlatforms []*v1.Platform
for _, m := range im.Manifests {
if m.MediaType.IsImage() && m.Platform != nil && m.Platform.Architecture == architecture && m.Platform.OS == linux {
foundArch = true
break
if m.MediaType.IsImage() && m.Platform != nil {
foundPlatforms = append(foundPlatforms, m.Platform)
}
}
if !foundArch {
return ImageSource{}, fmt.Errorf("index %s does not contain target architecture %s", pullImageName, architecture)
// now see if we have all of the platforms we need
var missing []string
for _, requiredPlatform := range platforms {
// we did not find it, so maybe one satisfies it
var matchedPlatform bool
for _, p := range foundPlatforms {
if p.OS == requiredPlatform.OS && p.Architecture == requiredPlatform.Architecture && (p.Variant == requiredPlatform.Variant || requiredPlatform.Variant == "") {
// this one satisfies it, so do not count it missing
matchedPlatform = true
break
}
}
if !matchedPlatform {
missing = append(missing, platformString(requiredPlatform))
}
}
if len(missing) > 0 {
return fmt.Errorf("index %s does not contain target platforms %s", pullImageName, strings.Join(missing, ","))
}
if err := p.cache.WriteIndex(ii); err != nil {
return ImageSource{}, fmt.Errorf("unable to write index: %v", err)
return fmt.Errorf("unable to write index: %v", err)
}
if _, err := p.DescriptorWrite(ref, desc.Descriptor); err != nil {
return ImageSource{}, fmt.Errorf("unable to write index descriptor to cache: %v", err)
if err := p.DescriptorWrite(ref, desc.Descriptor); err != nil {
return fmt.Errorf("unable to write index descriptor to cache: %v", err)
}
} else {
var im v1.Image
// try an image
im, err = desc.Image()
if err != nil {
return ImageSource{}, fmt.Errorf("provided image is neither an image nor an index: %s", image)
return fmt.Errorf("provided image is neither an image nor an index: %s", image)
}
log.Debugf("ImageWrite retrieved %s is image, saving", pullImageName)
if err = p.cache.ReplaceImage(im, match.Name(image), layout.WithAnnotations(annotations)); err != nil {
return ImageSource{}, fmt.Errorf("unable to save image to cache: %v", err)
return fmt.Errorf("unable to save image to cache: %v", err)
}
}
return p.NewSource(
ref,
architecture,
&desc.Descriptor,
), nil
return nil
}
// ImageLoad takes an OCI format image tar stream and writes it locally. It should be
@@ -227,27 +233,27 @@ func (p *Provider) ImageLoad(r io.Reader) ([]v1.Descriptor, error) {
// does not pull down any images; entirely assumes that the subjects of the manifests are present.
// If a reference to the provided already exists and it is an index, updates the manifests in the
// existing index.
func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) (lktspec.ImageSource, error) {
func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) error {
image := ref.String()
log.Debugf("writing an index for %s", image)
if len(descriptors) < 1 {
return ImageSource{}, errors.New("cannot create index without any manifests")
return errors.New("cannot create index without any manifests")
}
ii, err := p.cache.ImageIndex()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to get root index: %v", err)
return fmt.Errorf("unable to get root index: %v", err)
}
images, err := partial.FindImages(ii, match.Name(image))
if err != nil {
return ImageSource{}, fmt.Errorf("error parsing index: %v", err)
return fmt.Errorf("error parsing index: %v", err)
}
if err == nil && len(images) > 0 {
return ImageSource{}, fmt.Errorf("image named %s already exists in cache and is not an index", image)
return fmt.Errorf("image named %s already exists in cache and is not an index", image)
}
indexes, err := partial.FindIndexes(ii, match.Name(image))
if err != nil {
return ImageSource{}, fmt.Errorf("error parsing index: %v", err)
return fmt.Errorf("error parsing index: %v", err)
}
var im v1.IndexManifest
// do we update an existing one? Or create a new one?
@@ -255,11 +261,11 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
// we already had one, so update just the referenced index and return
manifest, err := indexes[0].IndexManifest()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to convert index for %s into its manifest: %v", image, err)
return fmt.Errorf("unable to convert index for %s into its manifest: %v", image, err)
}
oldhash, err := indexes[0].Digest()
if err != nil {
return ImageSource{}, fmt.Errorf("unable to get hash of existing index: %v", err)
return fmt.Errorf("unable to get hash of existing index: %v", err)
}
// we only care about avoiding duplicate arch/OS/Variant
var (
@@ -318,7 +324,7 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
appliedManifests[m.Digest] = true
continue
}
value, ok := m.Annotations[lktutil.AnnotationDockerReferenceDigest]
value, ok := m.Annotations[util.AnnotationDockerReferenceDigest]
if !ok {
manifest.Manifests = append(manifest.Manifests, m)
appliedManifests[m.Digest] = true
@@ -336,7 +342,7 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
im = *manifest
// remove the old index
if err := p.cache.RemoveBlob(oldhash); err != nil {
return ImageSource{}, fmt.Errorf("unable to remove old index file: %v", err)
return fmt.Errorf("unable to remove old index file: %v", err)
}
} else {
// we did not have one, so create an index, store it, update the root index.json, and return
@@ -350,18 +356,18 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
// write the updated index, remove the old one
b, err := json.Marshal(im)
if err != nil {
return ImageSource{}, fmt.Errorf("unable to marshal new index to json: %v", err)
return fmt.Errorf("unable to marshal new index to json: %v", err)
}
hash, size, err := v1.SHA256(bytes.NewReader(b))
if err != nil {
return ImageSource{}, fmt.Errorf("error calculating hash of index json: %v", err)
return fmt.Errorf("error calculating hash of index json: %v", err)
}
if err := p.cache.WriteBlob(hash, io.NopCloser(bytes.NewReader(b))); err != nil {
return ImageSource{}, fmt.Errorf("error writing new index to json: %v", err)
return fmt.Errorf("error writing new index to json: %v", err)
}
// finally update the descriptor in the root
if err := p.cache.RemoveDescriptors(match.Name(image)); err != nil {
return ImageSource{}, fmt.Errorf("unable to remove old descriptor from index.json: %v", err)
return fmt.Errorf("unable to remove old descriptor from index.json: %v", err)
}
desc := v1.Descriptor{
MediaType: types.OCIImageIndex,
@@ -372,21 +378,17 @@ func (p *Provider) IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor)
},
}
if err := p.cache.AppendDescriptor(desc); err != nil {
return ImageSource{}, fmt.Errorf("unable to append new descriptor to index.json: %v", err)
return fmt.Errorf("unable to append new descriptor to index.json: %v", err)
}
return p.NewSource(
ref,
"",
&desc,
), nil
return nil
}
// DescriptorWrite writes a descriptor to the cache index; it validates that it has a name
// and replaces any existing one
func (p *Provider) DescriptorWrite(ref *reference.Spec, desc v1.Descriptor) (lktspec.ImageSource, error) {
func (p *Provider) DescriptorWrite(ref *reference.Spec, desc v1.Descriptor) error {
if ref == nil {
return ImageSource{}, errors.New("cannot write descriptor without reference name")
return errors.New("cannot write descriptor without reference name")
}
image := ref.String()
if desc.Annotations == nil {
@@ -397,22 +399,18 @@ func (p *Provider) DescriptorWrite(ref *reference.Spec, desc v1.Descriptor) (lkt
// do we update an existing one? Or create a new one?
if err := p.cache.RemoveDescriptors(match.Name(image)); err != nil {
return ImageSource{}, fmt.Errorf("unable to remove old descriptors for %s: %v", image, err)
return fmt.Errorf("unable to remove old descriptors for %s: %v", image, err)
}
if err := p.cache.AppendDescriptor(desc); err != nil {
return ImageSource{}, fmt.Errorf("unable to append new descriptor for %s: %v", image, err)
return fmt.Errorf("unable to append new descriptor for %s: %v", image, err)
}
return p.NewSource(
ref,
"",
&desc,
), nil
return nil
}
func (p *Provider) ImageInCache(ref *reference.Spec, trustedRef, architecture string) (bool, error) {
img, err := p.findImage(ref.String(), architecture)
img, err := p.findImage(ref.String(), imagespec.Platform{OS: linux, Architecture: architecture})
if err != nil {
return false, err
}

View File

@@ -4,17 +4,20 @@ import (
"io"
"os"
"runtime"
"strings"
"github.com/containerd/containerd/reference"
v1 "github.com/google/go-containerregistry/pkg/v1"
cachepkg "github.com/linuxkit/linuxkit/src/cmd/linuxkit/cache"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
log "github.com/sirupsen/logrus"
"github.com/spf13/cobra"
)
func cacheExportCmd() *cobra.Command {
var (
arch string
platform string
outputFile string
format string
tagName string
@@ -42,7 +45,16 @@ func cacheExportCmd() *cobra.Command {
log.Fatalf("unable to find image named %s: %v", name, err)
}
src := p.NewSource(&ref, arch, desc)
plat, err := v1.ParsePlatform(platform)
if err != nil {
log.Fatalf("invalid platform %s: %v", platform, err)
}
platspec := imagespec.Platform{
Architecture: plat.Architecture,
OS: plat.OS,
Variant: plat.Variant,
}
src := p.NewSource(&ref, &platspec, desc)
var reader io.ReadCloser
switch format {
case "docker":
@@ -88,7 +100,7 @@ func cacheExportCmd() *cobra.Command {
},
}
cmd.Flags().StringVar(&arch, "arch", runtime.GOARCH, "Architecture to resolve an index to an image, if the provided image name is an index")
cmd.Flags().StringVar(&platform, "platform", strings.Join([]string{"linux", runtime.GOARCH}, "/"), "Platform to resolve an index to an image, if the provided image name is an index")
cmd.Flags().StringVar(&outputFile, "outfile", "", "Path to file to save output, '-' for stdout")
cmd.Flags().StringVar(&format, "format", "oci", "export format, one of 'oci' (OCI tar), 'docker' (docker tar), 'filesystem'")
cmd.Flags().StringVar(&tagName, "name", "", "override the provided image name in the exported tar file; useful only for format=oci")

View File

@@ -43,8 +43,9 @@ func readConfig() {
func newCmd() *cobra.Command {
var (
flagQuiet bool
flagVerbose bool
flagQuiet bool
flagVerbose int
flagVerboseName = "verbose"
)
cmd := &cobra.Command{
Use: "linuxkit",
@@ -54,7 +55,7 @@ func newCmd() *cobra.Command {
readConfig()
// Set up logging
return util.SetupLogging(flagQuiet, flagVerbose)
return util.SetupLogging(flagQuiet, flagVerbose, cmd.Flag(flagVerboseName).Changed)
},
}
@@ -69,7 +70,7 @@ func newCmd() *cobra.Command {
cmd.PersistentFlags().StringVar(&cacheDir, "cache", defaultLinuxkitCache(), fmt.Sprintf("Directory for caching and finding cached image, overrides env var %s", envVarCacheDir))
cmd.PersistentFlags().BoolVarP(&flagQuiet, "quiet", "q", false, "Quiet execution")
cmd.PersistentFlags().BoolVarP(&flagVerbose, "verbose", "v", false, "Verbose execution")
cmd.PersistentFlags().IntVarP(&flagVerbose, flagVerboseName, "v", 1, "Verbosity of logging: 0 = quiet, 1 = info, 2 = debug, 3 = trace. Default is info. Setting it explicitly will create structured logging lines.")
return cmd
}

View File

@@ -3,5 +3,4 @@ package main
const (
defaultPkgBuildYML = "build.yml"
defaultPkgCommit = "HEAD"
defaultPkgTag = "{{.Hash}}"
)

View File

@@ -54,9 +54,9 @@ require (
github.com/Code-Hex/vz/v3 v3.0.0
github.com/equinix/equinix-sdk-go v0.42.0
github.com/in-toto/in-toto-golang v0.5.0
github.com/moby/sys/capability v0.3.0
github.com/spdx/tools-golang v0.5.3
github.com/spf13/cobra v1.8.0
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635
gopkg.in/yaml.v3 v3.0.1
)

View File

@@ -103,8 +103,6 @@ github.com/docker/cli v26.1.3+incompatible h1:bUpXT/N0kDE3VUHI2r5VMsYQgi38kYuoC0
github.com/docker/cli v26.1.3+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/distribution v2.8.2+incompatible h1:T3de5rq0dB1j30rp0sA2rER+m322EBzniBPB6ZIzuh8=
github.com/docker/distribution v2.8.2+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w=
github.com/docker/docker v27.0.3+incompatible h1:aBGI9TeQ4MPlhquTQKq9XbK79rKFVwXNUAYz9aXyEBE=
github.com/docker/docker v27.0.3+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker v27.2.0+incompatible h1:Rk9nIVdfH3+Vz4cyI/uhbINhEZ/oLmc+CBXmH6fbNk4=
github.com/docker/docker v27.2.0+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
github.com/docker/docker-credential-helpers v0.8.0 h1:YQFtbBQb4VrpoPxhFuzEBPQ9E16qz5SpHLS+uswaCp8=
@@ -247,6 +245,8 @@ github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
github.com/moby/sys/capability v0.3.0 h1:kEP+y6te0gEXIaeQhIi0s7vKs/w0RPoH1qPa6jROcVg=
github.com/moby/sys/capability v0.3.0/go.mod h1:4g9IK291rVkms3LKCDOoYlnV8xKwoDTpIrNEE35Wq0I=
github.com/moby/sys/mountinfo v0.7.1 h1:/tTvQaSJRr2FshkhXiIpux6fQ2Zvc4j7tAhMTStAG2g=
github.com/moby/sys/mountinfo v0.7.1/go.mod h1:IJb6JQeOklcdMU9F5xQ8ZALD+CUr5VlGpwtX+VE0rpI=
github.com/moby/sys/sequential v0.5.0 h1:OPvI35Lzn9K04PBbCLW0g4LcFAJgHsvXsRyewg5lXtc=
@@ -331,8 +331,6 @@ github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsT
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/surma/gocpio v1.0.2-0.20160926205914-fcb68777e7dc h1:iA3Eg1OVd2o0M4M+0PBsBBssMz98L8CUH7x0xVkuyUA=
github.com/surma/gocpio v1.0.2-0.20160926205914-fcb68777e7dc/go.mod h1:zaLNaN+EDnfSnNdWPJJf9OZxWF817w5dt8JNzF9LCVI=
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 h1:kdXcSzyDtseVEc4yCz2qF8ZrQvIDBJLl4S1c3GCXmoI=
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
github.com/tonistiigi/fsutil v0.0.0-20240424095704-91a3fc46842c h1:+6wg/4ORAbnSoGDzg2Q1i3CeMcT/jjhye/ZfnBHy7/M=
github.com/tonistiigi/fsutil v0.0.0-20240424095704-91a3fc46842c/go.mod h1:vbbYqJlnswsbJqWUcJN8fKtBhnEgldDrcagTgnBVKKM=
github.com/tonistiigi/units v0.0.0-20180711220420-6950e57a87ea h1:SXhTLE6pb6eld/v/cCndK0AMpt1wiVFb/YYmqB3/QG0=

View File

@@ -16,6 +16,9 @@ import (
"strings"
"github.com/containerd/containerd/reference"
v1 "github.com/google/go-containerregistry/pkg/v1"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
// drop-in 100% compatible replacement and 17% faster than compress/gzip.
gzip "github.com/klauspost/pgzip"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/moby"
@@ -84,6 +87,8 @@ func OutputTypes() []string {
return ts
}
// outputImage given an image and a section, such as onboot, onshutdown or services, lay it out with correct location
// config, etc. in the filesystem, so runc can use it.
func outputImage(image *moby.Image, section string, index int, prefix string, m moby.Moby, idMap map[string]uint32, dupMap map[string]string, iw *tar.Writer, opts BuildOpts) error {
log.Infof(" Create OCI config for %s", image.Image)
imageName := util.ReferenceExpand(image.Image)
@@ -91,7 +96,7 @@ func outputImage(image *moby.Image, section string, index int, prefix string, m
if err != nil {
return fmt.Errorf("could not resolve references for image %s: %v", image.Image, err)
}
src, err := imagePull(&ref, opts.Pull, opts.CacheDir, opts.DockerCache, opts.Arch)
src, err := imageSource(&ref, opts.Pull, opts.CacheDir, opts.DockerCache, imagespec.Platform{OS: "linux", Architecture: opts.Arch})
if err != nil {
return fmt.Errorf("could not pull image %s: %v", image.Image, err)
}
@@ -280,9 +285,30 @@ func Build(m moby.Moby, w io.Writer, opts BuildOpts) error {
lowerPath := strings.TrimPrefix(lower, "/") + "/"
// get volume tarball from container
if err := ImageTar(location, vol.ImageRef(), lowerPath, apkTar, resolvconfSymlink, opts); err != nil {
return fmt.Errorf("failed to build volume tarball from %s: %v", vol.Name, err)
switch {
case vol.ImageRef() == nil || vol.Format == "" || vol.Format == "filesystem":
if err := ImageTar(location, vol.ImageRef(), lowerPath, apkTar, resolvconfSymlink, opts); err != nil {
return fmt.Errorf("failed to build volume filesystem tarball from %s: %v", vol.Name, err)
}
case vol.Format == "oci":
// convert platforms into imagespec platforms
platforms := make([]imagespec.Platform, len(vol.Platforms))
for i, p := range vol.Platforms {
platform, err := v1.ParsePlatform(p)
if err != nil {
return fmt.Errorf("failed to parse platform %s: %v", p, err)
}
platforms[i] = imagespec.Platform{
Architecture: platform.Architecture,
OS: platform.OS,
Variant: platform.Variant,
}
}
if err := ImageOCITar(location, vol.ImageRef(), lowerPath, apkTar, opts, platforms); err != nil {
return fmt.Errorf("failed to build volume OCI v1 layout tarball from %s: %v", vol.Name, err)
}
}
// make upper and merged dirs which will be used for mounting
// no need to make lower dir, as it is made automatically by ImageTar()
tmpPath := strings.TrimPrefix(tmpDir, "/") + "/"

View File

@@ -11,6 +11,7 @@ import (
"github.com/containerd/containerd/reference"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/moby"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/opencontainers/runtime-spec/specs-go"
log "github.com/sirupsen/logrus"
)
@@ -175,14 +176,15 @@ func tarPrefix(path, location, refName string, tw tarWriter) error {
return nil
}
// ImageTar takes a Docker image and outputs it to a tar stream
// ImageTar takes a Docker image and outputs it to a tar stream as a merged filesystem for a specific architecture
// defined in opts.
// location is where it is in the linuxkit.yaml file
func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter, resolv string, opts BuildOpts) (e error) {
refName := "empty"
if ref != nil {
refName = ref.String()
}
log.Debugf("image tar: %s %s", refName, prefix)
log.Debugf("image filesystem tar: %s %s %s", refName, prefix, opts.Arch)
if prefix != "" && prefix[len(prefix)-1] != '/' {
return fmt.Errorf("prefix does not end with /: %s", prefix)
}
@@ -197,9 +199,8 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
return nil
}
// pullImage first checks in the cache, then pulls the image.
// If pull==true, then it always tries to pull from registry.
src, err := imagePull(ref, opts.Pull, opts.CacheDir, opts.DockerCache, opts.Arch)
// get a handle on the image, optionally from docker, pulling from registry if necessary.
src, err := imageSource(ref, opts.Pull, opts.CacheDir, opts.DockerCache, imagespec.Platform{OS: "linux", Architecture: opts.Arch})
if err != nil {
return fmt.Errorf("could not pull image %s: %v", ref, err)
}
@@ -237,7 +238,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.PAXRecords[moby.PaxRecordLinuxkitSource] = ref.String()
hdr.PAXRecords[moby.PaxRecordLinuxkitLocation] = location
if exclude[hdr.Name] {
log.Debugf("image tar: %s %s exclude %s", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s exclude %s", ref, prefix, hdr.Name)
_, err = io.Copy(io.Discard, tr)
if err != nil {
return err
@@ -248,7 +249,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.Size = int64(len(contents))
hdr.Name = prefix + hdr.Name
hdr.ModTime = defaultModTime
log.Debugf("image tar: %s %s add %s (replaced)", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s add %s (replaced)", ref, prefix, hdr.Name)
if err := tw.WriteHeader(hdr); err != nil {
return err
}
@@ -263,7 +264,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.Typeflag = tar.TypeSymlink
hdr.Linkname = resolv
hdr.ModTime = defaultModTime
log.Debugf("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
log.Tracef("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
if err := tw.WriteHeader(hdr); err != nil {
return err
}
@@ -274,12 +275,12 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
}
} else {
if found, ok := touch[hdr.Name]; ok {
log.Debugf("image tar: %s %s add %s (touch)", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s add %s (touch)", ref, prefix, hdr.Name)
hdr.ModTime = found.ModTime
// record that we saw this one
touchFound[hdr.Name] = true
} else {
log.Debugf("image tar: %s %s add %s (original)", ref, prefix, hdr.Name)
log.Tracef("image tar: %s %s add %s (original)", ref, prefix, hdr.Name)
}
hdr.Name = prefix + hdr.Name
if hdr.Typeflag == tar.TypeLink {
@@ -304,7 +305,7 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
sort.Strings(touchNames)
for _, name := range touchNames {
if touchFound[name] {
log.Debugf("image tar: %s already found in original image", name)
log.Tracef("image tar: %s already found in original image", name)
continue
}
hdr := touch[name]
@@ -326,9 +327,9 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
hdr.Size = 0
hdr.Typeflag = tar.TypeSymlink
hdr.Linkname = resolv
log.Debugf("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
log.Tracef("image tar: %s %s add resolv symlink /etc/resolv.conf -> %s", ref, prefix, resolv)
}
log.Debugf("image tar: creating %s", name)
log.Tracef("image tar: creating %s", name)
if err := tw.WriteHeader(&hdr); err != nil {
return err
}
@@ -356,6 +357,79 @@ func ImageTar(location string, ref *reference.Spec, prefix string, tw tarWriter,
return nil
}
// ImageOCITar takes an OCI image and outputs it to a tar stream as a v1 layout format.
// Will include all architectures, or, if specific ones provided, then only those.
// location is where it is in the linuxkit.yaml file
func ImageOCITar(location string, ref *reference.Spec, prefix string, tw tarWriter, opts BuildOpts, platforms []imagespec.Platform) (e error) {
refName := "empty"
if ref != nil {
refName = ref.String()
}
log.Debugf("image v1 layout tar: %s %s %s", refName, prefix, opts.Arch)
if prefix != "" && prefix[len(prefix)-1] != '/' {
return fmt.Errorf("prefix does not end with /: %s", prefix)
}
err := tarPrefix(prefix, location, refName, tw)
if err != nil {
return err
}
// if the image is blank, we do not need to do any more
if ref == nil {
return fmt.Errorf("no image reference provided")
}
// indexSource first checks in the cache, then pulls the image.
// If pull==true, then it always tries to pull from registry.
src, err := indexSource(ref, opts.Pull, opts.CacheDir, platforms)
if err != nil {
return fmt.Errorf("could not pull image %s: %v", ref, err)
}
contents, err := src.OCITarReader("")
if err != nil {
return fmt.Errorf("could not unpack image %s: %v", ref, err)
}
defer contents.Close()
tr := tar.NewReader(contents)
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return err
}
// force PAX format, since it allows for unlimited Name/Linkname
// and we move all files below prefix.
hdr.Format = tar.FormatPAX
// ensure we record the source of the file in the PAX header
if hdr.PAXRecords == nil {
hdr.PAXRecords = make(map[string]string)
}
hdr.PAXRecords[moby.PaxRecordLinuxkitSource] = ref.String()
hdr.PAXRecords[moby.PaxRecordLinuxkitLocation] = location
hdr.Name = prefix + hdr.Name
if hdr.Typeflag == tar.TypeLink {
// hard links are referenced by full path so need to be adjusted
hdr.Linkname = prefix + hdr.Linkname
}
if err := tw.WriteHeader(hdr); err != nil {
return err
}
_, err = io.Copy(tw, tr)
if err != nil {
return err
}
}
return nil
}
// ImageBundle produces an OCI bundle at the given path in a tarball, given an image and a config.json
func ImageBundle(prefix, location string, ref *reference.Spec, config []byte, runtime moby.Runtime, tw tarWriter, readonly bool, dupMap map[string]string, opts BuildOpts) error { // nolint: lll
// if read only, just unpack in rootfs/ but otherwise set up for overlay
@@ -489,7 +563,7 @@ func ImageBundle(prefix, location string, ref *reference.Spec, config []byte, ru
return err
}
log.Debugf("image bundle: %s %s cfg: %s runtime: %s", prefix, ref, string(config), string(runtimeConfig))
log.Tracef("image bundle: %s %s cfg: %s runtime: %s", prefix, ref, string(config), string(runtimeConfig))
return nil
}

View File

@@ -5,20 +5,23 @@ import (
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/cache"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/docker"
lktspec "github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// imagePull pull an image from the OCI registry to the cache.
// If the image root already is in the cache, use it, unless
// the option pull is set to true.
// if alwaysPull, then do not even bother reading locally
func imagePull(ref *reference.Spec, alwaysPull bool, cacheDir string, dockerCache bool, architecture string) (lktspec.ImageSource, error) {
// imageSource given an image ref, get a handle on the image so it can be used as a source for its configuration
// and layers. If the image root already is in the cache, use it.
// If not in cache, pull it down from the OCI registry.
// Optionally can look in docker image cache first, before falling back to linuxkit cache and OCI registry.
// Optionally can be told to alwaysPull, in which case it always pulls from the OCI registry.
// Always works for a single architecture, as we are referencing a specific image.
func imageSource(ref *reference.Spec, alwaysPull bool, cacheDir string, dockerCache bool, platform imagespec.Platform) (lktspec.ImageSource, error) {
// several possibilities:
// - alwaysPull: try to pull it down from the registry to linuxkit cache, then fail
// - !alwaysPull && dockerCache: try to read it from docker, then try linuxkit cache, then try to pull from registry, then fail
// - !alwaysPull && !dockerCache: try linuxkit cache, then try to pull from registry, then fail
// first, try docker, if that is available
if !alwaysPull && dockerCache {
if err := docker.HasImage(ref, architecture); err == nil {
if err := docker.HasImage(ref, platform.Architecture); err == nil {
return docker.NewSource(ref), nil
}
// docker is not required, so any error - image not available, no docker, whatever - just gets ignored
@@ -31,5 +34,44 @@ func imagePull(ref *reference.Spec, alwaysPull bool, cacheDir string, dockerCach
}
// if we made it here, we either did not have the image, or it was incomplete
return c.ImagePull(ref, ref.String(), architecture, alwaysPull)
if err := c.ImagePull(ref, []imagespec.Platform{platform}, alwaysPull); err != nil {
return nil, err
}
desc, err := c.FindDescriptor(ref)
if err != nil {
return nil, err
}
return c.NewSource(
ref,
&platform,
desc,
), nil
}
// indexSource given an image ref, get a handle on the index so it can be used as a source for its underlying images.
// If the index root already is in the cache, use it.
// If not in cache, pull it down from the OCI registry.
// Optionally can look in docker image cache first, before falling back to linuxkit cache and OCI registry.
// Optionally can be told to alwaysPull, in which case it always pulls from the OCI registry.
// Can provide architectures to list which ones to limit, or leave empty for all available.
func indexSource(ref *reference.Spec, alwaysPull bool, cacheDir string, platforms []imagespec.Platform) (lktspec.IndexSource, error) {
// get a reference to the local cache; we either will find the ref there or will pull to it
c, err := cache.NewProvider(cacheDir)
if err != nil {
return nil, err
}
// if we made it here, we either did not have the image, or it was incomplete
if err := c.ImagePull(ref, platforms, alwaysPull); err != nil {
return nil, err
}
desc, err := c.FindDescriptor(ref)
if err != nil {
return nil, err
}
return c.NewIndexSource(
ref,
desc,
platforms,
), nil
}

View File

@@ -13,10 +13,10 @@ import (
"github.com/containerd/containerd/reference"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/spec"
"github.com/linuxkit/linuxkit/src/cmd/linuxkit/util"
"github.com/moby/sys/capability"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/opencontainers/runtime-spec/specs-go"
log "github.com/sirupsen/logrus"
"github.com/syndtr/gocapability/capability"
"github.com/xeipuuv/gojsonschema"
"gopkg.in/yaml.v3"
)
@@ -76,10 +76,12 @@ type File struct {
// Volume is the type of a volume specification
type Volume struct {
Name string `yaml:"name" json:"name"`
Image string `yaml:"image,omitempty" json:"image,omitempty"`
ReadOnly bool `yaml:"readonly,omitempty" json:"readonly,omitempty"`
ref *reference.Spec
Name string `yaml:"name" json:"name"`
Image string `yaml:"image,omitempty" json:"image,omitempty"`
ReadOnly bool `yaml:"readonly,omitempty" json:"readonly,omitempty"`
Format string `yaml:"format,omitempty" json:"format,omitempty"`
Platforms []string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
ref *reference.Spec
}
func (v Volume) ImageRef() *reference.Spec {
@@ -446,7 +448,7 @@ func AppendConfig(m0, m1 Moby) (Moby, error) {
// NewImage validates an parses yaml or json for a Image
func NewImage(config []byte) (Image, error) {
log.Debugf("Reading label config: %s", string(config))
log.Tracef("Reading label config: %s", string(config))
mi := Image{}
@@ -781,7 +783,7 @@ func assignStringEmpty4(v1, v2, v3, v4 string) string {
func getAllCapabilities() []string {
var caps []string
for _, cap := range capability.List() {
for _, cap := range capability.ListKnown() {
caps = append(caps, "CAP_"+strings.ToUpper(cap.String()))
}
return caps

View File

@@ -43,7 +43,9 @@ var schema = `
"properties": {
"name": {"type": "string"},
"image": {"type": "string"},
"readonly": {"type": "boolean"}
"readonly": {"type": "boolean"},
"format": {"enum": ["oci","filesystem"]},
"platforms": {"$ref": "#/definitions/strings"}
}
},
"volumes": {

View File

@@ -37,7 +37,6 @@ func pkgCmd() *cobra.Command {
HashPath: hashPath,
Dirty: dirty,
Dev: devMode,
Tag: tag,
}
if cmd.Flags().Changed("disable-cache") && cmd.Flags().Changed("enable-cache") {
return errors.New("cannot set but disable-cache and enable-cache")
@@ -65,6 +64,9 @@ func pkgCmd() *cobra.Command {
if cmd.Flags().Changed("org") {
pkglibConfig.Org = &argOrg
}
if cmd.Flags().Changed("tag") {
pkglibConfig.Tag = tag
}
return nil
},
@@ -88,7 +90,7 @@ func pkgCmd() *cobra.Command {
cmd.PersistentFlags().StringVar(&argOrg, "org", piBase.Org, "Override the hub org")
cmd.PersistentFlags().StringVar(&buildYML, "build-yml", defaultPkgBuildYML, "Override the name of the yml file")
cmd.PersistentFlags().StringVar(&hash, "hash", "", "Override the image hash (default is to query git for the package's tree-sh)")
cmd.PersistentFlags().StringVar(&tag, "tag", defaultPkgTag, "Override the tag using fixed strings and/or text templates. Acceptable are .Hash for the hash")
cmd.PersistentFlags().StringVar(&tag, "tag", piBase.Tag, "Override the tag using fixed strings and/or text templates. Acceptable are .Hash for the hash")
cmd.PersistentFlags().StringVar(&hashCommit, "hash-commit", defaultPkgCommit, "Override the git commit to use for the hash")
cmd.PersistentFlags().StringVar(&hashPath, "hash-path", "", "Override the directory to use for the image hash, must be a parent of the package dir (default is to use the package dir)")
cmd.PersistentFlags().BoolVar(&dirty, "force-dirty", false, "Force the pkg(s) to be considered dirty")

View File

@@ -327,7 +327,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
case bo.pull:
// need to pull the image from the registry, else build
fmt.Fprintf(writer, "%s %s not found in local cache, trying to pull\n", ref, platform.Architecture)
if _, err := c.ImagePull(&ref, "", platform.Architecture, false); err == nil {
if err := c.ImagePull(&ref, []imagespec.Platform{platform}, false); err == nil {
fmt.Fprintf(writer, "%s pulled\n", ref)
// successfully pulled, no need to build, continue with next platform
continue
@@ -470,7 +470,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
// - potentially create a release, including push and load into docker
// create a multi-arch index
if _, err := c.IndexWrite(&ref, descs...); err != nil {
if err := c.IndexWrite(&ref, descs...); err != nil {
return err
}
}
@@ -490,7 +490,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
if err != nil {
return err
}
cacheSource := c.NewSource(&ref, platform.Architecture, desc)
cacheSource := c.NewSource(&ref, &platform, desc)
reader, err := cacheSource.V1TarReader(fmt.Sprintf("%s-%s", p.FullTag(), platform.Architecture))
if err != nil {
return fmt.Errorf("unable to get reader from cache: %v", err)
@@ -562,7 +562,7 @@ func (p Pkg) Build(bos ...BuildOpt) error {
if err != nil {
return err
}
if _, err := c.DescriptorWrite(&ref, *desc); err != nil {
if err := c.DescriptorWrite(&ref, *desc); err != nil {
return err
}
if err := c.Push(fullRelTag, "", bo.manifest, true); err != nil {
@@ -617,7 +617,7 @@ func (p Pkg) buildArch(ctx context.Context, d dockerRunner, c lktspec.CacheProvi
if err != nil {
return nil, fmt.Errorf("could not resolve references for image %s: %v", p.Tag(), err)
}
if _, err := c.ImagePull(&ref, "", arch, false); err == nil {
if err := c.ImagePull(&ref, []imagespec.Platform{{Architecture: arch, OS: "linux"}}, false); err == nil {
fmt.Fprintf(writer, "image already found %s for arch %s", ref, arch)
desc, err := c.FindDescriptor(&ref)
if err != nil {

View File

@@ -240,21 +240,24 @@ type cacheMocker struct {
hashes map[string][]byte
}
func (c *cacheMocker) ImagePull(ref *reference.Spec, trustedRef, architecture string, alwaysPull bool) (lktspec.ImageSource, error) {
func (c *cacheMocker) ImagePull(ref *reference.Spec, platforms []imagespec.Platform, alwaysPull bool) error {
if !c.enableImagePull {
return nil, errors.New("ImagePull disabled")
return errors.New("ImagePull disabled")
}
// make some random data for a layer
b := make([]byte, 256)
_, _ = rand.Read(b)
descs, err := c.imageWriteStream(bytes.NewReader(b))
if err != nil {
return nil, err
return err
}
if len(descs) != 1 {
return nil, fmt.Errorf("expected 1 descriptor, got %d", len(descs))
return fmt.Errorf("expected 1 descriptor, got %d", len(descs))
}
return c.NewSource(ref, architecture, &descs[1]), nil
if len(platforms) != 1 {
return fmt.Errorf("cache does not support multiple platforms %s", platforms)
}
return nil
}
func (c *cacheMocker) ImageInCache(ref *reference.Spec, trustedRef, architecture string) (bool, error) {
@@ -359,9 +362,9 @@ func (c *cacheMocker) imageWriteStream(r io.Reader) ([]registry.Descriptor, erro
return []registry.Descriptor{desc}, nil
}
func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.Descriptor) (lktspec.ImageSource, error) {
func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.Descriptor) error {
if !c.enableIndexWrite {
return nil, errors.New("disabled")
return errors.New("disabled")
}
image := ref.String()
im := registry.IndexManifest{
@@ -373,11 +376,11 @@ func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.De
// write the updated index, remove the old one
b, err := json.Marshal(im)
if err != nil {
return nil, fmt.Errorf("unable to marshal new index to json: %v", err)
return fmt.Errorf("unable to marshal new index to json: %v", err)
}
hash, size, err := registry.SHA256(bytes.NewReader(b))
if err != nil {
return nil, fmt.Errorf("error calculating hash of index json: %v", err)
return fmt.Errorf("error calculating hash of index json: %v", err)
}
c.assignHash(hash.String(), b)
desc := registry.Descriptor{
@@ -390,7 +393,7 @@ func (c *cacheMocker) IndexWrite(ref *reference.Spec, descriptors ...registry.De
}
c.appendImage(image, desc)
return c.NewSource(ref, "", &desc), nil
return nil
}
func (c *cacheMocker) Push(name, remoteName string, withManifest, override bool) error {
if !c.enablePush {
@@ -402,9 +405,9 @@ func (c *cacheMocker) Push(name, remoteName string, withManifest, override bool)
return nil
}
func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descriptor) (lktspec.ImageSource, error) {
func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descriptor) error {
if !c.enabledDescriptorWrite {
return nil, errors.New("descriptor disabled")
return errors.New("descriptor disabled")
}
var (
image = ref.String()
@@ -417,11 +420,11 @@ func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descrip
// write the updated index, remove the old one
b, err := json.Marshal(im)
if err != nil {
return nil, fmt.Errorf("unable to marshal new index to json: %v", err)
return fmt.Errorf("unable to marshal new index to json: %v", err)
}
hash, size, err := registry.SHA256(bytes.NewReader(b))
if err != nil {
return nil, fmt.Errorf("error calculating hash of index json: %v", err)
return fmt.Errorf("error calculating hash of index json: %v", err)
}
c.assignHash(hash.String(), b)
root := registry.Descriptor{
@@ -434,7 +437,7 @@ func (c *cacheMocker) DescriptorWrite(ref *reference.Spec, desc registry.Descrip
}
c.appendImage(image, root)
return c.NewSource(ref, "", &root), nil
return nil
}
func (c *cacheMocker) FindDescriptor(ref *reference.Spec) (*registry.Descriptor, error) {
name := ref.String()
@@ -443,8 +446,8 @@ func (c *cacheMocker) FindDescriptor(ref *reference.Spec) (*registry.Descriptor,
}
return nil, fmt.Errorf("not found %s", name)
}
func (c *cacheMocker) NewSource(ref *reference.Spec, architecture string, descriptor *registry.Descriptor) lktspec.ImageSource {
return cacheMockerSource{c, ref, architecture, descriptor}
func (c *cacheMocker) NewSource(ref *reference.Spec, platform *imagespec.Platform, descriptor *registry.Descriptor) lktspec.ImageSource {
return cacheMockerSource{c, ref, platform, descriptor}
}
func (c *cacheMocker) assignHash(hash string, b []byte) {
if c.hashes == nil {
@@ -473,10 +476,10 @@ func (c *cacheMocker) GetContent(hash v1.Hash) (io.ReadCloser, error) {
}
type cacheMockerSource struct {
c *cacheMocker
ref *reference.Spec
architecture string
descriptor *registry.Descriptor
c *cacheMocker
ref *reference.Spec
platform *imagespec.Platform
descriptor *registry.Descriptor
}
func (c cacheMockerSource) Config() (imagespec.ImageConfig, error) {

View File

@@ -20,6 +20,7 @@ import (
type pkgInfo struct {
Image string `yaml:"image"`
Org string `yaml:"org"`
Tag string `yaml:"tag,omitempty"` // default to {{.Hash}}
Dockerfile string `yaml:"dockerfile"`
Arches []string `yaml:"arches"`
ExtraSources []string `yaml:"extra-sources"`
@@ -60,6 +61,7 @@ func NewPkgInfo() pkgInfo {
return pkgInfo{
Org: "linuxkit",
Arches: []string{"amd64", "arm64"},
Tag: "{{.Hash}}",
GitRepo: "https://github.com/linuxkit/linuxkit",
Network: false,
DisableCache: false,
@@ -257,9 +259,16 @@ func NewFromConfig(cfg PkglibConfig, args ...string) ([]Pkg, error) {
}
}
}
tagTmpl := pi.Tag
if cfg.Tag != "" {
tagTmpl = cfg.Tag
}
if tagTmpl == "" {
tagTmpl = "{{.Hash}}"
}
// calculate the tag to use based on the template and the pkgHash
tmpl, err := template.New("tag").Parse(cfg.Tag)
tmpl, err := template.New("tag").Parse(tagTmpl)
if err != nil {
return nil, fmt.Errorf("invalid tag template: %v", err)
}

View File

@@ -6,6 +6,7 @@ import (
"github.com/containerd/containerd/content"
"github.com/containerd/containerd/reference"
v1 "github.com/google/go-containerregistry/pkg/v1"
imagespec "github.com/opencontainers/image-spec/specs-go/v1"
)
// CacheProvider interface for a provide of a cache.
@@ -19,7 +20,7 @@ type CacheProvider interface {
// ImagePull takes an image name and pulls it from a registry to the cache. It should be
// efficient and only write missing blobs, based on their content hash. If the ref already
// exists in the cache, it should not pull anything, unless alwaysPull is set to true.
ImagePull(ref *reference.Spec, trustedRef, architecture string, alwaysPull bool) (ImageSource, error)
ImagePull(ref *reference.Spec, platform []imagespec.Platform, alwaysPull bool) error
// ImageInCache takes an image name and checks if it exists in the cache, including checking that the given
// architecture is complete. Like ImagePull, it should be efficient and only write missing blobs, based on
// their content hash.
@@ -30,20 +31,20 @@ type CacheProvider interface {
// Cache implementation determines whether it should pull missing blobs from a remote registry.
// If the provided reference already exists and it is an index, updates the manifests in the
// existing index.
IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) (ImageSource, error)
IndexWrite(ref *reference.Spec, descriptors ...v1.Descriptor) error
// ImageLoad takes an OCI format image tar stream in the io.Reader and writes it to the cache. It should be
// efficient and only write missing blobs, based on their content hash.
ImageLoad(r io.Reader) ([]v1.Descriptor, error)
// DescriptorWrite writes a descriptor to the cache index; it validates that it has a name
// and replaces any existing one
DescriptorWrite(ref *reference.Spec, descriptors v1.Descriptor) (ImageSource, error)
DescriptorWrite(ref *reference.Spec, descriptors v1.Descriptor) error
// Push an image along with a multi-arch index from local cache to remote registry.
// name is the name as referenced in the local cache, remoteName is the name to give it remotely.
// If remoteName is empty, it is the same as name.
// if withManifest defined will push a multi-arch manifest
Push(name, remoteName string, withManifest, override bool) error
// NewSource return an ImageSource for a specific ref and architecture in the cache.
NewSource(ref *reference.Spec, architecture string, descriptor *v1.Descriptor) ImageSource
NewSource(ref *reference.Spec, platform *imagespec.Platform, descriptor *v1.Descriptor) ImageSource
// GetContent returns an io.Reader to the provided content as is, given a specific digest. It is
// up to the caller to validate it.
GetContent(hash v1.Hash) (io.ReadCloser, error)

View File

@@ -10,12 +10,12 @@ import (
// ImageSource interface to an image. It can have its config read, and a its containers
// can be read via an io.ReadCloser tar stream.
type ImageSource interface {
// Descriptor get the v1.Descriptor of the image
Descriptor() *v1.Descriptor
// Config get the config for the image
Config() (imagespec.ImageConfig, error)
// TarReader get the flattened filesystem of the image as a tar stream
TarReader() (io.ReadCloser, error)
// Descriptor get the v1.Descriptor of the image
Descriptor() *v1.Descriptor
// V1TarReader get the image as v1 tarball, also compatible with `docker load`. If name arg is not "", override name of image in tarfile from default of image.
V1TarReader(overrideName string) (io.ReadCloser, error)
// OCITarReader get the image as an OCI tarball, also compatible with `docker load`. If name arg is not "", override name of image in tarfile from default of image.
@@ -23,3 +23,14 @@ type ImageSource interface {
// SBoM get the sbom for the image, if any is available
SBoMs() ([]io.ReadCloser, error)
}
// IndexSource interface to an image. It can have its config read, and a its containers
// can be read via an io.ReadCloser tar stream.
type IndexSource interface {
// Descriptor get the v1.Descriptor of the index
Descriptor() *v1.Descriptor
// Image get image for a specific architecture
Image(platform imagespec.Platform) (ImageSource, error)
// OCITarReader get the image as an OCI tarball, also compatible with `docker load`. If name arg is not "", override name of image in tarfile from default of image.
OCITarReader(overrideName string) (io.ReadCloser, error)
}

View File

@@ -25,23 +25,38 @@ func (f *infoFormatter) Format(entry *log.Entry) ([]byte, error) {
}
// SetupLogging once the flags have been parsed, setup the logging
func SetupLogging(quiet, verbose bool) error {
func SetupLogging(quiet bool, verbose int, verboseSet bool) error {
// Set up logging
log.SetFormatter(new(infoFormatter))
log.SetLevel(log.InfoLevel)
if quiet && verbose {
if quiet && verboseSet && verbose > 0 {
return errors.New("can't set quiet and verbose flag at the same time")
}
if quiet {
switch {
case quiet, verbose == 0:
log.SetLevel(log.ErrorLevel)
}
if verbose {
case verbose == 1:
if verboseSet {
// Switch back to the standard formatter
log.SetFormatter(defaultLogFormatter)
}
log.SetLevel(log.InfoLevel)
case verbose == 2:
// Switch back to the standard formatter
log.SetFormatter(defaultLogFormatter)
log.SetLevel(log.DebugLevel)
// set go-containerregistry logging as well
ggcrlog.Warn = stdlog.New(log.StandardLogger().WriterLevel(log.WarnLevel), "", 0)
ggcrlog.Debug = stdlog.New(log.StandardLogger().WriterLevel(log.DebugLevel), "", 0)
case verbose == 3:
// Switch back to the standard formatter
log.SetFormatter(defaultLogFormatter)
log.SetLevel(log.TraceLevel)
// set go-containerregistry logging as well
ggcrlog.Warn = stdlog.New(log.StandardLogger().WriterLevel(log.WarnLevel), "", 0)
ggcrlog.Debug = stdlog.New(log.StandardLogger().WriterLevel(log.DebugLevel), "", 0)
default:
return errors.New("verbose flag can only be set to 0, 1, 2 or 3")
}
ggcrlog.Progress = stdlog.New(log.StandardLogger().WriterLevel(log.InfoLevel), "", 0)
return nil

View File

@@ -0,0 +1,3 @@
[codespell]
skip = ./.git
ignore-words-list = nd

View File

@@ -0,0 +1,6 @@
linters:
enable:
- unconvert
- unparam
- gofumpt
- errorlint

View File

@@ -0,0 +1,90 @@
# Changelog
This file documents all notable changes made to this project since the initial fork
from https://github.com/syndtr/gocapability/commit/42c35b4376354fd5.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.3.0] - 2024-09-25
### Added
* Added [ListKnown] and [ListSupported] functions. (#153)
* [LastCap] is now available on non-Linux platforms (where it returns an error). (#152)
### Changed
* [List] is now deprecated in favor of [ListKnown] and [ListSupported]. (#153)
### Fixed
* Various documentation improvements. (#151)
* Fix "generated code" comment. (#153)
## [0.2.0] - 2024-09-16
This is the first release after the move to a new home in
github.com/moby/sys/capability.
### Fixed
* Fixed URLs in documentation to reflect the new home.
## [0.1.1] - 2024-08-01
This is a maintenance release, fixing a few minor issues.
### Fixed
* Fixed future kernel compatibility, for real this time. [#11]
* Fixed [LastCap] to be a function. [#12]
## [0.1.0] - 2024-07-31
This is an initial release since the fork.
### Breaking changes
* The `CAP_LAST_CAP` variable is removed; users need to modify the code to
use [LastCap] to get the value. [#6]
* The code now requires Go >= 1.21.
### Added
* `go.mod` and `go.sum` files. [#2]
* New [LastCap] function. [#6]
* Basic CI using GHA infra. [#8], [#9]
* README and CHANGELOG. [#10]
### Fixed
* Fixed ambient capabilities error handling in [Apply]. [#3]
* Fixed future kernel compatibility. [#1]
* Fixed various linter warnings. [#4], [#7]
### Changed
* Go build tags changed from old-style (`+build`) to new Go 1.17+ style (`go:build`). [#2]
### Removed
* Removed support for capabilities v1 and v2. [#1]
* Removed init function so programs that use this package start faster. [#6]
* Removed `CAP_LAST_CAP` (use [LastCap] instead). [#6]
<!-- Doc links. -->
[Apply]: https://pkg.go.dev/github.com/moby/sys/capability#Capabilities.Apply
[LastCap]: https://pkg.go.dev/github.com/moby/sys/capability#LastCap
[List]: https://pkg.go.dev/github.com/moby/sys/capability#List
[ListKnown]: https://pkg.go.dev/github.com/moby/sys/capability#ListKnown
[ListSupported]: https://pkg.go.dev/github.com/moby/sys/capability#ListSupported
<!-- Minor releases. -->
[0.3.0]: https://github.com/moby/sys/releases/tag/capability%2Fv0.3.0
[0.2.0]: https://github.com/moby/sys/releases/tag/capability%2Fv0.2.0
[0.1.1]: https://github.com/kolyshkin/capability/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/kolyshkin/capability/compare/42c35b4376354fd5...v0.1.0
<!-- PRs in 0.1.x releases. -->
[#1]: https://github.com/kolyshkin/capability/pull/1
[#2]: https://github.com/kolyshkin/capability/pull/2
[#3]: https://github.com/kolyshkin/capability/pull/3
[#4]: https://github.com/kolyshkin/capability/pull/4
[#6]: https://github.com/kolyshkin/capability/pull/6
[#7]: https://github.com/kolyshkin/capability/pull/7
[#8]: https://github.com/kolyshkin/capability/pull/8
[#9]: https://github.com/kolyshkin/capability/pull/9
[#10]: https://github.com/kolyshkin/capability/pull/10
[#11]: https://github.com/kolyshkin/capability/pull/11
[#12]: https://github.com/kolyshkin/capability/pull/12

View File

@@ -1,3 +1,4 @@
Copyright 2023 The Capability Authors.
Copyright 2013 Suryandaru Triandana <syndtr@gmail.com>
All rights reserved.

View File

@@ -0,0 +1,13 @@
This is a fork of (apparently no longer maintained)
https://github.com/syndtr/gocapability package. It provides basic primitives to
work with [Linux capabilities][capabilities(7)].
For changes, see [CHANGELOG.md](./CHANGELOG.md).
[![Go Reference](https://pkg.go.dev/badge/github.com/moby/sys/capability/capability.svg)](https://pkg.go.dev/github.com/moby/sys/capability)
## Alternatives
* https://pkg.go.dev/kernel.org/pub/linux/libs/security/libcap/cap
[capabilities(7)]: https://man7.org/linux/man-pages/man7/capabilities.7.html

View File

@@ -1,8 +1,9 @@
// Copyright (c) 2013, Suryandaru Triandana <syndtr@gmail.com>
// Copyright 2023 The Capability Authors.
// Copyright 2013 Suryandaru Triandana <syndtr@gmail.com>
// All rights reserved.
//
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package capability provides utilities for manipulating POSIX capabilities.
package capability
@@ -60,26 +61,27 @@ type Capabilities interface {
Apply(kind CapType) error
}
// NewPid initializes a new Capabilities object for given pid when
// NewPid initializes a new [Capabilities] object for given pid when
// it is nonzero, or for the current process if pid is 0.
//
// Deprecated: Replace with NewPid2. For example, replace:
// Deprecated: Replace with [NewPid2] followed by [Capabilities.Load].
// For example, replace:
//
// c, err := NewPid(0)
// if err != nil {
// return err
// }
// c, err := NewPid(0)
// if err != nil {
// return err
// }
//
// with:
//
// c, err := NewPid2(0)
// if err != nil {
// return err
// }
// err = c.Load()
// if err != nil {
// return err
// }
// c, err := NewPid2(0)
// if err != nil {
// return err
// }
// err = c.Load()
// if err != nil {
// return err
// }
func NewPid(pid int) (Capabilities, error) {
c, err := newPid(pid)
if err != nil {
@@ -89,33 +91,34 @@ func NewPid(pid int) (Capabilities, error) {
return c, err
}
// NewPid2 initializes a new Capabilities object for given pid when
// it is nonzero, or for the current process if pid is 0. This
// NewPid2 initializes a new [Capabilities] object for given pid when
// it is nonzero, or for the current process if pid is 0. This
// does not load the process's current capabilities; to do that you
// must call Load explicitly.
// must call [Capabilities.Load] explicitly.
func NewPid2(pid int) (Capabilities, error) {
return newPid(pid)
}
// NewFile initializes a new Capabilities object for given file path.
//
// Deprecated: Replace with NewFile2. For example, replace:
// Deprecated: Replace with [NewFile2] followed by [Capabilities.Load].
// For example, replace:
//
// c, err := NewFile(path)
// if err != nil {
// return err
// }
// c, err := NewFile(path)
// if err != nil {
// return err
// }
//
// with:
//
// c, err := NewFile2(path)
// if err != nil {
// return err
// }
// err = c.Load()
// if err != nil {
// return err
// }
// c, err := NewFile2(path)
// if err != nil {
// return err
// }
// err = c.Load()
// if err != nil {
// return err
// }
func NewFile(path string) (Capabilities, error) {
c, err := newFile(path)
if err != nil {
@@ -125,9 +128,17 @@ func NewFile(path string) (Capabilities, error) {
return c, err
}
// NewFile2 creates a new initialized Capabilities object for given
// file path. This does not load the process's current capabilities;
// to do that you must call Load explicitly.
// NewFile2 creates a new initialized [Capabilities] object for given
// file path. This does not load the process's current capabilities;
// to do that you must call [Capabilities.Load] explicitly.
func NewFile2(path string) (Capabilities, error) {
return newFile(path)
}
// LastCap returns highest valid capability of the running kernel,
// or an error if it can not be obtained.
//
// See also: [ListSupported].
func LastCap() (Cap, error) {
return lastCap()
}

View File

@@ -1,8 +1,9 @@
// Copyright (c) 2013, Suryandaru Triandana <syndtr@gmail.com>
// Copyright 2023 The Capability Authors.
// Copyright 2013 Suryandaru Triandana <syndtr@gmail.com>
// All rights reserved.
//
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package capability
@@ -12,62 +13,53 @@ import (
"fmt"
"io"
"os"
"strconv"
"strings"
"sync"
"syscall"
)
var errUnknownVers = errors.New("unknown capability version")
const (
linuxCapVer1 = 0x19980330
linuxCapVer2 = 0x20071026
linuxCapVer1 = 0x19980330 // No longer supported.
linuxCapVer2 = 0x20071026 // No longer supported.
linuxCapVer3 = 0x20080522
)
var (
capVers uint32
capLastCap Cap
)
func init() {
var hdr capHeader
capget(&hdr, nil)
capVers = hdr.version
if initLastCap() == nil {
CAP_LAST_CAP = capLastCap
if capLastCap > 31 {
capUpperMask = (uint32(1) << (uint(capLastCap) - 31)) - 1
} else {
capUpperMask = 0
}
}
}
func initLastCap() error {
if capLastCap != 0 {
return nil
}
var lastCap = sync.OnceValues(func() (Cap, error) {
f, err := os.Open("/proc/sys/kernel/cap_last_cap")
if err != nil {
return err
return 0, err
}
defer f.Close()
var b []byte = make([]byte, 11)
_, err = f.Read(b)
buf := make([]byte, 11)
l, err := f.Read(buf)
f.Close()
if err != nil {
return err
return 0, err
}
buf = buf[:l]
fmt.Sscanf(string(b), "%d", &capLastCap)
last, err := strconv.Atoi(strings.TrimSpace(string(buf)))
if err != nil {
return 0, err
}
return Cap(last), nil
})
return nil
func capUpperMask() uint32 {
last, err := lastCap()
if err != nil || last < 32 {
return 0
}
return (uint32(1) << (uint(last) - 31)) - 1
}
func mkStringCap(c Capabilities, which CapType) (ret string) {
for i, first := Cap(0), true; i <= CAP_LAST_CAP; i++ {
last, err := lastCap()
if err != nil {
return ""
}
for i, first := Cap(0), true; i <= last; i++ {
if !c.Get(which, i) {
continue
}
@@ -98,138 +90,33 @@ func mkString(c Capabilities, max CapType) (ret string) {
return
}
func newPid(pid int) (c Capabilities, err error) {
switch capVers {
case linuxCapVer1:
p := new(capsV1)
p.hdr.version = capVers
p.hdr.pid = int32(pid)
c = p
case linuxCapVer2, linuxCapVer3:
p := new(capsV3)
p.hdr.version = capVers
p.hdr.pid = int32(pid)
c = p
default:
err = errUnknownVers
var capVersion = sync.OnceValues(func() (uint32, error) {
var hdr capHeader
err := capget(&hdr, nil)
return hdr.version, err
})
func newPid(pid int) (c Capabilities, retErr error) {
ver, err := capVersion()
if err != nil {
retErr = fmt.Errorf("unable to get capability version from the kernel: %w", err)
return
}
return
}
type capsV1 struct {
hdr capHeader
data capData
}
func (c *capsV1) Get(which CapType, what Cap) bool {
if what > 32 {
return false
}
switch which {
case EFFECTIVE:
return (1<<uint(what))&c.data.effective != 0
case PERMITTED:
return (1<<uint(what))&c.data.permitted != 0
case INHERITABLE:
return (1<<uint(what))&c.data.inheritable != 0
}
return false
}
func (c *capsV1) getData(which CapType) (ret uint32) {
switch which {
case EFFECTIVE:
ret = c.data.effective
case PERMITTED:
ret = c.data.permitted
case INHERITABLE:
ret = c.data.inheritable
switch ver {
case linuxCapVer1, linuxCapVer2:
retErr = errors.New("old/unsupported capability version (kernel older than 2.6.26?)")
default:
// Either linuxCapVer3, or an unknown/future version (such as v4).
// In the latter case, we fall back to v3 as the latest version known
// to this package, as kernel should be backward-compatible to v3.
p := new(capsV3)
p.hdr.version = linuxCapVer3
p.hdr.pid = int32(pid)
c = p
}
return
}
func (c *capsV1) Empty(which CapType) bool {
return c.getData(which) == 0
}
func (c *capsV1) Full(which CapType) bool {
return (c.getData(which) & 0x7fffffff) == 0x7fffffff
}
func (c *capsV1) Set(which CapType, caps ...Cap) {
for _, what := range caps {
if what > 32 {
continue
}
if which&EFFECTIVE != 0 {
c.data.effective |= 1 << uint(what)
}
if which&PERMITTED != 0 {
c.data.permitted |= 1 << uint(what)
}
if which&INHERITABLE != 0 {
c.data.inheritable |= 1 << uint(what)
}
}
}
func (c *capsV1) Unset(which CapType, caps ...Cap) {
for _, what := range caps {
if what > 32 {
continue
}
if which&EFFECTIVE != 0 {
c.data.effective &= ^(1 << uint(what))
}
if which&PERMITTED != 0 {
c.data.permitted &= ^(1 << uint(what))
}
if which&INHERITABLE != 0 {
c.data.inheritable &= ^(1 << uint(what))
}
}
}
func (c *capsV1) Fill(kind CapType) {
if kind&CAPS == CAPS {
c.data.effective = 0x7fffffff
c.data.permitted = 0x7fffffff
c.data.inheritable = 0
}
}
func (c *capsV1) Clear(kind CapType) {
if kind&CAPS == CAPS {
c.data.effective = 0
c.data.permitted = 0
c.data.inheritable = 0
}
}
func (c *capsV1) StringCap(which CapType) (ret string) {
return mkStringCap(c, which)
}
func (c *capsV1) String() (ret string) {
return mkString(c, BOUNDING)
}
func (c *capsV1) Load() (err error) {
return capget(&c.hdr, &c.data)
}
func (c *capsV1) Apply(kind CapType) error {
if kind&CAPS == CAPS {
return capset(&c.hdr, &c.data)
}
return nil
}
type capsV3 struct {
hdr capHeader
data [2]capData
@@ -292,7 +179,8 @@ func (c *capsV3) Full(which CapType) bool {
if (data[0] & 0xffffffff) != 0xffffffff {
return false
}
return (data[1] & capUpperMask) == capUpperMask
mask := capUpperMask()
return (data[1] & mask) == mask
}
func (c *capsV3) Set(which CapType, caps ...Cap) {
@@ -401,15 +289,12 @@ func (c *capsV3) Load() (err error) {
return
}
var status_path string
if c.hdr.pid == 0 {
status_path = fmt.Sprintf("/proc/self/status")
} else {
status_path = fmt.Sprintf("/proc/%d/status", c.hdr.pid)
path := "/proc/self/status"
if c.hdr.pid != 0 {
path = fmt.Sprintf("/proc/%d/status", c.hdr.pid)
}
f, err := os.Open(status_path)
f, err := os.Open(path)
if err != nil {
return
}
@@ -423,11 +308,17 @@ func (c *capsV3) Load() (err error) {
break
}
if strings.HasPrefix(line, "CapB") {
fmt.Sscanf(line[4:], "nd: %08x%08x", &c.bounds[1], &c.bounds[0])
_, err = fmt.Sscanf(line[4:], "nd: %08x%08x", &c.bounds[1], &c.bounds[0])
if err != nil {
break
}
continue
}
if strings.HasPrefix(line, "CapA") {
fmt.Sscanf(line[4:], "mb: %08x%08x", &c.ambient[1], &c.ambient[0])
_, err = fmt.Sscanf(line[4:], "mb: %08x%08x", &c.ambient[1], &c.ambient[0])
if err != nil {
break
}
continue
}
}
@@ -437,6 +328,10 @@ func (c *capsV3) Load() (err error) {
}
func (c *capsV3) Apply(kind CapType) (err error) {
last, err := LastCap()
if err != nil {
return err
}
if kind&BOUNDS == BOUNDS {
var data [2]capData
err = capget(&c.hdr, &data[0])
@@ -444,14 +339,14 @@ func (c *capsV3) Apply(kind CapType) (err error) {
return
}
if (1<<uint(CAP_SETPCAP))&data[0].effective != 0 {
for i := Cap(0); i <= CAP_LAST_CAP; i++ {
for i := Cap(0); i <= last; i++ {
if c.Get(BOUNDING, i) {
continue
}
err = prctl(syscall.PR_CAPBSET_DROP, uintptr(i), 0, 0, 0)
if err != nil {
// Ignore EINVAL since the capability may not be supported in this system.
if errno, ok := err.(syscall.Errno); ok && errno == syscall.EINVAL {
if err == syscall.EINVAL { //nolint:errorlint // Errors from syscall are bare.
err = nil
continue
}
@@ -469,16 +364,19 @@ func (c *capsV3) Apply(kind CapType) (err error) {
}
if kind&AMBS == AMBS {
for i := Cap(0); i <= CAP_LAST_CAP; i++ {
for i := Cap(0); i <= last; i++ {
action := pr_CAP_AMBIENT_LOWER
if c.Get(AMBIENT, i) {
action = pr_CAP_AMBIENT_RAISE
}
err := prctl(pr_CAP_AMBIENT, action, uintptr(i), 0, 0)
// Ignore EINVAL as not supported on kernels before 4.3
if errno, ok := err.(syscall.Errno); ok && errno == syscall.EINVAL {
err = nil
continue
err = prctl(pr_CAP_AMBIENT, action, uintptr(i), 0, 0)
if err != nil {
// Ignore EINVAL as not supported on kernels before 4.3
if err == syscall.EINVAL { //nolint:errorlint // Errors from syscall are bare.
err = nil
continue
}
return
}
}
}
@@ -547,7 +445,8 @@ func (c *capsFile) Full(which CapType) bool {
if (data[0] & 0xffffffff) != 0xffffffff {
return false
}
return (data[1] & capUpperMask) == capUpperMask
mask := capUpperMask()
return (data[1] & mask) == mask
}
func (c *capsFile) Set(which CapType, caps ...Cap) {

View File

@@ -0,0 +1,26 @@
// Copyright 2023 The Capability Authors.
// Copyright 2013 Suryandaru Triandana <syndtr@gmail.com>
// All rights reserved.
//
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build !linux
package capability
import "errors"
var errNotSup = errors.New("not supported")
func newPid(_ int) (Capabilities, error) {
return nil, errNotSup
}
func newFile(_ string) (Capabilities, error) {
return nil, errNotSup
}
func lastCap() (Cap, error) {
return -1, errNotSup
}

View File

@@ -1,11 +1,14 @@
// Copyright (c) 2013, Suryandaru Triandana <syndtr@gmail.com>
// Copyright 2024 The Capability Authors.
// Copyright 2013 Suryandaru Triandana <syndtr@gmail.com>
// All rights reserved.
//
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package capability
import "slices"
type CapType uint
func (c CapType) String() string {
@@ -301,9 +304,27 @@ const (
CAP_CHECKPOINT_RESTORE = Cap(40)
)
var (
// Highest valid capability of the running kernel.
CAP_LAST_CAP = Cap(63)
// List returns the list of all capabilities known to the package.
//
// Deprecated: use [ListKnown] or [ListSupported] instead.
func List() []Cap {
return ListKnown()
}
capUpperMask = ^uint32(0)
)
// ListKnown returns the list of all capabilities known to the package.
func ListKnown() []Cap {
return list()
}
// ListSupported retuns the list of all capabilities known to the package,
// except those that are not supported by the currently running Linux kernel.
func ListSupported() ([]Cap, error) {
last, err := LastCap()
if err != nil {
return nil, err
}
return slices.DeleteFunc(list(), func(c Cap) bool {
// Remove caps not supported by the kernel.
return c > last
}), nil
}

View File

@@ -1,4 +1,4 @@
// generated file; DO NOT EDIT - use go generate in directory with source
// Code generated by go generate; DO NOT EDIT.
package capability
@@ -90,8 +90,7 @@ func (c Cap) String() string {
return "unknown"
}
// List returns list of all supported capabilities
func List() []Cap {
func list() []Cap {
return []Cap{
CAP_CHOWN,
CAP_DAC_OVERRIDE,

View File

@@ -1,8 +1,9 @@
// Copyright (c) 2013, Suryandaru Triandana <syndtr@gmail.com>
// Copyright 2024 The Capability Authors.
// Copyright 2013 Suryandaru Triandana <syndtr@gmail.com>
// All rights reserved.
//
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package capability
@@ -79,9 +80,7 @@ type vfscapData struct {
version int8
}
var (
_vfsXattrName *byte
)
var _vfsXattrName *byte
func init() {
_vfsXattrName, _ = syscall.BytePtrFromString(vfsXattrName)

View File

@@ -1,19 +0,0 @@
// Copyright (c) 2013, Suryandaru Triandana <syndtr@gmail.com>
// All rights reserved.
//
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
// +build !linux
package capability
import "errors"
func newPid(pid int) (Capabilities, error) {
return nil, errors.New("not supported")
}
func newFile(path string) (Capabilities, error) {
return nil, errors.New("not supported")
}

View File

@@ -511,6 +511,9 @@ github.com/moby/locker
## explicit; go 1.19
github.com/moby/patternmatcher
github.com/moby/patternmatcher/ignorefile
# github.com/moby/sys/capability v0.3.0
## explicit; go 1.21
github.com/moby/sys/capability
# github.com/moby/sys/signal v0.7.0
## explicit; go 1.16
github.com/moby/sys/signal
@@ -622,9 +625,6 @@ github.com/stretchr/testify/require
# github.com/surma/gocpio v1.0.2-0.20160926205914-fcb68777e7dc
## explicit
github.com/surma/gocpio
# github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635
## explicit
github.com/syndtr/gocapability/capability
# github.com/tonistiigi/fsutil v0.0.0-20240424095704-91a3fc46842c
## explicit; go 1.20
github.com/tonistiigi/fsutil

View File

@@ -0,0 +1 @@
FROM alpine:3.20

View File

@@ -0,0 +1,3 @@
org: linuxkit
image: image-with-tag
tag: file

View File

@@ -0,0 +1,16 @@
#!/bin/sh
# SUMMARY: Check that tar output format build is reproducible
# LABELS:
set -e
# Source libraries. Uncomment if needed/defined
#. "${RT_LIB}"
. "${RT_PROJECT_ROOT}/_lib/lib.sh"
linuxkit pkg build --force .
# just run docker image inspect; if it does not exist, it will error out
linuxkit cache ls 2>&1 | grep 'linuxkit/image-with-tag:file'
exit 0

View File

@@ -0,0 +1 @@
FROM alpine:3.20

View File

@@ -0,0 +1,2 @@
org: linuxkit
image: image-with-tag

View File

@@ -0,0 +1,16 @@
#!/bin/sh
# SUMMARY: Check that tar output format build is reproducible
# LABELS:
set -e
# Source libraries. Uncomment if needed/defined
#. "${RT_LIB}"
. "${RT_PROJECT_ROOT}/_lib/lib.sh"
linuxkit pkg build --force --tag cli .
# just run docker image inspect; if it does not exist, it will error out
linuxkit cache ls 2>&1 | grep 'linuxkit/image-with-tag:cli'
exit 0

View File

@@ -0,0 +1 @@
FROM alpine:3.20

View File

@@ -0,0 +1,3 @@
org: linuxkit
image: image-with-tag
tag: file-new

View File

@@ -0,0 +1,22 @@
#!/bin/sh
# SUMMARY: Check that tar output format build is reproducible
# LABELS:
set -e
# Source libraries. Uncomment if needed/defined
#. "${RT_LIB}"
. "${RT_PROJECT_ROOT}/_lib/lib.sh"
linuxkit pkg build --force --tag cli .
# just run docker image inspect; if it does not exist, it will error out
linuxkit cache ls 2>&1 | grep 'linuxkit/image-with-tag:cli'
# specifically, the `file` tag should not exist, so check that it does not exist
if linuxkit cache ls 2>&1 | grep 'linuxkit/image-with-tag:file-new'; then
echo "ERROR: image with tag 'file-new' should not exist"
exit 1
fi
exit 0

View File

@@ -21,7 +21,7 @@ logfile=$(mktemp)
# do not include the sbom, because the SBoM unique IDs per file/package are *not* deterministic,
# (currently based upon syft), and thus will make the file non-reproducible
linuxkit build --no-sbom --format tar --o "${NAME}-1.tar" ./test1.yml
linuxkit build -v --no-sbom --format tar --input-tar "${NAME}-1.tar" --o "${NAME}-2.tar" ./test2.yml 2>&1 | tee ${logfile}
linuxkit build -v 2 --no-sbom --format tar --input-tar "${NAME}-1.tar" --o "${NAME}-2.tar" ./test2.yml 2>&1 | tee ${logfile}
# the logfile should indicate which parts were copied and which not
# we only know this because we built the test2.yml manually