Update vendor

This commit is contained in:
Ettore Di Giacinto
2020-12-14 19:20:35 +01:00
parent 0b9b3c0488
commit 193f6872a0
311 changed files with 33633 additions and 10509 deletions

View File

@@ -0,0 +1,280 @@
# `tarball`
[![GoDoc](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/tarball?status.svg)](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/tarball)
This package produces tarballs that can consumed via `docker load`. Note
that this is a _different_ format from the [`legacy`](/pkg/legacy/tarball)
tarballs that are produced by `docker save`, but this package is still able to
read the legacy tarballs produced by `docker save`.
## Usage
```go
package main
import (
"os"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/tarball"
)
func main() {
// Read a tarball from os.Args[1] that contains ubuntu.
tag, err := name.NewTag("ubuntu")
if err != nil {
panic(err)
}
img, err := tarball.ImageFromPath(os.Args[1], &tag)
if err != nil {
panic(err)
}
// Write that tarball to os.Args[2] with a different tag.
newTag, err := name.NewTag("ubuntu:newest")
if err != nil {
panic(err)
}
f, err := os.Create(os.Args[2])
if err != nil {
panic(err)
}
defer f.Close()
if err := tarball.Write(newTag, img, f); err != nil {
panic(err)
}
}
```
## Structure
<p align="center">
<img src="/images/tarball.dot.svg" />
</p>
Let's look at what happens when we write out a tarball:
### `ubuntu:latest`
```
$ crane pull ubuntu ubuntu.tar && mkdir ubuntu && tar xf ubuntu.tar -C ubuntu && rm ubuntu.tar
$ tree ubuntu/
ubuntu/
├── 423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954.tar.gz
├── b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7.tar.gz
├── de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d.tar.gz
├── f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b.tar.gz
├── manifest.json
└── sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c
0 directories, 6 files
```
There are a couple interesting files here.
`manifest.json` is the entrypoint: a list of [`tarball.Descriptor`s](https://godoc.org/github.com/google/go-containerregistry/pkg/v1/tarball#Descriptor)
that describe the images contained in this tarball.
For each image, this has the `RepoTags` (how it was pulled), a `Config` file
that points to the image's config file, a list of `Layers`, and (optionally)
`LayerSources`.
```
$ jq < ubuntu/manifest.json
[
{
"Config": "sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c",
"RepoTags": [
"ubuntu"
],
"Layers": [
"423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954.tar.gz",
"de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d.tar.gz",
"f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b.tar.gz",
"b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7.tar.gz"
]
}
]
```
The config file and layers are exactly what you would expect, and match the
registry representations of the same artifacts. You'll notice that the
`manifest.json` contains similar information as the registry manifest, but isn't
quite the same:
```
$ crane manifest ubuntu@sha256:0925d086715714114c1988f7c947db94064fd385e171a63c07730f1fa014e6f9
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 3408,
"digest": "sha256:72300a873c2ca11c70d0c8642177ce76ff69ae04d61a5813ef58d40ff66e3e7c"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 26692096,
"digest": "sha256:423ae2b273f4c17ceee9e8482fa8d071d90c7d052ae208e1fe4963fceb3d6954"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 35365,
"digest": "sha256:de83a2304fa1f7c4a13708a0d15b9704f5945c2be5cbb2b3ed9b2ccb718d0b3d"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 852,
"digest": "sha256:f9a83bce3af0648efaa60b9bb28225b09136d2d35d0bed25ac764297076dec1b"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 163,
"digest": "sha256:b6b53be908de2c0c78070fff0a9f04835211b3156c4e73785747af365e71a0d7"
}
]
}
```
This makes it difficult to maintain image digests when roundtripping images
through the tarball format, so it's not a great format if you care about
provenance.
The ubuntu example didn't have any `LayerSources` -- let's look at another image
that does.
### `hello-world:nanoserver`
```
$ crane pull hello-world:nanoserver@sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b nanoserver.tar
$ mkdir nanoserver && tar xf nanoserver.tar -C nanoserver && rm nanoserver.tar
$ tree nanoserver/
nanoserver/
├── 10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0.tar.gz
├── a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053.tar.gz
├── be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a.tar.gz
├── manifest.json
└── sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6
0 directories, 5 files
$ jq < nanoserver/manifest.json
[
{
"Config": "sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6",
"RepoTags": [
"index.docker.io/library/hello-world:i-was-a-digest"
],
"Layers": [
"a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053.tar.gz",
"be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a.tar.gz",
"10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0.tar.gz"
],
"LayerSources": {
"sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e": {
"mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
"size": 101145811,
"digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
"urls": [
"https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
]
}
}
}
]
```
A couple things to note about this `manifest.json` versus the other:
* The `RepoTags` field is a bit weird here. `hello-world` is a multi-platform
image, so We had to pull this image by digest, since we're (I'm) on
amd64/linux and wanted to grab a windows image. Since the tarball format
expects a tag under `RepoTags`, and we didn't pull by tag, we replace the
digest with a sentinel `i-was-a-digest` "tag" to appease docker.
* The `LayerSources` has enough information to reconstruct the foreign layers
pointer when pushing/pulling from the registry. For legal reasons, microsoft
doesn't want anyone but them to serve windows base images, so the mediaType
here indicates a "foreign" or "non-distributable" layer with an URL for where
you can download it from microsoft (see the [OCI
image-spec](https://github.com/opencontainers/image-spec/blob/master/layer.md#non-distributable-layers)).
We can look at what's in the registry to explain both of these things:
```
$ crane manifest hello-world:nanoserver | jq .
{
"manifests": [
{
"digest": "sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "windows",
"os.version": "10.0.17763.1040"
},
"size": 1124
}
],
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"schemaVersion": 2
}
# Note the media type and "urls" field.
$ crane manifest hello-world:nanoserver@sha256:63c287625c2b0b72900e562de73c0e381472a83b1b39217aef3856cd398eca0b | jq .
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1721,
"digest": "sha256:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
"size": 101145811,
"digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
"urls": [
"https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
]
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1669,
"digest": "sha256:be21f08f670160cbae227e3053205b91d6bfa3de750b90c7e00bd2c511ccb63a"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 949,
"digest": "sha256:10d1439be4eb8819987ec2e9c140d44d74d6b42a823d57fe1953bd99948e1bc0"
}
]
}
```
The `LayerSources` map is keyed by the diffid. Note that `sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e` matches the first layer in the config file:
```
$ jq '.[0].LayerSources' < nanoserver/manifest.json
{
"sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e": {
"mediaType": "application/vnd.docker.image.rootfs.foreign.diff.tar.gzip",
"size": 101145811,
"digest": "sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053",
"urls": [
"https://mcr.microsoft.com/v2/windows/nanoserver/blobs/sha256:a35da61c356213336e646756218539950461ff2bf096badf307a23add6e70053"
]
}
}
$ jq < nanoserver/sha256\:bc5d255ea81f83c8c38a982a6d29a6f2198427d258aea5f166e49856896b2da6 | jq .rootfs
{
"type": "layers",
"diff_ids": [
"sha256:26fd2d9d4c64a4f965bbc77939a454a31b607470f430b5d69fc21ded301fa55e",
"sha256:601cf7d78c62e4b4d32a7bbf96a17606a9cea5bd9d22ffa6f34aa431d056b0e8",
"sha256:a1e1a3bf6529adcce4d91dce2cad86c2604a66b507ccbc4d2239f3da0ec5aab9"
]
}
```

View File

@@ -0,0 +1,17 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package tarball provides facilities for reading/writing v1.Images from/to
// a tarball on-disk.
package tarball

View File

@@ -0,0 +1,401 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package tarball
import (
"archive/tar"
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"io/ioutil"
"os"
"sync"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
"github.com/google/go-containerregistry/pkg/v1/types"
"github.com/google/go-containerregistry/pkg/v1/v1util"
)
type image struct {
opener Opener
manifest *Manifest
config []byte
imgDescriptor *Descriptor
tag *name.Tag
}
type uncompressedImage struct {
*image
}
type compressedImage struct {
*image
manifestLock sync.Mutex // Protects manifest
manifest *v1.Manifest
}
var _ partial.UncompressedImageCore = (*uncompressedImage)(nil)
var _ partial.CompressedImageCore = (*compressedImage)(nil)
// Opener is a thunk for opening a tar file.
type Opener func() (io.ReadCloser, error)
func pathOpener(path string) Opener {
return func() (io.ReadCloser, error) {
return os.Open(path)
}
}
// ImageFromPath returns a v1.Image from a tarball located on path.
func ImageFromPath(path string, tag *name.Tag) (v1.Image, error) {
return Image(pathOpener(path), tag)
}
// Image exposes an image from the tarball at the provided path.
func Image(opener Opener, tag *name.Tag) (v1.Image, error) {
img := &image{
opener: opener,
tag: tag,
}
if err := img.loadTarDescriptorAndConfig(); err != nil {
return nil, err
}
// Peek at the first layer and see if it's compressed.
if len(img.imgDescriptor.Layers) > 0 {
compressed, err := img.areLayersCompressed()
if err != nil {
return nil, err
}
if compressed {
c := compressedImage{
image: img,
}
return partial.CompressedToImage(&c)
}
}
uc := uncompressedImage{
image: img,
}
return partial.UncompressedToImage(&uc)
}
func (i *image) MediaType() (types.MediaType, error) {
return types.DockerManifestSchema2, nil
}
// Descriptor stores the manifest data for a single image inside a `docker save` tarball.
type Descriptor struct {
Config string
RepoTags []string
Layers []string
// Tracks foreign layer info. Key is DiffID.
LayerSources map[v1.Hash]v1.Descriptor `json:",omitempty"`
}
// Manifest represents the manifests of all images as the `manifest.json` file in a `docker save` tarball.
type Manifest []Descriptor
func (m Manifest) findDescriptor(tag *name.Tag) (*Descriptor, error) {
if tag == nil {
if len(m) != 1 {
return nil, errors.New("tarball must contain only a single image to be used with tarball.Image")
}
return &(m)[0], nil
}
for _, img := range m {
for _, tagStr := range img.RepoTags {
repoTag, err := name.NewTag(tagStr)
if err != nil {
return nil, err
}
// Compare the resolved names, since there are several ways to specify the same tag.
if repoTag.Name() == tag.Name() {
return &img, nil
}
}
}
return nil, fmt.Errorf("tag %s not found in tarball", tag)
}
func (i *image) areLayersCompressed() (bool, error) {
if len(i.imgDescriptor.Layers) == 0 {
return false, errors.New("0 layers found in image")
}
layer := i.imgDescriptor.Layers[0]
blob, err := extractFileFromTar(i.opener, layer)
if err != nil {
return false, err
}
defer blob.Close()
return v1util.IsGzipped(blob)
}
func (i *image) loadTarDescriptorAndConfig() error {
m, err := extractFileFromTar(i.opener, "manifest.json")
if err != nil {
return err
}
defer m.Close()
if err := json.NewDecoder(m).Decode(&i.manifest); err != nil {
return err
}
if i.manifest == nil {
return errors.New("no valid manifest.json in tarball")
}
i.imgDescriptor, err = i.manifest.findDescriptor(i.tag)
if err != nil {
return err
}
cfg, err := extractFileFromTar(i.opener, i.imgDescriptor.Config)
if err != nil {
return err
}
defer cfg.Close()
i.config, err = ioutil.ReadAll(cfg)
if err != nil {
return err
}
return nil
}
func (i *image) RawConfigFile() ([]byte, error) {
return i.config, nil
}
// tarFile represents a single file inside a tar. Closing it closes the tar itself.
type tarFile struct {
io.Reader
io.Closer
}
func extractFileFromTar(opener Opener, filePath string) (io.ReadCloser, error) {
f, err := opener()
if err != nil {
return nil, err
}
close := true
defer func() {
if close {
f.Close()
}
}()
tf := tar.NewReader(f)
for {
hdr, err := tf.Next()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
if hdr.Name == filePath {
close = false
return tarFile{
Reader: tf,
Closer: f,
}, nil
}
}
return nil, fmt.Errorf("file %s not found in tar", filePath)
}
// uncompressedLayerFromTarball implements partial.UncompressedLayer
type uncompressedLayerFromTarball struct {
diffID v1.Hash
mediaType types.MediaType
opener Opener
filePath string
}
// foreignUncompressedLayer implements partial.UncompressedLayer but returns
// a custom descriptor. This allows the foreign layer URLs to be included in
// the generated image manifest for uncompressed layers.
type foreignUncompressedLayer struct {
uncompressedLayerFromTarball
desc v1.Descriptor
}
func (fl *foreignUncompressedLayer) Descriptor() (*v1.Descriptor, error) {
return &fl.desc, nil
}
// DiffID implements partial.UncompressedLayer
func (ulft *uncompressedLayerFromTarball) DiffID() (v1.Hash, error) {
return ulft.diffID, nil
}
// Uncompressed implements partial.UncompressedLayer
func (ulft *uncompressedLayerFromTarball) Uncompressed() (io.ReadCloser, error) {
return extractFileFromTar(ulft.opener, ulft.filePath)
}
func (ulft *uncompressedLayerFromTarball) MediaType() (types.MediaType, error) {
return ulft.mediaType, nil
}
func (i *uncompressedImage) LayerByDiffID(h v1.Hash) (partial.UncompressedLayer, error) {
cfg, err := partial.ConfigFile(i)
if err != nil {
return nil, err
}
for idx, diffID := range cfg.RootFS.DiffIDs {
if diffID == h {
// Technically the media type should be 'application/tar' but given that our
// v1.Layer doesn't force consumers to care about whether the layer is compressed
// we should be fine returning the DockerLayer media type
mt := types.DockerLayer
if bd, ok := i.imgDescriptor.LayerSources[h]; ok {
// Overwrite the mediaType for foreign layers.
return &foreignUncompressedLayer{
uncompressedLayerFromTarball: uncompressedLayerFromTarball{
diffID: diffID,
mediaType: bd.MediaType,
opener: i.opener,
filePath: i.imgDescriptor.Layers[idx],
},
desc: bd,
}, nil
}
return &uncompressedLayerFromTarball{
diffID: diffID,
mediaType: mt,
opener: i.opener,
filePath: i.imgDescriptor.Layers[idx],
}, nil
}
}
return nil, fmt.Errorf("diff id %q not found", h)
}
func (c *compressedImage) Manifest() (*v1.Manifest, error) {
c.manifestLock.Lock()
defer c.manifestLock.Unlock()
if c.manifest != nil {
return c.manifest, nil
}
b, err := c.RawConfigFile()
if err != nil {
return nil, err
}
cfgHash, cfgSize, err := v1.SHA256(bytes.NewReader(b))
if err != nil {
return nil, err
}
c.manifest = &v1.Manifest{
SchemaVersion: 2,
MediaType: types.DockerManifestSchema2,
Config: v1.Descriptor{
MediaType: types.DockerConfigJSON,
Size: cfgSize,
Digest: cfgHash,
},
}
for i, p := range c.imgDescriptor.Layers {
cfg, err := partial.ConfigFile(c)
if err != nil {
return nil, err
}
diffid := cfg.RootFS.DiffIDs[i]
if d, ok := c.imgDescriptor.LayerSources[diffid]; ok {
// If it's a foreign layer, just append the descriptor so we can avoid
// reading the entire file.
c.manifest.Layers = append(c.manifest.Layers, d)
} else {
l, err := extractFileFromTar(c.opener, p)
if err != nil {
return nil, err
}
defer l.Close()
sha, size, err := v1.SHA256(l)
if err != nil {
return nil, err
}
c.manifest.Layers = append(c.manifest.Layers, v1.Descriptor{
MediaType: types.DockerLayer,
Size: size,
Digest: sha,
})
}
}
return c.manifest, nil
}
func (c *compressedImage) RawManifest() ([]byte, error) {
return partial.RawManifest(c)
}
// compressedLayerFromTarball implements partial.CompressedLayer
type compressedLayerFromTarball struct {
desc v1.Descriptor
opener Opener
filePath string
}
// Digest implements partial.CompressedLayer
func (clft *compressedLayerFromTarball) Digest() (v1.Hash, error) {
return clft.desc.Digest, nil
}
// Compressed implements partial.CompressedLayer
func (clft *compressedLayerFromTarball) Compressed() (io.ReadCloser, error) {
return extractFileFromTar(clft.opener, clft.filePath)
}
// MediaType implements partial.CompressedLayer
func (clft *compressedLayerFromTarball) MediaType() (types.MediaType, error) {
return clft.desc.MediaType, nil
}
// Size implements partial.CompressedLayer
func (clft *compressedLayerFromTarball) Size() (int64, error) {
return clft.desc.Size, nil
}
func (c *compressedImage) LayerByDigest(h v1.Hash) (partial.CompressedLayer, error) {
m, err := c.Manifest()
if err != nil {
return nil, err
}
for i, l := range m.Layers {
if l.Digest == h {
fp := c.imgDescriptor.Layers[i]
return &compressedLayerFromTarball{
desc: l,
opener: c.opener,
filePath: fp,
}, nil
}
}
return nil, fmt.Errorf("blob %v not found", h)
}

View File

@@ -0,0 +1,170 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package tarball
import (
"bytes"
"compress/gzip"
"io"
"io/ioutil"
"os"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/types"
"github.com/google/go-containerregistry/pkg/v1/v1util"
)
type layer struct {
digest v1.Hash
diffID v1.Hash
size int64
opener Opener
compressed bool
compression int
}
func (l *layer) Digest() (v1.Hash, error) {
return l.digest, nil
}
func (l *layer) DiffID() (v1.Hash, error) {
return l.diffID, nil
}
func (l *layer) Compressed() (io.ReadCloser, error) {
rc, err := l.opener()
if err == nil && !l.compressed {
return v1util.GzipReadCloserLevel(rc, l.compression), nil
}
return rc, err
}
func (l *layer) Uncompressed() (io.ReadCloser, error) {
rc, err := l.opener()
if err == nil && l.compressed {
return v1util.GunzipReadCloser(rc)
}
return rc, err
}
func (l *layer) Size() (int64, error) {
return l.size, nil
}
func (l *layer) MediaType() (types.MediaType, error) {
return types.DockerLayer, nil
}
// LayerOption applies options to layer
type LayerOption func(*layer)
// WithCompressionLevel sets the gzip compression. See `gzip.NewWriterLevel` for possible values.
func WithCompressionLevel(level int) LayerOption {
return func(l *layer) {
l.compression = level
}
}
// LayerFromFile returns a v1.Layer given a tarball
func LayerFromFile(path string, opts ...LayerOption) (v1.Layer, error) {
opener := func() (io.ReadCloser, error) {
return os.Open(path)
}
return LayerFromOpener(opener, opts...)
}
// LayerFromOpener returns a v1.Layer given an Opener function
func LayerFromOpener(opener Opener, opts ...LayerOption) (v1.Layer, error) {
rc, err := opener()
if err != nil {
return nil, err
}
defer rc.Close()
compressed, err := v1util.IsGzipped(rc)
if err != nil {
return nil, err
}
layer := &layer{
compressed: compressed,
compression: gzip.BestSpeed,
opener: opener,
}
for _, opt := range opts {
opt(layer)
}
if layer.digest, layer.size, err = computeDigest(opener, compressed, layer.compression); err != nil {
return nil, err
}
if layer.diffID, err = computeDiffID(opener, compressed); err != nil {
return nil, err
}
return layer, nil
}
// LayerFromReader returns a v1.Layer given a io.Reader.
func LayerFromReader(reader io.Reader, opts ...LayerOption) (v1.Layer, error) {
// Buffering due to Opener requiring multiple calls.
a, err := ioutil.ReadAll(reader)
if err != nil {
return nil, err
}
return LayerFromOpener(func() (io.ReadCloser, error) {
return ioutil.NopCloser(bytes.NewReader(a)), nil
}, opts...)
}
func computeDigest(opener Opener, compressed bool, compression int) (v1.Hash, int64, error) {
rc, err := opener()
if err != nil {
return v1.Hash{}, 0, err
}
defer rc.Close()
if compressed {
return v1.SHA256(rc)
}
return v1.SHA256(v1util.GzipReadCloserLevel(ioutil.NopCloser(rc), compression))
}
func computeDiffID(opener Opener, compressed bool) (v1.Hash, error) {
rc, err := opener()
if err != nil {
return v1.Hash{}, err
}
defer rc.Close()
if !compressed {
digest, _, err := v1.SHA256(rc)
return digest, err
}
reader, err := gzip.NewReader(rc)
if err != nil {
return v1.Hash{}, err
}
defer reader.Close()
diffID, _, err := v1.SHA256(reader)
return diffID, err
}

View File

@@ -0,0 +1,440 @@
// Copyright 2018 Google LLC All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package tarball
import (
"archive/tar"
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"sort"
"strings"
"github.com/google/go-containerregistry/pkg/name"
v1 "github.com/google/go-containerregistry/pkg/v1"
"github.com/google/go-containerregistry/pkg/v1/partial"
)
// WriteToFile writes in the compressed format to a tarball, on disk.
// This is just syntactic sugar wrapping tarball.Write with a new file.
func WriteToFile(p string, ref name.Reference, img v1.Image, opts ...WriteOption) error {
w, err := os.Create(p)
if err != nil {
return err
}
defer w.Close()
return Write(ref, img, w, opts...)
}
// MultiWriteToFile writes in the compressed format to a tarball, on disk.
// This is just syntactic sugar wrapping tarball.MultiWrite with a new file.
func MultiWriteToFile(p string, tagToImage map[name.Tag]v1.Image, opts ...WriteOption) error {
refToImage := make(map[name.Reference]v1.Image, len(tagToImage))
for i, d := range tagToImage {
refToImage[i] = d
}
return MultiRefWriteToFile(p, refToImage, opts...)
}
// MultiRefWriteToFile writes in the compressed format to a tarball, on disk.
// This is just syntactic sugar wrapping tarball.MultiRefWrite with a new file.
func MultiRefWriteToFile(p string, refToImage map[name.Reference]v1.Image, opts ...WriteOption) error {
w, err := os.Create(p)
if err != nil {
return err
}
defer w.Close()
return MultiRefWrite(refToImage, w, opts...)
}
// Write is a wrapper to write a single image and tag to a tarball.
func Write(ref name.Reference, img v1.Image, w io.Writer, opts ...WriteOption) error {
return MultiRefWrite(map[name.Reference]v1.Image{ref: img}, w, opts...)
}
// MultiWrite writes the contents of each image to the provided reader, in the compressed format.
// The contents are written in the following format:
// One manifest.json file at the top level containing information about several images.
// One file for each layer, named after the layer's SHA.
// One file for the config blob, named after its SHA.
func MultiWrite(tagToImage map[name.Tag]v1.Image, w io.Writer, opts ...WriteOption) error {
refToImage := make(map[name.Reference]v1.Image, len(tagToImage))
for i, d := range tagToImage {
refToImage[i] = d
}
return MultiRefWrite(refToImage, w, opts...)
}
// MultiRefWrite writes the contents of each image to the provided reader, in the compressed format.
// The contents are written in the following format:
// One manifest.json file at the top level containing information about several images.
// One file for each layer, named after the layer's SHA.
// One file for the config blob, named after its SHA.
func MultiRefWrite(refToImage map[name.Reference]v1.Image, w io.Writer, opts ...WriteOption) error {
// process options
o := &writeOptions{
updates: nil,
}
for _, option := range opts {
if err := option(o); err != nil {
return err
}
}
size, _, mBytes, err := getSizeAndManifest(refToImage)
if err != nil {
return sendUpdateReturn(o, err)
}
return writeImagesToTar(refToImage, mBytes, size, w, o)
}
// sendUpdateReturn return the passed in error message, also sending on update channel, if it exists
func sendUpdateReturn(o *writeOptions, err error) error {
if o != nil && o.updates != nil {
o.updates <- v1.Update{
Error: err,
}
}
return err
}
// sendProgressWriterReturn return the passed in error message, also sending on update channel, if it exists, along with downloaded information
func sendProgressWriterReturn(pw *progressWriter, err error) error {
if pw != nil {
return pw.Error(err)
}
return err
}
// writeImagesToTar writes the images to the tarball
func writeImagesToTar(refToImage map[name.Reference]v1.Image, m []byte, size int64, w io.Writer, o *writeOptions) (err error) {
if w == nil {
return sendUpdateReturn(o, errors.New("must pass valid writer"))
}
imageToTags := dedupRefToImage(refToImage)
tw := w
var pw *progressWriter
// we only calculate the sizes and use a progressWriter if we were provided
// an option with a progress channel
if o != nil && o.updates != nil {
pw = &progressWriter{
w: w,
updates: o.updates,
size: size,
}
tw = pw
}
tf := tar.NewWriter(tw)
defer tf.Close()
seenLayerDigests := make(map[string]struct{})
for img := range imageToTags {
// Write the config.
cfgName, err := img.ConfigName()
if err != nil {
return sendProgressWriterReturn(pw, err)
}
cfgBlob, err := img.RawConfigFile()
if err != nil {
return sendProgressWriterReturn(pw, err)
}
if err := writeTarEntry(tf, cfgName.String(), bytes.NewReader(cfgBlob), int64(len(cfgBlob))); err != nil {
return sendProgressWriterReturn(pw, err)
}
// Write the layers.
layers, err := img.Layers()
if err != nil {
return sendProgressWriterReturn(pw, err)
}
layerFiles := make([]string, len(layers))
for i, l := range layers {
d, err := l.Digest()
if err != nil {
return sendProgressWriterReturn(pw, err)
}
// Munge the file name to appease ancient technology.
//
// tar assumes anything with a colon is a remote tape drive:
// https://www.gnu.org/software/tar/manual/html_section/tar_45.html
// Drop the algorithm prefix, e.g. "sha256:"
hex := d.Hex
// gunzip expects certain file extensions:
// https://www.gnu.org/software/gzip/manual/html_node/Overview.html
layerFiles[i] = fmt.Sprintf("%s.tar.gz", hex)
if _, ok := seenLayerDigests[hex]; ok {
continue
}
seenLayerDigests[hex] = struct{}{}
r, err := l.Compressed()
if err != nil {
return sendProgressWriterReturn(pw, err)
}
blobSize, err := l.Size()
if err != nil {
return sendProgressWriterReturn(pw, err)
}
if err := writeTarEntry(tf, layerFiles[i], r, blobSize); err != nil {
return sendProgressWriterReturn(pw, err)
}
}
}
if err := writeTarEntry(tf, "manifest.json", bytes.NewReader(m), int64(len(m))); err != nil {
return sendProgressWriterReturn(pw, err)
}
// be sure to close the tar writer so everything is flushed out before we send our EOF
if err := tf.Close(); err != nil {
return sendProgressWriterReturn(pw, err)
}
// send an EOF to indicate finished on the channel, but nil as our return error
_ = sendProgressWriterReturn(pw, io.EOF)
return nil
}
// calculateManifest calculates the manifest and optionally the size of the tar file
func calculateManifest(refToImage map[name.Reference]v1.Image) (m Manifest, err error) {
imageToTags := dedupRefToImage(refToImage)
for img, tags := range imageToTags {
cfgName, err := img.ConfigName()
if err != nil {
return nil, err
}
// Store foreign layer info.
layerSources := make(map[v1.Hash]v1.Descriptor)
// Write the layers.
layers, err := img.Layers()
if err != nil {
return nil, err
}
layerFiles := make([]string, len(layers))
for i, l := range layers {
d, err := l.Digest()
if err != nil {
return nil, err
}
// Munge the file name to appease ancient technology.
//
// tar assumes anything with a colon is a remote tape drive:
// https://www.gnu.org/software/tar/manual/html_section/tar_45.html
// Drop the algorithm prefix, e.g. "sha256:"
hex := d.Hex
// gunzip expects certain file extensions:
// https://www.gnu.org/software/gzip/manual/html_node/Overview.html
layerFiles[i] = fmt.Sprintf("%s.tar.gz", hex)
// Add to LayerSources if it's a foreign layer.
desc, err := partial.BlobDescriptor(img, d)
if err != nil {
return nil, err
}
if !desc.MediaType.IsDistributable() {
diffid, err := partial.BlobToDiffID(img, d)
if err != nil {
return nil, err
}
layerSources[diffid] = *desc
}
}
// Generate the tar descriptor and write it.
m = append(m, Descriptor{
Config: cfgName.String(),
RepoTags: tags,
Layers: layerFiles,
LayerSources: layerSources,
})
}
// sort by name of the repotags so it is consistent. Alternatively, we could sort by hash of the
// descriptor, but that would make it hard for humans to process
sort.Slice(m, func(i, j int) bool {
return strings.Join(m[i].RepoTags, ",") < strings.Join(m[j].RepoTags, ",")
})
return m, nil
}
// CalculateSize calculates the expected complete size of the output tar file
func CalculateSize(refToImage map[name.Reference]v1.Image) (size int64, err error) {
size, _, _, err = getSizeAndManifest(refToImage)
return size, err
}
func getSizeAndManifest(refToImage map[name.Reference]v1.Image) (size int64, m Manifest, mBytes []byte, err error) {
m, err = calculateManifest(refToImage)
if err != nil {
return 0, nil, nil, fmt.Errorf("unable to calculate manifest: %v", err)
}
mBytes, err = json.Marshal(m)
if err != nil {
return 0, nil, nil, fmt.Errorf("could not marshall manifest to bytes: %v", err)
}
size, err = calculateTarballSize(refToImage, mBytes)
if err != nil {
return 0, nil, nil, fmt.Errorf("error calculating tarball size: %v", err)
}
return size, m, mBytes, nil
}
// calculateTarballSize calculates the size of the tar file
func calculateTarballSize(refToImage map[name.Reference]v1.Image, mBytes []byte) (size int64, err error) {
imageToTags := dedupRefToImage(refToImage)
for img, name := range imageToTags {
manifest, err := img.Manifest()
if err != nil {
return size, fmt.Errorf("unable to get manifest for img %s: %v", name, err)
}
size += calculateSingleFileInTarSize(manifest.Config.Size)
for _, l := range manifest.Layers {
size += calculateSingleFileInTarSize(l.Size)
}
}
// add the manifest
size += calculateSingleFileInTarSize(int64(len(mBytes)))
// add the two padding blocks that indicate end of a tar file
size += 1024
return size, nil
}
func dedupRefToImage(refToImage map[name.Reference]v1.Image) map[v1.Image][]string {
imageToTags := make(map[v1.Image][]string)
for ref, img := range refToImage {
if tag, ok := ref.(name.Tag); ok {
if tags, ok := imageToTags[img]; ok && tags != nil {
imageToTags[img] = append(tags, tag.String())
} else {
imageToTags[img] = []string{tag.String()}
}
} else {
if _, ok := imageToTags[img]; !ok {
imageToTags[img] = nil
}
}
}
return imageToTags
}
// writeTarEntry writes a file to the provided writer with a corresponding tar header
func writeTarEntry(tf *tar.Writer, path string, r io.Reader, size int64) error {
hdr := &tar.Header{
Mode: 0644,
Typeflag: tar.TypeReg,
Size: size,
Name: path,
}
if err := tf.WriteHeader(hdr); err != nil {
return err
}
_, err := io.Copy(tf, r)
return err
}
// ComputeManifest get the manifest.json that will be written to the tarball
// for multiple references
func ComputeManifest(refToImage map[name.Reference]v1.Image) (Manifest, error) {
return calculateManifest(refToImage)
}
// WriteOption a function option to pass to Write()
type WriteOption func(*writeOptions) error
type writeOptions struct {
updates chan<- v1.Update
}
// WithProgress create a WriteOption for passing to Write() that enables
// a channel to receive updates as they are downloaded and written to disk.
func WithProgress(updates chan<- v1.Update) WriteOption {
return func(o *writeOptions) error {
o.updates = updates
return nil
}
}
// progressWriter is a writer which will send the download progress
type progressWriter struct {
w io.Writer
updates chan<- v1.Update
size, complete int64
}
func (pw *progressWriter) Write(p []byte) (int, error) {
n, err := pw.w.Write(p)
if err != nil {
return n, err
}
pw.complete += int64(n)
pw.updates <- v1.Update{
Total: pw.size,
Complete: pw.complete,
}
return n, err
}
func (pw *progressWriter) Error(err error) error {
pw.updates <- v1.Update{
Total: pw.size,
Complete: pw.complete,
Error: err,
}
return err
}
func (pw *progressWriter) Close() error {
pw.updates <- v1.Update{
Total: pw.size,
Complete: pw.complete,
Error: io.EOF,
}
return io.EOF
}
// calculateSingleFileInTarSize calculate the size a file will take up in a tar archive,
// given the input data. Provided by rounding up to nearest whole block (512)
// and adding header 512
func calculateSingleFileInTarSize(in int64) (out int64) {
// doing this manually, because math.Round() works with float64
out += in
if remainder := out % 512; remainder != 0 {
out += (512 - remainder)
}
out += 512
return out
}