github/distribution
1
0
Fork 0
mirror of https://github.com/distribution/distribution.git synced 2025-09-17 23:59:57 +00:00
Code Issues Projects Releases Wiki Activity

Pass the last paging flag to storage drivers

Browse Source 
Storage drivers may be able to take advantage of the hint to start
their walk more efficiently.

For S3: The API takes a start-after parameter. Registries with many
repositories can drastically reduce calls to s3 by telling s3 to only
list results lexographically after the last parameter.

For the fallback: We can start deeper in the tree and avoid statting
the files and directories before the hint in a walk. For a filesystem
this improves performance a little, but many of the API based drivers
are currently treated like a filesystem, so this drastically improves
the performance of GCP and Azure blob.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
This commit is contained in:
James Hewitt
2022-07-10 03:04:50 +01:00
parent 3a44c2e10e
commit e22f7cbc73
13 changed files with 390 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
registry/storage/driver/storagedriver.go
View File

@@ -32,6 +32,19 @@ func (version Version) Minor() uint {
// CurrentVersion is the current storage driver Version.
const CurrentVersion Version = "0.1"
// WalkOptions provides options to the walk function that may adjust its behaviour
type WalkOptions struct {
// If StartAfterHint is set, the walk may start with the first item lexographically
// after the hint, but it is not guaranteed and drivers may start the walk from the path.
StartAfterHint string
}
func WithStartAfterHint(startAfterHint string) func(*WalkOptions) {
return func(s *WalkOptions) {
s.StartAfterHint = startAfterHint
}
}
// StorageDriver defines methods that a Storage Driver must implement for a
// filesystem-like key/value object storage. Storage Drivers are automatically
// registered via an internal registration mechanism, and generally created
@@ -88,8 +101,9 @@ type StorageDriver interface {
// from the given path, calling f on each file.
// If the returned error from the WalkFn is ErrSkipDir and fileInfo refers
// to a directory, the directory will not be entered and Walk
// will continue the traversal. If fileInfo refers to a normal file, processing stops
Walk(ctx context.Context, path string, f WalkFn) error
// will continue the traversal.
// If the returned error from the WalkFn is ErrFilledBuffer, processing stops.
Walk(ctx context.Context, path string, f WalkFn, options ...func(*WalkOptions)) error
}
// FileWriter provides an abstraction for an opened writable file-like object in