Collin Shoop
e4af4dc3a6
storagedriver/s3: Optimize s3 Delete test cleanup.
2021-08-11 13:00:06 -04:00
Collin Shoop
5d8fa0ce94
storagedriver/s3: Fixed a case where Delete was not working when the subpath immediately followed the given path with a "-" and quit early with noop
2021-08-11 12:17:24 -04:00
Collin Shoop
1e3b6b67a8
storagedriver/s3: Added Delete tests to s3_test
2021-08-11 12:16:43 -04:00
Collin Shoop
26e412898e
Merge pull request #17 from CollinShoop/storagedriver-s3-walk-optimization
...
registry/storagedriver S3 Walk optimization
2021-06-30 08:57:02 -04:00
Collin Shoop
dec90d3b1a
storagedriver/s3: More comment wording
2021-06-30 08:56:01 -04:00
Collin Shoop
d257d6ce28
storagedriver/s3: Updating the directoryDiff naming and replace sort with reverse
2021-06-30 08:56:01 -04:00
Collin Shoop
b9b0cac122
storagedriver/s3: Major change to the S3 Walk impl to infer directories to walk between files. This is needed for manifest enumeration among others
2021-06-30 08:56:01 -04:00
Collin Shoop
8d38cde0f3
storagedriver/s3: Updating comments
2021-06-30 08:56:01 -04:00
Collin Shoop
9618ba7231
storagedriver/s3: Added Walk test case for dealing with errors
2021-06-30 08:56:01 -04:00
Collin Shoop
9a4da20cf1
storagedriver/s3: Reverting unintentional change.
2021-06-30 08:56:01 -04:00
Collin Shoop
05a2d10b76
storagedriver/s3: Cleaning up tests.
2021-06-30 08:56:01 -04:00
Collin Shoop
b2bdea24aa
storagedriver/s3: Forgoing any interface conformance tests and moved it all into S3 tests.
2021-06-30 08:56:01 -04:00
Collin Shoop
70573f92c7
storagedriver: Adding an interface conformance test for Walk, cont.
2021-06-30 08:56:01 -04:00
Collin Shoop
fbb45bd8a0
storagedriver: Adding an interface conformance test for Walk
2021-06-30 08:56:01 -04:00
Collin Shoop
847738fd5e
storagedriver/s3: fixed a bug in s3 Walk impl preventing ErrSkipDir from stopping gracefully
2021-06-30 08:56:01 -04:00
Collin Shoop
aea873cc92
storagedriver/s3: test refactor. fixed a bug in WalkFallback preventing ErrSkipDir from stopping gracefully
2021-06-30 08:56:01 -04:00
Collin Shoop
8b726cc377
storagedriver/s3: Reverting a few changes from previous WalkFiles impl that was left in.
2021-06-30 08:56:01 -04:00
Collin Shoop
e20be1ead5
storagedriver/s3: Reverting WalkFiles method. Instead, optimizing the Walk method.
2021-06-30 08:56:01 -04:00
Collin Shoop
1c3ee66061
storagedriver/s3: continue refining tests
2021-06-30 08:56:01 -04:00
Collin Shoop
c98c694d51
storagedriver/s3: refining tests
2021-06-30 08:56:01 -04:00
Collin Shoop
a08cc38c05
storagedriver/s3: additional tests
2021-06-30 08:56:01 -04:00
Collin Shoop
88420f2c7c
Added unit testing for storage driver WalkFallback and WalkFilesFallback
2021-06-30 08:56:01 -04:00
Collin Shoop
cff975b29f
Fixed the fallback implementation of WalkFilesFallback that wasn't working & added a Files Removed test for WalkFilesFallback.
2021-06-30 08:56:01 -04:00
Collin Shoop
48e2373d5b
storagedriver/s3: Simplified conditional in Walk impl
2021-06-30 08:56:01 -04:00
Collin Shoop
c83e9ae10a
Replaced usage of storagedriver.Walk with storagedriver.WalkFiles for linked/blobstore
2021-06-30 08:56:01 -04:00
Collin Shoop
f31195b2e8
storagedriver: Added a new WalkFiles method (later removed)
2021-06-30 08:56:01 -04:00
Collin Shoop
9f05a10c1b
Merge pull request #15 from CollinShoop/cshoop/add-mtime-to-blob-descriptor
...
Add mod time to registry blob descriptor
2021-06-24 16:53:18 -04:00
Collin Shoop
0e4a61bbae
Add mod time to registry blob descriptor
2021-06-24 10:45:15 -04:00
wayne
e465862185
Merge pull request #14 from digitalocean/wwarren/set-ContentLength-on-HeadObjectOutput
...
s3: work around buggy s3 api client implementation
2020-11-04 15:33:55 -06:00
Wayne Warren
bdbf05f6f8
flyby: set ContentLength on GetObjectOutput also
2020-11-04 15:03:52 -06:00
Wayne Warren
a8c352063e
s3: nit - ignore HeadObjectOutput if it doesn't have ContentLength
2020-11-04 00:06:41 -06:00
Wayne Warren
b0fc5ddc8b
s3: use a request handler to set ContentLength on HeadObjectOutput
2020-11-04 00:05:55 -06:00
Adam Wolfe Gordon
78f71c18b4
Merge pull request #13 from digitalocean/awg/quotas
...
Handle S3 QuotaExceeded errors appropriately
2020-10-26 11:12:58 -06:00
Adam Wolfe Gordon
c1574b87be
handlers: Handle QuotaExceededError and return a DENIED to the client
...
If we get a QuotaExceededError back from the storage driver when doing a write
for either manifests or blobs, return a DENIED to the client with the message
"quota exceeded".
Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
2020-10-23 15:46:04 -06:00
Adam Wolfe Gordon
d192a974fb
s3: Handle QuotaExceeded errors
...
Add a new storagedriver error to be used when quotas are exceeded, and return it
from the S3 driver.
Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
2020-10-23 15:29:39 -06:00
Adam Wolfe Gordon
f2331f9d03
s3: Parse S3 errors returned during multipart operations
...
The S3 implementation of `Writer` creates a multipart upload and then uploads to
it as data is written. Previously errors were returned verbatim, but in order to
handle things like quotas we should parse the S3 errors so we can return more
meaningful errors to callers of `Write`.
Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
2020-10-23 14:25:36 -06:00
wayne
122552b40b
Merge pull request #12 from digitalocean/wwarren/s3-driver-dont-use-KeyCount
...
s3: don't use s3.ListObjectsV2Output.KeyCount
2020-10-22 00:04:25 +00:00
Wayne Warren
a10b3a9ab4
s3: don't use s3.ListObjectsV2Output.KeyCount
2020-10-20 16:51:33 -05:00
wayne
6255476b3b
Merge pull request #11 from waynr/fix-azure-autorest-dependencies
...
vendor: Update github.com/Azure/go-autorest
2020-09-03 12:41:29 -05:00
Wayne Warren
37be5b31d2
vendor: Update github.com/Azure/go-autorest
2020-09-03 16:27:22 +00:00
wayne
0e6ff0e95e
Merge pull request #1 from waynr/add-vacuum-removelayerlink-func
...
Add Vacuum.RemoveLayerLink function
2020-09-03 09:05:16 -05:00
Wayne Warren
28166f7383
simplify RemoveLayerLink function
2020-09-02 21:54:36 +00:00
Wayne Warren
ec64b15a77
fix signature on RemoveLayerLink
2020-09-02 21:53:48 +00:00
Wayne Warren
5fcc8f55e2
need schema2 import
2020-09-02 21:53:48 +00:00
Wayne Warren
3b7741805d
Remove layer links along with manifest
2020-09-02 21:53:48 +00:00
Wayne Warren
4366eee39a
add Vacuum.RemoveLayerLink function
2020-09-02 21:53:48 +00:00
Adam Wolfe Gordon
1e1c67e13f
Merge pull request #10 from digitalocean/awg/resume-commit-race
...
Return a sensible error in races between blob upload completions
2020-08-24 14:59:01 -06:00
Adam Wolfe Gordon
611f55e79a
Return a sensible error in races between blob upload completions
...
Completing a blob upload is a multi-step process, especially on a storage
backend like S3 which lacks a "move" primitive. If a client times out (or
disconnects) during a commit, then retries immediately, we'll end up invoking
the "resume" logic for the blob upload while the completion from the previous
request is still in progress. The resume tries to find the upload's data, but
the commit has already moved it to its new location, resulting in a PathNotFound
error from the storage driver.
Previously, this error was passed back to the HTTP handler and translated into a
500 Internal Server Error. Return ErrBlobUploadUnknown instead, so that the HTTP
handler will return a 404.
2020-08-24 13:49:09 -06:00
Adam Wolfe Gordon
d9adf97319
Merge pull request #9 from digitalocean/awg/fix-unlinked-blobs
...
Revert "OCI: GET request to deleted blob URL should yield 404 response"
2020-08-17 11:02:58 -06:00
Adam Wolfe Gordon
37ab304645
Revert "OCI: GET request to deleted blob URL should yield 404 response"
...
Using the linked blob store for blob requests fixes a conformance issue and
works correctly for newly-pushed images. However, any images that were pushed
before the change could be missing layer links (since previously it was possible
to fetch a blob without the layer link existing) and it's not possible to pull
these iamges after the change.
Revert the change to use the linked blob store, for now, until we can figure out
how to fix this without breaking existing repositories.
This reverts commit 71f1c25220
2020-08-17 10:37:50 -06:00
