github/distribution
1
0
Fork 0
mirror of https://github.com/distribution/distribution.git synced 2025-09-15 14:48:31 +00:00
Code Issues Projects Releases Wiki Activity

add concurrency limits for tag lookup and untag

Browse Source 
Harbor is using the distribution for it's (harbor-registry) registry component.
The harbor GC will call into the registry to delete the manifest, which in turn
then does a lookup for all tags that reference the deleted manifest.
To find the tag references, the registry will iterate every tag in the repository
and read it's link file to check if it matches the deleted manifest (i.e. to see
if uses the same sha256 digest). So, the more tags in repository, the worse the
performance will be (as there will be more s3 API calls occurring for the tag
directory lookups and tag file reads).

Therefore, we can use concurrent lookup and untag to optimize performance as described in https://github.com/goharbor/harbor/issues/12948.

P.S. This optimization was originally contributed by @Antiarchitect, now I would like to take it over.
Thanks @Antiarchitect's efforts with PR https://github.com/distribution/distribution/pull/3890.

Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
This commit is contained in:
Liang Zheng
2024-04-18 15:56:26 +08:00
parent a5882d6646
commit a2afe23f38
10 changed files with 153 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
docs/content/about/configuration.md
View File

@@ -141,6 +141,8 @@ storage:
usedualstack: false
loglevel: debug
inmemory: # This driver takes no parameters
tag:
concurrencylimit: 8
delete:
enabled: false
redirect:
@@ -521,6 +523,26 @@ parameter sets a limit on the number of descriptors to store in the cache.
The default value is 10000. If this parameter is set to 0, the cache is allowed
to grow with no size limit.
### `tag`
The `tag` subsection provides configuration to set concurrency limit for tag lookup.
When user calls into the registry to delete the manifest, which in turn then does a
lookup for all tags that reference the deleted manifest. To find the tag references,
the registry will iterate every tag in the repository and read it's link file to check
if it matches the deleted manifest (i.e. to see if uses the same sha256 digest).
So, the more tags in repository, the worse the performance will be (as there will
be more S3 API calls occurring for the tag directory lookups and tag file reads if
using S3 storage driver).
Therefore, add a single flag `concurrencylimit` to set concurrency limit to optimize tag
lookup performance under the `tag` section. When a value is not provided or equal to 0,
`GOMAXPROCS` will be used.
```yaml
tag:
concurrencylimit: 8
```
### `redirect`
The `redirect` subsection provides configuration for managing redirects from