Harbor sends 404 for blobs in proxy-cache project

[markafarrell]

Issue:

When we perform a HEAD on a blob in a proxy-cache project, a 404 Not Found is returned if the blob does not yet exist in the proxy-cache but DOES exist upstream.

Harbor should return the 200 with headers corresponding to what is found upstream.

This causes an issue when attempting to run a pull-through cache using https://distribution.github.io/distribution/recipes/mirror/ as it first sends a HEAD to check if a blob exists upstream prio to doing a get. This results in Blob Not Found error.

Once the image (and associated blobs) are pulled directly from harbor the issue goes away.

Reproduction:

Follow instructions in https://github.com/markafarrell/registry-double-cache-repro

Expected Results:

All blobs return 200 with expected Content-Length

e.g.

HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:8e44f01296e3a6fdc31a671bee1c2259c5d5ee8b49f29aec42b5d2af15600296
HTTP/1.1 200 OK
Content-Length: 28228449
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:98ac9c138461f51b44fee531c455b5ef0a51e320b0c84c2250440532ba4b8866
HTTP/1.1 200 OK
Content-Length: 1104
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:5d05fbd0b692b8098e49cf7df768d58ef3ff3d4e5e62c73a0508f459832fdb1d
HTTP/1.1 200 OK
Content-Length: 873
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:b4c7acd54b972ee44c6c89af4ad2ed9b66608f954c56ddb281867be9192bcb6e
HTTP/1.1 200 OK
Content-Length: 24751537
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:8116b2f58ddba907f8ec043a678960430130275fcc213ac03ac73bc6673b6bab
HTTP/1.1 200 OK
Content-Length: 97
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
HTTP/1.1 200 OK
Content-Length: 32
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:5325bfd41068d3d7fa2fad9d9a19e3fcdf571ca19d1eb4e055f8807284d9c870
HTTP/1.1 200 OK
Content-Length: 2115

Actual Results:

Harbor report 404 for blobs until the image is first pulled through the harbor proxy-cache project

HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:8e44f01296e3a6fdc31a671bee1c2259c5d5ee8b49f29aec42b5d2af15600296
HTTP/1.1 404 Not Found
Content-Length: 133
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:98ac9c138461f51b44fee531c455b5ef0a51e320b0c84c2250440532ba4b8866
HTTP/1.1 404 Not Found
Content-Length: 133
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:5d05fbd0b692b8098e49cf7df768d58ef3ff3d4e5e62c73a0508f459832fdb1d
HTTP/1.1 404 Not Found
Content-Length: 133
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:b4c7acd54b972ee44c6c89af4ad2ed9b66608f954c56ddb281867be9192bcb6e
HTTP/1.1 404 Not Found
Content-Length: 133
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:8116b2f58ddba907f8ec043a678960430130275fcc213ac03ac73bc6673b6bab
HTTP/1.1 404 Not Found
Content-Length: 133
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
HTTP/1.1 404 Not Found
Content-Length: 133
HEAD https://127.0.0.1:38469/v2/dockerhub/library/redis/blobs/sha256:5325bfd41068d3d7fa2fad9d9a19e3fcdf571ca19d1eb4e055f8807284d9c870
HTTP/1.1 404 Not Found
Content-Length: 133

[bupd]

It is by design the proposal. Harbor doesnt send HEAD requests to check blobs. it will only send HEAD to check the right manifest exist upstream with this (there is few cases where we hit rate limits).

if we add HEAD reqs to blobs this might result in getting rate limited very quickly.

[bupd]

Here is a link to the proposal - https://github.com/goharbor/community/blob/68d7320aaa671b4247f3de6b4ba555aa4dc6fdae/proposals/new/pull_through_proxy_cache.md?plain=1#L11

we can add this as a feature instead. Please feel free to create a proposal. we can discuss this.

[markafarrell]

@bupd Would it be reasonable to proxy the HEAD to upstream if (and only if) the blob is not found locally.

My assumption is that this would be fairly rare and potentially would not significantly towards rate limits.

The current behaviour means you can't run distribution/registry as a pull-through cache behind harbor with a proxy-cache project.