The cache was the easy part.
Now watch what it quietly throws away.

01The build that "hung"

The mirror as a machine: upstream intakes, one cache shelf, one serving nozzle.

I added zot after a CI pull hit Docker Hub's anonymous rate limit and the retry wrapper hid the 429. At the time of the incident, this site's build-stage pull was anonymous.

Docker Hub does not go silent when it limits you. It answers with HTTP 429 and the error code toomanyrequests. Under the current published limits, unauthenticated users get 100 pulls per six hours per IPv4 address (or IPv6 /64); a single-platform image counts as one pull, a multi-architecture image counts once per architecture pulled, version checks do not count, and a HEAD request can read the rate-limit headers without consuming a pull. A separate abuse limiter covers all request types with its own 429 form. The silence in this incident was manufactured on my side, by retries swallowing the answer.

The swallower, recovered from the build script's git history, was a double retry stack: an outer shell wrapper allowing four attempts, wrapped around buildah build --retry 3. Buildah's flag counts retries - one initial try plus up to three more, applying to registry push/pull operations. Four outer attempts, each containing up to four inner attempts: up to sixteen requests for one repeatedly failing registry operation (a build with several pulls can produce more), each billed against the limit it was trying to outlast. Retry amplification turns a throttle into an outage and hides the evidence while doing it.

retry() {
  local n=0 max=4
  until "$@"; do
    n=$((n+1)); [ "$n" -ge "$max" ] && { echo ">> failed after ${max} attempts" >&2; return 1; }
    echo ">> attempt ${n} failed, retrying in $((n*8))s..." >&2; sleep $((n*8))
  done
}
retry buildah build --retry 3 --retry-delay 5s ...
# --retry 3 = one attempt + three retries; the wrapper multiplies that by four.
# A 429 answered every one of them, and the backoffs read as a hang.

Pinning the base image by digest stopped the tag re-resolution but not the round-trip: an ephemeral runner with a cold store still fetches the manifest behind that digest every build. The repeated origin pulls justified a shared cache. This page takes it apart, including the parts that turned out not to work the way I first believed.

02What the mirror actually does

The mirror is zot: one registry, one 50Gi cache volume, digest-pinned like everything it serves. Five public registries front it in on-demand pull-through mode - a miss fetches and caches, a hit serves from the shelf. zot also offers polled mirroring and pre-seeding; Docker Hub is on-demand only, per its documentation.

The storage block, from the live config:

"storage": {
  "rootDirectory": "/var/lib/registry",
  "commit": true,
  "dedupe": true,
  "gc": true,
  "gcDelay": "1h",
  "gcInterval": "24h"
}

commit asks zot to commit writes to disk immediately instead of relying on buffered flushing; it narrows the buffered-write window, while end-to-end power-loss durability still depends on the filesystem, volume and disk. dedupe uses hard links on local filesystem storage (remote backends implement it differently), saving capacity when five upstreams ship the same base layers under different names - at the cost of a startup reconciliation pass that matters operationally when toggled on existing data. And gc with its two timers looks innocuous here; section 06 is about why it is the most consequential block on this page.

Consumers point at it via the platform's machine-level registry config:

machine:
  registries:
    mirrors:
      docker.io:
        endpoints:
          - https://zot.bztmon.org
      ghcr.io:
        endpoints:
          - https://zot.bztmon.org

The mirror is the only listed endpoint, yet not a hard dependency: Talos tries endpoints in order "and by default the last implicit endpoint is the original upstream registry", unless skipFallback: true. An early version listed the upstream explicitly as endpoint two; it was removed as redundant - origin fallback here is a Talos default, not an endpoint this project maintains.

One asymmetry decides section 07's rollout order. Talos's v1alpha1 machine-config reference states for registry auth: "changes to the registry auth will not be picked up by the CRI containerd plugin without a reboot" - matching what this lab saw on v1.13.4. Mirror endpoint changes applied live here (same versions; the docs are silent on that half, so it stays a lab observation, not a cross-version guarantee). Secrets the registry consumes from a mount rotate with a pod restart. Put each credential on the side that can move.

One pod, one PVC, one node is an accepted failure domain here, recorded as such; section 09 covers what changes when the requirements do.

03Request routing and five upstreams

Five registries feed one endpoint: when a node asks the mirror for pause:3.10, how does it know which origin that means? Three facts:

1. containerd names the origin - mirror requests carry it as a query parameter: /v2/pause/manifests/3.10?ns=registry.k8s.io (documented).
2. zot v2.1.17 ignores it - no ns handling exists in the deployed request path; using it for upstream selection is open feature request zot 4187. The URL path alone selects the local repository.
3. This deployment has no per-upstream prefixes - the live sync config gives all five upstreams prefix: "**" and no destination. On a miss, zot tries upstreams in config order until one has the path; the docs' own multi-registry example instead gives each a distinct destination.

The flat namespace has not bitten because the origins use largely disjoint paths - Hub's official images live under library/ (the implicit prefix behind bare names like alpine), NGC under nvidia/. Largely disjoint is a probability: an organisation existing on both GHCR and Quay would collide, and config order would pick the winner. Accepted risk, recorded; not a design.

04Bootstrap, fallback and failure modes

The dependency ring: one of the orbiting parts is the mirror itself - its own image is served through the thing it is.

Four circular dependencies live in this design; each got a different answer.

Loop 1 - the mirror's own image comes through the mirror

zot runs as a container whose image lives on a registry zot mirrors. When the hosting node boots, it asks the mirror - which is not running, because the node is booting. The loop breaks on the documented fallback: mirror endpoints exhaust, containerd falls through to the origin. Keeping skipFallback unset is the load-bearing choice.

Loop 2 - the recovery tooling ignores the mirror

The fleet's rescue tooling does not pull through the mirror: its image is cached on the operations host, outside the cluster, so it remains available while zot or its cluster is down.

Loop 3 - authentication cannot flip everywhere at once

Enforcing auth needs every node to carry credentials that only apply after a reboot, so anonymous read must survive until the last node is proven - and the proof must be designed not to lie. Section 07 covers it.

Loop 4 - the mirror's host is also its customer

The node hosting the mirror boots its own workloads through it, including the tunnel serving this page - so restarting that node carries a blast radius beyond the mirror, and its reboot runbook lists every public-facing dependant and the order they return.

Fallback is a policy decision. This lab keeps it on for bootstrap survivability and availability through mirror outages; the cost is that pulls can bypass the mirror unnoticed and land on origin rate limits. A regulated or disconnected estate makes the opposite call - skipFallback: true, egress restriction, preseeded release stock, a tested recovery path - because there the bypass is the failure mode. Failure classes differ in symptom, path and consequence:

05Digests, manifests and the supply chain

Half the mistakes on this page came from conflating these objects:

• A tag is a mutable pointer; zero or many tags can reference one manifest.
• A manifest digest identifies exact manifest bytes; a multi-platform index, each platform manifest, and every layer and config blob carry their own.
• name:tag@digest is client-side grammar, not wire protocol: the OCI distribution spec takes a tag or a digest in the URL. Clients resolve the digest and ignore the tag (Kubernetes documents this), so digest-pinned pulls are immune to tag moves. This page once mislearned the combined form as a cache-miss bug; retesting on v2.1.17 showed the historical zot rejection no longer reproduces.
• Digest pinning provides content integrity - not provenance, not authorisation, not availability, and not freedom from network fetches on a cold store.

For a mirror, digests carry one sharp operational rule: zot converts Docker-schema manifests to OCI by default, and conversion changes the digest - breaking every digest pin and signature downstream. The pairing that prevents it (preserveDigest: true per upstream, with http.compat: ["docker2s2"]) is mandatory in both directions: the registry refuses to start with one and not the other (the Loop 1 field note is this rule being learned). This deployment runs both on all five upstreams.

Digest preservation and artefact completeness are different properties. The docs tie preserveDigest/compat to keeping mirrored manifest bytes and media types - and therefore signature validity - aligned with upstream. Whether signature, SBOM and attestation objects arrive is separate: OCI 1.1 referrers ride the subject field and their own API, legacy Cosign signatures ride tag-schema conventions, and both depend on origin support and this version's sync behaviour. This deployment has not verified a complete referrer graph for any origin and performs no cryptographic signature verification; disconnected verification would also need locally held trust material. Both gaps sit in the open register. Production estates promote releases by digest and decide per policy whether referrers travel too; this lab runs only the digest half.

06Retention, GC and the digest-only trap

Cached bricks queued behind a jammed intake: availability problems here are quiet, not loud.

This section corrects the largest error in this page's first edition, which said "no retention config means keep everything". True for tags; false for everything else - and "everything else" includes the content this mirror exists to hold. The semantics, from upstream documentation and the deployed version's source:

• With no retention configuration, all tags are retained and all untagged manifests are deleted by GC (unless referenced by an index or artefact). Timing is two-stage: the manifest ages past gcDelay (1h here), then deletion happens at the next sweep (gcInterval, 24h here) - the survival window depends on where creation falls relative to that sweep, not a fixed one-hour fuse.
• Digest-only pull-through entries are stored as untagged manifests. That is the upstream fix's own problem statement: pull an image through the mirror by digest, and the cache entry has no tag to protect it.

Combined with section 04's revalidation behaviour: tag pulls are cached but contact upstream anyway; digest pulls serve locally but their entries are GC-eligible between sweeps - a digest-pinned fleet caches exactly the entry class GC may delete. On v2.1.17 this mirror is a rate-limit shield and a latency win, not yet a disconnection shelf; the first edition implied otherwise and was wrong.

Retention policy semantics - two nearby systems use opposite matching rules:

• Retention policies match per repository, first match wins - order the list most-specific-first. A repository matching no policy keeps its tags.
• Within a matched repository, one keepTags rule inverts the default: non-matching tags in that repository become deletable. The inversion is scoped to the repository, not global - the first edition of this page overstated it.
• Untagged manifests are governed separately (deleteUntagged, default true), and on the deployed version no retention rule can protect them. Pull-aware keepUntagged exists upstream from v2.1.19.
• Authorisation policies (section 07) use longest match, not first match. Two adjacent config blocks, two opposite precedence rules. Label which one you are reasoning about.

Mitigations, ranked: upgrade to v2.1.19+ for pull-aware keepUntagged (the designed fix, schema to be validated against the actual binary before rollout); until then, widen gcDelay/retention.delay (the upstream maintainer's interim suggestion), or set deleteUntagged: false, which protects every untagged manifest at the cost of unbounded cache growth. Disabling GC entirely swaps eviction for storage exhaustion with no reclaim path.

07The authentication migration

The proving gate: a node counts as migrated when an authenticated pull succeeds where an anonymous one cannot.

Target state: the mirror refuses anonymous pulls. Current state: anonymous read on, per-node credentials staged inert in machine configs until each reboot. The sequencing problem has one central hazard: while anonymous read is on, an ordinary pull says nothing about auth - a node whose credentials never applied is served as an anonymous reader and false-passes the gate. The canary must live in a repository that denies anonymous read, so success can only mean an authenticated pull.

The design is upstream-supported: authorisation resolves per-repository policies by longest match (** is the default for anything unmatched), and maintainer guidance confirms anonymous and authenticated access evaluate independently. So a canary/** entry granting named identities read, with no anonymousPolicy, denies anonymous on that path while the glob keeps the rest of the shelf open. The staged policy:

"accessControl": {
  "repositories": {
    "**": {
      "anonymousPolicy": ["read"],
      "policies": [
        { "users": ["zot-push"], "actions": ["read", "create", "update"] },
        { "users": ["zot-pull"], "actions": ["read"] }
      ]
    },
    "canary/**": {
      "policies": [
        { "users": ["zot-pull", "zot-push"], "actions": ["read"] }
      ]
    }
  }
}

The gate, per node, and what each step proves:

1. Stage credentials in the node's machine config (inert; documented as requiring reboot).
2. Reboot the node at a planned window.
3. Identity check: pull the protected canary through the node's own runtime. Success = authenticated (anonymous cannot); 401 = back to step 1. The expected split - 401 for missing/wrong credentials, 403 for a valid identity lacking the action - sits in the open register, since the docs state the 403 case only for OIDC identities.
4. Interception check, separately: the canary is a direct mirror reference, so it says nothing about whether docker.io/... references route through the mirror. That needs an original-upstream reference pulled on a node with a cold runtime store, correlated with the mirror's request log - node-local content would satisfy the pull without any network and fake a pass.
5. Anonymous read comes off the glob only when every node passes both checks. Rollback is the previous config commit, named before the flip.

Five trust domains never share material: node pull credentials (machine config, reboot-bound), the push credential (used by the operator-run build host for registry logins, not embedded in CI configuration), the mirror's upstream credentials (one mounted secret, pod-restart-bound), TLS trust, and human admin access (SSO in front of the UI). The push credential carries this page's oldest incident: in July 2026 its bcrypt hash and plaintext drifted apart in the secrets manager, rotation became impossible, and pushes stayed dead for twelve days - three applications shipped the day the rebuilt flow landed. Today the hash and plaintext are separate entries in the same secrets manager, rotated as a pair by a human; the registry consumes only the derived htpasswd file, each plaintext reaches only its consumer, and the sync identity is read-only so automation cannot half-rotate the pair. Residual cost, stated: one secrets-manager project holds both halves, so its compromise yields verifier and credential together - versioned, consumer-scoped secret objects remain a possible refinement.

08Observability that means something

The worst incident in this mirror's life: an in-flight sync wedged, every pull from that upstream hung, and the health endpoint returned 200 throughout:

"image already demanded, waiting on channel"
$ kubectl -n zot rollout restart deploy zot   <- recovery action (not a root cause)
# probe an actual manifest, not the process's opinion of itself:
$ curl -sfS --max-time 10 -o /dev/null -w '%{http_code}\n' \
    https://zot.bztmon.org/v2/library/busybox/manifests/latest \
    -H 'Accept: application/vnd.oci.image.index.v1+json'
200                                           <- THIS is "the mirror works"

The deployed source refines that story: the log line is coalescing by design - concurrent requests join the first sync, which runs on a detached background context with a three-hour default timeout and survives client disconnects. Waiters blocking on a stalled sync until that timeout matches what we saw; the restart was recovery, and the root cause was never isolated. Two knobs this config leaves unset - sync maxRetries (disabled by default upstream) and a tighter syncTimeout - are the first change on recurrence.

From the running version's source: /livez, /readyz and /startupz exist as health endpoints (absent from this version's docs), alongside the spec's /v2/. The Prometheus metrics extension exists upstream - series names include zot_http_requests_total, zot_repo_storage_bytes, zot_repo_downloads_total and zot_storage_lock_latency_seconds - and is not enabled here; wiring it, with scrape auth, is on the open list.

The layered checks, one question each:

The caching test matrix (digest rows: source-verified and consistent with behaviour here; disconnected rows: not yet run in this lab):

09The production translation

A central curated source and site-local registries: each site holds its own shelf, so a cut WAN idles nothing.

The first edition of this section asserted what "production" does; production chooses from patterns against requirements:

• Single registry, accepted failure domain - this lab's shape. Legitimate wherever a mirror outage degrades to origin pulls (fallback on) or a paused deploy window (fallback off), with the cost accepted in writing.
• Sync-based HA - documented active/standby or active/active pairs of independent instances, each with its own storage, mirroring each other behind a load balancer. The caveat: the synchronisation window between polls bounds what failover can lose.
• Scale-out clustering - instances shard repositories by hash and proxy to the owner, classically over shared S3-compatible storage. Horizontal scale for load; the shared storage becomes the availability story - a different property from the HA pair.
• Edge estates - a central source promotes releases by digest; each site runs a local registry holding its release set on a schedule, so a severed WAN idles nothing at pull time. A disconnection claim is valid only when content, required referrers, trust roots, credentials and the registry's own bootstrap image are already local (diagram D's right-hand column, per site).

Whatever the topology: recovery time and point objectives get numbers before an incident supplies them; backups are restore-tested copies off the failure domain (array snapshots and RAID protect against disk loss, not against the array, the site or operator error); cache capacity is planned from the section 06 retention policy; and the section 04 circular dependencies get drawn per site, because each exists at every scale - only the cost of ignoring them changes.

10Verified, open, and where the claims come from

Proof of life is one authenticated byte transfer through the real path - not a status page.

When a node powers on, the fleet's admission check performs an authenticated manifest pull through the node's own runtime before the node counts as a member: a process-health response validates neither routing nor authorisation. Most of this page condenses into that one command.

Open verification register

1. Digest-entry eviction reproduction in a disposable instance (section 06 procedure).
2. Canary repo negative test: anonymous 401 on canary/** while anonymous read elsewhere still succeeds; observed 401-vs-403 boundaries recorded.
3. Mirror interception proof: original-reference pull on a cold node correlated with the mirror's request log.
4. Disconnected-serve drill: cached digest pull with upstream blocked, disposable environment first.
5. Upgrade evaluation: v2.1.19/v2.1.20 for keepUntagged, with the retention config written and reviewed before the upgrade, not after.
6. Namespace redesign migration plan (per-origin destinations + overridePath), including cache re-warm cost.

Verified against

• zot documentation (versioned): retention · storage · mirroring · authn/authz · high availability · scale-out
• zot upstream issues/PRs: 4148 digest-only eviction · 4191 keepUntagged (v2.1.19) · 4128 tag revalidation · 4187 containerd ns param · 2928 mixed anonymous auth · 2505 demand coalescing
• Talos: pull-through registry cache (v1.13) · containerd: registry hosts and the ns parameter
• OCI: distribution spec · image/distribution 1.1 (referrers) · signing: cosign verify
• Docker Hub usage and limits · buildah build --retry semantics