Public packages only need to be downloaded once

Every day, the world’s CI fleet re-downloads the same immutable bytes: the same numpy wheel, the same lodash tarball, on every fresh runner and every cache-evicted container. PyPI alone served 136.7 billion downloads in March 2026 — about 4.4 billion a day. You don’t get to that number with humans typing pip install; the overwhelming majority is CI runners and automated builds re-fetching bytes somebody else already fetched an hour earlier. (That last inference is mine; the download count is the report’s.) The cost lands as redundancy and latency everywhere, and on some surfaces — Docker Hub most famously — as rate-limit ceilings.

CoreLink — the content-addressed cache platform I’m building (closed source; this post is the design and its exact current state, with code references I can show) — treats this as a storage-identity problem: if an artifact is addressed by what it is, a byte that never changes only needs to enter the system once. The interesting part is what happens when “once” means once across every customer — and where that idea stops being safe.

Prior art, first

None of the ingredients are new; the combination is the design.

• The Go module ecosystem is the gold standard for “packages fetched once, verified”: proxy.golang.org is a global mirror and sum.golang.org is a transparency log of module checksums, so every go get can hit one shared cache and verify what it got. It is per-language, operated by Google, and private modules live outside the scheme.
• Public mirrors and registry CDNs already share public bytes globally — mirror.gcr.io, the Linux distros’ mirror networks, AWS ECR’s pull-through cache, and the donated CDN in front of PyPI and the registry CDNs in front of npm. They solve transfer; they don’t solve identity: none of them co-locates those bytes with your private artifacts and your build cache in one addressing scheme, and Docker Hub’s rate limits follow you through most of them.
• pnpm stores every package version once in a local content-addressable store and hard-links it into node_modules — the same dedup insight, scoped to one machine.
• Nix and Guix binary caches are hash-addressed (input-addressed, mostly, rather than content-addressed) and community-shared — but they cache their own build model, not the npm/PyPI/OCI artifacts teams already consume.
• Artifactory, Nexus, devpi, Verdaccio, Athens, and a registry run as a pull-through cache are the per-organization staple; SaaS registries like Cloudsmith and GitHub Packages host multiple ecosystems per-org in the cloud. Every org’s cache starts cold.

The narrow thing I wanted, which I couldn’t find: cross-tenant dedup of public, immutable bytes, co-located with private artifacts in one addressing scheme — on the same content-addressed store that already serves the build cache. A public wheel and a private Bazel action result become two entries in one keyspace, shared and isolated by namespace rather than by living in different systems.

The design: access is gated, content is shared

CoreLink exposes each ecosystem through an adapter that speaks its native protocol — npm registry (tarballs + metadata), the PyPI simple index (wheels/sdists), Homebrew bottles, and the OCI Distribution Spec v1.1 pull/push surface (blobs, manifests, tag lists; no referrers API yet, and push/pull only — no public read-through mirror yet, so it’s not the answer to Docker Hub’s rate limits today) — so docker, podman, containerd and Helm are designed to talk to it with nothing more than a registry config change.

Under the adapters there is one rule: the token gates access; the content is shared. It is live end-to-end on brew today — with HOMEBREW_ARTIFACT_DOMAIN and a HOMEBREW_DOCKER_REGISTRY_TOKEN bearer, a stock brew client pulls real bottles through the mirror — and the exact state of every other surface is below. Public upstream bytes are stored once, under a shared _public namespace, and served to any authenticated tenant. Tenant identity never comes from the URL — the tenant segment in the path is routing-only, and the real tenant is derived from the verified PAT on every request to these surfaces (PipPatResolver::resolve, BrewPatResolver::resolve; on OCI, from the bearer minted by the two-leg token exchange). On the package surfaces, a request without a valid token gets nothing; a request with one gets bytes some other tenant may have caused to be fetched last month.

The first time any tenant’s brew asks for a given bottle, the service fetches it from upstream (read-through), verifies it against the digest in its URL, stores it content-addressed, and serves it. Every request after that — from any tenant — is served from the store: fetched from upstream once, modulo two asides (concurrent first requests can race and fetch twice — both store identical bytes — and a damaged or missing blob is deliberately treated as a miss and re-fetched). Today nothing is evicted; what eviction should even mean for a namespace whose value is being permanently warm is an open question below.

Private artifacts live in the same system but never in _public: npm @scoped metadata stays per-tenant by construction (namespace_for_meta_key), and tarball bytes are per-tenant for everything — more on why in a moment. The public/private split is enforced in the keyspace, not in a review process.

That shared namespace is also the economics: each tenant’s ordinary traffic warms the cache for the next one — that’s the whole mechanism; there is no step two. And because the store is Cloudflare R2, which (as of current pricing) charges no egress, serving a cached bottle costs storage and a read, not a transfer fee. Per-org mirrors structurally can’t compound like this; every one of them starts cold.

Where “once” stops being safe

The part worth a post: cross-tenant sharing is tempting to justify with two properties — the artifact is immutable, and the client verifies it independently. But the second property doesn’t always hold: it does for lockfile and --require-hashes workflows, while a hashless pip install trusts the mirror for the index it serves and — once the wheel client path closes — the bytes too, exactly as it would trust any proxy.

So the rule I can actually defend is narrower, and it is about artifact bytes: share only artifact bytes that are immutable and digest-verified — verified by the mirror always, before they enter the store, and by the client wherever its workflow does. Where even that floor can’t be established, don’t share. That rule, one boundary that is pure engineering debt, and one surface not yet built at all are how the implementation is split today:

• Homebrew bottles: shared cross-tenant and live end-to-end. HOMEBREW_ARTIFACT_DOMAIN points a stock client’s bottle traffic at the mirror — a real brew fetch hello routes through it, returns the bottle byte-for-byte (sha256 verified against the client’s own copy), and the same blob URL without a token returns 401. Bottles use the canonical bottle URL as the lookup identity, resolving through a two-level cache (key → content hash → blob), so storage identity is still the content itself.
• PyPI wheels/sdists: the shared store is live; the client path is not yet. The cross-tenant wheel store and its pre-put sha256 verification are shipped code, and released PyPI files are immutable (yanking hides them; it doesn’t change bytes) — but the index the mirror serves does not yet rewrite file URLs to point at itself, so a stock pip install exercises the index cache and still downloads wheels from PyPI’s CDN. Closing that loop is the remaining client-path work, and until it lands, “live” for pip means the index.
• npm: tarball bytes are per-tenant today — the engineering-debt boundary, not a policy triumph. The storage port sees only (tenant, hash-of-URL); the package name, and with it the public-vs-@scoped signal, is invisible at that layer. Sharing blindly would risk leaking private scoped tarballs across tenants, so the code fails closed to per-tenant until the signal is threaded through — forfeiting, for now, the dedup win on the largest ecosystem. And — the same client-path gap as pip — the served metadata doesn’t rewrite dist.tarball, so a stock npm client fetches tarballs from npm’s CDN; the per-tenant tarball store is shipped but not yet on the client path. (npm’s unpublish history is a second, softer reason to move carefully.)
• OCI: a per-tenant push/pull registry today — read-through mirroring isn’t built in any form. The adapter has no upstream-fetch path at all; it stores and serves what tenants push, isolated per tenant. So the Docker Hub pain from the intro is the one surface this doesn’t yet touch. Tags are mutable by design and only digests are immutable; sharing public base layers is the obvious win, and the open decision is tracked in the module doc.
• The build-cache surfaces (sccache, Bazel, Turborepo) never touch _public at all — they cache your own build outputs, which are private by nature. (Mirroring crates.io itself — public, immutable, sha256-carried in the index — would be the easiest future addition by this post’s own rule; it doesn’t exist today.)

Cross-tenant byte sharing means a bug in the mirror’s write path is a supply-chain incident multiplied by your customer count, so the population rules deserve exact language. Tenants don’t write _public — population happens only through the service’s own upstream fetch.

Brew’s fetch path is the strict one: a pinned first-hop origin (redirects after it follow the same internal-IP-literal guard as pip’s), an allowlist of homebrew/core and homebrew/cask, a digest-in-the-URL requirement for anything cached, verification against that digest before the put, and re-verification of the content hash on read.

Pip’s path is close behind, with one structural difference. The index host is pinned, and the second hop — the wheel URL the index supplies — is checked against a two-host allowlist (the index origin plus files.pythonhosted.org), with redirects refused when they point at internal IP literals (hostnames aren’t resolved and checked — a narrower guard than it sounds). The stored key is the wheel’s own sha256, verified against the bytes before the put — so the attack surface is that verification step, not the keyspace. What pip lacks, structurally, is brew’s digest-in-the-URL: the sha256 the mirror verifies comes from the same index the mirror fetched — the trust-the-mirror problem above, now in structural form.

Sharing the bytes is half the failure surface; who pays to serve them is the other half. Pulls are governed like the work they are: on the OCI surface every method including GET is charged against a per-tenant monthly dollar ceiling that fails closed — for a resolved tenant, over-ceiling or unable-to-cost-check is rejected rather than silently eaten. An adversarial pass I ran against my own gate flagged unlimited authenticated pulls as a cost hole, and that carve-out is closed. The gap that remains is earlier in the pipeline, and flagged in the source: a request with no resolvable bearer at all passes the meter uncounted, pending a trust-path review.

And one comparison I owe the reader: on auditability, this is strictly weaker today than Go’s proxy + sumdb — there is no transparency log of what _public contains. That is the most important open question below, and I consider it roadmap.

Limitations

• CoreLink is closed source. You cannot audit these claims from this post; the symbol references exist so a reader — or interviewer — can ask me to show exactly them.
• npm and OCI cross-tenant dedup is design, not code. Today those bytes are per-tenant; the boundary is stated exactly above.
• There is no takedown path today beyond manual surgery. When upstream removes a malicious package or a legal notice arrives, stopping those bytes means hand-editing the store’s index and blobs: no shipped takedown endpoint exists (every erase surface is tenant-bound — the internal ones to a verified GDPR/DSR request, the client-facing DELETEs to the caller’s own namespace — and _public is not an authenticatable tenant, so no endpoint can target it), and deleting a blob without its index entry would just trigger a re-fetch. Until that’s built, “cached” can mean “still served after upstream removed it” — a real gap, not a footnote.
• The trust model assumes upstream is honest at fetch time. A compromised artifact fetched during the compromise window is mirrored faithfully; content-addressing makes the copy tamper-evident from that moment, not retroactively clean. Client-side verification remains the real integrity boundary where clients use it.
• A shared cache is observable. Timing a fetch can reveal whether some tenant already warmed a given public artifact — not who. For public packages I judge that low-sensitivity; it is still a property per-org mirrors don’t have.
• Losing the mirror should cost latency, never artifacts — but the fallback story is per-ecosystem. Brew falls back to upstream on its own (point it at a dead mirror and a stock client just re-requests the bottle from ghcr.io); a pip --index-url replacement does not fall back by itself. Precise failover semantics per surface are productization work, and I won’t pretend they’re uniform today.
• No hit-rate or latency numbers are claimed here. The mirror hasn’t been benchmarked under representative multi-tenant load; by the rule the rest of my writing follows, a number exists when a run produced it.
• Mutable metadata is the unglamorous half — and it’s the one mutable thing shared cross-tenant today. Package indexes can’t be immutable; pip’s is per-tenant, but unscoped npm metadata is cached cross-tenant on a 5-minute TTL. Nothing writes it but the service’s own pinned upstream fetch (the adapters are GET-only) — yet no checksum bounds it, because the checksum travels inside the metadata and stock npm clients follow dist.tarball verbatim to any host it names. The only real floor is client-side lockfile/integrity. It’s pip’s trust-the-mirror circularity shared across tenants: outside the artifact-bytes rule above, and on the attestation-feed roadmap below.

Open questions

• What is a defensible cross-tenant trust story for OCI base layers — digest-only sharing with tag resolution kept per-tenant? Signed provenance thresholds before a layer is eligible for _public?
• The sumdb gap: the concrete next artifact is a **signed, append-only attestation feed of _public‘s contents** — (ecosystem, name, version, upstream-sha256) → stored content hash — published where anyone can diff it against the registries’ own hashes, without my permission or cooperation. That would turn the weakest point of a closed-source mirror into the checkable one.
• Should _public population require k independent fetch confirmations, or a transparency-log witness, before bytes become shared — so a single poisoned fetch can’t seed every tenant?
• Garbage collection: what does eviction mean for a namespace whose whole value is being permanently warm?

The one-sentence version: the token gates access, the content is shared — and knowing exactly where that stops being safe is the product.

CoreLink is the platform behind the site’s journey demo.