Repository navigation
Conversation
Outside contributors' pipelines need the frozen universe's declaration index
and search corpus, about half a gigabyte. Jack chose on 2026-11-11 to have the
Worker stream it to agent-token holders ("Token-gated download").
src/bundles.ts serves a bundle from the private R2 bucket wikilean-contrib
(binding CONTRIB, read-only; unbound, 503):
- GET /api/index/bundles lists the live bundles to an owned agent token;
- GET /api/index/bundles/<manifest sha256> serves the MANIFEST.json bytes;
- GET .../files/<path> streams a listed file, whole or by one byte range
(If-Range, 206, 416), with an ETag of its sha256, If-None-Match 304 and HEAD
answered from the manifest, `private, no-store`, and the path taken as sent,
never decoded;
- POST /api/index/bundles registers an uploaded bundle (pipeline bearer only):
it reads MANIFEST.json from the bucket, checks its sha256, its schema and
that it carries its libraries' licences, and that the bucket holds every
listed file at its size.
A download needs an owned agent whose owner accepted CC0, a live bundle and
the index_downloads lever open. Each range served is one row of
index_downloads, written by a statement that re-checks all of that and the
owner's 4 GiB daily quota across their agents, so the table is the download
log too (no IP). Migration 0022 seeds the lever `closed` and creates both
tables; leverOpen reads such a lever shut unless it says open. The promoter
gates on index_downloads, the binding test pins both buckets, and
refuse/requirePipeline are now shared from submissions.ts.
Deicyde
marked this pull request as ready for review
November 13, 2026 16:15
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Worker half of the token-gated index download (Jack, 2026-11-11: "Token-gated download (Recommended)"). Outside contributors' pipelines need the frozen universe's declaration index and search corpus, about half a gigabyte; the Worker streams it from a private R2 bucket to the holder of an owned agent token. The bundle format and the uploader are #194.
Routes (
wiki/src/bundles.ts).GET /api/index/bundleslists the live bundles to an owned agent token, with whether downloads are open.GET /api/index/bundles/<manifest sha256>serves the bundle'sMANIFEST.jsonbytes.GET /api/index/bundles/<sha>/files/<path>streams one listed file from the bucket, whole or by one byte range (If-Range, 206, 416), with an ETag of the file's sha256.If-None-Matchanswers 304 and a HEAD is answered from the manifest, neither reading the bucket. The path is taken as sent and never decoded, since no listed path needs encoding. Responses areprivate, no-store.POST /api/index/bundles(pipeline bearer only) registers a bundlescripts/articles/upload_contrib.pyuploaded. It readsMANIFEST.jsonfrom the bucket, checks it hashes to the sha it is stored under, its schema, that it carries its libraries' licences, and that the bucket holds every listed file at its size.Who and how much. A download needs an owned agent whose owner accepted CC0, a live bundle, and the
index_downloadslever open. Each range served writes one row ofindex_downloadsin a statement that re-checks all of that and the owner's daily quota of 4 GiB across all their agents, so the table is also the download log (no IP). Levers that need no deploy: closeindex_downloads, retire a bundle (index_bundles.state), revoke the token, block the owner. There is no separate terms checkbox for the libraries whose use is restricted (ATLAS among them): the bundle carries their licence texts anddocs/AGENT-CONTRIBUTIONS.mdquotes ATLAS's line. Say if you want an acknowledgement before the first download as well.Migration 0022 seeds the lever
closed(INSERT OR IGNORE, so re-applying never reopens it) and createsindex_bundlesandindex_downloads;leverOpenreads a missing row or table as shut. The promoter now refuses a commit containing 0022 untilindex_downloadsis on D1.Release. Release-affecting (
wiki/src,wiki/wrangler.jsonc, a migration). Before the promotion that carries it: back up, apply and verify 0022, and create the private R2 bucketwikilean-contrib, since a deploy with theCONTRIBbinding needs it. It ships dark: nothing is downloadable until a bundle is registered and the lever opened. CLAUDE.md, AGENTS.md anddocs/AGENT-CONTRIBUTIONS.mdsay all of this.Checks. Worker, Python and browser CI green at 32c510d: https://github.com/Deicyde/WikiLean/actions/runs/43458736457. The migration-0019 test listed every
site_settingsrow; it now reads only the two levers 0019 seeds, andwiki/test/bundles.test.tspins the new one.