Skip to content

Worker: token-gated downloads of the declaration-index bundles - #195

Open
Deicyde wants to merge 3 commits into
mainfrom
claude/index-downloads
Open

Deicyde wants to merge 3 commits into
mainfrom
claude/index-downloads

Conversation

@Deicyde

@Deicyde Deicyde commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

The Worker half of the token-gated index download (Jack, 2026-11-11: "Token-gated download (Recommended)"). Outside contributors' pipelines need the frozen universe's declaration index and search corpus, about half a gigabyte; the Worker streams it from a private R2 bucket to the holder of an owned agent token. The bundle format and the uploader are #194.

Routes (wiki/src/bundles.ts).

  • GET /api/index/bundles lists the live bundles to an owned agent token, with whether downloads are open.
  • GET /api/index/bundles/<manifest sha256> serves the bundle's MANIFEST.json bytes.
  • GET /api/index/bundles/<sha>/files/<path> streams one listed file from the bucket, whole or by one byte range (If-Range, 206, 416), with an ETag of the file's sha256. If-None-Match answers 304 and a HEAD is answered from the manifest, neither reading the bucket. The path is taken as sent and never decoded, since no listed path needs encoding. Responses are private, no-store.
  • POST /api/index/bundles (pipeline bearer only) registers a bundle scripts/articles/upload_contrib.py uploaded. It reads MANIFEST.json from the bucket, checks it hashes to the sha it is stored under, its schema, that it carries its libraries' licences, and that the bucket holds every listed file at its size.

Who and how much. A download needs an owned agent whose owner accepted CC0, a live bundle, and the index_downloads lever open. Each range served writes one row of index_downloads in a statement that re-checks all of that and the owner's daily quota of 4 GiB across all their agents, so the table is also the download log (no IP). Levers that need no deploy: close index_downloads, retire a bundle (index_bundles.state), revoke the token, block the owner. There is no separate terms checkbox for the libraries whose use is restricted (ATLAS among them): the bundle carries their licence texts and docs/AGENT-CONTRIBUTIONS.md quotes ATLAS's line. Say if you want an acknowledgement before the first download as well.

Migration 0022 seeds the lever closed (INSERT OR IGNORE, so re-applying never reopens it) and creates index_bundles and index_downloads; leverOpen reads a missing row or table as shut. The promoter now refuses a commit containing 0022 until index_downloads is on D1.

Release. Release-affecting (wiki/src, wiki/wrangler.jsonc, a migration). Before the promotion that carries it: back up, apply and verify 0022, and create the private R2 bucket wikilean-contrib, since a deploy with the CONTRIB binding needs it. It ships dark: nothing is downloadable until a bundle is registered and the lever opened. CLAUDE.md, AGENTS.md and docs/AGENT-CONTRIBUTIONS.md say all of this.

Checks. Worker, Python and browser CI green at 32c510d: https://github.com/Deicyde/WikiLean/actions/runs/43458736457. The migration-0019 test listed every site_settings row; it now reads only the two levers 0019 seeds, and wiki/test/bundles.test.ts pins the new one.

Outside contributors' pipelines need the frozen universe's declaration index
and search corpus, about half a gigabyte. Jack chose on 2026-11-11 to have the
Worker stream it to agent-token holders ("Token-gated download").

src/bundles.ts serves a bundle from the private R2 bucket wikilean-contrib
(binding CONTRIB, read-only; unbound, 503):
- GET /api/index/bundles lists the live bundles to an owned agent token;
- GET /api/index/bundles/<manifest sha256> serves the MANIFEST.json bytes;
- GET .../files/<path> streams a listed file, whole or by one byte range
  (If-Range, 206, 416), with an ETag of its sha256, If-None-Match 304 and HEAD
  answered from the manifest, `private, no-store`, and the path taken as sent,
  never decoded;
- POST /api/index/bundles registers an uploaded bundle (pipeline bearer only):
  it reads MANIFEST.json from the bucket, checks its sha256, its schema and
  that it carries its libraries' licences, and that the bucket holds every
  listed file at its size.

A download needs an owned agent whose owner accepted CC0, a live bundle and
the index_downloads lever open. Each range served is one row of
index_downloads, written by a statement that re-checks all of that and the
owner's 4 GiB daily quota across their agents, so the table is the download
log too (no IP). Migration 0022 seeds the lever `closed` and creates both
tables; leverOpen reads such a lever shut unless it says open. The promoter
gates on index_downloads, the binding test pins both buckets, and
refuse/requirePipeline are now shared from submissions.ts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant