Repository navigation
Conversation
…tor bucket Outside contributors are to download the declaration index from a private R2 bucket through the Worker (Jack, 2026-11-11: "Token-gated download"; ATLAS included). A copy handed out cannot be recalled, so the licences travel with it. package_index.py (bundle schema /2): - --licence-texts DIR copies each library's LICENSE, LICENCE or COPYING file and any NOTICE into licences/<library>/, lists them in the manifest (licences.texts) and LICENCES.md, and refuses a library with none, or a folder that names no library; - the converter's provenance keeps its shas and counts but no absolute path of the operator's machine; - an index whose header names another universe than the universe document is refused; - the command prints the MANIFEST.json sha256, the bundle's identity. upload_contrib.py (new, operator only, a dry run by default) puts a bundle in the contributor bucket under index/v1/<manifest sha256>/: every key derived from content, never replaced (same bytes skipped, other bytes refused), MANIFEST.json last, each upload checked by its ETag and the prefix listed again at the end. It refuses a bundle with a library that records no licence or has no licence texts, and use restrictions not named exactly with --acknowledge-restricted. It reuses upload_store.py's SigV4 client, whose credential helpers now take the variable names, and reads its own token (WIKILEAN_CONTRIB_R2_*), which docs/SECRETS.md now lists and CI unsets.
Deicyde
marked this pull request as ready for review
November 13, 2026 14:43
…ork packages upload_contrib.py package <envelope> puts a package's transport envelope at packages/v1/<its sha256>.json, a dry run unless --execute, never replacing a key. It refuses an envelope that is not in canonical form, or whose files are not exactly the ones its package.json names, so the sha256 the Worker publishes it under is a function of the package.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first half of the token-gated index download (Jack, 2026-11-11: "Token-gated download (Recommended)", and "Include ATLAS"): the bundle carries its libraries' licence texts, and a script puts it in the contributor bucket. The Worker routes that serve it come in a separate, release-affecting PR; this one is release-neutral.
The bundle (
scripts/pipeline/package_index.py, schemaindex-bundle/2).--licence-texts DIRtakesDIR/<library id>/per library and copies itsLICENSE,LICENCEorCOPYINGfile and anyNOTICEintolicences/<library>/. They are listed in the manifest (licences.texts) and linked fromLICENCES.md. A library with no licence file is refused before anything is written, and so is a folder that names no library (a typo would otherwise drop one library's texts unnoticed). A copy handed to a contributor cannot be recalled, which is why the texts go inside it.pathordiris cut to its name: a published manifest had the operator's home directory in it.MANIFEST.json, which is the bundle's identity in the bucket and, later, in the Worker.The upload (
scripts/articles/upload_contrib.py, new, operator only, a dry run by default). It puts a bundle underindex/v1/<manifest sha256>/in the contributor bucket (wikilean-contrib). Every key is derived from content and never replaced: an object with the same size and MD5 is skipped, one with other bytes is refused, and so is an object under the prefix the bundle does not list.MANIFEST.jsongoes last, each upload is checked by its ETag, and the prefix is listed again at the end. Before any request it hashes every file against the manifest and refuses an unlisted file, a library that records no licence or has no licence texts, and use restrictions not named exactly with--acknowledge-restricted(ATLAS is one). It reusesupload_store.py's SigV4 client, whose credential helpers now take the variable names, and reads its own token,WIKILEAN_CONTRIB_R2_*: R2 scopes a token to buckets, and the articles upload must not be able to overwrite a bundle.docs/SECRETS.mdlists it (not created yet) andscripts/ci-python.shunsets it.Work packages.
upload_contrib.py package <envelope>puts one work package for the claims API (#196) atpackages/v1/<envelope sha256>.json, the keyPOST /api/packageschecks. It refuses an envelope that is not in canonical form (sorted keys, no spaces, ASCII) or whose files are not exactly the ones itspackage.jsonnames, so the published sha256 is a function of the package. Same token, same never-replace rule, a dry run unless--execute.Release.
scripts/anddocs/only, so release-neutral. Nothing is uploaded until the operator runs the script with the new token.Checks. CI on 4c7d13e: worker, python (the uploader's 19 tests, 4 of them for packages, and the packager's), browser and attribution passed.
https://github.com/Deicyde/WikiLean/actions/runs/43525480984