Repository navigation
Conversation
Phase 2 of the split: a locate run planned from a sealed
wikilean.statement-set/1 instead of a harvested cut, and the carry-over map
published so the reader can show the tags of the cut before it.
locate.py --statement-set (refused beside --inventory) reads the set through
statements.py, which refuses a sourceset that has moved on from the revision the
set was cut from and recomputes every span_sha256 and passage_key against that
revision's own render before a task is cut. run.json's locate.statement_set
names the set by id and seal sha256; the task manifests say nothing about it.
export_records.py --statement-set takes each passage's Cut from the set and
refuses a row whose own cut states another. It goes with --rendered-sources and
only with it, because the published cut is checked twice before a record is
written from it: against the render, and against the sources.jsonl the export is
about to seal — a record set whose sources named another layer would publish
spans that are offsets on nothing. The layers also hold every free-text field
the export writes to the quote cap, which is why a render is never read without
a cut to check it against.
**The sealed quote is derived, not copied.** A cut passage carries the quote the
worker found it at; a record carries the format's capped head
(records.cut_passage). Phase 1 publishes the second, so the passage a reader
tags and the location phase 2 writes show the same words. The passage's own
quote is still held to gate rule 2 — unique, at the span's start, inside the
span — so a span that does not hold the words it was cut at cannot seal as a
statement of the few characters it does hold. One consequence, and the only
place the two differ downstream: a locate task manifest planned from a published
cut hands the worker that capped head where a harvested one hands it the
worker's own quote. Every other byte of every task manifest is the same, so a
run planned from a harvested cut — batch 1 included — is unchanged.
The re-cut. carry.py takes a sealed set as either side (--from-set/--to-set),
derives each entry's old quote from the old layer the same way (so an orphaned
tag shows words the reader really saw), and --store-out writes one <pageid>.json
a page into a directory it holds to being that map and nothing else — a page
that drops out is removed rather than left to go on being published, and a map
with a refused page in it is not published at all, because in the store a page
nobody could map and a page nothing moved on look the same.
build_store.py --carry <dir> publishes them at articles/v1/carry/<pageid>.json,
refusing a document that is not a map of THIS article's previous cut: its new
side must be the article's own source_id and layer_sha256, its keys are checked
at every level, its `same_layer` is read off the two sides' own layer shas, and
its counts are recomputed from its entries rather than taken on trust. A tag
carries only on the `unchanged` fate, where the key is byte-identical and
nothing has to move, so the fates themselves are checked: every `into` entry is
{passage_key, item, span, scope} with a 64-hex key, an `unchanged` passage went
into ONE new passage carrying its own passage_key, `split` into two or more,
`removed` and `changed` into none, and the rest into exactly one — a document
that says `unchanged` of a passage that went somewhere else would offer the
reader a one-click re-apply onto another passage. --carry over a directory with
nothing in it builds the store it would have built. Nothing auto-moves.
Two refusals the publication needs on the pipeline side: --store-out is refused
beside --source, because it writes the WHOLE map into a directory it holds to
being that map alone, so a per-page run would delete every page it did not name;
and a sealed set used as a side is checked against the render, because the map's
keys are recomputed from the layer and a set whose sealed span_sha256 is not
that layer's would be published under keys the store never had.
An export from a published cut holds the row's SPAN to the sealed one as well as
its quote and span_sha256: a record carries its row's span beside the cut's
span_sha256 (records.subject_end), so a row with no cut of its own, or one whose
cut agrees while its span does not, would otherwise publish the hash of one
stretch of text over the offsets of another.
Tests: R16 again from a sealed set — the cut, the gate, the harvest, the seal,
both plans, the gate, the harvest, the tally and the record set — holding that
the task manifests agree byte for byte but for the quote (and that the staged
gate differs in nothing but those manifests), and that every record's
passage_key is the statement set's. Plus the carry map's new side, its store
documents and its CLI, the builder's --carry refusals, and a challenge plan fed
from statements.passage_index.
The gate fixture's inventory passages now carry `finder` and `fragment`, which
the inventory prompt asks a worker for and the gate ignores.
Release-neutral: scripts/ and docs/ only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of the split: a locate run planned from a sealed
wikilean.statement-set/1instead of a harvested cut, and the carry-over map published so the reader can show the
tags of the cut before it.
locate.py --statement-set(refused beside--inventory) reads the set throughstatements.py, which refuses a sourceset that has moved on from the revision the setwas cut from and recomputes every
span_sha256andpassage_keyagainst thatrevision's own render before a task is cut.
run.json'slocate.statement_setnamesthe set by id and seal sha256; the task manifests say nothing about it.
export_records.py --statement-settakes each passage'sCutfrom the set and refusesa row whose own cut states another. It goes with
--rendered-sourcesand only with it,because the published cut is checked twice before a record is written from it:
against the render, and against the
sources.jsonlthe export is about to seal — arecord set whose sources named another layer would publish spans that are offsets on
nothing. The layers also hold every free-text field the export writes to the quote cap,
which is why a render is never read without a cut to check it against.
The sealed quote is derived, not copied. A cut passage carries the quote the worker
found it at; a record carries the format's capped head (
records.cut_passage). Phase 1publishes the second, so the passage a reader tags and the location phase 2 writes show
the same words. The passage's own quote is still held to gate rule 2 — unique, at the
span's start, inside the span — so a span that does not hold the words it was cut at
cannot seal as a statement of the few characters it does hold. One consequence, and the
only place the two differ downstream: a locate task manifest planned from a published
cut hands the worker that capped head where a harvested one hands it the worker's own
quote. Every other byte of every task manifest is the same, so a run planned from a
harvested cut — batch 1 included — is unchanged.
The re-cut.
carry.pytakes a sealed set as either side (--from-set/--to-set),derives each entry's old quote from the old layer the same way (so an orphaned tag shows
words the reader really saw), and
--store-outwrites one<pageid>.jsona page into adirectory it holds to being that map and nothing else — a page that drops out is removed
rather than left to go on being published, and a map with a refused page in it is not
published at all, because in the store a page nobody could map and a page nothing moved
on look the same.
build_store.py --carry <dir>publishes them atarticles/v1/carry/<pageid>.json,refusing a document that is not a map of this article's previous cut: its
newsidemust be the article's own
source_idandlayer_sha256, its keys are checked at everylevel, and its
countsare recomputed from its entries rather than taken on trust.--carryover a directory with nothing in it builds the store it would have built. Atag carries only on the
unchangedfate, where the key is byte-identical and nothinghas to move; nothing auto-moves.
A record carries its row's
spanbeside the cut'sspan_sha256(
records.subject_end), so the row's span is held to the sealed one as well as its quote:a row whose span has moved — or a row with no cut of its own, which the export accepts
deliberately — would otherwise seal the hash of one stretch of text over the offsets of
another, and the page would highlight text nobody judged.
What the publication refuses, because a published map is something a reader acts on:
--store-outbeside--source(it writes the WHOLE map into a directory it holds tobeing that map alone, so a per-page run would delete every page it did not name); a sealed
set as a side whose sealed
span_sha256is not the render's (the map's keys arerecomputed from the layer, and those would not be the keys the store published); and, in
the builder, a document whose
intoentries are not{passage_key, item, span, scope},whose fates do not mean what they say (
unchangedis ONE new passage carrying theentry's own key — the whole basis for carrying a tag on it —
splittwo or more,removedandchangednone, the rest exactly one), whosescope_changedorconjuncts_changedis set where nothing went anywhere, or whosesame_layeris not whatits two sides' own
layer_sha256say.Tests: R16 again from a sealed set — the cut, the gate, the harvest, the seal, both
plans, the gate, the harvest, the tally and the record set — holding that the task
manifests agree byte for byte but for the quote (and that the staged gate differs in
nothing but those manifests), and that every record's
passage_keyis the statementset's. Plus the span refusal, the carry map's new side, its store documents and its CLI,
the builder's
--carryrefusals, and a challenge plan fed fromstatements.passage_index.The gate fixture's inventory passages now carry
finderandfragment, which theinventory prompt asks a worker for and the gate ignores.
Release-neutral:
scripts/anddocs/only.