Skip to content

Locate from a published cut, and publish what a re-cut did - #188

Open
Deicyde wants to merge 1 commit into
statements/schemafrom
statements/locate
Open

Deicyde wants to merge 1 commit into
statements/schemafrom
statements/locate

Conversation

@Deicyde

@Deicyde Deicyde commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Phase 2 of the split: a locate run planned from a sealed wikilean.statement-set/1
instead of a harvested cut, and the carry-over map published so the reader can show the
tags of the cut before it.

locate.py --statement-set (refused beside --inventory) reads the set through
statements.py, which refuses a sourceset that has moved on from the revision the set
was cut from and recomputes every span_sha256 and passage_key against that
revision's own render before a task is cut. run.json's locate.statement_set names
the set by id and seal sha256; the task manifests say nothing about it.

export_records.py --statement-set takes each passage's Cut from the set and refuses
a row whose own cut states another. It goes with --rendered-sources and only with it,
because the published cut is checked twice before a record is written from it:
against the render, and against the sources.jsonl the export is about to seal — a
record set whose sources named another layer would publish spans that are offsets on
nothing. The layers also hold every free-text field the export writes to the quote cap,
which is why a render is never read without a cut to check it against.

The sealed quote is derived, not copied. A cut passage carries the quote the worker
found it at; a record carries the format's capped head (records.cut_passage). Phase 1
publishes the second, so the passage a reader tags and the location phase 2 writes show
the same words. The passage's own quote is still held to gate rule 2 — unique, at the
span's start, inside the span — so a span that does not hold the words it was cut at
cannot seal as a statement of the few characters it does hold. One consequence, and the
only place the two differ downstream: a locate task manifest planned from a published
cut hands the worker that capped head where a harvested one hands it the worker's own
quote. Every other byte of every task manifest is the same, so a run planned from a
harvested cut — batch 1 included — is unchanged.

The re-cut. carry.py takes a sealed set as either side (--from-set/--to-set),
derives each entry's old quote from the old layer the same way (so an orphaned tag shows
words the reader really saw), and --store-out writes one <pageid>.json a page into a
directory it holds to being that map and nothing else — a page that drops out is removed
rather than left to go on being published, and a map with a refused page in it is not
published at all, because in the store a page nobody could map and a page nothing moved
on look the same.

build_store.py --carry <dir> publishes them at articles/v1/carry/<pageid>.json,
refusing a document that is not a map of this article's previous cut: its new side
must be the article's own source_id and layer_sha256, its keys are checked at every
level, and its counts are recomputed from its entries rather than taken on trust.
--carry over a directory with nothing in it builds the store it would have built. A
tag carries only on the unchanged fate, where the key is byte-identical and nothing
has to move; nothing auto-moves.

A record carries its row's span beside the cut's span_sha256
(records.subject_end), so the row's span is held to the sealed one as well as its quote:
a row whose span has moved — or a row with no cut of its own, which the export accepts
deliberately — would otherwise seal the hash of one stretch of text over the offsets of
another, and the page would highlight text nobody judged.

What the publication refuses, because a published map is something a reader acts on:
--store-out beside --source (it writes the WHOLE map into a directory it holds to
being that map alone, so a per-page run would delete every page it did not name); a sealed
set as a side whose sealed span_sha256 is not the render's (the map's keys are
recomputed from the layer, and those would not be the keys the store published); and, in
the builder, a document whose into entries are not {passage_key, item, span, scope},
whose fates do not mean what they say (unchanged is ONE new passage carrying the
entry's own key — the whole basis for carrying a tag on it — split two or more,
removed and changed none, the rest exactly one), whose scope_changed or
conjuncts_changed is set where nothing went anywhere, or whose same_layer is not what
its two sides' own layer_sha256 say.

Tests: R16 again from a sealed set — the cut, the gate, the harvest, the seal, both
plans, the gate, the harvest, the tally and the record set — holding that the task
manifests agree byte for byte but for the quote (and that the staged gate differs in
nothing but those manifests), and that every record's passage_key is the statement
set's. Plus the span refusal, the carry map's new side, its store documents and its CLI,
the builder's --carry refusals, and a challenge plan fed from
statements.passage_index.

The gate fixture's inventory passages now carry finder and fragment, which the
inventory prompt asks a worker for and the gate ignores.

Release-neutral: scripts/ and docs/ only.

Phase 2 of the split: a locate run planned from a sealed
wikilean.statement-set/1 instead of a harvested cut, and the carry-over map
published so the reader can show the tags of the cut before it.

locate.py --statement-set (refused beside --inventory) reads the set through
statements.py, which refuses a sourceset that has moved on from the revision the
set was cut from and recomputes every span_sha256 and passage_key against that
revision's own render before a task is cut. run.json's locate.statement_set
names the set by id and seal sha256; the task manifests say nothing about it.

export_records.py --statement-set takes each passage's Cut from the set and
refuses a row whose own cut states another. It goes with --rendered-sources and
only with it, because the published cut is checked twice before a record is
written from it: against the render, and against the sources.jsonl the export is
about to seal — a record set whose sources named another layer would publish
spans that are offsets on nothing. The layers also hold every free-text field
the export writes to the quote cap, which is why a render is never read without
a cut to check it against.

**The sealed quote is derived, not copied.** A cut passage carries the quote the
worker found it at; a record carries the format's capped head
(records.cut_passage). Phase 1 publishes the second, so the passage a reader
tags and the location phase 2 writes show the same words. The passage's own
quote is still held to gate rule 2 — unique, at the span's start, inside the
span — so a span that does not hold the words it was cut at cannot seal as a
statement of the few characters it does hold. One consequence, and the only
place the two differ downstream: a locate task manifest planned from a published
cut hands the worker that capped head where a harvested one hands it the
worker's own quote. Every other byte of every task manifest is the same, so a
run planned from a harvested cut — batch 1 included — is unchanged.

The re-cut. carry.py takes a sealed set as either side (--from-set/--to-set),
derives each entry's old quote from the old layer the same way (so an orphaned
tag shows words the reader really saw), and --store-out writes one <pageid>.json
a page into a directory it holds to being that map and nothing else — a page
that drops out is removed rather than left to go on being published, and a map
with a refused page in it is not published at all, because in the store a page
nobody could map and a page nothing moved on look the same.

build_store.py --carry <dir> publishes them at articles/v1/carry/<pageid>.json,
refusing a document that is not a map of THIS article's previous cut: its new
side must be the article's own source_id and layer_sha256, its keys are checked
at every level, its `same_layer` is read off the two sides' own layer shas, and
its counts are recomputed from its entries rather than taken on trust. A tag
carries only on the `unchanged` fate, where the key is byte-identical and
nothing has to move, so the fates themselves are checked: every `into` entry is
{passage_key, item, span, scope} with a 64-hex key, an `unchanged` passage went
into ONE new passage carrying its own passage_key, `split` into two or more,
`removed` and `changed` into none, and the rest into exactly one — a document
that says `unchanged` of a passage that went somewhere else would offer the
reader a one-click re-apply onto another passage. --carry over a directory with
nothing in it builds the store it would have built. Nothing auto-moves.

Two refusals the publication needs on the pipeline side: --store-out is refused
beside --source, because it writes the WHOLE map into a directory it holds to
being that map alone, so a per-page run would delete every page it did not name;
and a sealed set used as a side is checked against the render, because the map's
keys are recomputed from the layer and a set whose sealed span_sha256 is not
that layer's would be published under keys the store never had.

An export from a published cut holds the row's SPAN to the sealed one as well as
its quote and span_sha256: a record carries its row's span beside the cut's
span_sha256 (records.subject_end), so a row with no cut of its own, or one whose
cut agrees while its span does not, would otherwise publish the hash of one
stretch of text over the offsets of another.

Tests: R16 again from a sealed set — the cut, the gate, the harvest, the seal,
both plans, the gate, the harvest, the tally and the record set — holding that
the task manifests agree byte for byte but for the quote (and that the staged
gate differs in nothing but those manifests), and that every record's
passage_key is the statement set's. Plus the carry map's new side, its store
documents and its CLI, the builder's --carry refusals, and a challenge plan fed
from statements.passage_index.

The gate fixture's inventory passages now carry `finder` and `fragment`, which
the inventory prompt asks a worker for and the gate ignores.

Release-neutral: scripts/ and docs/ only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant