Repository navigation
Dialect authoring: a two-level inventory index whose top level is bounded for every accepted collection (#252) - #256
Conversation
…nded for every accepted collection (#252) The #250 one-level index was checked whole before any page was read, so an accepted Desk collection (450 revisions x 30 fields with 64-byte names) had a 2,180,240-byte index and was refused at the 2 MiB cap. The snapshot, candidate, draft checkpoints, preview and proposal now carry a top index: per original revision its field count and the SHA-256 and canonical byte length of its revision field index, plus the whole-inventory digest. Each revision field index (ordered names and declaration digests) is read through the read port's new readFieldIndex / ReadFieldIndex member and checked against its top entry; declarations are paged and checked as before. Bound proof (ADR-0028 amendment, a Python arithmetic test, and paging.json bounds rows driven in both languages): every top index the shape admits is at most 1,089,098 bytes, a Desk collection's at most 284,098; a Desk revision field index is at most 4,956 bytes and an unescaped 1,000-field one 255,186. Every envelope cap is unchanged. The version-2 record shape of #243 is in no release and changes in place; version-1 records still read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
Conflicts resolved: mutation-rows.json keeps main's agent#223 rows, then the agent#252 rows; both record-shapes files regenerated with the generator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
…limit; one blank set in both languages (#252) Review of record F1: a paging case refuses a field index whose canonical length differs from its top entry while its digest matches, and a bounds row refuses 1,001 revisions inside the byte cap. Two mutation rows (agent#252), Go twins applied by hand. F2: blank text is one explicit code point set in TypeScript and Go, the union of String.prototype.trim and strings.TrimSpace, so labels of only U+0085 or only U+FEFF are refused in both; bounds rows carry both cases and an accepted label that contains them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
|
Carry needed from base commit 6ed3dc4. |
Conflict resolved: mutation-rows.json keeps main's agent#253 rows, then the agent#252 rows; generated files regenerated with their generators. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: kikashy <35789537+kikashy@users.noreply.github.com>
|
Verdict: MERGEABLE: no HIGH. The bound proof holds for every collection the published Desk accepts. Two MEDIUMs to file: the length-only and revision-count safeguards are unpinned, and a pre-existing Go/TypeScript label parity gap. Two LOWs. Clean-room review: agent PR #256 (#252), two-level dialect inventory index
Method
Runs:
Bound proof: verified(a) Desk limits, with citations (Desk main
|
| claim | Desk source | enforced where | verdict |
|---|---|---|---|
| ≤ 1,000 records per collection | internal/desk/input_preparations.go:23 const inputRecordLimit = 1000 |
decodeInputPreparation refuses more (:116). It runs on every read (input_preparation_file.go:26-31) and on every write: writeInputPreparation decodes before writing (input_preparation_file.go:33-50). CompareAndSwap writes only through it (input_preparation_repository.go:130-154). The import route refuses at :244-246. The file adapter is the only repository adapter (input_preparation_repository.go:47-52). |
hard, server-enforced on every path that builds a collection |
| one original revision per record | internal/desk/input_research_contract.go:265-269: a research record carries one recipeVersion |
input_typed_research.go:8-31 pins a record's original declaration to record.Research.RecipeVersion |
consistent with "every inventory revision is the original revision of at least one record", so N ≤ 1,000 |
| 1–30 fields per recipe | input_research_contract.go:138 len(r.Fields) < 1 || len(r.Fields) > 30 |
validateInputResearch, called by decodeInputPreparation (input_preparations.go:122) |
hard for the current recipe; see the caveat below |
| names ≤ 64 ASCII bytes, no escaping | input_research_contract.go:97 ^[a-z][a-zA-Z0-9_]{0,63}$ |
:159 |
hard |
| labels ≤ 160 bytes | an agent-side limit. The Desk label is prep_<32 hex>:recipe:<int>, at most 64 bytes even for a 19-digit int64 revision. |
agent dialectIndex / researchIndex |
conservative: the proof uses 160 |
Caveats. None breaks the proof today.
- The adopter is unpublished. The Desk code that builds the inventory and serves
readFieldIndexis in neither Desk main nor v0.13.2: main pins harness v0.3.0, v0.13.2 pins v0.1.0. So "the inventory lists only revisions some record holds" is supported by the published join (input_typed_research.go), not by the adopter itself.- The collection also keeps a
recipes[]history (input_preparations.go:78-98, appended at:323). Its count has no bound other than the 8 MiB file limit. - An adopter that listed every
recipes[]entry instead of the records' original revisions could exceed 1,000 revisions. The agent would then refuse the collection; it would not overflow a cap. - The Desk should qualify this when it adopts. It is listed under not established.
- The collection also keeps a
- Historical recipes are checked only when written.
decodeInputPreparationvalidates only the currentrecipe.research. The 30-field bound on historicalrecipes[i]therefore holds because each was validated when it was current. No earlier, looser limit is visible (the file's history starts at the v0.13.0 squash). - No plan to raise a limit was found. None of the Desk issues searched mentions raising the record, field or name limits.
- Desk #605 (store contract) and the store-kind line would add repository adapters. A new adapter must keep the decode-time record check.
(b) Arithmetic, recomputed independently
| index | arithmetic (independent) | bytes | claimed |
|---|---|---|---|
| available top entry | 117 + digits(bytes) + digits(fields) + escaped label | 117 base | 117 |
| unavailable top entry | 56 + label | 56 | 56 |
| top wrapper | {"digest":"…","revisions":[…]} |
99 | 99 |
| largest admitted top index | 99 + 1,000 × (117 + 7 + 4 + 960) + 999 | 1,089,098 (built and measured) | 1,089,098 |
| largest Desk top index (160-byte labels) | 99 + 1,000 × (117 + 4 + 2 + 160) + 999 | 284,098 | 284,098 |
| issue family N = 450, top index | measured | 77,390 | 77,390 |
| issue family, one-level | 98 + 4,842 × 450 + 1,242 | 2,180,240, SHA 276ebf66… = #252 |
2,180,240 |
| Desk revision field index | 27 + 160 + 30 × 158 + 29 | 4,956 (4,860 with a real 64-byte Desk label) | 4,956 |
| largest unescaped field index | 27 + 160 + 1,000 × 254 + 999 | 255,186 | 255,186 |
| field index page binding | 154 + 6 × (512 + 512 + 128) | 7,066 | 7,066 |
| fully escaped field index (refused) | 27 + 960 + 1,000 × 1,054 + 999 | 1,055,986 > 1,048,576 | "refused by name" |
- Escape factor. The sixfold factor is right for both encoders. Only U+0000–U+001F (other than
\b \t \n \f \r) become\u00XX.- Go's
dialectCanonical(json.Marshal, then parse, thenquoteJS) and TypeScript's JSON.stringify both leave<,>,&, U+2028 and non-ASCII raw. - Labels and names are limited in UTF-8 bytes in both languages, so a 160-byte label costs at most 960 bytes.
- Go's
- Python test. It checks the same numbers. Its 2,178,098-byte "over cap" row is far over the cap (2,000 revisions); the one-byte boundaries are covered by my probes in (c).
(c) Built and checked at the boundaries, in both languages
| case | Go | TS |
|---|---|---|
| issue family N = 450: read, proposed, retained, every page and envelope measured (candidate tests) | pass | pass |
| largest Desk collection, 1,000 × 30: same | pass | pass |
| top index exactly 2,097,152 bytes (1,926 escaped entries) | not byte-refused (refused as malformed: more than 1,000 revisions) | same |
| top index 2,097,153 bytes | "exceeds its byte limit" | same |
| worst admitted top index, 1,089,098 bytes, 1,000 distinct escaped labels | accepted; the field index port is called | same |
top entry bytes 1,048,576 / 1,048,577 |
accepted / "a revision field index exceeds its byte limit" | same / same |
| 1,001 revisions under the cap | malformed | malformed |
| field index page exactly 1,048,576 canonical bytes (994 escaped names, field index 1,048,417) | accepted; the full inventory is recovered | same inventory, same result |
| field index page 1,048,577 | "a revision field index page exceeds its byte limit" | same |
Probes
| # | probe | Go | TS | result |
|---|---|---|---|---|
| P1 | entry bytes off by one, digest right, top digest recomputed |
refused (digest/length) | refused | held |
| P2 | top digest disagrees with entries |
refused | (covered by vectors) | held |
| P3 | duplicated revision | malformed | (vectors) | held |
| P4 | port serves revision F's field index (same names and declaration digests) for entry A | refused (digest) | refused | held |
| P5 | declarations of A served under label F (identical name and bytes) | refused (order/revision) | refused | held |
| P6 | fields reordered within a field index | refused (digest) | (vectors) | held |
| P7 | field index page from another inventory digest; page answering another entry | refused (binding) | (row-pinned) | held |
| P8 | empty revision whose entry digest is not the empty index's; unavailable entry with a digest; entry field count inflated | refused | (row-pinned) | held |
| P9 | page with an extra declaration | refused (declaration bound) | (vectors) | held |
| P10 | host pages of 1, 2, 4, 7 and 1,000 over revisions [3, 0, null, 1, 5, 3] | exact inventory recovered; empty and unavailable revisions never read | same | held |
| P11 | field index page whose fieldIndex getter answers differently on a second access (#250 copy-before-check) |
n/a | the copy is used; correct inventory | held |
| P12 | parity: Go-built top index over labels and names with ", \, <&>, U+2028/2029, U+0001/001F/007F, \n\t\r\b\f, é, 漢, an astral character, 160 × U+0001, 80 × é |
— | TS dialectResearchIndex equals Go's; both read it back |
held |
| P13 | label byte limits: 80 × é (160 bytes) / 81 × é | ok / malformed | ok / malformed | parity |
| P14 | label "U+0085" alone; label "U+FEFF" alone | malformed / accepted | accepted / malformed | parity gap, pre-existing (F2) |
| P15 | fixture expectedFingerprint |
dumped the Go candidate. My encoder gives b3df2503… = the fixture. The top index equals my independent top index of the fixture inventory. Swapping the old one-level index back in gives 93d69bd8…, which is main's value. |
— | legitimate: only the index shape changed it |
| P16 | version-2 witness in retention.json |
entry digests, sizes (246, 344) and the inventory digest 61da0032… recomputed from the old one-level field digests |
— | exact |
| P17 | TOCTOU, preview to retain | after preview, the factory re-reads and refuses a different inventory digest. The candidate, drafts and proposal carry only the checked top index; no declaration byte is retained. | same | held |
Integrity (item 1). No declaration or field-index byte reaches a candidate, preview, draft, retained proposal, checkpoint, model request or tool answer without both checks: the top entry against the field index (canonical length and SHA-256), and the field digest against the declaration.
- The read tool answers only after
verifyInventoryreads every field index and every page. - The page tool re-reads the snapshot, compares identity and inventory digest, and re-checks the field indexes it spans.
- Envelopes carry only the top index, which
dialectIndexchecked: whole digest, shape, entry bounds, and the empty-revision digests.
Records (item 3).
- Version 1:
- No version-1 code path changed in the diff.
- The version-1 witness in
retention.jsonis unchanged and its tests pass.
- Version 2 (no stale reader is left):
record-shapes --checkpasses, andgo generateregeneratescontract_gen.gowith no diff.dialect-draft.jsonindexRevisionis{revision, fields, digest, bytes}, with the if/then/else for nulls.indexFieldmoved todialect-authoring.json.- The certification scenario host serves field indexes.
- Every file that names
researchIndexwas checked, includingdialects-runtime.test.ts, which is unchanged and compares againstdialectResearchIndex.
- The fingerprint is legitimate (P15).
No regressions (item 4).
- Dialect authoring: carry the complete original inventory by digest and page declarations through the read port (#243) #250: the echo checks (id, packKey, revision, inventoryDigest, cursor) and copy-before-check are intact in
declarationPage, and the newfieldIndexPagecopies before checking (P11). - Run leases and effect-host fencing for unattended runs: owner, authoritative expiry, monotonic epoch, fence at commit and admission (#223) #251: the PR touches no lease, fence or effect-host file. The merge commit's diff against main is dialect files only.
Mutants
| mutant | TS | Go twin |
|---|---|---|
row: field index matches its top entry's digest and length (if(false)) |
caught (3 tests) | caught (TestDialectInventoryPagingVectors/revision_field_index_does_not_match_its_top_entry, TestDialectInventoryHostRefusals/field_index_differs_from_its_top_entry, …/empty_revision_with_another_field_index_digest) |
| row: field index page echoes its entry | caught | caught (…/field_index_page_echoing_another_entry) |
| row: unavailable top entry carries no field index | caught | caught (…/unavailable_entry_with_a_digest) |
own 1: drop only the length half (utf8Bytes(raw)!==entry.bytes / len(canonical) != *entry.Bytes) and keep the digest |
survived the dialect selection (171 tests). The checker's whole-suite fallback reported "caught" only through tests unrelated to dialects (provider matrix, record readers, corpus and Codex replays). None of them names dialects, and the provider-matrix test fails unmutated in the clean clone, so those are environment failures. | survived (full ./context package) |
own 2: drop the 1,000-revision limit (value.revisions.length>L.columns / len(index.Revisions) > DialectColumns) |
survived the dialect selection; the same spurious whole-suite failures as own 1 | survived (full ./context package) |
The Go ./certification package could not run under the 8 GB address-space cap: it spawns Node, which cannot reserve Wasm memory under that cap, and it fails the same way unmutated. Its dialect scenario host is honest, so neither own mutant is reachable from it.
Findings
| # | severity | finding | where | disposition |
|---|---|---|---|---|
| F1 | MEDIUM | Two safeguards the proof leans on are unpinned in both languages. (1) The length half of the field-index check: an entry whose bytes is wrong but whose digest is right passes when the length comparison is removed, and only the length differs (P1 refuses it today). (2) The 1,000-revision structural limit: the "largest admitted top index 1,089,098" and the cursor domain (≤ 1,000 × 1,000 = dialectCursorMax) rest on it, but no test sends more than 1,000 revisions under the cap; the over-cap row uses 2,000 and is refused by bytes. Add two vector rows (bytes off by one with the digest right; 1,001 revisions under the cap) and two mutation rows. |
dialects.ts fieldIndex, researchIndex; dialects.go dialectFieldIndex, dialectIndex |
file; pre-release batch |
| F2 | MEDIUM (pre-existing, not introduced here) | Label admission parity: Go dialectText uses strings.TrimSpace, TS text uses String.prototype.trim. A label of only U+0085 is refused by Go and admitted by TS; a label of only U+FEFF is admitted by Go and refused by TS. Unreachable from the Desk (ASCII prep_…:recipe:N). The same helpers guard other text members since #171. |
dialects.go:373, dialects.ts:125 |
file once; pre-release batch |
| F3 | LOW | Field index re-reads scale with page size. Each declaration page reads every field index in the next ≤ 1,000 flattened positions, even when the host returns fewer declarations; a host paging one declaration at a time re-reads the same field indexes up to 1,000 times. This is bounded, and trivial for the Desk (30-field revisions, under 35 field index reads per page), but a 1,000-field revision at the 255 KB unescaped size served in small pages is hundreds of MB of re-reads per verification. | dialectIndexedRange / indexedRange |
note; rides |
| F4 | LOW | Go has no copy-before-check twin. The TS pages are copied before checking (#250); the Go readers check the host's returned slices in place (page.Declarations is returned as served). This only matters for a host that mutates its own returned slice concurrently (a data race), so it is outside the threat model, but asymmetric with TS. Pre-existing for declaration pages; repeated for field index pages. |
dialectPage, dialectFieldIndexPage |
note; rides |
No HIGH:
- No unchecked byte reaches a model, record or retained dialect.
- The bound proof is true for every collection the published Desk accepts.
- Version 1 reads unchanged.
- No bound is bypassable: the 2 MiB top cap is checked before anything else and applies even without the revision-count limit.
Not established
- The unpublished Desk adopter. It builds the inventory and serves
readFieldIndex, and it is in neither Desk main nor v0.13.2. Whether it lists only the records' original revisions, rather than everyrecipes[]entry (whose count is unbounded), is the Desk's to qualify. If it listedrecipes[], a collection with more than 1,000 recipe revisions would be refused by the agent: a compatibility failure, not a cap overflow. - Future Desk store kinds. Any adapter beyond the file adapter (#605 and the store line) must keep the decode-time 1,000-record check.
- Suites CI owns. I did not run the full TS suite,
go test ./..., go-race, certification,parity.shagainst a Desk pin, packed-smoke or harness-check. Only the dialect subsets, the contract test, record-shapes and go-generate ran. - Toolchain. Go 1.26.10, the manifest's version, is not installed; the runs used 1.26.5.
- The other mutation rows. Six of the nine new rows, and the 14 Dialect factory bounds exclude accepted Desk collections with complete original inventories #243 rows, were not re-run here; the PR reports them caught and CI owns the push-to-main table.
Maintainer disposition (2026-10-30)
- F1 (MEDIUM, the length half of the field-index check and the 1,000-revision limit were unpinned): fixed in this PR at
9750c9f. A paging case with the right digest and a length off by one, and a bounds row of 1,001 revisions inside the byte cap, are refused in both languages. Each new mutation row is caught by its intended test, not the whole-suite fallback, and the Go twins were applied by hand. - F2 (MEDIUM, already on main: U+0085 and U+FEFF blank labels disagreed between Go and TypeScript): fixed in this PR. Both languages use one explicit set of blank code points, and new vectors cover both labels alone and beside a letter.
- F3, F4 (LOW): left as they are. They are bounded and do not change any verdict.
- Desk note: the Desk's inventory builder, not yet published, must take revisions from the records' original revisions, at most 1,000, and not from the whole recipe history. This goes in the v0.12.0 release notes.
- The PR merged main (Run leases: bind effect-host certification to the admitted instance; test expiry on the fenced commit paths (#253) #257) at
b1e79d9. CI is green at that head.
|
🎉 Thank you for your contribution to Judgment Pack! If you’d like another task, choose a path that interests you: |
Closes #252. Milestone v0.12.0. Follows #243 and #250 (ADR-0028); the maintainer decision of #243 holds: complete original provenance, by digest, read through the read port.
What was wrong
#250 carried a one-level index (every revision's ordered field names and declaration digests) and checked it whole against the 2,097,152-byte cap before any page was read. The issue's accepted Desk collection (450 revisions x 30 text fields, 64-byte names, 4,920,611 bytes) has a one-level index of
98 + 4842·N + Σdigits(1..N)= 2,180,240 bytes. Onmainthe read refuses it:Dialect authoring: the original inventory index exceeds its byte limit.(run against an export oforigin/main, same family and generator as the vectors).The contract now
researchIndex; carried by the snapshot, candidate, both draft checkpoints, the preview candidate and the proposal): per original revision{revision, fields, digest, bytes}(field count, and the SHA-256 and canonical byte length of that revision's field index; all three null when unavailable), plusdigest, the SHA-256 of the canonical JSON ofrevisions(the whole-inventory digest every fingerprint and currentness check already binds).{revision, fields:[{name, digest}]}, read through the read port's new memberreadFieldIndex(request, signal)/ReadFieldIndex(ctx, request). Request:id,packKey, collectionrevision,inventoryDigest,entry. The page echoes them and carriesfieldIndex. Refused, as tool-input failures: a page over 1 MiB, bound to another revision/inventory/id/pack/entry, a field index whose canonical length or SHA-256 differs from its top entry, another revision label, another field count, a malformed field, a missing port. An empty revision is not read but its field index is still checked against its entry. A top entry naming a field index over the page bound is refused at the read.dialectFieldIndex/DialectFieldIndexOf;readDialectDeclarations({readFieldIndex, readDeclarations}, …)andReadDialectDeclarations(ctx, readFieldIndex, readDeclarations, …).researchInventorysummary,cursor/nextCursor).Bound proof (ADR-0028 amendment; arithmetic in the Python contract test; sizes measured in both languages from
paging.jsonbounds)Host domain: a Desk collection holds at most 1,000 records, and every inventory revision is the original revision of at least one record, so N ≤ 1,000 (also this contract's existing structural limit). The 8 MiB byte limit alone does not bound N tightly enough (a minimal revision is a few hundred bytes), so the revision dimension rests on the record limit. Fields 1–30 per recipe, names
[a-z][a-zA-Z0-9_]{0,63}(≤ 64 ASCII bytes), labels ≤ 160 bytes.A field index page adds at most 7,066 bytes of binding. Envelopes: the inventory share of every envelope fell from up to 2,097,152 bytes to at most 1,089,098 (284,098 for Desk collections); no cap changed. Names needing JSON escapes near 1,000 × 160 bytes could exceed the page bound and are refused by name; Desk names never need escaping.
Records
The #250 version-2 shape is in no release (v0.12.0 is unpublished), so it changes in place; no version 3.
researchIndexnow holds the top index;dialect-draft.jsonindexRevisionbecomes{revision, fields, digest, bytes}with an if/then/else tying the nulls, and itsindexFieldmoves todialect-authoring.json, which gainsindexField,fieldIndex,fieldIndexRequest,fieldIndexPage. Version-1 records still read. Host-facing: GoDialectIndexRevision.Fieldsis now*intwithDigest/Bytes;DialectAuthoringPorts.ReadFieldIndexis required when the top index names a field;ReadDialectDeclarationstakes the field index reader first.Files
go/context/dialects.go,go/certification/scenarios_dialects.go(scenario host serves field indexes),go/context/dialects_inventory_test.go,dialects_test.go,dialects_retained_test.go,dialects_compatibility_test.go, regeneratedgo/contract/contract_gen.goandrecord_shapes.json.packages/harness/dialects.ts,testing/dialects.ts(test host serves field indexes),dialects-inventory.test.ts,dialects.test.ts,dialects-retained.test.ts,dialects-compatibility.test.ts,contract/record-shapes.json,scripts/mutation-rows.json.contract/README.mdrows,paging.json(top index,fieldIndexes,fieldIndexRequests, a new refusal case,boundsrows),inputs.json(field index request and page witnesses, new inventory digest),retention.json(version-2 witness carries the top index),authoring.jsonexpectedFingerprint(recomputed outside the package; the old value was reproduced first by the same independent encoder),tests/test_dialect_authoring_contract.py.docs/design/dialect-authoring.mdpaging section rewritten.No file moved from the Desk changed;
scripts/parity.shrows are unchanged and no new file was added.Tests (both languages; vectors handwritten, digests and sizes computed with Python's hashlib outside the package)
boundsrows (GoTestDialectInventoryBounds, TS "the Complete retained-inventory index still exceeds the 2 MiB cap for an accepted Desk collection #252 bound proof…"): the issue's family at N = 450 (fails on main as above, retained here, one-level size asserted 2,180,240); the largest accepted Desk collection (1,000 × 30, 160-byte labels, 64-byte names); the largest unescaped field index; the largest admitted top index (accepted at 1,089,098 bytes); a top index over its cap (refused); a field index over the page bound (refused). Retained rows measure every served page and the snapshot, candidate, draft and proposal against their caps, and the reader recovers the exact inventory.test_bound_proof_arithmetic(the table above from counted encodings) and the rewrittentest_paging_vector_digests_are_exact.Safeguards and mutants
Nine new TypeScript rows, origin
agent#252, grouped at the end; Go twins applied by hand to a scratch copy of the module. Each failed at its intended assertion (the expected refusal):Four #243 rows changed only their
findtext to stay unique (page from another revision, page byte bound, page echoes id / packKey); the index byte bound row's note now names the bound row. All 23dialect inventoryrows plus the two retained-inventory rows: 23 chosen, 23 caught; Go twin of the top-index byte bound caught.Checks
Candidate: head
ea30585b39bb4b6cea3b583c3f37972063d8f9b4, no uncommitted diff.jpdev check --changed origin/mainselected 24 of 30 steps. Local Go is 1.26.5 (the manifest names 1.26.10, not installed); Node 22; Python 3.12.Self-checks: (a) no boundary path changed. (b) every added schema member is carried by a vector (the Python member-deletion tests). (c) both readers pass all added vectors. (d) rows and twins above. (e) every new refusal is a tool-input failure on the existing tool_result path; no new event kind. (f) no execution-checkpoint member added; the top index rides the existing draft checkpoints. (g) ADR and design text read against the code.
CI owns the exact-head run and the push-to-main mutation table.
🤖 Generated with Claude Code