Admit bounded indexed diagnostic evidence before compute

docs/contracts/indexed-diagnostics.md

@@ -0,0 +1,116 @@
+# Indexed diagnostics v1
+
+`extension_field_policy_indexed_v1` is an explicit opt-in policy variant for the
+existing noncanonical `external_model_evidence_recipe_v1` and bound
+`external_model_evidence_recipe_v2` recipe families. It adds one required
+`diagnostic_capture` member. Canonical field policies are outside this variant.
+Legacy policies remain closed and cannot gain this member. An arbitrary legacy
+artifact named `diagnostics.json` or using a diagnostic role gains no index
+semantics.
+
+The existing `model_evidence_v1` envelope, profile capability lists, profile Ref
+meaning and public evidence projections are unchanged. The complete admitted
+policy and recipe/material/context bind the addon; its `capture_sha256` is the
+canonical hash of `diagnostic_capture`. A profile Ref alone does not identify
+the addon. Only opted-in execution material adds the shared SDK contract modules
+to its recorded host closure. Retained legacy host-name sets remain unchanged.
+
+## Declaration and admission
+
+`diagnostic_capture` is closed over these fields:
+
+| Field | Contract |
+| --- | --- |
+| `format` | `diagnostic_capture_v1` |
+| `payload_schema` | `diagnostic_json_object_v1`: bounded finite JSON object, with no inferred numerical meaning |
+| `index` | `{name, max_bytes}` matching a required declared output with role `diagnostic_index_v1` |
+| `shards` | Ordered, unique `{name, max_bytes}` entries matching optional declared outputs with role `diagnostic_shard_v1` |
+| `phases` | Unique flat names, at most the selected recipe's output count |
+| `coverage` | Ordered scopes `{case_id, checkpoint_point, phase, required_count, optional_count}` |
+| `records_per_shard` | Positive exact integer; each shard must fit that many maximum-size records plus its JSON envelope |
+| `max_record_bytes` | Maximum canonical UTF-8 bytes of one complete record |
+
+Case IDs must belong to the fixed admitted validation probe. Checkpoint points
+must belong to the complete admitted capture cadence. Phases refer to the
+explicit declared phase list. Scopes sort by admitted probe order, checkpoint
+point, then declared phase order, without repeated scopes. Within each scope,
+required records precede optional records. Contiguous global ordinals identify
+the complete declared population.
+
+The population must fit `len(shards) × records_per_shard`. Shards retain the
+existing selected per-file ceiling, at most 8 MiB. The recipe's existing output
+file-count and aggregate caps remain authoritative. This opt-in variant reserves
+the sum of **all declared output maxima** inside the profile aggregate budget;
+legacy policies keep their existing aggregate semantics. Admission also reserves
+an index large enough for all shard descriptors and uncompressed optional
+omission ranges. Impossible record identity, shard, index, population or aggregate
+reservations refuse before execution admission and grant consumption. Scientific
+array minimum-size checks still apply independently.
+
+Diagnostic names must be distinct from all common scientific documents and
+bound probe/prediction/reference files. They cannot replace scientific evidence.
+
+## SDK producer
+
+Use the supported `tuldok_extension_sdk.evidence.Evidence` writer to finish the
+complete scientific envelope, then call `writer.diagnostics(records, omitted)`.
+The writer checks the full record population, all ranges and serialized byte
+bounds before writing diagnostics. It never samples or truncates required data.
+`writer.inventory()` includes the exact bytes it wrote; this is construction
+metadata, not collection permission.
+
+Each record is closed over `{ordinal, case_id, checkpoint_point,
+checkpoint_sha256, phase, payload}`. Its checkpoint SHA-256 must equal the actual
+retained checkpoint descriptor at that admitted point. `payload` is a finite
+JSON object under the existing SDK JSON depth/node/string bounds. Its content
+does not establish a metric, physical operator or acceptance criterion.
+
+`omitted` is an explicit sorted nonoverlapping list of `{start, count, reason}`
+ranges. Each range stays inside one scope's optional population. Reasons are
+`capture_budget`, `not_available` or `not_requested`. Required ordinals cannot be
+omitted. Every remaining ordinal must occur exactly once, in order.
+
+## Retained index and shards
+
+The index is `application/json`, schema `diagnostic_index_v1`, closed over
+`{format, payload_schema, run_id, material_sha256, capture_sha256, shards,
+total_shard_bytes, coverage}`. It binds the current Run and exact composition
+material. Each ordered shard entry is `{blob, record_count, first_ordinal,
+last_ordinal}`. `blob` is `{name, bytes, sha256, media_type}` with media type
+`application/json`. Shards are consumed in the declaration's order; all except
+the final retained shard contain exactly `records_per_shard` records. Unused
+declared optional shard files must be absent.
+
+Each shard is closed over `{format, run_id, capture_sha256, records}` with format
+`diagnostic_shard_v1`. Every file has its exact size/hash, record count and ordinal
+range checked. `total_shard_bytes` equals the sum of retained shard sizes.
+Coverage is closed over `{total_count, required_count, optional_count,
+retained_required_count, retained_optional_count, omitted_count, omitted}`;
+counts must agree with the admitted population, retained records and omissions.
+Complete mandatory coverage and omitted optional detail remain distinct.
+
+Runs' existing outer artifact inventory retains its generic
+`application/octet-stream` transport media type. The addon specifies JSON's
+semantic media type within that unchanged byte transport.
+
+## Host verification and publication
+
+The existing evidence owner first verifies the complete scientific envelope and
+all current artifact bytes against Runs' retained inventory. Only then does the
+shared diagnostic contract verify those already retained bytes. The proof is
+private `_diagnostics`; existing query/publication evidence projections strip
+private members and retain their closed public grammar.
+
+Publication continuation, custody, permission, accounting, recovery and result
+Refs remain owned by Runs. Consumers inspect or collect the declared index and
+its named shards through the existing exact inventory and whole-hash artifact
+readers. No arbitrary path, glob, alternate store or new export authority exists.
+Recovery reruns the same host verifier. An invalid original scientific output
+still fails; an index cannot repair it or confer numerical acceptance.
+
+Focused executable witnesses are `tests/test_indexed_diagnostics.py` and
+`tests/test_indexed_diagnostics_publication.py`. The latter explicitly uses a
+tiny `/usr/bin/true` metadata runtime and harness serialization before the
+actual no-op child. It qualifies real custody, interrupted-copy recovery, exact
+readers and unchanged accounting; it proves no captured Python producer,
+training, CUDA, numerical or lab-failure classification result.

docs/contracts/run-preflight-publication.md

@@ -321,3 +321,21 @@ The read-only [scientific comparison](scientific-comparison.md) accepts exact
 attached publication-result Refs alongside completed Run operands. It reuses
 publication-result identity and authority checks, preserves the original failed
 or interrupted outcome, and grants no legacy promotion or candidate eligibility.
+
+## Opt-in indexed diagnostic capture
+
+The noncanonical `extension_field_policy_indexed_v1` policy adds a hash-bound
+`diagnostic_capture_v1` declaration. Admission reserves its exact bounded index,
+ordered shard outputs, record populations and complete mandatory coverage under
+the existing output/file/aggregate caps. This opt-in variant reserves all
+declared output maxima within the selected profile aggregate budget. Existing
+policies and closed evidence/public projection grammars retain their meanings.
+
+The current evidence owner verifies scientific evidence first and diagnostic
+bytes against the same retained artifact inventory. Publication recovery and
+exact-hash result readers expose the admitted index/shards under the original
+Run and result Ref; no replay or alternate permission path is introduced. See
+[Indexed diagnostics v1](indexed-diagnostics.md) for the complete producer/host
+contract and the distinction between required coverage and explicit optional
+omission ranges. This generic contract does not classify absent original lab
+diagnostic bytes or repair an invalid original output.

docs/plans/lab-workflow-reliability/reports/m1-indexed-diagnostics/composed-readers/indexed-final-9d563ceb-independent-review.json

@@ -0,0 +1,32 @@
+{
+  "checks_run": [
+    "Read-only frozen source/Metadata inspection",
+    "Built-in compile and stdlib AST/order checks only"
+  ],
+  "format": "indexed_final_fixture_ownership_independent_review_v1",
+  "limits": [
+    "No imports/application/DB/heavy test executed during M4 quiet window.",
+    "Actual2 qualification on9d is still root-owned and pending quiet release.",
+    "Flushed stdout ledger is runner provenance, not a power-loss durability claim."
+  ],
+  "metadata_source_sha256": "893d489f8e5d323ac1eed61350d206bbc1c0375801402f52cc586ac8423d62e4",
+  "prior_review": {
+    "receipt_path": "/tmp/tuldok-indexed-composed-readers-70dcaeb-independent-review.json",
+    "receipt_sha256": "be25642992e49f5a0d89cb5a34a7185470a5ce321443103bf4597ed81c577a04",
+    "source": "70dcaeb0382a97d051275319881acf589bfb931d"
+  },
+  "result": "Clear: exact fresh schema ownership is registered and flushed before Metadata allocation; original composed helper/accounting review remains applicable",
+  "reviewed_at_utc": "2026-10-19T23:03:47.661032+00:00",
+  "reviewed_file": "tests/test_indexed_diagnostics_publication.py",
+  "reviewed_file_sha256": "7e14d96a4d2a5d6e2be5aa884770a8b57a32653e009c890f68291f125551fa11",
+  "reviewed_source": "9d563ceb72fdfa7cb3ce520540b4b002cbe3fba3",
+  "verified": [
+    "UUID identity and catalog_<identity> schema assigned before construction; self.dataset=None and catalog cleanup registered before Metadata(...dataset_id=identity).",
+    "Ledger print uses flush=True before constructor; runner can attribute a later constructor failure to this exact fresh schema, without discovering or guessing older schema identities.",
+    "Metadata honors explicit dataset_id, validates UUID identity and derives exact catalog_<identity> schema. Current owner implementation closes its connection on initialization exception.",
+    "Schema equality asserted and initial Metadata closed before Dataset reopen. Registered cleanup closes self.dataset when present and uses parameterized SQL Identifier for only self.schema.",
+    "DROP SCHEMA IF EXISTS handles failure before schema creation; cleanup does not enumerate, infer, adopt or delete older unknown schemas.",
+    "Cleanup registration follows temporary-directory cleanup registration, so unittest LIFO closes/drops catalog before removing its temporary catalog directory.",
+    "AST equality proves all methods except fixture setUp/cleanup_catalog unchanged from70; composed batch/scientific helper, no-replay and accounting assertions preserved."
+  ]
+}

docs/plans/lab-workflow-reliability/reports/m1-indexed-diagnostics/composed-readers/qualification.md

@@ -0,0 +1,9 @@
+# Composed indexed evidence qualification
+
+At exact source `9d563ceb72fdfa7cb3ce520540b4b002cbe3fba3`, both actual PostgreSQL/custodian cases passed in 22.431 seconds with no skips. The original captured-SDK serialization, actual no-op child, copy interruption, cold recovery, mandatory/optional coverage, exact artifacts and accounting/event conservation assertions remain. The source manifest records the final production and test file hashes.
+
+The recovered indexed result was additionally inspected through a real cold inspect AgentClient: WorkflowClient collected its original index and two shards through one batch call, with no individual segment fallback, matching whole SHA256 and private 0600 files. The scientific comparison operation read the same exact Ref twice with exact point/operand correlation, diagnostic delta zero and unknown criterion. This tests composition; it does not compare different models or establish scientific acceptance. Serialization remains harness-authored before an actual /usr/bin/true no-op child, with dummy runtime packages. No captured Python producer execution, training or CUDA is claimed.
+
+Independent source review found a missing private destination directory before execution; commit70dcaeb fixed it. The final fixture at9d563ce additionally reserves and logs its exact fresh schema before Metadata construction. Both pre-registered schemas were independently queried absent after owner cleanup. The shared cleanup receipt also clearly labels two separate refined-performance schemas; those are not indexed test fixtures. Older unowned schema candidates remain untouched.
+
+Hosted lifecycle/browser acceptance and serial integration remain separate gates. The original lab diagnostic failure path/bytes and scientific criteria remain missing inputs.

docs/plans/lab-workflow-reliability/reports/m1-indexed-diagnostics/composed-readers/tuldok-indexed-composed-actual.log

@@ -0,0 +1,12 @@
+test_impossible_population_refuses_before_graph_run_grant_or_reservation (test_indexed_diagnostics_publication.IndexedDiagnosticPublicationTests.test_impossible_population_refuses_before_graph_run_grant_or_reservation) ... Reserved exact fresh schema before construction: catalog_b8674cd6642d40c59051fda95a7cb9b5
+ok
+test_real_custody_cold_recovery_and_exact_inventory_read_with_no_replay (test_indexed_diagnostics_publication.IndexedDiagnosticPublicationTests.test_real_custody_cold_recovery_and_exact_inventory_read_with_no_replay) ... Removed exact recorded fresh schema: catalog_b8674cd6642d40c59051fda95a7cb9b5
+Reserved exact fresh schema before construction: catalog_fc607ffa4125402994f62c319f632faa
+ok
+
+----------------------------------------------------------------------
+Ran 2 tests in 22.431s
+
+OK
+INDEXED_DIAGNOSTIC_QUALIFICATION {"accounting_and_events_unchanged": true, "artifacts": [{"bytes": 948, "media_type": "application/octet-stream", "name": "diagnostic-index.json", "role": "diagnostic_index_v1", "sha256": "351f54de24ac9824a69a641674b1d63246bc2541c2d40f43bfc19bcbd94419d3"}, {"bytes": 983, "media_type": "application/octet-stream", "name": "diagnostic-0.json", "role": "diagnostic_shard_v1", "sha256": "bb0f9f8361a1380c02dfad134f2352e40f40da8dfa1b79347c0b3a47aba90543"}, {"bytes": 443, "media_type": "application/octet-stream", "name": "diagnostic-1.json", "role": "diagnostic_shard_v1", "sha256": "b98746f69577f240ee39204f4b6030b91c9e46b1a674d4164afb52afc06ef133"}], "captured_python_producer_executed": false, "composed_readers": {"batch_calls": 1, "batch_collected_exact_artifacts": ["diagnostic-index.json", "diagnostic-0.json", "diagnostic-1.json"], "comparison": "same exact Ref diagnostic delta zero; no scientific acceptance", "individual_segment_calls": 0}, "counts_after": {"access_execution_consumptions": 1, "access_execution_grants": 1, "model_attempt_reservations": 1, "workspace_graphs": 1, "workspace_runs": 1}, "counts_before": {"access_execution_consumptions": 1, "access_execution_grants": 1, "model_attempt_reservations": 1, "workspace_graphs": 1, "workspace_runs": 1}, "coverage": {"omitted": [{"count": 1, "reason": "capture_budget", "start": 2}], "omitted_count": 1, "optional_count": 3, "required_count": 2, "retained_optional_count": 2, "retained_required_count": 2, "total_count": 5}, "custody_outcome": "completed", "numerical_acceptance": "not_evaluated", "original_state": "interrupted", "result_ref": {"catalog_id": "fc607ffa4125402994f62c319f632faa", "custody_receipt_sha256": "66d76987a01903d9bcfdb06a991d411b729064c0a2a29461ae62d295e5c4edec", "inventory_sha256": "552f458cfd5e606c3f306bf4878946636e2e78bf6a53c6f648074a9aba7d9a1b", "kind": "recovered_publication_result", "material_sha256": "4c17d900d38acccd75a2bf299beda456e8693091b9bb5a8f77056732ae5f801d", "run_id": "d6a762d587d5475d82a56c0dbe4d2384", "schema_version": 1, "sha256": "ff56088bdaaa60497e06eeab578df457e5f1a24b085680f28daa5296e59970fd"}, "run_id": "d6a762d587d5475d82a56c0dbe4d2384", "runtime_packages": {"numpy": "0.0", "torch": "0.0"}, "runtime_snapshot_sha256": "cd69d4b1de535e6c59efc66b6b376991e7c9afdc2d3e7cefcc8f0ea18523d164", "schema": "catalog_fc607ffa4125402994f62c319f632faa", "scope": "harness SDK serialization; actual no-op metadata runtime custody and semantic recovery", "training_executed": false, "verification_scope": "source_destination_and_cold_exact_ref_reader"}
+Removed exact recorded fresh schema: catalog_fc607ffa4125402994f62c319f632faa