Trajectory step 2: measurement provenance; pending ledger reader tolerates new fields

CHANGELOG.md

@@ -12,6 +12,7 @@ Versions follow [SemVer](https://semver.org).
 - `outerloop trajectory <run-id>` now ends with the run's refusal count and its author models, for release canary comparisons; the agent-text inventory includes operator notes (#487, part 3). Session events record `role` (`author` or `judge`). Upgrading: session events written before this kernel have no `role` and count as author sessions.
 - On macOS without an image, sessions, evaluations and panel lenses run under a packaged Seatbelt profile (`sandbox-exec`, #292): they write only the workspace and a scratch directory, read only system and toolchain paths, and connect out only to TCP port 443, DNS and local ports. Linux and Apptainer runs are unchanged.
 - Kernel events now append to a per-run `trajectory.jsonl` with sequence numbers and a SHA-256 chain. `outerloop trajectory <run-id> [--verify] [--json]` reads it without changing run state. Operational files and their formats are unchanged; older kernels ignore the new file. Upgrading: runs started before this kernel have no trajectory and read as "no trajectory" until their next recorded event. No backfill is attempted.
+- `measure` trajectory events carry provenance: the ledger's ruler digest (the same value a pending submission computes), the container image digest when an image is configured, and refs to the evaluation result files the kernel read. Self-reported measures record the ruler alone. Pending ledger records now ignore fields this kernel does not know, so a later release can add one without breaking this release's ledger read. Upgrading: no action needed; older kernels ignore the trajectory, and pending records are unchanged.
 - Add a printable agent-text inventory and protocol wording lint, with clearer refusal recovery guidance (#487, parts 1–2).
 - `BENCHMARKS.md` shows `—` instead of "provenance unknown" in the main-commit column of a baseline row that nothing has improved on yet, as on a freshly seeded repository.
 - The README and the getting-started guide link to the quickstart repository. It runs two CPU benchmarks with the local forge and local compute, and needs no GitHub account or cluster (#563). The docs and the startup warning no longer say an uncontained session runs in a sandbox: on Linux without an image, it gets a scrubbed environment and its own home, but can read any file the account can.

docs/design/trajectory.md

@@ -38,4 +38,12 @@ The chain detects internal edits and missing lines, but without a trusted
 external anchor cannot detect a rewritten chain or removal of its tail.
 Verification checks the event chain, not the contents of referenced artifacts.
 
-Tool calls, full measurement provenance and attestation are later steps.
+`measure` events carry provenance. They record the ledger's ruler digest (the
+measurement signature plus the non-solver tree entries, the value a pending
+submission also computes), the container image digest when one is configured,
+and refs to the evaluation result files the kernel read under the run
+directory. Self-reported measures record the ruler alone: they have no
+evaluation image or result files. A pending submission does not yet name the
+trajectory head: kernels before this one reject unknown pending fields, so the
+pending reader now ignores them, and a later release can add the field without
+breaking this one. Tool calls and attestation are later steps.

src/outerloop/attempt.py

@@ -89,10 +89,11 @@
     measurement_pending,
     observe_target,
     queue_pending,
+    ruler_identity,
 )
 from outerloop.limits import bound_limits, capped_limits, clamp_bound_limits, effective_limits
 from outerloop.markers import has_marker, marker
-from outerloop.measure import DispatchedMeasurer, DispatchSettings
+from outerloop.measure import DispatchedMeasurer, DispatchSettings, measured_image
 from outerloop.orchestrator import (
     MAX_REPORT_BODY,
     AttemptResult,
@@ -165,6 +166,16 @@ def _pr_number(pr_url: str) -> int:
 
 log = logging.getLogger(__name__)
 
+
+def _ruler_provenance(ws: Workspace, contract: Contract, bench: Benchmark, sha: str) -> str:
+    """Best-effort ledger ruler digest; a git read must not fail a wake."""
+    try:
+        return ruler_identity(ws, contract, bench, sha)[0]
+    except Exception as exc:
+        log.warning("could not compute ruler identity: %s", exc)
+        return ""
+
+
 # Where a climb job reads its keys unless the CLI flags say otherwise. The
 # tick preflights the panel key (and compares it against the author key —
 # role separation) before claiming/submitting.
@@ -3369,17 +3380,26 @@ def _resume_run(
     # more experiments, or concludes — instead of being decided by policy here.
     # Falls back to the plain-finish behavior (negative terminal / draft PR)
     # when the session cannot be resumed.
+    measure_data = {
+        "baseline": result.baseline,
+        "candidate": result.candidate,
+        "metric": bench.metric,
+        "credited": result.outcome == "improved",
+        "outcome": result.outcome,
+        "head_sha": candidate_sha,
+        "ruler": _ruler_provenance(ws, contract, bench, candidate_sha),
+    }
+    image_sha256 = measured_image(str(getattr(measurer, "image", "") or ""), result.eval_files)
+    if image_sha256:
+        measure_data["image_sha256"] = image_sha256
+    if result.outcome == "improved" and result.note:
+        # resume_attempt carries the cached-baseline note here on a credit
+        measure_data["baseline_note"] = result.note
     record_event(
         run_dir,
         "measure",
-        {
-            "baseline": result.baseline,
-            "candidate": result.candidate,
-            "metric": bench.metric,
-            "credited": result.outcome == "improved",
-            "outcome": result.outcome,
-            "head_sha": candidate_sha,
-        },
+        measure_data,
+        refs=file_refs(run_dir, *result.eval_files),
     )
     submitted_park = bool(stage.get("submitted"))
     if stage.get("panel_skip"):
@@ -5316,6 +5336,10 @@ def acknowledge(seq: int) -> None:
 
         parked: RunParked | None = None
         kept_ref = ""  # the ONE candidate snapshot ref that must outlive a park
+
+        def ruler_sha(sha: str) -> str:
+            return _ruler_provenance(ws, contract, _bench, sha)
+
         try:
             # the last-known score orients the brief only; the gate re-measures
             # both sides after the session, so None (a first run) is fine.
@@ -5365,6 +5389,7 @@ def acknowledge(seq: int) -> None:
                     else None
                 ),
                 tree_of=lambda sha: ws.git("rev-parse", f"{sha}^{{tree}}").strip(),
+                ruler_sha=ruler_sha,
             )
         except EndpointUnavailable as exc:
             # Reuse a jobless capacity park; its first wake starts the author.

src/outerloop/dispatch.py

@@ -424,7 +424,15 @@ def write_eval_job(
         task_dir.mkdir(parents=True, exist_ok=True)
         # a resubmitted eval must never be read as its predecessor: every prior
         # artifact — including a leftover extracted tree — goes before submission
-        for stale in ("exit-code", "stdout", "stderr", "setup.log", "submitted", "artifacts.log"):
+        for stale in (
+            "exit-code",
+            "stdout",
+            "stderr",
+            "setup.log",
+            "submitted",
+            "artifacts.log",
+            "image-sha256",
+        ):
             (task_dir / stale).unlink(missing_ok=True)
         for stale_node in task_dir.glob("node-[0-9]*"):  # a multi-node eval's per-node results
             stale_node.unlink(missing_ok=True)

src/outerloop/ledger_events.py

@@ -50,6 +50,25 @@ def display_leader(github: Forge, target: str) -> dict[str, LeaderEntry]:
         return {}
 
 
+def ruler_identity(
+    ws: Workspace, contract: Contract, bench: Benchmark, measured_sha: str
+) -> tuple[str, str]:
+    """The ledger's ruler digest and measurement signature for a measured tree.
+
+    Ruler identity includes harness contents, not just its command. Solver
+    edits do not change it; stewardship edits do, even with the same contract.
+    """
+    signature = json.dumps(bench.measurement_signature(), separators=(",", ":"))
+    ruler_files = []
+    for entry in ws.git("ls-tree", "-rz", measured_sha).split("\0"):
+        if entry:
+            _, path = entry.split("\t", 1)
+            if not steward_out_of_scope([path], contract):
+                ruler_files.append(entry)
+    ruler = hashlib.sha256((signature + "\0".join(sorted(ruler_files))).encode()).hexdigest()
+    return ruler, signature
+
+
 def measurement_pending(
     ws: Workspace,
     contract: Contract,
@@ -66,16 +85,7 @@ def measurement_pending(
     kind: str,
     provenance: str = "measured",
 ) -> PendingSubmission:
-    signature = json.dumps(bench.measurement_signature(), separators=(",", ":"))
-    # Ruler identity includes harness contents, not just its command. Solver
-    # edits do not change it; stewardship edits do, even with the same contract.
-    ruler_files = []
-    for entry in ws.git("ls-tree", "-rz", measured_sha).split("\0"):
-        if entry:
-            _, path = entry.split("\t", 1)
-            if not steward_out_of_scope([path], contract):
-                ruler_files.append(entry)
-    ruler = hashlib.sha256((signature + "\0".join(sorted(ruler_files))).encode()).hexdigest()
+    ruler, signature = ruler_identity(ws, contract, bench, measured_sha)
     return PendingSubmission(
         bench.name,
         bench.metric,

src/outerloop/measure.py

@@ -17,9 +17,11 @@
 
 from __future__ import annotations
 
+import contextlib
 import hashlib
 import json
 import logging
+import os
 import time
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -37,6 +39,37 @@
 
 log = logging.getLogger(__name__)
 
+# A container image is large and identical across measures in one process;
+# (path, size, mtime) says when a re-hash is actually needed.
+_image_digests: dict[tuple[str, int, int], str] = {}
+
+
+def image_digest(image: str) -> str:
+    """SHA-256 of the container image file, or "" when none is used/readable."""
+    if not image:
+        return ""
+    try:
+        stat = os.stat(image)
+        key = (image, stat.st_size, stat.st_mtime_ns)
+        if key not in _image_digests:
+            with open(image, "rb") as stream:
+                _image_digests[key] = hashlib.file_digest(stream, "sha256").hexdigest()
+        return _image_digests[key]
+    except OSError:
+        return ""
+
+
+def measured_image(image: str, eval_files: tuple[Path, ...]) -> str:
+    """The image digest(s) the measures ran under: those recorded at dispatch,
+    comma-joined when baseline and candidate differ, else the image's digest now."""
+    recorded = set()
+    for path in eval_files:
+        if path.name == "image-sha256":
+            with contextlib.suppress(OSError):
+                recorded.add(path.read_text().strip())
+    recorded.discard("")
+    return ",".join(sorted(recorded)) if recorded else image_digest(image)
+
 
 class MeasurementPending(Exception):
     """Raised when one or more measures have not completed. Carries the wake
@@ -354,6 +387,22 @@ def _slot(self, m: Measure) -> str:
     def _ev(self, m: Measure) -> Path:
         return self.run_dir / f"eval-{self._slot(m)}"
 
+    def result_files(self, measures: list[Measure]) -> tuple[Path, ...]:
+        """The eval result files the kernel reads, when they exist.
+
+        `read_eval_result` reads exit-code and stdout for the metric, and
+        stderr to explain a failure; provenance refs name the same files, plus
+        the image digest recorded at dispatch.
+        """
+        files: list[Path] = []
+        for m in measures:
+            ev = self._ev(m)
+            for name in ("exit-code", "stdout", "stderr", "image-sha256"):
+                path = ev / name
+                if path.exists():
+                    files.append(path)
+        return tuple(files)
+
     def _job_name(self, m: Measure) -> str:
         # A LIVENESS HINT, not a durable key: the cluster is asked "is this
         # measure's job live" by name. Slurm caps name length, so this hash is
@@ -505,6 +554,11 @@ def _dispatch(self, m: Measure) -> str:
                 raise MeasurementPending(
                     (), capacity_wait=True, note="retry state could not be saved; parked uncharged"
                 ) from None
+        # recorded before submission: the image file may be replaced while the
+        # run parks, and a crash after submit must not lose the digest
+        digest = image_digest(self.image)
+        if digest:
+            (self._ev(m) / "image-sha256").write_text(digest)
         try:
             job_id = submit_batch(
                 state_root(self.run_dir),