Per-run trajectory: a kernel-written, hash-chained event timeline

CHANGELOG.md

@@ -6,6 +6,7 @@ Versions follow [SemVer](https://semver.org).
 
 ## [Unreleased]
 
+- Kernel events now append to a per-run `trajectory.jsonl` with sequence numbers and a SHA-256 chain. `outerloop trajectory <run-id> [--verify] [--json]` reads it without changing run state. Operational files and their formats are unchanged; older kernels ignore the new file. Upgrading: runs started before this kernel have no trajectory and read as "no trajectory" until their next recorded event. No backfill is attempted.
 - Add a printable agent-text inventory and protocol wording lint, with clearer refusal recovery guidance (#487, parts 1–2).
 - `BENCHMARKS.md` shows `—` instead of "provenance unknown" in the main-commit column of a baseline row that nothing has improved on yet, as on a freshly seeded repository.
 - The README and the getting-started guide link to the quickstart repository. It runs two CPU benchmarks with the local forge and local compute, and needs no GitHub account or cluster (#563). The docs and the startup warning no longer say an uncontained session runs in a sandbox: it gets a scrubbed environment and its own home, but can read any file the account can.

docs/design/trajectory.md

@@ -0,0 +1,41 @@
+# Run trajectory
+
+The kernel alone writes `runs/<run-id>/trajectory.jsonl`, outside the session's
+workspace and writable sandbox mounts. Operational state, launch ledgers,
+inbox/outbox and session records continue unchanged. The trajectory is a
+best-effort timeline, not a replacement for those recovery records.
+
+Each line is a JSON object with sorted keys: `seq` (starting at zero), `ts`
+(Unix seconds), `type`, `actor`, `data`, `refs`, and `prev`. `prev` is SHA-256
+of the previous event's exact bytes excluding its newline; the first event
+uses SHA-256 of the UTF-8 run id. Actors use `kernel`, `author:<agent-id>`,
+`judge:<name>`, or `human:<login>`; these kernel actions use `kernel`.
+Refs contain run-relative `path` and `sha256` fields. Native session logs are
+hashed once at session end, and the report is hashed at ending.
+
+Events cover session start/end, syscall acceptance/refusal, launch submission
+and results, gate results, acknowledged inbox delivery, wakes, PR opening
+(including reconciliation of an already opened PR), and endings. Inbox events
+contain sequence ranges, kinds and origins, never message bodies. Session start
+may have an empty id until the backend returns one. Exceptions produce a session
+end outcome when the host can catch them; process death can leave an unmatched
+start. Measurement credit describes the gate decision, not publication or merge.
+
+A sibling `trajectory.lock` serializes appenders with exclusive `flock`. A torn
+last line is preserved and separated by a newline; the chain continues from
+the last parseable event. Verification reports malformed lines even after
+recovery. A complete JSON event missing only its newline is retained and its
+newline repaired on the next append. No old bytes are rewritten. Writes stop
+at 4 MB and warn once per file per kernel process; I/O failures warn without
+failing a tick. Events are not transactional with the operational files, so a
+crash or write failure can leave missing events or retries.
+
+`outerloop trajectory <run-id> --verify` checks sequence gaps and byte hashes;
+`--json` emits events (and verification problems when requested). Missing legacy
+files read as an empty history and verification says "no trajectory". No history
+is reconstructed from older operational files. Older kernels ignore this file.
+The chain detects internal edits and missing lines, but without a trusted
+external anchor cannot detect a rewritten chain or removal of its tail.
+Verification checks the event chain, not the contents of referenced artifacts.
+
+Tool calls, full measurement provenance and attestation are later steps.

docs/reference/cli.md

@@ -280,3 +280,12 @@ outerloop migrate-ledger --target TARGET --main-sha MAIN_SHA [--force] [--dry-ru
 | `--main-sha` | required | the full current default-branch commit SHA |
 | `--force` | off | replace an existing branch ledger |
 | `--dry-run` | off | print the table without writing |
+
+### `trajectory <run-id> [--verify] [--json] [--root PATH]`
+
+Read the kernel's `runs/<run-id>/trajectory.jsonl` timeline. Text output shows
+sequence, time, type, actor and short event data. `--verify` checks the hash chain
+and sequence gaps; damaged chains exit 1. Missing legacy files show "no trajectory"
+and exit 0. JSON output is an event array, or an object with `events` and `problems`
+when verification is requested. This command is read-only.
+See [Run trajectory](../design/trajectory.md) for the format and limits.

docs/storage.md

@@ -185,7 +185,7 @@ The `.outerloop/` directory inside the workspace is the kernel's channel. It hol
 
 ## Run state and leases
 
-A run's record is one JSON file, `runs/<run-id>/state.json`, written by atomic rename. A crash mid-write leaves the previous record intact, and the sweep reasons only from these files plus Slurm, never from process memory. The same run directory holds the wake lease, the inbox, the outbox of replies kept before posting, and the redacted transcript of every session. Beside the run directories, a baseline cache holds the measured baseline that a `cached` contract shares among all attempts on one base commit.
+A run's record is one JSON file, `runs/<run-id>/state.json`, written by atomic rename. A crash mid-write leaves the previous record intact, and the sweep reasons only from these files plus Slurm, never from process memory. The same run directory holds the wake lease, the inbox, the outbox of replies kept before posting, and the redacted transcript of every session. The kernel also appends `trajectory.jsonl`, a [hash-chained event timeline](design/trajectory.md), under a sibling `trajectory.lock`. Beside the run directories, a baseline cache holds the measured baseline that a `cached` contract shares among all attempts on one base commit.
 
 Unknown fields are ignored on read, and new formats for run records are migrated as the kernel reads them, so an upgrade needs no migration step. That compatibility runs forward only: an older kernel cannot interpret newer record formats, so a rollback past a format change needs its runs drained or their records translated by hand. After a run ends, its workspace and run home are removed after a grace period of 24 hours by default; the record, the report, the transcripts and the ledger entries stay.
 

src/outerloop/attempt.py

@@ -147,6 +147,8 @@
 from outerloop.syscall import install_tool as syscall_install_tool
 from outerloop.syscall import refresh_tool as syscall_refresh_tool
 from outerloop.syscall import write_siblings as syscall_write_siblings
+from outerloop.trajectory import file_refs
+from outerloop.trajectory import record as record_event
 from outerloop.verifier import MAX_CLAIM_CHARS
 
 REPLY_MARKER = marker("followup")
@@ -1162,6 +1164,11 @@ def persist() -> None:
         persist()
 
         def refuse(index: int, reason: str) -> None:
+            record_event(
+                run_dir,
+                "syscall.decision",
+                {"verb": "message", "outcome": "refused", "reason": reason},
+            )
             entry["recipient"] = record.run_id
             entry["message"] = asdict(
                 Message(
@@ -1275,6 +1282,12 @@ def refuse(index: int, reason: str) -> None:
             ids=[f"message-{e['counter']:020d}" for e in public],
             references=[str(e.get("in_reply_to", "")) for e in public],
         )
+        for _entry in public:
+            record_event(
+                run_dir,
+                "syscall.decision",
+                {"verb": "message", "outcome": "accepted", "reason": ""},
+            )
         for index, entry in enumerate(journal, 1):
             if entry.get("delivered"):
                 continue
@@ -1338,6 +1351,12 @@ def refuse(index: int, reason: str) -> None:
                         append(run_dir, message)
                     else:
                         append(recipient_dir, message)
+                        if message.kind == "agent-message":
+                            record_event(
+                                run_dir,
+                                "syscall.decision",
+                                {"verb": "message", "outcome": "accepted", "reason": ""},
+                            )
                         if recipient.run_id != record.run_id:
                             append(
                                 run_dir,
@@ -3312,6 +3331,18 @@ def _resume_run(
     # more experiments, or concludes — instead of being decided by policy here.
     # Falls back to the plain-finish behavior (negative terminal / draft PR)
     # when the session cannot be resumed.
+    record_event(
+        run_dir,
+        "measure",
+        {
+            "baseline": result.baseline,
+            "candidate": result.candidate,
+            "metric": bench.metric,
+            "credited": result.outcome == "improved",
+            "outcome": result.outcome,
+            "head_sha": candidate_sha,
+        },
+    )
     submitted_park = bool(stage.get("submitted"))
     if stage.get("panel_skip"):
         panel_lenses = ()
@@ -4221,6 +4252,11 @@ def publish(
     def refuse(
         text: str, head: str = "", *, moved: bool = False, quoted_text: str = ""
     ) -> AttemptOutcome:
+        record_event(
+            run_dir,
+            "syscall.decision",
+            {"verb": "submit", "outcome": "refused", "reason": redact(text, secrets)},
+        )
         append(
             run_dir,
             Message(
@@ -4637,6 +4673,7 @@ def refuse(
                 # read: visible, plainly not merge-ready
                 draft=draft,
             )
+        record_event(run_dir, "pr.open", {"url": pr_url, "head_sha": result.candidate_sha})
         # Arm auto-merge, best-effort, and ONLY when branch protection
         # requires a human review — the guard keeps bot-never-merges
         # enforced in code, not in per-repo config. Never arm a draft,
@@ -6266,6 +6303,12 @@ def finish_run(
             final, stage={**final.stage, "launch_afterany": "afterany:" + ":".join(sorted(jobs))}
         )
     save_record(run_root, final, now)
+    record_event(
+        directory,
+        "ending",
+        {"state": ending, "reason": note},
+        refs=file_refs(directory, report),
+    )
     ref = str(record.stage.get("candidate_ref") or "")
     if snapshot_attempted and not ref:
         ws = None

src/outerloop/cli.py

@@ -1245,6 +1245,11 @@ def main(argv: list[str] | None = None) -> int:
     p.add_argument("--file", type=Path, help="read the message text from this path")
     p = sub.add_parser("limits", help="show live operator ceilings and fleet GPU usage")
     p.add_argument("--root", help="state root (defaults to OUTERLOOP_ROOT or ~/.outerloop)")
+    p = sub.add_parser("trajectory", help="show a run's kernel event chain (read-only)")
+    p.add_argument("run_id")
+    p.add_argument("--root", help="state root (defaults to OUTERLOOP_ROOT or ~/.outerloop)")
+    p.add_argument("--verify", action="store_true")
+    p.add_argument("--json", action="store_true")
     p = sub.add_parser("status", help="show local runs and endpoint outages (read-only)")
     p.add_argument("--root", help="state root (defaults to OUTERLOOP_ROOT or ~/.outerloop)")
     p.add_argument("--json", action="store_true", help="print structured JSON")
@@ -1312,6 +1317,44 @@ def main(argv: list[str] | None = None) -> int:
         return 0
     if args.command == "pr":
         return pr(args)
+    if args.command == "trajectory":
+        from outerloop.runstate import run_dir
+        from outerloop.trajectory import read, verify
+
+        values = env_file_values(keys=("OUTERLOOP_ROOT",)) if not args.root else {}
+        root = Path(
+            args.root
+            or os.environ.get("OUTERLOOP_ROOT")
+            or values.get("OUTERLOOP_ROOT")
+            or DEFAULT_LOCAL_ROOT
+        ).expanduser()
+        if Path(args.run_id).name != args.run_id or args.run_id in (".", ".."):
+            parser.error("invalid run id")
+        directory = run_dir(root, args.run_id)
+        try:
+            events = read(directory)
+        except OSError as exc:
+            print(f"cannot read trajectory: {exc}", file=sys.stderr)
+            return 1
+        problems = verify(directory, args.run_id) if args.verify else []
+        if args.json:
+            print(
+                json.dumps(
+                    {"events": events, "problems": problems} if args.verify else events,
+                    sort_keys=True,
+                )
+            )
+        else:
+            for event in events:
+                print(
+                    f"{event['seq']} {event.get('ts')} {event.get('type')} "
+                    f"{event.get('actor')} {json.dumps(event.get('data'), sort_keys=True)[:160]}"
+                )
+            if not events:
+                print("no trajectory")
+            if args.verify:
+                print("; ".join(problems) if problems else "chain intact")
+        return 1 if problems and problems != ["no trajectory"] else 0
     if args.command == "status":
         from outerloop.status import collect_status, render_text
 

src/outerloop/launchlog.py

@@ -12,6 +12,7 @@
 
 from outerloop.provenance import producing_author
 from outerloop.syscall import Launch, LaunchResult, launch_jobs
+from outerloop.trajectory import record as record_event
 
 LEDGER = "launches.jsonl"
 # generous: a run is depth_k launches x sleep_k sleeps x the array width, far below this
@@ -37,6 +38,29 @@ def _append(run_dir: Path, rows: list[dict[str, Any]]) -> None:
             fh.write("\n")
         for row in rows:
             fh.write(json.dumps(row, sort_keys=True) + "\n")
+            record_event(
+                run_dir,
+                "launch.submit" if row["event"] == "submitted" else "launch.result",
+                {
+                    key: row[key]
+                    for key in (
+                        "event",
+                        "sleep",
+                        "name",
+                        "job_ids",
+                        "commit",
+                        "author",
+                        "minutes",
+                        "array",
+                        "concurrency",
+                        "state",
+                        "exit_code",
+                        "why",
+                    )
+                    if key in row
+                },
+                at=row["at"],
+            )
 
 
 def append_submitted(