Skip to content

Trajectory step 2: measurement provenance; pending ledger reader tolerates new fields - #586

Merged
renmengye merged 4 commits into
mainfrom
feat/trajectory-provenance
Oct 25, 2026
Merged

renmengye merged 4 commits into
mainfrom
feat/trajectory-provenance

Conversation

@renmengye

Copy link
Copy Markdown
Member

Trajectory step 2: measure events say what produced the number.

  • Ruler: ledger_events.ruler_identity() now computes the ruler digest that pending submissions already carry: the measurement signature plus the non-solver tree entries. measurement_pending and every measure event call it, so the event and the ledger row hold the same value. It's best-effort: a git failure logs and records ""; it never fails a wake.
  • Image: image_sha256 is the SHA-256 of the configured container image file, cached in-process by (path, size, mtime) so a tick hashes it at most once.
  • Result files: refs hold {path, sha256} for the evaluation's exit-code, stdout and stderr under the run directory, the files the kernel reads to get the metric. Contents are never inlined.
  • Self-reported measures record only the ruler.

Ledger link deferred, with forward compatibility added now. The plan also puts the trajectory chain head in the pending ledger record. But parse_pending built the record with PendingSubmission(**raw), so any unknown key made an older kernel fail its whole ledger read. This PR makes parse_pending ignore unknown fields, and adds a test that a newer record still parses. The chain-head field can then land in a later release without breaking this one. Pending records written by this kernel are unchanged.

Compatibility: no persisted format changes. Older kernels ignore trajectory.jsonl. This kernel reads every pending record an older kernel writes, and will tolerate fields a newer one adds.

Not verified live: a dispatched run with a real image (unit tests use a small temp file).

Gate: ruff check, ruff format --check, mypy, pytest (3143 passed, 7 skipped).

Built by DeepSeek-V4.1-Flash, with the forward-compatibility change by Claude; reviewed by Claude.

@renmengye renmengye added the outerloop:review re-request the advisory review label Oct 24, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 — reviewed head b619c331 — reviewer summarizer:hermes/gpt-5.6-terra over coverage+credentials+deployment+general+lifecycle+prose.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 2 blocking, 1 advisory.

2 findings attached to the lines below.

Completed measurements lose their result-file refs. [lifecycle] The final AttemptResult omits outcome.eval_files, so the measure event at lines 2516-2520 receives an empty tuple after a non-parked successful measurement and cannot record the files it read. (src/outerloop/orchestrator.py:2742; high confidence)

Verdict: changes requested. Blocking: the image digest is captured too late for parked dispatched runs, and synchronous completed measurements discard result-file provenance refs. Non-blocking: the provenance test stubs out the production collector. Rejected: none; no duplicated, mistaken, or prose findings were supplied.

Comment thread src/outerloop/orchestrator.py Resolved
Comment thread tests/test_orchestrator.py Resolved

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 (re-run on the same head) — reviewed head b619c331 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

@renmengye renmengye added outerloop:review re-request the advisory review and removed outerloop:review re-request the advisory review labels Oct 24, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 3 (re-run on the same head) — reviewed head b619c331 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: no defects found.

@renmengye

Copy link
Copy Markdown
Member Author

Summary finding (credited result drops eval_files): fixed in 7a9767f. attempt_once's credited AttemptResult now carries outcome.eval_files. The measure event inside attempt_once already had the files; only the returned result lacked them.

@renmengye renmengye added outerloop:review re-request the advisory review and removed outerloop:review re-request the advisory review labels Oct 24, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 4 — reviewed head 7a9767f0 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: 2 blocking, 0 advisory.

2 findings attached to the lines below.

Image provenance can be wrong after a dispatch crash or when a cached baseline used an earlier image.

Comment thread src/outerloop/measure.py Resolved
Comment thread src/outerloop/orchestrator.py Resolved
@renmengye renmengye added outerloop:review re-request the advisory review and removed outerloop:review re-request the advisory review labels Oct 25, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 5 — reviewed head 3ba194d9 — reviewer hermes/gpt-5.6-terra.

terra
Advisory findings from outerloop — the code owner decides. Reply to disagree; the outerloop:no-review label opts this PR out.

Verdict: nothing blocking — 1 advisory note.

1 finding attached to the lines below.

The ruler provenance digest changes for solver-only edits.

Comment thread src/outerloop/ledger_events.py Resolved
@renmengye
renmengye merged commit cfc8af4 into main Oct 25, 2026
5 checks passed
@renmengye
renmengye deleted the feat/trajectory-provenance branch October 25, 2026 01:52
@renmengye renmengye mentioned this pull request Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

outerloop:review re-request the advisory review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant