Repository navigation
The climb board: data, table, and chart of every attempt, kernel-published - #193
Conversation
…ished CLIMB.md (numbers + attempts table, rendered by GitHub), climb/<benchmark>.json (the data contract), and climb.html (a chart.js page over the JSON) land on the target's research-log branch as runs end — published by a tick service like the ledger, not a GitHub Action: Pages is public on a private repo's plan, an Action would rebuild kernel state, and Actions minutes stay spent on CI. Rows merge by run id and files are written only on change, so the service is idempotent and history survives a state-root cleanup. At the public flip a Pages site serves the same files unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Round 1 — reviewed head 0d38f9c8 — reviewer summarizer:hermes/gpt-5.6-terra over coverage+credentials+deployment+general+lifecycle+prose.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 5 blocking, 0 advisory.
5 findings attached to the lines below.
Verdict: block merge. The board can erase historical benchmarks from shared rendered views, rank max-direction metrics backward, leave views unrecoverable after a partial view upload, fail under the documented direct-from-clone workflow, and publish links to JSON that failed to upload. Rejected findings: none; the lower blocking assessments for the failed-JSON and historical-view findings were merged into the corroborated blocking defects rather than treated as separate claims.
…t data The board keeps its own memory (climb/index.json: benchmark -> direction), so a benchmark whose local records were cleaned up keeps its place in the views. Both views honor direction: max. Every file is compared against the branch each pass, which is also the retry for a failed upload. A benchmark whose fresh JSON could not be uploaded renders from its last published rows, never data that is not on the branch. The chart data is embedded in climb.html — a browser blocks fetch() on a file:// page, so the direct-from-clone view needs it inline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 1 — reviewed head 7fdd0cb5 — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 2 blocking, 0 advisory.
2 findings attached to the lines below.
The board uses the wrong direction for max benchmarks and embeds report text as executable HTML script data.
… escape its script The tick passes the contract's benchmark directions (it had loaded the contract lines earlier and passed nothing). "<" is escaped inside the embedded JSON, so an agent-written hypothesis carrying </script> stays data instead of running in the published page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 2 — reviewed head 9aed2bf1 — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: nothing blocking — 1 advisory note.
1 finding attached to the lines below.
A transient board-data read can erase an indexed benchmark from the published views.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 3 — reviewed head 489ddaae — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 1 blocking, 0 advisory.
1 finding attached to the lines below.
A transient failure while reading the board index can overwrite it and remove published benchmark history.
The index is read with the raising client call so a 404 (no board yet) and an outage are distinct: the first starts fresh, the second skips the index write for the pass — a transient failure must not shrink the board's memory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 4 — reviewed head a3211fad — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: nothing blocking — 1 advisory note.
1 finding attached to the lines below.
The per-benchmark row cap drops older published attempts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 5 — reviewed head 191eb014 — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 2 blocking, 0 advisory.
2 findings attached to the lines below.
Two data-integrity defects found in climb-board publishing.
…rwritten A run in review is not published until its PR's fate is known (a merged run lands with its final ending; the views count merged as a win). The per-benchmark JSON is read with the raising client call, like the index: a 404 starts fresh, an outage sits the pass out — published history is never replaced by a blind merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 6 — reviewed head 0b3ac590 — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 3 blocking, 0 advisory.
3 findings attached to the lines below.
The climb board can discard data for an allowed benchmark name, after failed reads, and for GPU-hour totals.
…es terminal transitions Data files live under climb/data/ (benchmark names are contract- controlled and 'index' is legal), and _clear_stage keeps gpu_hours_used when it wipes the waiting bookkeeping, so the board reads real spend after the transition instead of zero. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stat chips, a card with a faint grid, and light/dark from the viewer's system theme — the page people will actually watch deserves more than default chart.js on white. Content and behavior unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 8 — reviewed head b7f9b83c — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 1 blocking, 0 advisory.
1 finding attached to the lines below.
The board cap can remove pre-existing attempts whose reports were never archived.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 9 — reviewed head 0514c118 — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 1 blocking, 0 advisory.
1 finding attached to the lines below.
The full test suite passes (894 tests), but a transient board-data read can erase published history.
…ever overwritten Commit 0b3ac59 said this and did not do it: the patch script died on an anchor mismatch before writing the module, the follow-up applied only the cosmetic remainder, and the commit landed under the original message unverified. This commit carries the real changes — collect publishes only ENDED runs (a merged run lands with its final ending), and the per-benchmark JSON is read with the raising client call so an outage sits the pass out instead of blindly overwriting history — plus the tests that pin both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A scatter dataset ignores category labels (it wants x/y pairs on a linear axis), so the mixed chart rendered nothing. All three datasets now share the category axis; candidates are a point-only line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 10 — reviewed head 66f15420 — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: nothing blocking — 1 advisory note.
1 finding attached to the lines below.
One advisory data-retention issue found. Verified with the full test suite: 895 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Round 12 — reviewed head 636efb15 — reviewer hermes/gpt-5.6-terra.
terra
Advisory findings from autoresearch — the code owner decides. Reply to disagree; the autoresearch:no-review label opts this PR out.
Verdict: 1 blocking, 1 advisory.
2 findings attached to the lines below.
Two climb-board correctness defects were found. Full test suite passed: 895 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Summary
The climb finally gets a face. When runs end, the tick publishes to the target's
research-logbranch:climb/<benchmark>.json— one row per terminal attempt (agent, date, hypothesis one-liner, outcome, baseline/candidate, GPU-hours, PR link), merged by run id — published history is never rewritten, and it survives a future state-root reaper.CLIMB.md— headline numbers + the attempts table, newest first (plain markdown; GitHub renders it).climb.html— a self-contained chart.js page over the JSON (candidate scatter colored by outcome, best-so-far step line, baseline dashes, hypothesis tooltips). Open from a clone today; a Pages site serves it unchanged at the public flip.Why not a GitHub Action + Pages now: Pages access control is Enterprise-only (a private repo's dashboard would be public), an Action would reconstruct state the kernel already has, and Actions minutes stay reserved for CI.
Idempotent by construction (merge by run id, write only on change), advisory like the ledger (a failure never stops the tick).
Test plan
🤖 Generated with Claude Code