A durable execution runtime for long-horizon agent workflows — checkpointed, forkable, crash-recoverable — verified by deterministic simulation testing.
Every badge number is reproducible by a command in this README — that's a house rule. Mock-bench scores follow a hardcoded fixture curve and are never presented as measurements. Sections marked planned are not built yet and are labeled so.
LangGraph gives a single thread checkpoint persistence and resume. This project adds what it doesn't have: branch lifecycle as durable state, lease-based claiming with fencing tokens across processes, crash reconciliation on boot, and fork-from-arbitrary-checkpoint with state mutation — and proves those mechanics with a seeded, fault-injecting simulator instead of hope.
The reference workload is a self-improving coding-agent harness search (after the Stanford Meta-Harness paper, arXiv:2603.28052). The workload has to run; it is not the claim.
flowchart TD
subgraph L1 [" refinement loop · planned "]
R1[detect recurrence] --> R2[attribute component + step] --> R3[gate on evidence]
end
subgraph L2 [" continual harness · Prime Agent "]
H1[memories] --- H2[skills] --- H3[prompts] --- H4[subagent specs]
end
subgraph L3 [" durable runtime · this repo "]
D1[checkpoints] --- D2[branches + fences] --- D3[crash recovery] --- D4[replay]
end
L1 --> L2 --> L3
LangGraph expresses the agent loop as a state machine and checkpoints it, so a run is a sequence of addressable points rather than an opaque process. Its scope is one thread: pause, resume, one child.
Prime Agent supplies
what rides inside that state — the Continual Harness, which keeps supplemental
prompts, memories, skill descriptions, and reusable subagent specifications as
durable state, refined through small evidence-backed updates. Its /refine
reviews the current trajectory and is local to the session by default.
Meta-Harness wraps both. It turns a checkpoint from something you can resume into something you can fork: many children from one point, each under its own lease and fencing token, safe to run concurrently and compare.
The seams are clean because each operates at a different scope — context is per-call, LangGraph state is per-thread, the continual harness is per-project, and branch lifecycle is what lets you version the last of those.
Every published harness-evolution system evaluates a candidate by rerunning tasks from scratch. That makes evaluation cost the binding constraint, and it is why refinement cycles are expensive and promotion thresholds get set at effect sizes too small to detect.
A checkpoint you can fork changes the arithmetic. If you know where a trajectory went wrong — not just that it did — you fork at that step and run candidates forward from identical state:
trajectory T
step 0 ──────────────────► step 39 ──┬──► H1 tail
├──► H2 tail
└──► H3 tail
Everything before step 39 — model sampling, tool ordering, environment noise, task difficulty — is shared exactly and cancels in the comparison. Only the tail executes. The three branches parallelize by construction, each with its own fence (I5), and a stalled one cannot corrupt a sibling (I1).
That is what the invariants below are for. The runtime is the part that has to be correct before any of it means anything.
sequenceDiagram
autonumber
participant C as client (MCP / dashboard)
participant S as StateStore (Postgres)
participant A as worker A
participant B as worker B
C->>S: fork_from_checkpoint(mods)
A->>S: claim branch → fence = 1
loop iterate
A->>S: fenced record + checkpoint
end
Note over A: 💀 kill -9
Note over S: lease TTL expires
B->>S: claim branch → fence = 2
A--xS: any late write REJECTED (stale fence 1)
B->>S: resume from last checkpoint
C->>S: get_branch_status
S-->>C: running · lease_generation = 2
B->>S: finish(completed) with fence 2
This exact sequence is verified three independent ways:
| Proof | Command |
|---|---|
| Real processes, real SIGKILL | cd backend && uv run pytest tests/test_worker_recovery.py -q |
| Driven end-to-end by an outside MCP client | cd backend && uv run pytest tests/test_mcp_acceptance.py -q |
Watchable live — the dashboard's chaos tab has per-worker kill -9 buttons and a fence badge that ticks gen N → N+1 |
META_HARNESS_CHAOS=1 uv run uvicorn app.main:app + two uv run meta-harness worker |
The spec is docs/INVARIANTS.md; tests are named after
the invariant they cover, and the simulator asserts all seven after every
seeded run.
| ID | Invariant |
|---|---|
| I1 | No double execution — no iteration lands twice across any crash/resume sequence |
| I2 | No orphans — after boot reconciliation, no branch stays running without a live lease |
| I3 | Resume convergence — crash + resume reaches the same final state as an uninterrupted run |
| I4 | Fork isolation — a branch's writes never mutate parent thread state |
| I5 | Lease safety — at most one worker executes a branch (fencing tokens, not just TTLs) |
| I6 | Durable cancel — a cancelled branch never resumes, not even after restart |
| I7 | Event integrity — per-run SSE sequence numbers are monotonic with no gaps |
I4 and I5 are the two that candidate evaluation rests on: sibling branches must not see each other's writes, and two workers must never execute the same branch.
Branch lifecycle — every transition outside this diagram is a bug:
stateDiagram-v2
[*] --> created : fork_from_checkpoint
created --> running : claim (fence++)
created --> cancelled
running --> running : reclaim after lease expiry (fence++)
running --> created : boot reconciliation requeue
running --> completed
running --> failed
running --> cancelled : durable cancel (fence++)
completed --> [*]
failed --> [*]
cancelled --> [*]
A seeded scheduler drives the orchestrator against the real in-memory
StateStore under a virtual clock, injecting crashes, stalls past lease
expiry, cancels, clock skew, and duplicate/reordered NOTIFY delivery — then
asserts I1–I7. A failure prints its seed; the seed replays the exact
interleaving.
cd backend && uv run python -m sim.run --seeds 10000 # → 10000 seeds run, 0 failedIt found two real bugs (both documented with their seeds in
docs/INVARIANTS.md):
- DST-1, seed 7 — the historical unfenced check-then-append protocol
double-appends when a stalled worker wakes past a reclaimed lease.
Fix shipped: an atomic, fence-guarded
record_iterationat the data layer. - DST-2, seed 9270 — a zombie worker lands one stale trailing checkpoint after the rightful owner finishes (LangGraph checkpoint writes are unfenced). Documented benign: terminal branches never resume, requeues re-execute idempotently.
Every DST run is a pure function of its seed, so a trace file is the bug
report. The /replay page (fully static, no backend) scrubs the timeline:
amber ticks are injected faults, red ticks are invariant violations, and the
violation is highlighted at the exact step it surfaces.
Seed 7 mid-run — worker w2 owns branch b1 at fence 1, lease already expired, the reclaim is coming:
…and the end of the same timeline, where the unfenced protocol pays for it:
cd backend && uv run python -m sim.export --seed 4471 --mode fenced_store -o trace.json
# load trace.json in /replay — the export is byte-identical every timeflowchart LR
subgraph clients [" clients "]
MC["Claude Code /<br/>any MCP client"]
UI["Next.js dashboard<br/>(chaos tab · trajectory tree · /replay)"]
end
MCPA["MCP adapter<br/>6 tools · zero logic"]
API["FastAPI<br/>REST + SSE"]
PG[("Postgres<br/>branch_runs · leases + fences<br/>checkpoints · run_events")]
W1["worker 1"]
WN["worker N"]
MC -- stdio --> MCPA --> API
UI -- "REST + SSE" --> API
API --> PG
PG -. "LISTEN/NOTIFY (gap-free seq)" .-> API
W1 -- "claim · heartbeat · fenced writes" --> PG
WN -- "claim · heartbeat · fenced writes" --> PG
- Workers are separate processes from the API. They claim
branch_runsrows withFOR UPDATE SKIP LOCKED+ a lease TTL; every claim incrementslease_generation(the fencing token) and every write carries it — a stale fence is rejected by the data layer itself. - Events cross the process boundary durably: a per-run event log with
gap-free monotonic sequence numbers, fanned out via
LISTEN/NOTIFY; the SSEid:is the sequence number, soLast-Event-IDreconnects survive API restarts (I7). - The workload runs inside it: two LangGraph state machines (outer propose→validate→benchmark→update_frontier, inner orient→plan→act→verify→submit), 6 fixed tools, 11 override points, cross-run memory.
The run_events table is also the substrate the planned refinement layer reads
from: a gap-free ordered event stream per run means a failure pattern has a
step index, which is what makes a fork point addressable.
Six tools — start_run, fork_from_checkpoint, get_branch_status,
list_branches, cancel_branch, resume_run — as a thin stdio adapter over
the REST API. One implementation of the branch lifecycle, no parallel logic.
.mcp.json registers it for Claude Code, or:
claude mcp add meta-harness -- uv --directory backend run meta-harness-mcpThe acceptance scenario — a client with no knowledge of this repo starts a
run, forks a mid-point checkpoint, survives its worker being kill -9'd, and
reads the fence increment off its next poll — is automated:
cd backend && uv run pytest tests/test_mcp_acceptance.py -q # 1 passed, ~13s, 4 processesPrerequisites: Python 3.11+ with uv, Docker, Node 20+ (dashboard only). No API key needed for anything below.
git clone https://github.com/jstxw/Meta-Harness.git
cd Meta-Harness
cp .env.example .env
uv sync
docker compose -f infra/docker-compose.yml up -d postgres
# the suite (live-LLM test skips without ANTHROPIC_API_KEY)
cd backend && uv run pytest tests -q # → 128 passed, 1 skipped
# a full durable run with the deterministic mock workload
uv run meta-harness loop --proposer mock --mock-bench --budget 2 --fresh
# kill it mid-run, resume from the last Postgres checkpoint
uv run meta-harness resume <run-name>
cat ../runs/<run-name>/evolution_summary.jsonl | jq -r .iteration | sort | uniq -d # empty (I1)
# the chaos demo
META_HARNESS_CHAOS=1 uv run uvicorn app.main:app & # API
uv run meta-harness worker & # worker(s)
cd ../frontend/dashboard && npm install && npm run dev # dashboard → chaos tabTwo sandbox modes for workload trials, recorded in every eval-result.json:
| Mode | Boundary |
|---|---|
subprocess (default) |
fresh temp dir, rlimits — same trust boundary as the host, labeled honestly |
META_HARNESS_SANDBOX=docker |
one container per trial: --network none, 512MB, 1 CPU, workspace bind mount |
docker build -t meta-harness-sandbox -f infra/sandbox.Dockerfile infra
cd backend && uv run pytest tests/test_sandbox_docker.py -q # 4 passedWhy Docker and not wasmtime: the spike write-up is
docs/PHASE5_SANDBOX.md — WASI has no
subprocess/shell and the frozen tool contract includes run_bash.
Prime Agent's own docs note that its worker and kernel processes improve lifecycle isolation and recovery but are not a security sandbox, so candidate evaluation keeps this Docker boundary regardless of which agent runs inside it.
Being precise about this is a house rule, not modesty.
Proven. I1–I7, three independent ways — real processes under SIGKILL, an outside MCP client, and 10,000 seeds of deterministic simulation with zero failures. Two real bugs found, each with a replayable seed. These are deterministic claims: they hold, or a counterexample exists.
Not proven. That harness refinement improves coding performance. The reference workload is mock-benched, so there is currently no measured evidence on that question. It is a statistical claim, and answering it needs a measured noise floor, paired comparison, held-out tasks, and pre-registered thresholds.
The forking argument above is precisely what makes that question affordable to answer — but it is a hypothesis about evaluation cost until the ratio is measured.
Three pieces sit between this runtime and a working refinement loop. None are built:
| Piece | What it does |
|---|---|
| Recurrence detection | deterministic predicates over run_events — blind retry, redundant scan, unfocused test suite — counted across trajectories, each carrying the seq where it fired |
| Attribution | given N supporting trajectories, decide whether the cause is the harness or the environment, and name the component and step |
| Evidence gate | paired comparison of candidate against parent on held-out tasks, with the threshold derived from a measured noise floor rather than chosen |
The first of these is what Prime Agent's /refine does not cover: it reviews
the current trajectory, not four of them. Cross-trajectory counting is the
missing input, and this repo's event log is where it comes from.
| Doc | What it is |
|---|---|
documents/REPOSITIONING_PLAN.md |
The plan this repo executes — phases, non-goals, honesty rules |
docs/INVARIANTS.md |
The spec: I1–I7, state machine, fencing-token analysis, DST findings |
documents/MCP_SERVER_SPEC.md |
Phase 6 expanded: the MCP surface and acceptance scenario |
docs/PROJECT_STATUS.md |
Current verified state — every number with its reproduction command |
docs/PHASE5_SANDBOX.md |
The wasm spike and the Docker outcome |
KICKOFF.md |
Session kickoff prompt for agent-driven development |
Historical design docs (docs/BUILD_ORDER.md, docs/DEFINITION_OF_DONE.md,
docs/PROJECT_KNOWLEDGE_BASE.md, docs/TEAM_HANDOFF.md) predate the
repositioning and contain synthetic demo-arc numbers — do not quote them as
results.
- Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn — Meta-Harness: End-to-End Optimization of Model Harnesses, arXiv:2603.28052.
- Prime Intellect and collaborators for Prime Agent and the Continual Harness (arXiv:2605.09998), whose harness representation this runtime is designed to version.
- The LangChain team for LangGraph's checkpointing and time-travel primitives.
- Martin Kleppmann's fencing-token argument, which the simulator re-derived the hard way at seed 7.
MIT — see LICENSE.