[kikashy]

Closes #173. Design note: docs/design/store-interface.md.

What moved

1. The contract (internal/runner/store_contract.go). Reads, outside or inside a transaction: release(id), job(id), requestedRun(job, caller, key) (the request's digest, and the record decoded only when asked) and entrySequence(run). Reads outside a transaction only: run(id), oldestQueued(), entryLine(run) and chainPage(after, limit). Writes go through update(fn), one serializable transaction that keeps what fn wrote only if fn returns nil and the commit succeeds. Inside a transaction: activeRuns, createRun (once per id and once per request key), putRun(r, expected) (a versioned update: the run's state is its version), runningRuns (in queue order), lastEntry, createEntry (once per sequence and run; a sequence below 1 is refused), createRelease (once per id) and appendJournal. Faults are typed: absent, taken, conflict, durability, unreadable and refused. An error keeps its cause's text, and errors.Is still reaches the cause (sql.ErrNoRows included). There is also a kind registry: sqlite is the default, and the boot line names no kind, so no other kind can be selected.
2. The sqlite kind (internal/runner/store_sqlite.go). This is the code that kept this state, moved. The lock, the database file's create-once and private check, sql.Open with one connection, the pragmas and schema, the trigger and chain schemas, the identity check and the migration (migrateStore becomes (*sqliteStore).migrate) all run in the same order. The Runner's working directories run in between through afterLock, and the manual-admission check through opened. The boot line's directory reaches the store only through openSQLiteStore and sqlitePreparedStore.
3. Routed through the contract: release/job/run, the submission's transaction and its earlier mapping-v2 read, dispatchNext's read of the queue, changeRun, finishRun and appendRunEntry, restart (its occurrences, which are outside the contract, run in the same transaction through sqliteTxOf), recordStop, saveRelease, runEntry/runEntrySequence/chainPage, and the 404 mapping (absent).
4. The in-memory kind (internal/runner/store_memory_test.go, test only) and the conformance suite (internal/runner/store_conformance_test.go), run against both kinds.
5. CI: the suite runs in the existing unit-tests step under go test -race ./.... The step then reruns TestStoreKindsConform -v and fails unless both sqlite and memory passed (scripts/checks/unit-tests.sh, which jpdev.json already maps). There is no new job and no new directory to map.

Not behind the contract yet: triggers, occurrences, job creation and revision, briefs, workflow executions, the manual-admission journal, metadata, the journal's reads, and the listings. These stay on the sqlite kind's database handle. Manual store preparation, offline relocation and the source worker's store hold their SQLite files themselves. Section 6 of the design note lists each one, and every statement left on runs, releases and jobs outside the kind.

What is unchanged, and the evidence

• Statements. The SQL literals removed from store.go, events.go and run_chain.go are the same set as those added in store_sqlite.go: 23 distinct, none removed without being added and none added without being removed. A script compared them over git diff origin/main...HEAD. The multi-line pragma and schema block is byte-identical. The migration's body is identical apart from s.db. becoming k.db..
• Transactions. Each transaction is the one it was, with its statements in the same order: submission, claim, a run ending in the queue, finish (run, chain entry, journal entry), restart (start entry, interruptions, occurrences), release, stop. An identical retry still ends its transaction without a commit (errKeepNothing).
• Error text. Every error keeps its cause's text. Errors the Runner returns inside a transaction (apiError, errNoAuditBytes, errManualProtocol) come back unchanged, and TestACompletionWithoutTheRecordsBytesIsNotRecorded still compares errNoAuditBytes by identity.
• Existing tests. Every existing test passes, and only one file changed. run_chain_test.go passed a *sql.Tx to appendRunEntry, which is now a moved internal; its two call sites wrap it as &sqliteTx{...}, and its assertions are unchanged.

What differs

• changeRun (the dispatcher's claim, and a run's end in the queue: expired, or set aside) now requires the store to hold the run queued. Otherwise it writes nothing and returns conflict. This is the claim's fence: a run leaves queued once. The comparison sits between the two statements the claim already made. With one dispatcher per store it never refuses, and every existing test that calls changeRun passes. finishRun and restart write whatever the state is, as before. There is no lease, because a claim does not expire and the next process's lock ends a run left running.
• Close now closes the attempts directory before the store. It used to close it between the database and the lock. The lock is still released last.
• No public surface changes: routes, schemas, vectors and output are untouched.

Conformance cases (TestStoreKindsConform, both kinds)

A release is created once · a run is queued once under its id and once under its request key · the queue is served in the order runs were queued (ids sort against it) · a run read queued is claimed once, and an update expecting another state writes nothing · concurrent claimers (4 producers, 6 claimers, 40 runs) grant every run once and lose none, retrying on conflict with no lock in the test · a transaction that fails keeps nothing · a commit that fails is durability, never a success, and keeps nothing (the sqlite kind's commit seam, the memory kind's seam) · the chain gives a run one entry and is read in order, and a completion without its record's bytes is not recorded · chain entries are read in order of sequence whatever order they were made in · nothing held reads absent · a record that does not read is unreadable. sqlite only: a reopened store holds what it held (a claimed run still running, a completed run and its entry, the queue order, the release, the request, the journal) · one process holds the store.

Mutants

Each mutant ran on a scratch copy against go test -race -run '^TestStoreKindsConform$', and each copy was deleted after its run. A copy that failed to build counted as not run.

#	Safeguard broken	Result
m1	sqlite putRun's expected-state check dropped (version check)	caught: claimed once, concurrent claimers
m2	memory kind's own lock dropped from update (claim exclusivity)	caught: data races under -race; also the concurrent case without -race, 3 of 3 runs
m3	changeQueuedRun claims at any state	caught on both kinds
m4	memory createRun's create-once check dropped	caught: queued once
m5	sqlite's taken typing of PRIMARY KEY/UNIQUE dropped (create-once)	caught: release once, queued once, chain
m6	sqlite update ignores a failed commit (durability)	caught: commit that fails
m7	memory commit seam ignored (durability)	caught: commit that fails
m8	oldestQueued ordered by id (list order)	caught: queue order, reopened store
m9	runningRuns ordered by id (list order)	caught: queue order
m10	the lock taken shared instead of exclusive (flock LOCK_SH)	caught: one process holds the store
m11	memory chain insert unordered (list order)	first survived. The case "entries read in order of sequence whatever order they were made in" was added, and the mutant is now caught
m12	memory update works on the live state, not a copy (atomicity)	caught: fails keeps nothing, commit fails
m13	chainPage ordered by run (list order)	first survived. Caught by the same added case
m14	memory putRun's expected-state check broken	caught: claimed once, concurrent claimers
m15	sqlite one-connection limit dropped	caught: concurrent claimers
m16	sqlite absent typing dropped	caught: absent cases
m17	memory unreadable typing dropped	caught: unreadable

Self-checks (AGENTS.md)

(a) The boundary path internal/runner shows no admission or verdict change. The change is the store. The one added refusal is changeRun's conflict, a storage error that stops the dispatcher like any other, and it never fires with one dispatcher. (b)–(f) do not apply: no schema member, vector, second reader, mutation row, request refusal or checkpoint member is added. (g) The design note's operation tables quote the kind's statements as they are in store_sqlite.go, and its sections 6 and 8 were checked against the code.

Checks

The installed development kit (jpdev 0.1.0 on this machine) rejects this repository's dev.json (its handover member) and has no --changed or push. I ran the repository's own runner under the same caps (the machine lock, ulimit -v 12000000, a timeout, a GOCACHE of the build's own) and pushed with the gh credential helper form AGENTS.md gives.

scripts/dev/check.sh --changed origin/main selected 11 of 15 steps:

== check summary (dev.json) ==
handover: boundary 2 paths, shared 2 paths; see AGENTS.md "Before you hand over"
PASS     local-paths              0s
PASS     agent-docs               0s
    sync-agent-docs: CLAUDE.md matches AGENTS.md
PASS     diff-check               0s
PASS     go-fmt                   0s
PASS     go-vet                   0s
PASS     build-pinned             1s
PASS     manual-gate              20s
    13 lines matched; last: ok  	github.com/Judgment-Pack/judgment-pack-runner/internal/runner	12.823s
FAIL     unit-tests               606s, exit 1
    panic: test timed out after 10m0s
NOT RUN  chain-tests
NOT RUN  build
NOT RUN  third-party-notices
== 7 passed, 1 failed, 0 skipped, 3 not run ==

unit-tests did not fail on a test. go test -race ./... reached go test's default 10-minute limit in internal/runner on this machine, with every test that had run passing. On CI, main's last run took 449 s and 517 s for this package. I ran the step once more with only -timeout 40m added to its go test -race ./... (a scratch copy of the script; the committed script is unchanged), and then the three steps that had not run:

ok  	github.com/Judgment-Pack/judgment-pack-runner/cmd/jpack-runner	90.445s
ok  	github.com/Judgment-Pack/judgment-pack-runner/internal/buildinfo	1.017s
ok  	github.com/Judgment-Pack/judgment-pack-runner/internal/cloudgoogle	1.034s
ok  	github.com/Judgment-Pack/judgment-pack-runner/internal/derivation	1.034s
ok  	github.com/Judgment-Pack/judgment-pack-runner/internal/runner	625.528s
store conformance: the sqlite and memory kinds ran and passed

== check summary (dev.json) ==
PASS     chain-tests              12s
PASS     build                    3s
PASS     third-party-notices      0s, python (Python 3.8.20)
== 3 passed, 0 failed, 0 skipped, 0 not run ==

Before that, with the pinned binaries, the targeted store tests (Restart|Durab|Idempot|Chain|Journal|Interrupt|Queue|Dispatch|Completion|Append|Release|Manual|Store) passed. The conformance suite alone takes about 5 s under -race.

🤖 Generated with Claude Code

[kikashy]

Review record: runner PR #179 (issue #173), the store behind an interface

Clean-room review at a359150 (base 28b0c81). Same-vendor review, not a cross-vendor claim. No HIGH, no MEDIUM. Both stated behaviour differences are unobservable today and are documented in design note §8. Statements, schema, migration and transaction boundaries are unchanged. 7 probes pass on both kinds, and 18 mutants are caught by the conformance suite or by the existing tests.

Dispositions

#	Finding	Sev	Evidence	What to do
1	(a) A claim, or an end in the queue, now needs the run still queued. Nothing can observe this today. No path puts a run back to queued: runs are queued only at creation (store.go:522), and the only UPDATE runs is putRun. Crash recovery (restart) ends a left-running run interrupted (anyState) and never re-queues it, so no legitimate re-claim exists to refuse. One worker per store runs under the flock, restart runs before the worker starts, and no other writer changes a run's state.	none	grep of UPDATE runs and of State = "queued". Probes P2, P3 and P5	none; documented in §4 and §8
2	(b) Close releases the attempts directory before the store, and the lock still goes last. Nothing is lost or leaked: every write commits (WAL, synchronous FULL) before the releases start, and the kernel drops the flock and the fds on death. The Open error path keeps the base order (db, lock, attempts).	none	Probe P7: a child killed between the two releases leaves the journal [journal.began runner.started runner.stopped runner.started], and the next Open succeeds	none; documented in §8
3	§7 says a failed commit "keeps nothing", but the contract comment on faultDurability says it "may or may not have kept it". The suite proves "keeps nothing" only through the seam, which never calls the real COMMIT.	LOW	store_contract.go and §7. P4: a real COMMIT failure (deferred FK) kept nothing, because the driver rolls back on a failed commit	Align §7 with the contract: the caller takes nothing as kept
4	§8 does not say what a conflict does to the Runner. It stops the dispatcher like any storage error, and health goes false. That is fail-closed, and the PR body says so.	LOW	store.go worker returns on any error and sets unhealthy	Add one sentence to §8
5	The conformance suite does not pin a Runner function's transaction boundary. Splitting the claim's state write and its journal entry into two updates survives it. The existing SQLite-only TestAFailedChangeLeavesNeitherItsStateNorItsEntry catches the split.	LOW (info)	Own mutant M8b	none now; keep the events test when a second kind ships

Probes (scratch copy, go test -race, both kinds unless noted): all pass

• P1: a duplicate createRun by id, inside one tx and across txs, and by request key returns taken. The failed tx keeps nothing, and the same key under another caller is accepted.
• P2: a stale claim (run read queued, claimed, ended, then claimed again) returns conflict and writes nothing. putRun with the wrong expected state returns conflict, and an absent id with an expected state returns absent.
• P3: 30 rounds of 8 goroutines racing to claim one queued run: exactly 1 claim each round, the rest conflict, and 30 run.started entries.
• P4: a commit failure. On SQLite a real one (a deferred FK fails at COMMIT) gives durability with "FOREIGN KEY constraint failed (787)". On memory the seam gives the same. Neither keeps the claim, the release or a journal entry, and the next claim succeeds.
• P5 (SQLite): a child process claims a run, then is killed with SIGKILL. While the child lives, a second process gets "another runner owns this workspace". After the kill, the claim is durable (running, same StartedAt), runningRuns holds it, and the queue head is still in order.
• P6: 60 chain entries made in a shuffled order read back 1..60 through chainPage(after, 7), and lastEntry is 60.
• P7 (Service): the process dies between Close's two releases (see Run Jobs from durable schedules, events, and Google Cloud signals #2).

Mutants (scratch copy, -race -run ^TestStoreKindsConform$): all caught but M8b-conformance

Builder's mutants re-run: m1 sqlite version check, m14 memory version check, m2 memory lock (DATA RACE), m3 claim at any state, m4 memory create-once (both the key and the id variants), m5 sqlite taken typing, sqlite INSERT OR IGNORE (variant), m6 sqlite durability, m7 memory durability, m10 LOCK_SH, m8 queue by id, m9 running by id: all caught.
Own mutants: M7a memory queue newest-first: caught. M8a memory journal kept outside the tx: caught (fails-keeps-nothing, commit-fails). M8b claim plus its entry split into two txs: survives conformance, caught by TestAFailedChangeLeavesNeitherItsStateNorItsEntry.
CI gate: with the memory kind renamed, unit-tests.sh's gate fails ("TestStoreKindsConform/memory did not run and pass"). Unmutated, it passes.

Verified unchanged

• SQL: every SQL literal in the package's non-test files, extracted on both sides, gives the same distinct set; only the counts drop, where reads are now shared. The SQLite kind holds 23 distinct statements. The pragma/schema block is byte-identical (sha256 cb94e4fb…), and migrate is identical to migrateStore except for s.db becoming k.db. The schema constants are untouched.
• Transactions: submission (key, job, release, manual, mapping, count, insert, entry), claim, end in queue, finish (lastEntry, chain insert, sequence, state, update, entry), restart (started, running, interruptions, occurrences through sqliteTxOf), release and stop each stay one transaction with the same statement order. The replay still ends without a commit (errKeepNothing). The mapping-v2 pre-read stays outside a tx.
• Opening order: lock, working dirs plus attempts root, db file create/private, sql.Open, one connection, manual check, schema, identity, migrate. The driver rolls back a failed COMMIT (modernc v1.59.0 tx.go).
• Boundary: *sql.DB/*sql.Tx appear outside the kind only in the categories §6 lists (triggers, occurrences, jobs create/revise, briefs, workflows, manual admission, metadata, listings, journal reads/writes, source worker, relocation, manual preparation). http.go no longer imports database/sql. cfg.Dir reaches the store only via openSQLiteStore/sqlitePreparedStore; its other uses are the working-directory ones §5 names.
• Error text: wrapped errors keep their cause's text, and errors.Is(sql.ErrNoRows) still holds on the sqlite kind. No == sql.ErrNoRows comparisons exist.
• Suite: the whole internal/runner suite ran once under -race and passed. 14 manual/executor tests first failed only because the binary was invoked by a relative os.Args[0]; they passed when re-run by an absolute path. go vet ./... and gofmt are clean, and CI is green on Linux and macOS.
• Public text: no machine-local paths, personal names or organizations in the diff or the design note.

merge: yes