[opencdlee-dotcom]

Step S0 of the precision plan: the ground-truth harness every later step is scored against.

What

aegis.py backtest replay [--days N] [--reobserve] re-runs the recorded findings of the last N days (default 30) through the current pipeline in a throwaway :memory: store and scores the result against the incidents the operator closed as noise.

• Read-only. The live store is opened mode=ro in one read transaction. _event_connection is never called because it creates, migrates and chmods. _custody_remember is a no-op for the run, and no seen, sigcache or findings ledger is written.

• Same pipeline. Batches are grouped by observed_at, because no writer sets events.scan_id. Each batch goes through:

  • route_findings, using the live suppression memory with the learning period forced OFF and a replay-local seen ledger kept the way emit() keeps it
  • _record_finding_events, the scan's own record-and-fold, extracted from record_security_state
  • _apply_correlations: chains, lineage, risk and incidents

  Risk accumulation receives the live per-category dismissal weights.

• Scored. The report shows:

  • interrupts per category and the cases left open
  • every FALSE_POSITIVE incident, whatever its resolution, whose evidence would open an interrupt again, listed by id with title and path (noise re-opened: N of M)
  • open cases with no noise-labelled evidence (new interrupts from corpus: K)

• Recall. The findings built by the 21 assay lanes go through the same gate (assay recall: X/21). Predicate-only lanes and quarantine-roundtrip, which writes live state, are named in the output rather than skipped.

• --reobserve asks classify_signature and _grade_binary again about every process, listener, beacon or outbound subject still on disk, and re-derives severity from that sensor's own gate. That makes classifier and custody-ladder fixes scoreable, not only routing fixes.

• Rule 17. Loaded rows, batching, the noise set (compared with a direct join query), routes and assay rows are all checked against the store. A failed check prints at the top and exits 1.

The scan-tail closers are deliberately not run. They key on what a scan re-asserted, but the event log only holds what the live fold chose to record.

Supporting changes, with no change to scan behaviour:

• record_security_state's record-and-fold loop moved into _record_finding_events. The fold query now selects the incident id instead of 1.
• route_findings(seen=None) gained an optional ledger. When it is not given, the function reads SEEN as before.

Baseline (reference Mac, main 49e38b3 plus this PR, last 30 days)

62,604 findings in 2,347 scans; 213 noise-labelled incidents with evidence in the window.

run	noise re-opened	new interrupts from corpus	assay recall
plain	170 of 213	12	9/21 (11 predicate-only, 1 not run)
--reobserve	135 of 213	12	9/21

--reobserve re-observed 7,158 findings. For 422 of them the trust verdict changed; the largest group is 305 listener findings on Spotify that moved from unsigned to developer-id. For 2,091 the custody rung changed (actions-runner Listener → operator-vouched), and 117 would no longer be emitted.

Verification

• python3 -m pytest tests/test_backtest_replay.py -q: 11 passed. The learning-OFF and no-custody-write tests were mutation-checked; each fails once its override is removed.
• Full suite: python3 -m pytest tests/ -q -x passed on this Mac with 2082 passed, 6 skipped and 30 xfailed in 15m07s. After that run, the test file alone gained one platform-neutral stub (_exec_alert held open in the custody-write test) and passes 11/11 natively and under SIM_BODY=win|linux|mac.
• On the live store, the sha256 of custody.jsonl, sigcache.json and seen.json was identical before and after the --reobserve run.

🤖 Generated with Claude Code