feat(plugin): optimize-skill's execution track, measured end to end — plus the skill_triggered and --split fixes it surfaced

.claude/decisions/2026-08-20-anti-memorization-and-search.md

@@ -0,0 +1,40 @@
+# The anti-memorization preflight, and what `search_compare` is not
+
+Subject: `optimize/search.py::candidate_leaks`, `skill_text`, `search_compare`,
+`leak_detection.py`.
+
+## Why the leak preflight must be handed a DIRECTORY, not a file
+
+`candidate_leaks` compares TEXT; `skill_text(skill_dir)` is the reader, and the CALLER owns which
+of the two it passes. That split is why the obligation has to be stated rather than assumed: a skill
+is a directory, and handed `SKILL.md` alone the preflight comes back CLEAN for a candidate that
+bundled train-row content into `scripts/` or a reference file — byte-identical to a genuinely clean
+result, which is the worst shape a preflight can have.
+
+## Why it shares its primitive with CE061 rather than reimplementing it
+
+`LEAK_LOCATOR_FIELDS`, `LEAK_MIN_CHARS`, `string_leaves` and `graded_strings` are ONE declaration
+with TWO consumers pointing in opposite directions: CE061 asks whether a dataset row's PROMPT
+contains a value a criterion grades it on; `candidate_leaks` asks whether a candidate SKILL.md newly
+contains train-row content. A second copy would agree on ordinary input and diverge exactly where
+either was written for.
+
+They differ in one behaviour, and it is a parameter rather than a fork: `graded_strings(drop_type=)`.
+CE061 keeps the discriminator — a PROMPT saying "skill_triggered" is worth flagging — while
+`candidate_leaks` drops it, because a skill BODY discussing eval criteria names types legitimately.
+
+The primitive lives in its own module rather than on `optimize.gate` because a task-lint rule
+importing from the optimize gate inverts the dependency, the same separation `pricing.py` and
+`path_utils.py` already have.
+
+## `search_compare` is emphatically not a gate
+
+It is an accept/revert decision inside a search, over one arm at a time, and it does not correct for
+multiplicity. Calling it a gate invites a reader to promote on it, which is the one thing it cannot
+support: the gates are `activation_gate` and `execution_gate`, and both go through Holm.
+
+## A hole is absent, never zero
+
+Every front and every comparison here treats a missing measurement as missing. Folding it to 0.0
+makes an arm that failed to produce a number look worse than one that produced a bad one, which
+inverts the ranking rather than merely biasing it.

.claude/decisions/2026-08-20-instrument-provenance.md

@@ -0,0 +1,56 @@
+# Instrument provenance: what the two fingerprints cover, and why there are two
+
+Subject: `suite_fingerprint.py`, `models/optimize.py::RoundScores.suite_fingerprint` /
+`grader_fingerprint`, `optimize/store.py::suite_changed` / `grader_changed`.
+
+## Why a second fingerprint was needed
+
+The grader fingerprint covers the outcome track's script and answer key. It cannot see a `weight`
+change that re-blends `weighted_score` — the very number the execution gate's paired *t* compares —
+and the activation track has no script grader at all, so it had NO instrument provenance of any kind.
+A round's numbers were comparable across rounds only by assumption.
+
+## The ROWS are load-bearing, not decoration
+
+`activation.yaml` is `initial_prompt: ${row.prompt}` with `expected_skill: "${row.expected_skill}"`,
+so every prompt AND every label lives in the rows file. A digest over row IDS alone — which is what
+the first implementation did — is blind to a rewritten prompt and a flipped label: the commonest
+suite edit there is, on the track this digest is the sole provenance for.
+
+## A DENYLIST, not an allowlist, and this is the one place that is the safe direction
+
+`scoring_dump` minus `_NOT_SCORING_RELEVANT` (a reason-carrying denylist of exactly one field,
+`description`). An allowlist over `BaseSuccessCriterion`'s five fields silently omits every subclass
+parameter — which is how the first draft could not see `run_command.command` move from `verify.py` to
+`verify2.py`.
+
+## `run_limits` is hashed WHOLE
+
+Rather than through four curated caps: the three token caps abort a run exactly as `max_usd` does,
+and `stop_early` is the kill switch for every armed criterion, so it moves `f1.yes` itself.
+
+## Length-prefixing and section tags are REDUNDANCY here
+
+Both are present, as in `verify.py::fingerprint`, but `_canonical` — canonical JSON per part — is
+what actually stops a value forging a delimiter or a part migrating between sections. A mutation test
+proved the obvious attribution wrong.
+
+## Order sensitivity is deliberate and asymmetric
+
+Order-SENSITIVE across criteria, because `criterion_index` is positional everywhere in this family.
+Order-INSENSITIVE within a mapping and across rows.
+
+## What it excludes, and the one boundary that follows
+
+The TASK-LEVEL agent and sandbox blocks, so two checkouts agree. One stated consequence: an
+`agent_judge` criterion embeds its own agent config and is hashed whole — which is right, since the
+judge's model is part of what it measures, and which makes such a suite's digest machine-local.
+
+## Digest only, never the pre-image
+
+`measurements.json` is committed.
+
+## Three-valued, both of them
+
+`suite_changed` and `grader_changed` return "changed" / "unchanged" / "cannot tell" rather than a
+bool. A round with no recorded fingerprint is not a round whose instrument matched.

.claude/decisions/2026-08-20-stage-c-confirmation.md

@@ -0,0 +1,51 @@
+# Stage C: did the Stage B effect reproduce?
+
+Subject: `optimize/gate.py::classify_confirm`, `build_confirm_verdict`, `confirm_split_check`,
+`confirm_one_candidate`, `optimize/activation.py::confirm_gate`,
+`optimize/execution.py::confirm_gate_execution`, `optimize/activation.py::gate_seed_stability`.
+
+## A family of ONE, and that is correct
+
+Only the Stage B winner is confirmed, so there is no multiplicity to correct. A reader who expects
+Holm here is looking for a correction over hypotheses that were never tested. Holm is still applied
+at `m = 1`, purely so the carried block is a DECIDED one rather than rendering as `UNDECIDED`.
+
+## The train effect is READ, never recomputed
+
+It comes off the Stage B verdict, so the two numbers the block compares cannot disagree with the
+blocks they were reported in.
+
+## The margin is the confirm split's OWN MDE
+
+Which is what makes the rule per-track without a second declaration: each track passes its own
+gate's floor, on its own metric. Picking a different multiple on one track would be a second
+declaration of "how much shrinkage is real". A floor of `None` or 0.0 leaves the margin UNDEFINED
+and the outcome is `undecided` rather than silently SHRANK — 0.000 means the floor could not be
+priced, never that the suite can resolve anything.
+
+## Why the split check is shared and must not be an if/elif over the collapsed value
+
+`SplitProvenance.value` collapses to `UNRECORDED_SPLIT` when ANY pooled dir is unreadable. So a chain
+reading `if unrecorded: note / elif value != "test": refuse` drops the refusal entirely for three
+dirs recording `train` beside one unreadable `run.json` — and the confirm then classifies over TRAIN
+rows carrying only a "provenance is missing from 1 of 4" note. That is precisely the failure the
+refusal exists for.
+
+The execution twin takes ONE run dir and cannot reach that state, which is exactly why the rule may
+not live on each track separately: the safe one would keep working while the other drifted. The
+activation side had already gained a "not the Stage B winner" note the execution side lacked, while
+its docstring claimed both worked "for the reasons the execution twin's docstring gives".
+
+## A recorded `train` is a REFUSAL; an unrecorded split is a NOTE
+
+A recorded `train` means Stage C silently re-ran the train rows, at full price, with no error
+anywhere — an effect reproduces on its own training data by construction. An unrecorded split is a
+run predating the field.
+
+## Seed stability carries no single `promoted` field
+
+Collapsing three disagreeing seeds into one verdict is the exact thing it exists to prevent: a
+decision that flips with the seed is a coin flip, and reporting the majority's answer as *the* answer
+hides that. Its `promote_agreement` counts promotions at a family of ONE, which is not the round's
+decision when the round gated more than one candidate — stated because the number invites that
+reading.

.claude/decisions/2026-08-20-the-advisory-fronts.md

@@ -0,0 +1,53 @@
+# The three fronts, and why only one of them is a shortlist
+
+Subject: `optimize/fronts.py::arm_row_scores`, `pareto_front`, `instance_best_front`,
+`cost_quality_front`, `cost_quality_points`, `headroom_ceiling`.
+
+## Three fronts, three different jobs
+
+- **Pareto** is a DISCARD rule. An arm dominated on every axis cannot be the answer, so dropping it
+  costs nothing.
+- **Instance-best** is a MERGE shortlist: the arms that win at least one row. A candidate that wins
+  nowhere has nothing to contribute to a merge.
+- **Cost/quality is ADVISORY only.** It is a 2-D Pareto filter over (quality, cost) — the arms
+  nothing beats on both — returned in INPUT order, with no ranking inside the front. It is NOT a
+  ratio: a ratio has no defensible threshold, and inventing an order here is exactly what would turn
+  an advisory into a shortlist, inviting promotion of the cheapest arm that happens to score.
+
+Reading the wrong front as the others is the failure this separation exists to prevent, and it is a
+reading error rather than an arithmetic one, so the names carry the distinction.
+
+## A hole is absent, never zero — on all three
+
+An arm with no measurement on a row is missing there. Folding the hole to 0.0 makes an arm that
+failed to produce a number rank below one that produced a bad number, which inverts the front rather
+than biasing it. `is not None` rather than truthiness, because a free model is legitimately the
+cheapest arm and `0.0` is a real cost.
+
+## Domination is gated on row-set COVERAGE, and that conjunct is load-bearing
+
+Without `row_ids <= other_ids`, an arm measured on a SUBSET can dominate one measured on more rows.
+Measured: an arm that crashed 5 of 6 rows and scored 1.0 on the sixth knocked the incumbent off the
+front. A count cannot express this — two arms on four disjoint rows each would both look entitled to
+dominate the other.
+
+## `headroom_ceiling` bounds what is left to win, not what was won
+
+It answers "is another round worth paying for?" for ONE arm. Read as a score it is meaningless; read
+as a ceiling it is the only number in the family that can say *stop*.
+
+Two things about it were deleted once by an over-eager docstring trim and are the reason this section
+exists. The ceiling's denominator is the arm's FULL finite row count and never the selected subset —
+a rule failing 3 of 15 rows has a ceiling of 0.1, and dividing by the subset overstates it 5x, which
+makes every rule look promotable. And `rows=None` (every row) is a different question from
+`rows=set()` (an empty selection), which matters because `rule_row_map` OMITS a rule that failed
+nowhere: passing that missing entry as `None` reports the whole suite's ceiling under that rule's
+name.
+
+## A contaminated tree WARNS here rather than refusing
+
+The return types are vectors and fronts with nowhere to put a refusal, and the caller is a human
+reading a Stage A table rather than a gate deciding a promotion. So `arm_row_scores` reconciles and
+logs. That is a deliberate asymmetry with the two gates, and CE053 is what keeps the reconciliation
+from being dropped altogether: measured, without it `arm_row_scores` returned a stale row in its
+vector and all three fronts were computed over it.

.claude/decisions/2026-08-20-the-execution-gate-refusals.md

@@ -0,0 +1,98 @@
+# The execution gate's refusal causes, and why their ORDER is the rule
+
+Subject: `optimize/execution.py::execution_gate`, `_execution_diagnostics`, `_refuse_*`,
+`_below_mde_findings`, `models/optimize.py::ExecutionGateVerdict.gate_refusal`.
+
+## First cause wins, and program order IS the precedence
+
+Every cause answers the same question — *is this a result?* — with the same consequence, so they
+share one field, one headline and one prose token. They differ in REMEDY, and a later cause is
+usually an earlier one's consequence: if there was no comparison to make, the rows are moot; if the
+rows never loaded, whether their differences vary is moot. So the earliest cause is the one whose
+remedy comes first, and routing every setter through one sink says that once instead of leaving
+eleven `if gate_refusal is None` guards to be kept in agreement.
+
+The concrete case: a mistyped variant id makes that arm load ZERO rows as a consequence. Refusing on
+the consequence replaces a message naming the two ids the experiment actually carries with one that
+can only say "a wrong variant id, a wrong suite id or a wrong run directory".
+
+## Why an arm with no rows is a refusal rather than a note
+
+This track's statistic comes from `experiment.json`, not from the row tree — so it computes
+perfectly well over rows that are not on disk, while every guardrail and integrity check reads green
+over nothing. A valid experiment file beside a mistyped path renders as PROMOTED with every check a
+green `— -> —`.
+
+## The below-MDE refusal is deliberately TWO-SIDED
+
+`mde` is the half-width of a bootstrap interval on a NULL difference, so a difference under it is
+indistinguishable from the suite's own run-to-run noise however small the p is. But under the null a
+candidate's difference is ALSO small: `abs(mean_diff) < mde` is true for nearly every candidate that
+simply does not work — measured, 40 of 40 true-null candidates. Refusing all of them would retire
+NOT PROMOTED almost entirely and send the reader to buy replicates for a candidate whose problem is
+that it is null.
+
+So the refusal is conditioned on the interval EXCLUDING zero. An interval that contains zero is the
+data agreeing the candidate is null: an ordinary negative result, and it stays one. What is left for
+the refusal is the pathology — a confident claim, in either direction, about an effect the
+instrument cannot see.
+
+## Zero variance splits into two messages
+
+At a constant difference of ZERO the arms behaved identically, which is a finding about the
+candidate that no number of extra rows can change, and `paired_t_test` reports p = 1.0 there rather
+than the 0.0 a non-zero constant shift gives. One message would state a p the block below it
+contradicts. The same split, for the same reason, as `holm_promote`'s `p_floor >= 1.0` branch.
+
+## The interval-tighter-than-floor case is a caveat, NOT a refusal
+
+The paired *t*'s interval comes from the BETWEEN-ROW spread of the differences, which is tiny
+whenever the arms differ by a similar amount on every row, while `mde` measures WITHIN-row noise the
+*t* never sees. So a real, large, consistent win reports an absurd p. Refusing it would be worse
+than the defect: measured, a genuine 8-row 0.30 win reports a half-width of 0.007, the same shape as
+the 0.400-on-every-row case. What is wrong there is the reported PRECISION, not the decision.
+
+## Every note is suppressed under a refusal
+
+A refusal says the comparison decided nothing; a note beneath it is a second, contradictory claim on
+a page a user pastes into a promotion ledger. The below-MDE note calls itself "an ordinary negative
+result", and it was the one rung that fired regardless — reproduced through the real gate, a
+zero-variance refusal printed it beneath `NOT A RESULT`. `promoted` was unaffected, so this was
+prose only.
+
+`refused_already` is OR-ed with the local cause because "nothing has refused yet" has to include
+what the diagnostics themselves decided three lines up. TWO paths arrive already refused and neither
+returns early: the stale-tree cause and the primary-index cause.
+
+## The primary-index refusal must be recorded BEFORE the diagnostics run
+
+That ordering is load-bearing, not tidy. Recorded after, it produced a `NOT A RESULT —
+primary_criterion_index=7 selected no usable row` headline above notes reading "this is an ordinary
+negative result and not a measurement problem" and "the paired interval is tighter than this suite's
+own noise floor" — measured.
+
+`require_valid_criterion_index` bounds only BELOW, deliberately, since rows may legitimately differ
+in criteria count and an over-long index should skip a row rather than raise. That is the wrong
+answer here: an over-long primary index makes `row_score` return `None` on every row, so the vector
+is EMPTY and indistinguishable from a suite whose rows all errored on that criterion.
+
+## Dead weight is a READING and can never gate
+
+Measured rather than argued: a constant criterion scales the paired difference vector without
+changing its shape, so it scales the mean AND the standard deviation by the same factor — the paired
+*t* is identical to 1e-12 between the grader-only and blended scales while the mean difference scales
+by 1/2.05. The bootstrap interval scales with the data, the MDE is measured on the same blended
+scale, and the guardrails never touch the blend, so EVERY conjunct of `promoted` is invariant to it.
+Wiring it into `integrity_checks` would force `promoted = False` on comparisons that are
+statistically sound — strictly worse than the presentational problem it would be fixing. The one
+case where dead weight genuinely invalidates a comparison is every criterion being constant, which
+is already the zero-variance refusal.
+
+## A stale tree FLIPS the answer rather than merely being reported
+
+`run.json` is written per INVOCATION while the tree is APPEND-ONLY, so a re-used `--run-dir` leaves
+an earlier call's rows — or, with a smaller `--repeats`, its replicates — on disk, and they are
+pooled into the comparison and into the checks that gate it. Measured on an identical winning
+candidate: four unrecorded incumbent replicates moved `completion_rate` from 1.0 to 0.667 and
+`promoted` from True to False, with no refusal and no note. Contaminate the candidate arm instead
+and the error runs the other way.