feat(mutants): RQ-65-MUTANTS (#1189) — mutation survey: 4 of 21 byte-changing mutants survive a named suite (19 %, upper bound); 3 controls KILLED; survivors enumerated; ledger CI-pinned

[avrabe]

RQ-65-MUTANTS (#1189) — how much of the code generator is correct by accident, measured by mutation

The number: of the sampled emitted-code decisions that change bytes when flipped, 4 of 21 survived the named suite — 19 % survival, an upper bound under the full CI board (a broader suite cannot un-kill a mutant). 17 of 21 were killed: 6 by execution differentials, 8 by unit/integration tests, 2 only by frozen-byte goldens, 1 by the compiler hanging. Every survivor is enumerated with its exact diff in docs/status/MUTATION_SURVEY.md; that list is what v0.66 is scoped from.

Red-first controls — all KILLED (run before the sample, same pipeline)

control	corpus objects changed	killed by
the #1189 aliasing copy disabled (if !live_home { → if true {, the pre-fix behaviour byte for byte)	7	execution: join-alias-1189-oracle, 18 s
the direct selector's rule_i32_select operands swapped (PR #1216's plant)	15	execution: cmp-select-oracle, 18 s
the startup R10 (linear-memory size) seed written into R9	325 (every self-contained image)	structure only: the base_cse_flip_468 / const_cse_reduction_242 byte goldens after 274 s of cargo test — no execution oracle in the suite boots the shipped Reset_Handler

A control that does not kill exits the harness with 1 and publishes no rate.

Frame, sample, suite — stated, not invented

• Frame: five regions ranked by RQ-65-PARITY (RQ-65-PARITY (#197): selector parity oracle — optimized vs direct vs wasmtime over the spec corpus; 11 findings, #1203 fixed, the rest pinned red-first #1216), located by function anchor: R1 routing (has_value_carrying_branch + compile_wasm_to_arm), R2 ir_to_arm_impl, R3 select_with_stack, R4 the shared post-merge tail (measured: the two selectors share no lowering code inside synth-synthesis; what they share is finish_allocated_stream/branch resolution/validation, the liveness.rs passes and the encoder's i64_effective_base), R5 generate_minimal_startup. 1,538 candidate sites.
• Operators: Reg::Rn→R((n+1) mod 13); Condition::X→inverse; a push(… ArmOp::Mov …) deleted; immediates +1/+4, startup register numbers +1; if COND {→if !(COND) { and negated ||/&& continuation terms; <↔<=, >↔>=, .len() - 1→.len(). Seeded (1189), stratified per region and operator, round-robin so any prefix is balanced.
• Sample: 38 drawn — 6 uncompilable (excluded), 32 compiled, 21 byte-changing over 200 corpus modules × 3 ARM configurations (--relocatable, self-contained, self-contained --no-optimize; 600 pairs hashed on .text/.data/.rodata + symtab). Target was 8 compiled per region; the run stopped at draw 38 when that mutant hung the compiler — R2 reached 6 compiled, R4 3 (four of R4's draws were uncompilable). Said plainly in the report.
• Suite, derived from ci.yml by the harness (each selected job's own run: steps executed verbatim, binary path substituted): L1 = wast-conformance-oracle, repro-sweep-arm-corpus-oracle, join-alias-1189-oracle, cmp-select-oracle, frame-slot-dce-242-oracle (12 steps, 23 s green on the unmutated binary, 0 red on baseline); L2 = the test job's cargo commands (274 s green). Left out with the measured reason: selector-parity-oracle — the coordinator's first suggestion and the strongest selector kill signal — takes >20 min per run here the CI way (single-threaded under oracle_run.py); the other 28 ARM oracle jobs are affordable singly, not per mutant; RV32/AArch64/Mach-O jobs are other crates; instrument-independence rebuilds the tree; claim-check/rivet are documentation gates. All recorded in the ledger's suite.unselected_jobs / suite.excluded_jobs.
• Triage before oracles: byte-identical mutants get no oracle run; a reach probe (the original token rebuilt inside { eprintln!(MARK); token }) tells EQUIVALENT (evaluated, byte-neutral) from DEAD (never evaluated on the corpus — a deletion candidate, not a proof).

Survivors, classified

	count
UNTESTED	4	ir_to_arm epilogue Mov dropped (8 objects); the direct selector's cmn-residual I32Eq→NE (2 objects — the gap #1216 predicted); R2 removed from the AAPCS dead-at-return set in liveness.rs (49 objects, conservative but unnoticed); the startup ROM→RAM data-copy count register moved (46 objects; self_contained_data_758, outside the subset, would see it)
DEAD	4	the VFP / frame-growth retry rungs and the literals arbiter in arm_backend.rs — unreached on the corpus; deletion candidates for the subtraction ratchet
EQUIVALENT	5	boundary values the corpus never hits
UNRESOLVED	2	declared, outside the denominator: a const array (no runtime probe); a < inside a multi-line format string (not a decision site — a sampling defect, recorded)

Per region: R1 0/2, R2 1/5, R3 1/6, R4 1/1, R5 1/7. All six R5 kills fell only to the structural layer.

The CI pin — the instrument's discrimination, not a re-run

• mutation-survey-discrimination job: mutation_survey.py ci replays the ledger's ci_subset — the 3 controls (their recorded killer step must go red) + 2 UNTESTED + 1 EQUIVALENT + 1 DEAD (their byte-triage set / reach verdict must reproduce) — and greps MUTANTS-CI subset=N controls=N non-killed=N failures=0. Verified locally on the rebased tree: subset=7 controls=3 non-killed=4 failures=0. It replays only already-declared scripts/repro oracles: no new # ci-checks: declaration, exact floor (re-derived: 345122) untouched.
• claims.yaml SYNTH-MUTATION-SURVEY-RQ65: the eighth kind: ratchet — mutants_untested (value 4, baseline 4, must FALL) over docs/status/mutation_survey.json; count-eq pins on 38 / 21 / 4, control_ok ×3 and the ci_subset sides (3 + 4); SYNTH-SUBTRACTION-PINS-DECLARED moved 7→8 / ceilings 4→5 with CLAUDE.md saying "Eight". artifacts/status.json regenerated.

Harness defects found and fixed while surveying (recorded in the report)

"mentions scripts/repro/" as the job-selection rule admitted coverage (a cargo llvm-cov of the workspace), claim-check and rivet → replaced by "invokes an oracle"; a 300 s per-compile timeout with no short-circuit let one hanging mutant burn >20 min and crash the run → 60 s and stop-at-first-hang, hangs classified KILLED/timeout; a killed run left a mutated selector on disk → SIGTERM now unwinds the restore; cargo's Running …/deps/ line shape was missed → structural kills re-attributed from the recorded test names (which is how the two freeze-only kills became visible).

Gates on this tree

cargo fmt --check 0 · cargo clippy --workspace --all-targets -- -D warnings 0 · status_evidence_check.py 0 (floor restated 0 times) · claim_check.py claims.yaml 64/64, mutants_untested 4/4 must FALL · oracle_wiring_check.py --exact-emulation-floor 345122 0 · mutation_survey.py ci failures=0 · rivet validate: exit 1 with 40 pre-existing sigil:* cross-repo dangling refs, 0 naming this artifact (the federated externals are not resolvable locally).

Refs #1189, #197, #1216

🤖 Generated with Claude Code

https://claude.ai/code/session_01YJK5LZZEkV5smCY1jKn18L

[avrabe]

Coordinator verification — the CI job re-runs the mutants; my first read was wrong

I went looking for the failure mode this release exists to find: a mutation-survey gate that only
re-reads its own committed ledger and therefore cannot fail. A keyword scan of cmd_ci showed 0
matches for apply_mutation, cargo, subprocess, run_suite, and I was ready to call it vacuous.

That read was wrong, and reading the function settled it. The code uses different vocabulary:
mutations are applied through an Edit(site, "after") context manager, the rebuild is build_synth(),
the oracle invocation is run_step(steps[0]). It genuinely applies the mutation, rebuilds, recompiles
the 200-module corpus, byte-diffs against the baseline, runs the recorded killer step, and asserts the
verdict:

if verdict != "KILLED":
    failures.append(f"{sid}: a mutation the ledger records as KILLED now SURVIVES its killer — the oracle lost its power")

Four properties that make this a gate rather than a record, all present:

• assert git_clean(site["file"]) after each mutant — the tree is proven restored, not assumed.
• if len(subset) < 4: sys.exit("VACUOUS: ...") — its own population floor.
• a check that the recorded killer step still exists in ci.yml, so deleting the oracle that kills a
  mutant fails the job rather than silently passing it.
• ledger drift handled honestly: if the live unmutated corpus differs from the recorded baseline it
  says so and compares byte-triage against the live baseline, rather than reporting a stale diff.

The ci.yml grep requires subset>=4 controls>=2 non-killed>=2 failures=0, so it is non-vacuous in both
directions — controls must still die AND survivors must still reproduce.

One small brittleness, not blocking. The grep's numeric classes are [4-9][0-9]* / [2-9][0-9]*,
which match 4-9 and 40-99 but not 10-39. Growing ci_subset from 7 to, say, 12 would red this job
for no real reason. It fails in the safe direction (a false red, immediately visible), so it is worth a
follow-up rather than a block — but it will bite whoever next widens the subset.

On the number. 4 of 21 byte-changing mutants survive, 19 %, relative to a named 5-job suite plus
the workspace tests, with the parity oracle excluded for cost and the rate correctly framed as an
upper bound — more oracles can only kill more. The control that most deserves attention is the
third: seeding R10 into R9 in the startup blob was caught by byte goldens only, because no
execution oracle in the named suite boots the shipped Reset_Handler. PARITY builds exactly that
capability, which is a concrete argument for putting it in the suite when the cost allows.

Refs #1189

[avrabe]

Coordinator: the third control's finding is a systematic blind spot, not an incidental one

The survey reports that the startup-r10-seeded-into-r9 control was killed by byte goldens only,
"no execution oracle in the suite boots the shipped Reset_Handler". I checked the five named-suite
oracles directly, and the shape is stronger than "the suite happens not to include one":

oracle	boots Reset_Handler	seeds R11 itself
wast_conformance_928_differential.py	0	0
arm_corpus_sweep_973.py	0	2
join_alias_1189_differential.py	0	2
cmp_select_two_move_differential.py	0	2
frame_slot_dce_differential.py	0	2

Four of the five supply the register contract the startup blob exists to establish. An oracle that
writes R11 before executing cannot observe a compiler that writes the wrong R11 — the property is
assumed by the harness, so it is untestable by construction rather than merely untested.

This is not hypothetical: it is exactly how #1203 survived. RQ-65-PARITY found it only because it
booted the shipped startup instead of seeding registers, and the lane's own note from that work reads
"boot the SHIPPED startup — harnesses seeding R11 themselves missed #1203".

Worth stating plainly in the report, because it changes what the survivor list means: the startup
region's mutants are not "covered by byte goldens" in the same sense as the others. A byte golden
detects that the bytes CHANGED; it cannot say the new bytes are wrong. For every other region an
execution differential can make that call, and for R5 nothing in this suite can.

Note the repo does have oracles that boot the real startup — cabi_arena_bind_418,
self_contained_data_758, call_indirect_275_selfcontained, brif_outer_740,
mem761_linmem_r9_overlap and others, plus the parity differential itself. They were excluded from the
named suite for cost, which is the honest and documented choice. The actionable follow-up is narrow:
adding ONE startup-booting oracle to the suite would move R5 from "byte-golden only" to genuinely
execution-covered, and the survey now says exactly which mutants that would affect.

Refs #1189, #1203, #197

[avrabe]

Coordinator: the ledger must be RE-ANCHORED on the rebase, not carried

#1232 (RQ-65-MVPCORE) merged and grew crates/synth-cli/src/main.rs from 11,143 to 11,405 lines.
The seven R5-startup mutants — and the CONTROL/startup-r10-seeded-into-r9 control — are anchored by
file:line:col, so on the rebased tree every one of them points at different code:

id anchor	on this branch	on main
main.rs:9369	encode_thumb2_movw(12, (w & …))	.with_binding(SymbolBinding::Global)
main.rs:9365	encode_thumb2_movw(9, (base…))	// Add symbols
main.rs:9359	if !globals_words.is_empty() {	.with_addr(flash_base)
main.rs:9340	encode_thumb2_movw(11, (lin…))	a comment
main.rs:9324	if enable_fpu {	}
main.rs:9295	encode_thumb2_movw(2, (data…))	// Build complete flash image
main.rs:9280	let src_patch_off: Option<usize> = …	// Set handlers …

cmd_ci fails loudly when a site has vanished ("site no longer exists on this tree — re-anchor the
ledger"), which is the right behaviour and will catch most of this. The case it cannot catch is the
one to worry about: a surviving file:line:col that still holds a mutable expression of the same
operator kind, where the replay would run a different mutation and report a verdict about code the
survey never measured.

So the rebase must re-derive the ledger, not carry it — the same rule this release applies to the
emulation floor and the ratchets, one level up: a ledger is a cumulative counter whose units are
sites. Re-run the enumeration on the rebased tree and re-anchor; if any R5 verdict changes, that is a
finding to publish, not a number to restore.

Two things this does NOT invalidate. The 19 % is relative to a stated frame and a stated commit,
and it remains a true measurement of that tree — it just needs re-anchoring to stay checkable on
this one. And the R5 control's finding stands independently: no execution oracle in the named suite
boots the shipped Reset_Handler, so those mutants are byte-golden-covered only. #1232 does not change
that.

Method note: my first two attempts at this comparison both came back "DIFFERS on every line" with an
empty column, because "$H:crates/…" triggers bash parameter expansion on ${H:…} and eats the
colon. Braces ("${H}:crates/…") fixed it. The real result above happens to agree with the broken
one, which is exactly why the broken one was worth catching.

Refs #1189, #1017

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[avrabe]

Coordinator verification — the published 19 % is sound; my attempted re-survey was the broken one

I tried to re-measure this survey on the merged tree and got 0 of 29 surviving. That number is an
artifact and is not in the release. Recording the whole episode because the near-miss is the useful
part.

Why my re-run was invalid — two independent environmental faults on this machine:

1. 147 of the 196 L1 steps invoke bare python, which does not exist here (only python3).
   Verified under the harness's own env: cargo_env inherits PATH, and
   bash -c 'python --version' returns exit 127 in ~0.1 s. The harness scores a non-zero oracle
   exit as a KILL, so 147 dead oracles read as 147 working ones.
2. Even with a python shim, fact_spec_div_494_differential.py --expect-decline is red on the
   UNMUTATED tree here (compiles=2 exit=1) while green in CI.

Together those made one cheap early step appear to kill every mutant — including all three controls.

What stopped it was this survey's own control check, not my judgement:

CONTROL/1189-copy-disabled came back KILLED (expected KILLED by join_alias_1189_differential.py)
1 control(s) not killed — the survey measures nothing; do not publish the rate

Requiring a control to die to its specific recorded oracle is what separates "every mutant was
caught" from "every oracle was dead". A gate that only checked that mutants died would have
certified the 0 %.

And the ledger you shipped is verifiably NOT affected. Its recorded killers are varied and
specific — 10 cargo test, 4 #989 local.set/tee WAR-aliasing execution differential, 1 ARM corpus
sweep, 1 #973 i64-cmp select, 1 compiler hang, and the three controls each dying to their own
oracle. Zero kills attributed to the divisor step that dominated my broken run. The oracles that
produced 19 % genuinely ran.

So the only change made here is the one that is environment-independent: changed-set 49 -> 53
for R4-shared/REG/liveness.rs:7025:36, re-derived by BYTE TRIAGE (cargo + synth only, no oracle
suite), agreeing with CI's own measurement. The site never moved; 13 of 609 baseline corpus entries
did, under #1232 and #1227. The classification is explicitly NOT re-verified and the report says so.

Follow-up worth having (not filed as a blocker): the harness cannot distinguish "the oracle failed
because the mutation broke something" from "the oracle failed because it could not run". Treating
exit 127 / command-not-found as a KILL biases a mutation survey toward optimism — the direction that
flatters. A cheap fix is to probe each step's interpreter once at baseline and refuse to start when
steps are unrunnable, rather than scoring them.

Refs #1189